Python2's urllib2 is a miserable libary

Python2's urllib2 is a miserable piece of shit of a library.  Use requests instead.

QA Notes

Testing

• Who is our largest QA audience?  Our users.  There's no escaping the fact that our users will find more bugs than our QA team.  Therefor it is in our best interest to capture what they are doing with logging and sane error messages.  There needs to be logging servers which handle the logs coming in from various environments (QA/Prod/Various different versions of prod) that allow us to grep back through and find out exactly what it was that the user was doing when he got the error. We can then replicate what he was doing in QA with debugging on, but if we are logging all error messages, this might not even be necessary.  Better exception handling is also a must.

Tests

• Unit tests - Tests developers write to make sure code isn't broken, no syntax errors, handles all edge cases etc.
• Acceptance Tests - Supposed to be written by Product Team (yes, they're supposed to be developers, they are supposed to have the good idea's for the product and understand how the features are going to work into the current code base), ensure that the code that the developer wrote is what the actually wanted in the feature
• Integration Tests - Test that the code works correctly in the entire stack, a replica as close as possible to production

More on Assertive Programming

(from The Pragmatic Programmer: From Journeyman to Master by Andy Hunt and David Thomas)

There is a common misunderstanding about assertions, promulgated by the people who write compilers and language environments. It goes something like this:

Assertions odd some overhead to code. Because they check for things that should never happen, they'll get triggered only by a bug in the code. Once the code has been tested and shipped, they are no longer needed, and should be turned off to make the code run faster. Assertions are a debugging facility.  

There are two patently wrong assumptions here. First, they assume that testing finds all the bugs. In reality, for any complex program you are unlikely to test even a miniscule percentage of the permutations your code will be put through (see Ruthless Testing). Second, the optimists are forgetting that your program runs in a dangerous world. During testing, rats probably won't gnaw through a communications cable, someone playing a game won't exhaust memory, and log files won't fill the hard drive. These things might happen when your program runs in a production environment. Your first line of defense is checking for any possible error, and your second is using assertions to try to detect those you've missed. Turning off assertions when you deliver a program to production is like crossing a high wire without a net because you once made it across in practice. There's dramatic value, but it's hard to get life insurance.

Python Utils

http://wiki.python.org/moin/PythonTestingToolsTaxonomy

Macs

You don't use a mac if you know what you're doing. You use a mac if you don't know what you're doing and you just want someone who does know what they're doing to configure a machine with all the basics set to the most popular settings.

The Last of the 8s (Strange Adventures in Math)

Today I learned that I am one of the last of the 8s... I was born in 1978 and this year has a unique property that 19+78 = 97 the middle two numbers of the year 1978. Nifty. What other years have this?

Well assuming that we're just looking for years between 1000 and 10,000, we can write some python code to figure that out:

for i in range(1000,10000):
a = i/100
b = i%100
c = (i-(i/1000)*1000)/10
if (a+b == c):
  print(i)

This produces something kinda cool... but that I don't really understand:

1208
1318
1428
1538
1648
1758
1868
1978
2307
2417
2527
2637
2747
2857
2967
3406
3516
3626
3736
3846
3956
4505
4615
4725
4835
4945
5604
5714
5824
5934
6703
6813
6923
7802
7912
8901

That's 8 years that end in 8, 7 that end in 7, 6 that end in 6 etc down to one year that ends in 1 (8901). But where are the 9's? They're in adding the first value and the last value together. 36 of them (9*4?).. There's some other patterns in there as well. I'm sure there's a whole dissertation in mathematics that's been written on this, but I'm not even sure where to start searching for more info about it (it's the Loblaw principle... named after it's founder, Bob Loblaw..)

Coach Johnny D's quote..

Our deepest fear is not that we are inadequate.
Our deepest fear is that we are powerful beyond measure.
It is our light, not our darkness, that most frightens us.
Your playing small does not serve the world.
There is nothing enlightened about shrinking so that other people won't feel insecure around you.
...We are all meant to shine as children do.
It's not just in some of us; it is in everyone.
And as we let our own lights shine, we unconsciously give other people permission to do the same.
As we are liberated from our own fear,
our presence automatically liberates others.

Marianne Williamson via Coach Johnny D

Epic Pub Crawl Remembered

Almost a year ago, we had an end of soccer party on a Saturday which turned into an epic pub crawl (my first ever). It was myself, JR, Jay, and Josh Jetson (seriously). We started drinking at Venice Beach Wines, then made our way to the beach where we got lunch. I remember seeing the smoke from the Station fire that was raging in the hills behind Pasadena while we were walking along the shore. From there we made our way back to Abbot-Kinney where we stopped at the Otheroom and hung out drinking various beers for about 5 hours. People we knew stopped by and eventually Enos showed up and we had a great time just chatting about random stuff. We were there so long that Jay took off and got some sandwiches for us all for dinner. From there we wandered over to Roosterfish, a gay bar with cheap drinks and got drunk on wiskey... we were basically there alone since it was still early in the evening (9pm). After that we continued on to The Brig where we continued drinking various stuff (PBR!.. for sheer trendiness) until around midnight when we decided to head for home. But first we stopped at one of the taco trucks that had showed up and grabbed some tacos. On the way home was basically when I realized that I was really drunk. I was walking along fuzzy headed and couldn't stop hiccuping. We got to Jay's place and continued drinking and chatting.

A pub crawl, starting at 11am and ending at 3am in Venice beach. It was the most fun I think I've ever had and all we did was talk, eat and drink.

Quicklaunch ballistic space cannon

Invisible Flying Spaghetti Monster In My Garage...

The Dragon In My Garage

by Carl Sagan
"A fire-breathing dragon lives in my garage"

Suppose (I'm following a group therapy approach by the psychologist Richard Franklin) I seriously make such an assertion to you. Surely you'd want to check it out, see for yourself. There have been innumerable stories of dragons over the centuries, but no real evidence. What an opportunity!

"Show me," you say. I lead you to my garage. You look inside and see a ladder, empty paint cans, an old tricycle -- but no dragon.

"Where's the dragon?" you ask.

"Oh, she's right here," I reply, waving vaguely. "I neglected to mention that she's an invisible dragon."

You propose spreading flour on the floor of the garage to capture the dragon's footprints.

"Good idea," I say, "but this dragon floats in the air."

Then you'll use an infrared sensor to detect the invisible fire.

"Good idea, but the invisible fire is also heatless."

You'll spray-paint the dragon and make her visible.

"Good idea, but she's an incorporeal dragon and the paint won't stick." And so on. I counter every physical test you propose with a special explanation of why it won't work.

Now, what's the difference between an invisible, incorporeal, floating dragon who spits heatless fire and no dragon at all? If there's no way to disprove my contention, no conceivable experiment that would count against it, what does it mean to say that my dragon exists? Your inability to invalidate my hypothesis is not at all the same thing as proving it true. Claims that cannot be tested, assertions immune to disproof are veridically worthless, whatever value they may have in inspiring us or in exciting our sense of wonder. What I'm asking you to do comes down to believing, in the absence of evidence, on my say-so. The only thing you've really learned from my insistence that there's a dragon in my garage is that something funny is going on inside my head. You'd wonder, if no physical tests apply, what convinced me. The possibility that it was a dream or a hallucination would certainly enter your mind. But then, why am I taking it so seriously? Maybe I need help. At the least, maybe I've seriously underestimated human fallibility. Imagine that, despite none of the tests being successful, you wish to be scrupulously open-minded. So you don't outright reject the notion that there's a fire-breathing dragon in my garage. You merely put it on hold. Present evidence is strongly against it, but if a new body of data emerge you're prepared to examine it and see if it convinces you. Surely it's unfair of me to be offended at not being believed; or to criticize you for being stodgy and unimaginative -- merely because you rendered the Scottish verdict of "not proved."

Imagine that things had gone otherwise. The dragon is invisible, all right, but footprints are being made in the flour as you watch. Your infrared detector reads off-scale. The spray paint reveals a jagged crest bobbing in the air before you. No matter how skeptical you might have been about the existence of dragons -- to say nothing about invisible ones -- you must now acknowledge that there's something here, and that in a preliminary way it's consistent with an invisible, fire-breathing dragon.

Now another scenario: Suppose it's not just me. Suppose that several people of your acquaintance, including people who you're pretty sure don't know each other, all tell you that they have dragons in their garages -- but in every case the evidence is maddeningly elusive. All of us admit we're disturbed at being gripped by so odd a conviction so ill-supported by the physical evidence. None of us is a lunatic. We speculate about what it would mean if invisible dragons were really hiding out in garages all over the world, with us humans just catching on. I'd rather it not be true, I tell you. But maybe all those ancient European and Chinese myths about dragons weren't myths at all.

Gratifyingly, some dragon-size footprints in the flour are now reported. But they're never made when a skeptic is looking. An alternative explanation presents itself. On close examination it seems clear that the footprints could have been faked. Another dragon enthusiast shows up with a burnt finger and attributes it to a rare physical manifestation of the dragon's fiery breath. But again, other possibilities exist. We understand that there are other ways to burn fingers besides the breath of invisible dragons. Such "evidence" -- no matter how important the dragon advocates consider it -- is far from compelling. Once again, the only sensible approach is tentatively to reject the dragon hypothesis, to be open to future physical data, and to wonder what the cause might be that so many apparently sane and sober people share the same strange delusion.

--- Excerpt from The Demon-Haunted World

Blogging for more than a year..

It's been more than a year since I started this blog. Wow. When I started the blog, I was working on finishing off the Shopgreen site, a site that was very well tested and laid out, but got no love (it really was kind of a dry project) from the higher ups.

A year ago today, I had just started on the CNN Trivia project for Red. Wait scrap that, I was waiting to get started on the trivia project and still working on shopgreen site while waiting for the 'producer' to figure out what to do. That project pretty much soaked up my time from July until it eventually finished after some delays in October, but was still ready for the re-launch of the Cnn site.

After that project I basically sat around for 3 weeks until I was let go the week of thanksgiving. Thanks, assholes. After that, I was annoyed with the whole prospect of having to work for another crappy non-engineering company and took the month of December off.

I started looking again in January, and with no work and no interviews beyond one at the beginning of the month, I started to get worried about paying to bills and actually applied for unemployment. But the interview earlier in the month panned out and I was able to tell the employment office that I didnt need unemployment after all.

February started with OpenID.. a crappy configuration hell project which made me doubt my skills as a python programmer due to bug after frustrating bug, most of which were in the repoze.who openid implementation. But at least I learned a ton about wsgi.

March brought the end of the OpenID project as it turned out OpenID didnt really provide what the tech lead wanted after all. At the end of the month I started on the Asset Admin project which entailed finally having to give up avoiding javascript/html/css. I've actually become rather good at it, but I'm still not entirely enthused about having to work on js/html projects.

In April I finished the first portion of the Asset Admin, the Playlist Editor, but due to work overload on the tech lead, it was never released. I completely rewrote Playlist Editor 5 times due to having to learn js/jquery at the same time, but in the end I was proud of the result. At the end of April I started on the second part of the Asset Admin, the Media Picker which was basically a continuation of what I had learned from Playlist Editor.

During May I continued working on the Media Picker, updating Playlist Editor as I learned more javascript/jquery tricks. Around this time I took the plunge into really understanding CSS. I got the two sub projects looking pretty good. I finished Media Picker and started on the last piece, the Asset Editor towards the middle of the month. I worked hard to make it so that the Asset Editor could be used anywhere (with Playlist Editor or Media Picker) and still look the same (no shifting of images or changing of heights).

But in June, the whole backend project which the Asset Admin was an administrative front end for was basically canceled through a decision by the parent company to allow serving of media through youtube and facebook etc. Everyone scrambled around looking for work until we started work on the current project which I'm not too keen on working on, the dashboard project, which is basically just a nice display of a bunch of CEO porn (aka random statistics about various things that are applicable to the company).

Non work related, I played soccer! From February '09 to August '09, I played soccer with the team we had at Red. Not since I was 17 in high school had I last played, but I was still one of the better players on the team. Surprisingly, one of the most artsy guys at the office was the best player on the team. We played for two seasons in which I had the most goals for our team the first season (3: headed the ball over the defender then blasted it in, the penalty kick, and the loose ball calm shot) but only got one second season (left footed shot!) That being said, we were a pretty sorry bunch with more than half the team having never played soccer before. Most scores were around 8-0 or some such.

In August, for the end of soccer party, I got drunk for the first time during an epic 14 hour pub crawl. Just a bunch of guys from the soccer team talking and drinking. Good times.

In October, the trainer I had been working out with for the past year finally opened his own gym. I trained with him in the gym as he was still building it; before the pull up bars were up, before the rubber mats were down. Previously we had been working out at one of those small gyms that big apartment complexes provide as an incentive to move in, but nobody really uses. And before that we were at some place in Monrovia which was owned by some guy who reminded me of a shorter smaller version of Mitch.

In November I got drunk for the second (and probably last) time during another pub crawl for JR's birthday. Wasn't as great as the first time as I ended up getting sick at the end of the night and had to crash at JR's place.

In December I did almost absolutely nothing.

In January I managed to find a job while continuing to save tons of money.

In February, my gf got a roborovski hamster (Hammie). A neurotic and seemingly suicidal hamster that is bascially crippled by fear. After a few weeks of having the hamster hurl itself from great heights to escape us, the gf got a second hamster. This one is a much more hamster-y hamster which we named 'Cookie'. Cookie lets us pet her and is very aware of heights and is generally adorable.

In April, for my birthday, the gf bought a second damn dog on complete impulse, chopped all her beautiful hair off, put on 15lbs, and was a complete bitch. The three weeks after my birthday she was basically the most unfriendly and bitch that I've ever known her to be. On several occasions I was bound and determined to call it quits, but I remained calm and kept telling myself that she was just going through some rough times. She eventually came out of it.

And so there's a year.

The "Big Lebowski" Office Management Method

Keesey: We'll just let stuff happen, man

jbum: "go with the flow"

Keesey: It'd be beautiful, you dig?

jbum: if a meeting wants to happen, it'll happen, dude

jbum: otherwise, it wasn't meant to be

Keesey: free your calendar and your mind will follow

wickedSA: the "big lebowski" office management method

Suppression.

"There are many hypotheses in science which are wrong. That's perfectly alright. It's the aperture to finding out what's right. Science is a self correcting process. To be accepted, new idea's must survive the most rigorous standards of evidence and scrutiny.

The worst aspect of the Velikovsky affair is not that many of his ideas were wrong or silly or in gross contradiction to the facts. Rather, the worst aspect is that some scientists attempted to suppress Velikovsky's ideas.

The suppression of uncomfortable ideas may be common in religion or in politics, but it is not the path to knowledge and there is no place for it in the endevour of science. We do not know beforehand where fundamental insights will arise from about our mysterious and lovely solar system and the history of our study of the solar system shows clearly that accepted and conventional idea's are often wrong and that fundamental insights can arise from most unexpected sources."

- Carl Sagan: "Cosmos" Ep.4 - Heaven and Hell. It's on Netflix.

How to ask (and answer) a question

Asker (1): Heys, is the requirement that the service provider 'client' part be in a wsgi module?
Asker (2): I've got my test stuff running in a tornado instance simply for ease of development (dont have to setup/configure apache... simple yes, but still dont have to do it)
Answerer (1): i want to wrap wsgi things with it, in a wsgi serving framework -- sounds like it'd be a wsgi middleware of some sort
Answerer (2): tornado is completely different
Answerer (3): we're not gonna redo everything to use tornado
Answerer (4): you don't need apache for anything
Asker (3): ok, that's why I was asking

So the above is a recent q/a session that took place a few moments ago in the real world. There are a number of problems with this q/a session. We'll start with the asker first.

Asker line 1 is fine. It asks the question 'is it a requirement that something be done in a certain way'. Asker line 2 is trouble. It's adding potentially confusing details to the question which the answerer has to evaluate and determine if to ignore it or not.

Answerer line 1 is a perfect answer to line 1, but as we can see in answerer line 2, the answerer picked up on details in asker line 2 which were extraneous to the actual answer. This is common for people who are very detail oriented and don't let needless details slide. In doing this though, answerer is now making assumptions about asker and briefly discussing Tornado and Apache, which the asker didn't ask about. This is even more of an issue on the internet and potential flame bait when it comes to online discussions since asker can become annoyed that the answerer has assumed that asker doesn't know what tornado is or what apache is needed or not needed for and respond with inflammatory comments such as 'that wasnt what I asked, asshole' or some such.

This simple example illustrates the reason that brevity is not only the soul of wit, but the key to good communication. Ask simple questions, give simple answers, ignore needless details when required (either answering or asking).

New Year's Python Meme

So I'm a bit late to the party, but there's meme going around python developers to discuss what they've done/discovered in the past year in a q/a manner. So here's mine.

1. What’s the coolest Python application, framework or library you have discovered in 2009 ?

Would have to be Twisted. I've avoided Twisted for a while now since I was always told that it'd hurt your head if you tried to get into it and that it wasn't very pep8 friendly, but it turns out it has some pretty good tutorials (the Finger tutorial) and that a lot of the stuff is useful right off the bat (such as the chatserver).

Non-Python:
The Lua language. It's not that it's all that cool, it's just another language and not as nice as python, but it's pretty cool in that it lets you create mods for games pretty quickly (in this case Demigod.. which is a great game that needs some help at this point).

2. What new programming technique did you learn in 2009 ?3. What’s the name of the open source project you contributed the most in 2009 ? What did you do ?
New programming technique would have to be constructor patterns that I learned from a very good Flash developer and the Refactoring book that he introduced me to. I mainly learned it by working through the refactoring example with my girlfriend on a automation project she had for her work.

The open source project I contributed the most to would have to be the PIL and python-payflowpro and only in the sense that I submitted bug reports (tho I submitted work around code for the issue with PIL).

For 2009, I worked entirely on two large web projects. One an online trivia game, and the other a credit offset reseller site. The credit offset site was fantastic in that the code was completely developed with TDD and had about 10-15 tests per function. I didn't start a single class or function without having the test code written beforehand. The trivia game had the potential to be fun, but I had to make accommodations for stupid managers and constant changes to what was wanted. I simply should have said 'No' more often, but it's hard to do when your excited and want to make everything cool.

4. What was the Python blog or website you read the most in 2009 ?

I read Planet Python which is an aggregation blog of a bunch of well known Python bloggers... so I guess I've been reading lots of blogs. Also, lots of StackOverflow.

5. What are the three top things you want to learn in 2010 ?

3 things for python in 2010?

1. Hadoop and map/reduce. I'm already familiar with the concepts, but I'd like to get some actual experience with it
2. pyGTK and pyGame. I'd really like to get into gui programming.. even tho cli rules.
3. Ctypes.. I've been meaning to read up or do a tutorial on ctypes for a long time.

Non-Python:

1. How to make mods for games (in particular, Demigod). For a long time I've played games and kept notebooks on what was cool, what was stupid and what I would do differently in the hopes that I would eventually learn how to create a mod of a game. I've been investing a lot of time in games and there's a lot of stuff that I'd like to tweak. Demigod is very open to modding with the Lua language so I'd like to finally start and complete a mod for a game.
   

Decemeber 2009: my month of quiescent perseveration

We repeat ourselves. The mind perseverates. The road is rutted and grooved, and beside the road are familiar ditches.
-- Eric Maisel

da Vinci quote

Those who are enamored of practice without science are like a pilot who goes into a ship without rudder or compass and never has any certainty where he is going. Practice should always be based upon a sound knowledge of theory.

Well.. so much for javascript..

I have always despised javascript and html for it's hacky approach to...well.. everything. Html was well done in the 90's, maybe, but at the moment html and javascript just look and feel and appear to work like one huge cobbled together kludge that nobody can fix because the entire world uses html/javascript and sorta knows how it all fits together. Years ago when I was the webmaster for a small college, I dealt with javascript/html extensively and hated it intensely (back in the age of ie3 and Netscape and just the beginnings of mozilla 1.0).

So after my latest little adventure in javascript and getting everything to work perfectly in FF3, my girlfriend, who does QA for another web company, asked 'Does it work in Internet Explorer?' So I tried my simple little <input> hider (along with all the concessions I made for I.E) in I.E 8. I was expecting that there might be some small issues that would popup, but nothing major. Instead what I got was I.E 8 completely freezing to the point where I couldn't even kill it in the task manager. I got the Vista little spinning ring of death and in order to fix the issue, I had to get to the ctrl-alt-delete menu and hit 'logout'.

Now I know that there's lots of people out there who are experts in making javascript things in sing in both ie8 and ff3, but at what cost? I work at a company who primarily works with flash and there's always a big discussion/name-calling-fest whenever a site is made entirely in flash when it 'should' be made in html/javascript. The main reason people usually defend the decision to use flash is something along the lines of "we have lots of very good flash developers and it's faster for them to just develop the site in flash", which is true. The 'it works properly in all browsers with a minimum of fuss' is rarely even brought up.

But given the amount of design considerations, QA, potental major headaches that have to be taken into account when developing an html/javascript site, I would have to say that these issues are very significant. Developing in flash is significantly easier simply because the majority of these issues are minimized. You simply develope the app, QA it and deploy it. It works as expected in all browsers. No special considerations that need to be learned and tested, no design decisions that have to be abandoned because the feature will work in one browser, but not in another.

Flash wins.

Update: HTML 5 to control them all.

Err messages....

For the project I'm working on currently, they don't want to let 'unfriendly' (meaning too technical) error messages through.. I think all error messages should be unfriendly because when they are relayed to the devs on the backend, it helps them track down exactly what the issue is much faster. But then again, my code does contain the error message:

"Error: Oh my god you've ruined EVERYTHING, AGAIN! You old bag!"

This is really quite good at being unfriendly, see, cuz old people have probably ruined everything at least once at their age and probably live their lives dreading that they might do it again...

Tall in the bathroom stall.

You know what would suck about being 7 foot tall? In most public bathrooms you can see over the stall walls and look at the guy who's currently wiping his ass.

Course you never hear 7 foot tall people complaining about this because usually they play for the NBA and are evil rich and don't have to use public restrooms. Thus, if you see a 7 foot tall guy looking over the stall at you while you take a dump, take pitty on him since he's most likely so rich that he's going to hell.

Adages

Red sky in morning, sailors take warning;
Red sky at night, sailors delight.

or was it
Red sky in morning, sailors in mourning;
Red sky at night, sailors in plight.

Yeah, that second one is it.. it basically means if you're a sailor and the sky is Red, your screwed.

And yes if the sky is red instead of blue for anyone else, it means you're going to hell.

Such a lame ending to a great game, Fallout 3

Finally finished Fallout 3. Unfortunately, the ending was just about the most horribly stupid ending that I can remember. I've not been watching many movies lately, so maybe I'm more susceptible to stupid STUPID endings, but the ending was a contrived turd strained out in pain. Especially when it was already established that there was another character who could accomplish the final 'deed' without being harmed in anyway, as that character had already done before. A super mutant, Fawkes, who was resistant to radiation, had gone into a highly irradiated chamber to retrieve a device called the G.E.C.K, but when asked to do the same task and enter in a simple code into a computer, his response was something along the lines of 'you have your own destiny to follow'. Apparently there's another character, Charon, who REGENERATES HEALTH from radiation, who basically says the same thing.

Poor, POOR form.