QA Notes

Testing

• Who is our largest QA audience?  Our users.  There's no escaping the fact that our users will find more bugs than our QA team.  Therefor it is in our best interest to capture what they are doing with logging and sane error messages.  There needs to be logging servers which handle the logs coming in from various environments (QA/Prod/Various different versions of prod) that allow us to grep back through and find out exactly what it was that the user was doing when he got the error. We can then replicate what he was doing in QA with debugging on, but if we are logging all error messages, this might not even be necessary.  Better exception handling is also a must.

Tests

• Unit tests - Tests developers write to make sure code isn't broken, no syntax errors, handles all edge cases etc.
• Acceptance Tests - Supposed to be written by Product Team (yes, they're supposed to be developers, they are supposed to have the good idea's for the product and understand how the features are going to work into the current code base), ensure that the code that the developer wrote is what the actually wanted in the feature
• Integration Tests - Test that the code works correctly in the entire stack, a replica as close as possible to production

More on Assertive Programming

(from The Pragmatic Programmer: From Journeyman to Master by Andy Hunt and David Thomas)

There is a common misunderstanding about assertions, promulgated by the people who write compilers and language environments. It goes something like this:

Assertions odd some overhead to code. Because they check for things that should never happen, they'll get triggered only by a bug in the code. Once the code has been tested and shipped, they are no longer needed, and should be turned off to make the code run faster. Assertions are a debugging facility.  

There are two patently wrong assumptions here. First, they assume that testing finds all the bugs. In reality, for any complex program you are unlikely to test even a miniscule percentage of the permutations your code will be put through (see Ruthless Testing). Second, the optimists are forgetting that your program runs in a dangerous world. During testing, rats probably won't gnaw through a communications cable, someone playing a game won't exhaust memory, and log files won't fill the hard drive. These things might happen when your program runs in a production environment. Your first line of defense is checking for any possible error, and your second is using assertions to try to detect those you've missed. Turning off assertions when you deliver a program to production is like crossing a high wire without a net because you once made it across in practice. There's dramatic value, but it's hard to get life insurance.

Python Utils

http://wiki.python.org/moin/PythonTestingToolsTaxonomy