Testing: The Developer Strikes Back

DjangoCon 2011Sandy Strong

About Me● Developing with Django since fall 2009● Started using Python shortly before● Did server side dev, reporting, "devops", and trolling at

Mahalo.com for 1.5 yrs○ I wrote a lot of tests at Mahalo○ Broke a lot of things at Mahalo

■ Because I didn't write (the right) tests● PyLadies board member

Outline● Unit test organization strategies for Django projects● Dealing with increasingly complicated testing scenarios● Improve coding habits through test writing● Testing a virgin codebase● Where fixtures, sqlite, and cache fit into testing● Testing tools that you should know about● Graceful code degradation● How to sell testing at your Django shop● Testing infrastructure

Testing: it's hard to do (right)● Delegate test responsibilities correctly● Won’t (always) get it right on the first try● Test refactoring opportunities present themselves● Tests are living code, documentation, evolve over time● Can't predict everything that will happen in production● Worth doing right

What is "unit testing"?A method by which individual units of source code are tested to determine if they are fit for use.

A unit is the smallest testable part of an application.

Making sure a "unit test" tests a unit● Each app in your project needs its own test module

○ Each module should have (many) sub-modules■ Each contains classes that test units of code

● Organize your suite however you want○ Just be consistent so it's easy to find things

● Minimally, separation should exist between the testing of:○ Business logic○ Datastores○ 3rd party APIs

● And other considerations, as applicable to your project...○ Network latency○ Concurrency

Organize and optimize your tests● As your project becomes more complex...

○ Test suite will grow larger○ Increasingly complex test scenarios arise

■ cache, NoSQL, message brokers, 3rd party APIs○ Separate code and service/infrastructure testing

■ suite execution speed, accuracy

● Don't keep all of your tests in one monolithic file named test.py in the root of your project!

● Test suite should mirror the organization of application code○ Test each method○ Make multiple assertions

Don't do this...

myproject/ accounts/... store/... test.py <--- 6000 lines

● Have fun debuggingyour test whensomething inside a 500 line method breaks!!!

● Often results in a few monolithic classes○ with tens of test methods

■ methods that contain multiple points of responsiblitiy

Project | Mirror* vs Embeddedmyproject/ accounts/ api/ main.py lib.py views.py models.py urls.py store/ rest.py models.py views.py cart.py

myproject/ accounts/ ... store/ ... test/ accounts/ __init__.py api/ main.py lib.py views.py store/ __init__.py rest.py views.py cart.py

myproject/ accounts/ ... tests/ __init__.py api/ main.py lib.py views.py ... store/ ... test/ __init__.py rest.py models.py views.py cart.py

*Easier to exclude single directory when deploying

Creating test sub-modules (embedded)myproject/ account/ api/ lib.py main.py... tests/ __init__.py api/ lib.py main.py

#myproject/account/tests/__init__.py

from myproject.account.tests.api.main import *from myproject.accounts.tests.api.lib import *

Are sub-module or root test.py files ok?

● Certain methods, used throughout entire project○ test.py file at project root is ok (i.e. mysite/test.py)

■ Ex: certain helper functions (unique IDs or tokens)

● Need for repetitive object creation inside app○ test file at base of testing sub-module inside app

■ Ex: ObjectCreator classes and other frequently used app-specific functionality

● Not necessarily bad

● Use sparingly

● Be careful what responsibilities you delegate there

An example unit test classclass TestCreateItem(object): ''' Test class for item creation functionality '''

def setUp(): ... def tearDown(): ... def test_assert_create(): '''Use ObjectCreator''' def test_update_price(): '''3rd Party API (mock response)''' def test_assert_invalid_category(): '''Business logic''' def test_assert_permission_denied(): '''Business logic'''

Why use ObjectCreator classes?● Makes the supporting code for your tests...

○ Loosely coupled○ More maintainable

● Easier to find bugs in the object creation process, because the bulk of the work is kept in one place

● Keep the creation of objects that will be used to test business logic, out of the way of testing said logic

● DRY-er test code

ObjectCreator Class/Usage Example#myproject/store/test.pyclass StoreAppObjectCreator(ObjectCreator):

def make_item(self, data, user='test'): item = Item() item.name = data['name'] item.price = data['price'] item.created_by = user item.save()

return item

#myproject/store/api/lib/test.py def test_item_purchase(self): data = {'name': 'Book', 'price': 50.00} item = StoreAppObjectCreator.make_item(data) ...

Give Mock a try● http://python-mock.sourceforge.net/

● Use Mock to create objects that mimic the interface of the class that the unit you are testing depends on

● Examine the Mock object to see what methods have been called and what parameters were used in the call

Beyond the business logicAs your project becomes more complex, you will find new and exciting scenarios to test (sometimes the hard way)

For many web applications, this means figuring out how to test 3rd party APIs and cache

The successful execution of your business logic may be dependent upon external APIs or cache, but they are separate units and so should be tested separately

"Facebook won't let me down! Oh shi--"● Remember that time Facebook went down for a few hours

recently?

● Easy to take 3rd party APIs for granted -- especially when it's Google or Facebook

● It's not practical to call out to 3rd party APIs each time you

run through your test suite

● Bottom line: there are two things you need to know about your application as it relates to 3rd party APIs

○ How it behaves if the 3rd party API is down

○ How it behaves if the 3rd party API changes

Mock 3rd party API responses● Test app behavior against mocked 3rd party responses

○ Mock a variety of possible response types

○ Can be done safely inside of your primary test suite

● Maintain a separate test suite that makes calls to 3rd party

○ This test suite will run more slowly, generate false failures more often than you may think

○ Won't impact the testing of your application logic

○ Will catch 3rd party API changes that break your app

Dealing with cache● Web apps are dynamic, models change often● Web apps frequently rely (somewhat) heavily on cache

● Development environments typically don't have long-term or complete sets of cached data

○ Many staging environments are also lacking in this way

● Ideally, your cache should be "agnostic" to model changes○ Version objects when they are cached○ Do object structure comparison when pulling from cache○ Caching Django Model objects rather than Python objects

is convenient...■ This one time, I was upgrading Django...

■ The class for one of the objects I had cached changed

How do I test cache?● This topic pertains more to

integration testing than it does unit testing

● What you're trying to test isn't cache itself (unless you're testing your infrastructure)

○ You need to know and understand how your modified application behaves when it interacts with a cache that was generated by your application in a previous state.

● Use a cache filler to warm up "old" cache○ Good for staging env, not practical for dev env

Writing tests can improve coding habits

● Smaller, modular code is testable, 200-line functions aren't

● Write more tests

○ Discover easily-testable coding patterns ■ This is a good thing■ Functions should perform a single function■ Units of code should be true to the definition

○ There will always be exceptions■ Units come in different sizes■ Some functions need to perform several tasks

"What about T.D.D.?"

"Test Driven Development? I don't think it exists."

What is TDD, really?

Step 1: Write tests that fail

Step 2: Write code until all tests pass

TDD sounds pretty simple, right?Then why aren't we all doing it 100% of the time?

It's complicated...

● Deadlines● Non-technical management

● Competing obligations● Poor planning/estimates

● Some code doesn't lend itself easily to TDD● In some cases, TDD doesn't make sense

Meh, I don't even like TDD... I just need my code to be well-testedTDD:

● Goes against prototyping● Requires full team buy-in to really work● Business owners (probably) won't get it

Well-tested code: a happy (enough) medium?● More realistic● More practical● Allows for more individual "style" in dev cycle

○ i.e. "I hate writing tests up front"

Tests are alive

"If you do your tests right, you will never encounter a scenario in production that your tests did not anticipate."

● Tests are not "set it and forget it"○ Don't think of them as something that has to "get done"

● Regardless of how/when you get your test-writing done, you will revisit them on a regular basis

○ Documentation○ New test cases

How do I enforce testing on my team?● If developers understand the value of testing, and are made

aware of the ways in which they are beneficiaries of testing, they will want to write tests

○ Fewer 2AM fires○ Iterate faster and with confidence

● If all else fails

○ Use git pre-commit hooks!■ No checkin without tests

○ Make a game out of it with coverage.py

○ And then there's always good ol' fashioned public shamings via the continuous integration IRC bot, for folks who break the build!

How much testing is enough testing?● No simple answer for that● This one time, I ruined two database clusters

○ Employed request object inside of a DB Router class ○ Needed to know what tables were modified in request

● Oops... when there's norequest object, what happens?

○ DB operation is delegated to random DB server

■ Sometimes to slave, andsometimes to master

■ DB-level write restrictions on slaves failed

... so yeah, that wasn't enough testing.

Junction between unit and integrationDifficult areas to test, because behavior is dependent upon environment

Example:● App code that used DB router was unit tested● I performed integration testing

○ Query/access logging at DB and Application level

● The failure: ○ There were several scripts that ran out of cron against

the DB and did not mock a request object○ The scripts did not run on dev/staging environment○ Slave DBs were not respecting their read-only setting

Testing a virgin codebase: 0 - 100%● You may find yourself faced with a project that has never

known the suite love of tests

● Test driven refactoring

○ Almost guaranteed, you'll find yourself rewriting application code as you begin writing tests

■ Units, reusability, testability

● One file at a time

○ Methodical approach pays off

Testing the Mahalo codebase● Django since spring 2009, unit tests began summer 2009

● One person on team writes a few tests, establishes framework, on-boards the rest of the team

● First broadly tested app was accounts○ User permissions

■ Good place to start, lots of scenarios to test■ Code fairly modular

○ I wrote most of these tests, made some big mistakes■ Relied on cache■ Had one or two monolithic test classes■ This was all refactored and reorganized by me and

others

Successful strategies● Require unit tests for all code going forward

● Reserve time/points to clear out "test debt"○ Good for new team members (learning codebase)○ Everyone takes a turn on rotating basis

● Establish a good foundation to build on

● Every bug is two bugs○ One in the code that broke, one in the test that failed to

detect the bug

● Later: implement continuous integration to monitor coverage (Jenkins) and git hooks to reduce untested code making it into the repo

Tests need data-- where to get it?Making data available to your tests is done primarily via:

● Loading fixtures from flat file into DB● Creating objects in the setUp method of your test classes

When to use which?● Fixtures are great for loading data that is required to

bootstrap your application○ User types/permissions, relatively "constant" stuff○ Mocked responses from 3rd parties/services

● Use ObjectCreator classes for models that change frequently (fixtures will become stale quickly)

○ Items in a store, user profiles

Use SQLite (default for Django test runner)● Fast to create/update● Exists in memory

Should I get test data from cache?● No - adds unnecessary complexity to unit tests

● Code units should not be dependent on cache○ ...therefore tests should not be dependent on cache

● If you find yourself writing tests that only succede if you have certain data available in cache, you may have found an opportunity to refactor some of your application code

● Cache invalidation is hard, bug-prone○ Handle it with Django signals

■ Abstracts caching functionality out into its own unit which can be tested independently

Graceful code degradation● Developers need to think outside their dev instance

○ How will the code I'm writing behave under condition X when it interacts with service Y?

● Service unavailable shouldn't mean your site is unavailable○ Mocking a variety of response types from 3rd party as

well as infrastructure components adds value

● If failure is catastrophic (and sometimes it will be), the best you can do is know how your app will behave during the event

○ What your app can survive depends greatly on how smart your code is

Survivable vs. Degraded vs. ReadOnly

With good test coverage and strategies....

Survivable:● Search Engine, 3rd Party API, Cache

Degraded:● Cache, Message Consumer (Celery)

Read Only:● Databases

Forcing awareness?

● Code should not be written in such a way that it won't work (at least in some fashion) if certain services or infrastructure fail

● Removing these dependencies has the side effect of simplifying the local Dev environment

○ Cool, your stack includes 5 different servers or services!■ Should each Dev have to mimic this setup and run 5

servers/services locally? No...

Test Infrastructure● Staging environment should be identical to production

○ I've never worked somewhere where this was actually true...

○ Strive for as close a match as possible● If you have servers like Solr or RabbitMQ

○ Run them on staging, not dev environments○ Leverage logging where useful

■ i.e. Celery has the option to bubble up exceptions■ Don't over-do it on logging, this will slow down test

suite execution

Testing tools you should know about● Nose <pypi.python.org/pypi/nose>● Coverage <pypi.python.org/pypi/coverage>● Mock <pypi.python.org/pypi/mock>

How to sell testing at your Django shop● There're lots of things you can do wrong in testing

○ The biggest mistake is not investing time to write tests○ "Code not tested is broken by design"

● How to sell it to your boss?○ Saves money○ Saves development time ($$$)○ Happier developers○ Saves on QA time ($$$)○ Confidence

Sourceshttp://ragefac.es/http://en.wikipedia.org/wiki/Unit_Testhttp://python-mock.sourceforge.net/http://media.tumblr.com/tumblr_lkbm74Ud3R1qztbgd.jpg