Preventing Project disIntegration

Loose Definitions

Let's define unit-testing as the testing of a small, targeted selection of classes as part of development. Unit-tests should be fast and run continually as part of development. Because of the need for speed, unit-tests typically use mock objects so that they stay within the app's process boundary.

When I first wrote unit-tests, I broke this rule all the time. I would write tests that crossed as many process boundaries as possible: touching a real DB and all the way back. After many arguments with people, I eventually realized that these were integration tests and not unit-tests. As a unit-tester, I was being evil by writing long-running tests that slowed things down and missed the point of unit-testing.

Rockin' with the Mockin'

Then, for a time, I went the other way: all unit-tests, all the time. The mantras were "Speed is king!" and "Who cares about the DB when you are concentrating on a composite-flyweight-visitor pattern?"

Well, I know someone one a project who cares about the DB: the project is working through some serious i18n issues. In my estimation, the problem lies with too few integration tests.

Unit by day, Integration by night

My new philosophy is the best of both worlds. The famous, celebrated unit-tests should be run during the day, as always. The nightly build should not only run these tests but also a serious suite of integration tests.

These tests should expressly cross system boundaries and deal with more with "use-cases" than mere "test-cases". Unlike unit-testing, where (semantic) redundancy should be avoided, I think redundancy should take a back seat with integration testing (i.e. don't worry if the same thing is being tested more than one way -- this doesn't mean code redundancy).

The focus of integration testing should be in asking the big questions.

Here are some examples:

Question: Is my app internationalized?

Tactic: Using canned input in French/Russian/Japanese, a test calls the app's service layer to store that input in a given object/DB, and then reads it back through the service layer to ensure that i18n is stored correctly.

Question: Does my app handle a given use-case? e.g. In a content-management system, a user adds a resource to a document, and is warned that the resource is being shared and cannot be modified. The user continues to include the resource and tries to modify it anyway.

Tactic: Use a testing DB to set up the raw materials for these kinds of use-cases, possibly restoring it after every test (slow, yes, but these will be running at 4 am!) Then integrate individual test-cases through composition to re-create the story above. The finished test will probably read like a narrative that describes the above scenario.

Bottomline

We don't want surprises late in the dev cycle, or else our fine projects will disintegrate into chaos. The rock-star Unit Test should be given its due, but don't forget its older brother Integration Test.