Thursday, April 17, 2008

The Golden Rule of Testing and JUnit Assumptions

The Golden Rule of Testing:

  • There are two times when all the unit and integration tests must run and pass: immediately prior to a check-in and on the build server.

Great. But what about the time when the developer is actively working on a task? That zone of "dev time" is a gray area.

I think most developers run the particular unit test for the class of interest. Lately, I have been running integration tests with the wonderful dbUnit.

In this case, each test is working with Hibernate transactions. It is very important that each test perform its setup and teardown properly, as one error can impact other tests, and this is tricky to debug. Consequently, during dev time, I want to run all of the tests fairly often.

Well, all of them except one called Test Q. Test Q takes 90+ seconds. It has become my nemesis, my rival. I see it running when I close my eyes at night. It haunts me.

Options

There are plenty of options for dealing with Test Q. We could break it up into different tests. We could write a test suite. And so on.

However, with an article and a JUG presentation, Charles Sharp has opened my eyes to the possibilities in JUnit 4.x.

Assumptions

JUnit 4 (and other modern testing frameworks) have a concept of theories and assumptions. These ideas are interesting, but outside of the scope of this post. Check out this article for more.

For my needs, suffice it to say that an assumption is used to declare assumptions about the data coming into a unit test. If the assumption is not true, then the test automatically passes.

Here's an example:


import org.junit.*;
import static org.hamcrest.CoreMatchers.*;
import static org.junit.Assume.*;

public class AssumeTest {
@Test
public void testA(){
int x = 1;
assumeThat( x, is(7) );
System.out.println("assumption is true!");
}
}

The assumeThat statement assumes that x is 7. If not (as in this silly example), the test will quietly pass. This may seem horrifying, but it makes sense within the realm of theories.

It also blows the door open in terms of dev time. There are two advantages: (a) the test passes and (b) x is a runtime variable. Compare this to another approach with a timeout variable.


public class AssumeTest {
@Test(timeout = 5000)
public void testALongRunningTest() {
}
}

In the above example, the test will timeout after 5 seconds. However, the test will fail and the timeout value must be a constant. During dev time, I don't want Test Q to fail; during the Golden Rule, I don't want it to timeout. Effectively, I want to skip it on my terms.

Configuration via Assumptions

I don't know yet if what follows is merely a parlour trick. I have not yet used it in production, but one can do some pretty cool things with these assumptions, outside of the theory arena. We can make assumptions about the state of the JVM in order to determine if we are in "dev time".

For example, we can do the following:
  • Configure a test so that it can be disabled via a JDK parameter (take that Test Q!).
  • Configure tests so that only a given percentage of them will run at a given time.
Both can be done with assumptions. And in both cases, the Golden Rule is not violated.

Here is an example of the first configuration:


import org.junit.*;
import static org.hamcrest.CoreMatchers.*;
import static org.junit.Assume.*;
import static java.lang.System.*;

// class under 'test'. I work in biology
class Bacteria {}

public class BacteriaTest {

@Before
public void setUp() {
out.println("hello from setup");
}

@After
public void tearDown() {
out.println("hello from teardown");
}

@Test()
public void veryLongTestA() throws Exception {
// if DEV is defined, then skip and auto-pass
assumeThat( System.getProperty("DEV"), nullValue() );

out.println("running long test A");
Thread.sleep(90 * 1000);
}

@Test
public void reasonableTest() {
out.println("reasonable test");
}
}


To run this example (with JUnit 4.4+ on the classpath):


apt BacteriaTest.java
java org.junit.runner.JUnitCore BacteriaTest
OR
java -DDEV=true org.junit.runner.JUnitCore BacteriaTest


The idea is simple: when the Golden Rule applies, no JDK params are provided and everything runs. But the dreaded test can be skipped when needed.

Admittedly, it does require code within the test, but I'm not sure that it is intrusive.

This is especially true in the next example, which combines both ideas: the one mentioned above but also another. This second idea is: during dev time, can we run some of the tests? The thought is that we don't want to run them all but don't want to run only one either. Think of it as a testing heuristic of sorts. The example below is neat in that the choice of tests is random.

Here's the code:


import org.junit.*;
import static org.hamcrest.CoreMatchers.*;
import static org.junit.Assume.*;

import java.math.*;
import static java.lang.System.*;

// class under 'test'. I work in biology
class Bacteria {}

abstract class BaseTest {
// if DEV is defined, auto-pass test (i.e. skip)
public void binarySwitch() {
assumeThat( System.getProperty("DEV"), nullValue() );
}

// if random value < THRESHOLD, auto-pass test (i.e. skip)
public void probabilitySwitch() {
int threshold = 101;
String thresholdStr = System.getProperty("THRESHOLD");

if( thresholdStr != null ) {
threshold = Integer.parseInt( thresholdStr );
}

int value = (int) Math.floor( Math.random() * 100 );
boolean doRun = ( value < threshold );
assumeThat( doRun, is(true) );
}
}

public class BacteriaTest extends BaseTest {

@Before
public void setUp() {
out.println("hello from setup");
}

@After
public void tearDown() {
out.println("hello from teardown");
}

@Test()
public void veryLongTestA() throws Exception {
binarySwitch();

out.println("running long test A");
Thread.sleep(90 * 1000);
}

@Test
public void reasonableTest() {
probabilitySwitch();

out.println("reasonable test");
}
}


Again, to run this example (with JUnit 4.4+ on the classpath):


apt BacteriaTest.java

// Golden Rule
java org.junit.runner.JUnitCore BacteriaTest

// or Dev Time
java -DDEV=true -DTHRESHOLD=25 \
org.junit.runner.JUnitCore BacteriaTest


In this case, we've refactored the first idea into the binarySwitch() method and introduced the probability heuristic into the probabilitySwitch() method. The reasonableTest() might run and it might not. This is silly in the small example, but consider it over a large test base.

Again, this doesn't violate the Golden Rule because the assumptions will be true. But it may have potential for development time, when they can be falsified. Because JUnit 4.x completely supports the 3.x format of tests, I might try swapping out the jars at my day gig.

Test Q, you are going down!

4 comments:

Eric Burke said...

Can you run the slow test in a thread while all of the other tests run? Maybe if 4 tests ran concurrently no matter what? (that sounds like a good number for a dual core system)

Michael Easter said...

@Eric

I was wondering about threading as well. AFAIK, JUnit 4 does not run tests concurrently.

If Charles or another JUniter reads this, they can set me straight. I'm curious if this is in the works at all.

@All

Btw, 2 other general points as a post-script.

As mentioned, JUnit 4 supports the 3.x tests. This is a brilliant move. So one can replace the jar and migrate over time.

Also, as Charles pointed out, they gave up on the Swing UI runner and some of the others. They adapted to the reality of usage (automated runners and text runner) and concentrated on that. Another wise idea to just cut bait and go for bang-for-the-buck.

Scott Bale said...

Thought-provoking stuff, thanks. I dzone'd it here:

http://www.dzone.com/links/the_golden_rule_of_testing_and_junit_assumptions.html

Michael Easter said...

Thanks Scott.... 'Thought-provoking' was my reaction to Charles' excellent presentation.

As mentioned, this is new stuff (for me): there seem like a lot of neat possibilities, even if these specific examples don't pan out.