The Golden Rule of Testing and JUnit Assumptions

The Golden Rule of Testing:

• There are two times when all the unit and integration tests must run and pass: immediately prior to a check-in and on the build server.

Great. But what about the time when the developer is actively working on a task? That zone of "dev time" is a gray area.

I think most developers run the particular unit test for the class of interest. Lately, I have been running integration tests with the wonderful dbUnit.

In this case, each test is working with Hibernate transactions. It is very important that each test perform its setup and teardown properly, as one error can impact other tests, and this is tricky to debug. Consequently, during dev time, I want to run all of the tests fairly often.

Well, all of them except one called Test Q. Test Q takes 90+ seconds. It has become my nemesis, my rival. I see it running when I close my eyes at night. It haunts me.

Options

There are plenty of options for dealing with Test Q. We could break it up into different tests. We could write a test suite. And so on.

However, with an article and a JUG presentation, Charles Sharp has opened my eyes to the possibilities in JUnit 4.x.

Assumptions

JUnit 4 (and other modern testing frameworks) have a concept of theories and assumptions. These ideas are interesting, but outside of the scope of this post. Check out this article for more.

For my needs, suffice it to say that an assumption is used to declare assumptions about the data coming into a unit test. If the assumption is not true, then the test automatically passes.

Here's an example:

import org.junit.*;
import static org.hamcrest.CoreMatchers.*;
import static org.junit.Assume.*;

public class AssumeTest {
    @Test
    public void testA(){
        int x = 1;
        assumeThat( x, is(7) );
        System.out.println("assumption is true!");
    }
}

The assumeThat statement assumes that x is 7. If not (as in this silly example), the test will quietly pass. This may seem horrifying, but it makes sense within the realm of theories.

It also blows the door open in terms of dev time. There are two advantages: (a) the test passes and (b) x is a runtime variable. Compare this to another approach with a timeout variable.

public class AssumeTest {
    @Test(timeout = 5000)
    public void testALongRunningTest() {
    }
}

In the above example, the test will timeout after 5 seconds. However, the test will fail and the timeout value must be a constant. During dev time, I don't want Test Q to fail; during the Golden Rule, I don't want it to timeout. Effectively, I want to skip it on my terms.

Configuration via Assumptions

I don't know yet if what follows is merely a parlour trick. I have not yet used it in production, but one can do some pretty cool things with these assumptions, outside of the theory arena. We can make assumptions about the state of the JVM in order to determine if we are in "dev time".

For example, we can do the following:

• Configure a test so that it can be disabled via a JDK parameter (take that Test Q!).
• Configure tests so that only a given percentage of them will run at a given time.

Both can be done with assumptions. And in both cases, the Golden Rule is not violated.

Here is an example of the first configuration:

import org.junit.*;
import static org.hamcrest.CoreMatchers.*;
import static org.junit.Assume.*;
import static java.lang.System.*;

// class under 'test'. I work in biology
class Bacteria {}

public class BacteriaTest {

    @Before
    public void setUp() {
        out.println("hello from setup");
    }

    @After
    public void tearDown() {
        out.println("hello from teardown");
    }

    @Test()
    public void veryLongTestA() throws Exception {
        // if DEV is defined, then skip and auto-pass
        assumeThat( System.getProperty("DEV"), nullValue() );

        out.println("running long test A");
        Thread.sleep(90 * 1000);
    }

    @Test
    public void reasonableTest() {
        out.println("reasonable test");
    }
}

To run this example (with JUnit 4.4+ on the classpath):

apt BacteriaTest.java
java org.junit.runner.JUnitCore BacteriaTest
OR
java -DDEV=true org.junit.runner.JUnitCore BacteriaTest

The idea is simple: when the Golden Rule applies, no JDK params are provided and everything runs. But the dreaded test can be skipped when needed.

Admittedly, it does require code within the test, but I'm not sure that it is intrusive.

This is especially true in the next example, which combines both ideas: the one mentioned above but also another. This second idea is: during dev time, can we run some of the tests? The thought is that we don't want to run them all but don't want to run only one either. Think of it as a testing heuristic of sorts. The example below is neat in that the choice of tests is random.

Here's the code:

import org.junit.*;
import static org.hamcrest.CoreMatchers.*;
import static org.junit.Assume.*;

import java.math.*;
import static java.lang.System.*;

// class under 'test'. I work in biology
class Bacteria {}

abstract class BaseTest {
    // if DEV is defined, auto-pass test (i.e. skip)
    public void binarySwitch() {
        assumeThat( System.getProperty("DEV"), nullValue() ); 
    }

    // if random value < THRESHOLD, auto-pass test (i.e. skip)
    public void probabilitySwitch() {
        int threshold = 101;
        String thresholdStr =  System.getProperty("THRESHOLD");

        if( thresholdStr != null ) {
            threshold = Integer.parseInt( thresholdStr );
        }

        int value = (int) Math.floor( Math.random() * 100 );
        boolean doRun = ( value < threshold );
        assumeThat( doRun, is(true) );
    }
}

public class BacteriaTest extends BaseTest {

    @Before
    public void setUp() {
        out.println("hello from setup");
    }

    @After
    public void tearDown() {
        out.println("hello from teardown");
    }

    @Test()
    public void veryLongTestA() throws Exception {
        binarySwitch();

        out.println("running long test A");
        Thread.sleep(90 * 1000);
    }

    @Test
    public void reasonableTest() {
        probabilitySwitch();

        out.println("reasonable test");
    }
}

Again, to run this example (with JUnit 4.4+ on the classpath):

apt BacteriaTest.java

// Golden Rule
java org.junit.runner.JUnitCore BacteriaTest

// or Dev Time
java -DDEV=true -DTHRESHOLD=25 \
org.junit.runner.JUnitCore BacteriaTest

In this case, we've refactored the first idea into the binarySwitch() method and introduced the probability heuristic into the probabilitySwitch() method. The reasonableTest() might run and it might not. This is silly in the small example, but consider it over a large test base.

Again, this doesn't violate the Golden Rule because the assumptions will be true. But it may have potential for development time, when they can be falsified. Because JUnit 4.x completely supports the 3.x format of tests, I might try swapping out the jars at my day gig.

Test Q, you are going down!