Pramatr Blog

Lots of my development effort is taken up writing unit tests. These tests form a nice warm security blanket for the code that I’ve developed. If the code is broken, typically these tests are the first feedback that something is wrong. Although some people fear the little red bar, it’s become a common part of development. So with this in mind, it’s vitally important that when a test fails, is crystal clear what really went wrong.

Over the past few weeks, I’ve been reading xUnit Test Patterns (review to follow in another post), which covers making tests easier to write, understand and maintain. One important issue it discusses, is the use of assertion methods.

Assertions are a fundamental part of writing unit tests, and are typically very simple to use. Within each test, the expected outcome is expressed using assertions, each of which is boolean expression. The assertion should evaluate to true, so if the evaluation of the assertion instead returns false, the test will be failed. JUnit provides the Assert class to handle various types of common assertion. Although Assert provides a number of very simple methods, I’ve seen these frequently used inappropriately, causing unit test failures to be unnecessarily difficult to track down.

Assertion Selection

When reviewing unit tests, one of the typical activities is to ensure that assertions are being used appropriately. When developers first start to write unit tests, assertions are often used inappropriately and as such reviews can become very time consuming. Tools play a major part in development and can automate many of these kind of manual review activities. PMD is one such tool, and it provides a rule set to deal with the common JUnit problems.

One of the most common selection problems is the use of the stated outcome assertion instead of the equality assertion. When the equality assertion evaluates to false, it includes information about the two objects that should have been equal in the failure message. This information allows a developer to quickly get an idea what went wrong in the test. If the developer had used a stated outcome, (such as assertTrue), no information would have been recorded about the object state. This typically leads to extra log statements being added, or the developer has to being debugging the test.

Ensuring that the appropriate assertion is used within each unit test, is a very simple thing to do, but it gives very clear feedback about failures and allows developers to quickly track their cause.

// assertTrue("code worked", true);
// this should be removed as it serves no purpose

// assertTrue("code didn't work", false);
// Fails a test with the given message.
fail("code didn't work");

// assertTrue("actual is not empty", !actual.isEmpty());
// Asserts that a condition is false.
assertFalse("actual is not empty", actual.isEmpty());

// assertTrue("object is null", actual == null);
// Asserts that an object is null.
assertNull("object is null", actual);

// assertTrue("object is not null", actual != null);
// Asserts that an object isn't null.
assertNotNull("object is not null", actual);

// assertTrue("objects are the same", expected == actual);
// Asserts that two objects refer to the same object.
assertSame("objects are the same", expected, actual);

// assertTrue("objects are not the same", expected != actual);
// Asserts that two objects do not refer to the same object.
assertNotSame("objects are not the same", expected, actual);

// assertTrue("objects are equal", expected.equals(actual));
// Asserts that two objects are equal.
assertEquals("objects are equal", expected, actual);

Assertion Configuration

The equality assertion method provides a very simple contract, requiring the expected and the actual value. Although the contract is simple, one of the most common problems I have encountered, is that the arguments passed in to the method are in the wrong order (e.g. the expected value argument is actually the actual value and vice versa).

Although this is a trivial issue to correct, I have seen developers waste large amounts of time trying to work out what’s wrong with the code when a unit test fails. The developer immediately start to scratch their head as the values don’t seem to make sense. Without reviewing the unit test, they dive into the code to solve the problem.

I have seen this happen countless times, and every time it happens, the developer in question can’t believe they didn’t have a look at the unit test first. The equality assertion isn’t difficult to use, you just have to remember; expected, actual…. expected, actual…. expected, actual…. expected, actual….

// assertEquals("objects are equal", actual, expected);
// Asserts that two objects are equal.
assertEquals("objects are equal", expected, actual);

Assertion Messages

xUnit Test Patterns provides an great discussion about the assertion message pattern. Most developers I’ve spoken to, do add assertion messages to their tests, and I also personally find them very useful. I completely agree with Meszaros when he says;

A well-crafted Assertion Message makes it very easy to determine which assertion failed and exactly what the symptoms were when it happened.

The assertion message should spell out exactly what the author of the test was expecting to happen. When the test fails, the message should provide enough detail for the developer to understand exactly what the problem was. The time taken to write a well-crafted assertion message, is more than paid back when the assertion fails!

// assertTrue(actual.startsWith(expected));
// Asserts that a condition is true.
assertTrue( "Actual should start with expected but actual was '"
+ actual + "' and expected was '" + expected + "'", actual.startsWith(expected));

Conclusion

If you are writing unit tests, spend a couple of minutes reading over the reference manual to see what assertion methods you have to work with. xUnit Test Patterns breaks down the types of assertion into; Single Outcome, Stated Outcome, Expected Exception, Equality, Fuzzy Equality. Understand what each assertion does and more importantly, understand when and where to use assertion. Test failures become much easier to track down, when it’s clear what went wrong. Use the right assertion, use the assertion correctly and give it a meaningful message. It really is a simple as that, what could possibly go wrong?