Introducing Examples

31 August 2013

This weekend I was working on Seaturtles, a toy implementation of the Raft consensus algorithm. Consensus algorithms are notoriously tricky to implement, so I’ve been writing a lot of tests as I go. At first, I was using the testing package from the Go standard library. The package provides a solid set of primitives and I’ve been happy using it on its own before; all the tests in Braintree Go, for example, are written with it. However, something about the tests I needed to write for Seaturtles made the flaws of the testing package particularly annoying, so I decided to write something to help: Examples, a lightweight behavioral testing library.

Examples provides a descriptive layer on top of standard Go tests – this makes it easy both to organize your tests and to document exactly what assertions about the code under test you want to make. I find this makes it much easier to reason about your test coverage and write clean, focused tests.

Here’s a pair of tests from before I refactored Seaturtles to use Examples:

func TestAppendEntriesRejectsEntryLowTermCalls(t *testing.T) {
    follower := createFollower(1, 5)
    appendEntryCall := AppendEntryCall{LeaderId: 2,
      Term: 3, PreviousLogIndex: 1, PreviousLogTerm: 1}

    response := follower.AppendEntry(appendEntryCall)

    if response.Success {
        t.Error("Follower accepted AppendEntry call with lower term than
          its own.")
    }
}

func TestAppendEntriesSendsCurrentTermWhenRejecting(t *testing.T) {
    follower := createFollower(1, 5)
    appendEntryCall := AppendEntryCall{LeaderId: 2,
      Term: 3, PreviousLogIndex: 1, PreviousLogTerm: 1}

    response := follower.AppendEntry(appendEntryCall)

    if response.Term != 5 {
        t.Error("Follower did not respond with its own term when
          rejecting a call")
    }
}

There’s a lot of suboptimal stuff here. First of all, the test names are extremely verbose and borderline unreadable, in part because they repeatedly describe the state and method call. Worse, a lot of information they contain is repeated in the error message of the test, often at the expense of more specific information. For example, in the second test, a better error message would contain the expected and actual value of response.Term.

With the examples package, we can both organize the tests better and get more informative error messages, all while reducing the amount of test code. Here’s the new code:

func TestAppendEntriesWithBadClient(t *testing.T) {
    e.When("the calling node has a lower term than the receiver node", t,
        e.It("rejects the request", func(ex *e.Example) {
            follower := createFollower(1, 5)
            appendEntryCall := AppendEntryCall{LeaderId: 2,
              Term: 3, PreviousLogIndex: 1, PreviousLogTerm: 1}

            response := follower.AppendEntry(appendEntryCall)

            ex.Expect(response.Success).ToBeFalse()
        }),

        e.It("responds with its own term", func(ex *e.Example) {
            follower := createFollower(1, 5)
            appendEntryCall := AppendEntryCall{LeaderId: 2,
              Term: 3, PreviousLogIndex: 1, PreviousLogTerm: 1}

            response := follower.AppendEntry(appendEntryCall)

            ex.Expect(response.Term).ToEqual(5)
        }),
    )
}

In addition to being a few lines shorter, these tests will fail with much better error messages thanks to the quiz library by Ben Mills. More importantly, though, the visual organization of the code here makes it obvious to the reader that these are two tests about the same situation. Before, you’d have to read the test setup carefully in order to realize that. Now, it’s implied by the fact that the tests are grouped together.

Tests are useful because they provide a verifiable description of how the code under test behaves. That description is only useful, however, insofar as it is easy for humans to read and understand. I hope that Examples will allow people to write tests that are shorter, more comprehensible, and ultimately, more useful.

(If you’re interested in behavioral testing with Go, you should also check out Mao and Zen.)