How to test your automated tests

Olli Ahonen • Senior Consultant

One of the more uncertain things in software development is whether your automated tests actually catch the mistakes they're supposed to.

The short version

Embedded content:

The longer version

There's a trinity: the test fixture, test assertions, and business logic.

1. Test fixture

Test fixture is what you set up the universe to be before running a test case. This includes things like what you insert in a database, how you set up mocks to behave, and so on. For this post, I'm going to include the things you give as input to the application in the concept of the fixture.

Any meaningful change* in your fixture must make at least some of your tests fail. If they don't, your tests don't work like they should, or you have meaningless crap in your fixture.

So what if I have meaningless crap in the fixture?

No-one else wants to have anything extra in the codebase. Get rid of it.

2. Test assertions

Test assertions are you telling the computer what the program should output in a test case. This and that must be equal to 200, for example.

Any meaningful change in an assertion must make at least some of your tests fail. If it doesn't, your tests don't work like they should, or you have meaningless crap in your assertions.

Same deal as with the fixture.

3. Business logic

Strip away all the boilerplate and user authentication and whatnot, and you're left with the purpose of the program: the business logic.

Any significant change in the business logic must make at least some of your tests fail. If it doesn't, you need more test coverage. If you think coverage is already good enough, then that particular piece of logic isn't significant, by definition.

The takeaway

Use these three rules to verify your automated tests.

*) What's a meaningful change? That is highly context specific. I would define it as 'A change affecting how business rules are applied and thus affecting the relevant outputs of the program.' Let's try to clarify this with an example. If your business logic deals with, say, the outside temperature, you might want to set it to a certain value for some test cases. A change from 14.8°C to 13.1°C is not meaningful at all if the business rules only depend on whether water freezes outside or not (assuming atmospheric pressure at sea level). On the other hand, changing the outside temperature from 14.8°C to -25°C should most likely break your tests. (Unless you've cleverly constructed your test assertions so that they are calculated based on your test precondition values, which you shouldn't do—but that's a topic for another post.)