A Quick Cost/Benefit Perspective of Unit Tests

I work in a business domain—the sport-tech—in which business logic is relatively stable and performance is much more about speed than scalability. Furthermore, the user experience has to be rigorously bullet-proof. The sport-tech is a field in which things must just happen no matter what. The show must go on regardless.

Just me

We don’t do unit tests, if not occasionally. Integration tests, much more relevant for our purposes—are not even structured and repeatable over and over again via scripts. We see it like we are actors on stage preparing for the show. We study our script, we play it in front the mirror (aka, “it works on my machine”) and we go through a series of rehearsal sessions. How many? Until we all feel confident it would work in the real world. In doing so, if we find any bugs we fix and deploy on staging.

So according to a large share of software experts out there (half the audience?) we’re old and unprofessional dinosaurs. Got it!

Unit tests are extra pieces of code to be written, maintained, tested and run with the sole purpose of ensuring that a given class, and ideally every single method of every given class, works as expected. To be fair, for every method you should be testing every possible combination of input parameters (including zeros, empty strings, null pointers) and weird assignment of values. All this even for a plain stupid method that does just one thing and is invoked from only one place.

On the other hand, this is 100% coverage, baby!

Speaking of that, you can’t just take coverage as a metric of code quality, meaning lack of (patent) bugs. Who guarantees that the hundreds of micro-methods someone wrote are relevant? Let’s take the most common definition of 100% coverage: all statements in the codebase are tested at least once. What if, I test a given statement once but miss another, more business relevant, case?

Furthermore, if the business logic changes, tests must be updated and retested and reconsidered and reevaluated and the mythical 100% coverage recalculated.

Unit tests are a significant development cost. Period.

The interesting question is, when is this cost worth? Unit tests provide some value that can be summarized as below:

• Ensuring a given behavior is obtained
• Enabling safer refactoring
• Code self-documentation

Unit tests are just one way to get this value. If you can reasonably obtain the same benefits otherwise, why spending time on unit tests?

In our environment the first two points go hand in hand. When we change some feature, we just test enough to feel confident it works or it can be repaired quickly. Deep knowledge of the business domain is a must here and—admittedly—without that unit tests are the minimal evil.

The point of self-documentation is trickier. We’re a small team and turnover didn’t touch us just yet. The team shares technical information and the complexity of the functionality reflects the business domain. But the prerequisite is that we have strong business domain knowledge. Which is not a secondary point in software development. We’ve experienced increase of productivity in developers—especially senior devs—only when they matured enough awareness of the domain.

It goes without saying that outside this boundary, unit tests are an acceptable way to make sense of the code you, as a company, receive whether as a contribution, pull request or outsourcing.

The key takeaway of this rant is, don’t take anything, including unit tests, too religiously. Like anything else, they bring some value and come at a cost. Finally, unit tests are code. As such, they could be bugged and may not deliver the expected behavior.

Not because you have tests your code works.