2024-02-21
* when writing unit tests
Info
blog adaption of my notes on unit testing.
Testers of the system, unit [sic]! You have nothing to lose but your (mock dependency) chains!
…in which we discover unit test philosophy and enlightenment. And may end up on a list for googling the communist manifesto.
Like most software development terminology, [unit testing is] very ill-defined, and I see confusion can often occur when people think that it’s more tightly defined than it actually is.
We need a common language for conversation. These are my own definitions.
| Term | Definition |
|---|---|
| unit | system that loses its identity and function when divided |
| interface | defined set of messages a unit accepts and emits |
| inputs | messages sent to the unit e.g. intake queues, runtime stack |
| outputs | messages sent from the unit e.g. output queues, runtime stack, includes exceptions |
| unit client | system that sends messages to or receives messages from the unit |
| side effects | any unit-affected change that is observable outside its defined interface, e.g. I/O |
| dependency | an external system the unit relies on to do work e.g. other units via message, the system clock |
| system under test (SUT) | set of units tested together; the set can be singular |
| unit test | program that executes and observes a SUT |
| solitary unit test | a unit test with isolated dependencies and inputs |
| sociable unit test | a unit test that relies on other units to fulfill some behavior |
| mock | a stand-in object for a dependency or input; for purposes of this page, mocks are automatically generated via DSL or framework and provide an API to make assertions on data it observes |
| integration test | test that observes side effects |
| functional test | test with user-visible effects and assertions |
| pure function | function or sub-routine with no side effects; the outputs are determined entirely by the inputs |
The fundamental question
Should a unit test’s scope be identical to the named unit under test, or is the unit better understood as “system under test” - unit plus dependencies?
Two schools prefer Solitary and Sociable unit tests, respectively.
Martin Fowler calls these the mockist and classic styles.
See section Why Paul is a Socialist.
The solitary tester asserts the unit under test should be observed in isolation with explicitly controlled inputs, outputs, and dependencies.
They believe it is possible to consistently layer abstractions to facilitate this style of testing.
Ergo, tests should prefer mocking out all dependencies and inputs.
any() and lenient() - the tests tend toward false signal.The sociable tester asserts it is better to include production code dependencies in a test rather than depend on an anemic approximation of the production code.
They believe nothing is a substitute for production dependencies.
Ergo, tests should default to executing production dependencies as much as possible, with mock implementations used as-needed to address execution time and side effect concerns.
The fundamental difference between the two styles is one of unit scope.
The solitary tester asserts:
A with a production dependency on B_1 (A -> B_1) …B_2 - a mock - is sufficiently representative of B_1 behavior that …A -> B_2 yield inferences about system A -> B_1The sociable tester asserts:
A -> B_2 are merely testing A -> B_2 at best, and performatively testing the mocks at worst.Mocks create an illusion of functional purity that does not exist in the real world.
But, but …
B_1 is a pure function?B_2 so tests for A are completely isolated?Is B_1 execution time slow?
If not, a deterministic B_1 is not inferior to a mock. Mocking out pure data or pure functions is as performative as it gets from both a practical and theoretical view.
Relying on end-to-end tests to cover the unintended blind spots created by mock-heavy unit tests is problematic.
Lemma
A fast set of “unit” tests that overlaps integration and functional test use cases is more valuable than a fast set of “unit” tests that only test pure functional properties.
A common argument against testing dependency code in unit tests is it encourages leaky abstractions. This is a fair criticism and where such implementation dependence impacts test reliability and speed, I believe it is worth providing a substitute implementation.
Outside that constraint, abstraction for the sake of it is performative.
Your mock object is a joke; that object is mocking you. For needing it.
[Tests that heavily rely on mocks] are reliable and fast, but they tend to “lock in” implementation, making refactoring difficult, and they have to be supplemented with broad tests. It’s also easy to make poor-quality tests that are hard to read, or end up only testing themselves.
For performance or test reliability, it’s not always possible to use production code in unit tests. What should be done when this is the case?
For an alternative to mocks, see Nullables in Testing Without Mocks. These are typically hand-crafted and do not rely on a mocking framework. Instead they require carefully constructed interfaces around side effects.
Side-effecting dependencies (clients for DBs, queues, workflow engines, etc) write all their logic with a narrowly defined interface abstracting the external system. There are two implementations of that interface
Some might even call this a mock.
The important differences from a mock are:
Related tags:
email comments to paul@bauer.codes