The sections "If you think you're using enough assertions, you're wrong" and "Bonus: Sanity Checking" can be considered a subset of "Design by Contract"[0].
Regarding:
However, comments can become out-of-date, and you might
not be lucky enough to stumble across the right comment
before you embark on a refactoring adventure.
Not if comments are treated as first-class citizens, just as important as the code change(s) which make them out-dated. Better still is to ensure unit tests are "executable documentation"[1].
Regarding "Prefer end-to-end tests":
In my experience, this leads to unit test atrophy, degrading system verification to rely solely on integration tests ("end-to-end tests"). This has often resulted in higher coupling and always resulted in interminable build times.
I think it depends a lot on the nature of the code.
I prefer to use unit tests to probe tricky parts of the code. It's basically like a recyclable debugging session. Sometimes with an actual debugger attached, but otherwise just an assertion is fine. If I've had reason to investigate the code once there's probably a reason to keep that inspection hatch because bugs rarely come alone.
Unit tests for the sake of unit tests is kinda stupid though and can actually border on a sort of technical debt. If no matter what you change, you need to fix/disable a bunch of unit tests, failing tests becomes expected and it becomes very easy to fall into just fixing the assertions or disabling the test when that happens without actually thinking about why it fails. Not even mentioning how much this sort of code base will bog down the rate of change.
That said, it's a tricky balance. You can definitely have both too many and too few unit tests.
It does depend on the nature of the code. My rule of thumb is integration code -> integration tests work better. Algorithmic/calculation code -> unit tests work better.
A debugger attached to an end to end test probably better if you're testing integration code. With a unit test you might have to do a lot more extra work to, e.g. construct an exact replica of a request object or a database object or something else that comes "for free" when you precisely mimic the software being used in the real world.
That said, if mimicking the real world in your code is as simple as 'assert my_function("some input")' and checking the output then yea, unit tests are way quicker.
I beg to differ - that’s a whole new layer of bugs and race conditions. And I’m extremely sceptical you can speed them up enough that I can run them every couple of seconds on every micro-change.
It is very hard (perhaps impossible) to write good unit tests around code that wasn’t designed to be unit tested. This is why the advice is to write the test first, even if you then throw away half the tests.
In my tests, I’m looking for something that tells me where the problem is quickly. End-to-end tests can’t do that - they will highlight that a happy path is broken and thus the build is broken. They usually fail to cover many unhappy paths, and they usually highlight a symptom but not the root cause.
> Unit tests will often just mirror the behavior of the code. This is not useful.
Can you go into that more? I definitely agree that end-to-end tests give some really strong benefits and verification, but I'm a bit lost on the "mirror the behavior of the code" comment. I like that if I write a function and then have associated unit tests for all of the case I'm considering (which probably won't hit all at first), I will have confidence that that code functions as intended and if something changes that makes that code function differently (when not intended), we'll know right away by test suite.
End to end tests give you more "real world" test, but it probably won't give you a good granular view of the issue. I don't see what the issue is with unit tests and your "mirror the behavior" comment, but I'm interested if you have the chance to clarify.
For self contained algorithmic or calculation code with simple inputs and outputs, TDD with unit tests really shines. If your app is mostly lower level code that does a lot of complex "thinking" especially with simple inputs and outputs then you probably should have mostly unit tests.
Integration/E2E testing algorithmic/calculation on that kind of code leads to tests that could take 100x longer to run - unnecessarily.
However, it's very common to have very little code of this type. If an app is very integration code heavy (it sends/receives messages, does CRUD, calls APIs, sends emails, etc.) then the opposite is true - you only want to have integration / end to end tests.
Unit testing that kind of code leads inevitably to mimetic tests.
End to end tests can make it hard to get a good granular view of an issue. I agree with this. However, this can be solved by integrating high quality debugging tools to the end to end tests - automatically grabbing screenshots, logs, external API calls, etc.
> Unit tests will often just mirror the behavior of the code. This is not useful.
Then they are characterisation tests, not unit tests. Which may or may not act at the unit level.
As for whether characterization tests are useful; they're extremely useful for, well, characterising existing code behaviour.
But they suck at detailing pre-code specification; that's what unit tests are for (at least at the unit level).
Now I'm not naive enough to believe most people adhere to this distinction appropriately. It's true that people sometimes treat X as Y in practice. But that's one thing, and a totally different thing to complain about X because people do Y instead.
Agree a lot. Just want to say, since the article also touched on this, that parallelization of test execution can again introduce nondeterministic failures...
They make no assumption as to the implementation of the thing, they test that the observable behaviour is correct and that’s it. You could refactor the whole codebase without touching your tests: the tests are decoupled from the runtime code
I view it differently. I view all my code through end to end tests and use very few mocks. If I need a mocks/fakes for things that aren’t external to the code (eg I/O to network services), it usually means I haven’t designed my code modular enough and refactor. The reason I view it that way is that all fakes (of which mocks are a subset) means you’re reimplementing your real code with a reimplementation specific to your test suite which means your test suite isn’t providing coverage and now you have two codebase to maintain hidden in one which grows very very difficult and time consuming to do (velocity of any complex refactor massively slows down). The tests themselves I view as a layered thing where each layer of the program has matching tests. So as I add a primitive, I’ll add tests to that piece + how it integrates into the next layer (if there’s observable changes). Sometimes primitives will cross-cut all the way into the end API/UI. In those cases there’s potentially new tests I have to add at every layer to make sure it’s stable all the way down.
This approach has a strong advantage aside from maintaining only one codebase and your tests being much easier to refactor in coordination with code changing along the way in smaller ways. It makes it easier to find and figure out bugs because if it’s not visible at a layer it means either you’re missing tests immediately below that layer OR your integration in that layer is broken. As you move up the layers, it’s often more complicated to debug issues at much lower levels. So if a very high level test fails, you can do a search through lower levels writing reduced test cases that reproduce your observation. That can be a good compliment in addition to reasoning about behavior from reasoning about what the failure might be from symptoms (which often can be faster if you understand your codebase but trickier problems that you can’t figure out and are stuck on can be tackled through a linear search or maybe even bisected search if your clever).
The problem with most tests is that they tend to capture “happy paths” you’ve thought about. Even though negative tests (which are hugely important to make sure to add about how you expect error scenarios / failures to behave) are hugely important, they have a similar characteristic. I like to use property check tests and fuzzing techniques more broadly to improve my coverage when I can (can get expensive in the test suite so it can sometimes be worth balancing testing it on every commit vs post-merge in the background).
Still. It’s a problem. Mutation testing feels like another important tool. I’ve not had success getting Stryker (mutation testing) running on any TypeScript project I maintain which is frustrating (I got it running kind of but it seems to struggle/I haven’t figured out how to generate a proper report because I’m using the workerd version of Miniflare where the code runs out-of-process). But mutation testing as a concept seems like a fantastic way to figure out code coverage and where you might have gaps that traditional testing / fuzzing (property checking is a subset) isn’t catching. We all have blindspots and mutation testing is a far more defensible position that your test coverage is good vs traditional techniques we’re taught in school like branch and line coverage (which in my opinion are extremely poor and misleading ways to measure code coverage).
The sections "If you think you're using enough assertions, you're wrong" and "Bonus: Sanity Checking" can be considered a subset of "Design by Contract"[0].
Regarding:
Not if comments are treated as first-class citizens, just as important as the code change(s) which make them out-dated. Better still is to ensure unit tests are "executable documentation"[1].Regarding "Prefer end-to-end tests":
In my experience, this leads to unit test atrophy, degrading system verification to rely solely on integration tests ("end-to-end tests"). This has often resulted in higher coupling and always resulted in interminable build times.
0 - https://en.wikipedia.org/wiki/Design_by_contract
1 - https://softwareengineering.stackexchange.com/questions/1546...