Green Build Surprises
Recently while working on our build process, I ran into an unexpected surprise - the build server reported no error (i.e.: it was displaying a green status, since all tests were passing), yet there was a major issue - not all of our tests were run!
Given that the situation was caused by working on the build itself, the
issue was discovered in time (though somewhat accidentally
during exploratory testing). However, this was a
nice reminder of the problem with the "unknown unknowns" - now that we
know of this issue, of course we have configured alerting for the case
when the current build ran less tests than the build before. But I
wonder how many times we may have released on a green build with
regression bugs...
Asking some others and looking back on my past experiences, this topic hasn't come up, so while I hope it is obvious and not news for you, dear reader, I blog it just in case I wasn't the last to get the memo...
And it turns out that every now and then we do mess up and commit code that makes some tests invisible to the test runner... So it wasn't just a one-off thing!
