Green Build Surprises

Recently while working on our build process, I ran into an unexpected surprise - the build server reported no error (i.e.: it was displaying a green status, since all tests were passing), yet there was a major issue - not all of our tests were run!

Given that the situation was caused by working on the build itself, the issue was discovered in time (though somewhat accidentally during exploratory testing). However, this was a nice reminder of the problem with the "unknown unknowns" - now that we know of this issue, of course we have configured alerting for the case when the current build ran less tests than the build before. But I wonder how many times we may have released on a green build with regression bugs...

Asking some others and looking back on my past experiences, this topic hasn't come up, so while I hope it is obvious and not news for you, dear reader, I blog it just in case I wasn't the last to get the memo...

And it turns out that every now and then we do mess up and commit code that makes some tests invisible to the test runner... So it wasn't just a one-off thing!

What do you think? I would love if you would leave a comment - drop me an email at hello@zsoldosp.eu, tell me on Twitter!

Posted on in continuous integration, monitoring, software by

Share this post if you liked it - on Digg, Facebook, Google+, reddit, or Twitter

Your email address