Problem Description

Our build structure is pretty stable, but the exact content of the steps varies as we discover more smoke tests that we'd like to add to, or when we rearrange the location of these checks.

The CI servers I've used made this a rather cumbersome process:

• First, I have to leave my development environment to go to the build servers configuration of choice - most of the time it is a web interface, and for some it is a config file



• I have to point and click, and if it's a shell script, I have to make my modifications without syntax highlighting (for the config files usually take the shell command to execute as a string, so no syntax highlighting)



• If it's a web interface, I have (or had) no versioning/backup/diff support for my changes (config files are better in this aspect).



• If it's a config file, then I need to get it to the build server (we version control our config files), so that's at least one more command



• I need to save my changes, and run a whole build to see whether my changes worked, which is a rather costly thing.



• Most places have only one build server, so when I'm changing the step, I either edit the real job (bad idea) or make a copy of it, edit it, and then integrate it back to the real job. Of course, integrating back means: copy and paste.



• If the build failed, I need to go back to the point and click and no syntax highlighting step to fix the failures



• Last, but not least, with web interfaces, concurrent modifications of a build step lead to nasty surprises!

Normal development workflow

• I have an idea what I want to do



• I write the tests and code to make it happen



• I run the relevant tests and repeat until it's working



• I check for source control updates



• I run the pre-commit test suite (for dvcs people: pre-push)



• Once all tests pass I commit, and move on to the next problem

Quite a contrast, isn't it? And even the concurrent editing problem is solved!

Quick'n'Dirty Inversion of Control for builds

Disclaimer: the solution described below is a really basic, low tech, proof of concept implementation.

Since most build servers at the end of the day

• invoke a shell command



• and interpret exit codes, stdout, stderr, and/or log files

we defined the basic steps (update from version control, initialize database, run tests, run checks, generate documentation, notify) using the standard build server configuration, but the non-built in steps (all, except the version control update and the notification) are defined to invoke a shell script that resides in the project's own repository (e.g.: under bin/ci/oncommit/runchecks.sh). These shell scripts' results can be interpreted by the standard ways CI servers are familiar with - exceptions and stack traces, (unit)test output, and exit codes.

Benefits

• adding an extra smoke test doesn't require me to break my flow, and I can more easily test my changes locally and integrating it back into the main build means just committing it to the repository, and the next build will already pick this up



• I can run the same checks locally if I would like to



• if I were to support a bigger team/organization with their builds, this would make it rather easy to maintain a standard build across teams, yet allow each of them to customize their builds as they see it fit



• if I were to evaluate a new build server product, I could easily and automatically see how it would work under production load, just by:
  
  
  
  • creating a single parameterized build (checkout directory, source code repository)
  
  
  
  • defining the schedule for each build I have
  
  
  
  • and then replaying the past few weeks/months load - OK, I still would need to write the script that would queue the builds for the replay, but it still is more effective than to run the product only with a small pilot group and then see it crash under production load
  
  
  
  
  
  

Shortcomings, Possible Improvements

As said, the above is a basic implementation, but has served a successful proof of concept for us. However, our builds are simple:

• no dependencies between the build steps, it is simply chronological



• no inter-project dependencies, such as component build hierarchy (if the server component is built successfully, rerun the UI component's integration tests in case the server's API changed, etc.)



• the tests are executed in a single thread and process, on a single machine - no parallelization or sharding

All of the above shortcomings could be addressed by writing a build server specific interpreter that would read our declarative build config file (map steps to scripts, define step/build dependencies/workflows), and would redefine the build's definition on the server. By creating a standard build definition format, we could just as easily move our builds between different servers as we can currently do with blogs - pity Google is not a player in the CI space, so the Data Liberation Front cannot help :).

Questions

Does this idea make sense for you? Does such a solution already exist? Or are the required building blocks available? Let me know in the comments!

1. bridge on 2012/02/21 14:55:14: What we do at our build server is something similar to this:
   
   We have ant steps defined, e.g. build, run checkstyle, run unit tests. We have different Hudson jobs for the same branch, one for each purpose.
   
   Applying this environment for your scenario (tell me if I missed something from the problem) would look like this:
   - we could create a new job for the integration tests.
   - for changing the integration tests, we could write new tests and add them to the corresponding ant step. We could write the code and the ant xml in our favorite IDE. We could run them in our environment before committing it.
   - we can use the output of the build step in the same way we usually do. This is right now stop/go; but I think there are other ways to do this out there.
   1. Peter Zsoldos on 2012/02/21 19:41:11: Yup, your example achieves the same purpose as what I described. I have used the shell script in this example because a) I haven't worked on java projects in ages and I forgot it exists b) we have a number of scripts (both shell and django runscripts/management commands as well) that we reuse
      
      When you say multiple jobs, you mean for one project (branch) you have 4-5 jobs or one job with multiple parallel steps?
2. bridge on 2012/02/22 13:55:01: When I say multiple jobs, we have 2 jobs for each branch. I don't know if we can make Hudson smarter - this is how the experts set it up initially, and nobody ever questioned it.
   
   Having predefined scripts, as placeholders to invoke other scripts - that is poor-mans-ant: you have a layer of abstraction between the CI server and the scripts that are actually running. Say for instance, Hudson invokes the smoketest.sh, which invokes the low-level step, something like this:
   
   #!/bin/bash
   compile.sh
   dist.sh
   run-unit-tests.sh
   check-login-ui.sh
   check-the-crazy-rollback-use-case.sh
   
   So all you had to do would be add a new line to this shell/perl file.
   
   Or, in perl (or python, whatever) you can add any other logic
   #!/usr/bin/perl
   if (system("dist.sh") != 0) {
   exit;
   }
   if (system("run-unit-tests.sh") != 0) {
   exit;
   }
   #...
   
   Here you can add alternative and parallel flows too. A bit ugly, programmatic, but works. Even more, you can keep it clean if you keep the actual step's scripts and the flow's scripts in different files.
   
   SoapUI is not a CI solution, but still, it's something similar as it's a testing tool. They suggest to use scripts in order to define custom logic between the test steps. http://www.soapui.org/Functional-Testing/controlling-flow.html