Displaying Build Numbers in Grails Apps

Being a fan of Continuous Delivery, identifiable builds, and Continuous Integration: I like to deploy web apps with a visible build number, or some other way of identifying the version. For example, having the build number on the login screen for example. In the Maven/Java world, this is straightforward. Or at least I know the idioms. I struggled with this a bit while working on a Grails app,  and wanted to share my solution. There may be other, better, solutions, but the ones I found approaches that didn't quite work they way that I'd hoped.

My requirements were:

• To display a build number from my CI tool, where the number was passed in on the command line. In Bamboo, for example you might configure a grails build as
  -Dbuild.number=${bamboo.buildNumber} war
• To only change build artifacts and not any source files.
• To not misuse the app version, or change the names of any artifacts.
• To be simple and idiomatic.

I realized that that Grails itself changes the application metadata (application.properties) when it does a build. So I modeled this approach after code in one the gant scripts in the Grails distribution.
The steps are:

• Hook into a build event to add a property in to application.properties.
• Access the property using the gsp tag.
• Pass the value into the the grails war command using a Java property.

To create the event hook, create a file in your project/scripts directory called _Events.groovy that contains a method like:

eventCreateWarStart = { warName, stagingDir ->
    def buildNumber = System.getProperty("build.number", "CUSTOM")
    println("Setting BUILD_NUMBER to "\+ buildNumber)
    ant.propertyfile(
       file:"${stagingDir}/WEB-INF/classes/application.properties") {
         entry(key:"build.number", value:buildNumber)
   }
}

Then you can access the property with a GSP tag like this

<g:meta name="build.number">


And you can now display build numbers (or any other properties that you want to pass in from you CI system) in your Grails app. This worked for me with Grails 1.3.7. The only down sides are:

•  This works on a per-project basis (That is probably easily fixable)
• The properties that we set are hard coded. (this could be fixed with a convention  that iterates through all of the system properties that start with "build" for example).

I welcome any suggestions for improving this, or alerts that I'm using the entirely wrong idiom :)

Things about Release Management Every Programmer Should Know

As I mentioned earlier I was privileged to contribute to the book 97 Things Every Programmer Should Know: Collective Wisdom from the Experts. In addition to the contributions about coding and design, I was pleasantly surprised to see the number of items that relate to release management. While I've long been interested in how to build architectures and processes that make deploying and releasing software easy, I sometimes get the impression that these items were often though of necessary evils that could be done at the end, often by the someone who isn't doing "more valuable work." Much like awareness of agile software development made it obvious that testing and quality assurance activities work best when they are integrated throughout the development lifecycle, agile has also made it more obvious why build and release engineering is something to work on as you go. This makes a lot of sense, as ease of release is closely tied to the physical architecture of the system, and your build process defines the physical architecture.

Some of the posts of interest are the following, though I could have added others than related to testing as well.

1. Deploy Early and Often by Steve Berczuk
2. Install Me by Marcus Baker 
3. Keep the Build Clean by Johannes Brodwall
4. One Binary by Steve Freeman
5. Own (and Refactor) the Build by Steve Berczuk
6. Put Everything Under Version Control by Diomidis Spinellis
7. Step Back and Automate, Automate, Automate by Cay Horstmann

Aside from being excited that programming experts understand the importance of build, release, and installation mechanisms, I was also excited to see that one topic that people associate with SCM wasn't mentioned: Branching.

Branching is a useful tool when used in the right context, but more often than not, branching is used as a way to avoid issue rather than to address them. Rather than branching because there is a true divergence in the code, we branch to avoid breaking existing code. The problem is that doing so simply defers a cost. Sometimes deferring the cost makes sense. Often it's better to invest in the techniques you need to enable incremental change while keeping the codeline working.

97 Things Every Programmer Should Know is about much more than just coding, or just release management. And that's the point: programming is a multi-faceted skill and to write good code, you need to know about more than just writing code.

Review of Adapting Configuration Management for Agile Teams: Balancing Sustainability and Speed

Adapting Configuration Management for Agile Teams: Balancing Sustainability and Speed is a book that bridges the gap between tradition SCM and Release Engineering and Agile teams. Mario asked me to be a reviewer of the draft manuscript and I knew that Mario had great experience in establishing SCM processes in larger organizations, and that he was also a strong advocate of Agile methods. I was pleased to discover a book that, while being able to help those from a traditional release management background adapt their processes to support agile, also addresses the needs of those transitioning to agile who can benefit from using appropriate SCM processes to help them.

This book will help you to understand how SCM can be an enabler for agile, and will also help you to understand how to fulfill the SCM definition of Identification, Control, Status Accounting, and Audit and Review while still being agile.

This is what I said about the book on my books page:

This book is a good guide to both CM and Agile principles, and it demonstrates how to use software configuration management to enable your team to be more agile. This book can guilde you to understanding how to manage releases in an agile environment, and how to apply basic CM concepts like build and branching successfully. While not a replacement for a book on your agile method, this book is a primer on agile for those with a traditional release management background, and and a primer on CM for those who understand agile. After reading it you will have enough background to be productive, and a good sense of what you need to learn more about. In addition, this book covers topics such as how to leverage cloud service providers for infrastructure, how to leverage SCM to make off-shore development less painful, and how to evolve your SCM process in an agile (incremental) fashion. With a good structure that allows you to navigate the book quickly, and a good use of metaphor to describe concepts, this book will help a release managers, project managers, developers and architects use the SCM process to get the most out of their agile teams.

If you are transitioning to agile and want to know how your release management team can help, rather than hinder you, give this book a look.

Continuous Integration of Python Code with Unit Tests and Maven

My main development language is Java, but I also some work in Python for deployment and related tools. Being a big fan of unit testing I write unit tests in Python using PyUnit. Being a big fan of Maven and Continuous Integration, I really want the  Python unit tests to run as part of the build. I wanted to have a solution that met the following criteria:

• Used commonly available plugins
• Keep the maven structure of test and src files in the appropriate directories.
• Have the tests run in the test phase and fail the build when the tests fail.

The simplest approach I came up with to do this was to use the Exec Maven Plugin by adding the following configuration to your (python) project's POM.

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>exec-maven-plugin</artifactId>
  <executions>
   <execution>
    <configuration>
     <executable>python</executable>
     <workingDirectory>src/test/python</workingDirectory>
     <arguments>
      <argument>unitTests.py</argument>
     </arguments>    
     <environmentVariables>
       <PYTHONPATH>../../main/python:$PYTHONPATH</PYTHONPATH>
     </environmentVariables>
    </configuration>
    <id>python-test</id>
    <phase>test</phase>
    <goals>
     <goal>exec</goal>
    </goals>
   </execution>
  </executions>
 </plugin>

This works well enough. Setting the PYTHONPATH environment variable allows your pyUnit tests to find the modules you are building in the project. What's less than ideal is that, unlike other maven plugins, the person running the build needs to have python installed and configured correctly. (You can allow for some variations between environments. And if you have a developer on your project who doesn't use python, and doesn't want to there is a property you can set on the exec plugin to skip the tests, so in the end only those who use python, and the continuous integration server, need the correct things installed.

This may be obvious to some, if not many, but in case anyone is looking for an answer to how to run unit tests as part of of your maven build, I hope that this is helpful.

Versions in Maven and Source

I'm a big fan of Maven, a build (and project) management tool. When you are working with Maven, each artifact  that you develop (jar, or war file for example) has a version that's distinct from version in your SCM system.  The Maven Book has a good discussion about how versions are managed, but some there are often questions on projects about how to use Maven versions when you also have the SCM version. "SCM Version 6453" gives more information that "Artifact version 1.1" for example. yet we have 2 version numbers to manage. Here's one approach that works well for simple projects.

I'm assuming that you know the basics of how to specify dependencies in Maven. If you need a quick intro to Maven, see the Maven Book.

If you are developing a project that has artifacts that external clients will incorporate with Maven, you need to change the artifact version with each release, as you  specify the artifact version in the dependency element in your pom. If you are just using maven to develop components that will never show up in an external repository, adding the artifact version dimension to the SCM version can appear to add little value at the expense of some overhead, unless you have a clear model of how the two interact.

Many teams using maven are using it to manage external dependencies, but their own artifacts are not published to external maven repositories. A simple webapp, for example may have one war file, and have some external dependencies, and for support and validation purposes, you can use the build number or version number. There are a couple of ways to figure out the source version that you built your webapp from.

If you use a continuous integration tool like Bamboo, you can pass in the build number on the build command line by adding -DbuildNumber=${bamboo.buildNumber} to make the property buildNumber available to your project. You can make this available the application, so that you can see the build number on a login screen or about page. From the build number you can infer the SCM version. Or you can often ask the CI system the SCM version number. For exampe, in Bamboo, you can specify -DscmRevision=${custom.svn.revision.number} if you are using subversion as your SCM.

You can also use the build-number maven plugin to embed the version in your war manifest or make it available to your flex application, for example.

Regardless of how you get the revision number the revision number still gives you enough information to identify what code you are working with. The Maven convention to keep in mind is that SNAPSHOT artifacts are for active development, and non-SNAPSHOT versions are for released artifacts, so while you are working on a new projects, you'd be building a 1.0-SNAPSHOT artifact, and when you are ready to create a release branch, you would change it to 1.0.

So when should you change the project version in maven?  Here is a simple approach: One maven major/minor version per codeline. Incremental versions stay on the same codeline.

So if you are using an active development line pattern (see Software Configuration Management Patterns) this means that each Release Line has a maven version, and your Mainline is use to create the SNAPSHOT version of the next release artifact.

For example:

• Start mainline development with 1.0-SNAPSHOT version
• When you have a release:
• create the release branch and change the artifact to 1.0.
• Change the mainline version to 1.1-SNAPSHOT (or whatever the next active version is)

To do support of the 1.0 release you have 3 choices:

1. Just work with the 1.1 artifact version and use build numbers to distinguish between the current release and the support release.
2. Branch off of the 1.1 codeline, change the version to 1.1.1-SNAPSHOT, and merge back when the 1.1.1 release goes out.
3. Change the version on the support branch to 1.1.1-SNAPSHOT and don't branch. Change the version to 1.1.1 when you are done, and create a tag for the release.

I prefer the third option as the simplest, and best option, assuming that there is only one stream of development work for the maintenance branch. It balances the need for identification, with the desire to minimize overhead.


To summarize:

• New development for M.N-SNAPSHOT versions on trunk.Use SCM version & Build numbers to identify components internally.
  
• M.N versions on their own branches.
• Use M.N.n versions for maintenance releases; SNAPSHOTS for development, remove the SNAPSHOT when done. 

I'm interested in hearing what others may have done to reconcile SCM and Maven artifact versions for internal components.

Begin with the End in Mind: Deploy Early

In many projects I've worked on the task of developing a deployment process is deferred until close to the release date. This is a bad idea. Deferring deployment-related decisions adds risk to a project as deployment to a target environment exposes development time assumptions and operational requirements that may need to be addressed by the engineering team before an application is released.

I suggest that a project develop (and use) a process to deploy the software to the target platform in the first iteration. Starting on deployment early gets you closer to the agile goal of shippable code at the end of each iteration.

Deployment is important, and something for which there should be a story in your process early on because:

• Deployment is the first thing a customer sees, but it's only an enabler, it doesn't directly deliver value, so you'd like for the process to be efficient. Incremental improvement of the process is a great way to make it work well.
• Design decisions can have an impact on the deployment and configuration process, and deployment models may suggest a different design
• Thinking about deployment early means that you'll have an identifiable, repeatable process. this will make your testing process more valuable.

By a deployment process I mean the mechanism for delivering your software to, and making it usable by a customer (or intially to a QA Team), including any required configuration steps. This process can be as manual or as automated as you like, ranging from a zip file and a README, to a fully automated installer. You can start simple and automate more as you get feedback.

Developing a deployment process early can look like Big Design Up Front if you start out with too detailed of a process that involve guesses about things you just don't know at the beginning of a project. But remember: you can change the process as you learn. Start with a simple process and change and refactor the process as you go to make it more suitable to the target user's needs. You may discover that a manual process works well enough, or you may find that adding some automation make the process simpler and lower risk with little added cost. If you have a test team that is using the deployed artifact, you may find that you can leverage automation developed to support test related deployments and use it in your customer process.

Deploying early and often can also improve your architecture too, as deployment exposes issues about the system architecture that are easy to overlook in an development environment. How the application is designed affects how it can (and should) be packaged (and the reverse is also true). While another groups may be responsible for the operational aspects of deployment, the engineering team is responsible for making the process work in a real environment. Consider configuration as just one example. Using a build tool like Maven or Ant, it's a simple matter to keep a number of configuration files in synch by using filtering, and passing a property in to the build. Once you deploy a package that needs to be configured at a customer site, the fact that there are now many configuration files that need to be edited in the same way becomes an obvious source of wasted effort and possible error.

Even seemingly trivial issues such as how to configure logging in production to debug a problem can influence, the choice of logging framework. The sooner you see the problem, the easier it is to fix with less risk. Michael Nygard 's book Release It, covers some of these issues and is an excellent resource for information about how to build software that can be deployed into production effectively.

Like all things agile, you can start small, and take small steps towards the end result. They key things to consider are:

• Packaging: have the build generate a deploy-able package. This can be used for Integration Testing, demo environments, and the like.
• Configuration: understand what parameters need to configured and how to configure them. The first pass could be build properties that are filtered into configuration files. Move towards understanding what needs to be configured at deploy time at a customer site.
• Requirements of the target platform and operations teams: How automated does the process need to be?
  

Even if you don't know all of the details of the production deployment, frequent early developments help you to understand what issues you need to decide, and which you can defer. And if you really do understand what you need to do, you may as well give it a try sooner rather than later. At best you'll be validated. At worst, you'll have lots of time to address issues.