Tips, Habits, Customs and Agility

I was listening to a Planet Money Podcast about tipping recently. According to the story, the custom of tipping has persisted even though the reasons that the practice was established no longer apply. According to the Planet Money story both waiters and customers think that tipping is useful, evidence to the contrary aside. The discussion led me to think about practices some teams do in the name of following a process.

Habits and routines are useful because they free you to focus on the important tasks.  Rituals and processes take a somewhat irrelevant decisions out of your hands, and conventions make it easier for others to understand code and other artifacts.  And when you are starting a new approach to work, following the rules by rote can help you understand the method. But circumstances change, and when the reason for the ritual has changed, there are a few possibilities:

• The practice is still useful, for reasons you didn't anticipate when you started it.
• The practice adds less value, but it doesn't add any overhead so there is no harm in continuing.
• The practice leads the team to spend energy addressing a problem that does not exist any more and causes waste.

To avoid instances of the last case that take up too much of you time, it is important to periodically reflect on why you are doing what you are doing, and it it still makes sense in your environment.  As your team adopts new tools, new practices, take time to think about whether what you are doing still makes sense. Periodic retrospectives are one way to capture this, so it's important to set aside time to review how you work. (And even to set aside time to review how you do retrospectives!)

Lots of teams (and individual) follow process  steps with good intentions, yet they end up causing unnecessary overhead. For example coding conventions that made perfect sense in the days of fixed length variable names, and text editors that add overhead when you have refactoring IDEs and follow clear naming conventions.

The podcast also reminded me of  one of of my favorite Gerry Weinberg quotes (which I referenced in Software Configuration Management Patterns):

 Survival Rules are not stupid; they are simply over-generalizations or rules we once needed for survival. We don't don't to simply throw them away. Survival rules can be transformed into less powerful forms so that we ca still use their wisdom without becoming incongruent

 (from Quality Software Management: First-Order Measurement)

Traditions and habits have their place. But if you are doing something simply out of habit you can benefit from acknowledging that fact.

Better: More Parallels between Medicine and Agile Software

I recently finished reading the book Better: A Surgeon's Notes on Performance by Atul Gawande. Having read The Checklist Manifesto: How to Get Things Right  (which I commented on earlier) I expected to learn not just about medical practice, but also about parallels between medicine and software development. Much of what Gawande says about performance in medical environments has can be said to apply to agile software teams, in particular, the need to focus on core practices, the value of retrospectives, and the value of generalists.

A discussion of how hand washing, a simple technique that is vital to infection control but that also requires culture change to implement with the discipline required to be effective, brought to mind how the challenges teams have being agile often center on the challenges of having teams begin to apply basic practices, without customization.

In the discussions of Polio vaccination and malpractice were excellent non-software examples of how we can benefit from retrospective practice, especially the prime directive, which reminds us that to improve we can't focus on blame, but rather on how decisions we made and how to make them better. A discussion of medical practice in war zones provided excellent anecdotes to relate when someone says that they can't gather data to measure performance and thus improve;  Gawande relates how combat doctors, those with the best excuses to not have time to do "paperwork" somehow find time to enter the data needed to track and improve the care of soldiers, while gathering data in hospitals seems to be hard.

In a profession where specialization is valued, Gawande discusses time spent in India, where understaffed hospitals mean that surgeons can't afford to specialize and yet get do well by collaboration, cross-training, and discussion; in short, how "generalizing specialists" help to eliminate roadblocks and help get the most value from a team.

Like all metaphors and comparisons, there are gaps. Doctors and Software Developers do different things, and work under different constraints. But by focusing on the differences you miss an opportunity to learn from those around you. This, to me, is the key lesson in Better.

Displaying Build Numbers in Grails Apps

Being a fan of Continuous Delivery, identifiable builds, and Continuous Integration: I like to deploy web apps with a visible build number, or some other way of identifying the version. For example, having the build number on the login screen for example. In the Maven/Java world, this is straightforward. Or at least I know the idioms. I struggled with this a bit while working on a Grails app,  and wanted to share my solution. There may be other, better, solutions, but the ones I found approaches that didn't quite work they way that I'd hoped.

My requirements were:

• To display a build number from my CI tool, where the number was passed in on the command line. In Bamboo, for example you might configure a grails build as
  -Dbuild.number=${bamboo.buildNumber} war
• To only change build artifacts and not any source files.
• To not misuse the app version, or change the names of any artifacts.
• To be simple and idiomatic.

I realized that that Grails itself changes the application metadata (application.properties) when it does a build. So I modeled this approach after code in one the gant scripts in the Grails distribution.
The steps are:

• Hook into a build event to add a property in to application.properties.
• Access the property using the gsp tag.
• Pass the value into the the grails war command using a Java property.

To create the event hook, create a file in your project/scripts directory called _Events.groovy that contains a method like:

eventCreateWarStart = { warName, stagingDir ->
    def buildNumber = System.getProperty("build.number", "CUSTOM")
    println("Setting BUILD_NUMBER to "\+ buildNumber)
    ant.propertyfile(
       file:"${stagingDir}/WEB-INF/classes/application.properties") {
         entry(key:"build.number", value:buildNumber)
   }
}

Then you can access the property with a GSP tag like this

<g:meta name="build.number">


And you can now display build numbers (or any other properties that you want to pass in from you CI system) in your Grails app. This worked for me with Grails 1.3.7. The only down sides are:

•  This works on a per-project basis (That is probably easily fixable)
• The properties that we set are hard coded. (this could be fixed with a convention  that iterates through all of the system properties that start with "build" for example).

I welcome any suggestions for improving this, or alerts that I'm using the entirely wrong idiom :)

Using Issue Tracking Systems Well

Many of us in the agile community have mixed emotions about issue tracking and issue tracking systems. Tracking is good, but issue tracking systems can become time sinks if not used well. I shared some thoughts on this topic in an article on Sticky Minds.com. Enjoy, and please comment (either here, or on the Sticky Minds site.)

Recent Writings (Dev Ops and Testing)

I recently wrote a couple of articles that might be of interest to those who read this Blogs.

On Sticky Minds.com, (a TechWell community) I wrote about unit testing, and how to overcome the inertia that stops people from testing in The Value of Really Dumb Tests

I also wrote an article for  CM Crossroads that gives an overview of DevOps.

If you read the articles and find them useful (or not) please comment here at the article site.

Continuous Delivery

 If you didn't already know that the key to reliably deploy quality software is to take a cross-functional, full-lifecycle approach, Jez Humble and David Farley's book Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation will help you to understand. Much like Jim Coplien describes in  Lean Architecture: for Agile Software Development the "secret" to successful lean projects is to work with "Everyone, All Together, from Early On."

While the authors have experience in, an a pre-disposition for, agile techniques, the principles described in  this book apply to any organization, whatever the process,  though if you take the approach to heart, you will find yourself becoming more agile, which is to say, more responsive to customer needs.

The book  covers the full lifecycle from requirements, and design, and coding, to acceptance testing, deployment and operations. There are even discussions on test design, data migration and performance optimization and capacity planning. All of this underscores the point that the goal of building software is delivering it to users, a point which some other books understate. Continuous Delivery develops the concept of a Deployment Pipeline, showing you the impact development and testing practices have on deployment and operations, and emphasizing the value of involving the operations team early in the development process.

There is a lot of material here, but there are also pointers to other resources. In the SCM and release management are, the authors build on works including Software Configuration Management Patterns: Effective Teamwork, Practical Integration, Release It!: Design and Deploy Production-Ready Software (Pragmatic Programmers) and Configuration Management Best Practices: Practical Methods that Work in the Real World.

Not only do you walk away from  this book with an understanding of how to start implementing a continuous deployment pipeline, you may also find yourself writing down a list of things to try, and tools to use. While not a tool-centric book, the authors provide many examples of tools to help you implement each phase of the process.

Having written about the version and release management process before, and how it relates to architecture and organization, I was excited get a review copy and see the authors give a thorough discussion of how the SCM and Release Management practices relate to the rest of the software development ecosystem.

While having a lot of material, the book is well organized and written. It's not a quick read, and you'll want to have a notebook or post-its handy to capture the idea it helps you to generate, but if you are interested in improving your deployment process you will find this book very valuable, whether you are a developer, tester, release engineer, or someone who manages people in those roles. When you finish this book you will not only have knowledge about how to implement a deployment pipeline, but  also the encouragement to know that it is possible, no matter how complex your project might seem.

What Agile QA Really Does: Testing Requirements

Teams transitioning to agile struggle with understanding the role of QA. Much of what you read about agile focuses developer testing.  Every project I've worked on which had good QA people had a much higher velocity, and higher quality. And there are limits to what developer Unit Testing can test.

In an "ideal" agile environment a feature goes from User Story, to Use Case, to implementation. With tests along the way, you should be fairly confident that by the time you mark a story done that it meets the technical requirements you were working from. If someone is testing your application after this point, and the developer tests are good, it seems like they should be testing something other than the code, which has already been tested.

Even though it's important that your QA team does not become the place where "testing is done" as that will sidetrack an agile project very quickly, it is  possible that the QA Team is testing (ideally in collaboration with developers) things that the team decided that developers could not test completely. Also, in practice,  problems sometimes slip through developer tests,  and the QA team is then testing the developer tests to give feedback on the types of things the developer tests need to be better at.  

Aside from "catching the errors that slipped through" a QA team doing exploratory testing is also exploring novel paths through the application, and may discover problems during some of these paths. In this case the team and stakeholders need to have a conversation about whether the error is something that users will see and care about, or whether it is something that isn't worth fixing.

In effect, exploratory testing is testing the requirements for completeness.

If you have a QA team, and they are doing exploratory testing, they are really testing:

• Requirements: Finding interactions between features and components that were not defined or understood when coding and developer testing began. 
• Developer Tests: Identifying where developer tests were not as good as they could have been (and how they could be better). This is a good use of automated "integration" tests.
• Systems tests that might be hard to test in a developer context.This might be the one set of tests that are the primary domain of QA.

And, as I've said in other places,  QA team can be extremely valuable testing user stories and requirements before they make it into a sprint backlog, by providing feedback about testability and precision. The QA team can also add value by testing how useful a product that meets the specification actually is.

Of course, this is an idealization.  But if you are disciplined about doing developer testing, you can still get a lot of value from exploratory testing, whether the people doing them are dedicated QA people, or people who are filling the role.