Recent Writings (Dev Ops and Testing)

I recently wrote a couple of articles that might be of interest to those who read this Blogs.

On Sticky Minds.com, (a TechWell community) I wrote about unit testing, and how to overcome the inertia that stops people from testing in The Value of Really Dumb Tests

I also wrote an article for  CM Crossroads that gives an overview of DevOps.

If you read the articles and find them useful (or not) please comment here at the article site.

Continuous Delivery

 If you didn't already know that the key to reliably deploy quality software is to take a cross-functional, full-lifecycle approach, Jez Humble and David Farley's book Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation will help you to understand. Much like Jim Coplien describes in  Lean Architecture: for Agile Software Development the "secret" to successful lean projects is to work with "Everyone, All Together, from Early On."

While the authors have experience in, an a pre-disposition for, agile techniques, the principles described in  this book apply to any organization, whatever the process,  though if you take the approach to heart, you will find yourself becoming more agile, which is to say, more responsive to customer needs.

The book  covers the full lifecycle from requirements, and design, and coding, to acceptance testing, deployment and operations. There are even discussions on test design, data migration and performance optimization and capacity planning. All of this underscores the point that the goal of building software is delivering it to users, a point which some other books understate. Continuous Delivery develops the concept of a Deployment Pipeline, showing you the impact development and testing practices have on deployment and operations, and emphasizing the value of involving the operations team early in the development process.

There is a lot of material here, but there are also pointers to other resources. In the SCM and release management are, the authors build on works including Software Configuration Management Patterns: Effective Teamwork, Practical Integration, Release It!: Design and Deploy Production-Ready Software (Pragmatic Programmers) and Configuration Management Best Practices: Practical Methods that Work in the Real World.

Not only do you walk away from  this book with an understanding of how to start implementing a continuous deployment pipeline, you may also find yourself writing down a list of things to try, and tools to use. While not a tool-centric book, the authors provide many examples of tools to help you implement each phase of the process.

Having written about the version and release management process before, and how it relates to architecture and organization, I was excited get a review copy and see the authors give a thorough discussion of how the SCM and Release Management practices relate to the rest of the software development ecosystem.

While having a lot of material, the book is well organized and written. It's not a quick read, and you'll want to have a notebook or post-its handy to capture the idea it helps you to generate, but if you are interested in improving your deployment process you will find this book very valuable, whether you are a developer, tester, release engineer, or someone who manages people in those roles. When you finish this book you will not only have knowledge about how to implement a deployment pipeline, but  also the encouragement to know that it is possible, no matter how complex your project might seem.

What Agile QA Really Does: Testing Requirements

Teams transitioning to agile struggle with understanding the role of QA. Much of what you read about agile focuses developer testing.  Every project I've worked on which had good QA people had a much higher velocity, and higher quality. And there are limits to what developer Unit Testing can test.

In an "ideal" agile environment a feature goes from User Story, to Use Case, to implementation. With tests along the way, you should be fairly confident that by the time you mark a story done that it meets the technical requirements you were working from. If someone is testing your application after this point, and the developer tests are good, it seems like they should be testing something other than the code, which has already been tested.

Even though it's important that your QA team does not become the place where "testing is done" as that will sidetrack an agile project very quickly, it is  possible that the QA Team is testing (ideally in collaboration with developers) things that the team decided that developers could not test completely. Also, in practice,  problems sometimes slip through developer tests,  and the QA team is then testing the developer tests to give feedback on the types of things the developer tests need to be better at.  

Aside from "catching the errors that slipped through" a QA team doing exploratory testing is also exploring novel paths through the application, and may discover problems during some of these paths. In this case the team and stakeholders need to have a conversation about whether the error is something that users will see and care about, or whether it is something that isn't worth fixing.

In effect, exploratory testing is testing the requirements for completeness.

If you have a QA team, and they are doing exploratory testing, they are really testing:

• Requirements: Finding interactions between features and components that were not defined or understood when coding and developer testing began. 
• Developer Tests: Identifying where developer tests were not as good as they could have been (and how they could be better). This is a good use of automated "integration" tests.
• Systems tests that might be hard to test in a developer context.This might be the one set of tests that are the primary domain of QA.

And, as I've said in other places,  QA team can be extremely valuable testing user stories and requirements before they make it into a sprint backlog, by providing feedback about testability and precision. The QA team can also add value by testing how useful a product that meets the specification actually is.

Of course, this is an idealization.  But if you are disciplined about doing developer testing, you can still get a lot of value from exploratory testing, whether the people doing them are dedicated QA people, or people who are filling the role.

Risks of Manual Integration Testing in the Context Rapid Change

You have probably come across a situation like this: It's close to a release deadline. The QA team is testing. Developers as testing and fixing problems, and everyone is focused on getting the best product they can out the door on time. During this time, you may notice that someone on the QA team, while working late has found an interesting problem. And they clearly spent a lot of time investigating the problem, identifying the expected results, and the details of why what's happening is wrong. If this is a data intensive application, there may well be SQL queries included to allow you to pinpoint the issue quickly. In short, an ideal problem report.

Except for one thing. Your team found and fixed the problem hours before.  And the effort to find and document the problem could have been spent on something else.

It's hard to avoid this kind of overlap when you don't have a complete end-to end automated testing process. And it's probably impossible (for now, anyway) to create an automated test process that will completely replace exploratory  testing. Any manual testing process has to balance the timeliness with the "flow" of the testers. If you update a deployment hourly, then you'll reduce the risk of redundant bug reports,  but the testers will experience too many interruptions. If you have a paradigm where you code, deploy, and stop coding until you see the next set of issues, but that means time wasted, and probably reduced quality. The answer is better communication between the the testers and developers about the state of the application. Under ordinary circumstances, you might have this sort of exchange in your daily scrum.

You're still early in adopting agile, you'll likely have a week or two at the end of a project where you'll have more effort put into manual testing, and that manual testing will find errors. So the simple thing to do is  to query among the team before spending too much time documenting an issue.

Some of the options are:

• Searching the issue tracking system. This sound like a good idea, but sometimes it's hard to find the right query. And it's possible that something got fixed without and issue being generated.
• Asking.  Yelling out a question, sending and email, or posting a message in a team chat room gives you the benefit of being able to be a bit vague in your query. 

I tend to prefer the "ask the room" approach. But this isn't effective when you're working in different time zones, or at different hours. On the other hand, since software development is a collaborative activity, you'll need to address the question of how yo collaborate across time. (The simple answer is to try to avoid close coupling between activities that are likely to happen when the teams are on different schedules).

Situations like the one that I described can happen. But it's worth figuring out how often they happen, and if there are simple ways to reduce their incidence and impact. Good communication channels are importanty and sometimes the lower tech ones work better.

To Scrum, Prepare

Agile methods have some sort of daily all-team checkpoint meeting as part of  the process. The idea behind the Daily Scrum (Scrum) or Daily Standup (XP) is good:  replace status meetings (or someone walking around asking about status) with one short daily meeting where everyone has a chance to communicate about what they are doing and what they need help with. This ensures that there is at least one chance each day for everyone to understand the big picture of the project, and to discover unexpected dependencies.

But just having everyone in the room doesn't make for an effective, focused scrum. You need to be be prepared. Once I was on a team where the scrums started going off track. They took longer. People's updates were often "I don't remember what I did yesterday," or they became long unfocused rambles that didn't convey much information.  I suggested that we all take a few minutes before Scrum to organize our thoughts. This got a lot of resistance. "It feels like a pre-meeting meeting, and with Scrum we're supposed to spend less time in meetings."

While Daily Scrum's are meant to be lightweight, it's respectful of everyone else's time to think about what 's worth sharing with the team. Most days you might just be working on one thing, in which case a quick glance at the Scrum board might be enough. But if you want to do what's best for your team, why not take 2 minutes before Scrum (either in the morning, or even the day before) jotting down what you want to share with the team that addresses the questions:

• What did I do yesterday?
• What do I plan to do today?
• What were my roadblocks?

Starting each day with a clear picture in your head of the answers those questions is probably not a bad thing from a professional development perspective anyway.

Sure, everyone will have off days where they don't get around to this, but if your Scrum's are losing focus frequently, consider:

• Posting a sign that reminds people of the agenda of the scrum (as Tabaka suggests in Collaboration Explained: Facilitation Skills for Software Project Leaders)
• Whether there are people participating who aren't needed.
• Whether your work isn't being structured in a way that moves the team towards the sprint goal.

The Daily Scrum (or standup) is a useful tool for being agile and responsive, but just being in the room does not mean that you are having a Scrum.

The Checklist as Empowerment tool

In an earlier post I talked about how many of the ideas in The Checklist Manifesto: How to Get Things Right  support for agile values. One of the  observations in the book that caught me by surprise was that checklists help people function as a team by making it easier to distribute decision making and empower individual team members. Checklists also help teams make better decisions by making it easier to distribute decision making. A team of empowered cross functional people, working together to decide how to get work done sounds a lot like the model of an agile team.

Checklists can help by institutionalizing a process where someone other than "the expert" is the center of decisions. In a discussion of a surgical checklist, Gawande discusses that nurses are the best people to own the checklist process but they needed to be able to stop a surgeon who skips a step without risking disciplinary action by a surgeon who feels free to avoid the process. Making the checklist part of the endorsed process clears a path toward a more empowered team.

An interesting case study from the book involved construction management. Since a building project, like a software project, involves a number of disciplines and teams.  And as a software pattern enthusiast, I've learned much from the study of Christopher Alexander's patterns. Much like agile and lean software development techniques have show us the advantages of moving away from a command and control process, the construction industry has learned similar lessons. According to Gawande:

But by the middle of the twentieth century the Master Builders were dead and gone. The variety and sophistication of advancements in every stage of the construction process had overwhelmed the abilities of any individual to master them. In the first division of labor, architectural and engineering design split off from construction. Then, piece by piece, each component became further specialized and split off, until there were architects on one side, often with their own areas of sub-specialty, and engineers on another, with their various kinds of expertise; the builders, too, fragmented into their own multiple divisions, ranging from tower crane contractors to finish carpenters. The field looked, in other words, a lot like medicine, with all its specialists and superspecialists.

Or for that matter, a software project with people with expertise in various technologies like database design, user experience, etc. He further describes how a construction process uses checklist to remind people to think of the other aspects of a project.:

Pinned to the left-hand wall opposite the construction schedule was ...  a “submittal schedule.” It was also a checklist, but it didn’t specify construction tasks; it specified communication tasks. ... The experts could make their individual judgments, but they had to do so as part of a team that took one another’s concerns into account, discussed unplanned developments, and agreed on the way forward. While no one could anticipate all the problems, they could foresee where and when they might occur. The checklist therefore detailed who had to talk to whom, by which date, and about what aspect of construction—who had to share (or “submit”) particular kinds of information before the next steps could proceed.

The line from the book that seemed most like it could have been just at home in a book on Agile Software Development was:

They had made the reliable management of complexity a routine. That routine requires balancing a number of virtues: freedom and discipline, craft and protocol, specialized ability and group collaboration.

By focusing on the shared responsibility of everyone to get the job done, we can strive for quality:

“That’s not my problem” is possibly the worst thing people can think

He sums up the role of checklists:

Just ticking boxes is not the ultimate goal here. Embracing a culture of teamwork and discipline is.

By embracing the idea of a team-based approach  to solving problems, we may need to give up some of our beliefs of what makes a person a valuable contributor:

It somehow feels beneath us to use a checklist, an embarrassment. It runs counter to deeply held beliefs about how the truly great among us—those we aspire to be—handle situations of high stakes and complexity. The truly great are daring. They improvise. They do not have protocols and checklists. Maybe our idea of heroism needs updating.

So, how can you figure out  how to use checklists to make teams more effective? Start with the basics and write down what you should be doing: The flow of a standup, minimally good coding practices, etc. Use iteration reviews and release retrospectives to identify what other issues can be avoided by adding a line to a checklist. Also review your current checklists and revise and improve them, especially if they get too long, or are mocked or ignored. Agile software adoption is as much about cultural change, as it is about specific skills, practices, or  tasks.

Lean Architecture

When I was a new programmer, the career path that appealed to me was to be an software architect. The architect was the person who had the vision of how the system worked, and the work of the architect (if done correctly) set the stage for all good things in a project, coordinating the development, requirements, and anything else  that you need to build a system. One thing that troubled me was that many architects I knew didn't code, considering the coding a distraction. Having worked on a project or two early in my career with a non-coding architect who was reluctant to spend time helping the team address how difficult his vision was to execute given the languages and frameworks we were implementing with, I thought that something was amiss with the idea of a non-coding architect.

One of the off-shoots of specification heavy projects (no doubt staffed by many analysts and non-coding architects) was the agile software development movement, which has as one of it's principles minimizing Big-Up-Front- Design. In some cases people took that (incorrectly) as meaning to always let the architecture evolve organically.

As I had hoped when I received my review copy,  Jim Coplien's recent book Lean Architecture: for Agile Software Development explains how agile principles and architecture are complimentary, and how, with everyone working collaboratively, a good, lightweight architectural framework can help enable agility, rather than being a barrier to it.  With his usual iconoclastic style, Coplien dispels the myth that agile doesn't need architecture.



As a C++ programmer in the early 90's Coplien's  Advanced C++ Programming Styles and Idiomswas a source of interview material when looking for programmers. It's a good bet that this book may fill the same role for those looking to see if candidates for architect roles understand what it means to be an architect in a Lean or Agile Organization. This book dispels the myth that Agile and Architecture don't go together and explains the balance between Agile architecture and too much Big Up Front Design.

This book emphasizes the importance of frequent collaboration between stakeholders in defining a good architecture and helps you to understand the importance of architecture to the success of agile projects. With code examples throughout,  this book demonstrates that architecture and coding must go together.  After describing some general principles of how architecture can add value to an agile project, the authors explain the Data Context, Interaction (DCI) architecture, which provides an framework for building lean architectures.  My one complaint is that the transition between the general discussions of lean architecture and the focused discussion of DCI was a bit abrupt. This could almost have been two books: one on lean architecture principles and a second (short) book demonstrating how DCI is an useful framework to apply the principles from book one. But this was a minor distraction from an enjoyable and informative read.

Rich with citations and historical context, this book will be useful for anyone who is struggling with how to build systems that need to support complicated user interactions.