Risks of Manual Integration Testing in the Context Rapid Change

You have probably come across a situation like this: It's close to a release deadline. The QA team is testing. Developers as testing and fixing problems, and everyone is focused on getting the best product they can out the door on time. During this time, you may notice that someone on the QA team, while working late has found an interesting problem. And they clearly spent a lot of time investigating the problem, identifying the expected results, and the details of why what's happening is wrong. If this is a data intensive application, there may well be SQL queries included to allow you to pinpoint the issue quickly. In short, an ideal problem report.

Except for one thing. Your team found and fixed the problem hours before.  And the effort to find and document the problem could have been spent on something else.

It's hard to avoid this kind of overlap when you don't have a complete end-to end automated testing process. And it's probably impossible (for now, anyway) to create an automated test process that will completely replace exploratory  testing. Any manual testing process has to balance the timeliness with the "flow" of the testers. If you update a deployment hourly, then you'll reduce the risk of redundant bug reports,  but the testers will experience too many interruptions. If you have a paradigm where you code, deploy, and stop coding until you see the next set of issues, but that means time wasted, and probably reduced quality. The answer is better communication between the the testers and developers about the state of the application. Under ordinary circumstances, you might have this sort of exchange in your daily scrum.

You're still early in adopting agile, you'll likely have a week or two at the end of a project where you'll have more effort put into manual testing, and that manual testing will find errors. So the simple thing to do is  to query among the team before spending too much time documenting an issue.

Some of the options are:

• Searching the issue tracking system. This sound like a good idea, but sometimes it's hard to find the right query. And it's possible that something got fixed without and issue being generated.
• Asking.  Yelling out a question, sending and email, or posting a message in a team chat room gives you the benefit of being able to be a bit vague in your query. 

I tend to prefer the "ask the room" approach. But this isn't effective when you're working in different time zones, or at different hours. On the other hand, since software development is a collaborative activity, you'll need to address the question of how yo collaborate across time. (The simple answer is to try to avoid close coupling between activities that are likely to happen when the teams are on different schedules).

Situations like the one that I described can happen. But it's worth figuring out how often they happen, and if there are simple ways to reduce their incidence and impact. Good communication channels are importanty and sometimes the lower tech ones work better.

To Scrum, Prepare

Agile methods have some sort of daily all-team checkpoint meeting as part of  the process. The idea behind the Daily Scrum (Scrum) or Daily Standup (XP) is good:  replace status meetings (or someone walking around asking about status) with one short daily meeting where everyone has a chance to communicate about what they are doing and what they need help with. This ensures that there is at least one chance each day for everyone to understand the big picture of the project, and to discover unexpected dependencies.

But just having everyone in the room doesn't make for an effective, focused scrum. You need to be be prepared. Once I was on a team where the scrums started going off track. They took longer. People's updates were often "I don't remember what I did yesterday," or they became long unfocused rambles that didn't convey much information.  I suggested that we all take a few minutes before Scrum to organize our thoughts. This got a lot of resistance. "It feels like a pre-meeting meeting, and with Scrum we're supposed to spend less time in meetings."

While Daily Scrum's are meant to be lightweight, it's respectful of everyone else's time to think about what 's worth sharing with the team. Most days you might just be working on one thing, in which case a quick glance at the Scrum board might be enough. But if you want to do what's best for your team, why not take 2 minutes before Scrum (either in the morning, or even the day before) jotting down what you want to share with the team that addresses the questions:

• What did I do yesterday?
• What do I plan to do today?
• What were my roadblocks?

Starting each day with a clear picture in your head of the answers those questions is probably not a bad thing from a professional development perspective anyway.

Sure, everyone will have off days where they don't get around to this, but if your Scrum's are losing focus frequently, consider:

• Posting a sign that reminds people of the agenda of the scrum (as Tabaka suggests in Collaboration Explained: Facilitation Skills for Software Project Leaders)
• Whether there are people participating who aren't needed.
• Whether your work isn't being structured in a way that moves the team towards the sprint goal.

The Daily Scrum (or standup) is a useful tool for being agile and responsive, but just being in the room does not mean that you are having a Scrum.

The Checklist as Empowerment tool

In an earlier post I talked about how many of the ideas in The Checklist Manifesto: How to Get Things Right  support for agile values. One of the  observations in the book that caught me by surprise was that checklists help people function as a team by making it easier to distribute decision making and empower individual team members. Checklists also help teams make better decisions by making it easier to distribute decision making. A team of empowered cross functional people, working together to decide how to get work done sounds a lot like the model of an agile team.

Checklists can help by institutionalizing a process where someone other than "the expert" is the center of decisions. In a discussion of a surgical checklist, Gawande discusses that nurses are the best people to own the checklist process but they needed to be able to stop a surgeon who skips a step without risking disciplinary action by a surgeon who feels free to avoid the process. Making the checklist part of the endorsed process clears a path toward a more empowered team.

An interesting case study from the book involved construction management. Since a building project, like a software project, involves a number of disciplines and teams.  And as a software pattern enthusiast, I've learned much from the study of Christopher Alexander's patterns. Much like agile and lean software development techniques have show us the advantages of moving away from a command and control process, the construction industry has learned similar lessons. According to Gawande:

But by the middle of the twentieth century the Master Builders were dead and gone. The variety and sophistication of advancements in every stage of the construction process had overwhelmed the abilities of any individual to master them. In the first division of labor, architectural and engineering design split off from construction. Then, piece by piece, each component became further specialized and split off, until there were architects on one side, often with their own areas of sub-specialty, and engineers on another, with their various kinds of expertise; the builders, too, fragmented into their own multiple divisions, ranging from tower crane contractors to finish carpenters. The field looked, in other words, a lot like medicine, with all its specialists and superspecialists.

Or for that matter, a software project with people with expertise in various technologies like database design, user experience, etc. He further describes how a construction process uses checklist to remind people to think of the other aspects of a project.:

Pinned to the left-hand wall opposite the construction schedule was ...  a “submittal schedule.” It was also a checklist, but it didn’t specify construction tasks; it specified communication tasks. ... The experts could make their individual judgments, but they had to do so as part of a team that took one another’s concerns into account, discussed unplanned developments, and agreed on the way forward. While no one could anticipate all the problems, they could foresee where and when they might occur. The checklist therefore detailed who had to talk to whom, by which date, and about what aspect of construction—who had to share (or “submit”) particular kinds of information before the next steps could proceed.

The line from the book that seemed most like it could have been just at home in a book on Agile Software Development was:

They had made the reliable management of complexity a routine. That routine requires balancing a number of virtues: freedom and discipline, craft and protocol, specialized ability and group collaboration.

By focusing on the shared responsibility of everyone to get the job done, we can strive for quality:

“That’s not my problem” is possibly the worst thing people can think

He sums up the role of checklists:

Just ticking boxes is not the ultimate goal here. Embracing a culture of teamwork and discipline is.

By embracing the idea of a team-based approach  to solving problems, we may need to give up some of our beliefs of what makes a person a valuable contributor:

It somehow feels beneath us to use a checklist, an embarrassment. It runs counter to deeply held beliefs about how the truly great among us—those we aspire to be—handle situations of high stakes and complexity. The truly great are daring. They improvise. They do not have protocols and checklists. Maybe our idea of heroism needs updating.

So, how can you figure out  how to use checklists to make teams more effective? Start with the basics and write down what you should be doing: The flow of a standup, minimally good coding practices, etc. Use iteration reviews and release retrospectives to identify what other issues can be avoided by adding a line to a checklist. Also review your current checklists and revise and improve them, especially if they get too long, or are mocked or ignored. Agile software adoption is as much about cultural change, as it is about specific skills, practices, or  tasks.

Lean Architecture

When I was a new programmer, the career path that appealed to me was to be an software architect. The architect was the person who had the vision of how the system worked, and the work of the architect (if done correctly) set the stage for all good things in a project, coordinating the development, requirements, and anything else  that you need to build a system. One thing that troubled me was that many architects I knew didn't code, considering the coding a distraction. Having worked on a project or two early in my career with a non-coding architect who was reluctant to spend time helping the team address how difficult his vision was to execute given the languages and frameworks we were implementing with, I thought that something was amiss with the idea of a non-coding architect.

One of the off-shoots of specification heavy projects (no doubt staffed by many analysts and non-coding architects) was the agile software development movement, which has as one of it's principles minimizing Big-Up-Front- Design. In some cases people took that (incorrectly) as meaning to always let the architecture evolve organically.

As I had hoped when I received my review copy,  Jim Coplien's recent book Lean Architecture: for Agile Software Development explains how agile principles and architecture are complimentary, and how, with everyone working collaboratively, a good, lightweight architectural framework can help enable agility, rather than being a barrier to it.  With his usual iconoclastic style, Coplien dispels the myth that agile doesn't need architecture.



As a C++ programmer in the early 90's Coplien's  Advanced C++ Programming Styles and Idiomswas a source of interview material when looking for programmers. It's a good bet that this book may fill the same role for those looking to see if candidates for architect roles understand what it means to be an architect in a Lean or Agile Organization. This book dispels the myth that Agile and Architecture don't go together and explains the balance between Agile architecture and too much Big Up Front Design.

This book emphasizes the importance of frequent collaboration between stakeholders in defining a good architecture and helps you to understand the importance of architecture to the success of agile projects. With code examples throughout,  this book demonstrates that architecture and coding must go together.  After describing some general principles of how architecture can add value to an agile project, the authors explain the Data Context, Interaction (DCI) architecture, which provides an framework for building lean architectures.  My one complaint is that the transition between the general discussions of lean architecture and the focused discussion of DCI was a bit abrupt. This could almost have been two books: one on lean architecture principles and a second (short) book demonstrating how DCI is an useful framework to apply the principles from book one. But this was a minor distraction from an enjoyable and informative read.

Rich with citations and historical context, this book will be useful for anyone who is struggling with how to build systems that need to support complicated user interactions.

Are You Done Yet?

Johanna Rothman recently wrote, commenting on Joshua Kerievsky's proposed definition of done. Both posts are worth a read, if for no other reason than to better understand why we have such a difficult time defining what "done" is, and why defining "done" is one of the major challenges for teams  trying to adopt agile practices.

Thinking about both Joshua's and Johanna's points I wonder if the difference isn't similar to a discussion of whether principles or practices are more important to be successful when adopting agile methods. On the one hand following  practices diligently allows you to develop good habits and even to get some good results early on. The challenge comes when it's time to reflect and improve on your practices. Without a good understanding of practices it's hard to optimize.

Similarly, defining done earlier in the process can cause problems if you are thinking about the meaning of "done" the wrong way.  If "done" means washing your hands of issues ("we met the spec..."),  evaluating done as late as possible makes sense, enforcing the idea that you are not done until the customer is happy is a useful driver.

If, on the other hand, you understand (and believe) that your goal as a developer is to deliver useful, quality software,  and if the customer understands that that they may not have understood the problem until they had a working system in hand, defining done for earlier steps means that you have more tools with which to evaluate your effectiveness, status, and progress.   Done closer to the developer means that you have more, rather than fewer, chances to evaluate, learn, and improve.  By embracing the principle that delivering a useful end product is the goal, you can benefit from having some local completion criteria.



Having the definition of done closer to the team (as Johanna recommends) allows you to measure progress and identify risk. You also need to be able to acknowledge that it is possible that completing all the stories may still mean that there is still work to do. Then you have to inspect, adjust, and adapt. Which is to say: be agile.

A Review of Drive by Dan Pink

This is one of those books that describes something extremely obvious and intuitive that at the same time goes against what you were taught was "common sense." This would be a good book just for the survey of the (long) history of the study of the theory of motivation. It also concludes with a number of things you can do to create an environment that encourages mastery (as opposed to simply meeting goals) in your work and school. 


If you're an agile software developer you'll have a few aha! moments when you understand how agile practices really encourage flow and create environments where teams and individuals can be highly productive. If you're a manager, this book will encourage you to think about how teams work and how some common practices are counter-productive. 


If you're trying to understand why self organizing teams work, but with a perspective outside of software development, this is a quick read that will get you thinking and learning. 


Some other books on related topics:

• Behind Closed Doors: Secrets of Great Management (Pragmatic Programmers)
• Pragmatic Thinking and Learning: Refactor Your Wetware (Pragmatic Programmers)
• Flow: The Psychology of Optimal Experience (P.S.)

Motivation Visibility, and Unit Testing

I've always been interested in organizational patterns (such as those in Organizational Patterns of Agile Software Development). I've recently found myself thinking a lot about motivation. I'm now reading Drive: The Surprising Truth About What Motivates Us and just finished Rob Austin's book on performance measurement. Being the parent of a three year old, I'm finding more and more that "because I said so, and I'm right" isn't too effective at home. My interests in motivation are closely related to my interest in writing software effectively. Writing software is partially a technical problem about frameworks, coding, and the like, but the harder (and perhaps more interesting) problem is how to get a group of people working together towards a common goal. Agile practices, both technical and organizational, build a framework which makes having the right amount of collaboration and feedback possible. But there's a bootstrapping process: How do you get people to start doing the practices, especially technical ones, such as unit testing?

In an ideal world, everyone will know how to write unit tests,  understand their value, and want to write them. In an organization transitioning to agile having all three parts in place is not a given.  Most people understand why unit tests are useful, in principle. The problem is the execution. In  my experience, there are two reasons people who claim that that want to write unit tests give for not writing them:

• They are too hard. The overhead of the test can make the cost of developing a feature excessive. 
• They are too easy. Some functionality appears to be trivial, so why would we want to test it.

Both of these reasons can have merit at times. Testing getters and setters, and trivial calls to a system library and other simple coding constructs really don't add value. And writing a complicated test using a hard to use framework to verify something non business critical that could be quickly validated by visual inspection ("make the background color of the home screen orange") may well be not worth the effort. The problem is that most people have bad intuitions about where the lines line until they start practicing the skill of unit testing. Most teams need experience with testing to get a true feel of what is a trivial test, and what is a seemingly trivial test that can unmask a major problem

Rather than frame the testing challenge with the default being the old way of not testing:

Write a test when it makes sense.

Change your perspective to the default being to test:

Write a test unless you can explain why you did not.

The key to this approach is to make encourage people to think through their rationale for not testing. There are a few ways to do this, but one approach is to tie the explanation mechanism into something every developer works with every day: your source code repository. Have the team agree that, in addition to the source files, every commit will have a change to a test or a rationale for why you didn't write a test.  For example if I do a commit without a test I could write:

ISSUE-23: Fixed the spelling of the company name. NO TEST: it was a typo. 

or

ISSUE-26: Fixed the rendering mechanism. NO TEST: we don't have a good framework for testing this sort of thing

or even

ISSUE-28: Fixed a serious logic issue. NO TEST: I didn't feel like writing one.

By developing a team agreement to add tests or explain why not, you are starting with a small change of behavior that paves the way for a greater change based on understanding. Even if a message like the last one is acceptable, many will be uncomfortable admitting to laziness, and think harder for a reason. By reviewing the commit messages later on you can get a sense of impediments to testing (technology, organizational,  or attitude), and use that data in a retrospective to decide how to improve.

By being creative you can help people on your team understand the value of process changes, and start a conversation about how to evolve practices to suit your team.