Site Reliability Engineering; The Book and The Practices

Site Reliability Engineering

It’s difficult to walk into a software development organization without hearing about the discipline of Site Reliability Engineering (SRE) though you may discover that SRE means different things to different teams. The practices of Site Reliability Engineering are all well known, and successful teams practiced them before there was a collective name for them. I read Site Reliability Engineering in an attempt to get a handle on what the discipline covers, and what SRE teams do. I do feel like I have a better understanding both of what canonical SRE is, and helped explain why different organizations practice the SRE discipline in slightly different ways. I recommend the book to those want to learn more about deploying systems at scale, though with some caveats.

The book is very Google centric, which is appropriate since the book is subtitled “how Google Runs Production Systems.” How well you can apply the lessons in the book to your team depends on your prior expertise, and the specific topic.

The book is a collection of articles by a variety of contributors, rather than a single sourced book and as a result the writing is a bit uneven. Some chapters do a good job of walking you though the subject area, and distinguish how what Google does could apply to your organization and too chain. Others are Google centric to a fault, describing internal tools and approaches with little if any reference to similar more generally available or even open source tools, or the tradeoffs to consider when evaluating that decision for your team.

The principles in the book are all generally valid and you will definitely walk away from the book knowing more about the issues to consider in keeping production systems running at scale. You many or may not have a clear idea about how to implement the lessons you learned.

Google is a successful company and has solved some challenging problems, and we can learn a lot from the Google practices. It’s important to remember that Google is also unique, in terms of history and problem space, so one should consider adapting the Google Way, rather than adopting it without interpretation. This book is a great launching point for discussion, and it’s worth having a copy if you deploy systems at scale. Just don’t take it as The Way to do Site Reliability Engineering.

Designing your life: Design and Agile Tools Applied to Life.

Designing your Life: How to Build a Well-Lived, Joyful Life by Bill Burnett and Dave Evans explains how you can apply design thinking to life choices. While the true test of the information in the book is to do the exercises and embrace the process, and look at the results (which I have not yet done) I finished the book with an excellent understanding of the possibilities and the feeling that I could use the tools to better understand my goals. Participating in a mini-workshop that Bill Burnett helped to validate that the process can be very powerful. A lot of work, but powerful.

As I mentioned in an earlier Techwell article (which was based on an interview) many of the concepts here may be familiar If you are a student of agile principles. Some of the exercises are reminiscent of those in Innovation Games. There is an exercise that is reminiscent of a career timeline exercise that Johanna Rothman has proposed, and the discussion of job searches and job descriptions is very consistent with some of what Johanna has written about the subject.

There are other elements that sounded familiar as well. The discussion of problem reframing will be familiar to those who work with software requirements, and the discussion of the qualities of a good “design team” will be familiar to anyone who has studied teams and read books such as Extraordinary Groups. Even if the concepts are familiar, there is value in seeing them applied to life design.

That the tools are familiar should not lead you you dismiss the value of the book. (After all, one could argue that much of the “agile toolbox” derives from other disciplines.) Like any tool box, tools can have many applications and it's useful to have a guide to how to use a tool to solve your problem with the right techniques, and to understand that there are versions of the tool that are more finely tuned to your purpose. I hesitate to say "solve your problem efficiently" because life design is neither a problem with a single solution end point (it's a process) nor is it simple. By applying the collection of straightforward steps in this book you can start on the path to understanding how to design a like that is congruent in all dimensions that matter.

Regarding solving as compared to exploring, the authors emphasize that there is difference between engineering problems and design problems is that design problems in that engineering problems are more about solving, and design ones about building forward. While I believe that to be a good engineer you need to understand design, I agree that there are different perspectives, and engineers don’t often switch between “explore options” and “solve a defined problem” appropriately.

This is an easy to read book that can be useful in helping with evaluating both the big picture and specific aspects of your life. You may even gain some insight into problems related projects, since the tools are similar. Reading the book is valuable. Working through the exercises with a team can be more so. The book is full of are exercises and is supplemented by a web site with worksheets and other resources to use. This is an easy to read book that can be as useful as you want it to be.

Excitingly Well Run Meetings

The group of people huddled together in a room “until the job is done” is often used as a demonstration of virtues like “diligence” and “commitment.” And these scenes are often the stuff of dramatic moments in popular culture. We rarely, if ever, see any effusive praise for the well-run meeting that ends on time with a useful outcome. The former is more compelling. The latter is often more valuable and much harder to do. And while there are times that the former is the right thing to do, more often than not, it’s an indication of a problem. Well run meetings are useful meetings.
While it seems a bit pedantic to talk about meetings and schedules, and teams often present a ‘meeting-free culture’ as an ideal, the reality is that software development is a collaborative activity and we need to interact with (or “meet”) with people. When you need collaborate with more than a couple of people — perhaps from different teams — scheduling a meeting is inevitable. People have other commitments and time constraints, and respecting these commitments is good for the organization and the individuals.

Respect

There are many reasons for keeping a meeting on schedule. Starting late or ending late, wastes people’s time, which has a financial cost. But the primary reason keeping to schedule from perspective its it is respectful, and not honoring a schedule shows a lack of respect. By starting or arriving late you are disrespectful to the attendees who might be on time, by not ending on schedule you are being disrespectful to anyone the attendee was supposed to meet after a meeting and anyone who needs the room (or other resources) you are using. Of course, it is OK to you extend a discussion by mutual agreement, but you also need to agree on the basic ground rules.

Meeting Principles and Protocols

There is much written about good meeting structure and facilitation, and if you are in a role where you need to collaborate with people (as most of us are) it’s worth the effort to learn a bit about the subject. An effective meeting has, at minimum, two things: a reason, and a schedule.

A Reason

In general, a meeting worth having should have:

• A reason, which attendees understand.
• A goal, which you can decide if you have met or not.
• An agenda, to help everyone understand if they are on track.
• Attendees who understand why they are there.
• A start and end time that is sufficient to have the discussion you need, and which allows for people to meet their next commitment.

All of these items can have more or less definition depending on the culture of the team, and the nature of a problem, and how close you are to having a common baseline of understanding. A Sprint Planning Meeting might have a more well defined reason, goal, and agenda, than an “explore design options” meeting. But it is important to be able to express these things.

Start Times and End Times

It’s easy to think of meetings as being (say) 1 hour long, starting and ending at hour boundaries. This makes no sense if you assume that anyone else has to be somewhere else after your meeting. It’s helpful to have a protocol where you allow for a buffer at the start and the end of the meeting to allow for transitions. There are many options. for a 1:00 - 2:00 meeting you could:

• Start at 1:05 and end at 1:55
• Start at 1:00 and end at 1:50
• Start at 1:10 and end at 2:00

or any variant. Pick what works for your organization. In my experience the first option seemed to work best, as people tend to be more keyed to “round” boundaries. But as long as everyone is consistent, people will have time to regroup and move between meetings.
Be sure to plan the agenda to include time before the end of the meeting to review what you all decided, what followup is necessary, and to decide who will do that followup.

When “Until we are done” makes sense

The opposite side of watching the clock is the idea that you don’t want to cut short a productive exchange of ideas. Ideally you would have allocated enough time to allow for opportunistic conversation. If you do that, you will need to take care that a non-productive meeting doesn’t expand to fill the allocated time. In some cases you can’t do that, and “in the room until the problem is solved, regardless of schedule” is the right thing to do and makes sense but when certain things are true:

• The issue under discussion needs to be resolved soon.
• The issue is the top priority for everyone in the room who is asked to be there, and everyone understands this.
• If you are using a shared resource, such as a conference room, the issue at hand is the top priority for the organization, and you have priority for using the resource.

All of these things are rarely true at the same time. Most people have other commitments, and in many organizations, meeting rooms are a scarce resource. It’s worth taking the time to solve the hard problem of how to get useful work done on a schedule (and to identify what problem fits in that schedule).

Next Steps

While “meetings” may always have a subtext of “something that distracts from work,” if you make the effort to be respectful of people’s time you’re likely to get more done. While having a company culture or guidelines that supports this approach is good, you can establish these guidelines are part or your working agreements at any level of scale.
Regardless or what guidelines you use, the most important thing is to be mindful of the goals of the meeting, and the reality the people may have other commitments.

Books to Make Discussion Easier

For a variety of reasons, I’ve recently found myself in quite a few conversations about social and political issues, both in person, and on Facebook and other social media. Even when I was engaging with someone with a different view that I had, I learned a lot, both about my views and contrary ones . Other conversations were more frustrating. The difference between the enjoyable ones and the frustrating ones seemed to be that the arguments I heard didn’t always seem to be either relevant or logical. Rather than (always) walk away I took these challenging conversations as an opportunity to practice focusing on understanding, rather than (only) and opportunity to win. (Though sometimes walking away is the best thing...)
I found these books, which I’ve recently read, to be a useful part of a toolkit for more productive arguments about controversial subjects:

• Don’t Think of an Elephant
• Mastering Logical Fallacies
• Humble Inquiry

Don’t Think of an Elephant by George Lakoff is the most partisan of the books. It is upfront as being about being a “guide for Progressives,” though it also explains the concept of “framing,” and the role it plays in how people interpret information.
As an engineer I tend to think that the best way to argue point is with facts and data. This doesn’t aways work, especially in political discussions, even when the data are clear, quantifiable, and not disputed by reasonable people, because the words are sometimes framed in a way that re-enforces another view point. Don’t think of an Elephant explains how framing works from the perspective of linguistics and cognitive science and the importance of framing in discussion and debate. Lakoff emphasizes that this book is more action oriented than academic, and he points to more scholarly works on the topic for those who are interested.
While the book is about political advocacy, and geared at Progressives, it can be useful for a number of audiences. For Progressives, you can better understand how to frame your arguments when trying to influence others. The book also has a discussion of what the Conservative mindset is, and awareness of the perspective of someone who thinks differently that you can help bridge gaps. Conservatives who are interested in having better conversations with their more progressive friends and associates might also get some insights from the book.
The book heavy emphasizes political discourse, but the concept of frames and framing is something you can apply to communicating your perspective in various contexts.
Partisan as the book, is, it can also be useful for bridging gaps. This book might provide a guide for Progressives to make their points in a more effective, less reactive way, and to have more productive conversations with their Conservative friends.

Mastering Logical Fallacies by Michael Withey is a bit less political, and more generic, but still relevant to political conversations. I got a copy of this book for my 10 year old so that he’d be exposed to the idea of good arguments early. Then I decided that I needed to read it myself, in part as a reaction to having been in far more conversations around recent political events where some of the arguments made no sense to me. The book helped me understand how to recognize and address those kinds of arguments. It also has helped me to take a step back in discussions in other domains, including technical discussions at work.
Having a framework for understanding these kinds of fallacies can help you to put a conversation in context, and be able to (more) calmly address the issues people are raising, rather than react emotionally and perhaps commit the same kind of fallacies yourself.
While I can’t speak fully to the thoroughness of the discussion of the fallacies (maybe if I had either taken the forensics class in High School, or considered the Debate Team!) I found this to be a really good bit of background. My one complaint is that some of the examples are a bit forced, but the author still makes his point most of the time.
I still plan to share the material with my 10 year old so that he can learn how to have good discussions -- something it’s never too early to learn! This book is a tool that can help you navigate conversations (especially political ones) be they on Facebook or in person.
While the first two books are more tactical, in that they provide guidelines for how to deal with specific conversational situations, [_Humble Inquiry_] by Edgar Schein is more about mindset.
This is a short, pragmatic, easy to read, book that can help you be better at something both essential and often neglected: How to work together better. By chance I first opened the book to a page with the heading “The Main Problem: valuing tasks over relationships” and I knew that I made the right choice when I got the book. Teamwork is essential and building an environment where people trust each other enough to work as a team is hard.
This book explores a technique that can help to work across cultural, organizational and hierarchical boundaries. With theory, examples, and practices to try, it becomes easy to understand what Humble Inquiry is. The practice will take work.
Humble Inquiry is as much a mindset as a technique, since, as Shein points out, it is hard to be authentic when simply “acting humble” and people will notice, and you will thus erode rather than build trust.
Any leader, team member, spouse, or even parent can learn valuable things from reading this book. I would even argue that if you interact with anyone there are lessons you can learn, or at least have reenforced. This book is a small investment in both time and money for a large reward.
There are certainly any more books that cover this space, but these three seemed to cover a range that might help be weather the more difficult conversations.