Perspective

“What is important is seldom urgent and what is urgent is seldom important.” – Dwight Eisenhower

There are things in our lives that are urgent and important, urgent and unimportant, not urgent and important, and not urgent and not important. The idea is, you first want to get your urgent and important things out of the way, then put a lot of energy into important but not-so-urgent priorities. DevOps adoption, as it’s currently being practiced, tends to focus on not-urgent or not-important problems – and sometimes both.

Some Background:

When the term “DevOps” was first introduced in 2009, the largest barrier to adoption was awareness. Managers and leaders didn’t know what DevOps was, much less why they needed it. As an industry, we tackled this problem with two main strategies: we appealed to the unicorns (i.e., Amazon is doing production deployment every 11 seconds), and we built grassroots proof of concepts. At the time, this was both urgent and important. Urgent because our competition was reading the same articles about the industry leaders, and important because the first step towards solving a problem is to recognize it and name it.

Seven years later the unicorns have all been spotted, rounded up, studied, analyzed, reported on, photographed and imitated.  So why do we still feel that it’s important to drive awareness? Certainly there is still a good deal of uncertainty about what DevOps really means, and this needs to continue to be refined.  But the awareness is there. IT VPs are beginning to put DevOps initiatives in their budgets.  The elusive “DevOps Engineer” is a hot commodity for enterprise recruiters. Go to any technical conference, and there’s sure to be a session on DevOps for developers. Undoubtedly, many organizations have at least one team that can showcase automated builds, tests and deployments. It has now become urgent and important to begin realizing the gains that have been promised by all the grassroots champions.

It’s Not About the Tools:

Any student of the DevOps movement will tell you that it’s not about the tools – it’s about building the culture. And yet, everyone is organized around building and leveraging those tools. We are seeing teams that are able to check in code and automate processes all the way to deployment into QA, but then things stop. The change advisory board is still there.  IT security is still there. Ops is still on edge about frequent deployments. But let’s not forget what the whole point of all of this is: to deliver business value quickly. It is now unimportant if you go from 1 build a day to 10 builds a day, if you’re still sitting at 1 deployment per quarter.  It’s not urgent to lower your WIP from Dev to QA if you’re letting it all stack up on your staging servers. Clearly the important work is to address the culture, relationships, security, and change control processes that have previously been stuck in a not-urgent/not-important quadrant.

What You Can Do:

This shift towards focusing on non-technical matters is hard for developers. When we’re staring at our screens thinking, “what can I do to make this better?” we usually come up with technical answers like writing a script to automate a process, or creating a tool to assist with build metric collection. These are great ideas, but a better answer may be to go find an ops guy and take him out to lunch or to sit in on a release management meeting to better understand why getting through production control is so difficult.

Rule of Thumb:

If you’ve already developed some automation, then do not favor more automation over relationship development with operations. I witnessed one mid-sized company focus entirely on agile development practices and build automation without ever addressing the production control variable. Sprint velocity and story acceptance was high.  Builds were automatically deployed to development servers on code check-in, and were push-button deployable to QA servers when QA was ready. All final binaries were automatically packaged up for deployment without any developer involvement. Yet, during a 12-month period not a single line of code was deployed to production. Clearly automation didn’t provide any business value here.

Conversely, I worked with a Fortune 500 company that was struggling with large, complex software releases. Big-bang releases would frequently be rolled back, causing the deployment complexity to continually ratchet up. Some rollbacks were the result of configuration or usage issues, rather than bugs. Failed deployments became a self-fulfilling prophecy. While developers were improving the build pipeline no new build or release automation was put in place to mitigate the deployment issues. Instead, the problem was solved by establishing a better relationship with our operations partners. We figured out that if we had the right players involved, we could collaboratively deploy smaller incremental releases to a limited production audience. This allowed developers to work side-by-side with end users to quickly certify changes and develop fail-forward plans in the event of mistakes. Release frequency went from a monthly ordeal with regular rollbacks, to multiple weekly deployments with nearly zero rollbacks.

It is time for the development managers and directors to start playing a more active role. Managers need to recognize that to fully realize the benefits of DevOps, they can’t simply delegate all the work down to the technical staff. Someone needs to champion the organizational changes that need to occur to get development, operations, QA, change management and security all working to take the same hill. Someone needs to start the grassroots work of building the organizational tools (not technical tools) to get the products through security and release management teams. These are the highly important, and increasingly urgent tasks that are needed to solve the DevOps puzzle.