Is DevOps Possible in Government Agencies?

Tweet Fri 09 February 2018

Editorial note: I originally wrote this post for the Techtown Training blog. You can check out the original here, at their site. While you’re there, take a look at their DevOps eBook to understand how your role fits into your organization's DevOps transformation.

Let’s not beat around the bush. Yes, it’s possible.

And before you close this tab because you’re not working for the government, let me say that what you’ll read in this post will provide a good example of how big organizations could implement DevOps—organizations where several teams have to agree on things and documents have to be filed before anyone can act. That’s usually the case with many big enterprises. Bureaucracy and coordination between teams are often what prevent organizations from providing value at the exact right time.

But what does this all have to do with DevOps and government? Well, DevOps is about providing value to someone that will benefit from the product the organization is providing. It’s not just about helping organizations to speed up delivery and profit. Governments have a yearly budget, so profit might not be a big win in this case. Instead, they’re interested in a more social cause: improving human life. The software they produce could affect someone’s life to the point of not getting healthcare coverage. We’ll talk about that in a moment. The point is that we all depend on software now, so it’s not just a matter of profitability. It’s also a matter of providing life-enhancing value.

A common mistake is to think that DevOps is only for certain companies—that you will need to have the smartest people on your team. But that’s not true. I overheard that someone ask Adrian Cockcroft how Netflix was able to get the team they have. He just answered, “I took them from you.” I’m not pretending DevOps is for everyone, but I will say that I haven’t found any industry of software development where this can’t be applied. It’s applicable even in firmware industry. So let’s see how we can take the DevOps approach to some of the common problems in government.

Culture, Processes, and Architecture Are Too Rigid

Government agencies typically have serious silos. That’s because they often work with a waterfall model, and each phase represents a different group. And it’s not just that. They usually have to deal with a series of contractors, and all of them use a different tool to communicate. This especially applies to, say, a ticket system to report bugs. As Conway’s Law says, “Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.” Communication is exactly one of the things that DevOps tries to improve.

The way you introduce DevOps in such a scenario is by working on how teams interact with each other. When you start to understand the needs of the other teams that are also struggling when delivering something, you’ll anticipate those needs and work on removing their hassles.

Let’s talk about security since it’s a big concern in government. You don’t need to wait until you’re about to deploy a change to recertify the software. Instead, you could start injecting those security checks from the beginning. And that list is pretty big in government. Doing this will not only help you to deliver faster but will also leave tracks as the change is being pushed across the different environments.

And because you will be leaving tracks all the way down, you’ll also have the ability to make sure the software is compliant. Everyone will know how a change got to production. You can even get to a point where no manual intervention will be required. Government agencies have strict regulations about what gets sent to production, as well as where and how it gets sent. Compliance is a big concern in this realm, but DevOps brings the benefits of continuous visibility.

Resistance To Change Until It’s Urgently Needed

Usually, bureaucratic shops resist change culturally. Most of the time it’s because, in government, many people have lots of years in the same job. Some might be afraid to lose those jobs, and others simply don’t see the need for change. For them, work might be slow, but at least progress is being made. But what about when there’s an external pressure—a political one that you can’t control? You might remember when the launch of the healthcare.gov platform failed. Everybody had something to say about what went wrong and how they should have done things differently.

The healthcare.gov failure forced the government to change. There are some interesting tales about how they survived during that time and how things worked a year after that failure. There were big struggles with communication. No one really knew exactly the impact and importance of that project. But it’s interesting that after some years, you see that there are platforms like cloud.gov that take care of all the security checks (a document about 4,000 pages long) that government needs to pass so that people can focus only on software development. That platform is even fedRAMP authorized now, and AWS has a dedicated region for these type of workloads.

So it is possible to introduce practices like DevOps in government. Some will resist the change wholeheartedly. Others will say that DevOps is great, but it’s just not for them. They will always put forth a valid argument or excuse, but they often come around to viewing DevOps as the right move when they’re forced to change. It’s exemplified by the Bill Palmer storyline in the book The Phoenix Project: after a big failure in a project, they had to find a way to do things better. They did, but only after the external pressure of outsourcing, IT came.

Working With Bureaucracy In Small Batches

So, you're in an environment where every little thing requires six levels of approval. Do you want to use Jenkins? Great! You just need 15 committees to approve it over two years, and you're good! So how do you work with this? Startups can easily take the DevOps approach because there are not many people who have to agree. But in government or big enterprises, that’s not the case.

Some time ago, I had the wrong idea about my boss and his feelings about introducing something new. I thought he didn’t like it when I’d innovate or try new things. In reality, he was concerned about the stability and quality of our product. But then he encouraged me in a different way. He said that it was OK to experiment in another environment that wouldn’t affect anyone else, especially our customers. That way, we were able to try a new technology, architecture, or tool without risking our business. Also, we’d make sure that we weren’t introducing something that wouldn’t work for us.

That’s the approach you can take when introducing a practice like DevOps. Prove that it’s working, start small, but make sure it will have an impact. Start making your allies and choose wisely. Talk with key team members in each team—those that you know have a voice. Listen carefully to all their concerns, and work on having an anticipated answer. Then, you can offer a few options. And don’t expect that you’ll be implementing DevOps overnight. You’ll have to give in to certain things that you might not necessarily agree with in order to convince the committee that will approve the initiative.

Rome Wasn’t Built In a Day

It will take time for some organizations to adopt DevOps. And we all can learn about other industries that have needed to do things differently. What worked for some might not work for you and vice versa. Especially in software development, there will always be tradeoffs. The majority of the problems in such bureaucratic industries concern culture and process. It’s not about individuals, per se, nor is it about technology stack or tools. I’ve had problems with my friends when coordinating where to go to dinner. Imagine how complicated it can be with the government, where there are so many people and aspects to take into consideration.

There will always be resistance to change. We humans don’t like it. Even in our personal lives, we try to avoid it. Change seldom comes until you need it or you’re forced into it. So instead of trying to avoid it, let’s embrace it.

So yes, DevOps is possible in government agencies. And if it’s possible in this area, chances are that it’s possible everywhere else. The secret is in the strategy we choose to introduce these types of changes into complex organizations.