Christian Melendez

Kubernetes Autoscaling: Building Efficient and Cost-Optimized Clusters with KEDA and Karpenter

2025-09-01T00:00:00+02:00

I'm excited to share the news about my first upcoming book: "Kubernetes Autoscaling: Building Efficient and Cost-Optimized Clusters with KEDA and Karpenter" 🤩 ... this book reflects all the lessons learned after working with many, many customers around the world to build efficient Kubernetes clusters with Karpenter and KEDA. I can't count how many times I've talked about Karpenter already, but all those customer conversations have helped shape the content for this book.

The idea started as a focus only on Karpenter, but if we're honest, that's only one piece of the puzzle when we talk about autoscaling in Kubernetes. Therefore, I took on the mission to start with the basics so that we all understand why such tools exist today, and why it's important to know what impacts the performance of your application, its application patterns, and that rightsizing has a massive impact on efficiency. Moreover, as I really love getting hands-on, the book is packed with dozens of hands-on tutorials to help you learn by doing. So, if you're up with me in the mission of "saving the planet, one pod at a time," make sure you give this book an opportunity.

That being said, the book is soon to be published and it's ready for pre-order on Amazon and Packt. Also, if there's something you'd like to see in the book, there's still time to give me your feedback. So, please let me know either in the comments or message me; I'd be very interested to know what you'd expect to learn.

Where tu buy it?

If the link is broken, just search for "Kubernetes Autoscaling book", and you should find it. Otherwise, let me know.

Have a nice day!! I'll keep you posted about the official release date.

What is site reliability engineering (SRE) and how is it different from DevOps?

2019-04-05T00:00:00+02:00

Site reliability engineering (SRE) is Google’s model of service management where software engineers run production systems using a software engineering approach. It’s clear that Google is unique, and they usually need to tackle software bugs and errors in different and non-conventional ways. But having software engineers doing a job that is traditionally done by professionals with a systems administration background sounds impractical. Well, it was good enough to succeed, so Google decided to share their wisdom in a book format a few years ago.

After the book was released, SRE roles were embraced across the board by modern software teams like Atlassian.

In today’s post, I’m going to explain what SRE is and why SRE helps maintain software quality in production systems. You might notice that SRE looks like DevOps, so I’ll also explain how these two concepts relate to each other. Finally, I’ll share a few tips on how the hiring process changes with the SRE approach. Let’s get into it.

A software engineering approach to run production systems

Benjamin Treynor is the founder of Google’s SRE, and he explains that “SRE is what happens when you ask a software engineer to design an operations team.” Treynor’s background has always been software engineering, so it wasn’t a surprise that he decided to design an operations team with mostly development skills. Traditionally, operations tasks don’t scale well when they rely too heavily on humans doing things manually. Google’s SRE team ended up automating their tedious manual tasks, even if they were complex.

Just because the team was bored doing the same manual tasks every day wasn’t the only reason for adding automation. A common challenge when operating a production system is to keep it both up and running and as performant as possible. Usually, systems go down because we need to release new features or because the system infrastructure is running out of capacity. The latter can be automated. But the first one is more difficult because something is always going to go wrong.

That’s why SRE accepts failure and will try to fail in a controlled manner. One way SRE achieves this is by using an error budget.

Error budgets

An error budget tells you the minimum amount of reliability required. A system that is reliable 100 percent of the time is expensive, and users might not even notice if the system goes down for one minute. But users will notice when the system is down for an extended period, even if you don’t. Then what should you do?

Well, start by defining what level of reliability is acceptable for the system. This level will become the target you shouldn’t violate, but it should be achievable enough that it doesn’t lower the pace of change or affect innovation. When a deployment fails, you’ll be able to decide when to stop it and when to keep it going—and you’ll be able to do it in a controlled manner.

Reliability is the most important feature of an SRE team.

Now let’s talk about what the team does to be reliable.

What does a site reliability engineer do?

How do SREs keep to the error budget and have a reliable system? To answer this question, let me talk about the four core SRE principles an engineer will implement on a daily basis.

1. Ensuring an engineering focus

SREs intentionally spend a certain amount of time on reducing human labor, sharing knowledge among teams, and creating a blameless culture. Keeping track of system’s reliability. Knowing what’s happening inside the systems error reporting software is crucial. Engineers design the software to automate routine tasks results in a self-healing system. Humans will be notified only when their decision criteria are needed.

2. Bringing the system back online

How the team responds to emergencies is what allows them to keep on error budget when something goes wrong. Software engineering removes the human factor and helps to alleviate the pain of failing by recovering rapidly.

3. Maintaining compliance with change management

When removing the human factor from the equation, change management needs to have automation. By leaving a trail, this not only increases confidence that company rules aren’t ignored, but it also increases the deploy and release velocity by removing the time needed to make a decision.

4. Forecasting and provisioning the capacity of the system

SRE teams will provide the capacity when it’s needed and optimize the resources when they’re not required. Ensuring the capacity the system will need when there’s a demand for it is vital to maintaining the system’s availability.

The above list of practices and principles will not be a surprise for an operations team—it’s what they traditionally do. What’s different about the software engineering approach is that it’s working to keep a system reliable under the limits of its availability. The team will be less bored performing manual tasks, they’ll be interrupted less, and as a result, the team will become more creative in finding solutions to complex problems.

How does site reliability engineering relate to DevOps?

At this point, you’ve probably noticed some similarities in SRE with DevOps. {Link} DevOps isn’t a detailed guide on how to run production systems, which is why everyone has their own way of doing DevOps. Some people even say that DevOps is mainly a cultural mindset. Therefore, SRE looks like DevOps because of the similarity in the practices and principles. But as Google’s engineer says, SRE is the implementation of DevOps with a clear set of practices.

Let’s look at that idea a little more closely. The DevOps movement began with the goal of closing gaps between development and operations but without giving any detailed information on how to do it. That’s one reason why DevOps looks different in every company that decides to implement it. Whereas in SRE, the gap between teams is mostly closed by having the same set of tools and practices.

At Google, an SRE acts initially as a consultant for a developer’s team, working together with the developers to build the service. Then, developers can hand over managing the system to the SRE when there’s a clear understanding of what to do when something goes wrong—and by this time the system will be quite fault-tolerant and self-healing.

So SRE is not just another way of saying DevOps, but it’s also not a competitor trying to get rid of DevOps.

How to hire a site reliability engineer?

By now, an SRE team might look like a good fit for your company.

So how do you hire a site reliability engineer? Like Benjamin Treynor, that person should have a solid background in software engineering with experience in Unix system internals and networking.

In an SRE role, the engineer will be in charge of running production systems and automating >human labor through software.

An SRE needs to not only understand code but also be good at creating something from scratch. They should have experience with at least one programming language like Go, Python, or Ruby. An SRE who knows these languages will be able to extend tools like Ansible, Chef, Kubernetes, Docker, and Terraform.

A traditional sysadmin is still a good candidate for this role if they are willing to spend significant time figuring out how to reduce toil. (Toil is a term explained in the SRE book—it’s all that repetitive manual work that can be automated but is still being done by a human.) As the system grows, toil grows linearly, demanding more humans for the job. An SRE’s main function will be to reduce toil as much as possible.

An SRE will also have excellent troubleshooting skills to find solutions to issues and prevent them from happening again. Therefore, an SRE will use meaningful metrics to trigger automatic remediations and start troubleshooting when there’s a new problem, or if automation didn’t solve the problem. An SRE engineer might like to lean on an error monitoring tool.

Regarding infrastructure, an SRE needs to have experience with infrastructure as code, immutable infrastructure, distributed systems, configuration management, continuous integration, and continuous delivery. In other words, they need the expertise to build a resilient system that is easy to change and evolve over time.

In a nutshell, if you’re looking to build an SRE team, you should be looking for software engineers with infrastructure experience.

SRE isn’t just for Google

There’s a lot more to talk about regarding error budgets and SRE practices. I didn’t get into that many details, but hopefully, SRE is more clear now. Even though this way of working was born in Google, it’s not just for the giants of the software world. SREs are now employed by modern software teams to ensure their software is reliable for their customers.

As software developers take increased responsibility in SRE, there will be an increased reliance on software monitoring tools like crash reporting, APM, and real user monitoring.

If you want to learn more about SRE, there’s a video series from Google that’s worth watching it. The first SRE book is also available for free, and the latest one is an SRE workbook. More and more companies are already starting to implement SRE, which means that it will only continue to be adopted.

Why Transformational Leadership Is Needed in Every DevOps Initiative

2019-02-04T00:00:00+01:00

Every DevOps initiative needs support from leadership. Without DevOps leadership support, teams won’t be able to move forward smoothly. Leaders have a significant impact on results—not just with DevOps, but with every change initiative in an organization.

This time we won’t talk about DevOps as a tool or from a cultural perspective—although leaders very much influence culture. There are a few foundations in leadership that guarantee the success of a DevOps initiative, so that’s what we’ll be looking at in this post.

So let’s start by properly defining what transformational leadership is.

What Is Transformational Leadership?

Leadership is about showing your group the way toward its goals. A team is guided by its leader, but that’s not because the leader demands they follow.

Good leaders are the types of people that others follow because they identify with them. But only a few leaders have the power to truly transform the team. Wikipedia defines transformational leadership as:

A style of leadership where a leader works with teams to identify needed change, creating a vision to guide the change through inspiration, and executing the change in tandem with committed members of a group.

This type of leadership is focused on leaders inspiring and motivating the team to achieve its goals. Leaders will make sure to create the environment so the team feels safe to work with new things, like DevOps. Every DevOps initiative needs support from leaders, who in turn need to understand the value and the outcomes of implementing DevOps.

But more importantly, leaders need to be clear that DevOps is not something that has a finish date. DevOps is about continuously improving how we deliver software.

Some aspects of leadership are essential in every DevOps initiative. According to this study on transformational leadership theory, there are five dimensions that explain transformational leadership. Keep reading to find out what they are and how you can include them in your DevOps initiative.

Vision

Leaders need to know and understand the organization’s vision so that they can align the team’s initiatives with the company’s goals. There are some initiatives that leaders don’t think add value to the organization. And the team might think that leaders just don’t like to try new things because, like an old dog that can’t learn new tricks, they’ve always done things a certain way.

That’s why both leaders and the team need to understand why DevOps leadership is important. It’s so they’ll be able to stick to the initiative even when it seems easier to go back to old ways. When there’s a clear vision of where the company is heading, all DevOps efforts must be aligned with that. For example, if the company’s goal is to be 100% cloud-based, then you need to consider that in every decision you make about automating deployments. Integrating things like packaging and configuration management and creating resilient applications will become a must.

Vision is what will keep the DevOps initiative alive, so make sure you’re clear why you decided to start the journey. But more importantly, every member of the team needs to be on the same page, including the leader.

Intellectual Stimulation

There will be times the team won’t feel the need to change. They’ve been doing things the same way for a long time, so why change it if it’s working? The problem is that the team might not be aware of a critical fact: if they don’t innovate, they’ll take too much time to deliver a change, and the company could be losing a portion of the market share.

Leaders must challenge the team and not let them settle for practices that have become a comfortable habit. Always ask your team why they’re doing things a certain way.

The biggest trap a team can fall into is thinking that things will remain static. This is especially true in our industry, where new technologies and new ways of doing things constantly arise. A good leader will break the status quo and try to get the most from everyone. For example, my boss is always challenging us and creating competition inside the team to get the most from us.

Or to take another example, in a DevOps initiative, the leader could convince the team that doing manual deployments will continue to be frustrating. Instead (the leader may argue), they should automate deployments so that they can do more interesting things—things like re-architecting and decoupling portions of the system so they can easily achieve continuous delivery.

Inspirational Communication

Good leaders inspire and motivate the team by being aware of the team’s morale and then appealing to their emotions. As a leader, you should pay attention to what the team is complaining about and what they get excited about. Find ways to always keep emotions up. It’s impossible to be positive all the time, but at least make an effort to create an environment where the team feels motivated.

And motivation isn’t always about money. A developer is way more motivated when they feel they can have career advancement and increased autonomy in what they’re doing.

When you get to know your team, you’ll know how to help them to do awesome things.

The team needs to understand that change is good. And they need to know that they won’t lose their jobs if they automate the things they’re doing manually. As a leader, you’ll want to help them understand that what they’re doing has an impact in the overall system. Say things like “If we automate the release pipeline of this system, marketing will be able to run A/B tests to figure out how to increase sales.”

Things are going to continue to go wrong sometimes when a team implements DevOps. That means the team will need someone that’s supportive and constantly motivating them to keep pushing through, which bring us to the next dimension.

Supportive Leadership

Simon Sinek has a TED talk about supportive leadership called “Why Good Leaders Make You Feel Safe,” and I highly recommend you watch it. He noted that “in the military, they give medals to people who are willing to sacrifice themselves so others might gain” to make the point that leaders make you feel safe. When people feel safe, they start doing things they know the leader would do for them too.

Instilling fear is never a good way to manage your team. Trust is key. You don’t hire people to follow your orders—you can program a machine to do that. You hire people so that they can solve problems in the best way possible. Sometimes they’re going to make mistakes, but they shouldn’t be punished for that. When they’re punished, they become afraid to try new things and innovation stops.

As a leader, you should expect and embrace failure. A team will do interesting things in an environment where they can practice without the fear of doing things wrong. They know the leader trusts them. DevOps fosters innovation because it’s built on the notion that you can recover from failure quickly.

Personal Recognition

Who doesn’t like to hear a “thanks, well done!” when they’ve just finished their work? Everyone wants to know that their efforts are successful and appreciated.

When the team feels that the work they’re doing is being noticed, they’ll continue doing what leaders reward. If the team is staying late or coming in on weekends to do deployments, always be thankful for it.

To help your team avoid frustration and burnout, you should apply inspirational communication and intellectual stimulation to challenge them. That way they don’t have to work extra time and the company will be able to deliver changes more often.

If a person is proud to work with your team, they’ll highly recommend the company to others. That’s a good sign that the environment is the proper one. One way Google was able to make teams do a post-mortem analysis after every incident was to give a visible recognition in the all-hands meeting from the CEO. Just imagine how proud a person would be if what they did had such an impact that even the CEO was informed.

In your DevOps initiative, you need to recognize all milestones of improvement. Implementing DevOps will require a lot of discipline and will not be easily achieved. Celebrate your team’s accomplishments along the way!

Once You’ve Done It, Just Keep Doing It

There’s never, ever going to be enough transformational leadership—once you’ve done it, you need to keep doing it. Good leaders need to keep pushing the team to achieve new goals. Sharing the company vision with the team will help keep you focused because everyone knows where they’re heading.

Try to get the most out of your people. Always challenge them to do better. The worst thing you could think is that there’s nothing left to improve—things will always change. You easily get the maximum performance from a team if they’re motivated and you inspire them to take action.

Lastly, support your team. Things are going to continue to go wrong, but don’t punish them. Instead, learn from mistakes. Whether you resolve to do better training or you enforce automatic validations, above all, let your team know that mistakes are an opportunity to improve.

Pitfalls with DevOps at Scale

2019-01-28T00:00:00+01:00

Let's get started by defining what DevOps is.

I know, I know; there are tons of definitions. But the one I like most is from Gene Kim:

DevOps is those set of cultural norms and technology practices that enable the fast flow of planned work from, among others, development, through tests into operations while preserving world-class reliability, operation and security. DevOps is not about what you do, but what your outcomes are.

Some might think that because DevOps is hard to implement, it’s not for everyone, especially not large organizations. That’s not true. Outcomes are what really matter, not how you get there. That’s why I like this definition; it shows that DevOps really is for everyone.

As with almost everything in IT, there are pitfalls that can keep you from reaching your expected outcomes. But before we get into them, remember that for the purposes of this article, we’re looking at DevOps at scale. So first, let’s define our terms.

What Does “At Scale” Mean?

An organization that’s operating at scale is able to grow to meet greater demand without too much hassle.

In a small startup, disruptive actions aren’t that difficult to implement—there’s not much to lose, and you need to meet deliverables fast in order to get more funds. With only a few people, it’s easier to communicate and reach agreements. But when more people are involved and the application becomes more critical, things start to change.

Organizations that run at scale have a whole host of issues to consider that small startups do not. They usually need to deal with compliance and auditors. They have to coordinate many developers, each with a different mindset and knowledge base. And sometimes, they depend on partners in order to solve problems faster, no matter that doing so costs more money in the short term.

Big, scaled-up organizations also like to make longterm plans and infrequently release app changes because the impact of things going wrong would mean losing tons of money. It’s understandable, then, why these organizations are very cautious of anything that might speed up their work (like DevOps).

So yes, things are different at scale. And at scale, DevOps has some specific pitfalls that you need to take care of.

Difficult Coordination Between Teams

The more people are involved, the more difficult it is to coordinate teams. That’s not just because as humans we’re not great communicators. It’s also because having more people means that there are more dependencies and applications are more complex.

Not knowing the application’s dependencies is a big pitfall that can keep you from being able to confidently release code. If you don’t know how parts of your application relate to each other, you might end up fixing one thing while breaking another. It’s necessary to have a shared understanding of the impact of each component in the system.

I’ve heard that at Netflix there’s no single person that knows every part of the system in detail. That’s problematic, and that’s why Jeff Bezos says that, at Amazon, developers should always think about exposing services through APIs. Doing so means that every interaction between different teams is clearly documented, and everyone works in a culture that values knowledge-sharing.

Picking the Wrong Projects

You can’t eat an elephant in one piece; you need to eat it in small chunks. And you can’t apply DevOps to the organization all at once; you need to do it little by little. Every time you choose a project, you need to know all its dependencies, its impact, and how stable it is. This is especially crucial for the first project.

You first need to establish some base knowledge and start simple. Don’t get overwhelmed because everyone is talking about cool things like infrastructure as code, containers, or microservices. Sure, those things can help, but you don’t need all of that to improve the outcomes of DevOps. Why don’t you simply start by getting rid of manual changes like deployments and leaving a trace to make changes visible?

Find out what aspects of your application are not adding value and focus on those. Automation is just one example.

When starting out, pick a project that is complex enough to do interesting things and that has a low enough impact on revenue so as not to make too big a mess things don’t go well. You can then scale and replicate what you learn in the process. Soon you’ll see better outcomes as an organization, not just by project.

Picking the wrong projects could make everyone think that DevOps is making things worse.

Lack of a Well-Established Framework

Employees last an average of two years in the big tech companies. Make sure you’ve established a framework that works for you before scaling DevOps. This will help with both onboarding new employees and ensuring that when someone leaves, his or her knowledge will be retained.

Just make sure you don’t create strict rules that are too difficult to change. For instance, you could make practices like infrastructure as code mandatory because you’ve noticed that the labor of maintaining infrastructure is not adding value. But you could let the team choose which tool to use to implement infrastructure as code.

Other practices you build into your framework could include trunk-based development, feature flags, test-driven development (TDD), not SSH’ing to servers and using centralized logging, among others.

Let’s take the Google example. Google runs production systems at scale by having site reliability engineers (SREs) that sometimes function as temporal consultants. SREs make sure that the team stabilizes the project such that it gets more reliable. They have a checklist of things that the team needs to do to make sure ensuing changes comply with Google’s standards.

Large organizations need a standardized way of working; otherwise, the turnover will be too problematic.

Lack of Production-Like Environments

Production-like environments could be difficult and expensive if you don’t start out with the mindset of working with homogeneous environments when creating the systems.

Preparing test environments is no easy task; preparing production-like environments is even trickier. But if this process is not automated as self-service, the path to deploying in production could be painful.

Having a production-like environment doesn’t mean you’ll have an exact copy of the environment for development or testing. It means that if in production you have, say, a load balancer, you also have a load balancer everywhere else.

Your production-like environments do not have to be at the same scale as those in production. But quantity and capacity of the resources should be your only difference—unless you need to do performance testing, but that should be a matter of scaling out the infrastructure.

Why is having homogeneous environments important? Well, they’ll let you do deployments the same way in all environments and run experiments before you go live. You don’t want any surprises when you make changes available to your users.

Lack of Meaningful Metrics

How sure are you that the things you’re doing are adding value? What’s the impact of automating certain manual processes? If you’re automating something that’s rarely used, the effort may not be worth it. If you’re using trunk-based development and spending too much time resolving merge conflicts, maybe you should switch to short-lived branches instead.

It’s important that every decision you make is based on data, not assumptions or because everyone is doing it. According to the state of DevOps report from last year, these are the key metrics you should be measuring:

Deployment frequency. Small batches are less risky.
Time it takes to do deployments. It should be a deterministic and boring task.
Number of times a deployment has failed. There might be things you didn’t consider before going live.
Time it takes to recover from a failure. Things like rollbacks or self-healing architectures could help.

Measure meaningful things. Otherwise, you won’t know what’s working to improve the outcomes of DevOps.

Know Your Weakness and Get Stronger

DevOps outcomes take time to be seen. That’s to be expected—you need cultural change before you can make real progress with different processes like DevOps. And changing people’s mindsets is usually hard to do. It’s in our nature as humans to avoid the feeling of discomfort that accompanies change.

In your continuous DevOps journey, measurement is crucial. You need to know what and where your weaknesses are in order to get stronger and achieve better outcomes for the organization. All industries and organizations are different. Some will encounter all the pitfalls I mentioned in this list; others, only a few.

Remember: DevOps can benefit your organization only as long as you set yourself up for success. If you’re putting time and effort into improving outcomes but aren’t adding value, you aren’t serving the organization. Check to make sure you aren’t falling prey to one of these pitfalls before applying DevOps at scale.

Getting Started Quickly With Go Logging

2019-01-21T00:00:00+01:00

It's time to talk about how to get started with logging again. The languages we've covered so far are C#, Java, Python, Ruby, Node.js, and JavaScript. Today, we're going to be talking about the Go programming language, also known as Golang. Go is a statically compiled, open-source programming language created by Google in 2009. It runs on Windows, Linux, and Mac and has been increasingly adopted in the last couple of years because of its simplicity and concurrency mechanisms. And in case you didn't know, Docker and Kubernetes were written in Go. Go is quite an interesting language when we talk about logging. Why? For one thing, it includes a really simple native library for logging. But the most interesting thing is that in Go there are no exceptions—only errors. That means when you want to log application errors, things get interesting. We'll cover that later in this post. But let's start by taking a look at how easily you can get started logging in Go.

The Simplest Go Logging That Could Possibly Work

Personally, I prefer to use the JetBrains offer for Go, but in this tutorial, I'm going to use VS Code because it's free. Let's start by creating an empty folder. In my case, I created the folder at C:\Projects\go\go-logging. Open VS Code. Then, click on File > Open Folder. Go to the location of the folder you just created and select it by clicking on the Select Folder button: In the left panel of VS Code, click on the icon indicated below to create a new file. Name the new file console.go. The left panel should look like this now: As I mentioned before, Go has a simple native logging library that you can use. By default, the output goes to the standard error of printing the date and time of the messages logged in the application. The simplest code that you can use to write logs is the following:

package main

import (
    "log"
)

func main() {
    log.Print("Logging in Go!")
}

In order to easily test the code without leaving VS Code, let's enable the terminal by clicking on View > Integrated Terminal. Because Go is a compiled language, we need to compile it by running the following command in the terminal:

go build console.go

There shouldn't be any errors when running this command—the absence of a message means it succeeded. In the same terminal, run the following command to execute the compiled program:

.\\console.exe

The terminal will display the date, time, and message logged, like this: Great—it works! But what if we need to store the logged messages in files? For that, you need to create a new file (or open an existing file), then set that file as the output of the log, like the following code:

package main

import (
    "log"
    "os"
)

func main() {
    file, err := os.OpenFile("info.log", os.O\_CREATE|os.O\_APPEND, 0644)
    if err != nil {
        log.Fatal(err)
    }

    defer file.Close()

    log.SetOutput(file)
    log.Print("Logging to a file in Go!")
}

Now go back to the terminal and build and run the program. This time, no output will appear in the terminal because it was written to the file. In the left panel, you'll see the info.log file. Open it, and the same message you saw in the terminal should appear here: And that's it! You just created your first Go logger. But as simple and easy as this could be, hello-world examples are not enough. Log messages could get messy and difficult to read if we continue logging manually like we just did. We need to know a few other things before we can seriously integrate logging into our applications.

What Is Application Logging?

Let's take Erik's definition from the first logging series post:

Application logging involves recording information about your application’s runtime behavior to a more persistent medium.

So what exactly does that mean? When problems arise, we want to know what caused the issue. I wish I could be physically inside the server to watch what's happening in real time to spot the problem. But as far as I know, humans can't be inside a server physically or virtually. Right? But seriously, in a world where containers are getting more attention every day and the cloud is becoming de facto, instances of our application are very volatile. One day you could have access to logs on a server, and the next day the server could be gone. Even if you stored your logs in files locally, you lose valuable information when things like this happen. You need to have a persistent medium so that you can do post-mortem analysis more accurately.

Why Bother Logging? What’s the Motivation?

I remember my first days at a new job several years ago. I was learning how the application worked, and one day I decided to pair-troubleshoot in production with one of the original programmers. At that time, the company was a startup, so developers had access to production servers. In order to know what was wrong with the application, we needed to scratch the log files. But the application depended on several microservices, and there were times that one of those services was down. The problem wasn't really that one service was down. The problem was that we had to go through several files before we knew what was happening. So we decided to change the approach and started sending the log files to a centralized logging platform—a more persistent medium. When you have application logging in place, you'll be able to:

Resolve problems faster because you won't need to go to each server to read logs.
Have a standard format to quickly spot issues. (If you read logs on a server, log formats can vary greatly from one application to another.)
Enable visibility to everyone that needs it. When you move logs to a centralized place, you can easily give access to developers so they can see for themselves what's happening with the application without having access to the servers. It also helps you avoid the temptation to implement a fixhr inside the servers that could make things worse.
Automate notifications so you know when problems are happening. You might be able to catch and fix problems before your users notice it.

In a nutshell, you need application logs to go back in time and understand how the application behaves in the real world.

What Should You Log?

The data that companies are storing with applications are becoming more relevant each day. Want proof? Think of the companies going to court to explain what they're doing with the data they collect. Or consider compliance initiatives like GDPR, SOX, or HIPAA. We need to be careful about what we put on those log files. And let's be honest—there's a lot of data we'll never need. Let me give you a short list of what I've found useful to log:

Date and time of the event in a standard timezone like UTC.
In distributed systems, a tracking flag to replicate the request across all the different points of the system.
Data that will help you to understand and make diagnostics of the application—things like error stacks, the external service name, header's request, the URL path and its parameters, and similar kinds of data that will help you to reproduce the error and understand what happened.
Context data that you might need for all events like the ones I mentioned before so logging doesn't get bulky in code.
The environment where the application is running—you could easily ignore anything else that's not production.
The IP address of the server so you'll know where the problem is.
Log levels such as panic, fatal, debug, or informational logs to give more context.

With the above data, I've been able to spot problems quickly and easily. Now, let's take a look at how you can easily log in Go by using a logging framework. With the simple approach we saw before, we might have some spaghetti code that will be hard to maintain and extend.

Enter the Logging Framework

Logging is an already-solved problem, and there are logging frameworks that will help you log easily. A framework will let you do the same things we did at the beginning of this guide but with fewer lines of code and a standard way of logging. Logging frameworks cover almost all common scenarios, so it's better that you choose a framework rather than do everything manually like in my first example. (The only case where I'd say you need to think twice about which framework you use is if your application's performance is crucial.) In the Go ecosystem, there are actually a lot of logging frameworks. You can see them listed on the Awesome Go page on GitHub. The two most popular logging frameworks are Glog and Logrus. Google is the author of Glog. It keeps the simple style from Go, but no one has updated the GitHub repository for two years now. On the other hand, Logrus has been constantly supported by the community. In this guide, I'm going to use Logrus. Now, even though I like Glog's simplicity, there are valuable things from Logrus that might be worth taking a look.

Getting Started With Logrus

Let's start by installing the Logrus module by running the following command in the terminal of your computer:

go get "github.com/Sirupsen/logrus"

We'll keep updating the code that we used before just to demonstrate how easily you could introduce logging frameworks in Go without having to change too many things in the code. So let's replace the log import with the Logrus import, like this:

import (
    "os"

    log "github.com/sirupsen/logrus"
)

Keep everything as it's now in the code and run it. The output of the log will look like this:

2018/05/22 17:02:43 Logging to a file in Go!
time="2018-05-22T17:33:03-06:00" level=info msg="Logging to a file in Go!"

A new line was added but in a different format: the default Logrus format. We're still seeing the timestamp and the messages logged, but we're also seeing the info level. What does "level=info" mean? It's the category of a message in the log. And why would you need levels in logs? Well, let's explain that with some readable code from logrus GitHub page:

log.Debug("Useful debugging information.")
log.Info("Something noteworthy happened!")
log.Warn("You should probably take a look at this.")
log.Error("Something failed but I'm not quitting.")
// Calls os.Exit(1) after logging
log.Fatal("Bye.")
// Calls panic() after logging
log.Panic("I'm bailing.")

Order lines here are very important. I'll explain why later in this guide. Log levels don't just give more context to the logged message by categorizing in groups. Log levels are also useful when you don't need to have too much information about the application's behavior. And vice-versa, there will be times when you'll need to have all the information available to make a better analysis. This will also help you make better use of the storage or data transfer in your infrastructure. Let's add some interesting code here. Then I'll explain the lines that are adding something different to the application that we've been coding.

package main

import (
    "os"

    log "github.com/sirupsen/logrus"
)

func main() {
    file, err := os.OpenFile("info.log", os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0644)
    if err != nil {
        log.Fatal(err)
    }

    defer file.Close()

    log.SetOutput(file)
    log.SetFormatter(&log.JSONFormatter{})
    log.SetLevel(log.WarnLevel)

    log.WithFields(log.Fields{
        "animal": "walrus",
        "size":   10,
    }).Info("A group of walrus emerges from the ocean")

    log.WithFields(log.Fields{
        "omg":    true,
        "number": 122,
    }).Warn("The group's number increased tremendously!")

    log.WithFields(log.Fields{
        "omg":    true,
        "number": 100,
    }).Fatal("The ice breaks!")
}

Let's start with this line:

log.SetFormatter(&log.JSONFormatter{})

One of the main purposes of using a framework is to have log data standardized. This is not only because it will be easy to read and understand; it's also because the idea should be to move this log data to a centralized logging platform so that you can read logs from several sources in one place. JSON is the standard format when you want to move logs somewhere else. That's because JSON's key/value style will make it easier to parse the data.

log.SetLevel(log.WarnLevel)

This line sets the log level we talked about before. Remember when I said earlier that the order of lines is important? The reason is that the log level also sets the importance of the logged message. The first level, debug, is the least important. And the last level, panic, is the most important. When you set a level in the code, as the line above does, you're saying that all the log messages from that level down will be logged. In this case, all debug and info messages won't be written to the log.

log.WithFields(log.Fields{
    "omg":    true,
    "number": 100,
}).Fatal("The ice breaks!")

I also mentioned that it's important to include some context to the logged messages. In this line, we're doing it by using the WithFields and Fields functions. That's dummy data, but in the real world, you could use a data structure like the request parameters so that you can reproduce the problem. Let's compile and run the application. The log will look like this now:

2018/05/22 17:02:43 Logging to a file in Go!
time="2018-05-22T17:33:03-06:00" level=info msg="Logging to a file in Go!"
{"level":"warning","msg":"The group's number increased tremendously!","number":122,"omg":true,"time":"2018-05-22T17:52:47-06:00"}
{"level":"fatal","msg":"The ice breaks!","number":100,"omg":true,"time":"2018-05-22T17:52:47-06:00"}

You can now see the difference between this and other approaches we took in this guide. Only the warning and fatal log messages were written. If you want to take a look at the code, you can check out the different approaches I've uploaded to GitHub as examples. I can't stress enough that you need to move these log files to a centralized logging platform. Even though you now have a standard format and it's easy to add logging to your applications, you can't keep going to the servers to read logs. It will be time-consuming as the application keeps growing.

Where Do You Go Now?

Logging frameworks are there to help you add logs to the application with just a few lines of code. What I really like about these frameworks is that you can guarantee that important data (such as timestamps, context, and log levels) is logged. Log levels can become your best friend when problems arise and you need to troubleshoot. By using log levels, you avoid being ashamed in demos with funny log messages that you might accidentally leave in the code. Go's native library for logging is really simple, but if you want to know more about it, you can take a look at the docs site. In this post, I only just scratched the surface of Logrus logging framework. You can find useful information and examples by just going to the home page of the project on GitHub. You can configure hooks, log messages according to the environment, use formatters, and a lot of other things as per the docs site. Since we just learned how easy it is to add logs to your applications, there's no reason why you shouldn't! If you're worried about performance, just make sure you can turn off logging or easily change log levels. Trust me, you won't regret it—you'll need logs when you want to find out what's wrong.

Deployment Smells: The 5 Most Common Deployment Mistakes

2019-01-14T00:00:00+01:00

Deployment shouldn't be the most nerve-racking task for sysadmins when releasing software, but many times it is.

If we’re deploying application changes to a beta system, we don’t care that much if the system goes down. Neither will your users—they know that beta means “expect problems from time to time.” The same thing applies to new systems that don’t have a large user base yet.

But what if we’re talking about bigger systems, like those used by a major online retailer or a bank? Users might get frustrated if they have problems shopping online. They may choose to buy from a different store as a result. And they’ll be furious if they aren’t able to access their bank account to make a withdrawal. Even if it’s just for a small fraction of time, companies lose credibility every time their systems are down or malfunctioning.

Deployments tend to get really difficult. Just how difficult is proportional to the complexity of the system you’re working with. The more dependencies the system has, the riskier and more problematic deployments get. But what are the common mistakes that we’ve all made?

1 Not Enough Automation

Manual deployments lack consistency. So, when a significant portion of your procedures are manual, you wind up with a big ball of mud. I have suffered the consequences of doing deployments manually. I vividly remember one time when the site was down intermittently because I forgot to upload the binary files to 1/4 of the servers in the farm.

With the rise of DevOps, automation is becoming de facto. But don’t get overwhelmed if your process is still largely manual. You don’t need to stop what you’re doing and automate everything. I’ve always loved the boy scout rule to leave things better than you found them. It’s OK to start manually because it’ll help you understand how everything works. But do identify where automation can be used_._ You might want to be able to go from 0 to 100 to declare the task of automating your process “complete,” but automation should be done progressively and continuously.

The only manual thing you might want to have is someone approving the release, but even that should be a matter of just clicking a button. So, it’s not just about copying and pasting things; it’s also about configuration management, infrastructure, on-demand environments, and so on.

When you automate, you foster communication. If there’s only one way to make changes in deployment, everyone will know the what, why, and how of things. And if everyone is in the know, the guy or gal that solved a complex deployment problem can go on vacation without having their mai-tai-at-the-beach time interrupted by a clueless colleague back at the office.

2 Different Environment, Different Way

I once fell into the trap of automating deployments by environment. In my case, there were four environments: dev, QA, staging, and production. I took my time to finish with the dev environment. Then I needed to replicate those changes in the other environments. The problem was that the deployment process for each environment was completely different in most cases.

I’m a big fan of the twelve-factor app methodology. Regarding deployments, it says that you need to keep config values in the environment, not in the application or any file. It also says that environments should be as similar as possible—not in size, but in style.

In other words, if you lack production-like environments, you’ll be lying when you say you’re ready to deploy to production. I’ve seen it, I’ve suffered it, and I stopped treating production as some sacred environment to be feared.

When I was part of a project where we Dockerized the system, I understood this very clearly. I used concepts like immutable infrastructure, infrastructure as code, environment configuration management, cloud, and deployment definition files with Kubernetes as pieces of the deployment puzzle to make things easier.

3 Waiting for Low-Traffic Time Slots

Any time you hear (or say) “Let’s wait for the weekend to make this deployment,” it’s a sign the deployment pipeline isn’t trustworthy. You’re still treating production significantly differently from other environments.

Netflix and the other “unicorns” in the industry say they actually do deployments during regular work hours. After all, that’s when everyone is together and ready to react properly if there’s an incident. I know you might not be as big as the unicorns, but there’s nothing special about them aside from one thing: they experience problems at a really large scale. That forces them to change the way they’re doing things.

As I said before, you probably do deployments during low-traffic hours or over the weekends because you want to prevent any downtime in the system. But instead of living in fear of screwing things up, why don’t you set the foundation to experiment?

Do deployment strategies like feature flags, canary releases, blue/green deployments, or rolling updates ring a bell? Maybe, maybe not. Either way, these are great strategies that help you make deployments without users noticing it. And it’s a way to avoid stealing family time away from the staff.

4 Waiting Until Enough Changes Accumulate

Do you have changes that are ready to be deployed? Why wait? To take one out of Nike’s book, just do it!

OK, so maybe something official is delaying you. Maybe you’re waiting for the change advisory board (CAB) to approve the changes based on the risk. I’ve been there too. But maybe you just need to improve communication by automating some steps in the process. These changes don’t come overnight—it takes time for the CAB to trust the process.

Another reason why organizations wait to do deployments until enough changes have been accumulated is, again, to reduce downtime. Downtime usually happens because you’re changing something. It rarely happens because of nightmarish things like the cloud provider going down, fires, earthquakes, and so on.

But I’ll say it again: don’t wait! Work to build a culture that deploys frequently and in small batches. If it hurts, do it more often. Every sprint, you can deploy something that can easily be disabled with a feature flag. Once you decide it’s ready to be used by everyone, turn the feature on to release the changes.

When you deploy too many things at once, you don’t know which of those things might be causing problems. And when you don’t know what’s causing problems, you have to roll back everything, including perfectly functional features that may have been really critical for the system.

5 A Lack of Subsequent Feedback

You might have good monitoring and observability for your systems, but how sure are you that problems are being caused by a recent deployment and not by something else? There are lots of possible factors, such as internet problems, cloud provider issues, and an increase in traffic.

Another problem is that after a deployment, we tend to monitor changes for a while and then forget about it. But if you end up working with feature flags, it can take some time to activate or release a change. For that reason, it’s important that you have a smart way to correlate events, logs, and any other information you’ve found useful as you’ve learned the system’s behavior.

It’s also important that you have feedback after deployments because that’s how you’ll know if something needs to be rolled back. Without reliable feedback, any issues will just continue to affect your users. Let your team know when a deployment and release happen. It could be as simple as a Slack message in a channel. That way, people can “subscribe” to get notified, ask questions, and learn how the team solves problems.

Shift Deployments to the Left

In a nutshell, don't treat production differently from your other environments. If you want to have a high rate of successful deployments, practice them before going live. I've even been in the situation of announcing a release to the CAB for a certain date and then having to delay release dates because we found problems when we were launching the B side of our blue/green deployment.

I've also come to the conclusion that most of the things I've written here are more of a cultural change than anything else—a mindset that has to be adopted by the team that's involved in building the software, not just developers and operations.

Start small and keep analyzing areas where you're consistently struggling during deployment. Then improve. The end goal should be that, at some point, you might leave releases to marketing or management because you already took care of doing deployments with almost no downtime.

And how do you get there? Practice until you've mastered deployments.

A Detailed Guide to Canary Deployments

2018-10-22T00:00:00+02:00

Every time we need to deploy to production, we worry about how changes will affect the user experience. No matter what technique or strategy you use to make deployments, there are going to be times when the things that can go wrong will go wrong. It’s Murphy’s law. The fact of the matter is you can’t always prevent errors when deploying changes. But what you can do is prevent downtime after a deployment from becoming the norm. One way you can minimize downtime is by implementing canary deployments.

It doesn’t matter what type of application you have—canary deployments can be applied whether it’s a website, desktop, or mobile app. They also work with on-premises, cloud, or hybrid infrastructures. Containers? You bet. In this post, I’ll be demonstrating how to do canary deployments using Docker and Kubernetes as a hands-on example.

What is a canary deployment?

Let’s start with the origin of the name canary deployment, which comes from an old British mining practice. It’s a little bit cruel. According to Kat Eschner at The Smithsonian, this practice consisted of using “canaries in coal mines to detect carbon monoxide and other toxic gases before they hurt humans.” To make sure mines were safe for them to enter, miners would send canaries in first; if something bad happened to the canary, it was a warning for the miners to abandon the mine.

So how is this comparable to software development and deployments?

Well, when you want to release a new version of an application, you’ll use a canary to make sure the new changes can survive for a broader public. It’s just a matter of defining who the proverbial canary will be. For simplicity, some organizations decide that the canary will be five percent of the traffic. So if something goes wrong with the new changes, only five percent of users will be affected. You can also use more canaries of different colors and sizes in order to increase your confidence in the new changes.

After you feel confident that the new changes are good to go, you roll out the changes to the rest of users. You can now get rid of the canary since it was only used to make sure the miners (your broader audience) wouldn’t be negatively affected by the new changes.

How do you do canary deployments?

As I said before, if you want to do canary deployments, the type of application or infrastructure you’re working on isn’t too important—canary deployments are pretty versatile. Let me give you some specific examples of how to do canary deployments on different infrastructures using only one canary.

Load balancers

If you want to do canary deployments with load balancers, you’ll start by having two of them. Load balancer A will receive 95 percent of the traffic, and load balancer B will get the remaining five percent. We’ll use load balancer B for canary deployments, which means we’ll update the servers behind it first. It doesn’t really matter how many servers are behind each load balancer—the important thing is that traffic is split between the load balancers, not the servers.

(Note that you can do canary deployments without load balancers, but it’s easier with them because you can take the server you’re updating out of the rotation.)

When you’re done updating the servers, you monitor and test for a while. If you don’t like the results in load balancer B, you can roll back and finish the release. But if the new changes are good to go, you can proceed to update the servers that are behind load balancer A.

Amazon Web Services (AWS)

If you want to do canary deployments with AWS, you’ll use their service called Route 53, which lets you use weighted records to split the traffic 95/5, like we did before. Or you can use auto-scaling groups, if you don’t want to mess with DNS TTLs.

But if you don’t care that much about having a strict split like that, you can get the same results using Auto Scaling groups (ASG). The magic of ASG is that, even if you have more than one ASG, you only need one Elastic Load Balancer (ELB) because you’ll be attaching all the ASGs to the same ELB. This balances the load between ASGs. So let’s say that one ASG has nine instances (three for each availability zone). A second ASG has only one instance. This gives you a total of ten instances. Then, if we use the second ASG as a canary, you’ll have 1/10 instances for that purpose.

To do canary deployments with ASGs, you apply the same technique as with load balancers but modify it slightly. You update the ASG with the new Amazon Machine Image (AMI)—which has the new changes—and replace the current instance with the new one. You don’t have to worry about downtime because you can launch the new instance first. Once it’s registered in the load balancer, you can drain connections with the old instance and terminate it.

With this technique, rollback is as easy as updating the ASG with the previous AMI and then rotating instances. If you decide to continue with the deployment as is, you just do the same procedure with the other ASG that has nine instances.

Kubernetes

Now with Kubernetes, the approach to doing canary deployments is similar to the one used in AWS, but we use different terminology.

You have a service that could act as a load balancer for pods that are “registered” to the service. You also have “deployments,” which act like ASGs. So in theory, you apply the same techniques. But instead of just explaining it like I did above, let’s learn by practicing.

Containers and Docker

Now, let’s talk about how to do canary deployments with containers and Docker.

Containers are what you use to pack the application code and its dependencies so that you can move your application without too much hassle.

Docker is the company that made containers popular. When someone talks about containers, there’s a good chance they’re referring to the containerized technology that Docker created. Containers are pretty easy to use; all you need to have is a file called Dockerfile, where you define how you build the app and what dependencies it needs. This file is used to create what is called a Docker image, which is later used to instantiate a container.

Since all you need to do to use Docker containers is have the Docker daemon installed, you can move the application through all the environments without fear. Everything that’s needed for your app to run is right there in the container image that you built.

Kubernetes basic theory

Now that we’ve covered how to do canary deployments on different infrastructures, let’s talk about the Kubernetes theory you’ll need to know in order to do a canary deployment.

Deployments

Deployments are the desired state definitions for pods and replica sets. A pod is a group of one or more containers that you run together. Think of a pod like a virtual machine where you “install” and “configure” the application and its dependencies—we both know it’s not actually a virtual machine, but bear with me. A replica set is used to define the number of pods that are running. If for some reason a pod is stopped or destroyed, Kubernetes will create a new one to maintain the desired state.

Deployments then define which container images and versions will run in pods, how many of them will run, what strategy will be used to update pods, and what metadata can be used to interact with other resources.

Services

You can think of services as load balancers for pods. Let me explain: each pod has its own IP address, but if the pod is recreated or there’s more than one, consumers of those pods will need to keep track of those IP addresses. Instead of doing this manually, you could use a service. Services also provide an IP address that consumers can use, but the real benefit is that the service will balance the requests among the pods that are under it.

There are other use cases for services, but for this time, let’s leave it at that since we’ll be using services in our practice example.

Labels and Selectors

Finally, labels and selectors are the metadata that other resources need in order to interact with each other. Labels are key/value pairs that you can define as you want—they’re used to identify objects. Because you can define as many labels as you need, some objects won’t need to have the same labels. So a label selector is needed to select objects using one or more labels.

Labels and selectors are how everything we’ve just discussed is connected. When you define the pods in a deployment, you describe a set of labels that a service will use to balance the load. This is how pods are “registered” into a service.

Canary Deployment Step by Step

Enough theory; let’s get started. Let’s mimic what a developer’s day-to-day work will look like when using Docker and Kubernetes.

0. Prerequisites

Before we start, you need to have the following installed on your computer:

Docker Edge. If you’ve already installed Docker, you need to uninstall it and then install the Edge version. Why Edge? Simply because it includes a local Kubernetes installation that will make your life easier. It’s supported on Windows and Mac.
Minikube if you’re using Linux. It’s also supported on Windows, but then Docker needs to be installed in a non-native fashion.
kubectl is a must because this is how you interact with Kubernetes using the command line. It’s supported on Windows and Mac, too.

Once you have Docker, Kubernetes, and kubectl in your local environment, you’re ready to get started.

1. Create the Docker image for the application

You need an application to deploy in Kubernetes. In my case, I’ll use a .NET Core API template, but you can use an existing application.

If you want to use mine, the code is on GitHub. You simply need to clone it in the root folder of the repository and run a few commands to create the initial Docker image version. Replace my username “christianhxc” with your username.

Run the following command to create the Docker image version 1.0:

docker build -f HelloWorld/Dockerfile -t christianhxc/helloworld:1.0 .

Next, run the following command to push the Docker image version 1.0 (this is optional since our case is running locally):

docker push christianhxc/helloworld:1.0

You’ll see that the image is now visible in the Docker Hub:

2. Deploy version one to Kubernetes

Now that we have an initial version of the application, we can deploy it to Kubernetes.

Let’s start by creating the service definition file k8s-service.yaml. It should look like this:

apiVersion: v1
kind: Service
metadata:
  name: helloworld
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: helloworld

As you can see in line 8, this service will run in port 80. Note how the service is going to select pods in lines 9–10. It will use the label “app: helloworld” as the selector. The pods that have that label will be included in the service.

Now run the following command to create the service. If the service already exists, it will be updated:

kubectl apply -f k8s-service.yaml

Now it’s time to create the deployment. Let’s create the deployment definition file k8s-deployment.yaml, which should look like this:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: helloworld
spec:
  replicas: 2
  strategy:
    rollingUpdate:
    maxSurge: 1
    maxUnavailable: 1
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: helloworld
        track: stable
    spec:
      containers:
      - name: helloworld
        image: christianhxc/helloworld:1.0
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 50m
          limits:
            cpu: 100m

In this deployment definition file, you see that in lines 15–16 there are two labels: app and track. This is where the magic happens. Pods in this deployment will have the “app:helloworld” label that the service is looking for. The service will balance the load to the two pods (line 6) created by this deployment.

Let’s create the deployment in Kubernetes by running the following command:

kubectl apply -f k8s-deployment.yaml

You can check that pods are running with the following command:

kubectl get pods

Run this command to get the “EXTERNAL-IP” address of the service:

kubectl get service

This is where you will get a static IP address if you’re running on the cloud, but in this case, we’re running a local Kubernetes so the “EXTERNAL-IP” is localhost.

Go to http://localhost/api/values and you should see API Version 1.0 running:

3. Create the Docker image for version two of the application

If you decided to use my application, go to the HelloWorld/Controllers/ValuesController.cs file and change the version in the first method so it looks like this:

Run the following command to create the Docker image version 2.0:

docker build -f HelloWorld/Dockerfile -t christianhxc/helloworld:2.0 .

Then, run the following command to push the Docker image version 2.0 (again, this is optional as our case is running locally):

docker push christianhxc/helloworld:2.0

4. Deploy version two to Kubernetes using a canary

Version 2.0 is ready, but we want to deploy it safely.

Let’s create the canary deployment file definition k8s-deployment-canary.yaml. It should look like this:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: helloworld-canary
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 5
  template:
    metadata:
      labels:
        app: helloworld
        track: canary
    spec:
      containers:
      - name: helloworld
        image: christianhxc/helloworld:2.0
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 50m
          limits:
            cpu: 100m

In line 6, you can see that this deployment file is saying that it will create only one pod. As we saw in the “stable” deployment file, you now have two pods running. That means that the canary version will now receive 33 percent (1/3) of all traffic. In line 20, the new Docker version 2.0 is defined, so the new pod will run the version we just created.

5. Monitor version two for a while

It’s time to monitor the new version for a while. You need to have some visibility about performance.

If there are no errors, you’re ready to deploy (see step 6). But if something goes wrong (Murphy’s Law, people), jump ahead to step 7. Even if there were some errors, you don’t need to worry because only 33 percent of the users will be affected by the errors. (If you want to have a lower percentage, you need to add more pods to the stable version. For example, if the stable version created nine pods and the canary version created one, only 10% of the users would be affected.)

6. Roll out version two to the stable

Congratulations! If you’re here, it’s because you decided it’s time to deploy the new version. Now you can get rid of the canary and deploy to your entire audience.

Let’s update the k8s-deployment.yaml file in line 20 to use the Docker image version 2.0, which should look like this:

image: christianhxc/helloworld:2.0

Run the following command to apply the update:

kubectl apply -f k8s-deployment.yaml

Verify that everything is still running well. You should see the version 2.0 running for all pods.

7. Remove the canary version

If you’re here, it’s because something went wrong. You need to roll back, or maybe you just need to clean up. Remember, we’re only using the canary deployment to reduce the impact on our users.

Run the following command to delete the canary deployment:

kubectl delete -f k8s-deployment-canary.yaml

8. Go back to step 3 to deploy a new version

And that’s it. Every time you have a change that you want to deploy, you need to create a new Docker image version.

Go back to step 3 and make the changes you need, then deploy. Steps 4-7 don’t need to change. A nice thing here is that we’re running everything locally, so you can practice deployments before going to the production environment. You can automate which Docker image version gets deployed. Or you can pass it on as a parameter so that you just replace the new version with a placeholder in the deployment file.

It’s Not as Difficult as You Might Think

Canary deployments can be done in different ways with different platforms. But to me, the Docker and Kubernetes way feels more native. And as you saw, it’s not as difficult as you might think. Following these simple and sequential steps will help make your deployments more predictable. Of course, there’s no way to predict application errors, but that’s why you also need to have information about how the application is performing.

You’re not just going to deploy more rapidly as you automate these steps—you’re also going to deploy with more confidence. You’ll always have a chance to test the new version with a small fraction of the users. And if something goes wrong, you can easily roll back. If nothing goes wrong, you can proceed with the full rollout. Then you just shut down the canary version and repeat.

Amplify Feedback with Continuous Performance

2018-10-16T00:00:00+02:00

As cloud becomes the norm, we’re letting others manage much of our infrastructures for us. Cloud providers offer common metrics like CPU, memory, storage, and networking so that you can stay up to date on the health of your system. And we worry less and less about those metrics with serverless architectures. But there remains a major concern that we can’t just trust to the cloud provider, and that’s our applications. We need our applications accompanied by a good set of internal metrics that will tell us about their health, and no one else can take care of that for us.

It doesn’t matter where our applications are hosted. If we don’t know how well things are running, we’ll continue to add features to the point that some users won’t be able to use them.

And performance issues usually aren’t evident until it’s too late. After the code has been released, problems start popping up. That’s why it’s important that we amplify feedback by assessing performance continuously. Shifting performance to the left will help you to think about performance early. It won’t just be an afterthought; rather, performance will be a feature of the application.

But what does all this mean? Let’s start by figuring out why we need to optimize application performance.

Why Is Performance Important?

When you’re visiting a webpage, if it doesn’t load fast, you just leave. If an app is taking to long to work, you just close it. If applications are making users struggle to satisfy their needs, they’ll go somewhere else. We don’t usually think about performance until we need to.

We might not think that our application will be the next Facebook or Twitter—two applications that receive millions of visits per hour or minute. But it shouldn’t matter how many people use our applications. We should always be looking to have an application that performs well.

If we do this, we’ll realize the benefit of continuously adding features to the application because we won’t constantly have to fix performance issues in production. Some people claim they don’t have to worry about performance because “premature optimization is the root of all evil.” On the other side, there are the people who obsessively try to improve performance, thinking that performance is a feature. Which way of thinking is the correct one? It depends. There are always trade-offs.

Performance is not a new focus. As an industry, we’ve always had to worry about it. Several years ago, we needed to make sure that our code could function with limited resources. Languages weren’t as mature as they are now, so a common issue was to have stack overflow memory issues. Modern languages hide all of that complexity for us, but that comes at a cost.

As someone who’s been on both sides of the fence, I’ve found that premature optimization isn’t worth it—but that doesn’t mean that it isn’t important. It’s cheaper and easier to apply a fix if any problems are visible at the very beginning of the workflow.

Create a Culture of Continuous Improvement

When we talk about culture, it all starts with management. If the people at the top don’t understand the impact of having applications with low performance, they’ll see it as a waste of time. To fully grasp the problems, they’ll need to see the numbers and have feedback. We’ll come back to that later—the importance of performance should be visible and supported by data.

Once management buys in, a performance-valuing culture needs to be spread everywhere. It’s not enough to say it’s important. You need to motivate developers by rewarding them when they do something to improve application performance. And I’m not even necessarily talking about a financial incentive. Sometimes a simple “Well done, thank you!” will give them recognition within the team and will make them feel that performance really matters. As Joe Duffy says in his performance culture post, “You get what you reward.”

In this regard, everyone should be clear about what success means. Do you need to optimize latency of an API to the lowest number possible? No. As long as that number is within the acceptable range, latency can still be considered successful. Performance needs to be constantly evaluated because if it isn’t, progress will be interrupted when the team needs to pay down some of the accumulated technical debt. This leads to the sometimes counterproductive idea that the team should throw everything out and start over with a greenfield project.

Only having a few people who truly care about performance isn’t a good idea. Folks think, “It’s not my problem. I’m not the performance-focused team member.” It can create a culture of blame. Performance should be everyone’s responsibility, and it should be a part of our day-to-day processes.

Start by Shifting Performance to the Left

When I say you should shift development to the left, I simply mean you need to think about performance from the very beginning of the development process. We usually think about delivering first. When it comes to performance, we buy into one of the biggest lies we in IT tell ourselves: that we’ll do it later. In a way, this makes sense. You won’t know if what you’re building will be successful when it’s live. But if it is successful, you’ll have to be quick about fixing performance issues, or else you’ll lose the opportunity to get more customers. It’s tricky, I know.

So, what if we deliver quickly, and we continuously work on doing everything we can to make our applications better?

First, you need to build the machinery to deploy live changes quickly. If you deliver in small batches, you’ll be able to iterate more often and fix performance issues within hours instead of days or weeks because it’s easier to identify where the problem is. There will be fewer interruptions for the team because everyone will be taking care of issues as they arise; whereas normally developers would get interrupted immediately if there are problems in production—I’ve been there, and it’s frustrating.

You should start with your continuous integration (CI) pipeline. Don’t assume that the things you’re changing won’t affect performance. Measure performance all the time. Start by profiling the application on your machine. A good CI practitioner will not push untested code to master. Load testing should become part of your CI pipeline. You also need to include load tests for each environment in your continuous delivery (CD) pipeline. By simply making your different types of tests (unit, integration, smoke, etc.) run in parallel, you’ll be stressing the application. If you don’t automate it, you might only think about performance again when a problem arises.

Get Feedback on Each Deployment

You don’t need to wait until users start noticing performance issues to do something about it. Here’s how feedback should work.

Feedback Should be Accessible to Everyone

After the application has been updated, you need to have feedback. Developers need feedback, but they don’t usually have access to production servers. So it’s operations folks’ job to help them to get visibility. One way of achieving that is by having a centralized logging tool everyone can access. There, people will be able to spot not only performance issues, but problems in general.

Application performance monitoring is key because you need to get more information about what’s really going on. For example, you might share context error information that will help your team spot issues in a database.

Or what about alerts? Without those, it’s hard to be proactive about fixing problems. Of course, I’m not talking about having noisy alerts. Rather, you should set up alerts based on error rates, latency issues, networking issues, page abandon rates—things that will tell you if something’s wrong.

…and it Should be Useful

When you have feedback not only after doing deployments to production but also in other similar environments, you reduce the chances of releasing features with poor performance. Collect everything you can get, like syslog events. There are tools like webpagetest.org that give you a decent context of a site. I’ve been a huge fan of that service, and just recently I learned that you can have those same tools in your own environment.

You can also get performance feedback when monitoring the applications from a different perspective—namely, from a business perspective. Sometimes problems aren’t evident when we look at the infrastructure or error logs, so we need to keep an eye on things like the number of orders made in the application. I once helped a team that had an automated process not just to validate deployments but also to collect application metrics that were important, from a revenue perspective.

Once you have good quality feedback, the solution to a problem becomes quite obvious. This allows you to act sooner rather than later. Let’s say that after analyzing your site, you notice that you need a content delivery network (CDN) to reduce page load time. Or if the database is now a bottleneck that needs more and more time to give data back, you might add a cache layer to reduce the load in the database.

Without having intentional feedback on each deployment, users will complain. And they don’t always complain right after you released something, when it’s convenient for you to fix it. They might complain at a moment when fixing performance problems is not only difficult, but also costly.

Learn and Experiment Because Things Will Still Go Wrong

After you’ve been deploying constantly in small batches, including a certain type of performance testing in your pipeline, and getting feedback constantly so that it’s easier and cheaper to fix things, you might think that you’re covered. But things will still go wrong. It’s better if you’re prepared.

No production environment can beat the experience of real users. You might practice having a similar traffic load in previous environments, but you can’t predict tomorrow’s users. Even if you design and code the application in a way that can support 10 times the amount of users you typically have, things will always fail. Expect failure and embrace it.

Testing in production should be part of the process, not a phase that should be avoided and feared. It’s okay to wait for things to go live to really see how the application performs when receiving real usage. It will be hard and costly to have a copy of production where you can test changes. Quite frankly, you have other things to worry about. For example, you’ll be preparing the application to be resilient to failures. If the database is down, you should depend on the cache, a local copy, or default data. When something goes wrong, you learn. It’s okay to make mistakes; what’s not okay is failing to learn from them.

You’re not always going to be prepared to react when something goes wrong, and it might not be your fault. That’s why it’s important to be able to deploy quickly, and get feedback constantly. Implementing things like a circuit-breaker with Hystrix, doing canary releases, or using feature flags in the application will help you to be more resilient. As a bonus, only a fraction of your users will notice that something’s happening. Rollbacks should be quick, easy, and expected.

Everyone, not just operations folks, should know what a healthy system looks like. And more importantly, everyone should know what to do when it’s not healthy.

Performance is Not Static—It Continuously Changes

In order to have applications with good performance, you need to constantly work on your awareness, culture, and a good set of practices.

Start by Setting Up the Basics

To amplify feedback with continuous performance, it’s necessary to know why performance is important. That will help everyone to understand why you need to load test, even if the belief is that the recent change won’t have any significant impact. Then, you need to make performance everyone’s job, not just something for a few people to handle. Spread the news that performance should not be an afterthought or only something you think about when doing big refactors.

When you have the basics down, you’re ready to include performance feedback from the very beginning of the process: things like developers profiling to locally spot bugs. You can start by reserving a time for doing load tests, but eventually, as the set of test cases grows, it could be a matter of running all of them in parallel.

…and Then Keep Monitoring

Having feedback before releasing to production is important because you can fix things quickly and effortlessly. But feedback from production will tell you the real story, all the time. For that reason, everyone needs to have that visibility. Knowing where things are wrong instead of just knowing that things are wrong will make the difference in how much time you’ll need to fix problems. This is especially true for performance problems—those that the user might only understand as “the site is slow.”

Expecting failure will foster resiliency in the application code. Don’t get discouraged when things are still going wrong. Use those failures to learn and constantly experiment with how the application will react under duress. That will give you the feedback you need to be prepared for unintentional failures.

Performance is not something that you should take care of only when it’s needed, but it also shouldn’t be the only thing you care about. There should be a balance. But what you can’t have enough of is feedback. Work to get it, and get it continuously.

Why You Need an Error Budget and How to Make It Work

2018-09-26T00:00:00+02:00

How many times have you seen Google go down? Not many, I bet. You might not even notice it if it happened. If you did, you’d probably assume it’s an internet connection problem.

But Google isn’t perfect. As Werner Vogels says, “Everything fails, all the time.” If even Google doesn’t have 100% uptime, maybe we should accept we won’t either. Shouldn’t we instead focus on how to recover from failure? It’s been proven that systems fail when they change, but your customers probably wouldn’t like it too much if you simply never updated your software. So since we have to change, how do we know when to change?

Maybe you’ve had good uptime recently, and a small fail won’t hurt you so much. But you have to know whether that’s the case or not. You need metrics that tell you if it’s a good idea to freeze changes for a time or if having some errors is still acceptable.

Why You Need an Error Budget

If you’ve compared cloud services, you may have heard about the nines of availability. It’s basically a number that tells you how much of the time systems are going to be down. When someone talks about having four nines (99.99%) of availability for a system, that person is saying the system will be down only 52 minutes and 35 seconds a year. The more decimals, the more uptime. For instance, let’s say that you defined a rule specifying that the system has to respond in under 500 ms 99.99 times out of 100. If latency goes up, then your system is considered down because it’s above the 500 ms threshold.

This number is used to define the service level agreement (SLA) or service level objective (SLO). The error budget is how much time you’re willing to allow your systems to be down, and it will depend heavily on the SLA that you’ve defined with the product team. Everyone would like to have systems with 100% uptime, but you need to be realistic. How much availability are you willing to provide, based on how much your customers care? Are your users going to notice that your system is up 100% of the time? What about 99.99%? Or even 99%? They might not.

It’s important that you have an SLA and SLO that works for you so that, at the moment a deployment fails, you’ll think twice before trying to fix something in production or go back to a stable environment. Having an error budget helps support the plan of not pushing changes if people lack trust in those new changes.

Uptime vs Innovation: Should I Pick One?

High uptime has risk beyond financial costs and complexity. It’ll also put you in the position of worrying too much when deploying changes. Some might use error budgets to support their theory that every time a change happens, the stability of the system is affected. That means no more changes, in their mind. But I’d advise against having that mindset. It’s better to avoid risking stability in other ways.

Operations will always seek to have systems that are highly available by putting in place replication, redundancy, auto-scaling, backups, and everything that makes systems more robust. On the other hand, developers will try to write code that satisfies the requirements that came from the business. That’s how the DevOps movement started: people wanted to create a culture where these frictions are minimal.

If you care more about having several nines of availability than releasing new features, the result will be that innovation will stop. Sure, it might be better to be conservative than to take the risk, right? But let’s face it. No one will care very much about your reliability if your system doesn’t provide any value. Successful systems are those that solve a problem users have. There are always trade-offs, but keeping systems static won’t allow you to keep your customers happy.

How Do We Keep the Budget Positive?

It’s important to have room in your error budget in case something happens that’s external to deployments—something like internet connection issues, fires in the data center, cloud providers being down, and any problem that’s not in our hands to fix (and that complaining on Twitter won’t solve).

When you push changes gradually, you’re more in control of the error budget. If something starts to affect uptime, you can roll back immediately before it consumes the budget. Also, you might soar over your error budget if you don’t release in small batches. Deployment strategies like blue/green deployments or canary releases are a good option to keep numbers positive. Automation becomes your best friend here. Every second counts, especially when you need to do a rollback.

You can also start with the code. How does your code respond if the database has problems? Or what about Redis? Problems with dependencies will always happen, so it’s better if your application can support that. Let’s say your system is composed of several microservices. If one of those microservices is down, instead of failing, the client should have a default response or take data from the local cache. For example, Netflix has a really good library called Hystrix. If you’re not in the Java world, you can still internalize the principles behind other companies’ level of support for problems.

Fail, But Don’t Get Caught

Failure is an option, but the trick here is how you manage it. Netflix actually practices failure all the time with their chaos monkeys. They go to the extreme of bringing down entire clusters in production several times, all the time. Now, let’s be clear about this. If Netflix is down, users might get mad, but no one will die. They won’t lose money, either, because of their monthly plans system. But their reputation could be affected if the system is constantly down, impacting revenue months later through lost users.

So, should you start bringing servers down? What will be the business’s reaction when you tell them? They’ll probably freak out and respond with a solid no. It’ll depend on the impact downtime has on the users. But even if you don’t put your systems in failure situations in production, like Netflix, you should at least practice it in a testing environment. Doing that shows you care about reliability and are prepared for common failure scenarios.

When AWS went down some years ago, Netflix was one of the few that survived that failure. They failed but didn’t get caught.

Keep Innovating While Staying Up

With DevOps, some organizations are so focused on delivering fast that they sometimes don’t adequately assess risk. And if that’s you, developing an error budget can help you to be aware and respond properly. Most businesses prefer to be conservative, so you need to learn how to sell DevOps or even automation to management while accounting for risk. At the same time, you need to keep innovating without affecting reliability.

Having an error budget will force you to have metrics in place to know if you’re meeting expectations or not, and it will help you take action to reduce the chances of being unreliable.

Error budgets give you more than just a number. They’ll change your thinking when you’re delivering software. You’ll want to shift to the left everything that will make your systems more reliable.

Choosing a Deployment Strategy: A Manager’s Guide

2018-08-27T00:00:00+02:00

Some time ago, I was in charge of deploying to the main system in the company I was working at. Back in those days, I had the bad habit of doing deployments manually by copying/pasting assemblies and then RDPing into the servers to update it with the latest changes. One. Server. At. A. Time. Luckily for me, that system only needed four servers.

Those were bad days because something always went wrong—files were in the incorrect folder, configuration values or file permissions were missing, or I was so tired that I forgot to update one of the servers.

Things are way different now; there are more good days than bad days. For example, what I call a “bad” day now is because of something as simple as a missing test case.

Now that we live in the DevOps era, deployments are also different in the industry. Some organizations do several deployments per day. There are some strategies to increase the frequency of deployments that help to reduce the risks of downtime without customers noticing that something changed.

All or Nothing With Blue/Green Deployments

The blue/green deployment strategy consists of having two production versions of a particular system running at the same time. The catch is that only one is receiving live traffic. This means that the new version doesn’t necessarily have to be backward compatible because the blue and green deployments won’t be live at the same time. It represents a challenge for app dependencies like databases, but you can use the blue/green strategy for them too.

Let’s say for example that your blue side is the running version. The new version with the changes you want to deploy then is the green side. Because the green side isn’t receiving live traffic yet, you have the chance to perform real testing in a production environment before pushing it live.

When the green side is ready to be released, you just need to change the router that’s pointing to the blue side so that it points to the green side instead.

If something goes wrong for some reason, you can just switch the router back and diagnose what went wrong calmly. Rollback speed depends on how quickly you can switch versions, but it’s usually fast. The downside is that you’re duplicating infrastructure, which has an impact on cost—although this might get reduced with containers. But the more important thing is that you’re reducing risk by having a production version that you can test before deploying to real users.

Little by Little With Rolling Update Deployments

The rolling update strategy consists of pushing the new version little by little. You won’t be replacing everything at once, as you do with blue/green deployments, so backward compatibility is a must. This challenges developers because they need to make sure that the new version can keep working with the previous version for some period of time. But this will also give them a sense of how things actually work in production and, as a bonus, it fosters the habit of practicing deployments early in the workflow—which will shift deployments to the left.

Similar to the blue/green strategy, you start by spinning up the new version first when it’s ready to go live. Then, connections from the now-previous version start draining to be rolled out gracefully. The cool thing about this is that you do deployments gently. You also get to define whether you want to spin up and replace one or several instances at a time. The downside is that you aren’t able to run a set of tests on the new version before going live. Or rather, it’s not that you can’t, but it will make the deployment slower and more complex.

Before deployment finishes, you might see some problems with the new version. It will take some time to do rollbacks depending on how far along the update is. (It’s usually slow because you need to spin up new instances of the previous version again and then do the rotation.) An alternative that will help you reduce deployment and rollback time is to use containers—as long as there’s enough capacity do the rollout.

The benefit of this method is that costs stay low because you’re just creating copies of the new version in small portions, not a full copy every time you deploy.

Start Small With Canary Deployments

Canary deployments make the new version live for just a small fraction of the users. Technically, you decide which users will see the new version first, and you can always change the decision of which users to have in your test group. It’s usually as simple as redirecting a small portion of user traffic to the new version and the rest to the current version. When deployment is finished, both versions will be the same. But the good thing is that you get the chance to test changes with a portion of the real traffic before a full rollout.

Deployment could be done either with a blue/green deployment or a rolling update, as long as you only release the new version to a control group of users at a time. You also define how you want to split the traffic. It could be 5/95, meaning that, initially, only five percent of users will see the change. You could also use more than two groups, like 5/15/80. Or you can even start with a group of beta testers who are willing to look for problems and provide feedback.

You can keep the new version for some time (minutes, hours, or even days) to see how it goes when receiving real traffic. Again, it depends on how you want to do it. If nothing goes wrong, you can continue by making the new version available to the rest of the users. Or you can roll back to the previous version. The good thing is that only the few users who tested the new version will notice it.

The benefits and risks, rollback time, and costs (besides the ones that I just mentioned) will depend on whether you choose the blue/green or rolling update strategy.

Compete For Revenue With A/B Testing Deployments

A/B testing is similar to the canary release, but how you split the traffic is a business decision and is usually more complex. This strategy is commonly used for experimentation—when you want to try something that you’re not sure will work. You define groups based on user preferences, location, age, gender, or other factors that will tell you which version had the biggest positive impact on revenue. You can then decide if A or B is ready for all your users.

With this deployment strategy, two versions of the system compete with each other. It could be new vs old or A vs B. Whoever is responsible for making executive decisions in your company could choose which version converted the user to a customer the most, and that’s the version you keep. For that reason, it’s important that you have data to support your theory rather than make a decision simply because you think a certain version will work better.

Implementing this strategy is usually complex, but if you make use of feature flags, it gets simpler. You can be as creative as you want when testing your hypothesis. For example, you might start by dark launching using any of the previous strategies. Then, you’d turn on the feature flag to make only version A available. After some time, you’d turn it off and then turn on version B for the same amount of time as A. From there, you’d collect metrics and decide which version to leave running.

What Outcomes Are You Expecting To Have?

It doesn’t matter much which strategy (or a mix of strategies) you choose for your deployments. It depends on the outcomes you want. The key thing here is that you stop doing deployments manually and you have as little human interaction as possible. You also need to stay consistent, meaning that you’ll have to apply the chosen deployment strategy for development and any other environment in your workflow. It might be challenging to have production-like environments, but you need to practice deployment before updating the environment where your real users interact. Don’t treat your production environment any differently.

Every strategy has its trade-offs. But one benefit common to all of them is that, if you implement the strategy well, you’ll have zero downtime. You won’t have to wait for specific times when your users won’t notice that you’re changing something. Now, it won’t be easy to get to that point; your staff will have to put in some extra effort to achieve it. But they’ll appreciate not having deployments cut into their nights, weekends, or family time.

Last but not least, practice doing deployments frequently. If it hurts, do it more.

Which DevOps Metrics Matter?

2018-08-20T00:00:00+02:00

Some time ago, I decided to start dieting for the millionth time. All previous tries were a complete failure. But this last time was different.

It was different because not only was I going to a nutritionist, but also every time I visited her, she took my measurements. That way, she knew if I was following the recommended diet and exercise, and if I was, she knew what changes she needed to suggest based on the results. I was seeing a small amount of progress every week. It didn’t matter if I lost only one pound; it was something.

When building a product, we should always try to find ways of improving what we’re doing to produce—to deliver faster, cheaper and on time. There are practices and processes that help us, especially in software development. But when you implement something new in your process, you also need to know if it’s working or not. Even if everyone else says it works for them, you might be doing something a different way…maybe even the wrong way. How can you know if what you’re doing is adding value or not? Well, you need to measure.

When you start practicing DevOps, it’s very helpful to have numbers that tell you how good or bad it’s going. But which metrics would you use? Well, let’s explore the what, why, and how of some important metrics in DevOps.

How Many Times Are You Doing Deployments?

Organizations usually do deployments quarterly or monthly—or with the rise of Scrum, every two weeks, if user stories are complete.

You need a deployment frequency metric. That means counting the times you do deployments of the applications under your DevOps implementation. The more deployments you do (or can do), the better. You don’t necessarily need to do frequent deployments if you’re not changing the system very often. In other words, deployment frequency correlates to the times you complete changes in the code.

Is your change ready to be deployed? Go ahead, do it! Or maybe you’re good enough that you have it ready to go on the date that marketing set for a campaign. Awesome! But it doesn’t matter if you deploy when coding is done or not. You can use feature flags to deploy anytime you want with the feature turned off and then turn it on when you’re prepared to release.

It’s important that you know how many times you’re doing deployments because it’s a sign of how fast and easy a deployment is to do. If they’re not easy and you don’t deploy very often, your next task is to find out why and improve. It might be because your developers’ confidence in deployments is broken. Or maybe testing quality isn’t good enough. Don’t let these issues stand; you have to address them.

You don’t have to be Amazon, Facebook, or Flickr to do ten deployments a day. It should be possible when you need to.

Now, how to keep track of this? Well, every time you do a deployment, save a record in a database or any other service you use for logging, like ELK. Then, build dashboards to have a better view of the past. You’ll know if you’re improving or getting worse at a glance.

How Much Time Does It Take From Code Commit to Release?

The DevOps movement started because after developers finish coding, moving to production tends to be problematic. I’ve experienced this struggle in my job. Basically, we would spend two weeks coding and another two weeks trying to release. It didn’t always take that long, but the time to deploy after development delivered certainly wasn’t consistent. Add to that the pressure of management asking why it was taking so long to deliver, and you have one stressed-out team.

My second suggested DevOps metric measures the time between a developer saying “OK, I’m ready to have this change in production” to actually putting that change in production. If you don’t measure that time, you might not improve upon it—or it will be too hard to identify why it’s taking too long. In this case, the less time it takes, the better. So this metric number should be low, and as low as you can possibly make it because you need to consider compiling, publishing and testing.

Knowing that you take too long to deliver will incentivize you to identify where you’re spending most of that time. It could be that you’re wasting too much time fixing merge conflicts. If that’s the case, then the solution is to have a trunk-base development with feature flags. It could be that you’re doing things differently in each environment, so there’s always something new to fix. The solution is to have production-like environments.

The idea is to know you have a problem and that you need to do something to fix it.

So, how can you keep track of this? Well, because the time between delivery and deployment could be due to many things, start by recording that time in a database or app. Save the date and time the change was validated in a dev environment and ready to be released. Then, when that change is deployed, record the date and time again, do the math, and store the total time it took.

How Many Times Do Deployments Fail?

The metric I’ll speak about measures how many times you’re in need of rolling back or turning a feature off because of the increase in errors. The closer to zero this number is, the better. Measuring this will tell you about the quality of your process before the release. It can also tell you how many times you’ve practiced putting things in a production environment.

It’s important to have this metric because it will tell your customers or upper management how good or bad your DevOps efforts are going.

You need to measure this if you want to worry less about deployments. I know some organizations that reserve Christmas Eve to do high-risk deployments. Why? Users are too busy spending time with family, eating, and celebrating to care if the system is down. But you shouldn’t have to wait for a time when your users won’t notice you’ve failed.

How can you keep track of this? For this metric, it’s not about measuring time. And you might be already measuring what you need to, but not for the purpose of DevOps metrics. I’m talking about knowing how many 5XX and 4XX errors you have or how high latency is after a deployment. Even if your system isn’t a web app, it doesn’t matter: you need to keep track of system errors.

An even better metric is to have key performance indicators (KPIs). These will vary from one industry to other. Retailers may measure sales while banks may measure transactions, for instance. So your KPIs could be something like the amount of failed purchases, the number of times the user has to abandon the page—you get the idea. As long as you have a way of knowing the system is failing, you should use that information to know the number of times a deployment has failed.

How Much Time Does It Take To Recover?

The next metric I’ll mention requires you to measure the time it takes you to recover from a failed deployment. It’s also known as “mean time to recover” (MTTR). The less time it takes, the better.

You need to measure this because it will tell you how good you are at recovering the state of the system. If you’re putting stability above all else, your response to seeing a deployment fail will be to immediately roll back. And you don’t have to build a rollback process. It could be as simple as turning a feature flag off. If you read this blog, you’re already aware of how powerful feature flags can be. And they’re especially useful in these types of scenarios.

How can we keep track of how we’re doing on recovery time? It’s as easy as having a record of the time it takes you to roll back to a stable version. If you have one-click deployments, going to a stable state of the system shouldn’t take that much time. How much automation you have here will help you, definitely.

How Are Your Metrics Doing?

Correlate all metrics and you’ll have a better understanding of what’s happening with your DevOps implementation. You don’t need to have good numbers when you start, but make sure you know your initial state—like the before/after photos you see when someone succeeded in losing weight.

In the end, what you should seek to have is

A significantly higher number of deployments per day or per week. The more, the better.
A decrease in the amount of time you take to deploy a change after the code is finished. The less time, the better.
A smaller number of failures after doing a deployment. The fewer, the better.
A decrease in the amount of time you take to recover from a failure. The less time, the better.

You can start by having simple measurements of each of the above metrics—a simple number that includes the concept. Or you can have detailed metrics. You’ll need them when trying to improve the numbers. Either way, don’t make guesses about your progress with DevOps—use metrics to prove it with data.

Introducing Different Testing Strategies Into Your Delivery Pipeline

2018-08-14T00:00:00+02:00

With the rise of DevOps, everyone’s talking about delivering quickly. But while speed is important, so is confidence. Testing will help you increase confidence, and without tests, you can’t trust the deployment pipeline. Just like divers won’t explore the ocean without oxygen tanks they can trust, ops folks shouldn’t dive into the deployment process without having tests of good quality. You need something to protect you from creating a mess when you go live.

That’s why it’s important to include a good testing strategy in your delivery pipeline. There’s no need to introduce another silo by testing your code changes at the end of the workflow. Shift tests to the left because you need to test early, often, and with confidence. You need a way to certify that what you’re shipping satisfies the needs of the users.

But what’s a good way of introducing tests into your delivery pipeline without increasing the delivery time? Let’s explore some ideas.

Creating Tests That Run Quickly Even If the System Grows

How fast should the tests be? Well, how much time are developers willing to wait after pushing code changes? If tests take more than one hour, developers won’t be willing to test early and often, especially if we’re talking about the first set of tests that your application will be running against.

Tests should be integrated into your continuous integration (CI) workflow, but not just after a developer pushes the code changes into the master branch. One of the critical benefits of CI is that developers have rapid feedback before integrating their code with the rest of the changes from the team. That means that these tests should run fast—in a matter of minutes—and they should run locally. They also need to have a constant running time, even if the application keeps growing.

One way of keeping a steady time for tests is to parallelize. But don’t parallelize in code. Parallelization should be something external. Tests should be able to scale by running multiple groups at the same time—say, for example, a Jenkins job that triggers tests simultaneously by calling different JAR files at the same time. Or now that we’re in the era of containers, you can spin up a group of containers that will run several tests at the same time, optimizing resources and reducing run time.

Including a Minimum Set of Use Cases

A strategy that will help you be more confident is to include smoke tests. These give you the minimum amount of use cases you need to approve a change.

As you progress, tests can take longer per phase, and that’s alright. The closer you are to deploying to production, the more confident you need to be that a change isn’t breaking something. In the final production phase, you’d like to do more—add more real data—so it’s not a bad thing to take longer.

If you find a significant bug, make sure you translate that issue into a test that runs fast. That way you can check to make sure you’ve fixed the bug. Also, make sure it’s included at the very beginning of the pipeline. It could be something like converting a manual test to a unit test that will run fast. This doesn’t just apply to unit tests—it can be a more complex integrated test that runs quickly using real data. These complex tests can run while the rest of the unit tests are running.

You also may have heard about the testing pyramid, where you include a certain number of different test types in the pipeline. The idea is to have more unit than integration tests and to use unit tests to test locally. While that works, there are times when mocking (that’s why they run so fast) an object is more difficult than just using real data objects. So you can also include some integration tests at the beginning of the delivery pipeline. They won’t depend on certain data existing—they’ll create it and then remove it once they’re done. By doing this, these integration tests could also be testing other parts of the system, increasing test coverage.

So it’s not about which type of tests should go in each step. Rather, it’s a combination of speed and the importance of tests.

Using Acceptance Criteria, Not Just Code Coverage

I’ve heard that people sometimes add tests that assert to a static “true” value just for the sake of increasing code coverage. But is that the real purpose of the code coverage metric? No. The idea is that you have a test for each portion of the code. It’s really costly to have 100% test coverage, and it’s not worth it. Why? Because every time you need to make a change, most likely you’ll also need to change something in the tests.

Why don’t we focus on how many use cases we’re covering rather than just code coverage? Taking this approach will you avoid the temptation to create silly and useless tests. Practices like ATDD or BDD could be helpful because you’re focusing on the use of the system. These practices will also help you be more confident when doing refactors because you’ll be changing code, not behavior. So the code is being adapted to the tests, not the other way around.

Make sure the tests you’re including in the pipeline are providing value. When they stop doing that, evolve them or get rid of them. A good sign that a test needs to be retired is how much time you invest in fixing or adapting it to new changes.

Constructing Tests In Parallel With Code

Tests should be treated the same way as the application code. You might construct them using a different source code or even a different language. You can also benefit by using versioning and packaging them (i.e., JARs, containers, etc).

What else should you do with the application code? Put it in version control. Don’t use long-lived feature branches. Push frequently to master and use feature flags. The same principles that apply to the code apply to construct tests.

Was a test ready before the application code but now it’s causing failures in CI? No problem. Just make use of the same feature flag you’ll use in the application code and turn it on when the changes are pushed.

It’s not necessary to add another silo, like a separate team that constructs the tests. Testing should be a developer’s job. It doesn’t mean that the same developer that wrote the code writes all tests. It’s about having different roles during development sprints. If Developer A writes the code and some unit tests, then Developer B should write another set of tests that will increase confidence in the code. Next time, those developers will exchange roles. And if for some reason in your organization there’s a testing team, then it’s important that sometimes they pair program together.

Testing shouldn’t be an afterthought when a developer finishes coding. It must be integrated into your delivery pipeline, in the same way you integrate code. You might not run all tests in all stages, but it’s important that they’re built together.

Continuously Adjust Based On Results

Avoid keeping your testing strategy static. It must be evolving continuously.

You’ll always have bugs by missing tests. Get used to it and expect it. Your main job after this happens is to reproduce the bugs by including a new set of tests in the pipeline. This will break something, but that’s the idea. It should be evident that the problem exists before you even start coding the fix.

In the end, it doesn’t matter how many tests you have before going live. There’s nothing like real users for testing, and that’s why you need to continuously adjust, adapt, and evolve your testing strategy. But with the ideas above, you’re amplifying feedback and becoming aware of any bugs early.

How Does DevOps Help You Make Database Changes?

2018-08-08T00:00:00+02:00

Databases have always been treated with a certain level of respect---sometimes to the point that when we want to make a change to the database, we have to create scripts that contain all the commands that need to be executed in the destination environment. Of course, these sets of commands need to be reviewed first by someone else (usually a DBA team).

We need to be this careful because if something goes wrong, data could be lost. The life of a company could go down the drain. Sure, there are always backups, but let's be honest: how many of us have tried to recover from those backups recently? And how many of us do it periodically? Not that many of us can proudly say that we do.

I know I'm not alone in thinking that working with databases can turn complicated, especially as they grow. We've all suffered this pain, or at least have heard of or seen someone else deal with it. But what about if we could apply some DevOps principles to databases? It's worked so well with application code. Why don't we give it a try?

Why Database Changes Are So Hard

Usually, database changes are not an automated task because each change is different. And tools can help you to manage them by creating scripts, but there are some tools that will include sentences that will first delete a table and then create it, as if it's always about a greenfield project. This is why DBAs like to take a final review before proceeding with the deployment to production.

Another reason database changes are hard is that databases tend to be really big, especially now that we're storing all sorts of information just for the sake of it. Everyone knows big databases are problematic. When we do a change in these databases, it could take hours to finish. And what about if we don't get the results we expected? Rollbacks could take twice the time.

And it's not just that. Because of the nature of how changes are promoted, it's probable that the different database environments are not alike. This causes problems for the DBAs when running the scripts in production. Developers get informed of it, but they have no idea what could have happened because it always worked. So they spend some more time troubleshooting.

We all get it. Changes in databases are really hard to promote...but what if they weren't?

Treat Your Database Like Code

There must be a better way, right? I mean, how are companies able to do several deployments a day without considering databases? Everyone has to interact with a database---no exceptions. It doesn't matter if it has good architecture or not. Systems are there to collect information, process it, and then produce a result. That result can't just be sporadic. It needs to be preserved.

The good news is that there is a better way. What we need is to start treating our databases like code. You might have heard this before with infrastructure as code, where we define infrastructure by using a descriptive language. The scripts we've been talking about could go under version control. But it's more than that---it's about changing your mindset. If we want this to succeed, we should avoid promoting database changes manually.

All database changes need to be scripted, pushed into version control, and introduced in your current delivery pipeline. These scripts need to have a special characteristic. They should be idempotent, meaning that it doesn't matter how many times that script is run---it should always produce the same result. It's as simple as adding an "if," just to check if the field the script is about to add already exists.

But you don't have to create your own script. There are tools and frameworks out there that help you to treat your databases like code.

Treat Every Database Change Like a Migration

There's a wonderful project that I love in the .NET stack called Entity Framework Core (EFC). When you use this framework, it doesn't really matter what database vendor you're using. You interact with the database by using objects, because it's an object-relational mapper (ORM). Although this is incredibly powerful, that's not the point of this post.

Still, what I like about EFC is that you can use migrations to promote changes. It's as easy as running a set of commands. For example, say you need to start with the following to create the initial migration script, including the initial table structure like columns, indexes, and keys:

dotnet ef migrations add InitialCreate

It will translate all the objects you've defined in the code to a SQL format. When you need to promote changes to another environment, you just run the following command:

dotnet ef database update

That command will check if the change was already applied to the database. If it wasn't, it applies the change.

When you keep iterating and doing changes, you can continue doing them in the code. Then it's just a matter of creating a new migration by running the first command you ran when you started. To promote, you just run the update command again.

Those commands are repeatable, which makes it easy to introduce them into your delivery pipeline. EFC will take care of making sure that those migrations are idempotent. Does this mean that DBAs will be out of the equation? No, there are sentences that DBAs might want to review before proceeding. The difference now is that they'll be happy because there will be a standard format in scripts. And they'll be involved early in the process, not just before a release.

But...What About Rollbacks?

It's really important to be able to manage rollbacks in databases easily. Let's keep using the EFC example. EFC creates a migration history table when you start using it. So every time you run a migration, it adds a tuple to that table to keep track of what migrations it applied to the database. Having this process is really helpful because you can also roll back the last migration. You can keep rolling back until the database is in the desired state.

Another important aspect for rollbacks is that your application needs to be able to live with new and previous versions of the database. Changes need to be backward and forward compatible. This will help you to decouple database changes from code changes. The idea is that it shouldn't matter which goes first, the code or database changes---you shouldn't have any downtime. There might be occasions when this won't be possible because a change is too complex and existing data is too big. But try to make that the exception and not the rule.

It's Not Just Automation

Tools are important. Honestly, they make our jobs easier. Automation will be easier to implement. But it's about more than tools; it's about a change in mindset for developers, testers, DBAs, managers, and everyone else involved.

It's possible to introduce database changes in DevOps. It won't be as easy as with code, but it's definitely doable. My recommendation will be, as always, to start small. Pick a non-critical or new project where you can apply these principles. It will help your team to get used to this new way of working. As one manager once told me, "I'm not sure if what you're suggesting will work, but I'm willing to be proven wrong."

So prove to your boss or the blockers that they're wrong about how DevOps principles can help you make database changes. Treat your changes as if they were code and use migrations as a way to promote them. It will amplify feedback and visibility for the team. The result will be that you can iterate your product faster, and you can go back and forth as needed.

Test Data Management and Its Role in DevOps

2018-07-24T00:00:00+02:00

In my career, there've been many times when I've experienced the false joy of my code change being ready to be released to production. I say false joy because everything worked as expected on my computer, in dev, in testing and also staging. But in production, my recent code changes were causing intermittent problems.

You know the types of problems. It's always something little, like data being longer than expected for certain fields. No matter how careful I was when testing my change, there was a scenario that I forgot or I didn't know existed. If only I could have good data that helped me to do my job better! My joy wouldn't be diluted by those errors.

Having data for good quality testing is key. And that's where test data management (TDM) comes into play. But what's its role in DevOps? Is it possible to integrate TDM? And how would we go about it?

Let's find out.

What Is Test Data Management?

Missing a use case of our app: it's a common problem in all types of organizations that develop software. No matter how we think our app will be used, users tend to exceed our expectations for creativity when using it. And production is production. It serves as a reminder to us that our test cases aren't invincible. If only we could test our changes in production.

But what if we could have production-like environments? And what if these could be production-like environments not just in terms of infrastructure, but also in terms of data?

TDM is the process of creating production-like data for testing purposes. In some cases, there's no difference. But when there's sensitive data, things change. That sensitive data needs to be masked; then, if it's compromised, the impact will be low.

Tests are important in DevOps. Without tests, it's easy to lose confidence in our applications, and deployments tend to be scary. You need data for good tests. TDM won't prevent you from introducing bugs, but it will help you to reduce the chances by giving you the ability to build data of good quality. That's because if you're able to reproduce an error in production, you'll be able to fix it and make sure it won't happen again. Bugs will continue emerging, but they won't be the same ones over and over. Thus, it's important that anyone in the team can access the data they need when they need it.

Test Data Management Should Be Self-Service For Everyone

Well, we know now that there's a process to prepare data and there are tools for the job. The next obvious thought then will be to automate it and give access to the team. That way, operations and DBAs stop being the bottleneck.

When you're building this process, everyone should be involved. You should agree on what data can and should be used, what data will be masked, and how much data will be needed---much like a data discovery phase, if you will. But after there's an agreement, the team should work on having this process automated so that everyone can create, update, or duplicate data for testing. DBAs will appreciate it; it's one less thing to worry about.

You don't have to reinvent the wheel. There are already some tools for this job.

People shouldn't have to wait too long to get the data they need---and, in fact, they won't wait. They'll find ways to get around things that take time and will shift testing to the right. That's why it's key that you choose the proper tool and plan ahead what data will be needed.

Test Data Management Should Help You Keep Healthy Data

It's also important to keep in mind that we're talking about having production-like data in other environments. Initially, that might not be a problem. But as the data grows, it could be costly. This process will also force you to keep your data healthy, not bloated.

My recommendation is that you start by having all the data you need, but as you grow, start generating the data for each test case. Or you can even have a mix of pre-populated data and static data that you generate in the code for each test case. Look forward to having more static data because it will be cheaper and you'll have more control over it!

It will always depend on the use case, but I've seen databases with tables that have data from years, not months or days. It affects not only costs by using more storage but also performance when writing. When this happens, why don't we consider keeping just the data that's needed and then moving historical data for reporting somewhere else? Or what about working with tables per day, week, or month? It's complex, but as with everything, there are always tradeoffs you need to consider.

The plan is to include TDM in your delivery pipeline. So always keep an eye on the time it takes to prepare data, and make sure to optimize. It's key to reduce lead time to deliver your software, so the less time it takes for TDM, the better.

It Should Be Part of CI/CD Processes

Once TDM is a self-service process that testers and developers can use when they need it (and once it runs fast), it's time to implement it in your continuous integration and delivery process.

In DevOps, every process or task that increases silos are shifted to the left. Shifting to left means that you take to doing it at the very beginning of the workflow. We've seen this happen with deployments, security, testing, and basically everything we need to develop and deliver software.

If we're talking about shifting to the left, it means that we should even start including TDM on a developer's machine. Same for testers and any other team member involved in the process. Some argue that developers and testers should generate static data for testing and that they should invest heavily in unit testing, mocks, and stubs. Some even say you should even use containers. It's not just cheaper to go this route---it's also faster. I get it, and actually, I'm an advocate for that. But I won't lie; doing that is not an easy task.

So while you work on increasing test coverage with unit tests, the easiest way to start is by integrating TDM into your workflow. It's better that your process become reliable first. Then you can optimize and improve it.

Good Testing Data Increases Reliability

Want to increase deployment reliability and at the same time reduce the lead time? Invest in TDM and make it part of your process. Shift it to the left. Don't let testing become an afterthought. A sign that you're not shifting to the left enough is when development is finished and you still need to wait for tests to validate those changes.

Automate your tests. It's OK that you started testing manually, but try to take the time to automate as much as possible---even the process of preparing the data. After that, include it in your DevOps implementation. Make it part of your delivery process. But you also need to pay attention to the time it takes to generate testing data. This will force you in some way to think constantly in your data architecture.

And it's good to be thinking. If you do so, you'll always improve, and you'll decouple things that are not just hard to change but also to test.

7 Container Design Patterns You Need to Know

2018-07-04T00:00:00+02:00

Containers are popular right now because they help move applications forward in a consistent, repeatable, and predictable manner, reducing labor and making app management simpler.

But how do you know if you're using containers properly? That's where container design patterns come into play. Here's what you need to know about container design patterns and why you need them, along with seven common design patterns to consider and how to choose the right one for your needs.

Design patterns exist to help you solve common problems with containers. They also provide a common language when communicating about the architecture of the applications. That way, everyone can understand what's going on.

Design patterns ultimately help make containers reusable. Users of those containers will give each their own purpose. There are times when I don't need to have a complex configuration to test locally, but at the same time, I don't want to modify the architecture so much that I lose consistency when testing. That's why having a baseline is helpful: to reuse containers and make things simpler to test.

1. The Single-Container Design Pattern

Employing the single-container pattern means just putting your application into a container. It's how you usually start your container journey. But it's important to keep in mind that this pattern is all about simplicity, meaning that the container must have only one responsibility. That means it's an anti-pattern to have a web server and a log processor in the same container.

Containers are commonly used for web apps, where you expose an HTTP endpoint. But they can be used for many different things.

In Docker, you have the ability to change the behavior of a container at runtime, thanks to the CMD and ENTRYPOINT instructions. So I'm not limited to using containers for HTTP services. I can also use them for any bash script that accepts some parameters at runtime.

By letting containers change behavior at runtime, you can create a base container that can be reused in different contexts. So you'd use the single-container pattern to expose an HTTP service or to reuse a script for which you don't want to worry about its dependencies. And it would be a good choice, as long as you keep in mind that containers should solve only one problem.

2. The Sidecar Design Pattern

So containers should have only one responsibility. But what about the use case I mentioned before, where you have a web server with a log processor? Actually, that's one of the exact problems that the sidecar pattern aims to resolve.

Using the sidecar pattern means extending the behavior of a container. In our example of the log processor for the web server, the log processor could be a different container reading logs from the web server.

The web server will need to write those logs to a volume. In Docker, volumes can be shared with other containers. It's preferable to have this separation because it makes packaging, deployment, resiliency, and reuse easy—and also because not all containers will need or use the same resources.

With this pattern, you're decoupling your system in different parts. Each part has its own responsibilities, and each solves a different problem. You're eating the elephant in small chunks.

3. The Ambassador Design Pattern

If you're using the ambassador pattern, it means you have a proxy for other parts of the system. It transfers the responsibility to distribute the network load, retries, or monitoring to something else. A container should have one responsibility and be as simple as possible. For a container, the communication to the outside world will simply be an endpoint. It won't know (or care) if what's out there is a set of servers or just one server.

This is the pattern you'd use when you want microservices to interact with one another. They don't know exactly where other microservices are; they just know they can find them by name. And for that, they need a service discovery. This discovery could be at the DNS level, or it could be at an application level, where microservices register. Service discovery will be in charge of keeping only healthy services.

In Docker, this is possible because containers can live on the same virtual network. When you use Docker Compose and you link containers, it basically modifies just the "hosts" file so the call to a service is by name, not by IP address. Also, Docker supports environment variables to inject values such as subdomains for a proxy server that you can change depending on the environment.

4. The Adapter Design Pattern

Using the adapter pattern means keeping communication between containers consistent. Having a standard way of communicating via a set of contracts helps you to always make requests in the same way, and lets you expect the same response format. It also helps you easily replace an existing container without the consumer or client noticing because the contract won't change—just the implementation changes. You can also reuse this container somewhere else without having to worry about managing other application logs.

Analyzing logs from different sources can be a pain if you don't have a standard format. When you have a container that works as an adapter, it will receive raw logs. It will standardize and store data in a centralized place. The next time you need to consume the logs, you'll have a consistent format, and so it will be easier to understand, correlate, and analyze logs.

The main premise here is that the adapter pattern allows a container to reuse a solution for a common problem in the system.

5. The Leader Election Design Pattern

If you're using the leader election pattern, it means you're providing redundancy for consumers of containers that need to have highly available systems. You can see this pattern in tools such as Elasticsearch, an open-source stack. Elasticsearch's architecture consists of more than one node, and each node will have chunks of data (shard) for replication and redundancy purposes.

When the service starts, a node is elected as the leader. If the service goes down, the rest of the nodes elect a leader based on certain criteria, keeping the cluster healthy.

So how is this related to containers?

Well, you can spin up a bunch of containers that communicate with one another without needing service discovery. Elasticsearch containers will elect a new leader, and then you can spin up a new one in just seconds, either manually or automatically, by using an orchestrator such as Kubernetes. Doing the same thing with virtual machines or physical servers could take minutes or even hours.

6. The Work Queue Design Pattern

The work queue pattern dictates that you split up a big task into smaller tasks to reduce running time. You can think of this as the producer-consumer problem. Say a user requests that you transform 1 million records. This will take a lot of time. So to speed up the process, you'd employ the work queue pattern and transform the data into smaller chunks of 100 records each. The code that does the processing work can be packed into a container, and then you can spin up 10 containers at the same time.

Containers are really useful for batch processes. You might need to worry about resources being able to support concurrency, but if you don't, there are tools or services such as AWS Batch that help you manage resources. You just need to provide a container and launch a set of execution jobs.

Containers will help you make the code reusable and portable. But coordination is a problem better solved by container orchestrators.

7. The Scatter/Gather Design Pattern

The scatter/gather pattern is quite similar to the work queue pattern in the sense that it splits a big task into smaller ones. But there's one difference. Containers will immediately give a response back to the user. So instead of launching a bunch of tasks and forgetting for a moment about the actual response, in this pattern, you'd need to combine all small responses into just one. A really good example of this pattern is the MapReduce algorithm.

To implement this pattern, you need two containers. The first will do the partial computation that returns all small chunks needed (map), usually not in an ordered way. This container will then do a request to the second container you need, the one in charge of merging all parts, to return data that makes sense to the user.

With this pattern, you're only focusing on developing each part independently, and you can spin up and use as many containers as needed.

Which Design Pattern to Choose?

Which pattern you should pick out of these seven depends on several factors. There's no silver bullet. Each design pattern has its own purpose and solves a different type of problem. Actually, you might want to apply more than one at the same time in the same system.

These design patterns for containers let you focus on developing the mindset to comprehend distributed systems. They give you the ability to reuse code and have fault-tolerant and high-availability architectures with optimized resources.

I've just scratched the surface of each pattern. Hopefully, you've learned enough to know which ones may be right for your application. I encourage you to further explore the patterns you think will best target the problems you face.

AWS Batch: A Detailed Guide to Kicking Off Your First Job

2018-06-29T00:00:00+02:00

There are always the kind of tasks that need to run periodically, either to analyze and process information (like fraud detection), or to simply do things like send email reports. But for that, we need to have a tool to schedule computer resources and, of course, the script.

But what if we only need to worry about coding the script?

Introducing AWS Batch. It’s a free service that takes care of batch jobs you might need to run periodically or on-demand. And you only pay for the resources you use.

In this tutorial, you’ll learn how to kick off your first AWS Batch job by using a Docker container.

What Is AWS Batch? A Quick Overview

Before we dive in, let’s do a quick overview of the subject at hand. AWS Batch is a service that lets you run batch jobs in AWS. You don’t have to worry about installing a tool to manage your jobs. AWS Batch will do that for you.

There are a lot of features you might not need when you’re first starting out, but let’s explore a few of them anyway:

Instances will run only for the time that’s needed, taking advantage of the per-second billing. You can also lower your costs by using spot instances.
It’s possible to configure how many retries you’d like for any job.
It offers queues where you send the jobs. Each queue could be configured with a certain priority so you can configure which jobs will run first. You can also have queues that use better resources to speed up the process.
It supports Docker containers so that you can focus only on your code.
And more…

So, enough theory. Let’s get our hands dirty.

Kick Off Your First Job

Before we start, there are some prerequisites that will make this tutorial easy to follow, and it will include some good practices in regards to security. If you think you need more details, you can check the setup page in AWS’s official docs.

Prerequisites

Have an AWS account.
Create an IAM user with administrator permissions. To do this, you can just follow this tutorial. I recommend you give granular permissions to the user that will do the provisioning.
Install and configure AWS CLI.

If something from the above doesn’t work, it might be because a permission is missing, or the CLI is not configured properly. I’ll let you know exactly what’s needed in the following steps.

Go to AWS Batch

You’ll see a screen like the following:

Click the “Get started” button. Then, this next screen will appear:

Click the “Skip wizard” button. We’re not going to follow this wizard because I want to explain each step to you. Also, after this, you’ll probably use AWS CloudFormation or something else to provision, not the wizard.

Create a compute environment

The jobs will run on a compute environment. Here, you’ll configure the instance type, family, and some other things that we’ll see in a bit.

It’s important that you know we’re not going to create any instances now. AWS Batch will create one when it’s needed. You can also configure things to create instances right away, speeding up job scheduling, but we won’t tackle that in this post.

Click the “Compute environments” link that appears in the left menu. You’ll see the following screen:

Instance type and permissions

Now click the “Create environment” blue button so you can start defining the compute environment. You’ll start configuring the environment in the following screen:

For simplicity, we’re going to choose all default values. You just need to name the environment. I called it “first-compute-environment.”

You don’t have to worry about creating a service or instance role right now. Just choose the option “Create new role” for both, and AWS will create them for you with the proper permissions. It will help you see which permissions are needed and adjust them if you want to.

Leave the EC2 key pair blank because we don’t need to access the servers for now.

Compute resources

Scroll down a little bit, and let’s talk about the compute resources section. You’ll see the following screen:

This is where you get to choose if you want to use on-demand or spot instances. For simplicity, let’s choose “On-demand.”

The “Allowed instance types” field is where you define which family type you’d like these environments to create. This is where things get fun because you can create compute environments that are CPU-intensive and choose between C family instance types. Or if there are jobs that are memory intensive, you can choose M family instance types. You’re limiting which instance types can be created. I chose “optimal,” so AWS decides for me which instance is better based on the configuration of job queues.

Now, vCPUs are one of the most important things here in order for your first job to run.

If you’re familiar with running workloads using ECS, you might get confused here. You might configure so many vCPUs that AWS won’t be able to create the environment. And even if there are a few instances running, jobs won’t run until the environment is ready. So keep in mind that vCPUs are virtual CPUs, not CPU units that you configure in a container when running in ECS.

I configured a maximum of four vCPUs. It means that if at some point the cluster has four vCPUs among all instances, it won’t create more. Jobs will run slowly, but your costs will remain controlled. I also put one vCPU as desired, just so it starts creating an instance right now. AWS will adjust this later if needed, and you can change it when submitting a job if you’re in a hurry.

Networking

Scroll down a little bit, and you’ll now configure the networking section and tags. You’ll see a screen like this:

Leave the VPC and subnets as default for now. Click the “Create” blue button and wait a bit while the environment is created.

Create a job queue

Now you need a queue where you’ll send the jobs to get executed. This queue will be attached to a compute environment so the AWS Batch service will create the resources needed based on the load of the queue. It will use the min, max, and desired vCPUs configuration to know how many instances to create.

Click the “Job queues” link in the left menu and you’ll see the following screen:

Then, you can click the “Create queue” blue button. You’ll see this:

Let’s put a name to the queue so it’s easy to identify. I called it “first-job-queue.”

In the priority, make sure you type a value that lets you play with lower priority queues later. I put “100” in case I need to create a lower priority queue later—say, for example, one with 50.

Enable the job queue. By default, this checkbox will be checked. You should leave it that way.

You now need to connect this queue to one or more compute environments. I chose the one I just created, the “first-compute-environment” one. If there were any other environment, this is where you’d choose it.

Why would I like to have more than one compute environment? Well, it’s useful if you want to speed up a job’s processing time by creating more instances using the spot market. You can have an on-demand compute environment where you always have resources available. And if the load increases, you can create spot instances if there are any available, based on the bid you configured.

Click the “Create queue” blue button.

Create a job using Docker

We’re going to use a “hello world” job that AWS evangelists have used for demo purposes. I couldn’t find a repository with all the files they’ve used, so I created one with all the files we’re going to need. You can find it on GitHub here.

Let’s explore what’s in there, as well as why and how to use those files to create our first job in AWS Batch.

Docker image

We’re going to create a simple job that will pull a Bash script from S3 and execute it. The Dockerfile and the script that does what I just described is located in the “job” folder of the repository.

I won’t explain either the script or the Dockerfile just yet—we’ll just use it. So let’s build the Docker image and push it to the Docker hub. You need to have Docker installed on your machine, a Docker hub account, and a login for your computer.

Let’s build the Docker image. You can skip this step and use my image located here, or you can run the following command and tag the image using your username instead of mine:

docker build -t christianhxc/aws-batch-101:latest .

Now, let’s push the image. You need to be logged in with your user ID. And make sure you push the image that has your username in the tag. Run the following command:

docker push christianhxc/aws-batch-101:latest

That’s it! You now have the Docker image that will download a Bash script from S3 and run it.

A Bash script

Let’s create the Bash script. You can use the one I have in the repo. That script simply puts a Fibonacci sequence in a DynamoDB table. It uses an environment variable called FOO to create the series of numbers, and it uses an argument just to print it in the console.

This script is in the root of the GitHub repository I linked before, and it’s called mapjob.sh

Now, because this is outside the scope of AWS Batch, I’m just going to list the actions you’ll need for this guide to work. We’ll need to do the following:

Create a DynamoDB table in the Virginia region. Mine is called “fetch_and_run.” If you decide to enter a different name, make sure you change it at the end in the mapjob.sh script.
Create an S3 bucket in the Virginia region. Mine is called “cm-aws-batch-101.” Don’t make it public.
Upload the mapjob.sh script in the bucket you just created.
Create an IAM role for an ECS service task with permissions to the S3 bucket and the DynamoDB table. If you don’t know how to do that, follow these instructions. I called my IAM role “aws-batch-101.” We’ll use this one next.

You’re almost ready to kick off your first job. You already have a script and a Docker image to use.

Let’s create the job definition in AWS and then submit a job.

Create a job definition

At this point, you’ve defined the environment where your jobs will run and the queue, which means AWS takes care of creating resources only when they’re needed. Now you need to run the job definition. And this is where things get more interesting.

Click the “Job definitions” link in the left menu and you’ll see the following screen:

Click the “Create” blue button and let’s start defining the job.

Enter any name you’d like. I put “first-job.” Next, choose the IAM role you created previously; mine is “aws-batch-101.” Now pick a name for the container image. Like I said before, for simplicity, you can use mine. I called it “christianhxc/aws-batch-101:latest.” These values can’t be changed when submitting a job, but the ones we’re about to explore can be changed.

Scroll down a bit and let me explain what’s there:

The command field describes the command passed to the container. It maps to the COMMAND parameter to docker run. Here, we’ll type the name of the script that will run the container and its parameters. Because we can override this value, we’ll leave it as it is right now.

Now, here’s another trick to be able to run a job. Unfortunately, you can’t configure CPU units to a container, only vCPUs. It means that, at minimum, the container will have 1024 CPU units because that’s the equivalent to one vCPU. You can configure the CPU, then, in blocks of 1024. This is important because I entered 256, thinking that this was CPU units, and the job never started. It sticks in the RUNNABLE state if there’s nowhere to run it.

Configure how much memory this container will need. I put 256. Leave the rest as it is.

Submit a job

You’re now, finally, able to submit a job.

Click the “Jobs” link in the left menu, and you’ll see the following screen:

Click the “Submit job” blue button. Let’s submit one!

Next, name your job submission. I called it “first-attempt.” Choose the job definition and the queue we just created, and choose “Single” as a job type.

Scroll down a little and let’s override some values here:

In here, you’ll need to put the name of the script in the S3 bucket and the Fibonacci number as parameter. But these are only for reference. I used “mapjob.sh 60.” Type in “1” for vCPU and “256” for memory. As for job attempts, 1 is fine.

Scroll down some because our scripts need environment variables in order to work. Let’s add the corresponding values:

Let’s add the environment variables. For FOO, enter the Fibonacci number. I used 60. For BATCH_FILE_TYPE, put “script”, and for BATCH_FILE_S3_URL, put the S3 URL of the script that will fetch and run.

Click the “Submit job” blue button and wait a while. You can go to the computer environment and changed the desired vCPUs to 1 to speed up the process.

It will start creating one instance. When the instance is ready to process a job, the job will transition from RUNNABLE to SUCCEEDED.

And you’ll see a new entry in the DynamoDB table.

You can keep submitting jobs and change the FOO var to generate a different sequence of numbers. When you don’t submit any other job, AWS Batch will terminate the instance it created.

It’s Your Turn Now

You now have the basics to kick off a job in AWS Batch. Once you’ve finished this guide, it’s up to you which scripts or code you’ll put in a container. AWS Batch will manage all the infrastructure, scheduling, and retries for you.

Now the challenge is in how to code your application so that you can submit several instances of a job. AWS Batch will run them as you submit them and will scale out/in when it’s needed, saving you some money. You can start by migrating any existing cronjob, but don’t stop there. Let’s be creative!

Growing a High-Performance DevOps Culture

2018-06-08T00:00:00+02:00

Culture is one of those things where we all know what it is but can’t explain it. Well, according to Wikipedia, culture is “the social behavior and norms found in human societies.” But in simple words, it’s all about people: how they interact, how they behave, how they talk, and what they practice. And culture is the foundation of a successful implementation of DevOps.

John Willis, an established speaker and writer on the subject of DevOps, coined the term CAMS (culture, automation, measurement, sharing) at a talk where he explained that DevOps culture is about breaking down silos. But what I find most striking about his discussion of culture, as summarized in the DevOps Dictionary, is the observation that “fostering a safe environment for innovation and productivity is a key challenge for leadership and directly opposes our tribal managerial instincts.” So the starting point for your DevOps journey is good leadership. After that, it’s just about how to grow your team to become a high-performing one.

A high-performing team in DevOps, according to recent research, is one that

Does deployments often, meaning several times a day.
Delivers a change with a fast lead time (minutes) after it’s been pushed to a shared repository.
Has a short (again, minutes) mean time to recover (MTTR).
Has a small change failure rate (described here).

So how do you grow a high-performance DevOps culture? You create a culture that will produce a team that delivers on time with confidence in a predictable manner. Here are the things that will help you get there.

Increase Deployment Frequency

Increasing deployment frequency could be seen as a contradiction to delivering in a reliable manner. But working in small batches isn’t anything new. Scrum and Extreme Programming are two management systems for software development that emphasize delivering in short cycles. While deployment frequency varies from one user to the next, people who practice Scrum typically deploy biweekly. And sure, that may seem fast if you’re new to agile. But there are some companies that would find it laughable to call two weeks a “short cycle.”

High performers are known for doing several deployments per day. The average high-performance team does four daily deployments, and some do way more than that. Etsy does eighty, and Amazon and Netflix deploy thousands of times every day, according to this study. That seems crazy, but doing so increases delivery throughput and lets organizations get feedback all the time. Customers or stakeholders don’t have to wait for the sprint to finish. As soon as development has finished coding and verified that the change adds value (and doesn’t break anything), there’s no reason to keep waiting to deploy.

Working in small batches is more than a practice. It’s a mindset. It requires that all members of the organization—developers, operators, businesspeople, and most importantly, leaders—think about it all the time. The result will be that you’ll increase customer collaboration because they’ll have something to try every day.

When you incubate an idea for too long, it becomes obsolete and useless. Deliver, and do it often.

Find Countermeasures to Increase Speed

How will you know if something’s working or not if you don’t measure it?

Measuring plays a big role in building a high-performance team. It’s important that you identify what’s stopping you from delivering on time. High performers have all sort of metrics in place to help them do that.

One very important metric is lead time. Lead time is how long it takes to implement a change in a production environment after the code has been committed in version control. Basically, it’s how much time it takes to deploy and test code changes. This is important because it won’t matter how fast the developer codes; a change could take weeks to be delivered.

Do you know what’s causing big lead times in your workflow? You might not.

For example, you might be working with long-lived feature branches. Are you measuring the time it takes to merge the feature into the master branch? If you’re spending too much time on this task that doesn’t have much value, it’s time to find a countermeasure.

According to lean training company Velaction, countermeasures are “the actions taken to reduce or eliminate the root causes of problems that are preventing you from reaching your goals.” So a good countermeasure in the above example is to use trunk-base development. You’ll always commit to the master branch, reducing the issue of having to spend too much time fixing merge conflicts.

After you’ve applied a countermeasure, you might need to apply another one somewhere else. But it will be hard to identify which if you’re not measuring everything you’re doing.

Having a culture of continuous learning in your organization will help you become better across the board. You won’t stop improving once you’ve made the transition from doing deployments once a month to two weeks. You’ll continue working to identify what’s preventing you from doing daily deployments. And deployments are just one example. What about code coverage? Manual testing? Immutable infrastructure? The list goes on. But the idea is that you’ll always have a chance to improve what remains.

Stop Pointing Fingers

There’s nothing more harmful in a culture than pointing fingers when something goes wrong. No one that provoked a downtime said, “I’m bored. Let’s bring the system down, just for fun.” (Well, unless they’re hackers.) And if colleagues make someone feel guilty after causing a downtime, will that person admit fault the next time? Not likely.

Failure is inevitable, so organizations should embrace it. It’s better to find mechanisms to reduce the time it takes to recover from failure than to try to avoid mistakes. One way to avoid failure is to do deployments less frequently, but that stops innovation.

What you need in your organization is a blameless culture.

This is a culture that accepts that we all make mistakes sometimes. Being able to say, “I screwed up. I’m sorry. Can someone help me fix it?” will help your organization focus on a solution rather than the problem. And if there’s no stigma associated with guilt, the team can be honest about who else contributed to the problem.

It’s important to build trust in your team. Making mistakes shouldn’t be a punishable offense. Instead, you should anticipate failure and then fail on purpose. Netflix does it all the time—they bring servers down just to test their fault-tolerant architecture.

Now, telling management that you want to intentionally cause problems doesn’t sound good. But what if you did it in an environment other than production? What if it only affected certain users? These are ways that you can enable experimentation and build trust in the team.

Let your people learn from their mistakes. Find ways to prevent those things from happening again. When something bad happens, it’s not about one person. Stop pointing fingers so everyone accepts responsibility for their mistakes. A healthy organization embraces failure and learns from it.

Enable Production-Like Environments

High performers have a low failure rate. Deployments don’t usually fail because high-performing teams deploy in small batches, they have practices in place that reduce lead time, and most importantly, they have a way to recover quickly in case of any failure.

How is that possible?

Well, one option is to have a production-like environment. It lets everyone experiment with what it’s like to work on a production server. And it lets the team practice deployments as if they were doing it in a production environment.

For example, if you’re using Jenkins as your CI tool, you can make sure every developer has their own copy of Jenkins installed locally. Why? Having it running locally on their machines will let developers test their code before integrating it into the source control. The flow will be like this: do some changes, pull the latest changes from the repository, fix any conflicts, simulate the integration locally, and if it works, push the changes. By doing this, the practice of continuous integration will have the team always ready to deliver.

Work on building a culture that’s always ready by practicing before going live. This includes building, packing, testing, deploying, rolling back…you name it. You’ll avoid any surprises when going live, and that will reduce your failure rate, increasing the trust in the team.

Do It Again, but Better Every Time

You’ll start seeing the benefits of everything I described here once you’ve made it a habit. After all, practice makes perfect. People might not see the value of these initiatives right away, but don’t get frustrated. Just keep pushing until everyone gets it.

Fostering a safe environment for innovation and productivity should be the main goal in every DevOps implementation. Building trust in the team brings enormous benefits because members will have a feeling of ownership when doing things.

When you reward certain behaviors, you see that behavior more often. So make sure everything you reward contributes to building a high-performance culture.

But I’m a Dev, Not a DevOps!

2018-05-25T00:00:00+02:00

My experience with DevOps began before I even knew there was a name for the approach, when my boss asked me for some help in operations. The company I worked for was small at that time, so I always had the opportunity to get my hands dirty in the release automation process. I knew a few things about servers and Linux, so I was up for the challenge. To my surprise, I loved it. I knew it wasn’t the classic way of doing operations by manually managing physical servers, firewalls, virtual machines, and the like. We were using a cloud vendor. This meant that to spin up a new server, it wasn’t necessary to know which buttons to click.

The cloud vendor had his own API and SDKs for several languages, so I never really felt like I stopped programming. Of course, that was just the tip of the iceberg because systems administration is not just about spinning up new servers, adding more storage or rebooting servers. I had to take care of the architecture and which cloud services were needed for the job. But I was sure I could apply some development skills to operations, and I did. I created some scripts that launched a new environment from scratch, made backups, and restored databases.

Then, I found out about DevOps and all its practices. And because my background was in development, I was able to work with developers and explaining in their language how they could be destroying our log files and why it was important.

So if you’re a developer new to this DevOps world, trust me. You’ll like this new way of working.

You Won’t Be Writing Less Code

Because things often go wrong when deploying their latest changes in code, developers get interrupted constantly, keeping them away from the keyboard and writing less code. At first, I thought maybe this just happened to me. But when I found out this was also an issue for Ben Treynor, who created the first Site Reliability Engineering (SRE) team at Google, I didn’t feel alone. In an interview, he described SRE as “what happens when you ask a software engineer to design an operations function.”

Can you imagine that? A developer tasked with operating a production system? At first, that sounds like a scary thing. (I was scared the first time I did it.) But it turns out that this experiment went amazingly well for Ben Treynor. SREs are now lightening the workload by automating what they can and worrying less and less about the mundane, boring operations tasks.

If you’re worried about writing less code, you don’t have to be. I think we all know coding is an important skill to have. You’ll actually be interrupted less frequently if you work with the operations team, building all the machinery to do boring deployments. And if you decide to make a DevOps career path switch, you’ll be writing more code as you automate the world’s repetitive tasks.

You Won’t Have to Wait Weeks to Push New Changes

As developers, one of the things we like the most is seeing our lines of code come to life. But realistically speaking, most of us don’t get to see this every day; deployments are done every two weeks or even more. And those deployments can be really painful. Some are scheduled late at night, on weekends, or when few users are connected. There’ve been times I had to roll back because something went wrong and the maintenance window wasn’t wide enough. That’s frustrating, but it’s the nature of deployments. It’s our reality.

So…did you buy that? That this has to be our reality?

I don’t. There’s a better way to do deployments.

According to the 2017 State of DevOps Report, some organizations have been able to do four deployments per day. Now, I’m not going to lie to you. It’s not easy to get there. But it’s possible. And no, it’s not just about automation. You’ll need a critical set of practices in place to be successful. I’ve had the joy of experiencing fast deployments. In the past, we deployed every two weeks at my company. We created new infrastructure and had to test everything over and over again with a mix of automated and manual testing. One day, we made some changes in our architecture and adopted containers. We now do several deployments per week with a push of a button, and each of them takes five minutes.

For this to work, you’ll need to make adjustments at every stage of your process. The most critical change will be to increase code coverage with automated testing. It’s not just about speed. It’s also about consistency and reliability.

There will be some growing pains when you try to reduce your deployment time, but it’s really worth it.

You Won’t Have to Keep Waiting for Feedback

When you’re able to get your changes to production fast, you can’t just restart the server to solve problems. The old ways won’t work. It’s mandatory to know, in detail, what’s happening every time you do a deployment. What metrics changed? When did problems start? Everyone should have the ability to see the system status all the time. Long gone are the days when only operations had this visibility. It’s not enough to have infrastructure-level metrics; you need metrics that matter to the business, too. What those metrics should be will depend on the purpose of your software. For a site with a shopping cart, for example, you’ll be interested to know the number of orders taking place.

One of the hardest things to do is to deploy in the same way across all environments, including your local computer. It’s possible most of the time, but it gets complicated when you use PaaS in production because the idea is to have production-like environments everywhere. This doesn’t apply to data, but if you have a load balancer in production, you need it in every environment, too. You can run local VMs or mimic some dependencies. For example, you can use a local NGINX server to represent a load balancer in your cloud provider.

If you have the chance to work with containers and Kubernetes, you can use Minikube to run Kubernetes on your computer and do deployments locally. Excuse my excitement here, but I think that’s awesome! Now, it’s of tremendous importance that you can get feedback locally because that’s the fastest feedback possible. And it’s essential you have a production-like environment running on your computer. But with this, developers can truly increase credibility when they say, “It works on my machine.”

You Won’t Lose the Opportunity to Experiment

A natural effect of deploying faster and getting feedback afterward is that you’ll now be able to experiment. You’ll have the ability to say, “Hmm. What if I do this change? Is revenue going to increase? Let’s run an experiment.” Your changes will be accompanied by a set of reliable, automatic tests, and then you can wait for feedback and make a decision. I know, it sounds simple—and it is! And now that you’ve implemented these other changes, you can easily run A/B tests with your users as well.

You Will Deliver Faster

I’ve made a career change by moving from development to operations. And because of how operations has evolved now, I don’t regret it. I still love coding, and I do as much of it as I did before—the cool thing is that I’m applying development principles to operations. You’re going to delight in this new way of working, even if you stick with development. When you see how DevOps changes the way of working at an organizational level—not just at a team level—you’ll be smitten. And the way you think about your code will change too. You’ll no longer just stop thinking about your code once you finish because now you know you might be only halfway to getting your code live.

These are exciting times for a developer. You’ll be writing more code and creating interesting new things every time. More importantly, you’ll be interrupted less frequently, freeing up time for learning and allowing you to have less stress. You’ll see your changes go live rapidly, and you’ll get feedback quickly as well. That means you can remain focused on a task and do less context switching, which kills productivity. And when deployment is so frequent it becomes natural, the doors to experimentation will be wide open to you. It’s not just that bug fixes get done rapidly. The pace of innovation will also be positively impacted. Everyone will be happy.

Taking the DevOps route won’t be easy, but it’s worth it: for everyone, not just for you.

Deploy to Production: 5 Tips to Make It Smoother

2018-05-23T00:00:00+02:00

Who hasn’t had issues when deploying a software change to production? No one.

I once accidentally deleted a bin folder from a .NET application in one of four servers when deploying a change. The site was working intermittently as a consequence, so I had to check all the servers again to find the one that was having problems. Luckily for me, there were only four servers.

Everything that can go wrong in a deployment change to production will go wrong. It’s Murphy’s law. For that reason, we have a sort of respect that borders on fear when we think about doing deployments. There’s too much at risk, so sometimes we avoid the struggle. But in reality, deployments should be the most boring task of the workflow. The trick is knowing how to do it.

With that in mind, let’s talk about some ways to smoothly deploy to production without risking quality.

1. Automate As Much As Possible

Let computers do the repeatable work for you—they’re better at that than we are. I prefer to invest some hours working on a script to do deployments, like a recipe of commands. Then I test the script locally and in the development environment. That way, next time I want to deploy a change, I won’t have to worry about missing any steps like creating a folder or giving permissions to some files. I’ll just need to run the script again.

But I’m not talking about just a bash script that can be hard to read and understand. In the last few years, new things have emerged. You have at your disposal tools like Terraform, Ansible, and Jenkins, which allow you to use code principles. You can have all your scripts in version control, and that will be the only source of truth. That means everyone will know what changed, as well as when and why it changed, not just in code but also in infrastructure and deployment pipelines.

All of this means that deployments can be as easy as clicking a button. Even reverting the deployment is a button-click away. You can get to the point where anyone, even non-technical people, can do deployments.

Automation makes deployments smoother by letting you move forward and backward with ease. The process becomes repeatable and reliable.

2. Build and Pack Your Application Only Once

The build process usually takes a while, so it will make things run so much more smoothly at deployment time if you only build once. You can keep coding and push changes all the time without worrying about promoting code that’s not ready.

After you build your app, the next step is to pack it and to make sure the package remains untouchable. That way, you’re constantly taking snapshots of your app. There will be times when the app is packed with errors or incomplete code. But that’s no problem because you get to choose which package will be deployed.

You’ll move the same package across all your environments. This means that when you generate it for development, you’ll use the exact same version of the app to update the production environment. You can easily achieve this by doing adequate configuration management. Just replace placeholders with the respective values of the environment—like database endpoints, for example.

You’ll want to practice continuous integration to make packing easier, and one of its fundamental principles is that you build only once. Practices like this allow you to integrate constantly in a shared repository, verify the process with automated tests, and fix things quickly if needed.

Do all of these things, and you’ll smooth the deployment to production because what you tested in development is the same thing you tested in other environments, including production.

3. Deploy the Same Way All the Time

There’s no value in applying the previous tips if you don’t deliver the same way in all circumstances. You packed once to deploy the same code everywhere, and you need to deliver the same way. By doing this, you’ll avoid any surprises in production.

For this, you need to have production-like environments. This doesn’t necessarily mean that you’ll have the same data everywhere—that can be expensive and a possible security risk. It just means that if you have a load balancer in production because you need to be able to scale out/in, you also have to have one in development. If you’re hosting production in the cloud, you need to do the same for development.

Of course, you can think about how to have cheaper environments. You just need to change configurations when you update an environment.

This deployment tip will help you to be bold when changing production. You’ll be fearless because you’ve already practiced it several times before actually doing it.

4. Deploy Using Feature Flags In Your Application

If you only take away one tip from this list, make it this one. Using feature flags will increase your confidence to deploy. And the benefit is even bigger when you combine them with strategies like blue/green and canary deployments. These strategies will help you to deploy without customers noticing there was a change.

It doesn’t matter which strategy you choose—all changes should be backward compatible. I recommend you choose a strategy that works for you. Blue/green deployments tend to be costly, depending on your infrastructure and how small your app is. But they’re the safest way to deploy because you’re duplicating the environment and testing it before pushing it live. In the case of canary deployments, you’re pushing changes little by little. You can do it server by server or by recreating containers one at a time.

But you can do it better using feature flags.

In fact, you can do it so much better that you can deploy all changes with the feature flag turned off. Then you can turn on the feature flag for a small fraction of users, monitor their behavior for a while, and keep enabling the feature for more users until you’re sure that the change is good to keep. And you can do this all without worrying about adding more complexity to your code or deployment strategy.

One good tool that makes feature flags easier to implement is Rollout. You set up your audience and roll out your changes little by little.

5. Deploy in Small Batches, and Do It Often

When you put all the previous tips in practice, deploying in small batches and doing it frequently becomes natural. It will take some time to get there, especially because deployments should be predictable and reliable. But it’ll be worth it.

Now, automated testing is a prerequisite. It doesn’t matter how much effort you put into automating your deployments—if at the end of the day you spend a lot of time manually testing your application, you won’t see the benefit. So try to spend time automating your test cases. Otherwise, there’s no point in doing automated deployments.

But back to the subject at hand. How small should your batches be, and how frequently should you deploy them?

Well, if you’ve ever played the hot potato game, you’ll have a good idea of what “small” and “frequent” mean. As soon as you have something ready to be deployed, do it. It doesn’t have to be complete, so long as you use feature flags. Treat deployments like coding. It’s pretty weird when someone spends many hours coding without testing. Without checking your code as you write it, you won’t know exactly what’s causing an error. And if there aren’t many lines of code, there are fewer things you need to check.

Amplify feedback by increasing the number of times you deploy. It might sound risky, but it’s not. The fewer changes you make, the easier it will be to know what’s wrong. Don’t wait until you have a lot of changes to deploy. Do it frequently.

Make Deployments a Mundane Task

When you practice one or all of the above tips, your deployments will become boring. And in this case, boring is a big win. You’ll find more exciting things to do rather than face the same problems over and over again.

The tools, services, and practices that exist today—especially those I’ve mentioned here—are wonderful. They help you avoid surprises, transforming all those scary nights or long weekends into peaceful, routine workdays. You can do things like automate many of your tasks, especially testing, to make going forward easier and more efficient. Also, you can pack your application; it will help you make deployments consistent. Note, too, that production environments don’t deserve to be treated differently. And make sure you use feature flags. They’ll help you be less stressed when deploying because you know you can always turn off features and everything will continue working. Lastly, work in small batches. It’s easier to spot bugs, and it’s less risky because you’re not changing too many things at the same time.

The Advantages of Using Kubernetes and Docker Together

2018-05-07T00:00:00+02:00

You might be hearing a lot about Kubernetes and Docker—so much that you might be wondering which one is better.

Well, there is no “better” because these aren’t equivalent things. Docker is like an airplane and Kubernetes is like an airport. You wouldn’t ask “Which should I use to travel—airport versus airplane?” So it goes with Docker and Kubernetes. You need both.

In this post, we’ll run through a deployment scenario, how containers and orchestrators can help, and how a developer would use them on a daily basis. You’ll walk away from this post with an understanding of how all the pieces of the puzzle fit together.

Everything Starts With Your Local Environment

So let me start with a typical day in the life of someone who struggles through every deployment. Then I’ll explain how these two technologies can help. For practical purposes, we’ll talk about the fictional developer John Smith. John’s a developer working for a startup, and he’s responsible for deploying his code to a live environment.

John has two apps: one in .NET Core and another in Node.js. He struggles every time a new version of the language, framework, or library comes out and he has to run an upgrade. The problem is when things aren’t compatible with what he’s installed. When something’s not working, he just installs, uninstalls, updates, or removes until finally things get back up and running. The struggle becomes even bigger when he has to push a new change after doing all of that to another environment. It’s kind of hard to remember all the steps when we’re in a rush.

One solution could be for him to work with virtual machines (VMs). That way, he can isolate all dependencies and avoid affecting any existing apps and their dependencies

While that could work, it doesn’t scale. Why? Because every time something changes, he has to take a new snapshot. And then he has to somehow organize all the different versions of those VM snapshots. He’ll still need to deploy changes in code and any dependencies to other environments. Now, he can screw things up in other environments too and then fix it, and that’s okay. But when we’re talking about production, things get risky. He has to work with production-like environments to ease deployments and reduce risk. That’s hard to do.

Even having automation in place, deployments might be too complex or painful. Maybe John even has to spend a whole weekend doing deployments and fixing all sorts of broken things.

We all wish deployments could be as boring as pushing a button. The good news is that that’s where Docker and Kubernetes come into play.

Use Docker to Pack and Ship Your App

So, what is Docker anyway?

Docker is a company that provides a container platform. Containers are a way to pack and isolate a piece of software with everything that it needs to run. I mean “isolate” in the sense that containers can assign separate resources from the host where it’s running. You might be thinking this sounds pretty similar to VMs, but the difference is that containers are more lightweight: they don’t need another OS to make software run. Containers let you be more agile and build secure and portable apps, which lets you save some costs in infrastructure when done well.

I know that sounds like a textbook definition, so let’s see how this is beneficial by following the day in the life of John.

Let’s say John decides to start his containers journey. He learns that Docker containers work with base images as their foundation to run an app. A base image and all its dependencies are described in a file called “Dockerfile.” A Dockerfile is where you define something like a recipe that you usually have in docs (or in your mind) for anyone who wants to run your app. He starts with the .NET Core app, and the Dockerfile looks like this. Take a look:

FROM microsoft/aspnetcore-build:2.0 AS build-env
WORKDIR /app

# Copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore

# Copy everything else and build
COPY . ./
RUN dotnet publish -c Release -o out

# Build runtime image
FROM microsoft/aspnetcore:2.0
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "hello.dll"]

As you can see, it’s as if you were programming. The only difference is that you’re just defining all dependencies and declaring how to build and run the app.

John needs to put that file in the root of the source code and run the following command:

docker build -t dotnetapp .

This command will create an image with the compiled code and all of its dependencies to run. He’ll only do the “build’ once because the idea is to make the app portable to run anywhere. So when he wants to run the app, only Docker needs to be installed. He just needs to run the following command:

docker run -d -p 80:80 dotnetapp

This command will start running the app on port 80 of the host. It doesn’t matter where he runs this command. As long as port 80 isn’t in use, the app will work.

John is now ready to ship the app anywhere because he’s packed it in a Docker container.

So why is this better? Well, John doesn’t have to worry about forgetting what he installed on his local computer or on any other server. When the team grows, a new developer will rapidly start coding. When John’s company hires an operations guy, the new hire will know what exactly what’s included in the container. And if they want to do an upgrade of the framework or some dependency, they’ll do it without worrying about affecting what’s currently working.

Use Docker to pack and ship your app without worrying too much about whether the app will work somewhere else after you’ve tested it locally. If it works on your machine, it will work on others’ machines.

Use Kubernetes to Deploy and Scale Your App

So, John now just needs to go to each of the servers where he wants to ship the app and start a container. Let’s say that, in production, he has ten servers to support the traffic load. He has to run the previous command on all the servers. And if for some reason the container dies, he has to go to that server and run the command to start it again.

Wait. This doesn’t sound like an improvement, right? It’s not much different than spinning up VMs. When something goes down, he’ll still need to manually go and start containers again. He could automate that task too, but he’ll need to take into consideration things like health checks and available resources. So here’s where Kubernetes comes into play.

Kubernetes, as their site says, “is an open-source system for automating deployment, scaling, and management of containerized applications.” There are more of its type, but Kubernetes is the most popular one right now. Kubernetes does the container orchestration so you don’t have to script those tasks. It’s the next step after containerizing your application, and its how you’ll run your containers at scale in production.

Kubernetes will help you to deploy the same way everywhere. Why? Because you just need to say, in a declarative language, how you’d like to run containers. You’ll have a load balancer, a minimum amount of containers running, and the ability to scale up or down only when it’s needed—things that you’d otherwise need to create and configure separately. You’ll have everything you need to run at scale, and you’ll have it all in the same place. But it’s not just that. You can also have the ability now to have your own Kubernetes cluster running locally, thanks to Minikube. Or you can use Docker, because Docker now officially supports Kubernetes.

So, coming back to John. He can define how he wants to deploy an app called “dotnetapp” at scale.

Take a look at the “dotnetapp-deployment.yaml” file, where John defines how to do deployments in a Kubernetes cluster, including all its dependencies at a container level. In this case, besides launching the dotnetapp, it’s also launching the database using a container. Here’s how the file looks:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: dotnetapp
spec:
  replicas: 3
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 5
template:
  metadata:
    labels:
      app: dotnetapp
  spec:
    containers:
      - name: dotnetapp
        image: johndoe/dotnetapp:1.0
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 250m
          limits:
            cpu: 500m
        env:
        - name: DB_ENDPOINT
          value: "dotnetappdb"
---
apiVersion: v1
kind: Service
metadata:
  name: dotnetapp
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: dotnetapp

John now just needs to run this command to deploy the app in any Kubernetes cluster, locally or in another cluster:

kubectl apply -f .\dotnetapp-deployment.yaml

This command will create everything that’s needed, or it will just apply an update, if there is one.

He can run the exact same command on this computer or any other environment, including production, and it will work the same way everywhere. But it’s not just that. Kubernetes constantly checks the state of your deployment according to the yaml definition you use. So if a Docker container goes down, Kubernetes will spin up a new one automatically. John no longer has to go to each server where the container failed to start it up again; the orchestrator will take care of that for him. And there will be something monitoring the stake to make sure it’s compliant—meaning it’s running as expected—all the time.

That’s how you could easily get to doing several deployments a day that take around five minutes.

You’ll Deliver Quickly, Consistently, and Predictably

Now you know what Docker and Kubernetes are—and not just in concept. You also have a practical perspective. Both technologies use a declarative language to define how they will run and orchestrate an app.

You’ll be able to deliver faster, but more importantly, you’ll deliver in a consistent and predictable manner. Docker containers will help you to isolate and pack your software with all its dependencies. And Kubernetes will help you to deploy and orchestrate your containers. This lets you focus on developing new features and fixing bugs more rapidly. Then you’ll notice, at some point, your deployments stop being a big ceremony.

So, the main thing to remember is this: when you combine Docker and Kubernetes, confidence and productivity increase for everyone.

Agile and DevOps: Why Are You Keeping Them Separate?

2018-04-09T00:00:00+02:00

I first heard about agile many years ago. I learned then that the movement started in 2001 when a group of professionals met and created the Agile Manifesto. The manifesto describes a way of developing software faster by focusing more on how to provide value than anything else.

Personally, when I talk about agile, I like to share a video from Uncle Bob about demanding professionalism. Some of his aspirations include inexpensive adaptability and continuous improvement, and he believes we should always be ready deploy or to deliver software changes. The first time I saw the video, I thought, "Wow, this all sounds great. But how do we achieve everything he expects?" I had an idea of how to pursue some of the goals he mentioned, but for the rest, I had no idea.

That's because some of Bob's aspirations require involvement from operations folks. When I was a developer with practically no knowledge about operations, I believed agile was only for development. At least, that's what I thought until it hit home that my work didn't stop when I finished my code. Although my definition of "done" was "have the software working in a production environment," I had always simply focused on how to develop faster.

But what about the moment after? The operations team struggled every time developers were ready to deploy. Was it from a lack of communication between these two teams? Maybe, but that's how the DevOps movement got started. It came from the desire to break down silos between developers and operations. Maybe we need something more now. Maybe we need to inject agile into this way of working, too.

So why keep DevOps separated from agile? Is it worth it? Wouldn't there be a bigger benefit if they're combined? Let's find out.

DevOps Was Conceived as Agile Infrastructure

The moment developers start adopting agile, operations starts receiving requests for deployments every two weeks or less. And we all know that the production environment becomes unhealthy right after we start making changes. It shouldn't be the case, but it usually happens. I can't imagine the reactions of the crowd when Flickr announced they were doing 10 deployments a day. But they were able to pull it off by identifying the real problem. When development and operations start to work together, both teams will be aware of all the steps needed to prepare the software for deployment. It's mainly about communication issues, blockers, and lack of knowledge.

"Agile," in its shortest definition, means to be able to move quickly and easily. As humans, one of our main problems is interacting with one another in a way that promotes understanding, and this prevents us from moving quickly and easily. The Agile Manifesto starts with individuals and interactions over processes and tools.

What if we added the concepts from agile to the operations world and called it "agile infrastructure"? Well, we might start by immediately applying some of the concepts that succeeded in agile for the software development industry: things like working in small batches and delivering more frequently---or rather, deploying more frequently. This will help the operations team to pay, little by little, the interest they've accumulated through technical debt.

When you don't do anything to reduce technical debt, you'll keep feeling the pain of being paged and constantly interrupted during each release. This keeps you stuck in a perpetually stressful, overwhelmed mindset, enabling the excuse of "We don't have time to implement agile or DevOps now."

That's a shame because, with the amount of APIs that have emerged for infrastructure, it's easier to apply agile practices than ever. As you see, agile isn't only for software developers.

"Working Software" Doesn't Stop at Development

Agile was popularized in development organizations. Perhaps that's why some people relate agile only to development, forgetting about operations. (Well, to be fair, that's also probably because the Agile Manifesto starts by talking about uncovering new ways of developing software.)

The result is that when developers say they're done, they're not really done. They have to wait for operations to install and configure the latest changes in a server, with the chance of getting interrupted if something goes wrong. And most of the time, something does, in fact, go wrong. So one of the first things we need to do is define what "done" means in an organization. This will help to determine who's involved in the process and what the outcome will be.

Customers will expect to have working software in their hands, not in any previous environment, including the developer's machine. So we need to include operations in the workflow. And it's not just that. We should also include security, compliance, databases, management, and everything else that's involved in the process of getting the working software to customers.

But how can that be achieved? Glad you asked. You can make use of the practices that many of the agile frameworks created, and of course, you can use the practices under the DevOps umbrella.

The Practices, Not the Frameworks or Labels

When you take the approach of thinking about practices, you get the real benefit, because there are a lot of agile frameworks, and sometimes people tend to mix them up.

What about labels or definitions? Oh, you'll find a dozen definitions of agile and DevOps. So don't pay too much attention to that. Eventually, you'll have your own too.

What if instead, we turn our focus to the practices? Practices like TDD, BDD, infrastructure as code (IaC), configuration management, continuous integration (CI), continuous delivery (CD)---the list goes on. Knowing what they are and why they're useful makes a remarkable difference in the way you deliver software to your customers.

With agile and DevOps practices, you can reduce lead time and get feedback more rapidly and more frequently. That will let you, as the Agile Manifesto states, respond to a change over following a plan. But it's not just that. You can also run experiments once you've mastered the delivery process. So instead of trying to convince someone else (or even yourself) that agile and DevOps are the coolest, hippest things, focus on mastering the practices. I can't recommend just one of them, but I can tell you that when I got the "aha" moment by practicing them, I couldn't go back. It made me say, "If only I'd have known this before."

In the end, it won't matter too much which practice you choose. What matters is if your performance increases or not. And using practices of both worlds will definitely help you.

Combined, They're Stronger

Are you still going to keep agile and DevOps separate? You could. But there's a lot of synergy between these two. The IT industry started to change constantly after agile, for the good and the bad. Unfortunately, now we have a lot of frameworks, labels, and definitions around agile. But we also have a ton of practices with the spirit of being able to move quickly and easily. Customers expect value, and they don't care what you did to provide it to them---they only care when something goes wrong.

So, you start in development with good technical practices to get the software ready for delivery. Then, you continue with DevOps, which helps everyone to have a broader picture of what's needed to take software from "ready" to "working software in the hands of your customers." DevOps is about all the people that are immersed in the process of software development, not just developers and operations. And DevOps acts as the reminder that software development doesn't stop when a developer has finished coding, even if all tests passed. All that learning development has acquired through the years about agile is valuable, and it's applicable to infrastructure nowadays.

It's important to focus more attention and effort on the practices of agile and DevOps. When you succeed in this, the name of the movement becomes less important.

The Pitfalls of Feature Branching

2018-04-02T00:00:00+02:00

Working with feature branching reminds me of building a house. You don’t see general contractors insisting on making copies of their own houses when remodeling. Working in the house at the same time forces them to talk to each other and come to an agreement over a conflict. Imagine that a contractor finishes a room, tries to leave, and finds the door suddenly blocked by another wall. The same thing could happen in the code after you’ve been too focused on your feature changes. That’s why communication needs to be enforced—and with feature branches, that’s a thing that can easily get overlooked.

There are some situations in which working exclusively with feature branches might make sense. But generally speaking, if this can be avoided, a team’s productivity and confidence in the code will increase by a factor that you never imagined. For that reason, let’s talk about some of the pitfalls of working with feature branches in your development workflow.

You’ll Waste Time Fixing Unnecessary Merge Conflicts

Merge conflicts are the biggest pitfall of using feature branches. Nothing hurts more than spending unnecessary time fixing merge conflicts, especially when a feature branch has been there for a while. But time is not the only factor. The risk of removing any existing code or introducing new issues increases considerably. In some occasions, you’d also need to freeze any changes in code while you stabilize everything. Everyone on the team that was involved must make sure any code is removed or overridden.

I felt this pain when working with a remote team. They were always removing my recent changes. Every time they changed something, I had to stop what I was doing and review their work to see if any of my code was altered—or even removed all together. I don’t blame them. We only had one code repository for a big monolithic app. Instead, I blame merge conflicts.

When there are too many people working in the same source code repository, it could be a smell that you need to decouple your code. Divide your big application into smaller parts to avoid the suffering caused by merge conflicts. Some might call this going from monolith to microservices, although this doesn’t mean you’ll create a git repository for just two or three files. You can pack commonly used code.

You’ll Have No Consistency When Generating Artifacts

When you generate the artifacts that will be deployed in your environments, it’s recommended that you generate them only once. Building and packing should happen in the first stage of your workflow or delivery pipeline. That first stage is usually dev. You don’t want to get any surprises when releasing a change in a production environment. For that reason, it’s better if you deploy the same way everywhere, including production. A change in configuration is all that you’d expect.

Branching might sound good. Working this way, you can isolate changes that are ready to go from those that aren’t ready yet. But let’s think about a common scenario:

You have your “dev,” “master,” and “release-X” branches, and everyone is initially pushing their changes to the “dev” one. Everything that gets there is released in dev and any other test environment. When a change is ready to be released, you create a “release-bugfix” branch and generate the artifacts. After they’ve been released, you integrate changes to “master” because that’s where all stable code exists.

The problem is that everyone needs to update their branches from “master” and resolve any conflicts. A long time could have passed since their last update. But it’s not just that. They’d need to generate artifacts again, and they’d also need to repass the set of tests. It increases delivery time and risk, and you’ll lose consistency. The release process will begin to inspire a lack of confidence.

You Won’t Be Ready To Ship

Which feature branch should be released? Is there more than one? Which one should go first? These are the types of questions that you’ll face when working with feature branches. If you don’t know the answer, you’ll need to take some time to investigate. And it’s not only that. You’ll also need to coordinate with the teams that were working on certain branches to make sure no one deleted another’s code.

What about bug fixes? What if you need to push a change but you’re in the middle of a merge? Or if you have a branch in the middle of the pipeline that’s not ready to be released? Things are about to get complicated.

To make sure you don’t release anything unexpected, you need to roll back beforehand in order to have a clear path. Only then are you good to go by creating another branch for the fix.

It might sound too complicated, but I’ve seen this scenario many times. There are also some branching models like GitFlow that help you with this. But sadly, complexity doesn’t go away, and the chance of ruining things is still there. GitFlow can be harmful to others who work on the code. And the fact that the branching model needs to be this complicated is a sign that something needs to change in your workflow.

Release cadence is reduced by these types of problems. That’s something you need to take care of—the sooner the better. You shouldn’t be afraid of releasing new things.

Feedback and Collaboration Will Be Complicated

When you work with feature branches, the team won’t know what changed until the branch is merged. Feedback doesn’t come quickly. Collaboration is sacrificed because the group focuses only on the piece of code they changed in the feature branch. To avoid any delay, some might even decide not to update their branch until they’ve finished coding. Taking time to decide what code should get mixed in or deleted will also cause delays. All of this adds up to procrastination; people know that fixing a merge conflict will impact their delivery time, so they decide to leave that till the end.

Another downside here is that, at some point in time, there might be a common problem in two feature branches. Each team will solve it in a different way. And because of this, the code might start getting duplicated. More than one solution to the same problem might start to exist. Which one will you fix when a problem arises? Are you aware of all of them? Or just certain people’s? You’ll never know exactly.

That’s why pushing your changes to “master” will increase feedback and collaboration in the team. The feedback you receive is immediate. The team is forced to pull the latest changes more frequently. If there’s a conflict, they can easily review what just changed instead of needing to review a big list of changes when the branch is merged.

It becomes problematic when a long list of files need to be reviewed carefully in each merge request. It’s way better when a developer pushes the latest changes to the main branch and makes sure it didn’t affect anyone else’s work. But that needs to happen as soon as possible.

Knowing What Changed Will Be Difficult

Many things can happen in a feature branch. If more than one person was working on it, the most likely scenario is that there’s a branch from the feature branch for each one of those people. Most likely, many commit messages will be found when merging the feature branch. You could squash all your commits to make them appear as one, but if the code needs to be reviewed, the folks doing it will need to read an awful lot. And they’ll have questions…too many questions.

Even though there’s always a way that you can go back in history and see what changes were made, at the moment of deciding which code should be chosen when fixing a merge conflict, it’s better if you have a quick way to understand why that code was added, removed, or changed. Knowing the context will be key.

In Short, Avoid Using Long Feature Branches!

At this point, you might think, “OK. I understand that there are some pitfalls…but what do you propose then?” Well, I propose to work with feature flags and short-lived feature branches and to practice continuous integration. The most common reason people have for choosing to work with feature branches is that they want to minimize the risk of releasing incomplete or buggy code. But you shouldn’t be afraid of doing that. As long as a feature is turned off, you should be good to go. You can always toggle it when needed.

The pitfalls I described here are real, but this doesn’t mean you should go to the extreme of not using feature branches at all. They’re really helpful. But the key here is how much time they will live. The same thing happens when you don’t release frequently. Changes accumulate, and the risk is bigger and bigger the more time you wait. If you make the switch to working in small batches, you might keep using feature branches, but they will live a day at most. The difference is that feedback is amplified, and you can work on fixing things rapidly because there’s not too much to review. That’s the smart way to use feature branches!

HTTP Monitor: What It Is and Why You Need It

2018-03-26T00:00:00+02:00

One day, one of our main web APIs was down, and the first person that knew it was my boss. We were so worried about bringing the API up that we never paid attention to how he was able to be one step ahead of us. There were times when we even thought he had nothing else to do than constantly refresh the web page. But the truth is that he wasn’t doing that at all. He was using an HTTP monitor that emailed him every time the API was down, slow, or unresponsive.

It was actually lucky for us that he had that monitor: it helped everyone fix things before our clients could notice. But what is an HTTP monitor, anyway? And why else would you need it?

It’s a Friend Who’s Watching Out For You All the Time

Think of an HTTP monitor like it’s a friend you’re paying to constantly check if your HTTP endpoints are working. That friend will hit the endpoints you tell him to, and he’ll follow the exact orders you gave him. Those instructions will be something like, “Hey, I need you to hit this endpoint every five minutes. And only call me if they’re responding with anything other than OK or if you haven’t heard anything back from them in less then a second.”

You can use an HTTP monitor not just for web APIs but also for any of your websites. All that matters is that the application you want to monitor can be accessed using HTTP. It’s also important that they’re publicly accessed. If they’re not, you can make use of a bastion host for critical-no-public endpoints. This is needed because an HTTP monitor is something that’s outside your network. Imagine that you or your provider have massive downtime. Your application will be down, but you also have that good friend who’s supposed to tell that you’re down. People also use HTTP monitors as part of the integration testing suite to validate a deployment.

HTTP monitors are really simple, but fault tolerance and high availability in these type of systems make them complicated. And I bet you have enough of a time trying to keep your application up and running to want to worry about something else. Scalyr helps you to alleviate the load by offering HTTP monitors to probe your servers and measure availability, status, and performance.

Some Metrics You Need to Know Before You Start

It’s not just about whether it works or it doesn’t work. In order to make better use of HTTP monitors, we need to understand a handful of metrics and codes you can use to know how bad, exactly, your app is behaving.

Let’s explore these metrics and codes:

HTTP Response Codes

There are several codes that will give you more information about what happened in a request. But when monitoring HTTP endpoints, the only code that you’ll probably use is the “200 OK.” If you don’t receive this code in the response, something is wrong.

They are grouped like this:

1XX is only informational.
2XX means success.
3XX means a redirection happened.
4XX means something in the user’s request is not correct.
5XX means that something is wrong in your servers.

So, it doesn’t make sense to tell your monitor to watch for any code other than the “200 OK.” If you get something different than this code, it will be useful to troubleshoot, not to monitor.

Response Body

This is simply the payload you receive back when hitting the endpoint. It could be in various formats such as JSON, HTML, or even plain text. This is useful because you can configure your monitors to extract, parse, and analyze the text.

So for example, if the word “error” is found, you can trigger an alert even though you’ve received a 200-status code.

Or what about when, for example, an endpoint should return data for Ferraris and you’re getting back data for Lamborghinis? That could be caused by a data migration in the database. But it doesn’t really matter what caused it; it matters that you can be notified almost in real time automatically.

Response Time

The response time is the time the endpoint takes to respond with a status code and a response body. It’s also called latency. Response time will depend on the endpoint you’re monitoring and the SLAs you’ve defined for it. If it’s an API, it usually needs to be in a range of 100 ms to 500 ms, but if it’s a website, the rage varies from 1 s to 2 s. But it depends on your use case.

There are different metrics that serve different purposes. You can combine them so the monitor will alert you only when something really bad is happening, and not just because there was a temporary problem. And it’s important that your applications are returning the proper code too. I’ve seen cases where the endpoint is returning a 200 code, but when I looked at the response body, it was clear that the application was throwing an error.

You Can React When Something’s Wrong

After you know what data is important when probing HTTP endpoints, the next step is to configure the monitor so that it can notify the proper people. This part is tricky, but it’s the most important step because you don’t want your monitor to become the boy who cried wolf.

For this reason, you might want to only be alerted when something can’t be recovered or healed by itself. Otherwise, the next time you receive an alert, you’ll rapidly think, “Nah, I always get these alerts. The recovery notifications will come after—they always do.” But no, after five minutes you start receiving calls from everyone, including your boss, telling you that the site is down.

So let me give you some examples of HTTP monitor configurations that have worked for me. An alert will be triggered when at least one of the following happens:

The system received a status code that’s different than 200 OK.
It takes more than 800 ms to respond back.
The response body size in KBs is zero.
The response JSON doesn’t include the “success” value in the “status” property.
A custom header in the response isn’t found.
The HTTPS endpoint is timing out after 800 ms and the status code is not 200 OK.

The idea is that when you receive an alert, you previously verified all the contexts of the request. Downtime could be caused by a ton of different reasons. It could be something with a server, and you just need to replace it. Or maybe an external dependency like others’ API endpoints are having problems and you just need to disable them temporarily. What happened doesn’t matter too much at the moment of being alerted. What matters is that you can act soon to reduce your system’s downtime.

You’ll Always Be One Step Ahead

Don’t let your clients be the first in line when reporting a problem in your system. Always be one step ahead of them and find the mechanisms to recover quickly. HTTP monitors will help you to minimize downtime by being aware. They’ll do this boring job for you.

These type of monitors will inform you about the uptime percentage of your system endpoints. They’ll let you know when endpoints aren’t loading or taking too much time. And it’s not just that. You can do interesting things like parse the response body and look for any unexpected results. So it’s important that the dev team knows what status should be returned depending on the result.

Know what you have at your disposal, and take advantage of it.

The Top 5 Use Cases for Feature Flags

2018-03-01T00:00:00+01:00

The concept of feature flags is really simple: it’s a conditional block of code that you strategically put somewhere to enable or disable features of your app easily. But knowing the top use cases for feature flags helps you to plan ahead and be more deliberate with their implementation. There are some obvious use cases, but there are also some that maybe you haven’t considered before.

So let’s explore the top feature flag use cases!

1. Deployments With Less Hassle

Have you ever wondered how some companies release new things just after they’ve announced them at a live conference? Well, with feature flags, that’s possible—they just toggle a switch and enable the new feature. From the moment you start coding, you can wrap the new changes into a conditional block and deploy anytime you want, but with the new code disabled. Thinking of introducing the new change in secret, also known as a dark launch? No problem—just make sure all your tests are passing with and without the new code.

It doesn’t matter really which deployment strategy you choose. Whether it’s blue/green (B/G) or canary deployment, feature flags fit perfectly. Let’s talk about a B/G scenario in which you have the green version running and the blue version as a candidate to be released. The blue version will have the feature turned off. Blue and green versions will run together for some time. And then you turn on the new feature. If something goes wrong, instead of switching back to the green version (because fixing things could take some time), you can just turn off the new feature immediately. Taking this approach, you’ll know for sure which version has problems, and that will lead to reduced downtime.

The same thing goes for canary releases, only the benefit here is more evident. Imagine you encounter problems when almost finishing the rolling update. Normally, you’d have to roll back. With feature flags, you don’t have to worry about that. You just turn the feature off. Feature flags enable your apps to be backward compatible, and that’s critical for deployments with less hassle.

2. Experimentation Through A/B Testing

Most of us have seen the movie The Matrix. What pill would you choose, if you had Morpheus asking you? Red pill for the truth of reality? Or would you pick the blue pill and continue to believe an illusion?

What if you wanted to try both and then choose?

Unfortunately, that wasn’t possible in the movie, but in software, with feature flags, it’s possible.

Facebook, Netflix, and other companies claim that they run experiments all the time. In fact, this is how Amazon launched their recommendations section you see on each product page. A developer had the idea of making recommendations to customers by showing what others had bought when shopping for the same product. They ran the experiment for a while, showing this section to some users and tracking the results, using the A/B testing technique. We all know how it ended because now everyone sees that section.

But you don’t need to be Amazon to run A/B experiments. Let’s pretend we’re working for Amazon and we need to run the experiment I just mentioned. We’d add the code for a feature flag and decide which percentage of users, with the criteria we define, will see the new feature. Then, we’ll collect metrics to see how many more purchases came from those who saw the “customers who bought” feature versus those that didn’t see it. If the new version proved to be more profitable, we’ll just change the feature flag value to make it available to the rest of users and keep monitoring. The great thing about this is if for some reason we decide we don’t want the feature anymore, we don’t have to redeploy; we just need to turn the feature off.

3. Verbose Logging for Troubleshooting

There have been times when I wish I’d enabled the verbose logging in my code—especially when I receive reports that some users have been experiencing errors in code. The first thing I do is try to reproduce the error. If I’m lucky enough and succeed, the next step is to see logs. The sad part of the story is that, at this step, there’s nothing more than “something unexpected happened here.”

It’s understandable if you don’t want to have verbose logging because disks could run out of space. No one wants to be awakened in the middle of the night with this news. But what if you could launch your code with verbose logging turned off and only enabled when you need it? Yes, with feature flags, it’s possible. You could activate all different types of logging that you previously configured but released them deactivated. Everyone will be happy with this approach: you get the information only when you need it, and no one has to worry about being paged about a low space problem or purchasing more space just to avoid it.

4. Always Be Ready for Continous Integration

You don’t need to worry about integrating unfinished code with the rest of the team when using feature flags. A common approach is to use feature branches when working on a new change. The problem with this approach is the dreaded “merge conflict hell”—you know, that moment when you’re trying to merge your feature branch to master. The moment you’ll do anything to avoid. Some clever folks might even distract other developers so that they can merge their branch first and avoid this pain. It would be funny if it weren’t true.

The idea with this, number four in our list of feature flag use cases, is to minimize problems that will steal valuable time from the team. When you code something you know for sure will take more than the rest of your day, you’ll see the benefit of feature flags. That’s because you can always push your code with the feature toggled off. Merge conflicts will be reduced, and it will be much easier and cheaper to fix them because everyone is pushing their changes more frequently.

You might be afraid of pushing your changes to master; something incomplete could go live or the build could fail. But if that happens, other team members will look at the code to identify which flag was recently turned on. Then, they can simply turn it off and continue working. You’ll realize there’s no need for this fear any longer, thanks to feature flags.

5. Customize Behavior Based on User Preferences

Have you ever seen a message in a system saying they’re testing a new layout and asking you to opt into the new interface? I have. And in my experience, if I don’t like it, I can just simply opt out. Feature flags can be at the user level too. If users were selected to test a new feature, they could always just turn the new feature off if it didn’t suit them.

Feature flags can also help you enable certain things for premium users. How easy would it be to upgrade a user? Quite easy. Just turn the premium features on for that user, and you’re done.

At some point, your app might have a ton of features that your experienced users love. Just remember that you always have to think about your new users, too. They could get confused and stop using your app because they don’t even know where to start. A good solution for this is to turn off non-common features. Learn little by little how they’re using your app, and suggest features to them from time to time. Simplicity is the key. Try to avoid giving the impression that your app is hard to use or hard to learn.

Just Turn It On or Off…But Plan Ahead

This is how a few more lines in your code can let you experiment more. You’ll also enjoy the peace of mind that comes with just being able to turn things off when the result isn’t what you expected. When you plan ahead and write your code intentionally, good things emerge from it. Everyone’s productivity will increase because it’s not just developers that will be able to push unfinished code; also, the operations team will be able to roll back easily when it’s needed.

Even if I was able to convince you where and how to use feature flags, until you start practicing using them, all of the above will be just theory. And one last thing—you can use feature flags almost everywhere, and you can use them many times, but eventually, they’ll need to get retired. But meanwhile, apply the advice in the feature flag use cases listed here, and enjoy watching your stress levels decrease!

Sexy But Useless DevOps Trends

2018-02-23T00:00:00+01:00

What’s sexy but useless? A Ferrari in a traffic jam. It’s beautiful, but all that power means nothing. When trapped in traffic, it can’t live up to its full potential.

Same with DevOps. While there are some critical DevOps functions that you absolutely need, there are some sexy but useless DevOps trends that are good to be aware of. Truth be told, there’s no recipe that will tell you how to succeed in DevOps. Everyone will have different opinions, and what worked for others might not work for you. But you can trust one thing: there are some actions that will guide you directly to frustration with DevOps.

With the amount of information out there about DevOps, you might get overwhelmed and think it’s not for you. You also might think the learning curve is too steep—that you need to change too many things before you get started. Maybe you’ll need a new team, new tools, more metrics, more time… you name it.

My advice is this: don’t get distracted by all things that people say about DevOps. These things I’m going to talk about here, for instance, are all style and no substance.

1. Having a Separate DevOps Team

The idea of DevOps is to remove silos. It doesn’t make sense then to include another team that will be in charge of handling or easing the communication between developers and operations. Historically, these two teams have had conflicts, so adding a DevOps team is similar to adding therapists for each team. Having a DevOps team might look sexy because now you can tell the world you’re doing DevOps. Your team will hurry to update their LinkedIn profile. But in reality, you’re just adding more complexity.

What would a potential problem with a DevOps team look like? Well, developers and operations could start saying, “It’s not my job to automate the release; it’s the DevOps team’s responsibility.” And what about accountability and ownership? You’ll lose it.

Another downside is that you’ll need to have a “full-stack” team that has a background in development, operations, testing, and security. I don’t doubt there are people with these skills, but you lose a lot when someone isn’t specializing in one thing.

So instead of having a separate team that will create more handoffs, what you can do is identify where the problem really is. It could be as simple as changing a culture that doesn’t share knowledge. Are developers thinking about using a new NoSQL database? Great. The developers should let operations know so they can learn it too. Is operations thinking about adopting containers? Nice. They should let developers know about it—I bet the devs would appreciate a production-like environment.

So, please, don’t add a new team to your DevOps initiative. Say no to silos. Instead, target the real problems: the ones that made you think you needed a DevOps team in the first place.

2. “Going DevOps” By Virtue of Adopting Some Tool

As the great Grady Booch once said, “A fool with a tool is still a fool.” While using the fanciest, most-almighty-seeming tool for DevOps might seem sexy, it’s useless if you’re hoping it will magically make you a practitioner of DevOps. Same for things like cloud, microservices, or containers. You don’t need any of those things to provide value. If you don’t know what problem a tool or technology stack is solving, ask yourself why they even exist.

This doesn’t mean that tools aren’t important and they can’t be useful. Some of them will help you speed up, but you shouldn’t fully depend on them. They’re always evolving.

In fact, you can embrace DevOps without tools, and James Shore’s post on how to do continuous integration with the bare minimum sets a baseline for what I’d say about tools and DevOps. The post basically says that you only need an old computer that you’re not using anymore, a rubber chicken, a desk bell, a script, and Kool-Aid to do continuous integration. Sounds silly, but it helps you understand that you don’t need a tool to start things; you can use what you already have in place. You just need to be creative.

3. Having Developers Manage Production Environments

You might be tempted to do this, especially after hearing that companies like Netflix and Amazon are letting their developers push changes directly to a production environment several times a day. But you shouldn’t forget that, in order to achieve this, the delivery pipeline needs to be deterministic and accompanied by a trustworthy set of tests. While I believe that DevOps encourages freedom, I don’t think giving the keys to production to everyone will be beneficial. Folks have been fired on their first day of work because they were allowed—and proceeded—to screw things up badly. Imagine giving everyone at a bank a key to the vault. It’s like that.

I’m not saying that developers shouldn’t be trusted, nor am I saying that companies that give full permissions to developers are nuts. It’s just not for everyone. Giving this type of access could make you ask why you need an operations team.

Instead, consider giving developers indirect access to production. By indirect access, I’m saying the operations team should provide all the tools developers need to make better decisions. For example, you could give developers read-only access to the system you’re using to administer infrastructure (e.g., AWS or vSphere).

You could also give developers access to the log management tool or to whatever you use to store infrastructure metrics. If developers have visibility into things like CPU, memory, requests per second, error rates, and other such things, it will help them be accountable. You don’t need to give them full permissions. Just give them the proper tools to make their work easier. And in the end, remember that operations folks have way more to do than just releasing and operating systems, and most people are just trying to help.

4. Giving Yourself X Months to Become “DevOps Compliant”

Are you looking for a crystal ball that can tell you when you’ll finish with your DevOps initiative? That would be Ferrari-style sexy. You could, in fact, reach the point of doing several deployments per day, being always ready to deliver. That sounds next-level sexy, especially to top-level management. But the ambition to finally have “arrived,” as far as DevOps is concerned, is fool’s gold. This should be something that you constantly improve upon. What if suddenly, for some reason, you’re not delivering quickly, securely, and reliably? Your process might need to change.

Are you going to stop working on new features? I’d hope not. You should take a more conservative approach initially, but as you progress, know there’s always an opportunity to improve. The architecture will evolve, features will be removed, technology will need to be upgraded, etc. So as the Agile Manifesto says, you should always respond to change over following a plan. Don’t expect that you’ll need X time to be DevOps. It’s a continuous improvement process.

What Worked for Others Might Not Work for You

In conclusion, your own DevOps path might be easy. Or it might make you suffer a bit. But just as organizations are different, cultures and processes can be different—even in the same industry. So why bother with sexy-but-useless rules that not only may not apply to you but also might hold you back? Just keep learning from others, taking what you think will work for you, and then trying things out. Pinky and the Brain were trying to take over the world every night, and they never did it.

It’s important you’re aware that DevOps trends that are sexy but useless exist. But here’s some things that will help you.

Remove silos instead of adding more of them.
Know you don’t need a certain tool or technology to provide value to your customers. Some will make your life easier and will improve productivity, but they alone won’t solve all your problems.
Make sure every person in your organization knows how they are providing value. Doing so will reduce complaints about lack of access.
And most importantly, know that it’s helpful to take the approach of always improving in small batches.

Experiment, adjust, and don’t forget to enjoy the journey.

5 Critical DevOps Practices

2018-02-15T00:00:00+01:00

DevOps is like pizza. We can’t think of pizza without considering critical ingredients: dough, sauce, cheese, and your preferred choice for vegetables and proteins. Everyone likes different toppings. In my case, I can’t think about pizza without extra cheese and meat. You might choose differently, but I think we can agree there are some ingredients that are critical for this food to be called pizza. Quality and ingredients will vary, but some things will always remain true.

Well, it’s the same with DevOps practices. There are some critical practices, and you can’t think about DevOps without considering them. Everyone will have preferred choices regarding the tools and the process, but the practice will remain and each practice complements the other.

Every critical DevOps practice takes time to get down, but the end result will be magnificent. So, let’s discuss what they are and how to implement them.

1. Involvement From Everyone In the Process

Group involvement is like the dough of our pizza. It’s the foundation, the base. This is the DevOps practice that you need to start before any other. Gone are the days when you would define DevOps as a collaboration between just development and operations. What about QA? What about SecOps? And more importantly, what about management? In order to get a good base for DevOps, you have to stop thinking about teams or people. Everyone involved in the process of developing the product should have a mindset of collaboration.

If this doesn’t happen, you’ll constantly have people questioning everything or ignoring important aspects of the project. It’s not like you’re working with saboteurs—no one’s trying to destroy the initiative. But people have to understand the impact of their actions. When you involve everyone in the process, people can grasp the whole picture. And just like that, you’ve laid the foundation for DevOps.

2. Trust Only What You Have In Version Control

Trusting only what you have in version control is like the sauce of our pizza. It’s the first ingredient you put on it. The flavor will vary, but it’s a critical practice.

At work, every time someone interrupts me with a question about a certain functionality—especially one that I coded a while back—I answer with what I remember, but I always add, “Don’t trust me, though. Let’s go and see what’s in the repository.” Now it’s not just code. You’ll have to put in version control infrastructure specifications, dependencies, and configuration templates. Some of those configuration values will be visible to everyone, and others won’t. It becomes critical to have it just for the sake of repetition. Repetition and consistency are key.

The only source of truth should be what you have in version control. Avoid doing manual changes. If something needs to change, it should be done here. Not anywhere else. The team will then know the “what,” “when,” “why,” and “who” when it comes to changes, and that knowledge is the difference between a team with poor communication and a high-functioning DevOps team. As this practice becomes routine, team confidence will increase. There’s nothing like trusting the process.

3. Be Confident With Automated Testing

Cheese! Automated testing is like cheese. And there’s no pizza without cheese. You can’t afford to not have the practice of automated testing in place.

Forget about DevOps for a moment. This practice is critical now more than ever in our industry. If you want to succeed and deploy with confidence, a good suite of tests you can trust is crucial. You need tests that will fail if the app is not providing the expected output. Otherwise, the team will be slower to call something is done, thanks to manual testing.

There are several types of tests that you can choose, and today’s not the day to talk about them. But know that the speed of the tests is a critical factor. If it takes too much time to receive feedback, everyone will avoid them. Some people might go to the extreme of commenting them out or removing them. As long as almost no human intervention is needed and they run fast, your tests are good.

Without the practice of automated testing, all DevOps efforts will be compromised because you’ll instead be dealing with a constant barrage of errors.

4. Regularly Integrate Your Codebase

The regular integration of your codebase is like our pizza’s vegetables. In order to have a balanced diet, you need to include them. As kids, we didn’t like them that much, but as adults, we realized that we need them. You might not like this practice at the beginning, but you will after time. This practice consists of regularly integrating your codebase into the mainline, trunk or master (depending on the version control tool). Avoid long-lived feature branches, and integrate your code at minimum once a day. Everyone should always have the same working copy of the codebase each morning.

Things can go wrong, I know. That’s why, in addition to integrating your codebase, you’ll need to have an automated way to check if everything went well. Compile the code, do your static code analysis, check that all unit tests passed, and then pack to deploy the same code everywhere. You’ll build only once. If you don’t do this, you’ll add more time to delivery, and you might encounter new problems every time. This practice might sound crazy if it’s the first time you’ve heard of it.

Architecture plays a big role here. You will need to have a loosely coupled one. Things need to be easy to change and evolve all the time.

Are you still in doubt that this is necessary? I understand. The first time I read about this practice, I thought, “It’s not for me.” Try it out and see the difference it can make. You can leverage feature flags to always have the option of turning off what you’ve checked in.

5. Deterministic Deployments in All Environments

Deterministic deployments in all environments are like proteins. A good pizza will have them, and so will a good DevOps implementation. I never get tired of telling people that deployments should become a boring task: predictable, without randomness. You’ll need to have the above practices in place in order to practice this one, though. It’s about having a deployment pipeline. There are gates that require the human skill of critical thinking in order to continue.

It’s of the utmost importance that the delivery pipeline is the same in every environment. Remember, you don’t want to be surprised on release day. Development, QA, staging, and production should have the same pipeline. For development, you’ll want everything automated without anyone approving. In QA, you’ll want to have a bigger suite of tests approving it. Staging might be the same but with a different suite of tests, usually that take more time to validate. And for production, it should be a business decision.

Let me tell you also that it’s not just about code. It’s about everything involved in the process of deploying the app in a working state. This means that aspects like infrastructure and configuration should be considered too. Processes like infrastructure as code and configuration management are critical. Of course, automation will always be key because the idea is to provide value to your customers in a timely manner. That’s why this practice is critical. You can’t let fate take care of this. There’s nothing worse than calling things done when developers have delivered.

It’s All About Providing Value

As I’ve been saying, it’s all about pizza. Wait, sorry. It’s all about providing value with this set of practices.

It’s important to have at least these five elements in place in order to succeed. Practicing these will help to increase your team’s confidence more and more with each release. Persevere through the pain of implementing something new, because the more you practice these, the better you’ll become. Customers will be impressed when you’re mature enough in DevOps that you deliver value to them — and in a timely fashion — each time they make a request.

Is DevOps Possible in Government Agencies?

2018-02-09T00:00:00+01:00

Let’s not beat around the bush. Yes, it’s possible.

And before you close this tab because you’re not working for the government, let me say that what you’ll read in this post will provide a good example of how big organizations could implement DevOps—organizations where several teams have to agree on things and documents have to be filed before anyone can act. That’s usually the case with many big enterprises. Bureaucracy and coordination between teams are often what prevent organizations from providing value at the exact right time.

But what does this all have to do with DevOps and government? Well, DevOps is about providing value to someone that will benefit from the product the organization is providing. It’s not just about helping organizations to speed up delivery and profit. Governments have a yearly budget, so profit might not be a big win in this case. Instead, they’re interested in a more social cause: improving human life. The software they produce could affect someone’s life to the point of not getting healthcare coverage. We’ll talk about that in a moment. The point is that we all depend on software now, so it’s not just a matter of profitability. It’s also a matter of providing life-enhancing value.

A common mistake is to think that DevOps is only for certain companies—that you will need to have the smartest people on your team. But that’s not true. I overheard that someone ask Adrian Cockcroft how Netflix was able to get the team they have. He just answered, “I took them from you.” I’m not pretending DevOps is for everyone, but I will say that I haven’t found any industry of software development where this can’t be applied. It’s applicable even in firmware industry. So let’s see how we can take the DevOps approach to some of the common problems in government.

Culture, Processes, and Architecture Are Too Rigid

Government agencies typically have serious silos. That’s because they often work with a waterfall model, and each phase represents a different group. And it’s not just that. They usually have to deal with a series of contractors, and all of them use a different tool to communicate. This especially applies to, say, a ticket system to report bugs. As Conway’s Law says, “Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.” Communication is exactly one of the things that DevOps tries to improve.

The way you introduce DevOps in such a scenario is by working on how teams interact with each other. When you start to understand the needs of the other teams that are also struggling when delivering something, you’ll anticipate those needs and work on removing their hassles.

Let’s talk about security since it’s a big concern in government. You don’t need to wait until you’re about to deploy a change to recertify the software. Instead, you could start injecting those security checks from the beginning. And that list is pretty big in government. Doing this will not only help you to deliver faster but will also leave tracks as the change is being pushed across the different environments.

And because you will be leaving tracks all the way down, you’ll also have the ability to make sure the software is compliant. Everyone will know how a change got to production. You can even get to a point where no manual intervention will be required. Government agencies have strict regulations about what gets sent to production, as well as where and how it gets sent. Compliance is a big concern in this realm, but DevOps brings the benefits of continuous visibility.

Resistance To Change Until It’s Urgently Needed

Usually, bureaucratic shops resist change culturally. Most of the time it’s because, in government, many people have lots of years in the same job. Some might be afraid to lose those jobs, and others simply don’t see the need for change. For them, work might be slow, but at least progress is being made. But what about when there’s an external pressure—a political one that you can’t control? You might remember when the launch of the healthcare.gov platform failed. Everybody had something to say about what went wrong and how they should have done things differently.

The healthcare.gov failure forced the government to change. There are some interesting tales about how they survived during that time and how things worked a year after that failure. There were big struggles with communication. No one really knew exactly the impact and importance of that project. But it’s interesting that after some years, you see that there are platforms like cloud.gov that take care of all the security checks (a document about 4,000 pages long) that government needs to pass so that people can focus only on software development. That platform is even fedRAMP authorized now, and AWS has a dedicated region for these type of workloads.

So it is possible to introduce practices like DevOps in government. Some will resist the change wholeheartedly. Others will say that DevOps is great, but it’s just not for them. They will always put forth a valid argument or excuse, but they often come around to viewing DevOps as the right move when they’re forced to change. It’s exemplified by the Bill Palmer storyline in the book The Phoenix Project: after a big failure in a project, they had to find a way to do things better. They did, but only after the external pressure of outsourcing, IT came.

Working With Bureaucracy In Small Batches

So, you're in an environment where every little thing requires six levels of approval. Do you want to use Jenkins? Great! You just need 15 committees to approve it over two years, and you're good! So how do you work with this? Startups can easily take the DevOps approach because there are not many people who have to agree. But in government or big enterprises, that’s not the case.

Some time ago, I had the wrong idea about my boss and his feelings about introducing something new. I thought he didn’t like it when I’d innovate or try new things. In reality, he was concerned about the stability and quality of our product. But then he encouraged me in a different way. He said that it was OK to experiment in another environment that wouldn’t affect anyone else, especially our customers. That way, we were able to try a new technology, architecture, or tool without risking our business. Also, we’d make sure that we weren’t introducing something that wouldn’t work for us.

That’s the approach you can take when introducing a practice like DevOps. Prove that it’s working, start small, but make sure it will have an impact. Start making your allies and choose wisely. Talk with key team members in each team—those that you know have a voice. Listen carefully to all their concerns, and work on having an anticipated answer. Then, you can offer a few options. And don’t expect that you’ll be implementing DevOps overnight. You’ll have to give in to certain things that you might not necessarily agree with in order to convince the committee that will approve the initiative.

Rome Wasn’t Built In a Day

It will take time for some organizations to adopt DevOps. And we all can learn about other industries that have needed to do things differently. What worked for some might not work for you and vice versa. Especially in software development, there will always be tradeoffs. The majority of the problems in such bureaucratic industries concern culture and process. It’s not about individuals, per se, nor is it about technology stack or tools. I’ve had problems with my friends when coordinating where to go to dinner. Imagine how complicated it can be with the government, where there are so many people and aspects to take into consideration.

There will always be resistance to change. We humans don’t like it. Even in our personal lives, we try to avoid it. Change seldom comes until you need it or you’re forced into it. So instead of trying to avoid it, let’s embrace it.

So yes, DevOps is possible in government agencies. And if it’s possible in this area, chances are that it’s possible everywhere else. The secret is in the strategy we choose to introduce these types of changes into complex organizations.

Why IT and Software Delivery Service Cycles Are Getting So Much Faster?

2018-01-25T00:00:00+01:00

I had the opportunity to attend Amazon's AWS re:Invent conference this year. The number of new services they launched was amazing. And in November, Microsoft held the virtual conference called Connect(). They, too, announced plenty of new initiatives. But what does this have to do with delivery service cycles? Well, as I watched what Amazon and Microsoft have done, I realized that their end goal is to alleviate the pain companies have when developing and deploying new software. They’re letting everyone focus on what’s really important: delivering value to customers.

The existence of tools and services is one of the reasons why delivery speed is so fast these days, but it’s not just that. When you let others take care of the things that aren’t your primary business, you can easily innovate and deliver faster. For example, if you need to do a big migration to the cloud but you don’t have enough experience to do it, it’s better if you hire a contractor to help you. This doesn’t mean you aren’t capable of doing it–or even learning how to do it. But it may cost you, both financially and reputation-wise. It’s way more affordable–and more face-saving–in the long term to lean on someone else’s shoulder.

That’s how our industry is behaving now. There have been so many improvements regarding cultures and disciplines, tools and services, practices and the vast number of resources you can find out there that speed is resulting naturally. It’s letting organizations take software delivery time from months to minutes. So let’s explore some of the things that are making this speed possible.

Changes in Culture That Have Resulted in New Disciplines

One of the first software development methodologies I encountered was the waterfall approach. At the time, I liked the idea that there was an order and well-defined processes to follow. But when I joined a company where they heavily used this model, I was disappointed. I’d get so excited to work on new assignments, but then they’d take ages to deliver, thanks to the bureaucratic processes involved. It was clear that we needed a change in the process. Everyone needed to be involved. After working in waterfall fashion for some time, I was introduced to Scrum. Everything was different. For example, I noticed that with this model we could be paying more attention to creating working software than having extensive documentation.

When we switched to Scrum, we were deploying something new to production every two weeks. In the past, we’d deliver a simple change in a month, if we were lucky. Testing is another example of improvements that came with Scrum. Teams became more confident in what they were working on because something or someone was always verifying they were delivering what was expected. And testing opens the door to refactoring and to always be improving the code. That’s contrary to what usually happens: the older the code, the more difficult it is to make a change.

Now we have test-driven development (TDD), short cycles of development, all sorts of different tests (like unit, acceptance, and integration tests), pair programming, ownership from everyone, continuous integration, continuous delivery, SOLID principles, and even crazy things like chaos engineering. We’re still learning how to apply all of these principles and disciplines, but it’s good to know they’re there. We should take what makes our delivery service cycles faster and discard what doesn’t.

More Tools and Services That Let You Focus On Your Business

There’s one main reason you’re seeing tools and services being launched so frequently: those companies or communities are trying to make your life easier and more productive. Proof of that is all the serverless waves you may have been seeing recently. It’s a simple and fast way to get started. From prototyping to validating a concept to creating something new, you don’t need to worry about provisioning a server to host your application. And you just pay for what you use. Cloud providers are launching these types of tools with the sole purpose of helping you to become more productive. If there’s something you need that a cloud provider doesn’t offer, there might be a company already offering a software-as-a-service (SaaS) solution for that. Twilio or Auth0 are just two companies that specialize in this.

Also, for any tech stack that you choose, there’s likely to be a powerful IDE available. There are so many IDEs right now that are making people more productive. Features like live unit testing from Visual Studio or pair programming from Visual Studio Live Share or Cloud9 are good examples of productivity-boosters. Those features are appealing because you can receive feedback instantly, and this certainly improves your speed of development.

Another important tool is Jenkins, which helps in the field of deployments and code integration. You can have at your fingertips the possibility to deploy a new change to a live environment in minutes. Of course, that’s only after the change has passed a set of tests successfully and after the team knows it isn’t making the application worse, overriding anything, or causing any downtime. While these benefits aren’t 100% attributable to Jenkins, the tool is absolutely helping to achieve them.

Better Practices That Enable Ease of Change

When you hear that there are some organizations delivering software changes in minutes, your first impression might be that their system must not be that complicated—or that they’re nuts! But in reality, these organizations are leveraging automation. This is by far the more important practice that’s helping everyone to go faster. The rise of DevOps as a movement reinforces this practice of relying on automation more and more each day. Many practices in this area will help you to deliver with confidence: practices like configuration management, continuous integration, automated testing, infrastructure as code, immutable infrastructure, release strategies, and more.

But architecture also plays a big role here. You’ll need to keep it in mind that things will change. Technology will evolve. We can choose to evolve with it, or, if we wait too long, we’ll be forced to evolve with it. Functionality will change, too. This happens all the time. And if you don’t act wisely, you’ll accumulate technical debt. That technical debt, in the end, can force you to start over. That’s costly.

Instead, what’s happening right now is the emergence of new patterns that let us be less afraid of change. There are practices like implementing domain-driven design (DDD), using design patterns, building microservice architectures, making use of containers, and taking the serverless approach, just to mention a few. Of these examples, containers are the most acclaimed right now. Everybody is talking about containers and with good reason. Organizations are increasingly adopting them, thanks to the portability and packaging that this technology offers. Containers enable more collaboration between teams and reduce the time taken for deployments—and even for rollbacks. In some way, they enable you to go faster by forcing you to document each change, even at the infrastructure level. This makes everyone aware of what’s being pushed, and potential problems practically identify themselves. Plus, it’s easy to change something; if it’s easy to change, you can deliver fast.

Focus On What Your Customers Value Most

We’re living in wonderful times. It amazes me how good our industry is becoming at delivering useful things to the world and how collaborative we are right now. We’re not only focusing on delivering fast. We’re also paying more attention to what customers value most.

We’ve learned from our mistakes. We’re creating tools that make our lives easier and more enjoyable. New practices are becoming the default, thanks to the results they’re giving in return.

It’s true that things are changing fast, but the good thing is that you can easily tell when something is becoming relevant and useful if it stands out from the rest. You might be practicing some of the things I mentioned here. If you’re not, at least consider trying some of them.

How Does DevOps Handle Change Management?

2018-01-11T00:00:00+01:00

Tools today are wonderful. They help ease your task load, letting you get things done with only a few clicks. I remember reading a potential horror story about a developer promoting a change in the database to a production environment. He was trying to add a column to a table, and a tool generated the script with a "drop table" statement at the beginning. The developer didn’t notice it because he tested the script locally. Luckily for him, they had a DBA that always reviewed developer’s scripts. When he discovered that the script was going to recreate the table just to add a new column, he reported that back and no data was lost that day. Phew!

That’s change management (CM), the process by which you can understand why a change is needed and how to minimize risk as much as possible. My first encounter with CM was horrible. I was used to deploying new changes rapidly, without a formal announcement because the team was small. The problem with that was I didn’t have free time; when something was failing, I was the only one who’d know what was causing the problem. And that meant it was me that had to connect remotely and fix it.

I always thought that CM was only for big companies, not startups. After all, startups have to stay light and lean! But the idea behind CM isn’t to slow down company initiatives. Instead, one of its main purposes is to reduce risk when releasing new changes, which is something everyone, including startups, can use. A key point of CM is to provide documentation. And CM is also about having a human to review, analyze, and approve changes—to make judgments. After all, people are better able to make decisions than computers. CMs aren’t meant to prevent failure, but what if we can automate the things that can be done by a computer? We can have feedback frequently and rapidly, letting us humans take on fewer—and more interesting—decisions.

With the speed that DevOps brings, it’s only logical to think that more problems will arise and our systems will become riskier. DevOps isn’t meant to supplant CM. In fact, its purpose is to replace the need for human intervention. So CM and DevOps can live together. When we merge speed with less risk, the results are incredible. So how does DevOps handle change management?

Treating Changes Differently

Not all changes have the same impact. You can’t measure how risky a change is by counting the lines of code modified. It’s so easy for little things to create big problems, like just changing a font color in a web page. For example, ITIL has three type of changes we can use as a base to classify and, based on that, apply DevOps principles. Those changes are standard, normal, and emergency.

A standard change is a low-risk change that’s already well known, and it follows a strict procedure. Everyone loves these types of changes.
A normal change is, well, normal. You don’t know how risky these changes are; they’re either new or they’re those changes everyone is afraid to make because there’s always a problem with them. For these type of changes, you usually request approval from a change advisory board (CAB). The CAB analyzes it before going live, trying to deduce the impact on the system. Usually, IT Ops would like to avoid these types of changes.
Emergency changes are those that need immediate attention. You can’t wait for them to go through the formal approval process. Everyone would like to avoid these, but sometimes they’re inevitable.

Having this knowledge as a base, we’ll need to practice making those changes. In the beginning, you won’t have many standard changes. Most of them will be normal changes. But the end goal is to transform normal changes into standard changes by having automation in place.

DevOps will start treating your changes differently, with the purpose of improving lead time and reducing blockers by introducing automation all the way down. The CAB could reject a change because there’s missing documentation, and a much-needed fix could be delayed because of this. Next time, you can automate this process by automatically filling in the documentation from the user stories, emails, or whatever you use to document requirements.

Integrating With Existing Tools and Processes

You don’t need to stop doing what you’re currently doing and drop all your existing tools. Hack them. Find a way that blockers stop being blockers. The problem most of the time is lack of documentation and poor communication. You don’t always remember what steps are needed unless you do it all the time. Humans are awful at at repeatable tasks. Seek to have an improved process in which, after the developer pushes a change in the code, all the machinery is triggered and starts creating requests for changes (RFC) and JIRA tickets, sending emails requesting approval with links or summaries, etc. This will leave humans free to do what they’re best at it: applying judgment.

The main goal is to prove we have everything we need to make the change operate as expected when published—especially when the business hits the “publish” button. The CAB will appreciate all the effort we went to give them enough information and context for a change. And this new automated process will give them all the proof they want to see before releasing a change to the users.

Leaving Tracks All the Way Down

When you have automation in place, it becomes a trivial task to incorporate audit trails into the pipeline. This brings big benefits. Anyone who wants to can know how much time went into a recent change going live, why it was needed, who approved it, and whether all checkmarks were ticked off in previous steps. For example, the next time an auditor requests evidence that a change followed your process, it’ll just be a matter of following the trail backward. All information will be available. But along with these big benefits come big challenges. That’s especially true when you need to skip the process and make manual changes for an emergency change, for example.

Avoiding Manual Changes

When you understand this, you’ll gain so much confidence with deployments that you’ll never want to do manual changes again. In a perfect and immutable world, let’s say you eliminate the temptation to implement manual changes by locking down the doors and forcing everyone to avoid SSH’ing to servers. There may still be times when you will need to break the rules, but you’ll want to make that the exception, not the norm, and automate it. Having problems and need to see logs? Go to the log management tool. Running out of space? Ok, fix it. But next time, add an alert and trigger an automated process to increase storage. See? It’s doable.

Transporting From Dramatic to Non-Dramatic Changes

Now we’ll have enough information to prove that a pipeline is trustworthy. There’s proof that you’ve had fewer incidents with automation. The CAB will start seeing those types of changes as less risky, and less and less human intervention will be needed, even to approve changes. Why? Because to have a solid deployment pipeline, you have to have all types of evolving tests included. There are some companies that will still want to have the approval gate for business purposes (e.g., a change will go live on the announced date, not before).

The idea is to transport a normal change to a standard change and make emergency changes need less and less human intervention. This will take time, sure. But be patient and seek to have a green-light pipeline most of the time. Make sure that if it goes red, it’s because a test failed and not because there was something wrong with the automation process.

Use Your Weapons Wisely

All the things that have been blockers—you’ll transform them in a way that makes them easier to live with. (The “aha” moment for me was when I read The Phoenix Project, and I can’t recommend a better book for understanding why CM is needed and how DevOps helped a company escape the hole they dug themselves into.)

Meanwhile, relax. DevOps is not trying to remove CM. It’s trying to make it more powerful. You can still use your current processes and tools, but you can do it in a more strategic way. Focus on trying to have more standard changes. Start with one—the least problematic one—and prove the theory. ITIL practices have been in IT for quite a long time, and it’s for a reason. Embrace change management, but do it smartly by having an expected and predictable delivery pipeline.

Integrated Security: DevSecOps 101

2017-12-14T00:00:00+01:00

Everyone at some time or another has received an email from a service saying their password has been reset because data was compromised. Security has always been a big topic in IT, and it’s sad that some companies don’t take security as seriously as others. New vulnerabilities are found every day, and most of them are discovered long after they’re introduced. Equifax is currently feeling the pain of a breach, and it's a lesson companies should learn from. Luckily, many IT professionals already are thinking ahead about security. But we still have things to overcome.

Part of the problem is that we’re still using traditional methods. We’re thinking about security only at the final stages of the software development lifecycle (SDLC). But security isn’t something you should consider only at the end. Imagine if car manufacturers started considering security after the car had been designed and assembled. It’s too late by then. Now let’s use the same analogy for testing. In order for a change in code to get to the end user, it has to pass through a set of stages. It’s good practice to start testing at the beginning of the lifecycle of the project—not only because you're addressing security before it’s too late but also because it’s cheaper.

Just think of it. An application is ready to go live, and we just need to make sure it’s secure. Boom! The potential for a breach was found during testing, and it needs to get fixed in the code. This forces you to start all over again. That change in code needs to be validated to make sure it doesn’t break any existing functionality, meaning the end user will have to wait longer to get the fix or new feature in his/her hand.

There are other problems, too. What about the frustration of the application needing to be scanned just before the deployment? The security team can become a bottleneck. As the Agile Manifesto’s first principle says, “Individuals and interactions over processes and tools.” It’s more about a cultural change than just acquiring new tools to make apps more secure. It’s not enough to have leverage only on firewalls. We'll get better results when, instead of being a bottleneck in the process, we work with the team to make security everyone's responsibility. And that’s where development, security, and operations (DevSecOps) combine.

DevSecOps. It’s about feedback and a blameless culture. How will security teams know that someone found a hole in the app’s security? Well, the Ops team is good at this. They can provide the needed feedback by collecting metrics and reviewing logs. This same information can be shared with the development team so they can apply patches in the code.

Some say that DevSecOps is just a new buzzword. Others say that DevOps and DevSecOps are the same thing. In the end, the name doesn’t matter much. What matters is the intention to shift security to the left of the cycle.

To help move that shift along, let’s explore several topics worth seeing in more detail.

Security As Code

In order to innovate by pushing new changes in a timely manner and at the same time minimize risk, we need to pay attention to security right from the beginning. This becomes a no-brainer when you start treating your infrastructure as code (IaC). Think of security as code, like the test cases you’ll be running against the IaC. But don’t stop there. There are some tools out there that perform static application security testing (SAST) and dynamic application security testing (DAST) on the code, and they can help you go the extra mile.

SAST analyzes the source code by looking for common vulnerabilities—and it does this without needing the application to run. It won’t catch all problems, but it certainly reduces the attack surface. It’s a white-box testing methodology, testing from the inside out. In this regard, it’s similar to unit testing.

DAST covers scenarios by acting as if it was an external user. It runs against the real application, similar to integration tests. It’s a black-box testing methodology and tests from the outside. When you implement this type of testing in the SDLC, you’re making sure that, for example, SQL injections are not possible or that sensitive information is not being exposed to the end user. You’re also controlling what the user is able to see when there’s a managed and/or unmanaged exception in the code.

How much test coverage will you be able to get for on-premises? That will depend on if there’s an API you can use to interact with your infrastructure’s security. In the cloud, this can be achieved easily, as every interaction is done through APIs. You can run common checks, like seeing if the SSH port is open to the world, if some servers are in a private network, and if secrets are encrypted, among other things. Basically, you’ll need to make sure you’re giving least privilege and granular access to resources.

Injecting Failure Continuously

The only way you become better at something is by practicing. No matter how hard or painful it is, you have to do it. It’s just like when you were learning to ride a bike. We need to inject failure into our systems. The best example of this is the well-known Simian Army by Netflix, with its security monkey that’s constantly watching the state of the app’s security.

Don’t chase a final state for security. You can’t make security a project that has an end date. It never ends. New security holes are found every day, so it doesn’t make sense to stop thinking about how our apps can be protected if hackers are constantly trying to find new breaches. Vulnerability scans are not enough in today’s world; consider doing penetration tests as well, which can be performed as soon as you start development. Having exploit testing can bring many benefits. Instead of just relying on security scans, you’re actively monitoring.

Make use of the information you already have at hand and take actions based on the valuable information you can get from logs. All types of logs. Use log management tools like Splunk or the famous ELK (Elasticsearch + Logstash + Kibana). Gone are the days when you download logs from the servers and analyze them. Thanks to the cloud, it’s common to constantly generate new servers when something happens or when implementing blue/green deployments. You can’t afford to lose the last known state of a server. It can be your chance to find new ways to protect your application or see what the attack did.

Automate to Improve Agility

The time you take to react when a new vulnerability is found is crucial—especially if you have hundreds of servers (or containers). You need to act rapidly or be ready to face the consequences. You can take the approach of patching each and every server, as normal operations would. But if you’re using immutable infrastructure, you just need to create a new version with the patch applied and replace all servers. This is accomplished when you treat your infrastructure as code. There’s no better way to react in a consistent and predictable manner than with automation.

You can even protect yourself automatically when an attack is taking place. There’s an excellent example of this that I always like to present when talking about the value of automated protection. A site was constantly having DDoS attacks, and in order to remediate this, the administrator temporarily blocked the attacker’s IP addresses with Azure Functions. After some time, those IPs were again whitelisted, but the administrator was constantly monitoring the process. In other words, he was applying automation that he was actively supervising.

You can also use AWS Lambda functions to act based on API calls. For example, you can set up an alarm that will alert your team when a security group (like a firewall) changes. This is what the security monkey mentioned above does. You can even get to the point that if something changes in a way that’s outside of your defined parameters, you can revert it. That’s taking security to the next level.

Self-Service Reviews

Anyone should be able to see the state of security—especially auditors. It’s pretty common for them to request a static snapshot of an app’s security status, but that isn’t a very useful way to demonstrate that you’re applying security practices. This is like a picture. You always see the same thing when you look at it. Plus, it’s taking up someone’s valuable time, as they construct these boring reports. Even recording a video of how security checks are being performed limits you. You’re just watching a recorded period of time, with no idea what’s happening outside that window. And it’s still wasting someone's time. I was pretty disappointed when an auditor requested I simply send a set of screenshots of password policies, active users, groups, and roles for a certain tool. And easily-manipulated image files shouldn’t be the only source of information, should they? In this circumstance, the phrase “trust but verify” makes a lot of sense to me.

Why don’t you instead invest some time in creating the building blocks that make security visible to anyone? When auditors come around again and start asking you about security, you can just give them access to whatever tool you built and make them do the reviews themselves. This time, we’re not wasting people’s work hours, and we’re gaining so much. We’re demonstrating that security is important to us and that we’re practicing it constantly. And our report becomes more credible because it’s not subject to human error or fraud, intentional or unintentional.

Security Is Everyone's Responsibility

This is not just about a person or a team. We’ve already learned that isolating development and operations brings conflict. Don’t make the same mistake with an aspect of your application as important as security. DevSecOps is not introducing a new concept. It’s just making the point that we need to think about security for more than just a moment during development. It’s something that we should think about every day, and it’s everyone’s responsibility. It’s about collaboration. And it means that you can’t afford to have a finger-pointing culture, where everyone’s trying save his/her own skin. We all make mistakes, but having a team that supports you makes all the difference. This mindset is an enabler of experimentation, and that opens the door to innovation.

In the end, if you’re going to take away anything from this post, let it be this. Leverage automation to ensure your applications are secure, and act rapidly when there are potential breaches that could affect you.

Want to know more? Check out the DevSecOps manifesto and this awesome content list in GitHub full of links to tutorials, videos, tools, and more.

The Business Case for Container Technology

2017-12-07T00:00:00+01:00

When you see the word “containers”, you probably immediately think of Docker. But containers are not a new concept. It dates back to Linux-VServer, initially released in 2001. Then in 2008, containers improved with the release of the Linux Container Project (LXC). Docker came along in 2013, making things easier by using the command line interface to interact with LXC. But you didn’t come here for a history lesson. Even though “container” seems to be a buzzword right now, as you can see, they’re not something new. They've been around for about sixteen years now. And Docker isn't the only container technology, either. There are other projects in the container ecosystem. And rkt (another flavor) makes a good comparison on its site.

I also won’t give you the vendor speech about containers—I’m not a salesman. But let me tell you something: if you want portability, efficiency, and security for your applications, you should consider using containers. With containers, you can deploy the same code through all environments, without any modifications. You can make your database endpoint configurable so that it’s just a matter of configuring it for the environment in which you’d like to run the app. It can be on your computer, in the cloud, on a virtual machine, on bare metal or a mix of everything.

Before we dive in, you should know that containers are not a silver bullet. Using them requires a change in culture, not just in code. (In fact, you might not even need to change the code). But make sure you’re clear on all the benefits and requirements before adopting the container approach. Don't do it just because everybody is doing it. All that being said, let’s look at the business case for containers. This should give you a clear idea of where and why you’d use containers.

Portability as First-Class Citizen

One of the first things you’ll hear about containers is that they don't only work on your machine but also on others' machines. You can easily recover from a disaster and move your environment to another place—from on-prem to the cloud or from one cloud provider to another. You avoid vendor lock-in. Your application will continue working, guaranteed…or at least it will work as long as you make sure all dependencies are in place too, as with databases.

You can also reduce risk by repetition. By the time you’re ready to deploy to production, you’ve proven that the code works by testing it in familiar environments. It doesn’t matter if you don’t have automation in place; you’re just moving the container from one place to another. How can this be achieved, you might be wondering? Well, since you’re isolating the application in its own environment, you can make use of environment variables. That means making it behave differently is just a matter of how you run the container and what values you use.

You can pass a list of values to your container, like calling a parameterized function in code. Or you can use just one value to define the environment and then have the app grab the values it needs from another location. The compiled code inside the container will remain the same regardless. This way, you’re gaining consistency. You can increase the level of trust because your containers are completely isolated one from another, and they're especially isolated from the host. Gone are the days when you need to reboot a server because your app is having problems. Now, if you’re having issues with your app, you just kill the bad container and create another fresh one in seconds.

Be More Responsive to Change

How much time do you need to take your application code from a developer machine to a production server? How many validations in different environments do you need to perform before saying, “It’s ready to go"? And how many people are involved in that process? How many manual interventions need to happen? If the answer to these questions is “a lot”, start tracking how much time you actually invest in moving your code around, as opposed to developing new features. In the end, if you’re not an infrastructure company, you shouldn't need to bother with these activities. Make your life easier.

With containers, you focus on what’s important: providing value to customers. That means less time spent debugging problems and more time spent delivering useful features. When you pack your application code in a container with all its dependencies, you can iterate faster. That’s because you know the code will work in all environments. And you know the build won't fail before going to production since you only have to build once to deploy many times. You’ll get to the point where you won’t be adding more external dependencies or environment configurations. And finally, you'll make more changes in code than anywhere else. See? Fewer and fewer infrastructure headaches.

One of the nice things about packing the application code and its dependencies is that you can use versioning. Instead of modifying an existing version, you create a new one. It’s making your container immutable by going forward, not backward. But what if something’s wrong with the new release? If you’re practicing continuous integration, you’ll stop everything, make the correction, and deploy again. In the meantime, just use the previous container version image and deploy again to avoid any interruptions. But you have to be able to do this fast. Another way to do this is to just turn off a feature by changing the configuration (a.k.a. feature flags) and then deploy again.

It Can Be Adopted to Legacy Systems

Maybe you’re thinking that when you adopt containers, you’ll have to make big changes. Or maybe you think you’ll have to adopt a microservices architecture. Save those worries for later because they’re not valid in this case. Don’t believe me? I don’t blame you—I bet you’re thinking of that Windows service you’ve been running forever. Or maybe you’re thinking of that old app that you don't even dare to look at, let alone touch.

If that Windows case sounds familiar, there’s a Powershell module called Image2Docker where you’ll be porting your application using Docker without making any changes. Is your issue not a Windows one? Don’t worry, Docker likes challenges. They have a program called “Modernize Traditional Applications” (MTA). It won’t force you to modify the code or make other application changes. And the folks at Docker guarantee that they’ll help you in less than five days. If they need more time, it means things will need to be added to the platform.

Make Better Use of Infrastructure

All enterprises like to reduce cost, whether it be through cutting waste like idle resources or making sure servers aren’t overprovisioned. The cloud gives you a set of server types with predefined characteristics for memory, CPU, and networking. This means you’re more likely to overprovision resources. In the case of on-premises, the chances of idle resources increase. You have to plan ahead of time. But with containers, if you need more capacity, you just spin up more containers on the same host. You don’t have to worry about more infrastructure. Just make sure you have enough resources available. A good, simple example of this is the AWS service called ECS. In ECS, you configure your auto-scaling policies with the "reservation capacity” metric. It doesn’t really matter how many resources you use but rather how much you reserve. In the end, resource usage will serve as a metric to add more containers, whereas reserved capacity will be used to add more hosts.

Maybe you won’t see big savings at the beginning, but you’ll see a real benefit when you’re in need of scaling. Instead of adding big servers, you can have small servers and add just the blocks you need, little by little. On the other hand, without containers, you might need to buy the same size of server (on-prem or cloud) because that’s the capacity your app is currently needing. But you won’t be able to use all resources, leading to waste.

You might be asking yourself, “Aren’t we making things more complex?” Maybe. But for the example of on-prem environments, are VMs getting recycled when the app isn’t behaving well? No. At least not automatically. In the cloud, you have self-recovery built in, but on-premises, you need to manage that by yourself. If you’re using vSphere, for example, you can rest assured that if something goes wrong with the host, that VM will be moved quickly to another one without any downtime. That’s cool. But what if the host is OK? What if the VM is OK? What if the problem is inside the application because it’s using so many resources? In vSphere, you do have the ability to auto-scale, but it’s not something that comes built in, sadly.

Increase Security for Your Applications

People sometimes talk negatively about containers’ security. After all, if you have access to the host, you can inspect and log in to any container easily. But as the industry learns more about containers, that perception is rightly starting to change. This doesn’t mean that you won’t need to worry anymore about security in your applications (for example, in SQL injections). It means that if for some reason, the attacker is able to gain access to the container, you can limit the damage that can be done. This is thanks to the fact that containers can work with only what’s needed. You don’t need to have all OS capabilities inside a container unless you specifically need it.

For this reason, Docker recently launched a program that certifies images. If an image is certified, it means it’s been tested, it’s implementing best practices, and it doesn’t include common vulnerabilities. It’s also supported by Docker and the provider. So unless you can run a security scan, don’t use other people’s images, and you should be good on that front.

If for some reason an attacker gains access to the container, the attack surface is limited. That’s because, in Docker, containers run in an unprivileged way by default. Why is this important? Well, if that weren’t true, the attacker could gain root privileges and infect the host or other containers. But now, you don’t have to worry about that.

Containers are also isolated thanks to namespaces in Linux. You can reserve a certain capacity (memory and CPU) to containers thanks to cgroups. This means you have a truly isolated environment for containers. It doesn’t matter if one container uses all its resources. It won’t affect the rest of containers and, more significantly, it won’t affect the host. (By default, Docker terminates the container if it uses 100 percent of memory.) Next time you see that something bad is happening to a container, you can just kill it and create another one. And again, you can do this in seconds. Want to know more? Take a look at this great security cheat sheet here.

Container Adoption is Increasing Rapidly

The adoption of containers is finally starting to grow. More enterprises are signing on for the adventure. You can check out a great study from Datadog to get a better idea of how Docker adoption works. What’s interesting from that report is that large companies are the ones adopting containers. More and more tools are being released, and they’re being released frequently. Container vendors are also providing enterprise editions to companies that care about core features: portability, efficiency, and security. For example, Docker is offering an enterprise edition. CoreOS has an enterprise-ready Kubernetes called Tectonic. Red Hat offers OpenShift. There’s also Rancher, and Hashicorp has Nomad. That offering list is growing, and I just named a few.

So if you thought that containers were the new kids on the block, think twice. Maybe it’s time to modernize your application by using containers because containers are not the future: they are the present.

Infrastructure as Code (IaC): The Benefits

2017-12-01T00:00:00+01:00

When pushing application code changes to any environment, one of the most common problems you’ll encounter is that servers aren’t configured properly. How many of us have faced not having the same exact version of a dependency across all environments? I’d say everyone. When our application is failing and we’re in a rush, we tend to log into the server, find the error, and apply the fix. This might not be seen as a big problem. But when the same problem arises in another environment and we don’t remember what we did to fix it, it is a big problem. Sound familiar?

We humans often forget things, especially when it only caused us a small amount of pain. But we’ll never forget what we did wrong when we brought down a production application. The struggle at that moment is horrible; it’s sometimes traumatic. Maybe that’s one of the reasons we usually have a set of scripts and commands that take the risk of human error down a notch. There’s a ton of information out there on how to install and configure helpful services, like an Nginx server. The blog posts and articles that provide this information give a recipe of steps we need to follow, and those become our scripts.

Nowadays, not only could you install and configure services but you could also do so for infrastructure. This mostly applies to cloud applications, but that doesn't mean it can't be done for your on-premise infrastructure. You can spin up a set of servers in minutes by just running a command or making an API call. Here’s where the term “infrastructure as code” (IaC) enters the game. IaC means you have cookbooks to help you manage and provision infrastructure in a consistent manner. You no longer have to remember which buttons to click or which commands to run in order to get an environment up and running. When you take this approach, several benefits come included.

Treat Infrastructure Like Application Code

When you treat your infrastructure like your application code, you can be confident that the infrastructure’s file definition won’t get lost easily. Why? Because, as with your application code, you’ll store this file in a version control system. And you’ll enjoy additional perks like change history, ease in rolling back changes, redundant storage, and one source of truth, among others.

When you implement IaC, gone are the days when you made manual changes to the server so the application worked. You and your team will get used to having an automated way of making changes to servers. Every change will be made in the infrastructure definition file. As for what language you’ll be working with, it can be any descriptive language. The most common language is YAML.

Every time you need to make a patch to the server, you’ll define that change in code. A good example of this is Dockerfiles. All the dependencies that the container will need to run the application are defined in the Dockerfile. There, you’ll place the exact same commands you’ll run: say, for example, updating the packages repository and then installing a specific version of the package.

As you do with code, you can deliver small changes frequently. This helps to increase the speed of deployments. According to several reports, you reduce risk by releasing small changes frequently. Fail fast, fail often. If it’s hard to do, do it more frequently so you can become better at it. When you start implementing IaC, you might feel like it’s not for you. But just keep doing it until you get it right because you’ll eliminate the lack of knowledge about what’s installed and how it’s configured.

Same Code for All Environments

I know. The first thing you’re going to tell me is, “Production environments are way different than development environments—and more expensive, also.” But when you define your IaC in a way that lets you control the resources you assign or how many replicas you want, it’s possible to have a small version of the infrastructure for development and a bigger one for production. Environments then become similar.

IaC is not just about provisioning; it’s also about configuration management. You’re reducing the chance of misconfigurations in the application. All the values that are different in each environment will be under version control. Of course, this presents an interesting challenge because you wouldn't put sensitive information like passwords there. The most common approach here is to use encryption or make use of environment variables where only certain people have access, like your Ops team.

If you embrace end-to-end automation in your delivery pipeline, you’ll be promoting changes from one environment to another. In order to take full advantage of this, you shouldn’t allow yourself or your team to make any manual changes that weren’t tested in a different environment first. By testing first, you’re minimizing the chances of introducing instability and differences into each environment.

Anyone Can Build an Environment Anytime

If you have the code and the proper access, you’ll be able to spin up a similar environment from production. You don’t need to bother operations anymore. Imagine that! Maybe you just need to experiment with the newest version of a dependency of the server. You wouldn’t want to interrupt your team if you break something in the environment.

By implementing IaC, you’re able to have disposable and immutable infrastructure. Instead of making a change to the current servers, you’re replacing them with a new version built from your file definition. No more hidden or surprise changes to the environment. If something is broken by a recent change, you can always go back to a previous version and rebuild the environment. That’s the beauty of IaC.

Validate Infrastructure Before Deployment

When I said, “treat infrastructure like application code,” I meant it. The cloud came to revolutionize the way we thought about infrastructure. In the past, you didn’t have to worry about paying more money because you left-turned on a server for several hours. But when you’re in the cloud, you pay for every hour, minute, and even second the server is on. So instead of considering this a drawback, the industry found new ways to protect the budget. I myself had to deal with this problem. When I was developing the scripts to spin up instances and something went wrong, I had to pay for the fragment that instance ran. I needed to be very sure that my script was doing what I wanted.

That’s the reason you can even create unit tests for your infrastructure code. It’s also possible to do static code analysis (linting)—that will help you prevent some obvious problems. And as if that weren’t enough, you can also plan the changes your script will perform so you can preview how the infrastructure will look. It’s a pretty common feature in Terraform, and it was recently introduced in AWS CloudFormation. When the infrastructure is ready, you can also check that it’s properly configured by running some Serverspec tests. Could you ever have imagined these kinds of features with your infrastructure? I didn’t.

Always Know What Changed

You can keep track of all changes made to the infrastructure because you defined each one in a file. If you respect this rule, when something goes wrong in the deployment and you don’t know why, you can always check the history, revert, and rebuild.

Tracking your changes is also useful for compliance reasons. Auditors love to see that an organization keeps track of how a change went from one environment to another. When was the change done? Who made the change? Who approved the changed? Is there a separation of duties in place? What’s awesome about IaC and DevOps is that you can tell them that you closed all doors and the only way to promote changes is through code. Plus, it’s automated. No human intervention needed.

IaC Will Put a Smile on Your Face

If you decide to go with IaC, you’ll be setting the foundation for a consistent and automated delivery pipeline for your applications. Your entire team will always know how the application is architected at the infrastructure level because now they’ll have a standard language. Anyone can make suggestions or apply fixes. If you’d like to restrict who’s pushing the changes, you can protect the master branch and limit the merge only to the Ops team, for example.

Developers can play around with similar infrastructure-as-test or prod before pushing application code changes. You can apply some proven practices from the development world to Ops. And there’s one last important aspect: you’ll provision and configure your infrastructure in the most elegant way possible through code. So it’s time to give some formality to those valuable scripts you always use and start implementing IaC.

Why Is Jenkins Good for Dev Teams?

2017-11-22T00:00:00+01:00

Some time ago, a new product demanded a lot of attention and manpower. Many of my fellow team members were taken off our project and diverted to this new one. That left us pretty shorthanded. In my primary role as developer, things were looking to get quite interesting. I didn’t just have to worry about getting my changes ready; I also had to make sure those changes were in a production environment. I didn’t have any DevOps knowledge at the time (it didn’t even exist then, in 2008 or so), and I also thought that the project would end quickly. Silly me. My way of promoting our team’s changes was using a basic flow: build, copy and paste. And as I’m sure you know, many things can go wrong when we add any human intervention to the process.

We Learned the Wrong Way

There’s a funny anecdote from the days of this project, and those of us who were involved always joke about it. Here’s the story. We only had one operations guy when the company started. The number of applications kept increasing, and he couldn’t give our team’s application the attention it deserved. We decided to help him out by documenting the manual process we had for deployments (yes, again, manually), and it worked very well for the dev team. But we discovered that it was because we knew better than anyone how the application worked.

A new operations guy joined, and we told him we already had some documentation for our deployments and he just needed to follow the recipe. One day, there was a need to do an emergency deployment, and only the Ops team was still in the office (so convenient). Deployment started. Everything was going well...until they started receiving emails, SMSs, and calls reporting that the system was down. The OPs team, in accordance with the protocol for problems like this, called the lead developer for help.

When the lead developer arrived, he started reading all the error messages and discovered the problem. He fixed it right away, and errors stopped coming. What went wrong? Ops guy skipped a very important step in the recipe. Even though we had the most elaborate document explaining how to do deployments, we failed because we introduced human intervention to the process. But we knew there was something better: we had heard about Jenkins.

Jenkins is an open-source platform that started in 2004, and it’s maintained mainly by CloudBees. (The GitHub repo is located at https://github.com/jenkinsci/jenkins) It’s an automation server that’s most commonly used with builds, but it can also automate a wide variety of tasks. It supports all common version control tools like Git, and the functionality can be extended with plugins. You can configure it to be automatically triggered by a push in Git or schedule it via a cron-like mechanism.

It’s the Easiest Way to Start

We already had some automation scripts for deployments. The part we were missing was something to keep running them in the proper order. With Jenkins, it doesn’t matter what OS or language you’re using. It runs in Linux and Windows, and you can plug in any tool you need. Jenkins is written in Java, so the only prerequisite is to have Java installed. (You can download the Jenkins installer here: https://jenkins.io/download/)

You can start with a very simple job, which is a task or step in your process. You can start off with only one job, but the recommendation is to have multiple jobs (e.g., one for each script) so you can identify problems very easily and avoid having to start over again every time something fails in the process.

The very first thing you need to do to start taking full advantage to Jenkins is to document the set of steps you perform every time you need to deploy something. List the servers that you have for each environment, your source code repositories, and all the dependencies for your application to work. By doing this, you’ll identify what tools are going to be needed on the Jenkins server, what accesses or permissions are needed for your jobs, etc.

You Can Do Almost Everything

After installing Jenkins on your dedicated server, you can start installing everything you need. The Jenkins installation can be architected to have a master that can be used to authenticate users, manage security and permissions, and create the folder structure that you’ll need for your different application jobs. Because you might have different types of technologies, you’re better off having one Jenkins agent connected to the master for every line of business or every different stack (e.g., Windows for .NET or Linux for Java).

In order to have a Jenkins agent, you simply need to have Jenkins installed on the server and then register it to the master. The agent must have all the dependencies needed to execute the jobs you’ll choose for it (e.g., Git, Java, etc.). By doing this type of configuration, you enable your team to automate applications that have different versions of a stack (e.g., Java 7 and 8). When this configuration is ready, you’re able to choose where the Jenkins job will run.

Be aware that, as of the time of writing this article, there are 1,403 plugins that you can install on your Jenkins server. The Jenkins installation comes only with the essentials: a minimum set of plugins. But you’ll need to install extra plugins if you want to do things like notify the team via email or Slack about the status of any job, build .NET applications, use your cloud provider, and much more. (You can find all plugins you’ll need here: https://plugins.jenkins.io/) There are so many at your disposal that, when I need to automate anything, I’ll joke that “there must be already a Jenkins plugin for this.”

Improve Time to Deploy to Any Environment

It’s every developer’s dream to push code and get it to production without any suffering. Deploying to any environment, mostly to production, should be a boring activity. It takes time to get there, but it’s possible. When you understand that our main goal as developers is to provide value to the company, you’ll start focusing on what’s adding that value: the code. We all have witnessed and lived the pain of integrating everyone’s changes into code and pushing them to production. It can be avoided.

We live in a world of constant change where, if you don’t adapt, you die. Improving the speed and the frequency of pushing new changes to your users is a major determinant for success. Say you just pushed something that’s affecting your users. How fast can you react to that? Do you have an easy, fast, and reliable way of reverting or pushing a new change? Jenkins can help you respond favorably to these questions.

Of course, Jenkins by itself won’t solve all your problems. For example, you’ll always need a set of automated tests—like unit tests, integration tests, and smoke tests—to assure that you don’t expose your users to bugs..

Consistency and Repetition

One of the best ways to avoid downtime in production is to avoid pushing new changes. Many things can go wrong with servers. There can be CPU spikes, memory leaks, storage capacity—all sorts of things. But nothing compares to getting live new features or fixes. When you have a consistent and repeatable way of doing things, you minimize the risk.

With Jenkins, you can script all your deployment pipelines by using theirs instead. The Jenkins pipeline is a set of plugins that help you to have your delivery pipeline as code. It enables you to keep everything scripted and easy to review when something new is needed. It also gives you the confidence that comes with knowing that it doesn’t matter if someone made a change to a Jenkins job; you’ll always get the latest version from your source control. Almost the same way you treat code, right? This will be the first step to start practicing continuous delivery.

There’s No Excuse Not to Automate Deployments

Once you have an automated, consistent, repeatable, and reliable way of changing things in code, you and your team can focus on what’s really important: providing value. Jenkins is only the tool you’ll need to have a graphic representation of your deployment workflow, so you can visualize how your team pushes updates to end users. Check out Jenkins and start coding your delivery pipeline, too. You don’t want to keep banging your head against the wall when something goes wrong in production. When we humans know that we’re causing damage, we tend to freak out and lose common sense. On the other hand, machines will keep doing what you told them to do. That’s where Jenkins is helpful.

My First Bot with Microsoft Bot Framework in .NET and Xamarin

2017-08-01T00:00:00+02:00

As developers we always try to find the easiest way to solve problems. This time, I'm glad to introduce you an easy way to create your own Bot in just minutes, from your local computer to the cloud. But that's not enough, we will take one step further and connect our Bot with Xamarin, we'll see how easy is to get started.

So, before we start we need to download a few things:

0. Requirements

Download the template for Visual Studio here. By having this template you'll avoid having to install the Nuget Packages and some initial code.
To install the template, you need to copy the zip file (do not extract it) to your Visual Studio Templates folder, in my case is: C:\Users\Christian\Documents\Visual Studio 2017\Templates\ProjectTemplates\Visual C# ... Just replace "Christian" with your user.
Download and Install the Bot Framework Emulator here
Azure account to publish the bot (it could be anywhere, but it's easier here)

1. Create the Bot

Let's create the bot by following this steps:

Open VS and inside Visual C# you'll see Bot Application project (if you moved the zip file with VS opened, you need to close and open again), click there and then OK
Create a new folder in the project called Orders or choose a different name, it doesn't matter
Inside this new folder, create a new class called ComputerOrder, this is where we will define the questions of the Bot and the options. In this case we're choosing one easy way to create a guided conversation by using a FormFlow dialog, it is like we were creating a form by using properties in the class, it supports some basic types and in our case today we'll use only the Enumeration type, you can find more information in the docs here.
Add the [Serializable] attribute to the ComputerOrder class (if not the message won't be serialized when sending it to the API)

[Serializable]
public class ComputerOrder
{

}

Let's create our first question by creating an enum like this (create it in the same namespace or in a separate file, just make sure you reference it):

public enum Memory
{
    _4GB,
    _8GB,
    _16GB
}

In the ComputerOrder class add a new property called MemoryOption of the type Memory (the enum you just created). The name of the class should be in CamelCase in order to have a more readable question, in this case the question will be Please choose a Memory Option.

public Memory? MemoryOption { get; set; }

Now, one important aspect of a bot is that it needs to be presented with a message, to do this add it inside the BuildForm method of the ComputerOrder class. It should look like this now:

[Serializable]
public class ComputerOrder
{
    public Memory? MemoryOption { get; set; }

    public static IForm<ComputerOrder> BuildForm()
    {
        return new FormBuilder<ComputerOrder>()
            .Message("Welcome to the Computer Sale Bot!")
            .Message("With my help you can build your custom computer, just choose an option or type the number that you wish.")
            .Build();
    }
}

Go back to the controllers in the project and in the MessagesController you need to add the references for the FormFlow dialog, Bot Connector and the class that includes all questions:

using Microsoft.Bot.Builder.FormFlow;
using Microsoft.Bot.Connector;
using ComputerSaleBot.Orders;

Inside of the controller, create the method that will generate the dialogs for the bot:

internal static IDialog<ComputerOrder> MakeRootDialog()
{
    return Chain.From(() => FormDialog.FromForm(ComputerOrder.BuildForm));
}

Inside the Post action of the controller, you'll find something like this:

await Conversation.SendAsync(activity, () => new Dialogs.RootDialog());

You need to replace it with the following:

await Conversation.SendAsync(activity, MakeRootDialog);

Test your bot by launching the website locally (as you usually launch local website apps)
Open the Bot Framework Emulator and paste the URL that VS just launched and add the controller route /api/messages, something like http://localhost:3979/api/messages

In a real world, just one question is not enough, so you can keep adding more enums and properties to the class and the Bot will have more questions to ask to the user.

2. Publish the Bot

It doesn't matter if you didn't add more "questions", you're ready to publish the bot now. I'm going to use Azure to ease this task and this time I'll publish directly from Visual Studio but this is just to get started quickly, I don't recommend keeping it this way, instead we should configure an automated pipeline using tools like Jenkins or Team Services. So, for now you can follow these steps:

Right click in the Web project and click "Publish"
Choose "Microsoft Azure App Service" and "Create New"
Fill in the blanks (API App Name, Subscription, Resource Group and App Service Plan)
When choosing an App Service Plan you can create a new one and choose the Free option (I did it)
Click Create and wait some time, when finished VS will launch the API in the browser

3. Register the Bot

You need to register your Bot so you can easily add more integrations and focus the Bot only. So for this you just need to go to: https://bots.botframework.com, sign in with your account and then:

Click the "Create a bot" button
It will launch a new page, click the "Register" button
Fill in the blanks all the details of your Bot profile
In the configuration section, you need to put the URL of the bot endpoint (the one we published in Azure) using HTTPS, for example: https://computersalebot20170731064536.azurewebsites.net/api/messages
Click in the "Create Microsoft App ID and password" button to generate the keys, copy this generated values temporary because we'll use them to connect our Xamarin app with the Bot.
Agree to the terms of use (obviously)
Click "Register"
Now, this is very important, in order for this to work you need to deploy the Webapi again but with the generated keys you just copied when registering the Bot. To do this go to the Web.config file in the project and fill all the blanks in the appSettings section: BotId, MicrosoftAppId and MicrosoftAppPassword.
Publish again and you're done
You can go to the page where you registered the bot and test it, it will launch the same simulator you launched locally but in the browser, do this just to make sure the Bot it's well configured

4. Integrate with Xamarin

The way we're going to integrate the Bot with our Xamarin App is by using the Direct Line channel, this is a REST API that uses standard secret/token authentication, if you're interesting in knowing more go the docs here. This time, I'll give you an app where you just need to change the secret/token, you can go deeper by reading the code and extending it (I might write an article later). The idea here is to get your hands dirty as soon as possible and demonstrate how easy is to get started. So, follow these steps:

Go to your Bot page in https://dev.botframework.com/bots
Click in the name of the bot you just created
Under the "Add a channel" section, click in the Direct Line icon

Click in the "+ Add new site" button and put a name and click "Done"
The generated keys are the ones we're going to use to connect the Xamarin app with the Bot
Click in the "Show" link and copy the value
Click in the "Done" button at the left-bottom of the page
It's time to download the sample code, for this go here and clone/download it
Go to the MainPageViewModel.cs file and where it says "botSecret" (line #88) replace the text "SecretKey" with the one you just copied when creating the Direct Line channel
And that's it, launch the simulator in your computer (or deploy directly to your device) and start talking with your Bot

This is what you should see when the app is running ...

Reference Links

Bot source code: https://github.com/christianhxc/BotVentaDeComputadoras
Xamarin app source code: https://github.com/christianhxc/XamarinFormsBot
Slides (Spanish): https://www.slideshare.net/christianhxc/mi-primer-bot-bot-framework-xamarin
Samples: https://github.com/Microsoft/BotBuilder-Samples
Recorded workshop (Spanish): https://www.youtube.com/watch?v=v6fgn8GB5KY

This is my own version, the original content is from: @HJaimesDev and @JorgeCupi

I gave this talk as a workshop for the Xamarin Fest in Guatemala City, you can take a look to another great talk on how to integrate Custom Vision with Xamarin by Guillermo Zepeda @cloudzepeda in this [link]

Summary

I hope you have enjoyed this article and experienced by yourself how easy is to get started with the Bot Framework and not just that, connect it with your Xamarin App. This is just a jumpstart, I didn't do a deep dive on each section because the intention was to get you started fast (yes, I've said that so many times lol). If you'd like to do everything that I did here (with you hopefully) you can follow me in the Youtube video I put in the reference links, that's the video of my talk I mentioned, so even if you don't speak Spanish you can see how all steps connect in case you missed something or I didn't explain myself well enough ;)

What changes with Docker for Developers?

2017-07-24T00:00:00+02:00

Last week I was having a talk with a new friend (@rdiazconcha) about Docker and one of the virtual table topics was: “What are the implications of adopting Docker for developers? Do we need to keep doing the same things as we’ve been doing it in the last years (or months) or should we start considering other things too?” Well, in this article I’ll give my perspective and what changed for me and the good group of developers around me.

But first, I don’t think this is a complete list of topics that you need to consider (I’m assuming that you already have some basic knowledge about Docker) in fact I think some one of them could need more than one article to discuss, so you can add more in the comments.

Is not just about having the same environments

We always read that the common phrase “it works in my machine” it’s dead with Docker, but let me tell you that this isn’t always true, but why? Because we need to be aware that there are many other components that are needed for our app to work, say for example: redis, elasticsearch, mysql, memcached, etc.

We often have different environments too, where all tests need to be green (everybody test their code, right?), the most common ones are: local, dev, qa, staging and production (to name a few). And the idea of having Docker is that we can “build once run anywhere” so we need to have in mind that the app won’t run only on my machine but also in any other team member (yes, treat servers like a team member, lol). The usual flow is like this: I create my feature branch, make the changes, test the changes, push the code, make a merge request, my team lead approves it, it gets merged to master, then the CI tool is triggered to build the image and it’s deployed for the first time with the CD tool to the dev environment, after this we can just deploy the same image to the rest of environments … yes, when all tests passed.

Sounds good and easy right? Not always, sometimes there are inconsistencies that causes so much pain. For example, one of the most common problems we had is that the endpoints for the resources weren’t always in the same way, for redis some used the server IP (that sometimes was changed because whatever) other used the domain name but then we realized that we needed to have a standard there so we changed the domains again and we always forgot to change something because eveng though the tests were green, when we shut down an old server we started noticing that some things were still pointing there. So, even us having a flow similar to the one described above, we always had some problems between environments.

We never knew exactly what was happening (everything worked in our machines, remember?), but weren’t taking full advantage of something very useful (we’re in that): the debugger. Sometimes we have to go the limit of saying: “We don’t have the same environment as in production so we will need to debug there” and we end up having ugly debug messages that sometimes are exposed to our customers. No, no, no, we can’t keep working in that way, we do have a better way now because t’s possible to even debug our code inside a container, for example, if you’re a .NET developer, Visual Studio 2017 has this already integrated and for existing projects you just need to “Add Docker Support” to your projects and it will build all what’s necessary to run and debug locally your app, it will use some base structure and according to the version of your code it will create a proper Dockerfile, but hey, as Scott Hanselman said in the VS launch: “You can use this even to learn Docker” (you can do this with Java too: [here].

Testing my app can be easily automated

We can’t just test (and/or debug) locally, if we have full controll of the environment we’ll always find the hack for our app to work and as our app grows, more and more tests will be needed so this means that more and more time is going to be needed and this might start slow you down.

Yesterday, I found out that SQL server team use Docker to automate testing (link) and also Yelp says that they run millions of tests every day (link), that’s amazing! We have two examples of big companies on how we can take advantage of containers to test our app to the limit to even run those type of tests locally. The approach you’ve been using to test your app needs to change because like the SQL Server team, now you have the ability to run the database in a desired state as the first step, run the tests with several containers at the same time and wait some time for the results (you’ll need more resources but we leverage that to the cloud, right?).

Imagine that you can run all you test suite after you’ve deployed the app in the dev (or other one) environment with this approach, you’ll gain confidence the more you practice and also you’ll reduce not just lead time but risks when deploying something new to production.

Storage is ephemeral

You need to know that all changes that happen in a container when it’s running, might get lost forever because when you start again it will create a new container (unless you relaunch the same container). At the moment it gets stopped there’s no guarantee that your files will stay there, these can be logs, missing files when deploying and uploaded manually, new uploaded files when using the app, missing components that were installed, etc. I definetly recommend you reading this article [here] to know more about how this works.

Why this is important? Like I said before, when you have the control of the environment you will always find the way to make your app work and as humans (specially with deadlines) we tend to forgot what we did to fix it and that’s why our app stops working in other environments. Also, this means that you can’t depend on having the state inside a container, the container should always be stateless.

Let me give you an example, let’s say a user uploads their profile pic with your app and you store that pic in the container, the user is happy because the profile pic is being displayed but then something happens with the container and it’s terminated. Boom, the profile pic is not there anymore, at scale it’s annoying to go and re start a stopped container. In this specific case you should instead move the image to a permanent storage like S3, Azure Blob or whatever you prefer.

Environment variables

Now, this is one of the most important things where you’ll see significant gains if you applied this correctly. This was for me the “aha” moment for “run anywhere” because by implementing environment variables you just need to “inject” the values you need for the environment the container will run. Let’s use the database endpoint as an example, by using an environment variable like DB_HOST I can start making the changes needed to my app to take the endpoint from the environment, then I need to add that variable to my computer … right? … ahmm you can do that but you won’t be changing things that much, just making everything more complex because what you did in your computer you’ll need to do it in the rest of the environments.

So how can this be achieved? You need to make use of the environment variables when running the container, this way you’ll be injecting the desired values. This recommendation is based on the 12 Factor App guide and specifically for the config part, you can find more info [here] and I also wrote a post about using docker env vars with .NET Core [here].

Logs, logs, logs

Data might get lost, remember? So how do we know what happened after a container crashed? Logs. This is crucial in production so you need to be prepared by making sure you have this across all environments (even in your machine, yes, to verify it’s working). You need to know that by default all logs goes to the standard output (/dev/stdout) and standard error (/dev/stderr), take a look for example to an nginx container, the /var/log/nginx/access.log is a symbolic link to /dev/stdout. This way you can use the “docker logs” command to get the information without login in to the container or you can make use of other logging drivers and have a centralized logging server where all containers will push logs, as the 12 Factor App says “treat logs as event streams” [here]. But you have to choose just one, if you decide to go with the logging driver approach, you are no longer able to get logs with the “docker logs” command and I should mention that you can only have just one logging driver, at least at the moment (that sucks).

How this affects you as developer? You need to know that if you send logs to another path different that stdout or stderr (this is translated to all console writes your app can make) your logs might get lost (unless you send them to a shared volume, in this case the logs will be kept in the host, but there’s still a chance to lose them). So keep this in mind.

Port binding

You should find a way that this is not an impediment for your app, especially if you would like to scale out containers in the same host. The application inside your container will expose a port but the host is not going to necessary expose the same port (there is a default range but as developer you don’t need to know this, but if you’re curious go [here].

And this is also a characteristic of a 12 Factor App, you can find more info [here]. But, why is this important? By having this option, you don’t need to worry where the container will be living, in your local computer could be port 8080 but in development could be port 32888. You might start thinking that his will make things more complex to you because you need to know in which port the container is running, this gets complicated when you run more than one container to have a cluster be fault tolerant.

You also have some options to work with this and here they are: 1. You can use docker compose and link the containers, more info [here] 2. Use a load balancer, this will expose obviously a port that will always be the same and that you will use to call a container, the difference is that this will proxy the calls to the container’s port in the host that are registered to it, there are several ways to do this and here are some. And all the orchestrators have this feature in some way, but as a developer you don’t necessary need to know how to do implement it. 3. Ignore this section and continue working with a fixed port number, the downside here is that you will only be able to have one container per host

Tooling to have my local environment

Now, you don’t need to learn a bunch of tools to start but there are some things that you should consider when beginning this journey:

Docker installed, kind of obvious but anyway, more info [here]
In the case of Windows, you might need to share the drive where you have the code so you can debug or use volumes in that drive
In some cases, make sure you’re giving enough resources to the docker daemon, i.e. to run SQL Server containers you need to have at least 3.5 Gb of memory reserved
A script to start the container app and all its dependencies (i.e. local redis), this can be another topic table, just take note that you’ll need to use compose to ease the work. This way when a new developer comes in he/she will just need to pull the code, read the README.md, follow some instructions maybe, run docker compose and start hacking. This sounds easy, but it can get complicated to get to this point, but it will worth it.

Create the Dockerfile with the operations team

You need to have this scripted, it will make everyone’s life easier, trust me. You can launch a container with a base image (even FROM scratch), install what you need, copy your app and then test, after you are happy with everything you just commit the container, create a new image tag, push the image to the repo and that’s all. In other words, you could treat the container like a virtual machine, don’t do that, please.

Instead, make use of the Dockerfile, you can start with what you know you’re going to need for your app and then ask to the operations team for a review, they might have something to add or remove and they will know what the container will have installed. Same thing happens if other person is building the image, you would like to know what’s inside, right? Maybe something that it’s in there is what is making your app to crash or running slowly.

A few things to consider here: avoid using a Dockerfile for build locally, other for development and another one for production, it will get a mess, instead use the multi-stage builds, more info [here]. Another thing is the image size, remember that the idea is to have only what you need to run your app, nothing else, so ideally you’ll try to have your image size as low as possible, for example you can achieve this by using base images like Alpine or CoreOS (create your own with Linux Kit?) instead of using the Ubuntu one. Why is this important? Well, when you scale out and a new server is launched, the image needs to be pulled and if the image is serveral GBs of size, it will take some time before you can launch your first container, you users might not even notice it but it can affect someone.

Summary

I hope you have a good idea now what changes you might need to implement to your existing applications. But do not try to make all changes at once, start little by little, you will always have some benefits. Right from the very beginning you’ll be able to have a decent level of consistency between deployments, you might not be able to get the full advantages of Docker but at least give the first step: containerized your app :)

MSSQL Linux with Docker

2017-02-27T00:00:00+01:00

On February 18th I had the opportunity to give a talk about SQL Server running inside a Docker container in the SQL Saturday Guatemala event, more details here. I thought no one was going to attend due to at the same time Jason Horner (Microsoft Certified Master of SQL Server) was giving a talk about database performance tricks, but to my surprise some people were interested in the talk (yes, mostly developers, one DBA and one auditor), nice!.

So, I want to share the agenda with some useful reference links and I'm going to extend in the most important slide of the PPT, here it is:

Agenda

The agenda was pretty straight forward and short:

Docker
MSSQL + Linux
Use Cases
Demo

1. Docker

I started by giving a quick tour about what Docker is and why it's, the architecture behind and some adoption details in the recent years. Here's the list of reference links:

What is Docker? https://www.docker.com/what-docker
Docker Overview https://docs.docker.com/engine/understanding-docker/
Docker Adoption https://www.datadoghq.com/docker-adoption/

2. MSSQL Linux + Docker

Because there was another talk about MSSQL running on Linux I didn't focus on that and went straight to the instructions and considerations when running the container:

Run the SQL Server Docker image on Linux, Mac, or Windows https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-setup-docker
MSSQL for Linux Containers https://hub.docker.com/r/microsoft/mssql-server-linux/
MSSQL for Windows Containers https://hub.docker.com/r/microsoft/mssql-server-windows/
MSSQL Express for Windows Containers https://hub.docker.com/r/microsoft/mssql-server-windows-express/

Now, you need to consider the following when running the container (otherwise the container won't run):

Pull the image from the docker registry (docker pull)
Your computer needs to have at least 3.25 Gb of RAM (to be honest, I didn't like this but I still need to test if it runs with less memory assigned)
You need to accept the license agreement with the env var ACCEPT_EULA
You need to provide a strong password with the env var SA_PASSWORD
Beware that you're running the full version of MSSQL so you have a trial of 150 days
You can have persist data by attaching a volume, this is because by default the data is ephemeral and you will lose it once the container is stopped. You need to attach a volume to the "/var/opt/mssql" path

Here's the docker run command:

docker run -e 'ACCEPT_EULA=Y' e 'SA_PASSWORD=YourStrong!Passw0rd' -p 1433:1433 -d microsoft/mssql-server-linux:ctp1 linux:ctp1-2

Here's the docker run command with a volume attached:

docker run -e 'ACCEPT_EULA=Y' e 'SA_PASSWORD=YourStrong!Passw0rd' -p 1433:1433 -v <host directory>:/var/opt/mssql -d microsoft/mssql-server-linux:ctp1 linux:ctp1-2

If you're running on Windows, you can replace with for example C:/mssql

3. Use Cases

I gave some use cases that came to my mind, things that I will use to run MSSQL in a container, so if you have any other use case please share it in the comments, I'll glad to hear from the community:

Prototyping, things that you need to validate fast and don't want to have a full server for the DB
Local environment for developers, a developer can start working in the code from day 1 without having to worry about the DB version for example
Test/Practice environment for DBAs, there are times that we would like to test some scripts that might bring down the server or cause inconsistency
Migration script testing, we might want to practice the migration from one version to a newer version
Practice backups restore, we can configure a periodic practice to restore backups, we need to make sure that when we need them, they work
Separate processing layer from storage layer, you might want to have all data in the cloud or a SAN and not in your computer, this also gives us the ability to copy data from another environment or partner (although we have the risk of corrupted data)
Microservice Arquitecture, this is a good fit when you want to have a DB per service (it can also help you to migrate to a more lightweight storage like a no-sql db)

4. Demo

I followed the "Getting Started" page for developers, but I didn't have enough time to do everything, so here are the activities we can do to practice:

Configure environment
Create an application in C# connecting to a MSSQL
Change the app using Columnstore for a 100x faster app
mssql extension in Visual Studio Code

Reference link: https://www.microsoft.com/en-us/sql-server/developer-get-started/csharp-ubuntu

SQL Saturday Guatemala

Here are some photos of that day :)

Summary

As you saw, it's really easy to get started with MSSQL for Linux using a Docker container. I present to you some use cases that came to my mind, it might exist more and others would like to discuss some of them and that's why I want to make clear that my idea is not to define a rule, I just wanted to give some examples of how I would use this combination. I'm not sure yet if I would recommend use MSSQL in a Docker container for production environments, I see it more as a fast way to experiment.

Oh yes, here's the PPT of the talk for that day (it's in Spanish): Download here

Using Docker env vars in .NET Core

2017-02-06T00:00:00+01:00

One really good thing about packing your apps with Docker is that you're building once and deploying many times in many different places (environments), because of this it's a good idea to make your app configurable at the environment level (not just files). This means that you should avoid having config files in the container that can be different in each environment you run the container, "An app’s config is everything that is likely to vary between deploys" (The 12 Factor App).

But why environment variables?

The idea comes from the 12 factor app methodology (https://12factor.net/config), where it says that we should prefer using this type of configuration because having files can compromise sensitive information (security in Docker is not that mature yet because you can easily inspect the container and see all the environment variables, but that's an interesting talk for another time).

I'm not going to repeat what you already read in the previous link but I want to focus in the following: "In a twelve-factor app, env vars are granular controls, each fully orthogonal to other env vars. They are never grouped together as 'environments', but instead are independently managed for each deploy. This is a model that scales up smoothly as the app naturally expands into more deploys over its lifetime." (The 12 Factor App)

So, the idea is that we should always have our app prepared to behave accordingly in each environment, we should not have to re-compile our code just because we need to change some configuration, imagine doing that in all of the software you use in your computer.

That's good, but how can this be done in .NET Core?

I'm glad you ask. By this moment I hope you are convinced that env vars are a good option and lucky for us this is how Docker works, we just need to know how to achieve this behavior in .NET Core. For this, there's also good documentation here but in this post I will focus only on how to read env vars that comes from Docker.

.NET Core has a really good integration with env vars, actually the default template for a Web API use it as default, so we just need to make a few changes to our code in the Startup.cs class, create a new POCO class and use it in our Controllers by injecting the configuration we need.

Changes in the app

While I list the changes we need to make in the app, I'm going to give some explanation in each step. I have uploaded the code sample in Github, so in I couldn't explain myself enough you can see the source code here. Here's the list of steps:

Make sure that in the Startup.cs class of your Web API project you have the line to add the env vars. This should be in the constructor of the class where the builder of the configuration is being defined. The builder definition should include the call "AddEnvironmentVariables()". Something like this:

var builder = new ConfigurationBuilder()
  .SetBasePath(env.ContentRootPath)
  ... other method calls
  .AddEnvironmentVariables();

You need to create a POCO class where you define all the properties that you're interested to read from the environment. In my case I want to treat the profile name as an env var, so I created a class called EnvironmentConfig.cs and I included a public string property called "ProfileName" (his name should match with the env var, the framework makes reflection to fill the properties with env vars and because this is being done at startup, it won't affect performance when serving requests, this is also because in the container you will have only the environment variables you define when running the container).

public class EnvironmentConfig
{
     public string ProfileName { get; set; }
}

Add the new configuration class in the Startup.cs services definition. This means that in the method called "ConfigureServices" you should include this line:

services.Configure<EnvironmentConfig>(Configuration);

Finally, we just use it in the Controller (or wherever you want) by injecting the configuration in the constructor of the class. I recommend having a local var in the class, then you need to specify in the constructor the injection of the configuration class we just defined, but it should be done in the .NET Core way by using the 'IOptions' interface. In my case I want to include it in the ValuesController.cs so my constructor looks like this:

private readonly EnvironmentConfig _configuration;

public ValuesController(IOptions<EnvironmentConfig> configuration)
{
    _configuration = configuration.Value;
}

In case you don't see it, one of the cool things by doing it in this way is that if you don't specify an env var called "ProfileName" your app won't crash (unless you want/need to) because when you read the property of the configuration class it will be empty.

And that's all, if you want to deep dive in how configuration can be modified/used please read the following article here

How to test and debug this with VS?

I assume you already have added Docker support to your solution, if not take a look to this article. In your solution, open the docker-compose.override.yml file inside docker-compose section and in the environment list the ProfileName var, take a look:

Go and debug in Docker mode and the app should read that value from the container environment.

But what about Docker?

Yes, I know I almost forgot about this. But you know what? This is really simple, I'm going to put here the way I run my container so you can see how to define the env var that my .NET Core app will read.

docker run -e ProfileName='Christian' -d -p 9000:80 christianhxc/hellodocker:env

Now I need to go to: http://localhost:9000/api/values and boom! I see my name there. Of course, you don't need to specify all env vars in the run command, you can make use of docker compose to group them, but remember: don't include the any config file with the source code in version control.

You don't need to follow this guide to test it, just run the above command with different values :)

Hello World .NET Core and Docker with VS

2017-01-28T00:00:00+01:00

Today I want to share with you how you can create a Hello World API with .NET Core but running inside of a Docker container and not just that, we will be able to even debug the API with Visual Studio (VS) and then I'll explain how easy is to push an image that is ready for a production environment, so let's get started.

What I need to have first?

In order for this demo to work you need to have the following installed in your computer (I'm using Windows 10):

Visuatl Studio 2017 RC, you can download it here
Docker for Windows, you can download it here
If you're using more than one drive in your computer and you installed docker in a different drive than your project files, you need to share that drive in Docker, you can do it following this guide here. Read this part again, you might not be able to debug the API in VS if you don't do this.

Once you have everything installed/configured, you can continue.

Create a Hello World API

We're not just going to create a simple API (you might know how to do this already) but I'm going to focus on how you can integrate Docker to your project.

Open your VS as an Administrator (right click and "Run as Administrator")
Go to File -> New -> Project
Choose Templates -> Visual C# -> .NET Core and then "ASP.NET Core Web Application (.NET Core)"
Put a name to the project (in my case "HelloDocker"), choose the location of files and then click "OK"

After this, you should be seeing another screen where you need to enable docker support (you can do it later too).

Choose "Web API" under "ASP.NET Core Templates 1.1"
Check the "Enable Docker Support" box and click "OK"

Let's wait a little bit and you should be seeing something like this:

What's new here? If you notice you now have a new section in the solution called "docker-compose" and inside your project you have a "Dockerfile". I'm not going to explain why you have those files and what they do (for more details go here). Now, let's see the magic when debugging the API.

Open the ValuesController.cs file (In the web project, Controllers/ValuesController.cs)
Put a break point in the line where return a list of values (line #16)
In the solution explorer, go to "docker-compose", right click and then Debug -> Start new instance. If you do it in the project, it will launch the IIS Express and you don't want to do that right now. You should start seeing in the output how the image is being built by copying all the binaries inside the container.
When it finishes, it will launch a new window in your browser
In the same window, go to the /api/values URL and VS will catch that call in the break point we put

Now, this where you can start screaming excited about what just happened (I did). If you don't get it, let me explain what just happened: VS built a container with the code you have in the project, then it launched the container and attached it to your VS so you can debug it. This means that you're not just testing locally but you're testing in the same way that the API will run in the production server. That's really awesome, I loved it.

Close your mouth now please, we need to do some things to finish the demo :) ... In the same line that you put the breakpoint, let's add a new value, we're gonna push this to the Docker Hub.

What about deployment to Production server?

I know, that was my first question, so let's open the terminal window (I use Powershell) and let's do it, but first, we need to build the API in release mode.

In VS, change the configuration from Debug to Release mode and build the solution again. Why? It will start creating the image for a production environment (you never release your APIs in Debug mode, right?) and it chooses the best container base image that Microsoft team have configured for those environments, so it's not from the same image base we used to debug our code. More info here
Open your terminal window and run "docker images"

Now, you see that I remarked the one with the "latest" tag, that's the one we need to use for production, the other one (with the tag "dev") it's the one just for development. Ok, let's push it to your docker registry.

Login to your Docker Hub account here (If you don't have one, create it please)
Now, let's create a public repository, mine is called "hellodocker"
Go back to your terminal and run the "docker images" command
Copy the IMAGE ID of the hellodocker with the latest tag and tag it with the name of your repository that you just created. In my case I need to run this "docker tag 95904eeb5775 christianhxc/hellodocker"
List the images and you should see the one you just created
Login to your docker repository by running "docker login"
Push the image, in my case I ran "docker push christianhxc/hellodocker", wait a little bit and you should see the image in Docker Hub (it might take some time to see it)

How to run the container without VS?

Just open your terminal and run the following command:

docker run -d -p 8000:80 christianhxc/hellodocker

Open your browser in http://localhost:8000/api/values and see the latest version of the API

So that's all, now you can easily create .NET Core apps with Docker support :)

JMeter 101: Load Test a Web API

2017-01-24T00:00:00+01:00

One of the most crutial things for a Web API is to know how many resources you'll need to run it, this based on how much traffic you will (or expect to) receive, the response time of your API and the payload (the content that it will retrieve). And when I mention resources, I'm talking about how big/small your server should be, the good part right now is that we have the cloud, where we can experiment in minutes and just with a few clicks. But today, I'm not going to talk about the cloud, I'm going to focus on how to prepare our local environment to run this set of tests in our machines.

What will you need?

First thing first, we will need an API to test, for this we're going to create a simple one, actually let's use the one from the VS templates. We can do that by following these steps: 1. Open your Visual Studio (2015 or higher, try get up to date) 2. In the left pannel, choose: "Visual C#" and then ".NET Core" 3. In the middle pannel, choose: "Create a new ASP.NET Core Web Application (.NET Core)" 4. Put a name to your project (I put "HelloWorld") 5. Choose a location to save the all project files 6. Click "OK"

Easy, right? Here's an image of what I see:

Now, you need to download JMeter (you'll need Java to run it) here: http://jmeter.apache.org/download_jmeter.cgi. Extract it and go to the bin folder, you'll find a file called: "ApacheJMeter.jar", run it and the UI will appear.

OK, now you're good to go, let's start having fun.

How to create my first load test?

It's really easy and we're going to do it in just a few steps, at the end I'll share with you the link fo the xml configuration of the test, let's start:

Open JMeter (I guess you already did it after donwload it in previous step)
Right click in "Test Plan" and go to Add -> Threads (Users) -> Thread Group. This is where you will configure the users that JMeter launch, lets put 10 in "Number of Threads (users)" to simulate 10 concurrent users. You can also configure how much time the test will run, check the “Scheduler” box and in this case we will put 30 in “Duration (seconds)” so the load test runs for 30 seconds.
JMeter will try to send as much load as it can, so we need to put something to assure a constant throughput to the API, do to this you need to right click in "Test Plan" and go to Add -> Timer -> Constan Throughput Timer. This is the maximum number of requests that it will send in one minute, because we already configure the test for 10 users let’s assume that it will send 10 requests per second (rps), in one minute it will send 600 requests, so put 600 in “Target throughput (in samples per minute)” box.
We need something to test, an endpoint. So let’s run the API in Visual Studio (VS) in debug mode for now and let VS launch the IIS express, it will assign a port so make sure you have it handy.
Let’s get back to JMeter and in “Thread Group” right click and go to Add -> Sampler -> HTTP Request. You can configure now the endpoint, in “Server Name or IP” just put “localhost” (this where you will put the IP of the service or the subdomain name). Specify the port number that you were assigned in the previous step, in my case it was 62029, so I put that number in “Port Number” box. You need to also specify the route of the API to test, so in “Path” it’s the place to put it, in my case I put “/api/filelog” because that was the endpoint in my IP. You can also configure parameters in the request but we will leave this example as simple as possible.
The good thing about JMeter is that you can see some output while the test is running, so let’s add a few. In “Thread Group” right click and go to Add -> Listener -> Aggregate Report. With this you’ll be able to see the number of request sent in total, the average latency in milliseconds (ms), the median of the latency, 90th percentile latency, the 95th and the 99th (don’t trust in averages please), minimum latency, maximum latency, percentage of requests with errors from total sent, the throughput (rps), data received and sent in total.
You can also add a response time graph to the test, to do this go to “Thread Group” right click and go to Add -> Listener -> Response Time Graph. So, when the load test is running you can see a graph of how the latency is behaving.
You’re done, so let’s run the test by going to the “Run” (Alt + R) menu and click in “Start”

As you can see, in eight steps you were able to create a simple load test with JMeter, so now practice adding more listeners and keep playing with the configuration, add more users to the test and see how your API behaves, you might find pretty interesting things.

Here’s the final screen of how your test plan should look:

You can download the configuration file here, have fun!

AWS Lambda - Automated Snapshots

2017-01-10T00:00:00+01:00

This is my version based on the code from the blog post from Ryan S. Brown, I recommend reading his blog before mine, you can find it here. I'm also including a recipe of how to deploy, my opinion fo why you should use the funtion the way it is and how to use the function to backup and restore your EC2 Volumes. Enjoy!

What's the difference?

I added the functionality to include the ec2 tags to the snapshot, downside is that now it makes a call to the API for each snapshot, it doesn't update them all at the same time (with just one call).

What it does?

The purpose of this tool is to automate the process of backing up important servers in AWS, the idea is to have a way that we can control what servers will need daily backups and what retention period will need. For this we're going to be using some AWS services: Lambda and EC2 Tags.

By having daily snapshots we will have the opportunity to restore an instance easily, reasons can vary but common ones are: corrupted volumes, AZ failure, etc. So, how this process works? Here's how:

There are two lambda functions, one for taking snapshots and other one to act as janitor
Both functions are configured to run on daily basis
The backup lambda look for instances that have the 'Backup' tag (it doesn't matter the value), then it takes snapshots of all volumes attached to the instance and looks for the value in the tag 'Retention' to set an expiration date for the snapshot, if there's no tag it will use a default value (30 days). When creating the snapshot it adds the expiration date, so for example the function runs on November 1st and it has a renetion period of 7 days, it will add an expiration date of November 8th.
The janitor lambda will look for snapshots that has to delete, meaning that it will take the date when it's running and will search for snapshots that have the tag 'DeleteOn' with the date.

Why snapshots and not AMIs?

There are a couple of reasons for this decision:

AMIs - AMIs are cool because in order to restore we just launch a new instance with few clicks - Downside of this is that there are times that we will need to keep the IP, IAM Role assigned, Tags and SG ... and when an emergency comes taking care of all of those aspects is too risky and complex - There's no guarantee at 100% that when we take an AMI from a running instance that when restoring the AMI will keep working, because it was still running there might be some files corrupted (that's why the recommendation is to stop the instance first)

Snapshots - Are incremental, only the first time will take time to create it - It's easy to restore (read below for details)

For more details about them, please check the official docs: AMIs and Snapshots.

How to upload/configure the code to AWS Lambda?

For both functions, we'll need to do the following:

Create Lambda IAM Role with the permissions described in the policy from this repo (iam/policy.json)
Go to AWS Lambda and click on "Create a lambda function" button
Select "Blank Function" blueprint
Configure the trigger, click on the left button and choose: "CloudWatch Events - Schedule"
Put a rule name if it's new or choose from an existing one
Choose a schedule expression, in this case should be one day
Check the "Enable trigger" checkbox
Click "Next"
Configure the lambda function:
Put the name of the function
Choose "Python 2.7" as the Runtime
In the box below, paste the code of the lambda function in the src folder of this repo
For Role, select the option "Choose an existing role"
Choose an existing role (the one created in step 0)
Leave evyrthing else as default
Click Next

Ref: http://docs.aws.amazon.com/lambda/latest/dg/get-started-create-function.html

How to restore an instance from a snapshot?

Follow this instructions to restore an instance with previous snapshot:

Go to EC2
Identify the instance and the volume(s) with problems, take note of the volume id and the AZ of the instance
Go to Snapshots and search for snapshots from volume id with problems
Choose the most recent snapshot and create a volume in the same AZ from the instance
While the volume(s) is/are being created, stop the instance with problems
Detach the volume(s) from the instance, take note of the "Block Device" name (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-detaching-volume.html)
When the volume(s) is/are ready, attach the volume to the instance with the same "Block Device" name (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-attaching-volume.html)
Start the instance

Questions or suggestions?

Easy, just send me an email or open an issue here

AWS Lambda - Redshift Copy

2016-12-21T00:00:00+01:00

This is a short post about how to interact with Redshift inside a lambda function using python, it will also introduce the way we include external libraries in AWS Lambda. The main reason for writting this is that I had the need of automate the copy command to Redshift but couldn't find much information about how to do it, so this is why I decided to share this piece of simple code.

How it works and how can I configure it?

The script is coded in a way that you can easily set the desired configurations, the initial section has a list of global variables where I have added a description of what you should add there but I think it's worth explaining how it works:

First, this lambda receives a notification from S3, then it gets the bucket and the key, based on the prefix of the file it gets the copy query from another bucket and then do some replacements to finally build the copy query that will run in the destination. Before executing the copy, it checks if the file has not been copied yet, this is to make the lambda function idempotent.

What will I need to make it work?

S3 Bucket where you will store your copy queries (it can be one) 1.1 (Optional) A prefix where you will store your copy queries, i.e. queries/
IAM Role ARN that the destination Redshift has assigned
Host, user, (port?) and password of the destination Redshift

How to upload this to AWS Lambda?

You just need to follow this simple steps: 1. Zip all the content of the src folder (including the psycopg2 folder) 2. Upload the zip file to AWS Lambda (you will be able to see the code there but not the psycopg2 library)

Why do we need to have psycopg2 included?

AWS Lambda doesn't have native support (at least yet) to execute queries into Redshift and when a library is not included in the stack, you need to include all libraries needed in the package (zip) that you upload to lambda.

How is this useful?

Well, I've used this to replicate in real time the data from one Redshift to another Redshift cluster. For example, imagine that you need to store data from Firehose, you configure your stream in a way that it will convert the data in files and it will store the files in S3, so that it can be executed the copy query that will upload the data to Redshift, but what happens if you would like to upload this data to another cluster (maybe for development purposes). Easy, you just need to configure an event trigger in the S3 bucket, and done! How? this is how: Lambda with S3

Questions or suggestions?

Easy, just send me an email or open an issue here