Building a successful Devops team

There are no devops engineers but there are devops organizations. In a devops org you don't have mere mortals, you have superheroes who have the superpower of learning fast and helping others. Devops is about efficiency, effectiveness and collaboration. It does not mean one person takes on both the dev and ops role, but instead its a team formation that takes a historically manual and slow arm of the business and automates it.

There are a lot of definitions and interpretations of Devops but the outcome is the same: faster delivery of a product that will delight the customer. Devops is not a need of technology but it is demanded from the business. If you are in a business that is slow to evolve, has no competitors and the customer base is flat, then I would get out of that business ASAP.

In this writeup we will go over the attributes of a successful devops team and how can one transform from a regular org to a devops org.

Background: How did it all start

It started very simple, 2 tier apps and everyone was happy. As products became feature rich, architecture got complex as a result the teams grew and departments were created. This was the world we were living in. Employee count grew linearly with the userbase/product. The user growth was a linear chart and capacity planning was easy. User behavior was not changing overnight and taking downtime for upgrades and deployment was acceptable. Sprint cycles were a month and then Ops and QA were given 1-2 weeks to do their work. On top of that, deployment would happen once or maybe twice a month. So everyone got into a mindset of "this is what I do and this is how it is done." Not to mention, the work itself was mostly manual, as there was a dedicated person for each task. In order to make a small change to the product you needed rally all the teams: product, XD, legal, UX, developers from different components, QE and Ops. Then eventually release it to the customers. Going through the above flow means 6 months will pass before the product actually gets updated. The core of every conversation was the "Product" and what we "Offer".

Following is what typical non-devops shops look like. There could be a huge disconnect between business, tech and the users. Cycles in each component can take weeks and, by the time the loop completes, months have passed and the users have likely evolved. By the time this team delivers, the needs they were solving for would be old. This causes a lot of frustration as everyone is always playing catch-up and no one is getting time to innovate. Users suffer because their needs are not met, they are not the focus for the product owners. Product owners are busy at looking at competition. The business is not happy as it is losing customers to the innovators who are cruising and winning over the customer base. Tech is not happy because the technology is likely outdated, they're siloed and any kind of change requires so many approvals, it's impossible to move faster than a snails pace.

Now lets focus on the innovators. They posses characteristics such as being agile/nimble, are not process heavy, the design is simple and the best of them are targeting very specific problems. They understand the needs and requirements of the customers very well. When they speak, they are all about the "customer" and the "customer need".

As you can see in the below diagram the whole organization is built around customers. Business and Tech get bundled as one product team with a shared goal to build a product that will delight its customers. There are no silos, everyone is technical, understands the business and is a customer advocate. That is a true devops organization.

Devops started as collaboration between development and operations but it is no longer confined to that. It is structured to move faster, be nimble and deliver awesome. It is not just building pipelines, enabling telemetry or automating tests. It is also about educating employees about the business, making sure they feel ownership of and are passionate about the problem they are solving for. Every person on the team has the skillset of a product owner, QA , dev and ops.

Now that we have established what devops means, lets talk about how to build a devops tech-focused team.

Ways to build Devops Organization

There are 3 ways that I have seen/tried in the past. Lets go over them briefly.

1. Top Down
2. Bottom Up
3. Tiger Team

Top Down

In this approach, we start from the top. Senior leadership leads the methodology and we bring the teams together through requiring new processes.

Pros:

• Faster to implement
• Org buy-in granted, as coming from Upper management.
• Orgs gets structured accordingly.

Cons:

• Out of your control.
• You can propose methodology but implementation will be from SVP's/CXO.

Bottom Up

You start with your department, fix one piece at a time. Once the process is established for one, start evangelizing to get buy-in from others.

Pros

• you can get started today!
• easy to figure out the mindset and mold accordingly.

Cons

• Convincing people can be challenging.
• You will need to hire talent without breaking budget.

Tiger Team

Identify the critical players from each department and form a team. I find this approach a bit short sighted, but it can accomplish a lot in a short amount of time.

Pros

• Immediate results
• Easiest to implement
• Is focused on the project not people, culture.

Cons

• Not scalable
• Alienates team members
• Puts too much load on key talent. If you lose them, it can put the project in jeopardy.

I have tried and had success with the second model "bottoms up". I specialize in technical operations and have built teams that fit the devops model for operations (appops, techops, security, network, DB engineering, SRE, SA, infra support, tools team). In many organizations these teams would be separate which makes it difficult for the engineering org to be truly agile.

Lets go over a some prerequisites to moving in the devops direction.

Question the Status Quo

Find out the details of the existing process and how it came to be that way. Make sure you have a very good understanding of every use case. What would it look like if this process were automated and worked without human intervention?

Skillset

Skillset gap is the biggest issue I have seen. If this is the case then you need to start building the skillset (train, hire, borrow). I usually start with training and induct a few developers to train on dev tools and move forward.

Tools

Make sure you are using the right tools for the jobs. I have my personal arsenal that I prefer but every team has its own sets. If you want to get familiar with some of these tools, check out Devops-toolset.

Transparency and tracking

• Create a dashboard using any project management tool. Where does your team spend most of its time? What are the sources of these tasks? Talk to the stakeholders and see if they are looking for faster response. Likely, they'll say yes. Gauge the impact on the business if you optimize it.
• Publish tracked metrics to make sure everyone is aware of the tasks, WIP, bandwidth, ETA's and priority.

Automation

Now that you have figured out where the team spends most of its time, it is time to automate those tasks or build self service tools.

Data Drive

Measure anything that moves. Remove emotions from decision making, decisions should be data driven. It applies to feature on the product, priority of a task, priority of an incident.

Measure and keep a healthy trend

Monitoring plays a very critical role. If you are not measuring then it cannot be improved. Make sure that you have the capability to measure and then you can decide weather to mark it as a KPI or not.

Standup

Have a daily standup where everyone speaks for 30 seconds about what they did yesterday, are planning to do today, and any blockers holding them back. The takeaway for you will be what is slowing down the team. Try to remove the blockers and make sure the team sticks to DRY (Don't Repeat Yourself) with tasks.

Documentation

Document all designs, approaches and plans so the whole team can be aligned on the solution and every team member understands the context and can provide feedback.

Once you have a good handle on the team and work performed, you can now think about the next step. The next step involves engaging other teams in your processes. Figure out the inefficiencies in the system, take over the piece and simplify and deliver. To give a small example, if the above is implemented for operations, feel free to reach out to QA and see if you can help them with jenkins, selenium, code scan, maven , Cobertura and any other tools in use.

Once you have good momentum, start establishing SLA's across different functional units. Create and distribute reports to make sure teams can identify bottlenecks, leverage shared tools and solve the issues cohesively. It is an iterative process, this is what is called "Continuous improvement".

Moving from traditional to true devops org is not an overnight change. The change can take months or years and it never stops. Once set in motion it triggers perpetual improvement and will help you build the best teams and the product.The whole org is connected by one thread, "the product". Using the above attributes identify the areas of improvement.The goal is to delivery the product faster, with superior quality and innovation. Once you solve it for your org. Spread the message, you have the process, people and a plan. Reach out to the first dependency connection, understand their process, work with them to improve it. Set your expectations and march towards them.