DevOps session, Introduction to transforming software agility

What is a role of software? The only role of any software is to “Enable” business. We cannot run anything through software unless we create a real business value. From small startups to giant market leaders, our true achievement is to survive a challenging environment with end users to keep them up, running and before all “satisfied”.

So how we do this, we introduce too many concepts in our development models to keep up with “Change”. Change introduces all the amazing stuff we see in our products and platforms we build every day. But unfortunately, we have 2 sides to see this change.

In a fast view of the process we mostly follow for agility, we use a concept/change, turn it to a development team (which we will call it “DEV”) including all analysts, developers and quality engineers. They work on that change to generate the product in releases and give it to the support teams who operationally runs the customer environments (and those are the “OPS”) and here comes back the cash from a satisfied customer.

It looks great until we deploy, DEV uses agile methodology (Lean, XP, Scrum or whatever technique), we abide to our iterations and sprints and deliver to OPS. Then with the first issue reported from a customer, we get into the awful argument: “It works perfectly on my machine!”, and “It’s not a server issue, it IS your code!”. How did we get to that!

To diagnose our problem, let’s go back to our “change” topic. Change is required and crucial for us to grow. It should be adding value to our software so we must not fear it. But on the other side by introducing changes we should not be firefighting. It doesn’t mean service outage or performance leaks.

So DEV sees change always positive, we -software engineers- embrace change and love it. Even we create some changes ourselves by adding new technologies and layers in the platforms we develop. We love to add and modify features and leverage our technical power through our knowledge and skills.

On the other hand, stands our OPS team. They are meant to create stability in customer environments, they focus on creating/enhancing the service itself and improving customer satisfaction. After rounds of issues and slips, they are motivated enough to resist “change”. Sadly!

The problem:

• Everything needs software nowadays.
• Software must run on a server to become a service.
• Delivering a service from inception to its users is very slow and error prone.
• There are internal friction points that make this the case.
• This loses you money. (delay = loss)
• IT is frequently the bottleneck in the transition of “concept to cash.”

The symptoms:

• Defects are released into production, causing servers outages or slower services.
• Inability to diagnose production issues quickly, too many rounds of emails and connections where DEV and OPS teams are trying to figure why did it happen.
• Problems appear in some environments only. Your platform new release is successful at all clients’ installations except for one and you waste a lot of time figuring why?
• Blame shifting/finger pointing
• Long delays while DEV, QA and OPS waits on resources or for a response from other team members.
•  “Manual error” is a commonly cited root cause, there is always someone who made a mistake.
• Releases slip/fail, your commitments begin to trouble, and your client relationship sadly deteriorates.

Welcome to DevOps

DevOps is “The practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support.”

It is like what is “Agile” to software development is for software implementation. I will quote from Tom Limoncelli who said: “DevOps is the application of agile methodology to system administration.”

DevOps as a process has its core values and pillars. First the core values which we call it the CAMS values: Culture, Automation, Metrics and Sharing.

The first value of DevOps is “Culture“, it means to focus on the people then the process then the tools. It is all about how to facilitate powerful relationship and communications between team members. To attain proper relationships:

• Engage early and engage often. This keeps you closer to change and aware of everybody else’s position.
• Be open. There is nothing to fear to express or talk about.
• Stop finger pointing. We don’t blame we just move forward.
• Eradicate the “last-mile” syndrome. No more of “it worked on my PC” comments. We are here to do it together.

And to be great in communications:

• Talk is cheap (get out of your chair). Get around and involve responsible team members.
• Involve each other in core processes and decisions. DEV now can access customer shells and OPS will see the release plans earlier. Invite everyone to stand-up meetings and planning/retrospectives.
• Ask Questions. It is a must to ensure commitment is to have everybody’s questions answered.
• Never say “No”. Change is always good, planning it is the problem. Work on that.

The second value is “Automation”, with new trends in technology we can now think of nothing not to be done by a computer. So, let’s move our work to be done by computers as well. What can we automate?

Builds, Deployments, Testing, Monitoring, Self-healing, System rollouts, and System configurations.

And this leads to our third value which is “Metrics”. In DevOps, measure everything, simply like that. Always capture your product performance data, monitor your platform faults and analyze those figures. By doing this you will be able to learn and improve consistently.

And here comes the most important part of DevOps which is “Sharing”. DevOps is all about collaborative effort between DEV and OPS, embrace and amplify your feedback loops between team members. To do so, members must “share”. Share the ideas they have so we stand solid on a ground base. Share the metrics so we can all learn on how to enhance our process. Share the access to all resources so everybody knows where we are and how we do it.

And to start on a DevOps lifecycle, you should follow 3 pillars:

Infrastructure Automation: Automate everything and use services to do it. From development to build to integration to deployment then configuration.

Continuous Delivery: add CI “Continuous Integration” for builds and testing, and CD “Continuous Deployment” for deployment and integration testing. This can boost your deployment to multiple releases per day and minimize your MTTR (mean time to recover) below an hour.

Reliability Engineering: Foster a culture of responsibility, it is still your code even after commit and deployment to production. Design patterns should be used, they empower the resilience in your software. Continuously build, measure, learn and repeat for success.

This looks interesting, how can we build it?

Welcome DEV, you are now an OPS

As a developer you will need to do more work as part of DevOps like:

• Operational Knowledge: Look behind the functional requirements into the non-functional ones. Seek operational knowledge where you see your software from the aspects of reliability, availability, performance and security.
• Responsibility: Always take responsibility for what you built, your code passes test, gets deployed, and stays up for users is your responsibility –not someone else’s.
• Environment: Make your development environment better by having production-like environments and use power tools to have access to production shells to help you get closer to your code in real life.

Welcome OPS, you are now a DEV

And as an operations, development now is one of your tasks. Keep an eye on the following:

• Tools: Don’t do tasks for people. Build tools so they can do their own work. Ops is now a dev team, and the platform is their product.
• Automate: This means code and using coder practices. Learn how to use source control, build scripts and tools for reporting and measurements.
• Feedback: Build feedback loops back to DEV from production environments.
• Metrics: Ensure that monitoring, logging and metrics feedback is pushed periodically/on request into DEV (in addition to business scope).
• It’s Us: Encourage blameless incident postmortems. If something went wrong fix it with the team before working on who did the mistake.
• Support: Welcome your DEV team members on board, they will do on call production support with you.

This sums all, hope you enjoyed it.