How do you know if a software is ready for production?

This morning we had a great discussion in my team about the responsibilities for delivering an application to customers. We very quickly came to the conclusion that "the team" (consisting of Agile Coach, Product Owner, Developers and an Operations Specialist) is responsible for rolling out and ensuring the application is ready for production by a certain date.

So far so good. But what does this mean? Who finally decides to go online? Who is the point of contact in case of issues? Any kind of "manager" who does this, introduces a procedural bottleneck on the one hand and always accepts a high risk of being blind on several levels on the other.?

Actually, this question is one of the main reasons why we are implementing DevOps. If we can't answer this simple question, I would say that we have failed with it. DevOps gives us the opportunity to take responsibility as a team and at the same time give experts the chance to contribute their expertise.

Therefor we ask all Subject Matter Experts (SME) a supposedly simple question:?"What does it take to be ready for production?"

• Senior Developers know if their code is of good or bad quality.
• Operations experts know what measures must be in place to maintain and support any software on the longer run.
• Security specialists are aware of any threat to our applications.
• Product Owners can decide about functional and non-functional (mainly performance) business requirements.

Yes, please - give them guidance on the level of maturity we need: whether it's a proof of concept, a minimum viable product, or a kind of Mars rover project where there's very little room for error.... but let them decide about the required measures.

1. This is done by automation with appropriate quality gates in place. These gates are defined by the Subject Matter Experts.
2. The pipeline execute all tests steps, before an application will be public available. All tests base on a former task can be executed in parallel. So there is no need for a sequence like Dev Environment → QA Environment → Performance Environment → etc. In the best case, we spontaneous spawn and decommission environments according to the test cases needs.
3. It contains all measurements to be tested for approval by each expert group. And each test step provides a gate that must be passed before it is deployed to an externally accessible environment.

If the quality gates are in place, we don't need any kind of approval any more. Anyway, it makes a lot of sense to coordinate this Change (according to ITIL) similar to all other Changes with lower level of automation. However, it should go through a ITIL Change Advisory Board (CAB) as it is a very low risk/impact issue that does not require further approval or coordination. Since all approvals are automated, only the final execution can be scheduled manually. This provides some option for a timed deployment window, customer communication, and more transparency.

The following chart offers an (incomplete) set of tasks during the delivery process and the responsible experts.

Who is establishing the basic pipeline (not all of the gates themselves, but their order and execution)?

Actually it is code. So it depends a bit on the structure and knowledge of the team. It is often maintained by the developers, since the have a very good knowledge here. Also thinkable is to define an explicit role for this, like the Release Train Engineer according to the SaFE? Framework.

How about Scrum Master and Software Architects?

As you could see, there is not the role Architect or Scrum Master/Agile Coach defined. Because both roles are more a kind of enabler: Agile Coaches help the team to iteratively improve and sharpen these roles. Architects help the team to coordinate and document the technical implementation.

What about features which must be rolled out on a coordinated point of time (eg. by marketing constraints)?

In this case, the software can be rolled out anyway before the deadline. But we can use feature toggles or traffic balancing mechanisms to control the point of time when the application is public available. And yes, there are some rare cases (often caused by a bad software design) when this is not possible. In this case, the only option left is a manual roll-out, including a higher risk and a higher effort. A Change Advisory Board can than coordinate the process.

Isn't that kind of risky? To let a computer decide if a product is suitable for production?

Actually it is not a computer deciding this. There have been dozens of experts made these decisions in advance. But instead of writing the necessary conditions for being production ready on a paper, they wrote it directly into the delivery pipeline. And, by the way, all roll-outs are still scheduled and monitored by operations. Further best practices are still in place and independent of the delivery pipeline. Like feature toggles, green/blue deployments, canary releases or well defined roll-back strategies.

And finally: Yes, the team is responsible. But with automated quality gates, all experts within their expertise are involved.?