DevOps or Dogma

I recently posted an interview scenario I use to gauge the thought process of prospective candidates for DevOps Roles. This post blew up quite a bit. There were a few comments saying that there is no such thing as DevOps roles or DevOps Engineers. The conversation then turned into what is/isn't DevOps and what DevOps practitioners need to do to promote DevOps in their organizations. This got me thinking, A fictionalized account might shed more light on why I have not much hope in implementing DevOps principles in startups (at least not without making a bunch of compromises). I would be happy to be proven wrong!!!. This account contains a kernel of real life experiences. Some folks may even think they are working in such a startup. Sorry, but it is pure coincidence.

Let us consider the following example here. There is this startup called "Z". The founders of this startup have identified a huge, untapped (future??) market of zombies who need brains to satiate their hunger. Where there is demand(?), there is supply. The founders dropped out of school, customized a WordPress install with a few off the shelf plugins and created an order management system. The founders have zero knowledge about the market they are targeting. They put together a compelling pitch deck. VCs were bowled over by the hype around this vast future market and have dropped a few billions on this startup.

Now the pressure is on to find profitability and, funnily enough, product-market fit? The product folks and designers now want to develop multiple features and new product lines (vegan brains or plant-based brains, anyone?). They hire a few senior people and then hire a number of interns and people who have less to no experience. The developers begin to churn out features, setup infrastructure etc. with little to no regard for best practices, costs, scalability, reliability, availability or any of the other "ilities". The startup continues to raise funds (why not, the market is hot, right?) while also discovering costs are out of control, people are pushing directly to production, outages are frequent, Merge conflicts are an hourly occurrence etc. There have already been malware and ransomware attacks, and the startup has had to pay a few ransoms as well.

Lets assume, at this point, someone who has been exposed to the "One True DevOps Way" has been hired. This person is horrified at the mess and begins to put out feelers to the management and his teammates on fixing some of these issues. One of the very first things he notices is the lack of CI/CD or even access restrictions on production. Because things have been done in a non-standard way, it will take a certain amount of time and maybe 2 to 3 developers to resolve the issues. Cue the management folks who are not ready to allocate these people as things are running fine (for now!!!), why fix something that is not broken. "Z" continues to bumble thru and work in the same way until the infrastructure is now costing a couple of millions a month. Now they start searching for people who can bring this cost under control, setup CI/CD, fix logging, monitoring, auditing, security etc. Because the infrastructure was bought up organically over time, there is no tagging no cost attribution, no documentation etc. Every proposal to fix things by incorporating DevOps/DevSecOps principles is shot down either due to lack of manpower, knowledge, reluctance by the rest of the organization, prioritization or any of the other 100s of things that go on at a typical startup.

Meanwhile, senior executives from mature startups, Fortune 500s are hired to fix what ails the startup. The new management layer brings in their own spin on things, they may neither care nor want DevOps to get in the middle of realizing their ultimate dream. Some may be well-intentioned and want to support such initiatives only to run against the impenetrable wall of short deadlines, mounting costs, politics etc.

At this point, many would argue DevOps needs to be a grassroots effort, the person championing DevOps has to start finding allies and then build the DevOps culture over time. Yeah, that works only if there are people who are aligned or interested in learning something new or have enough time and resources to dedicate to the cause. Remember, the team is already putting in 14-16 hour days (even on weekends) for months on end.

Another take might be that a startup is far too immature for DevOps. Why don't you try this at a larger organization? DevOps goes to die in such organizations, and the death is a slow and painful one, mostly decided by committees or boards.

My observations then is

1. How do you convince an organization that neither wants nor cares about DevOps principles to incorporate these?

2. DevOps and Agile go hand in hand. The startup is experimenting rapidly, requirements change, projects are abandoned and people are shifted around into various teams. How do you Implement DevOps here in such a "Fr"agile environment?

3. Some developers are complaining that they have had to juggle Development, Infra work & CI/CD. They feel these are not aligning with their goals of only writing code. They don't care about code quality, DORA metrics (The only DORA they know is an explorer!) Or SPACE (It is what is up there, you know!). Do you then setup a release engineering team, infra team, SRE team, platform team etc. Will the startup slacken its pace while these folks are hired, trained and are fully up to speed?

4. what if the "One true DevOps way" person is the only one who cares about this and regardless of what they do, they are unable to make any head way??

5. Now rinse and repeat this same basic scenario over multiple startups or organizations, and then tell me if you would still adhere to the "One True DevOps way"?

The thing is, developers doing operations and operations folks learning to code is a pipe dream for many of us. We are stuck in situations where the choice of doing what can be done under the circumstances or having yet another aggravating discussion about DevOps implementation or best practices is a no-brainer. You need to know when to fight and when to give up, and prioritize your health and well-being.

Ultimately, The way DevOps is currently being done in some organizations and startups needs to DIE. A common standard needs to be established where the meaning of what is and isn't DevOps and a set of criteria/certification that you have to pass (Like ISO27001 maybe) might be the only way to figure out what is DevOps and what is Dogma. Maybe they are one and the same??

#devops #startups #security