Why non-production is as significant as production, unless you have mono-environments?

Every component of your software release chain is vital. You know that.

But, as is the case with roughly anything that involves human beings, or machines, or both, there are times when it is all too easy to assume that the most visible elements of a system or process are also the most important. And it is also all too easy to assume that the less visible elements of the process are not very essential at all.

In a typical DevOps continuous delivery chain, production is typically highly visible. It, after all, is where the wheel hits the road, the point at which your application or service meets your clienteles, and where any functional or performance issues are likely to not only become obvious, but also result in problems which require instantaneous remediation.

But the truth is that production is just the absolute stage of the delivery phase. Without development and testing, there would be very little to deliver.

So, you’ve at last been successful at convincing your organization to utilize Kubernetes and you have even got your first services deployed in Production.

You know the uptime of your production workload is of extreme significance; so, you set up your production cluster(s) to be as solid as could be expected under all circumstances. You include a wide range of monitoring and alarming, so if something breaks your SREs get informed and can settle it with the most elevated priority.

But, this is expensive, and you need to have staging and development clusters, too — maybe even a few play areas. Also, as spending plans are in every case tight, you begin considering the worth of having additional clusters.

What's with DEV? Surely can't be as vital as PROD, isn't that so? Wrong!

One of the fundamental objectives of modern technologies and techniques is to improve Developer Productivity. We need to engage developers and empower them to deliver better programming quicker.

In any case, on the off chance that you put less significance on the dependability of your DEV clusters, you are fundamentally saying "it's alright to obstruct my developers", which indirectly implies "it's alright to pay great cash to developers (inside and outside) and let them sit around a large portion of day without being able to work productively”

Additionally, no developer likes to hear that they are less significant than your customers, in essence when you run a platform they are your customers.

How about we take a look at a some of the issues you could keep running into, when putting less significance on DEV, and the effect they may have.

I didn't think of these. We've seen these all occur before in the course of the most recent year.

 Situation 1: K8s API of the DEV Cluster is down

Your nicely assembled CI/CD pipeline is presently spitting out a pile of errors. All your developers are currently obstructed, as they can't deploy and test anything they are building.

In reality this has a significant impactful in DEV than Production, as in PROD your most essential resources are your running workloads and those should be running even when the Kubernetes API is down (that is the beauty of kubernetes, isn’t it for another day). That is, on the off chance that you didn't assemble any solid dependencies on the API, you might not be able to deploy a new version, but your workloads are fine.

 Situation 2: Critical add-ons failing

In many clusters, CNI and DNS are critical to your workloads. If you utilize an Ingress Controller to get to them, at that point that counts as critical. You're extremely bleeding edge and are running a service mesh? Congrats, you included another critical piece there (or rather an entire bundle of them).

Now, if any of the above begins having issues (and they do halfway rely upon one another), you'll begin seeing workloads breaking left and right, or, in case of the Ingress Controller, them not being reachable outside the cluster any longer. This may sound little on the impact scale, yet simply taking a glimpse at our past postmortems, I should state that the Ingress Controller has the greatest share of them.

 Situation 3: Cluster is full/resource pressure

A few developers are currently obstructed from deploying their applications. Furthermore, if they attempt (or the pipeline just pushes new versions), they may expand the resource density.

Pods begin to get killed. Now, your priority and QoS classes kick in — you did make sure to set those, correct? Or on the other hand was that something that was not critical in DEV? Positively, you have at least safeguarded your Kubernetes components and critical add-ons. If not, you'll see nodes going down, which again escalates the stress on resources. Thought DEV clusters could do with less buffer? Reconsider.

This unfortunately happens substantially more in DEV as a result of two things:

? Heavy CI running in DEV

? Less attention on clean definition of assets, priorities, and QoS classes

What can happen?

 A flock of thinkable and impossible things can happen and lead to one of the scenarios above.

 Most often we’ve seen issues arising because of misconfiguration of workloads - maybe one of the below (the list is not conclusive).

? CI is running crazy and filling up your cluster with Pods without any limits set

 ? Defective TLS certs messing up your Ingress Controller

 ? Container’s occupying whole nodes and killing them

 Sharing DEV with a lot of teams? Gave each team cluster-admin rights? You’re in for some excitement. We’ve seen pretty much anything, from “small” edits to the Ingress Controller template file, to someone accidentally deleting the resources.

 Conclusion

In the event that it wasn't obvious from the above mentioned: non-production clusters are critical!

Simply think about this: If you utilize a cluster to work beneficially then it ought to be considered likewise imperative in terms of reliability as PROD.

Non production clusters for the most part should be reliable consistently. Having them reliable just at business hours is precarious. To begin with, you may have distributed teams and facades working at odd hours. Second, an issue that occurs at off-hours may very well get greater and after that take more time to settle once business hours begin.

 A few things you ought to consider (not just for DEV): 

? Be aware of problems with resource pressure when sizing your clusters ; Include buffers.Monitoring and Capacity Planning is important.

 ? Separate teams with namespaces (with appropriate access controls) or even different clusters to decrease the blast radius of mismanagement.

 ? Configure your workloads with the right requests and limits (including you CI jobs)

 ? Consolidate your Kubernetes and Add-on components against resource pressure.

 ? Confine access to critical gears and do not give out cluster-admin credentials.

 ? Have your team members on standby to look into non-production issues almost like a production incident

 ? If possible empower your developers to easily rebuild DEV or spin up clusters for development by themselves

 If you genuinely need to save cash, you can experiment with downscaling in off-hours. If you are really good at spinning up or rebuilding DEV, i.e. have it all programmed from cluster creation to application deployments, then you could experiment with “throw-away-clusters”, i.e. clusters that get tossed away at the end of the day and start a new one soon before business hours.