Auto Scaling, as if your life depended on it.
Biological Auto Scaling is amazing technology that significantly pre-dates our modern cloud based solutions. Our life depends on this sub-cellular level auto scaling, so it should not be surprising that it’s secrets may help us design better auto scaling mechanisms.
The Summary
This article is quite technical, so I’ll give the summary here, and if you want more detail, then please read on to the end. The cells in our bodies are powered by mitochondria, that convert molecules, derived from our food, into energy. The more mitochondria, the more energy that can be made available to do work, like solving a problem, running a race, lifting a weight or any other activity. When we do strength training in a gym, we stress the mitochondria, and they respond by producing dangerous and harmful chemicals called reactive oxygen species (ROS). This is like getting HTTP500 errors when there are too many users on a web site. However, the nasty chemicals released during an intense workout have a very useful and important signalling purpose. They effectively trigger an auto-scaling process that eventually results in bigger and better mitochondria (scale up) and more of them (scale out). This enables the delivery of much more energy to cells during the subsequent workouts, hopefully improving total performance. This biological Auto Scaling is ingenious and it’s mechanisms may be useful in the design of application stacks that are more resilient and are better at matching available resources with varying workloads.
The Detail
When I run a Load Test or a Stress Test against an Application, I am validating the capacity of that system to support the workload I generate in that test. When an IT system is able to Auto Scale, it is in theory, able to up-size its capacity in response to excessive workload, which is a means of delivering variable capacity to meet variable workload requirements. Capacity costs money, so organisations that want to conserve funds should seek to run the minimum possible infrastructure that is necessary to satisfy demand at any point in time. There are many ways that this can be configured from an IT Infrastructure perspective, but you may in interested to learn about how our human body Auto Scales when it is hit with demands that it can’t cope with.
We need energy to function. That energy, ingested in the food we eat, is broken down into molecules that can be used by cells to perform work. However, a cell is not just a blob of stuff, but contains multiple discrete sub-components. Each cell in each different tissue is fundamentally different, as each tissue has a specific function, but all cells need energy. Most cells contain mitochondria, which you may remember from High School biology, are known as the ‘Powerhouse of the Cell’. When we 'think', we process cognitive workload, and each neuron that fires in that process needs energy. Our brains make up only 2% of our mass, yet consumes around 20% of it's energy. Some researchers say that our cognitive processing capability is rate limited by energy supply, and energy is supplied by local mitochondria, so this is important not just to reaching new milestones at the Gym, but in making important decisions in our personal and professional lives.
When we work out at a gym, we perform a number of sequences of muscle contractions and relaxations. There are huge numbers of cells in each muscle group, and in each of these cells, there are hundreds or thousands of mitochondria. Over time, repeated exercise effectively recruits higher quality muscle fibres to the relevant locations. However, a key limiting factor to the ability to constrict and release a muscle is the ATP that is available to that muscle. The Mitochondria generate most of the necessary ATP, and by auto scaling they are able to increase their size and capacity (vertical scaling) as well as increasing their actual number (horizontal scaling) in each cell.
Considering Mitochondria as if they are Server Instances
Several steps are involved in converting chemical energy, from nutrients in food we ingest, into ATP, culminating in the last five steps known as the Electron Transport Chain (ETC) which take place in the inner membrane of the mitochondria. The last step, processed by a protein known as Complex 5, looks like a turbine, spinning at 150 revolutions per second, with each revolution ‘spring loading’ a third phosphorus ion into an ADP molecule, transforming it into an ATP molecule. Each mitochondria has hundreds of complex 5 proteins. If we imagine the mitochondria as a metaphor for a server in an application stack that is deployed in the cloud, then we could consider the five protein complexes of the ETC as different resources or processes in that server. Each of the 5 ETC steps involves various dependencies, and if a rate limiting shortage of a necessary component to any of those steps exist, then the process ‘backs up’, causing a number of undesirable side effects. A classic problem impacting application throughput in a server occurs when a thread limit is reached. This impact is similar to the way mitochondria are impacted when electrons can’t go all the way through the ETC. For example, if they can’t get to complex five due to an issue impacting on the fourth step of the ETC, then those electrons effectively give rise to Reactive Oxygen Species (ROS), usually in the form of a super oxide, which is the equivalent of a nasty error on a server.
Most people recognize both ROS and nasty server errors as ‘bad’ and something to be minimized. However, the genius of biological auto-scaling is that ROS is a also a powerful signalling mechanism. In small concentrations, ROS effectively stimulates the production of components to increase mitochondria size (scale up) and mitochondria number (scale out), even though that ROS could actually cause serious damage to Mitochondrial DNA or some other important protein. Interestingly, studies have shown that consuming antioxidants while performing high intensity training undermines the auto scaling process, by suppressing the ROS signalling, which is a sort of biological redirect to /dev/null.
Another vital role played by mitochondria is their orchestration process for cell apoptosis – commonly known as cell suicide. When the conditions are satisfied, healthy mitochondria can trigger cellular death, which is itself quite demanding in terms of energy consumption. However, if mitochondria are not healthy, then they are incapable of initiating cell death. Using our metaphor of a mitochondria being like a server in an application stack, this server could trigger a whole stack rebuild in a single availability zone. From an architectural perspective, this makes sense in the context of a non-centralized system, as mitochondria are well placed to know that cell death is necessary if they sense the required conditions. Disease such as cancer is possible however, when apoptosis is not carried out, as necessary, at a cellular level, which is little like auto scaling that adds new instances, but never terminates old defective ones.
Software Deployment and mtDNA
The software of each mitochondria, known as the mtDNA (Mitochondrial DNA), is completely separate and distinct from the Nuclear DNA that we each get from each of our parents. The mtDNA is inherited maternally, is much smaller and is much more susceptible to damage than our nuclear DNA. (It is also much less costly to duplicate and manage, a bit like the difference between an OS and a relatively small program that runs on an OS) It is not just one strand of DNA, but consists of hundreds of different strands that have been collected over many hundreds of generations. Their susceptibility to damage means that, to use a software metaphor, the production code contains defects. Change control and defect management in this software metaphor is achieved by terminating, splitting and recombining of mitochondria. The signalling that triggers these three actions is so effective that it is capable of shutting down ‘instances’ with a particular threshold of defective code, while splitting instances with relatively good code, and combining instances during downsizing in a way that conserves and even concentrates good code. When this does not occur, disease follows. Many cancer researchers are actively investigating the relationship between mitochondrial damage and cancer.
The DNA view of cancer sees the root of the problem in the Source Code (DevOps) , whereas the Metabolic view of cancer sees the problem in the mitochondria (SysOps).
Cellular auto scaling of mitochondria can also be triggered under conditions of caloric restriction. This is especially relevant in brain cells, as it is more difficult to subject them to the physical stresses of Strength Training. It makes intuitive sense that caloric restriction triggers mitochondrial up-regulation, as newly created mitochondria are usually more efficient, and efficiency is very important in a time of scarcity. This is an area of great personal interest to me, as I usually do a 16+ hour intermittent fast each day, and I hope that this regular caloric restriction is beneficial for my brain health. I also follow a ketogenic diet, which means that I make ketone bodies available to my brain cells, as a very clean burning fuel, in addition to glucose, which means that I can deliver more energy to my cognitive process, increasing and optimizing my cognitive performance.
Loosely coupled and decentralized
Our biological systems are loosely coupled, at the cellular level, in a way that allows for dynamic horizontal and vertical auto scaling, where both take place concurrently, and not always in the same direction. Rather than our brain attempting to simultaneously manage vast number of cells and their mitochondria, the operation of cells and mitochondria is largely distributed and autonomous, but they are subject to signalling of various chemical concentrations in the immediate area, which could be influenced by the brain. This is where I see the biggest difference between current IT based auto scaling strategies and biological auto scaling.
A traditional auto scaling configuration may be based on resource utilization, such as CPU usage or bandwidth, but a biological process does not have access to such data, due to its decentralized nature, and is instead limited to the number and type of ‘errors’ it observes in its immediate vicinity. These errors act as signals, which regulate processes that encourage increase or decrease in number, average size and total mitochondrial mass without the need for centrally managed orchestration.
When we embark on a strength training regime, we induce stress in our mitochondria in the skeletal muscles we are working on. We feel that stress as pain and fatigue, but our mitochondria sense it as a rise in ROS. The existence of ROS signals processes that result generation of bigger and better mitochondria over the next few days, resulting in the ability to push our bodies to even higher levels of achievement than were previously possible.
Our brains are more like sensory processors than logic machines. While cells may be largely autonomous, the brain can sense the aggregation of signals arising from cellular activity, and may present it to us as a 'feeling'. For example, "I don't think I can do a full workout today, as I've not yet recovered from yesterday's workout." This gives our bodies a distinct advantage over an IT System, as we are usually able to moderate the workload applied to our bodies, but an application that is open to every user on the internet is exposed to an unlimited upside of potential workload.
Problems and Possibilities with Auto Scaling
When I run a load test of an auto scaling solution, I frequently find that it is not configured to trigger an appropriate scale-out actions in a timely manner. This could be due to poor auto scale design, or unexpected behavior. For example, imagine that version 2 of an application is auto scaling nicely in production, using a CPU Utilization trigger. If version 3 of that application included substantial performance improvements which reduced CPU consumption at the expense of increased memory usage, then the trigger may never fire, as memory may be exhausted before CPU is able to rise to the threshold. Determining optimal scaling policies is non trivial, and much be validated with load testing using a mix of realistic workloads.
When considering optimal auto scaling policies, it may be that we should consider the biological auto scaling that we have depended on for every second of our lives. It may be that usage of ‘error based’ signalling to drive auto scaling is worth consideration. It also may be reasonable to consider that instances that generate dis-proportionally more errors should be marked for termination, mimicking the cellular processes which concurrently increase counts while terminating ineffective components. Regardless of your decisions on how to auto scale an application, it is pretty cool to think about how well our bodies implement scale-out, scale-in, scale-up and scale-down – as out lives do depend on it.
Other Information
If you are interested in more information, I highly recommend taking the following course from edX as a starting point. https://www.edx.org/course/cell-biology-mitochondria It took me about 20 hours to complete (and pass) but has enabled me to understand many research papers on subjects from neurological degeneration to optimizing athletic performance, as the mitochondria are a vital component in almost every cell in our body, and have a profound impact on our health. Other interesting information can be found https://www.sciencedirect.com/science/article/pii/S0896627312007568 , https://www.sciencedirect.com/science/article/pii/S0925443909001677 and https://www.amazon.com/Cancer-Metabolic-Disease-Management-Prevention/dp/0470584920 and https://www.amazon.com/Mitochondria-Future-Medicine-Understanding-Disease/dp/1603587675
Apart from helping customers optimize their IT systems through performance engineering, I’m also keen to help teams achieve optimal performance through awareness that their understanding of IT architecture can be leveraged to benefit their own health, well-being and performance. As I’ve improved my health (lost 38Kg, ceased long term medication and improved functional strength) I’ve also improved in my capacity to do my job, and that is a win-win that I would like to impart to others. Please reach out to me if you would like me to visit your organization for an inspirational and practical presentation.