Load Testing in production? It's up to you

On April 26, 1986, at 1:24 a.m., a chemical explosion occurred in reactor number 4 of the Vladimir Ilyich Lenin nuclear power plant, popularly known as Chernobyl. Undoubtedly it was, and still is, the most important nuclear accident that has ever happened (level 7 on the nuclear accident scale). In fact, the Fukushima accident was light years behind in terms of consequences.

However, as the ultimate reason for the accident we can conclude that it was due to choosing an inadequate scenario for a test. Translated to the software world, it was a test of an isolated stressed component, theoretically at least.

What kind of tests can be done in a production environment?

Logically we can't do tests that leave the environment in a mess. Let's remember that running performance or load tests in a production environment carries a risk that can be unpredictable. Ladies and gentlemen, NFT (Non Functional Testing) environments were designed for a reason. Why is it so difficult to convince IT managers to invest in cloning the production environment in order to have an isolated testing scenario? For not spending the euros they will tell you EXTRAPOLATE. ????Extrapolate???? Extrapolating in environments where there is no solvent mocks system leads us to take our finger to the wind and buy lottery tickets because it is not something simple. In simple systems perhaps an extrapolation exercise can be performed. But what happens when we talk about systems integrated in a thousand layers? Well, the weakest layer will condition any load test, thus hiding possible bottlenecks.

And we should not only talk about how to extrapolate the results. The fact of doing load tests (and almost any kind) can leave our production environment inconsistent (for example with fictitious orders registered in our system, with defective catalogs, etc.). What can we do in these cases? My recommendation: invest in ephemeral environments that you can quickly set up in the cloud and then discard while minimizing the cost.