Should we be hoarding gold like a dragon?

You step into the inner sanctum of an ancient tomb and gander upon a vast field of gold. As you step forward into the room, your eyes meet with its guardian, a fierce red dragon. With riches on your mind, you draw your sword and raise your shield, you attempt to strike the beast...

You completely miss and your opponent retaliates with a mighty fiery breath, the health bar on your screen goes drastically down and the game over screen appears, you lost again!

While getting slapped around by a dragon, one of the many brutally difficult bosses the video game Elden Ring has to offer, a thought popped up in my head - are there perhaps some useful lessons, we the brave and adventurous performance engineers can learn from these dangerous and mythical creatures?

So what is the deal with dragons and hoarding vasts amount of gold?

Within European folklore, dragons are commonly known to be guarding a vast treasure and sit upon a large pile of gold. The philosophical meaning behind this is synonymous with greed and usually ends with a handsome adventurer showing up to slay the beast and claim the prize as their own.

Dragons will, forever be attached to their gold and will hoard every single coin with a flaming passion. Knowing this, perhaps performance engineers could borrow some of that same passion for hoarding our greatest treasures.

What treasure would spark a roaring passion within a performance engineer?

Well, that would be observability data, for example, application metrics like response times and system resource counter are alongside other great metrics and are our greatest treasures and that is why we should take inspiration from how a dragon can cherish every single gold coin in its vast collection. Performance engineers should also not shy away from gazing at the beauty of every single measurement that our own coffers hold.

If keeping your measurements in their raw format is an unfamiliar topic for you then I would suggest you check out one of my earlier articles on the subject of test results or a great article from Stijn Schepers about the topic so you can become more familiar with the concept.

Nevertheless, how dragons go about collecting their coins, I leave to your imagination but gathering performance data in one place is technically not very difficult. Not to say there are no technical challenges as the objective is to create a data platform that allows you to store whatever measurements you want in a central place or you want to have some tool that can bring all that data together into one view.

Generally, this can be achieved in various ways from simply using batch scripts to append all your data into one big text file, to a more complicated approach by using a proper programming language to neatly start storing data into a central database or you can look into vendor solutions that can pull data from across your organization from multiple sources into one view.

The name of the game here is to make it easier for yourself to fetch and analyze the data that matters to you. That way it becomes possible to correlate data and spot interesting trends. What works for you and how you want to tackle this, completely depends on your own technical knowledge, context, and imagination, "the world is your oyster" I would say.

Start following in the steps of a dragon but don't start growing wings yet.

If we decide to follow in the dragon's footsteps and start hoarding our data, we don't want to fall into the same greed trap that they have found themselves in. We need to start thinking about when we have enough. Therefore, deciding which data you want to start storing and how you want to store it, is important. Also, you want to start thinking about how long you would want to store this data.

Finding an answer to these questions could be a difficult endeavor as it depends a lot on your situation and your organization. Generally, I believe it's best to start hoarding the measurements you know and understand first, such as response times and system resource usage counters, for example.

By starting with what you know and understand you can start setting up the necessary technical frameworks to hoard or pull in data efficiently and in a flexible way and immediately start gaining value from the act of storing them.

Once this infrastructure is there and you find more interesting measurements you will have all the tools at your disposal to swiftly add them to your collection or remove ones you no longer find interesting. When talking about data retention policies this really comes down to what you want.

It is very well possible that you know this from the very start and have it clearly defined. However, in most cases, nothing stops you from hoarding as much as you can, you can always change your mind and shape your data retention policy over time as new insights or technical speed bumps come to light.

Gazing upon your collection of treasures to make thoughtful observations.

Storing our results in a medium such as a database for example would allow us to easily query our data and create our very own pile of virtual data gold. This simple action of storing the results of our previously executed tests in a central repository like a database will suddenly open up a plethora of additional valuable insights.

Take the example below which demonstrates a Tableau dashboard digging through the latency results of a daily executed load test, highlighting the problems that occurred over the course of a couple of weeks and multiple releases:

As is seen in the graphic above at any point in the story it would be very easy for an engineer to witness a drop or improvement in the performance of our application and report that to stakeholders and start an investigation into why this deviates from the normal.

The benefits and perks of having all of your treasures in one place.

By just storing our data in an automated way and optionally calculating some additional statistics over this data with a programming language or a BI tool, we are able to take quicker action within key moments. As we can more easily spot a deviation from normal, compare certain runs with each other, or see a break in a trend because we have given ourselves?more oversight of our complicated test results.

We are also able to create a powerful and detailed story for our stakeholders about how the application's performance has changed over time. Besides that, being able to manipulate your results better and generate cleaner-looking and more high-quality graphics also helps you to better convey your message to your stakeholders.

This is for sure not the only benefit that this practice can bring as it also opens up the path to executing more complicated calculations on your test results for example automating your test-result analysis which requires some more advanced statistical voodoo magic and some historical data to be able to pull that off in a reliable and automated manner so it can fit into a CI/CD pipeline.

Concluding this tale and looking ahead to what is yet to come.

So, if you managed to reach till here, I want to thank you for taking out the time to read my article, I greatly appreciate it. The practice of stocking test results and other useful metrics into a database has been quite beneficial for me over the past years and has given me a lot of invaluable insights which I could communicate with my stakeholders.

This is why I felt motivated to encourage others to also start storing their raw data. Having all of this data at my fingertips was the perfect foundation to start doing more complicated things with my test results, for instance, automating the performance test result analysis using powerful statistics to free up more of my time.

As I want to start writing more articles and release content more often I will also soon release a blog post describing how you can also start automating your results analysis using the magical power of statistics, perhaps another fantasy-themed blog post?