The Pain of Research

At the end of last year, I asked one of our customers, “What was your aha moment when using Artemis?”

Without skipping a beat, he said, “The moment I combed through the insights and realized I no longer needed to keep all this context in the back of my mind, Artemis does the research and then surfaces issues for me.”

This was not what I was expecting. I thought he would highlight all the cool AI agent tech we had built that takes the insights we surface and then resolves the tasks for him. Instead, the value that got him was that we researched for him.

In this context, research related to the work needed to understand what problems were occurring in his data platform, why they were happening, and what a suitable fix was to resolve them.

This pain of research is deep, and the more we dig into it, it is a huge cost driver for data teams. A dbt labs survey in 2024 found that data teams spend 26% of their week fixing and maintaining their data platform. This is in addition to the fact that engineers spend 55% of their time maintaining or organizing data sets; a combined 80% of their time is spent on some form of maintenance.

If teams spend 80% of their week on maintenance, a considerable amount is spent on research. Why are teams spending all this time on research?

A Few Possible Reasons for So Much Research

• You didn’t write it: Most practitioners do not build the platform they hold the keys to. Data engineers spend their time understanding, reverse engineering, and dissecting the work of others to fix, maintain, and improve data platforms.
• Fragmentation means no one is a master: When building and maintaining a platform of 8-12 tools, all of which have quirks and ways of becoming efficient, there is much to learn. As an engineer, you won’t know every tool and won’t have the time to understand the intricacies of each. So, you learn enough to get by and take little refresher courses when you need to solve a specific problem.

This convoluted web of work leads to tech debt, but it's also the workspace most data practitioners work in daily.

The other side of this is the mental drain on you. We like the path of least resistance, and it’s discouraging when we see a ticket that should take 15 minutes to resolve to take three or more hours due to the added complexity and research needed.

The emotional side of this problem is deep-rooted. It can crush momentum and motivation for a team performing at a high level and seriously derail progress.

Research is necessary for solving problems, and data is no different. However, in the long run, will we continue to do the research ourselves as practitioners, or will AI do it for us?