Analytics Engineering

What is Analytics Engineering?

As the sole analyst of a fast-growing Sydney startup, Claire experienced the pain of the traditional analyst workflow—stuck on a hamster wheel, an ever-growing backlog, and numbers that never quite matched up. So she taught herself dbt, the command line, version control and brought all the rigor of analytics engineering to her team. Along the way, she fell so in love with dbt that she eventually packed up and moved to the US to lead the growing dbt community.

Analytics engineers provide clean data sets to end users, modeling data in a way that empowers end users to answer their own questions. Here are the market trends that gave rise to the newest role on modern data teams.

A year ago, I was preparing a presentation for an event and the title slide asked me to fill in my role. I had been hired as a “Data Analyst”, and when I started the role, I spent my time doing normal data analyst things. I pulled data for finance and marketing, analyzed trends and generated insights, and spent lots of time in Excel and Looker.

But my role had been changing dramatically. Finance and marketing were able to run their own reports. So a normal day for me involved preparing data for analysis by writing transformation and testing code, and writing really good documentation. My tools were no longer Excel and Looker, they were iTerm, GitHub, and Atom.

Was I still a data analyst?

I left the slide blank for the moment, and just before the event, I filled in: “Claire Carroll – Data Something.”

Since then, the industry has begun to adopt a title for what I was attempting to describe – analytics engineer.

What is an analytics engineer? #

Analytics engineers provide clean data sets to end users, modeling data in a way that empowers end users to answer their own questions. While a data analyst spends their time analyzing data, an analytics engineer spends their time transforming, testing, deploying, and documenting data. Analytics engineers apply software engineering best practices like version control and continuous integration to the analytics code base.

When did analytics engineering become a thing? #

The traditional data team

If you were on a “traditional data team” pre 2012, your first data hire was probably a data engineer. You needed this person to build your infrastructure: extract data from the Postgres database and SaaS tools that ran your business, transform that data, and then load it into your data warehouse.

You would then hire a data analyst to build dashboards and reports on top of this data. Analysts, like me, would maintain a mess of SQL files with names like monthly_revenue_final.sql, or maybe just bookmark their queries in a SQL web editor. Often we would need to supplement data in the warehouse with fancy Excel work.

The people consuming the data–CEOs, Marketing VPs, CFOs–would receive monthly reports, request special analysis as-needed, and send analysts a never-ending stream of requests to “segment by this” or “cut by that” or “hey, we’ve updated our definition of ‘account’”.

Being a data analyst was a hard and thankless job, and it didn’t have a ton of leverage. Because of this, it was often a junior role, one where you “did your time” and then moved on to something else