This new data role is making waves
The 5X Digest brings you insightful resources on all things data engineering. No spam, just carefully curated content.
This edition, at a glance ??
Rise of the Data Generalist
In the evolving data landscape, driven by AI and advanced data vendors, data teams are expected to become more compact, prioritizing efficiency over team size.
In a landmark blog post in 2017, Maxime Beauchemin talked about the "Rise of the Data Engineer." This marked the moment when data engineering became a recognized job title, with Facebook leading the way in 2012. But let's take a step back and see how we got here.
Looking back: Data boom and specialization
A decade ago, we saw the rise of cloud-based super apps like Facebook, Yelp, and Foursquare. These apps needed to handle vast amounts of data in the cloud, and we didn't have the right tools and infrastructure to make the most of this data.
Things like data pipelines (ways to move data), data storage, modeling (making data usable), business intelligence (BI), and other tools were either not around or just getting started. This deficiency in data management tools led to the need for specialization in the field.
The changing face of data roles
A new breed of professionals, known as "data engineers," emerged in response to this growing challenge. They were responsible for creating tools and infrastructure to handle and optimize data in the cloud, filling the crucial gap.
Over the past decade, as the data landscape has evolved, we've witnessed the emergence and transformation of various roles within the field to meet the changing demands of the data ecosystem.
1. Data Analysts: They translate structured, modeled data into actionable business insights. They play a vital role in interpreting data effectively, empowering organizations to make informed decisions.?
2. Analytics Engineers: Building upon the foundation of data analysis, analytics engineers, often associated with dbt, introduced software engineering practices to the data world. Their focus shifted towards building analytical models, reducing the need for ad hoc analysis, and progressively converging with data engineering responsibilities.
3. Data Scientists: The need for data scientists arose with the increased availability of structured and actionable data. These individuals possess expertise in statistics, programming, and domain knowledge, allowing them to extract valuable insights from extensive and intricate datasets. Their role involves informing decision-makers and uncovering data-driven patterns to shape business strategies.
4. Data Visualization Engineers: Specializing in transforming raw data into easily understandable visual representations, data visualization engineers combine data analysis, design, and technical skills. Their objective is to create visually appealing graphics that facilitate interpreting complex information.
But here's another reason we ended up with so many roles: the data world got really complicated. Each part of the data puzzle had its own set of tools. This complexity meant companies needed bigger teams to manage it all, translating to larger spends and management challenges.
This trend was fueled by buzzwords like 'data is the new oil' and cheered on by investors and pundits. As a result, companies often boasted about the size of their data teams.
However, 18 months ago, things took a turn...
Achieving more with less
It's time to refocus on the true purpose of these teams: driving business growth. It's no longer about building large teams or using trendy tools; it's about generating ROI for the business. The era of unchecked spending on data teams is behind us. Today, our goal is efficiency, which is going to be achieved by solving for both the platform and people:
Reducing the overhead of managing multiple vendors
As the data landscape becomes increasingly complex, "Fully Managed" is the newest category on the block. It eliminates the need for managing multiple vendors, handling vendor discovery, POCs, negotiation, procurement, integration, and maintenance.
There are over 500 vendors in 30 different data categories. The analogy we use is that all these vendors are selling car parts. Imagine walking into a Honda, and instead of selling you a Civic, they sold you an engine, and you had to build your own car.
Businesses spend much time figuring out which vendors to use and how to integrate them. This results in large, cumbersome data teams to manage these vendors, leading to rising costs and complexity.
The fully managed category will give you all of the advantages of an end-to-end platform with the flexibility of best-in-breed vendors. Depending on your industry, use case, size, and budget, you will be able to build a tailored platform for your business.? By 2025, I anticipate that 30% of data teams will migrate to a fully managed solution.
Creating a lean team that moves fast
As we start to consolidate tooling, we have an equally or arguably bigger opportunity to reduce team size; “Do more with less” is becoming a theme. People costs represent a substantial portion of a company's expenses, making this shift significant. Moreover, tooling has become more mature, and many current data roles would not justify dedicated titles moving forward.
As a result, we're witnessing the rise of a new archetype—the "data generalist"—who can operate around different areas of the data stack. Here are a few examples of consolidation in titles in light of this shift:
1. Data platform engineers - They typically comprise 20% of the data team. Tooling consolidation and full managed solutions will allow them to adeptly manage complex tasks like vendor integrations, access control, governance, and security, often without dedicated resources.
2. Data engineers -? The rise of data generalists will enable them to utilize automated ingestion tools to construct and manage data pipelines with increased efficiency.
3. Data analysts / Analytics engineers -The need for specialized analytics engineers and data analysts is on the decline. Since intuitive tools and concepts, like activity schema, have made data modeling simpler and more accessible. Moreover, some of these tools offer helpful insights and recommendations, making it easier to take on these tasks.?
4. Data scientists - We're going to see a lot of AI platforms that run complex models. These platforms require little knowledge to tweak and feed your data in. This will pave the way for generalists to operate at a higher level with less sophistication
5. BI engineers - The rise of conversational BI on top of the semantic layer has automated the tasks usually executed by BI Engineers. Additionally, LLM features like chat have made it far more intuitive to answer business questions.?
领英推荐
Sure, we will still need specialized talent, especially at larger companies, and will still have enough workflows for each role, but in general, smaller to medium-sized companies would need their data team members to have generalist skills as well.
Over time, there is a possibility that the next generation focuses on an all-rounded approach to data instead of specializing in data science, data engineering, or other skill sets. Existing specialists will also need to retrain to gather more rounded expertise. Specialists who don't retrain run the risk of getting left behind.
In conclusion
This shift has given rise to data generalists who can handle various tasks, making traditional roles less necessary. Larger teams may still specialize, but the skills gap is widening, prompting specialists to explore other paths.
Adaptability is key. Expect lean, agile, and highly effective data teams building on top of fully managed data platforms.
Business Intelligence Buyer's Guide
In today's data-rich business landscape, making informed decisions is paramount to success. That's where Business Intelligence (BI) tools come in. With BI tools, your business can:
Factors to consider when deciding on the right BI tool
Data Stack Demystified: Unlocking Growth with Insights
Leading data experts Tarush Aggarwal, Benn Stancil, & Ben Rogojan took a stab at the most pressing questions about the modern data stack in the most recent episode of our Data Dialogues. While you can watch the full webinar on-demand on our YouTube, here's a concise overview of the data stack trends discussed.
Key takeaways
Company objectives drive data team structure: Companies vary in how they view and use data, influencing their data team structures. Adaptation is crucial as perspectives on data evolve.
AI's role in data: Data's value is enhanced through AI for personalization and customer engagement. The adoption of AI tools varies, with some businesses creating custom solutions and others opting for off-the-shelf tools.
Uncertainty in data categories: Emerging data categories like observability and governance face uncertainty as standalone billion-dollar businesses.
Relevance of "Modern Data Stack": The term "modern data stack" may no longer fit the needs of some companies, especially larger enterprises, as their requirements exceed the original use cases.
What's new ??
5X is on a mission to remove the complexity of building, operating, & optimizing data platforms. In line with this, we've launched new features that free you up to drive impact, without the hassle of managing the underlying infrastructure:
Claim your complimentary data architecture audit ??
Know someone who might benefit from these insights? Please share this newsletter with them & help us grow our community!
Thanks for reading! Feel free to drop any questions or feedback in the comments. We're all ears.