Modernizing Data Stack, Data Products, LLMs for Data Engineering, Center of Excellence, Data Mesh Readiness and More
While moderating a?panel?at the Modern Data Stack Conference (MDSCon) this spring, I was struck at the diversity of viewpoints on what a modern data stack really is. Our smart panelists George Fraser of?Fivetran, Benn Stancil of?Mode, Lior Gavish of?Monte Carlo, and Sarah Catanzaro of?Amplify Partners?framed this concept as a specific collection of bleeding-edge tools and techniques. In contrast, I suggested the modern data stack is an ever-changing environment that balances the new and the old. In retrospect, my definition is actually more of a “modernizing data stack,” so this blog will use that term.
All types of companies need new tools to integrate, prepare, and deliver data as part of a modernizing data stack. But traditional companies—i.e., those born before the cloud boom—must balance their new tools with older technologies and processes that persist on premises. My blog in April?described?twelve “must-have” characteristics of a modernizing data stack with this balancing act in mind. This blog explores three big trends evident at MDSCon—economic uncertainty, AI disruption, and tool consolidation—and ways for companies to navigate them by balancing new and old elements in their stack.
When our 10-year-old makes something up he gets dimples on his cheeks. It’s a cute sign of mischief. But when a large language model makes something up it flashes no such alerts. That's one of the not-so-cute risks with this emerging form of generative AI.
LLMs are hugely popular because they make humans more efficient and creative with fast, articulate responses to questions or instructions. But the risks to data quality, privacy, intellectual property, fairness, and explainability deserve equal attention. Vendors and companies must help LLM’s early adopters—including?nearly half?of data engineers at last count—maintain control of their businesses.
This blog, the second in a series about LLM assistants for data engineering, explores the risks of this new technology as well as approaches for governing them. The?first blog?defined LLMs and examined use cases for managing data pipelines. The third blog will dive deeper into LLM platforms and tools, and the fourth and final blog will recommend guiding principles for successful adoption. The good news: if governed well, LLMs offer much-needed productivity benefits for data teams that have long struggled to support modern analytics.
Most data & analytics leaders say they need an Analytics Center of Excellence (ACoE), but few know what that is or why they need it. And those who have an ACoE disagree about how to shape and implement it.?
Truth be told, an Analytics Center of Excellence can take many shapes and forms. That’s because every organization is unique: it has a distinct combination of business domains, units, and departments with different information needs and data & analytic skills. As such, every Analytics Center of Excellence must be designed and shaped to match the singular contours of an organization.
What is it??
An Analytics Center of Excellence is a centralized resource of data & analytics experts whose primary objective is to help business teams and individuals make better use of data & analytics. It empowers business teams and individuals meet their own information needs without relying on or waiting for the IT department to do everything for them. It usually consists of three disciplines: business intelligence, analytics, and data science. It works together with the enterprise data team and local (or domain) development teams to optimize the use of data and analytics throughout the organization. (See figure 1.)?
Despite constant innovations in data architecture, infrastructure, and analytics, most organizations today still struggle to realize the promised value of data. While this state of perpetual change introduces many new possibilities, it also creates a moving target for achieving the business outcomes we strive for. This includes improving decisions and actions, uncovering potential problems, and creating new opportunities.?
领英推荐
Data mesh, conceived by Zhamak Dehghani, offers a new paradigm for conquering the ever-changing demands and opportunities of modern data. It rejects long-standing centralized data architectures such as the data lake and the data warehouse and their centralized teams. Instead, data mesh aims for flexibility and responsiveness by decentralizing data and distributing responsibility for it.?
A key pillar of data mesh is the data product, which is a reusable data asset designed for a particular use and delivered according to agreed-upon standards and schedules. In this article, we’ll explore how data products can help organizations get more value from their data, either as part of a data mesh initiative or as a stand-alone strategy.
The Four Pillars of Data Mesh
The data mesh approach centers on four pillars that encompass the organizational and technical changes it espouses.
Data mesh offers a new paradigm for fulfilling the promised value of data. It decentralizes both data ownership and the data itself, shifting them toward the functional domains that create and use data to run their business. But data mesh is not for everyone. An organization must have lots of highly diverse and distributed data. And companies must work within the bounds of what their culture, technologies, and resources will allow. In this article we’ll review the main principles of data mesh and propose criteria by which to evaluate your organization’s readiness for the changes it prescribes.
Who is Data Mesh For?
Data mesh is not for every organization. You need sufficient data size and complexity to justify the investment in a data mesh program.?
About Eckerson Group
Eckerson Group is a global research and consulting firm that focuses solely on data analytics. Our experts have substantial experience in data analytics and specialize in data strategy, data architecture, data management, data governance, data science, and data analytics.
Our clients say we are hard-working, insightful, and humble. It all stems from our love of data and desire to help organizations optimize their data investments. We see ourselves as a family of continuous learners, interpreting the world of data and analytics for you.
Get more value from your data. Put an expert on your side.?Learn what Eckerson Group can do for you!