What are some of the data engineering trends and innovations that you think are overhyped or underrated?

1 Overhyped: Data Lakehouses

Data lakehouses are a hybrid of data lakes and data warehouses, aiming to combine the best of both worlds: the scalability, flexibility, and low-cost storage of data lakes, and the reliability, performance, and governance of data warehouses. However, data lakehouses are not a silver bullet for data engineering challenges. They still require careful data quality management, schema evolution, and metadata orchestration. They also introduce new complexities and trade-offs, such as choosing the right file format, partitioning strategy, and query engine. Data lakehouses are not a one-size-fits-all solution, and they may not be suitable for every use case or organization.

2 Underrated: Data Mesh

Data mesh is a paradigm shift in data engineering, where data is treated as a product, and data domains are decentralized, autonomous, and self-serve. Data mesh enables data democratization, innovation, and agility, by empowering data owners and consumers to collaborate and share data across the organization. Data mesh also leverages modern technologies, such as cloud-native architectures, microservices, and event-driven design, to create a scalable, resilient, and adaptable data infrastructure. Data mesh is not a hype, but a logical evolution of data engineering, that responds to the increasing complexity, diversity, and volume of data.

3 Overhyped: No-Code/Low-Code Data Engineering

No-code/low-code data engineering is the idea of using graphical user interfaces, drag-and-drop tools, and pre-built components to create and manage data pipelines and platforms, without writing much or any code. While no-code/low-code data engineering can lower the entry barrier, increase productivity, and reduce errors for some simple and repetitive tasks, it cannot replace the need for skilled and experienced data engineers. No-code/low-code data engineering tools often have limited functionality, customization, and scalability, and they may not support complex or novel data engineering scenarios. Moreover, no-code/low-code data engineering tools can create technical debt, security risks, and maintenance issues, if not used properly and responsibly.

4 Underrated: Streaming Data Engineering

Streaming data engineering is the process of ingesting, processing, and analyzing data in real-time or near-real-time, as opposed to batch data engineering, which operates on data at rest or in fixed intervals. Streaming data engineering enables faster and more accurate insights, actions, and outcomes, by capturing and leveraging the dynamic and temporal nature of data. Streaming data engineering also opens up new possibilities and opportunities for data engineering, such as event-driven architectures, stream analytics, complex event processing, and stream machine learning. Streaming data engineering is not a fad, but a necessity, as more and more data sources and applications generate and consume data in streaming mode.

5 Overhyped: DataOps

DataOps is a set of practices, principles, and tools that aim to improve the quality, speed, and reliability of data engineering and analytics workflows, by applying the concepts of DevOps, agile, and lean to data. While DataOps can bring many benefits and value to data engineering and analytics teams, such as faster delivery, better collaboration, and higher trust, it is not a magic bullet or a one-off project. DataOps requires a significant cultural and organizational change, as well as a continuous and iterative improvement process. DataOps also depends on the maturity, context, and goals of each data engineering and analytics team, and it may not be feasible or desirable for every situation or problem.

6 Underrated: Data Governance

Data governance is the process of defining, implementing, and enforcing policies, standards, and rules for data quality, security, privacy, ethics, and compliance, across the data lifecycle and the data ecosystem. Data governance is often seen as a boring, tedious, or restrictive task, that hinders data engineering and analytics innovation and creativity. However, data governance is actually a critical and strategic enabler of data engineering and analytics success and value. Data governance ensures that data is accurate, consistent, reliable, and trustworthy, and that data is used and shared responsibly and ethically. Data governance also helps data engineering and analytics teams to align with the business objectives, regulations, and stakeholder expectations, and to demonstrate the impact and ROI of data.

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?