Selected Data Engineering Posts . . . October 2024

The most popular of my data engineering posts in October 2024 ... with additional references ...

Welcome to the October edition of "Selected Data Engineering Posts." This month, we’re looking at new tools and strategies that are making data engineering more efficient and impactful. Learn how generative AI coding tools from major tech companies are changing productivity and coding practices, with wider use expected soon. We’ll also discuss how to optimize Delta tables in Microsoft Fabric using techniques like Z-ordering and data compaction for large projects. AtScale's new Semantic Modeling Language aims to standardize semantic layers across different platforms, improving collaboration and reusability. Other articles cover important topics like using knowledge graphs for AI, AI in gathering software requirements, the competitive AI marketplace, effective data self-service, and data governance for secure insights.

Subscribe to stay updated on the latest advancements in data engineering.

This issue:

GenAI Coding Tools: Tech companies are leading the adoption of generative AI coding tools, which boost productivity and streamline code writing. Other industries are likely to follow, driven by significant efficiency gains, despite challenges like technical debt and varying code quality.

Delta Tables in Fabric: The article discusses advanced optimization techniques and maintenance strategies for managing Delta tables in Microsoft Fabric, including data compaction, Z-ordering, file size management, and commands like OPTIMIZE and VACUUM to enhance performance and efficiency in large-scale data operations.

Semantic Modeling Language: AtScale introduces the Semantic Modeling Language (SML) to standardize semantic layers across platforms. SML enables reusable, shareable models, supports multi-dimensional use cases, and promotes open collaboration through its open-source approach, aiming to streamline data analytics and model-building for business users and data scientists.

Knowledge Graphs: Knowledge graphs function like the human brain by efficiently storing, organizing, and retrieving data through semantic connections. They enhance AI by structuring data for accurate, scalable insights, supporting inferencing, and facilitating proprietary data management. This ensures reliable, context-driven AI applications.

Gathering Requiremens with AI: Generative AI, specifically using the RAG-based approach with Gemini 1.5 Pro, is used to enhance software requirements gathering. Manual tasks are automated, accuracy is improved, complex datasets are handled, and innovative requirements are generated, streamlining development.

AI Marketplace: The AI market is competitive. The article discusses the evolving AI market and the strategies of major vendors such as Microsoft, AWS and Google. It categorizes the providers based on the openness of their AI models and their monetization approaches and emphasizes the importance of aligning with partners who share similar business values when making AI decisions.

Self Service: Self-service in data management is an ongoing journey, requiring a deep understanding of business users. Successful implementation involves establishing appropriate data architecture and governance, creating a supportive operating model, and utilizing the right tools to empower users effectively.

Data Strategy: Organizations must evaluate their data management practices to ensure they align with business strategies and support real-time analytics and generative AI. A comprehensive data strategy enables effective data governance, democratization, and collaboration, ultimately unlocking business value and improving decision-making across the organization.

Effective Data Governance: This blog post discusses the importance of data governance and provides strategies on how to communicate it to various stakeholders. It emphasizes the need for data security, compliance, trusted data for decision making, accelerated insights and ROI on the data stack. By addressing these concerns and demonstrating the benefits of good governance, data stewards can effectively advocate for the necessary resources and drive successful data initiatives.

EEBO - Engineering Excellence to Business Outcomes: Engineering Excellence to Business Outcomes highlights EEBO metrics that bridge engineering efforts and business impact by aligning engineering work with financial results. The book emphasizes common metrics selection, clear communication, and accountability, helping organizations effectively demonstrate the value of engineering.

Looking forward to sharing these insights with you and supporting you in your quest for data excellence.

Enjoy reading!

???????????????????? ???? ???????????? ?????????? ???? ???????? ??????????????????

... are Gen AI Coding Tools the Gateway to Broader Industry Adoption?

In one of the latest articles, Deloitte points out that Generative AI (Gen AI) tools have been rapidly adopted by tech companies, particularly in software development. These tools are considered transformative and significantly increase productivity by automating tasks such as writing routine code and maintaining accuracy.

However, ?????????????????????arise, such as the risk of technical debt from lower-quality code and the need to address the tendency of AI to generate incorrect or biased results. Furthermore, the impact of Gen AI varies depending on the experience of the developer, with junior developers often benefiting the most.

??????????????????????????????:

• ???????????? ?????????????????? ????????: Ensure that the speed of AI-generated code does not compromise quality, increasing long-term system maintenance costs.
• ???????????????? ????-?????????????????? ????????: Regularly check AI-generated outputs for accuracy to avoid potential errors or biases.
• ?????????? ????????????????????: Provide guidance and training on AI tools to maximize productivity and ensure quality standards.
• ?????????????? ???????? ????????????????????: Evaluate the effectiveness of different Gen AI tools to optimize ROI and improve coding outcomes.

????????????????????: There are significant benefits to be gained from integrating AI coding tools, but it requires careful monitoring to avoid technical debt and ensure high code quality. By training developers and validating AI outputs, organizations can fully realize the transformative potential of these tools.

Go to Article

Further Reading

The 20 Generative AI Coding Tools Every Programmer Should Know About

11 generative AI programming tools for developers

7 Code Generation Tools That AI Insiders Have Tried And Tested

???????????????????? ?????????? ???????????? ???? ?????????????????? ????????????:?

???????????????? ???????????????????? ?????? ?????????????????????? ????????????????????

In this third part of his blog series on data ingestion with Spark in Microsoft Fabric Rajaniesh Kaushikk focuses on advanced optimization and maintenance for Delta tables. Efficient data management is critical for performance and reliability, particularly with large datasets and complex workflows.

?????? ???????????????????? ?????? ??????????????????????????????:

???????? ????????????????????: Small files can degrade performance. Use the 'OPTIMIZE' command to merge small files into larger ones to improve read performance.

???????? ???????? ????????????????????????: Configure file sizes for optimal performance. Enable automatic file optimization with 'spark.databricks.delta.optimizeWrite.enabled' and 'spark.databricks.delta.autoCompact.enabled'.

??-????????????????: This technique organizes related data to enhance query performance. Use the 'OPTIMIZE delta_table ZORDER BY (column1, column2);' command for efficient data retrieval.

?????????????????????? ????????????????????: Employ VACUUM to remove old files and manage storage costs. Use schema evolution and handle deletes to maintain optimal performance.

Effective optimization and maintenance of Delta tables are essential for high performance and efficient data operations in Microsoft Fabric. Implementing these strategies will ensure your data management remains robust and scalable, driving better outcomes from your data.

Go to Article

Further Reading

Table maintenance feature to manage delta tables in Fabric

Optimizing performance and cost in Microsoft Fabric Lakehouse

Automate Delta Tables Maintenance in a Microsoft Fabric Warehouse

?????????????????? ???????????????? ?????????? ???????????????? ???????? ????????-?????????????? ??????

... to establish a universal standard, enabling model portability and cross-platform collaboration.

SML is expected to drive industry-wide improvements by simplifying model sharing, reducing dependency on proprietary tools, and promoting widespread adoption of semantic layers.

According to David P. Mariani, AtScale, the lack of a universal standard for semantic modeling has long hindered seamless data analytics. Key challenges include data fragmentation, limited model interoperability, and vendor-specific solutions. In response, the Semantic Modeling Language (SML) was developed as an open-source initiative to standardize semantic modeling, allowing for more consistent and scalable data use.

?????????????? ?????????????????????????????? have been proposed to address these challenges:

• ?????????????????????? ???????????????? ????????????????: Open-sourcing SML aims to create a universally accepted standard, facilitating model portability and collaboration across platforms.
• ?????????????? ???????????????? ????????????: Pre-built models covering industry use cases enhance accessibility and allow for quick implementation.
• ?????????????????? ??????????????????-???????????? ????????????????????: Open access to SML encourages contributions from developers, leading to faster development of new tools and model converters.

The adoption of a standardized semantic modeling language is crucial for accelerating analytics consumption, enhancing interoperability, and democratizing data access for broader business use.

Go to Article

Further Reading

Github repo: https://github.com/semanticdatalayer/SML

What is SML?

?????? ?????????? ???? ?????????????????? ???????????? ???? ???????????? ???????? ????????????????????

The Future of Data Management with Knowledge Graphs

Knowledge graphs enhance data management and AI, improving insights and operational efficiency.

In this blog post, Guillaume Rachez, Perfect Memory and Doug Kimball, Ontotext point out that knowledge graphs offer efficient ways to access and manage knowledge by mimicking the human brain's ability to connect and organize information. They improve data handling by integrating various data points into a semantic framework, enhancing inference and insight generation.

????????????????????:

• ?????????????? ???????? ??????????????????????????: Companies struggle with understanding and managing intricate data relationships, leading to inefficiencies.

• ?????????????? ?????????????? ??????????????????????????: Traditional databases lack the capability to infer and connect disparate data points like a knowledge graph can.

• ???????????????????? ???? ????????????????: Ensuring generative AI models are trained on accurate, structured data to prevent errors and improve reliability.

??????????????????????????????:

• ???????????????? ?????????????????? ????????????: Utilize knowledge graphs to integrate and connect diverse data, improving access and insights.
• ?????????? ?????????????????????? ????????????????????????: Implement advanced inferencing to enhance data understanding and analytics.
• ?????????????????? ???????? ???????????????????? ????: Use knowledge graphs to train AI models, ensuring accuracy and reducing errors in AI-generated insights.

Knowledge graphs provide significant advantages in managing complex data and improving AI capabilities, offering a crucial tool for enhancing data insights and operational efficiency.

Go to Article

Further Reading

What Is a Knowledge Graph?

Knowledge Graphs 101: The Story (and Benefits) Behind the Hype

The Power of Knowledge Graphs in Data Management

Using a Knowledge Graph to Power a Semantic Data Layer for Databricks

How-to: Building Knowledge Graphs in 10 Steps

????????????????????????????????????????????????? ?????????????????? ???????? ?????? ?????? ??????????

?????????????????? ???????????????? ?????????????? ????????????????????

In this article,Hemank Lowe points out that traditional methods of gathering project requirements are time-consuming, error-prone, and often incomplete. Manual processes lead to missed details, miscommunication, and rework, impacting project success. However, using a Retrieval-Augmented Generation (RAG) approach powered by generative AI offers significant improvements.

???????????????????? ???? ???????????????????????? ??????????????????:

• ???????????????????? ?????? ??????????????????: Manual efforts often result in incomplete or inconsistent documentation.
• ???????? ?????? ????????????????-??????????????????: The traditional approach is laborious, increasing project costs and delays.
• ???????????????????? ???? ???????? ????????????????: Managing large volumes of diverse data adds to the complexity of requirement analysis.

??????????????????????????????:

• ???????????????? ???????????? ??????????????????: Leverage RAG-based models to reduce manual work and improve accuracy.
• ?????????????? ???????????????? ??????????????????????????: Use AI to grasp nuanced project details and stakeholder needs.
• ???????????? ????????????????????: Allow generative AI to create new content and foresee unanticipated requirements.

?????? ???????????????????????? ???? ?????? ?????? ???????????????????? ???? ?????????????????????? ?????? ?????????????? ???? ?????????????????? ????????????????????????, improves the quality of documentation, and reduces errors, leading to more successful software development outcomes.

Go to Article

Further Reading

The Potential of Generative Artificial Intelligence in Requirements Engineering

AI in Requirements Engineering: Clarity and Consistency Boost

Using AI for requirements analysis: A case study

???? ?????????????????????? ?????????? ????: ???????? ????. ????????????, ???????????? ????. ????????????????

... choose an AI partner whose business model and values align with yours

The AI market is rapidly evolving, with major players like Microsoft facing increased competition. Ayal Steinberg points out that this competition centers around differentiation: how vendors position themselves to attract customers.

?????? ???????????????????? ?????? ??????????????????????????????

• ???????????? ????????????????????: Microsoft's early lead in AI workloads on Azure stemmed heavily from its OpenAI partnership. However, this reliance on Nvidia chips for in-house GPU technology increases data center capital expenditures and potentially leads to higher operating margins.
• ?????????????????????????????? ????????????????????: Each vendor is leveraging existing strengths and business models to differentiate themselves. Frameworks like "AI Market Dynamics: Open vs. Closed, Direct vs. Indirect" can help understand this landscape.
• ???????? ????. ???????????? ????????????: This framework considers how open the underlying AI model is (open-source or proprietary) and how the vendor monetizes it (direct model sales or AI integration within existing products/services).

?????????? ???????????????? ????????????

The framework categorizes vendors into three groups:

• ?????????? ??: ???? ?????? ???? ?????? ?????????????? (????????????): These vendors (e.g., Microsoft, OpenAI) offer a complete suite of AI capabilities but limit them to their own closed ecosystem.
• ?????????? ??: ???????????????? ???? ?????????? (????????): These vendors (e.g., Meta, Google) primarily generate revenue from user bases through advertising. Their AI investments are focused on improving existing user experiences and services.
• ?????????? ??: ???????????????????? ???? ?????????? (????????): These vendors (e.g., IBM) primarily focus on enterprise software and services. Their AI strategy leverages open-source models combined with proprietary value-added services to deliver solutions for businesses.

???????????????? ???? ???? ??????????????: ?????????????????? ??????????????: Selecting an AI partner with a business model and values aligned with your own is crucial. Today's AI decisions will become the foundation for future applications.

Understanding the evolving AI market landscape and vendor strategies empowers organizations to make informed decisions when selecting an AI partner. Choosing the right partner, based on both technical capabilities and business model alignment, can significantly impact the success of AI initiatives.

Go to Article

Further Reading

How to Make Money with AI: Strategies for Success and 7 Proven Methods

OpenHPI Free Online Course: Profitable AI

The AI Marketplace

?????? ?????????????? ???? ????????-?????????????? ??????????????????: ?? ??????????

Empowering Business Users: Building a Foundation for Self-Service Success

Self-service analytics, where business users can independently access and analyze data, is a popular goal for data-driven organizations.

However - as Wayne Eckerson points out - achieving this can be challenging due to the complexities of data governance, architecture, and tool selection.

??????????????????????????????

• ?????????????????? ?????????????????????? ?????? ??????????????: Successfully implementing self-service requires striking a balance between empowering business users and maintaining data governance.
• ?????????????????????????? ???????????????? ??????????: Classify business users and tailor self-service solutions to their specific needs and capabilities.
• ?????????????????? ?????????????????? ???????? ????????????????????????: Create a data architecture that supports self-service by providing a foundation of building blocks for business users to extend.
• ???????????????????????? ?? ???????????? ?????????????????? ??????????: Implement clear data engineering standards and provide training and coaching to business users to enable them to build their own data structures.
• ?????????????????? ?????? ?????????? ??????????: Choose tools that empower power users to discover, validate, prepare, and analyze data efficiently.

True self-service is a process that requires careful planning and execution. By understanding the business requirements, implementing a solid data architecture and operating model, and selecting the right tools, organizations can empower their business users to make data-driven decisions.

Go to Article

Further Reading

Self-Service Data: Definition, Benefits, and Step-by-Step Guide

Self-Service Data Analytics as a Hierarchy of Needs

Self Service Analytics: What is It and Why is It Important?

Self-Service Data Platform: Definition & Must-Have Features

???????? ???? ?????? ?????????? ???? ?????????

???????????????? ?? ???????? ???????????????? ?????? ??????????????

In today's data-driven world, organizations grapple with managing vast amounts of information. As Databricks points out, a well-defined data strategy acts as a roadmap to unlock the true potential of this data and drive business value.

????????????????????:

• ???????? ?????????? ?????? ?????????????? ????????????????????: Fragmented data hinders analysis and decision-making.
• ???????? ???????????????? ??????: Lack of training hinders understanding and utilization of data.
• ???????????????????? ?????? ???????????????? ??????????: Uncontrolled data access raises security and legal concerns.

???????????????? ?? ???????????? ???????? ????????????????:

• ???????????? ??????????: Clearly outline the objectives for your data strategy, aligning with business goals like cost reduction or customer experience improvement.
• ???????????????? ???????? ???????? ??????????????????: Identify current data sources, access rights, and analytical maturity.
• ???????????????? ?? ??????????-???????????????????? ????????: Include stakeholders from various departments who can contribute diverse perspectives.
• ?????????????? ?? ??????????????: Define steps to achieve your goals, including timelines and resource allocation.
• ?????????????????? ???????? ????????????????????: Set clear policies and standards for data quality, access, and security.
• ???????????? ???? ????????????????????: Consider data lake solutions, integration platforms, and analytics tools.
• ???????????? ?? ????????-???????????? ??????????????: Train employees to work with data and understand its value.
• ?????????????? ?????? ??????????: Regularly assess progress and adjust the strategy based on evolving needs.

??????????????????????: A well-implemented data strategy empowers organizations to:

• ???????? ???????????????? ??????????????????: Data-driven insights guide better business strategies and resource allocation.
• ?????????????? ???????????????????? ?????? ???????????? ??????????: Eliminate wasteful practices and optimize workflows.
• ?????????????? ???????????????? ????????????????????: Gain insights into customer behavior to personalize interactions and drive loyalty.
• ???????????? ????????????????????: Fuel research and development with valuable data-driven insights.

By implementing a comprehensive data strategy, businesses can gain a competitive edge, improve decision-making, and enhance operational efficiency.

Go to Article

Further Reading

What is a Data Strategy & Why is it Important?

AWS: What is Data Strategy?

7 Elements of a Data Strategy

Databricks Data Strategy Blog

????????????????????????? ?????????????????? ???????? ????????????????????

How to craft the ultimate business case for data governance

Data governance is essential for organizations, but it often faces resistance due to perceived high upfront costs compared to unclear benefits. Prukalpa ? (Atlan) points out that it is not merely about risk avoidance but also about leveraging data effectively.

???????????????????? ???? ?????????????? ???????? ????????????????????:

• ???????????????? ????????????????: The advantages of governance can seem intangible against the investment required.
• ???????? ??????????: Lack of visibility into data creates risks regarding sensitive information, which can lead to fines and poor data quality.
• ?????????? ????????????: Stakeholders often distrust data, hampering decision-making and leading to inefficiencies.

??????????????????????????????:

• ?????????????????? ???????????????? ?????? ????????????????????: Emphasize the importance of safeguarding sensitive data and compliance to protect the company’s reputation.
• ?????????????? ?????????? ???? ????????: Foster a data-driven culture by ensuring stakeholders understand data sources and quality.
• ?????????????? ????????????????????: Ensure that teams can easily access and utilize data, thus increasing productivity and efficiency.
• ?????????????????????? ??????: Show how effective governance can enhance the performance of data assets and justify the investment.

Also, in light of the new EU AI Act, establishing effective data governance is critical for organizations seeking to maximize the potential of their data and ensure compliance. By addressing the challenges and implementing these recommendations, companies can derive significant value from their data.

Go to Article

Further Reading

Demonstrating the Value of Data Governance

7 benefits of data governance for your organization

How to Drive Business Value With Data Governance?

The Importance of Data Governance in Today’s Business Environment

???????????????? ?????? ?????? ?????????????? ?????????????????????? ?????? ?????????????????

?? ?????????? ???? ???????? ??????????????

EEBO metrics are crucial for aligning engineering efforts with business objectives. As Richard Gall points out, they provide a direct link between engineering activities and tangible business outcomes. By measuring engineering effectiveness and its impact on business value, organizations can make data-driven decisions, improve efficiency, and gain a competitive edge.

??????????????????????????????:

• ???????????????????? ?????? ????????????????????: Traditional metrics often focus on activity rather than value. Organizations must shift towards metrics that directly measure business impact.
• ???????????????? ???????? ??????????: Effective collaboration between engineering and business teams is essential for aligning goals and understanding the connection between engineering work and business outcomes.
• ???????????????? ?????? ?????????? ??????????????: Select metrics that are relevant to business objectives, actionable, and aligned with team behavior.

By implementing EEBO metrics and fostering collaboration between technical and business teams, organizations can bridge the gap between technical excellence and business success. This leads to more efficient and effective software development, better decision making and ultimately a stronger competitive position.

Go to Article

Further Reading

Introduction to EEBO Metrics

EEBO Metrics Create a Line of Sight from Engineering Excellence to Business Outcomes

How do you know engineering is powering business growth?

Takeaways

Key takeaways from this month's issue that will provide you with important strategies and insights for success in data technology:

GenAI Coding Tools: Tech companies should embrace generative AI coding tools to enhance productivity and code efficiency, while preparing for broader adoption across industries. Data engineers should explore these tools to automate routine tasks, improve coding speed, and maintain oversight to prevent technical debt, while continuously upskilling to maximize productivity and code quality.

Delta Tables in Fabric: To enhance performance and efficiency in managing Delta tables, implement advanced optimization techniques such as data compaction, Z-ordering, and effective file size management. Regularly utilize commands like OPTIMIZE and VACUUM to maintain data quality and streamline large-scale data operations.

Semantic Modeling Language: Explore and adopt the Semantic Modeling Language (SML) to enhance model portability, simplify data analytics workflows, and support standardized, reusable semantic models across platforms, fostering consistency and collaboration in data-driven environments.

Knowledge Graphs: Leverage knowledge graphs to efficiently organize and connect disparate data, enabling powerful inferencing capabilities and improving AI accuracy. Integrating knowledge graphs supports scalable, context-driven data management, which enhances AI applications and ensures reliable, accessible insights.

Gathering Requiremens with AI: Enhance requirements gathering by using generative AI and a retrieval-augmented generation (RAG) approach. It improves efficiency, accuracy, and scalability, automating manual tasks and creating more complete, innovative requirements while handling large volumes of data across various formats.

AI Marketplace: One should evaluate AI vendors based on their business models and revenue strategies, taking into account factors such as openness and direct monetization of AI. Working with vendors that share similar values can have a significant impact on future AI applications and the success of data-driven projects.

Self Service: To successfully implement self-service analytics, prioritize understanding business user needs. Establish robust data architecture and governance frameworks, develop supportive operating models, and select appropriate tools. This approach empowers users while ensuring data quality and consistency, ultimately driving effective decision-making across the organization.

Data Strategy: Focus on developing a comprehensive data strategy that aligns with business objectives. Prioritize creating a unified data architecture, removing silos, and fostering collaboration. By empowering users and ensuring effective governance, you can unlock valuable insights and enhance decision-making across the organization.

Effective Data Governance: To effectively advocate for data governance, one should focus on the tangible benefits it brings. By addressing data security, improving data quality, and accelerating insights, data governance can enhance the overall effectiveness of data engineering efforts. Point out the potential cost savings, increased productivity and improved decision making that result from well-managed data.

EEBO - Engineering Excellence to Business Outcomes: To demonstrate the impact of technical work, prioritize metrics that align with business goals, like EEBO. Collaborate with business stakeholders to select meaningful metrics, ensure clear communication and promote accountability - and ultimately demonstrate how technical work contributes to financial results and strategic success.

Conclusion

This issue shares important strategies for improving data engineering practices. Using generative AI coding tools can boost productivity and code quality while managing technical debt. Applying optimization techniques for Delta tables enhances performance in large data operations. The Semantic Modeling Language helps standardize data analytics, making collaboration easier. Knowledge graphs organize data effectively, improving AI accuracy and insights. Generative AI can streamline gathering requirements, making development faster and more innovative. Choosing AI vendors that align with your business values is crucial for success. Focusing on self-service analytics, solid data strategies, and good governance allows organizations to unlock valuable insights and make better decisions.

Stay tuned for our next issue, where we’ll explore the latest trends and innovations in data technology.

See you next month ...

#DataEngineering #DataGovernance #DataArchitecture #DataAnalytics #DataScience #MachineLearning #AI #DataManagement #BigData #DataQuality #DataSecurity #DataPrivacy #DataIntegration #DataTransformation #DataModeling #DataVisualization #DataOps #CloudDataWarehousing #CloudDataLakes #DataPipeline #ETL #ELT #DataLakes #DataWarehouses #DataMarts #DataCatalog #MetadataManagement #DataLineage #DataProfiling #DataCleansing #DataStandardization #DataGovernanceFramework #DataGovernancePolicy #DataGovernanceTools #DataGovernanceBestPractices #DataGovernanceCompliance