登录查看更多内容

How do you deliver data requirements to scientists and analysts?

由人工智能和领英社区提供技术支持

Data engineering is the process of designing, building, and maintaining data pipelines that collect, transform, and deliver data to various consumers, such as scientists and analysts. As a data engineer, you need to understand the data requirements of your end users and communicate them clearly and effectively. In this article, we will discuss some best practices for delivering data requirements to scientists and analysts, and how to avoid common pitfalls and challenges.

此文章中的业界达人

由社区从 4 条内容中精选。了解更多

1 Define the scope and purpose

The first step to deliver data requirements is to define the scope and purpose of the data project. You need to identify the business problem, the data sources, the expected outcomes, and the success criteria. You also need to clarify the roles and responsibilities of the stakeholders, and the timeline and budget of the project. By defining the scope and purpose, you can set realistic expectations, avoid scope creep, and align your data engineering goals with the business goals.

添加您的观点

Elena Alikhachkina, PhD

Digital-First Operating Tech & AI Executive | Fortune 100 Global businesses | CDO, CIDO, CDAi, CIO | Non-Exec Board Director
举报内容
For me, the initial step in fulfilling data requirements always begins with a focus on the data user. We must first understand the problem we aim to solve for this individual, whether it's a sales, marketing, or business executive, among others. It's crucial to delve into how this person operates, including their interactions with customers and their daily workflows. I've noticed a significant gap in prioritizing the "data user" and placing them at the center of every data solution.

已翻译

赞
Lasha Dolenjashvili

Senior Data Partner @ Tabby
举报内容
In my own practice, we often discuss the Data Retention during scope definition meetings. Data Retention is about how long the data should be kept and whether it should be deleted or archived after the retention period. Data Retention is crucial in many ways: - Regulatory Requiements: Many industries, e.g. healthcare or finance, companies have the legal obligations to retain specific type of data for a set amount of time. - Storage Limitations: I've been working in gambling industry, and for example Poker play data had much longer retention period than the Casino. The reason was the vast amounts of data Casino was generated that the company just couldn't maintain in the main analytical database system.

已翻译

赞

2 Document the data specifications

The next step is to document the data specifications, such as the data model, the data quality, the data format, and the data access. You need to describe the data structure, the data attributes, the data relationships, and the data constraints. You also need to specify the data quality standards, such as the accuracy, completeness, consistency, and validity of the data. Moreover, you need to indicate the data format, such as CSV, JSON, or Parquet, and the data access methods, such as APIs, SQL queries, or dashboards. By documenting the data specifications, you can ensure the data consistency, reliability, and usability.

添加您的观点

Elena Alikhachkina, PhD

Digital-First Operating Tech & AI Executive | Fortune 100 Global businesses | CDO, CIDO, CDAi, CIO | Non-Exec Board Director
举报内容
In addition to documenting data specifications, I highly recommend documenting business processes, data collection procedures, and sources of data, among other crucial elements. This comprehensive documentation ensures that everyone involved has a clear understanding of how data is utilized within the organization, leading to more effective data solutions and eventually to decision-making and data-driven processes.

已翻译

赞

3 Validate the data assumptions

The third step is to validate the data assumptions, such as the data availability, the data relevance, and the data compatibility. You need to verify that the data sources are reliable, accessible, and up-to-date. You also need to check that the data is relevant, meaningful, and sufficient for the analysis. Furthermore, you need to ensure that the data is compatible, interoperable, and scalable with the existing or planned data systems. By validating the data assumptions, you can avoid data gaps, errors, and conflicts.

添加您的观点

4 Collaborate with the data consumers

The fourth step is to collaborate with the data consumers, such as the scientists and analysts. You need to communicate the data requirements clearly and frequently, and solicit their feedback and input. You also need to demonstrate the data delivery process and the data results, and address their questions and concerns. Moreover, you need to establish a data governance framework, such as the data ownership, the data security, and the data maintenance. By collaborating with the data consumers, you can build trust, rapport, and alignment.

添加您的观点

Alekhya Achanta

Innovating at the Intersection of Data and Business Intelligence | Enabling Smarter, Faster Decisions
举报内容
Recognize that different data consumers may have varying needs. Offering customizable dashboards or reports can allow consumers to tailor the data presentation to their specific requirements, enhancing its utility. Easy access to data from any location is essential in today's diverse work environments.The world of data is ever-evolving. Encourage a culture of continuous learning among data consumers. This could involve workshops on new data tools, seminars on emerging data trends, or even courses on advanced data analysis techniques. Sometimes, a visual representation can communicate what raw numbers cannot. Offering robust data visualization tools can help data consumers derive insights more intuitively.

已翻译

赞

5 Optimize the data performance

The final step is to optimize the data performance, such as the data speed, the data efficiency, and the data quality. You need to monitor the data pipeline performance, and identify and resolve any bottlenecks, issues, or anomalies. You also need to test and evaluate the data delivery outcomes, and measure and report the data impact and value. Furthermore, you need to update and improve the data requirements, and incorporate any new or changing business needs. By optimizing the data performance, you can ensure the data satisfaction, retention, and loyalty of your data consumers.

添加您的观点

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?

添加您的观点

Data Engineering

+ 关注

给文章评分

我们借助人工智能创建了此文章。您认为这篇文章怎么样？

很棒不太好

举报此文章

查看全部

How do you deliver data requirements to scientists and analysts?

1

2

3

4

5

6

1 Define the scope and purpose

2 Document the data specifications

3 Validate the data assumptions

4 Collaborate with the data consumers

5 Optimize the data performance

6 Here’s what else to consider

Data Engineering

给文章评分

感谢您的反馈

更多Data Engineering相关文章

更多相关阅读内容

How do you deliver data requirements to scientists and analysts?

1

2

3

4

5

6

1 Define the scope and purpose

2 Document the data specifications

3 Validate the data assumptions

4 Collaborate with the data consumers

5 Optimize the data performance

6 Here’s what else to consider

Data Engineering

给文章评分

感谢您的反馈

查看其他技能