How do you identify data sources for data modeling and ETL?

2 Identify the types of data sources and formats

Data sources can be classified into different types and formats, based on their origin, structure, and content. Internal data sources are generated and stored within an organization, such as customer data or financial data. External data sources are collected and provided by external parties such as vendors or web services. Structured data sources have a predefined and consistent format, like spreadsheets or CSV files. Unstructured data sources do not have a fixed or regular format, like text documents or images. By understanding the types and formats of data sources that are pertinent to your project, you can decide which methods and tools to use to access, extract, transform, and load them. This can help you gain valuable insights and context from your data.

3 Evaluate the data sources based on criteria

When evaluating data sources for your project, you must consider their availability, quality, compatibility, and cost. Accessibility and maintenance are important factors in assessing availability, as well as accuracy, completeness, and relevance when considering quality. Compatibility requires examining the data elements, attributes, relationships, and formats of the data source. Cost is also a factor; you need to consider the investment in hardware, software, and personnel. Applying these criteria will help you select the most appropriate and reliable sources for your project.

4 Document and validate the data sources

Once you have selected the data sources for your project, you need to document and validate them. Documentation involves creating a data dictionary or a metadata repository that describes the data sources, their types, formats, elements, attributes, relationships, and quality issues. Documentation helps you to keep track of your data sources, communicate with your stakeholders, and facilitate your data modeling and ETL processes. Validation involves verifying and testing the data sources, their connections, and their extraction, transformation, and loading procedures. Validation helps you to ensure that your data sources are accurate, complete, consistent, and relevant, and that they meet your business objectives and requirements.

5 Review and update the data sources

Data sources are not static. They may change over time due to various factors, such as new data sources, new business objectives, new data quality standards, new data formats, etc. Therefore, you need to review and update your data sources periodically, to ensure that they are still suitable and reliable for your project. Reviewing and updating the data sources involves checking their availability, quality, compatibility, and cost, as well as their documentation and validation. By reviewing and updating your data sources, you can maintain and improve your data modeling and ETL processes.