How do you handle data governance issues when working with external data sources or partners?

Data profiling is crucial. A key part of this process is assessing the limitations of the data: is the data covering the full scope needed (will you be able to infer/impute missing data?); is it defined at the best level of granularity for your projects (what transformations will be needed to make it so?); will it be easily relatable to your existing data (what mapping will be required to get it to be?), etc. Defining the steps that will be needed in order to integrate and use this data will help you determine whether it is a viable source. This planned integration work is expecially important if there are more than one possible sources you can work with, meaning pros and cons of each will need to be quantified and compared.

data profiling, data cleansing and data validation are not tools but the methods to ensure data quality. A beginner would misunderstand about knowing actual tools used for these methods.

Classify the data obtained from external sources or partners based on sensitivity and criticality. Implement appropriate access controls and authorization mechanisms to restrict data access only to authorized individuals. Role-based access controls (RBAC), data anonymization techniques, and encryption can enhance data security and governance.

The type of pipeline you design will be dependent on the update frequency of the data. For data updated less frequently (yearly, quarterly), more in-depth data quality checks can be done (think about the effect this data will have on your projects: the less frequent the data is the cleaner it needs to be). Those QA checks might be more manual, as issues with the data might vary from cycle to cycle. Data format of the source is also more likely to change in 'slow' data. More frequent data ingestion should rely more on automated checks, which will by necessity be less thorough. Regular in-depth reviews of the data are advised, similar to the early 'data profiling' tasks: data that is suitable now might not still be so in 3 years.

In my experience it is important to consider data formats and data transformation because in different countries use different systems to measure data, it is common that for different diseases use numeric and alfanumeric scales ie. Cobb Scale and numeric scale for the same trait to measure and transformation is not straightforward, as well to transform from discrete scale to continuos scale could be a cumbersome because you can not get the same value after you transform data. I think the data should be stored as it comes using the corresponding scale, and then for output data you can transform it, keeping original data for future checking.

In GCP, one common way of controlling storage costs in Cloud Storage is by configuring the lifecycle configuration file to delete or move old data to cheaper storage classes automatically. Also, by setting partitioning and table expiration parameters, BigQuery automatically deletes expired tables and partitions. BigQuery has long-term storage pricing that drops the storage cost by 50% for files not edited in 90 days. Thus by carefully choosing the right data model and infra optimization, the storage cost can drastically be reduced.

Maintain visibility into how external data sources or partners use the shared data. Implement monitoring mechanisms to track data access, usage patterns, and data processing activities. Regularly review access logs, perform audits, and establish alerts for any suspicious activities. If possible, consider implementing data loss prevention (DLP) technologies or data monitoring tools to proactively identify and prevent unauthorized data breaches or misuse.

Promote data governance awareness and provide training to employees, external partners, and stakeholders involved in data sharing processes. Educate them about data privacy, security, and compliance obligations. Encourage a culture of responsible data handling and emphasize the importance of adhering to data governance practices throughout the data lifecycle.