How do you manage complex and diverse data?

I was assigned to prepare some dataset for business analysis while participating in a virtual internship program with KPMG. The goal is to have the dataset ready for business decisions. Accuracy of the data is critical. For these, it is very important to be exact about the data and type of data required.

Below are few standard steps to follow for any data pipeline 1. Scope your requirememt and align it with business need. 2. Define your assumption, accept or mitigate any risk or discrepancy with data. 3. Expect the unexpected - define cornor case and handle them gracefully. 4. Document all changes. 5. Test all posibilities with mocking data. 6. Regression test and integration test

An effective, and proven, way to develop a clear understanding of what the organization wants to achieve with data management is to execute a data management assessment. The assessment should cover all the functions of data management (data governance, metadata management, data quality management, data architecture, data security, etc.) and show which function(s) are done well, those that need additional attention, and what functions not practiced according to best practices and industry standards. This information will help to develop the proper scope, the most important activities, and the stakeholders for a data management strategy. Not a simple task - but not an overwhelming one if done with experienced people leading the effort.

My biggest “insider tip” is to sit down with the producers and consumers of the data and always, Always, chart the data flow. It looks like a traditional approach, but the “data flow mapping” gives the surest way of understanding the data need. And helps start face-to-face conversations about “data culture” at ground level. That tells you everything about the real data need in that project. Then comes the estimation of which data is live, which data is historical and which data is just too unwieldy for any next steps. Complex and diverse data just means this initial step has to be done employing ample communication and by checking alignment of in scope data with the big hairy goal / “why” that drives this exercise.

I agree, but the next step after goal setting in designing the data structure and data flows should be to create an org chart. The organization chart allows us to identify future users of the relevant features and relevant data that will be available. each access to data must also be carefully analyzed and justified. Everything else should be prohibited. Unfortunately, in analytical systems, access restrictions are not always thought about. but all other steps in choosing a data model, standards, and so on must also be considered from the point of view of who this data can be accessed.

Assessing data sources is the next step once the scope has been nailed down. It happens as part of estimating the data maturity levels across the teams in scope. Knowing which data is in which format and where (data source) enables designing a robust data model and planning for where to reside the output data post processing - in a data lake, a warehouse or an archive. Future scalability and all future changes in the data sources must also be taken into account at this step. When dealing with diverse data - find overarching themes based on scope to group the data types into. Then assess the data sources in respective groups together. It provides an apples-to apples estimate for in-scope data sources.

Knowing the current state of the data in each domain / subject area is an outcome of a good data management assessment. With this information it is possible to evaluate the critical data sources for the area and identify their maturity / challenges. This evaluation should include both business context (definitions, usage, etc.) and technical information (format, differences in format, data source / data target, etc.)

Assessing data sources is like a chef perusing the market to search for the highest quality ingredients to make a delicious dish. If you need high frequency internal reporting, the data needs to be available and accessible. If the data is going to be externalized on product it needs to be reliable and high quality. Relevance, completeness, accuracy, timeliness, consistency are the "flavor profiles" of the ingredients. Each dish needs a bit of salt, fat, acid, sweetness, etc. in order to work.

In addition to the ETL process, it is important to start designing other processes for organizing data flows inside the data warehouse. The processes of saving aggregated data, archiving data, extracting from the archive, logging the execution of all processes. Also, if necessary, the process of correcting the data at the source. Then all these processes will be refined and changed in the process of designing the repository itself.

Consistency among all sources of the same data is essential. For example, if various counties are preparing data for a statewide repository using different rules, standards, coding protocols, etc., the statewide repository is useless. Every data source has to be dancing to the same drummer.

During a virtual data analytics internship with TaTa, I was tasked with creating an analytical report for the Senior Management team that told a story through visualization. Understanding your audience and their level of data knowledge is critical in data storytelling and data visualizations. The correct structure and presentation style can help you get your point over to the audience. No matter how visually appealing your presentation is, if it is not conveying the right message to your audience, it is useless. Always keep in mind that poor data presentation might lead to disastrous business decisions.

Choosing the right tool has as much to do with cultural factors as it does with the technical features of the tool. Sometimes even more so. When it comes to choosing a tool, culture isn't the most important thing, it's the only thing. How congruent is this tool going to be with the existing and future company culture? Will you get buy in from Senior Leadership? Will they have any personal interest in your success? If not...you might be sunk before you leave the port. Who are the official influencers in the culture (those who have official authority in their role). Who are the unofficial influencers (those who don't have official authority but still influence the culture significantly). You will need buy in from both.

In my experience unfortunately sometimes stakeholders prefer to choose projects with eye-catching words “artificial intelligence”, “neural networks”, “BigData”. Although in really it is not always possible to work with BigData. And in really we can consider all AI-methods as applied statistics methods. And often it is enough to organize simple ETL process and to use Excel.

The amount of choices in data tools, and the wide variety of differentiating as well as semi-overlapping functions the tools perform are a testimony to the size of the data management market. Choosing the data tool is a bit of an analysis by itself, and ideally - the data engineering or data scientists team that will be working hands on with the tool should be involved in this decision making. The tool is ultimately an enabler, and a good technical team can do wonders with whatever software they estimate is appropriate for them. Technical team should estimate the current & future compatibility of any tool before choosing to invest training and set up for it.

Processes should precede tools - it is important to know which processes are necessary and how they need to be performed by business staff (and technical teams) before (repeat before) evaluating and selecting the tools. There are many good tools, but not all are appropriate for an organization - platform, business needs, staff experience, and goals for the usage.

Process boils down to one thing...checklists. Everyone involved in the workflow needs a customized checklist, and they must be cohesive and updated regularly. There also needs to be a quantified way to track progress. Data people love numbers, and we love a progress chart that goes up and to the right even more. Gamification can make the mundane work fun and exciting. Create some healthy (keyword healthy) competition and watch the sparks fly. Two killers of processes are vagueness and boredom. Add clarity (checklists) and fun (scoring and competition) and you will be set up for success. Put a lot of thought into choosing the right metrics and use no more than 5. Choosing the right KPI's can separate success from failure.

Implementation by itself takes many forms and shapes. A data parsing from diverse sources to an active system is a different implementation to processing from a data lake to create knowledge graphs. The steps at implementation depend on type of datum, format at source, complexity of required processing, actual outcome of the processing & final consumer of the processed data. Being able to nail the above factors down without facing any surprises during implementation is a “50% done” success. With data management just booming as a need, it would be difficult to try to generalise that a certain set of implementation steps will work for most projects. Each implementation team + stakeholders must evaluate what will work for them.

In my experience besides data cleaning, validation, integration, transformation, analysis, visualization, and reporting processes it is important to consider processes of archiving, storing of aggregate data and logging. And all processes should be designed taking into account the differentiation of access to data. Unfortunately often many people don’t remember about security.

La tentation est souvent de tout prendre et faire le tri ensuite. C’est une mauvaise idée. La réflexion doit précéder l’action et c’est la même chose au regard des données. Il faut vaincre le syndrome de diogène et sortir du ‘?a peut être utile’ pour penser ′est ce déjà utile’ compléter de ′est ce que ?a une chance d’être utile et si oui pourquoi et avec quel impac’. Si on ne fait pas on traine des systèmes inflationnistes mal ma?trisé et peu documenté .

Once the data outcomes are produced, upgrade and maintenance of the outcomes is a success when the ownership is taken on by the producers and consumers of the data outcomes. Those stakeholders may not be able to do the technical work, but they are the best to judge upgrade needs. Maintenance is an ongoing process, and needs an active data management & governance team to ensure integrity/ proper stewardship of the data outcomes. Garbage in = garbage out can snowball into disasters when it results in errors in data outcomes.

As data ecosystems are dynamic, data teams must stay agile. Automating monitoring workflows and reports saves time. Version controlling data assets enables reproducibility. Modular and adaptable systems are easier to maintain. Fostering a culture of learning and improvement is key. Data professionals should be empowered to refine skills and share knowledge. Connecting data initiatives to business outcomes justifies investments to mature capabilities. Continuous improvement of data outcomes ensures value is maximized over time. Monitoring provides the insights needed to iteratively enhance data processes, analytics, and decision-making.

The is value on designing a robust data governnance model which includes build-in data controls and data validation at the entry point. This will ensure the integrity and the accuracy of the data at the source. Waiting to do this later-on in the process, by valiatign the data outcome is too late and will cause rework to clean the data. Self-serving of the data outcomes should become the norm and should be designed as a capability up-front. Evaluating the data outcomes is best done by someone that is the most familiar with the data.

It is also important to educate users on the importance of data management and the value of data. If users do not see the value, then data will end up in a mess over and over again. Ideally, data managers and users should collaborate in monitoring and improving data management.

Communication with users at all levels is a critical component of any data plan. Working alongside users to understand how data processes can fit into and improve their processes will ensure better outcomes than trying to implement data processes with a top-down approach.

Domain knowledge and data must be a package deal. Without context, data isn't worth very much. It can also be easily misinterpreted and misrepresented. However, with context and domain knowledge, data can lead to additional insights and expertise. Conversely, domain knowledge without new data can be stale, have blind spots, or biases. Up to date data can help keep SME’s fresh and on the cutting edge. Every stakeholder would love perfect real time data. However, building an infrastructure to deliver such data in a consumable way is nuanced, complex, and unique to each application. Any data scientist will tell you, analyzing is 10% of the work. Gathering, cleaning, organizing, and providing context make up 90% of the workload.