How do you prepare data for analysis?

Prepare the data by clearly defining the goals for the analysis. Do this by checking in with stakeholders. Next, Clean the data. This is the biggest thing. Dirty Data = Dirty Analysis. Start by getting rid of irrelevant data points and null values that have no usage in the data set. Format data so that it is easy to read in a way that even a 5 year old could understand what the data is about if you give them some context.

This is the most difficult task for initial data analysis as every detail needs to be jotted down. We have to create a schema and select table formats and platforms for analysis. Often, business users give 2 line requirements which need to be translated to 2 page methodologies for a comprehensive data analysis plan. Once we have aligned business objectives with analysis output, expectations will be clear.

The first step of data preparation is to define your objectives and questions. What are you trying to achieve with your analysis? What are the specific questions you want to answer? How will you measure your results and success? These questions will help you narrow down the scope and focus of your analysis and guide your data selection and collection. You should also consider the ethical and legal implications of your analysis and ensure that you respect the privacy and consent of your data sources.

Keep the questions your data needs to answer: precise and short. Things get adjusted as you go, but it's important to always later come back to the original question of what the data is trying to answer.

Make your analysis "as simple as possible, but not simpler." Avoid overtraining your data but filter out the noise to focus on the key criteria that deliver value for your business.

To work with data effectively, understand input, development, validation, and manipulation processes. 1. Understanding Data Input Process (ETL or ELT) 2. Raw Data: Gain insights by understanding raw data sources. 3. Ensuring Data Validity: controls used for data validity and any exceptions. 4. Calculated/Manipulated Fields: identify such fields and validate the rules. 5. Data Constraints: the rules for data integrity, format, and limits. 6. Field Constraints: such as if null values are allowed etc. 7. Unloaded Data: Identify unutilized data in the target system 8. Inconsistencies in Fields: Address data inconsistencies 9. Interpreting Data from Different Sources: context and calculation methods from varied sources. 10. Review Meta Data

Data discovery phase is crucial. Define scope of KPIs so that different stakeholders (data owners or product/ platform owners) understand what data should be collected

Go back to the reason that you’re doing the research/project to begin with and double check the data you have collected.Make sure it is directly relevant to the issue at hand. Some information is just intermediate, meaning it helps you collect/discover the important/revel ant information that you actual need. Leave the intermediate data in the appendix, not at the forefront

Identify and correct inaccuracies, remove duplicates, and handle missing values to ensure the dataset is accurate and consistent.

Data cleansing and validation is crucial. I often run through multiple validation methods (spot checks, formulas in excel, visualization tools to quickly tell me if I have something that’s off base, etc). If you don’t have good data, your results are meaningless.

Validation is crucial. In the process of validation, we lay a foundation for exploratory analysis & we can expect the results to move in a certain direction - so we already know where we’re headed. Outliers need to be eliminated / treated at the right time or the conclusions from the analysis can take a flip.

Try to process & keep all raw data together in a very simplistic & usable format. Avoid going back to original raw datasets every now & then - this can cause human errors. Always follow best practices to process & structure data.

The transformation step is crucial. In the ETL process, data specialists apply modifications to enrich the data to the end data consumers. nowadays, companies are building curated datasets as a pragmatic solution. Imagine having a dataset that includes the most common KPIs aggregated on a daily basis that can be aggregatable to other levels. or one dataset that combines product, customer, and sales attributes together, considering identified data-quality issues. It will definitely reduce the cost of querying/processing, increase data-consumer productivity, and facilitate driving insights from a trustworthy dataset.

Visualization's objective is to answer questions in an understandable way by the audience. answers can be informative or insightful. imagine having a trend-line chart with 30 countries as a breakdown, Would it be easy to drive a conclusion? or spot a problem? As humans, we like things to look fancy, but is it insightful, simple, and pragmatic as well? Most companies use simple charts, like tables, trend lines, and bar charts. because it simply answers their questions and enables them to make a pragmatic decision.

Whenever possible, use a single tool to prepare and analyze your data. This will make the end-to-end process a lot easier, faster, and more auditable. It will also reduce the risk of errors and inconsistencies, while making the overall user experience a lot more enjoyable. Happy data practitioners produce happy data.

Try to retain as much information possible in your final processed data. You may need to answer additional questions which may not be in the scope of your initial objective, but going back to raw data at that moment, because you did not retain all info previously - can be time consuming, re-work & increases the possibility of errors / inconsistencies.

Defining Data strategy at Enterprise level is crucial. Initial step for capitalising data analytics is to identify sources of data - platforms like Service now, legacy CRM systems, provisioning platforms, back end databases for web apps and respective billing systems. Identify any roadmaps and programs actively migrating data from older version to newer versions Then strategise to store data in a central repository (data lake) using ELT. Here starts the complexity because by now leaders realise different departments and silos are owners of data. Collaboration is the key.

Key things to consider: 1) A bread-trail of the data preparation process is key, helps to track down evidence at later project stages; 2) workflows, new data collected undergoes the same/similar process. This helps to identify areas in the process that can be improved; 3) metadata, dates, people involved, project owners, or even equipment used (essential if working with regulated substances); 4) don't forget about qualitative data, basically record-keeping, text-documentation. Markdown highly effective, it allows you to use tag-based workflows as project IDs, which you can query later. 5) Finally, keep in mind data privacy, who can view/use the data, is it personal data, does it need encryption?