How do you implement the data analytics lifecycle, from data acquisition to deployment and feedback?

Step #1: Understand the business problem, identify the relevant internal and external data sources, check for data availability, data quality and sufficiency. These are important activities before embarking on data analytics journey. Step #2: Establish a data pipeline to extract, ingest and transform data. Apache Airflow is a good tool for building data pipeline Step #3: Perform EDA and design reporting data mart keeping end goal in mind (reports, AI/ML models, chatbots etc) Step #4: Develop canned / DIY reports using a reporting tool / framework like Apache Superset or Looker Studio. If you want to develop ML models use Sagemaker or Python or Azure ML Studio Step #5: Deploy and monitor pipeline, models and reports Hope this helps

We need to prepare a lot of infrastructure and define business processes before going straight to acquiring data. Do the hard work up front and scaling is a breeze. Some of the work below can be done in parallel: 1) Work with your team to define a standardized ETL architecture - rope in DevSecOps/SWE/etc. and do an architecture review to make sure everyone is aligned on the approach. What are the business use cases for the data? Example ETL for BI: [AWS S3, SNS, SQS, Lambda, RDS] -> [Tableau]. 2) Provision the resources (e.g., terraform) and configure security/permissions to ensure that the environment is secure to start pulling in data. 3) Define Data Model & Dictionary

You need clarity on the target to start this part. I usually share a list of questions answered by me with the data engineers and analysts. I created the lists based on their input and I usually improve this checklist from time to time. Having this north helps them to understand the goals of the delivery, including which client or KPIs will be impacted (since we are a SaaS company), from whom I received this, who will be the person that will consume the product, and add some extra insights that I have from others products that I oversee (some database already treated, some people to them talk to, old analysis or discussions, etc). This helps me a lot to give them full visibility of the delivery and not have the wrong product at the end.

Avoiding some common data pitfalls is the critical part for the success of data projects. The most common pitfall is the gap between real world and data collected, e.g. employee performance vs appraisal grade. Understanding the limitations of data is the key of data analysis.

I think it's important to note that data visualisation should also be used when we do data preparations. The data analyst would reach a better picture of the dataset e.g. the relations between the dataset, it's characteristic and the questions being answered as well as normality of dataset, data distributions, missing data, outliers, etc. The visualisation helps immensely in deciding which approach, methods, models, tools are most effective to use in the next stages of involvement data processes. Sometimes the analyst might realise that additional data is required to answer the research questions.

Data preparation is crucial in analysis, often taking up to 80% of a project's time. It's the foundation for accurate insights, so investing time very important.Garbage in is garbage out so if you did a really detailed analysis but on a bad quality data then your analysis is not accurate and can lead to wrong decisions.

It is crucial to share preliminary analysis and work-in-progress components before finalizing and moving on to create the end product, such as KPIs, dashboards, and the final database. The standard process involves: (i) Identifying the goal, (ii) Preparing the data, (iii) Analyzing the data and (iv) Providing a preview of numbers and results to align with the PM and the team. Only after completing these steps should we discuss the final product and begin the delivery process.

My most successful projects have been when I had full engagement with my stakeholders thought out the entire process from Acquisition to Presentation. Getting the data out in front of the customer as quickly as possible, getting feedback. Is the data as expected? Are there better sources or attributes that will answer those questions?

Think of data visualization as the storyteller of the data world. It's not just about charts or graphs, but crafting a tale that hooks your audience. When you pick the right tool, be it Excel, Tableau, or D3.js, you're picking your paintbrush. And just like any great story, the best visuals aren’t just clear—they captivate and inspire action. Dive deep, paint the picture, and let the data do the talking! Do not focus on the tool itself when learning data visualisation because each company has the tools used so invest time in learning concepts and how you can make visualisation simple without complicatiing it for audince. The goal for the audience to understand the message easily from your visualization/dashboard!

Capturing feedback on the data deployment is probably the most under-rated aspect of the life-cyle. This is often confused with immediately adoption. For e.g. how many users are logging in to the application or dashboard or how many are accessing the models or data sets? However, the really key piece of feedback that needs to be captured is , what did the dashboard enable the end-user to do and what outcomes could be derived from it. If we can capture that , then we are talking about ROI and chances are we will find the project has paid for itself.

I think everyone realizes that without trust worthy data , the end products (dashboard visualizations etc.) would never be successful. However, while solving the issue of trust , it is only regarded as a data quality problem , which is addressed in the data preparation stage. Although, this will cleanse the analytics side of the pipeline , it does very little to drive consistency and transparency of definition across the organization. This needs to be integrated with enterprise cataloguing and glossary initiatives, to drive consistency of interpretation in the results.

When you are building your portofilio, prioritize datasets that align with the industry you’re targeting or wish to join. Understanding sector-specific data can give you an edge when diving into real-world challenges.

In addition to Data acquisition, preparation, analysis, visualization and deployment - consider data access, security at each layer - who can access, how (direct query, through analytical tools) - operational data vs analytical data needs - deployed data monitoring each stage of data processing and quality A reliable data set comprehensive plan to support deployed data Ex Education, new usages, additional analysis, changes to acquisition sources, business processes It’s rarely deploy and done

In addition to all the response, following: Modelling - Involves developing analytical models to extract meaningful information from data. The choice of model depends on the type of data, business problem and the desired outcomes. Model Evaluation - this assesses the performance of the developed models to ensure their effectiveness is making accurate predictions/insights. Ensure Rigorous Data Quality. Transparency and reproducibility - data sources, methodologies and code should be documented and shared. Ethical data handling - Data privacy, informed consent and data protection protocols must be strictly adhered to. Contribution to knowledge - Data processes should aim to contribute new knowledge and insights.

It is also worth considering the cultural aspect of data utilization. - It's not just about the technical implementation; it's also about fostering a data-driven culture within your organization. Encourage collaboration between data professionals and business stakeholders. - Promote data literacy and empower teams to interpret and leverage data in their decision-making processes. --> A strong data culture ensures that analytics isn't just a process but a mindset driving continuous improvement and innovation. #DCTalks