What are the most important standards to follow in data analysis?

Great starting point. I'd like to add that it's important to begin with the problem rather than the data. Often, people in the organization express their needs through data requirements. We should revisit the purpose and communicate in the language of the business. What are the business goals? What are we trying to achieve?

Here are some of the key questions to answer before you consider the data sources, techniques and systems: What performance metric are you going to impact with your analysis? Can you quantify the business value in dollar terms? Answers to these questions will help you to set priorities, focusing on high value initiatives. Also, it will be easier to refine the business problem and the execution plan.

Even if it is called Data Science it is in reality Business Problem Solving using computers and algorithms. It starts defining what problem to solve in connection to a business strategy or priority.

In my early days, i used to think the first step to start a data science project is to gather and prepare your data but when I worked on some projects I realised "No one can shape the data as per requirements until they have defined objective of the project itself or should I say - Objective of the idea". Unless you understand the why this idea is being programmed you cannot make it work. Defining the objective is a easy task you just have to gather all the facts and think or create a line of the original idea into small segments such as 1. What type of data should it be given to keep the originality and accuracy of the project. 2. Which analysis tools and testing method to use. 3. Sources should be legitimate and strong 4. Goal oriented.

Prior to performing any data analysis, 1. We need to understand the business question or problem we are trying to solve and why. 2. Who are our end consumers of the data and what are their needs ? so that we can better understand goals and project scope. 3. Are there any key performance indicators (KPIs) that business is mainly interested in and which fulfill the requirement ? Answering these questions will give us a clear direction and enable us to derive insights in a more effective manner.

Well, when you are done defining the idea's objective, the next step is to choose the right data to feed. It's like Google Maps 'If you have fed the wrong address it will take you to the wrong street'. It's important to choose the right data so you can get accurate predictions. Try comparing the data with requirements with some bullet points and you will know the answer. Stay away from inconsistent, null, and out-of-bounds data as it will consume your time and effort and will interfere with prediction accuracy just like roadblocks.

In case you've never heard it before.. garbage in results in garbage out. When pursuing objectives, I ensure the selected data directly ties to answering my questions and is of the highest integrity. Before use, I carefully inspect for errors, inconsistencies, biases and missingness that could skew results if not addressed. Timeliness is also critical - outdated information may no longer relate to actual business conditions. These quality checks are embedded in my processes, whether done personally or through collaborative review. They set the work up for success while building stakeholder trust in findings and recommendations through rigor.

Organizations and individual analysts benefit from developing consistent first-line checks for data quality. Mine include tagging negative numbers and placeholders ("test", 9999, "foo", 12345, etc), counting blanks and missing values, identifying perfectly correlated dimensions, comparing 5th and 95th percentiles to 1 and 99th percentiles, count of unique values, and so forth. Also, strong considerations must be made for data integration, as poorly-joined datasets can introduce bias and spurious conclusions due to duplicates and dropouts. Time and effort in integrating data are often overlooked. Never ever skip EDA - it will come back to bite you as wasted effort that could've been avoided. It also makes great documentation of the dataset

Along with choosing the right data, another critical consideration is data integrity. This ensures that the data is trustworthy, maintaining accuracy, consistency, and reliability throughout its lifecycle. The synergy between data integrity and selecting the right data establishes a solid foundation for reliable and impactful data-driven actions. This ensures that the information used is not only accurate but also relevant, enhancing the effectiveness of data-driven decisions.

I'm usually not commenting these things, but I could not resist here: This is just bullshit. Real life do not have the data as described here. Data in real life is messy, incomplete, bias... The data as here described as a "need" do not exist... But don't panic, try your best and accept the limitations of the real data world.

Take your whiteboard and a marker and start writing - 1. Okay I have this problem and I have read this solution works best on this kind of problem. 2. You have to do analysis start writing all the analysis methods you know and try to find best one by comparing it to your objective and data. I would say try more than one approach to compare results. 3. You know statistical tools are helpful so you write down how and where to apply those so data gets more strong. 4. Always spend more than 60% of project time on EDA and feature selection, always. This step is like a filter in Water purifiers it will clean and give you the best data to work on. 5. Make a pipeline structure and document it so you always know how it started and ended.

One thing which I have found useful is the need to avoid methods that are inappropriate, irrelevant, or biased for your data and objective and need to ensure that your data is of high quality, and that you have performed the necessary preprocessing, cleaning, and transformation steps to make it suitable for your chosen methods. Also presenting your results in a clear, concise, and meaningful way, using appropriate visualizations, tables, charts, or graphs is vital. You need to highlight the main findings, insights, or patterns that emerge from your data analysis, and explain their significance and implications for your objective and domain.

We as humans, tend to have biases and preferences, this is no different for chosen analytical methods. To mitigate this bias, I weigh my potential methods on a case-by-case basis with the following criteria: 1. The nature of the data: The data type, volume and even the specific distribution may determine the chosen method i.e to report categorical data you ideally can't use a line graph. 2. Business Goals: The goal the business is trying to achieve with the data determines how it should be analysed. i.e. If the business simply wants to improve its visibility then visualisations and descriptive statistics may trump fancy ML models. 3. Feasibility: This refers to the cost, time, and know-how of the technique or method in question.

Depending on the objective, data characteristics, and the domain at hand, one has to choose the right techniques. - Descriptive analytics provides a snapshot of historical trends. -Exploratory analytics helps unearth hidden patterns. -Inferential analytics helps making predictions about a population based on sample data - Predictive analytics forecasts future trends. -Prescriptive analytics, on the other hand, recommends actions for optimal outcomes. -Statistical methods like hypothesis testing and regression analysis bring a quantitative rigor. - Mathematical techniques contribute to a deeper understanding of underlying structures. - Machine learning algorithms enable extraction of patterns from large datasets.

Descriptive analytics serves as the foundation for data analysis: measures of central tendency, dispersion, and graphical representations. Descriptive analytics is often the starting point to understand data. Analysts use exploratory methods to identify hidden patterns, relationships, or trends in the data. Inferential analytics focuses on making inferences about a population based on a sample of data. It involves hypothesis testing and estimating parameters to draw conclusions. Predictive analytics leverages historical data to build models that can forecast future events or trends. E.g. regression analysis, time series forecasting, etc. Prescriptive analytics is recommending specific actions based on the insights gained from models

It could be simpler, or it could be a bit dramatic depending on the audience. 1. If you are communicating approaches and results with your teammate, keep it simple. Ask them for their feedback on your approach and solution. 2. If you are presenting it to your senior or stakeholders, try not to bore them by explaining everything instead make some visual charts graphs and dashboards mentioning - the approach, Solution, and most importantly its long-term impact.

I would add that when communicating results it is important to use tools with diverse display options. For a technical audience, providing detailed dashboards with interactive features allows for in-depth exploration and analysis. Non-technical stakeholders often benefit from simplified visualzations that offer a clear understanding of the key trends and the main messege. This tailored approch ensures that data is accessible and meaningful across varied skill sets and preferences.

Communicating effectively in data analysis is crucial for several reasons. Firstly, clear communication ensures that findings and insights are accurately conveyed to stakeholders, preventing any possible misinterpretation. Additionally, it fosters collaboration among team members, enhancing the overall analytical process. Well-articulated reports help decision-makers understand complex data, facilitating informed choices. Moreover, effective communication aids in building trust, as transparent explanations of methodologies and results instill confidence in the analysis. Overall, it's a key standard to uphold for ensuring the value and impact of data analysis in any organization.

Amongst other things, documentation of steps, assumptions, data coverage and insights at each step has benefitted me. Steps -> In a project that I am working on, certain filtering of data is done upfront as the filtered out data requires a different, more complex computation . Unless documented, this will not make sense to stakeholders. Assumption -> Stakeholders are made aware under what circumstances this analysis will hold true. Data coverage -> Gives a sense of business impact. Eg: If a data analysis is valid for 70/100 stores for a company, decisions can be taken keeping the 70% coverage in mind. Insights -> Sometimes, insights in one step act as inputs in subsequent steps. Hence, documenting insights helps understand the flow.

It's better to start with the initial business goal and address it during the presentation process. Another good point is to have a middle process milestone to show the results and progress to stakeholders to initiate a discussion on if we there is a need to correct the initial objective due to specific findings.

Seek feedback from people with diverse experiences and who are most likely to disagree with you. They will have something to contribute to make you project better.

Feedback is important, but be careful whom you ask for feedback as a wrong input can cause a greater impact. Try to evaluate every feedback first before processing it or thinking about applying the changes. 1. Ask for feedback from your teammates or your superiors, who know that domain and can guide you to a better approach. 2. Sit back and think if there can be a different and better perspective.

Welcome disagreement with open arms. In my experience, diversity of opinion, while makes for uncomfortable conversation in the short term, almost always result in a higher quality outcome.

Seeking feedback in data analysis is akin to having a second set of eyes—it brings valuable insights and refines the process. From my experience, involving others in the analysis not only enhances the robustness of the results but also fosters a culture of continuous improvement. Constructive criticism is a stepping stone to growth, and being receptive to diverse perspectives contributes to more well-rounded and reliable analyses. The iterative nature of seeking feedback ensures that the analysis evolves, aligning with the dynamic nature of data and its interpretation.

Seeking feedback in data analysis is a crucial step to ensure the quality, accuracy, and relevance of your work. Here are some key practices for seeking feedback in the data analysis process: Clearly Communicate Your Analysis: Provide a concise and clear summary of your analysis, including the objectives, methods used, and key findings. Make sure that your audience understands the context and purpose of the analysis. Define Specific Questions: Clearly articulate the questions or issues you're seeking feedback on. Specify areas where you are uncertain or would like additional insights. Engage Stakeholders Early: Involve relevant stakeholders and team members in the analysis process from the beginning.

A resource that is available to all data scientists is their own company. Domain knowledge is a critical skill to maximize the value of data analysis and science. Meet with stakeholders across various business units and learn about them. What are their goals, processes, frustrations, areas of opportunity? To really generate value as a data scientist, the solutions need to positively impact the business. For that, you need to understand the business.

Be a curious mind always, what worked for you might not work for other people with the same set of data but different problem. Always, make a habit of reading new things and Jupyter notebooks on Kaggle. Give your best first and then see others work so that you are learning new things. That's what they say, If in a room you are the smartest then you are not doing something great, you need people to challenge you so that you can learn more being in the uncomfortable situations.

Best place to explore and learn is Kaggle where not only you can get exposure to real-world business problem via competition projects but also interact, collaborate with huge community of Data Science enthusiasts who are not only learning and sharing knowledge but bring diverse background and valuable experience.

I recommend every one, not only juniors, to make a conscious effort to regularly expand their skillset Stay on top of the latest techniques is the greatest long-term career insurance because it ensures you stay relevant I suggest trying different methods and stick with that works best for you: (1) social media creators (2) blogs and forums (3) linkedin articles (4) online courses (5) communities (6) podcasts I also suggest to budget learning time into your calendar My personal goal is to dedicate at least 5 hours per week I block these hours on my calendar to stay accountable Without self-accountability, new things will always get in the way, I recommend you to follow this approach to stay on track

"I recently ventured into the world of Data Analysis, driven by a long-standing curiosity about maximizing the potential of data to assist individuals and organizations in making informed, data-driven decisions. Throughout my journey of delving into the realm of Data Analysis, I've come to recognize that continuous self-learning stands as a fundamental skill for anyone in this field. The exciting part is the abundance of resources available to facilitate this ongoing growth: YouTube courses Books hosted on platforms like GitHub Insightful blogs Engaging LinkedIn Groups I've discovered that embracing these resources has allowed me to not only gain technical expertise but also to cultivate a mindset of adaptability and innovation.

For each project, I use a template to log the end-to-end process. This includes: - Stating clear objectives and hypotheses upfront to establish the intended purpose - Detailing all data sources with descriptors of features, collection/acquisition methods, and any adjustments from raw data collection - Outlining each iterative step of feature engineering, model tuning, and other data wrangling performed - Recording all code files with comments to allow full reproducibility - Commenting visuals and reports to explain intended insights and conclusions Maintaining this documentation standard: - makes my work transparent and defensible - allows proper oversight and peer review - revisit past notes help me continually improving my methodology

Uma quest?o a considerar é o conhecimento do negócio sobre o qual queremos fazer a análise. Acredito que esse conhecimento ajudará na defini??o do objetivo e na sele??o dos dados a serem obtidos para elabora??o da análise.

Keeping track of the value you've built is essential. It's also crucial to communicate this value through various channels within the organization.

In any event, the best most dependable most impactful efficacious and efficient standards to hold above all others are the ones collaboratively agreed to across silos tailored to optimizing collective corporate and shared OKRs. In short - never break the standard of processes that are working unless intend to fully rearchitect the processes and clearly supplant the status quos with a new higher baseline. When ones primed and equipped to deliver value with data science methods & mxms not in place - one delivers that value either as ad hocs(if clearly distinct isolated value) or otherwise brings that value proposition for incorporation into the cross corporate governance and flows.

In addition to the topics mentioned above, it's important to consider that data analysts need to have knowledge of the business they're analyzing the data for. This way, they'll have a better chance of finding the best insights to consider.