How do you decide which data experiments to prioritize?

As a data scientist, I prioritize experiments by closely aligning them with our business goals. This involves understanding our projects's objectives, KPIs and selecting data that directly impact these metrics. Collaborating with stakeholders, I ensure our data initiatives drive meaningful business outcomes, adapting flexibly to evolving goals.

I believe that a systematic approach is necessary to prioritize data experiments. Defining objectives is like setting the stage before the play begins. Similarly, establishing the aim of the experiment serves as a guiding light, illuminating the path to the team, while also assisting in selection of the right metrics to measure. Metrics help us understand if we are moving in the right direction. Choosing metrics that directly correlate with the organizational goals transforms them from mere numbers into meaningful indicators of success. Finally, maintaining a pragmatic scope for data experiments is crucial. Focusing on issues aligned with strategic objectives yields significant progress.

Prioritizing data experiments involves considering the project goals and the specific challenges you aim to address. Start by identifying the critical questions or problems you want to solve. Evaluate experiments that align with these objectives and have the potential for significant impact. Additionally, prioritize experiments that address uncertainties or risks in your project. Assess the feasibility and resource requirements of each experiment. Consider the potential value of the insights gained and how they contribute to your overall objectives. Regularly reassess and adapt your prioritization based on ongoing results and emerging project needs. This iterative process ensures a dynamic and effective approach to data experimentation.

One of the key qualities that Data Scientists possess is their ability to know what to do with the data the client has. Understanding the domain of the business, looking out for the KPI's by the client are crucial steps that tell what kind of action-steps are relevant ahead. Priroritize these steps over anything else when you deal with any data.

Whether it’s optimizing operational efficiency, understanding customer behavior, predicting market trends, or enhancing product performance, these objectives should be specific, actionable, and aligned with the company’s strategic priorities. We establish clear, measurable criteria that define the success of our data experiments. These metrics encompass aspects like accuracy, precision, recall, customer satisfaction scores, revenue uplift, or any other relevant KPIs that directly correlate with our defined objectives. Aligning our data experiments with the organization’s vision, mission, and strategy is imperative. Every experiment should tie back to the broader goals of the company, contributing meaningfully to its overarching mission.

Another option can be to explore unconventional data sources, like social media sentiment or unconventional data sources, to uncover unique insights. For example, analyzing emojis in customer reviews can provide nuanced emotional responses that traditional methods might overlook, adding a dimension to your data experiments.

To spur creative ideation for predictive models, I hosted hackathons asking “How might we predict churn most accurately?” Structured sessions generated wide model type options from logistic regression to neural networks. To organize further, I mapped 12 proposed models spanning both supervised and unsupervised methods against dimensions like complexity, required data infrastructure, and explainability. Having a wide range of potential model types ensured we didn't fixate on any one approach while grouping them into categories showed us where we lacked certain solutions to address.

You should ensure that your ideas sound like they come from important stakeholders. An often overlooked source is the Global Head of Diversity whose job is to say a wide range of comforting things they've head from other Diversity heads. Some can be connected with reality.

Utilizing YouTube, Newsletters and Blogs: Keep an "idea journal" where you note down interesting points or sparks of inspiration from your readings. Look for case studies, success stories, or even failures documented in these sources. Each can offer a learning opportunity. Identify an identify any new tools, techniques, or data sources mentioned in these readings that you can use.

I think the best way is to calculate mde (minimum detectable effect). That is, how much do we have to increase our key metric so that it cannot be considered a random chance. With many hypotheses, we can calculate mde for each of the AB tests. Then we will choose what to run, focusing on how realistic it is to get such an effect. This process helps us understand that many tests may not be run at all, as mde may be too high and not match realistic expectations. To better understand this, I recommend watching the A/B Testing course on Udacity.

I aim to gather perspectives on two factors: 1. Business value - how much the project moves key metrics if successful 2. Feasibility - can we actually pull this off at present with available data, software, skills and staff within reasonable time? There's no bad or right answers, scoring is relative. I've found using basic comparative rating scales works well for initial gut checks whether high/medium/low or 1-5 stars. No need to overengineer formal frameworks yet - keeping eval qualitative avoids paralysis by analysis when we're just directionally ranking order of magnitude ROI differences. The end goal is landing on a short list balancing fruitful impact potential against executable reality with tradeoffs transparently discussed.

The key to evaluating ideas for data experiments is to rank them by value vs. effort. This can be visualized in a 2D scatter plot. The ideas with the highest value and the lowest effort "quick wins" are often the preferred ones to pursue. When determining effort, it is important to include experienced technical stakeholders who understand what needs to be done. Effort also depends on the experience of the person implementing the solution. In agile contexts, a group vote can be achieved through the so-called "estimation poker". Often, a rough estimate is sufficient. Value estimation requires a holistic view, not a technical view, on the project. There can be direct customer impacts, but achieving strategic goals can also be valuable.

Leading data experiments means continually stress testing assumptions before overinvesting. Before fully committing to complex production builds, I've found it helpful to test proof of concepts on available sample data to check if assumptions actually held up or not when we peered under the hood. Experimenting on a smaller controlled scale revealed what we got right and wrong about different techniques when put to practice. It's much cheaper getting key feedback early to course correct theoretical design flaws than after months invested into a solution hitting walls. Quick tests may stifle crazy innovation but save so much pain down the road.

Validating and refining ideas is crucial in data science. Test your assumptions through methods like A/B testing or piloting. Post-validation, improve your experiment design with tools like SMART criteria or Lean Canvas. This iterative process enhances accuracy and relevancy.

I can not emphasise enough the importance of communication when it comes to running experiments! I once was hired to lead a data science team who was running personalisation experiments and had already reached really good results. But the entire company wanted to get rid of the project. Why? Poor communication, that's it. I always say we should over communicate about experiments because they can generate business anxiety, and when people don't know how the experiments are doing, they naturally make assumptions. So announce your plans, remind people of experiments' start dates, reassure them once they're running, and clearly communicate and explain the results.

- Keep presentations concise by focusing on 2-3 key questions and distilling insights into clear takeaways. - Back up recommendations with quantifiable metrics and visualizations that tell the data's story in an engaging format. - Develop a detailed experiment plan outlining objectives, methodology, timeline and expected outcomes for approval. - Track progress regularly through dashboards and status reports to ensure work stays on track and flag issues early. - Don't just present results - facilitate a discussion to help stakeholders apply learnings and next steps. - Reiterate value through follow up conversations on how analysis informed decisions and impacted goals.

I factor in the metric of value versus effort for each project. This involves evaluating the potential impact (value) against the required resources and time (effort). I prioritize experiments that promise high value with manageable effort, ensuring an optimal balance for effective and efficient resource utilization.

As a Data Scientist, let me elucidate the framework you should use in data experimentation: Objective Definition: Clearly define the goal. Use SMART criteria for precision. Data Collection: Gather relevant data, ensuring quality and representativeness. Hypothesis Formulation: State assumptions or predictions based on initial analysis. Experiment Design: Plan methodologies (e.g., A/B testing, control groups). Analysis: Apply statistical methods and data modeling. Interpretation: Draw conclusions, relating back to the objective. Actionable Insights: Provide clear, actionable recommendations based on findings. Feedback Loop: Implement, measure impact, and refine based on feedback.

Here is what I usually do before any data experiment: 1. Estimate the time and cost of each experiment and rank them: Leverage the Pareto principle to identify low-hanging fruits that can reap significant monetary or time gains with small efforts. 2. Right data, right results: As you list down the experiments, also list down the data/KPIs you might need to run the experiment. Most of the time, the KPIs we might be looking for might not be available or may exist in a different format that needs to be extracted. 3. Set clear expectations and goals for each experiment: Define the problem statement, KPIs to track, goals that will be achieved through the experiment, and what the success criteria are.

It is worth spending sufficient time properly framing the problem, deriving good hypotheses, and using frameworks like logic trees to prioritize interdependent experiments. Achieving satisfactory outcomes in an early experiment may obviate the need to carry out certain follow-on experiments (i.e., pruning branches of the logic tree).