What are common mistakes when conducting experiments?

Neglecting to consider the application perspective and bypassing crucial design details represent significant oversights in the execution of experiments.

The absolute first step, as with most things, is to speak to the people affected by your actions. Do they even want you to experiment at all? And if so, what are the pain or friction points they prioritise? All experimentation should be focused on problem solving; not experimenting for experimenting's sake.

The importance of a well-defined hypothesis is often overlooked while running growth experiements. A hypothesis should be based on the SMART criteria and the experiment should be aimed at proving the hyposthesis right.

If instructions for participants or procedures for data collection are ambiguous or poorly communicated, it can lead to variability or misunderstanding, potentially compromising the reliability of the results. researchers should strive for clarity in all aspects of the experimental process, from designing the study and communicating instructions to documenting procedures and reporting results. Clear and transparent communication enhances the credibility of the research and facilitates the advancement of scientific knowledge.

La falta de claridad es una situación con la que hay que lidear no solo en la fase de experimentación, sino de todo el estudio o investigación. Considerando que son procesos subsecuentes, al momento de la experimentación nos encontramos con 2 situaciones: poca claridad de los objetivos del estudio o investigación, o poca claridad de la información empírica que obtendremos o validaremos con la experimentación. Es necesario realizar el ejercicio permamente de ajuste y alineación tanto de la parte operativa del experimento, como de la parte general que contiene al experimento.

Selecting the appropriate type of experiment is also key (e.g. qualitative vs quantitive, generative vs evaluative, painted-door, concierge, judo ...). Sites like Kromatic are a great reference source and inspiration for those interested in a deeper dive on this topic.

ailing to adequately plan the experiment can lead to biased results or an inability to draw meaningful conclusions. This includes issues like inadequate sample size, inappropriate control groups, or failure to randomize.

An experiment should be described in advance exactly what you want to test. The hypothesis itself, what exactly you want to test, how you can measure it and how the success is described. This also ensures that the objectives for a successful experiment are not a moving target.

If you haven't read Kahneman and / or Ariely, please do so before designing your next experiment. Bias is the greatest danger to experiment design, in my experience. As humans we are wired to focus on the short-term. Consequently, the benefits of a well-designed experiment which invalidates a future market launch and saves millions on deployment tend to be lost on the product manager in charge of prior development, for example. Experiment design is better off in the hands of neutral parties - internal or external - which can ward off most bias. I personally prefer to conduct A/B testing whenever possible, to get a basic comparison if a broader benchmark is not possible / affordable.

Um outro ponto muito importante é a comunica??o e também o engajamento das diferentes equipes envolvidas no projeto. Além do planejamento do experimento a comunica??o eficaz é um das maiores for?as para o resultado a ser alcan?ado.

It is imperative to remember and stay focused on the reason 'why' you are collecting the data. It is easy to go off track and miss the important points of gaining this data. What are you gathering the data for, how is this relevant to the market? Should always be asked! Making sure the data is useful, because if we have our own opinion or slant included, instead of going to the market to obtain what the market wants, we can easily go sideways instead of forward. The right sort of data in the right market is very powerful, this should always be the focus for collecting any data. Useful 'Evidence based results' has the power to influence. If you can't measure it, you can't control it, if you can't control it, you can't improve it!

Além da insuficiência dos dados, os vieses precisam ser levados em considera??o. A maneira como os dados est?o ou ser?o expostos influencia grandemente o usuário/leitor dos dados. Por isso, garantir a transparência e clareza da informa??o é imprescindível para diminuir a influência dos vieses

A common mistake in experimental design is the failure to differentiate between interventional and observational studies, often resulting in biased conclusions. Many data scientists employ predictive modeling to tackle business challenges that are interventional in nature, then validate their methods through biased A/B experiments. This lack of distinction between study types can result in flawed experimental designs and unreliable conclusions. Observational studies focus on exposure-outcome relationships. Interventional studies directly assess the impact caused by interventions. Helpful resources: "The Effect" by Nick Huntington-Klein "Trustworthy Online Controlled Experiments" by Ron Kohavi, Diane Tang, and Ya Xu

To address insufficient data, researchers should carefully plan their experiments to ensure adequate sample sizes, diverse participant groups, comprehensive data collection, and sufficient duration to capture relevant trends or effects. Additionally, using appropriate measurement tools and techniques can help improve the precision and accuracy of the data collected.

Ignoring feedback A crucial misstep in experimentation is disregarding the feedback received during or after the testing phase. Feedback, whether from customers, users, or stakeholders, provides invaluable insights into the viability and impact of your solution. Ignoring feedback can lead to the development of products or services that don't meet market needs or expectations, resulting in wasted resources and missed opportunities. To prevent this, actively seek and incorporate feedback at every stage of the experiment. Utilize platforms for user feedback, conduct focus groups, or employ beta testing to ensure that the feedback is integrated into the iteration process, enhancing the solution's alignment with user needs and market trends.

Experimentation is without a doubt a good starting point to gain insights about the market resonance and user response to an innovation. However, often innovations don't fail because of the solution to an existing problem they provide, but because of a lack to scale the innovation that it's becoming either affordable (economic viability) or an established utility (usability and ease of use). Specially when innovations do solve complex problems, it's important to follow up with questions about the economic viability and how to become a utility after a successful experimentation or Proof of Concept phase.

To avoid premature conclusions, researchers should approach their data with caution, conduct thorough analyses, consider alternative explanations, and critically evaluate the strength of their evidence. Peer review and collaboration with colleagues can also help provide valuable perspectives and identify potential weaknesses in the interpretation of results. Ultimately, scientific progress relies on careful and rigorous evaluation of evidence before drawing conclusions.

In many wags, we might always assume outcomes and that hinders the processes and progress which would be entirely different.

To sidestep such pitfalls, establish clear, upfront criteria for what constitutes a conclusive experiment. This involves not only reaching a statistically significant dataset but also ensuring that the results align with your predefined goals and metrics. Utilize decision-making frameworks like the Pivot or Persevere model to guide your next steps based on the experiment's outcomes. Moreover, adopting a structured approach to evaluate the findings—perhaps through a detailed Experiment Report—can offer clarity and direction, helping to distill the data into actionable insights.

En la practica lo que he visto es este problema dependerá si es un producto nuevo o una mejora a un producto ya existente. Si el equipo y la dirección está realmente comprometida con aportar un valor a?adido al cliente/usuario se basa el feedback que realmente permitirá mejorar el producto, servicio o experiencia. Si el equipo y los tomadores de decisiones no están realmente comprometidos en un proceso de innovación o existe conflicto de intereses entre las partes involucradas en los diferentes niveles de la empresa (esto más habitual de lo que os imagináis) la información que emplearán será selectiva y más cercana a no generar cambios radicales.

Once the experiment has been carried out successfully, the findings must be evaluated accordingly. Meaning, which hypothesis did you test, what did you observe during the experiment, what is the learning from this experiment and how do you want to incorporate the learning into the new idea. What is also helpful is what was a eureka moment during the experiment, because it is precisely these insights that lead to completely new & interesting possibilities.

Un punto crucial en la innovación es la capacidad de los equipos para gestionar el feedback negativo. No se trata de verlo como un error o un fallo, sino como una oportunidad para aprender y mejorar el proceso de dise?o. Las empresas que dise?an un marco claro para gestionar el "fracaso" obtienen mejores resultados de experimentación. Esto permite que sus innovaciones sean más asertivas y exitosas. ?Cómo hacerlo? Establecer una cultura de aprendizaje: El "fracaso" es parte del proceso de innovación y debe ser visto como una oportunidad para aprender. Definir métricas claras: Medir el éxito y el fracaso de forma objetiva ayuda a identificar áreas de mejora.

lack of learning in experimental research, researchers should cultivate a mindset of curiosity, open-mindedness, and continuous improvement. This includes actively seeking to understand all aspects of experimental outcomes, embracing unexpected findings as opportunities for discovery, and fostering a culture of collaboration and knowledge-sharing within the scientific community. Additionally, documenting and disseminating both successful and unsuccessful experiments can contribute to collective learning and help avoid the repetition of mistakes in future research endeavors.

I have often seen that in simple experiments, such as interviews, a classic mistake is made, that you like to show your idea and end up fishing for conformation. the client confirms that the idea is a good one, but you don't get any insights and you don't test your important critical hypotheses. In the end the customer says he likes it or has small changes, but in the end it is not bought = weak evidence.

Moreover, the true utility of experimentation is only realized when the insights gained are actively applied to refine products or services. To counteract these pitfalls, fostering an organizational culture that prioritizes documentation, sharing, and application of experimental insights is essential. Tools like Learning Cards or an Innovation Portfolio can be instrumental in this process, serving as repositories and conduits for the distilled wisdom from each experiment, ensuring that every trial, whether successful or not, contributes to the cumulative intelligence and agility of the business

One of the most important considerations when experimenting is to be clear on what constitutes success. A successful experiment is one from which you learn. A successful experiment is not necessarily one that proves the hypothesis. We mustn't talk about something as a failed experiment if it didn't work. What we did instead is prove that something doesn't work. So what else can we try?

When conducting #BusinessInnovation experiments, the most common mistake is not Poor Design, Insufficient Data, Premature Conclusions, or Lack of learning. The most common mistake is a hyper-focus on expected outcomes rather than a broader focus on possibilities. 'Innovate' is "making changes in something established, especially by introducing new methods, ideas, or products." Pre-conceived outcomes and tightly regulated test criteria can mute creative innovation.

A good summary of many experiments and their goals is the book by Alex Osterwalder (Strategyzer) Testing Business Ideas. The book is a library of 44 experiments. Evaluated by cost, strength of evidence, set-up time, duration and whether it fits the desirability, feasibility or viability.

Working with project and research teams, I’ve learned that beyond the usual suspects, here are other pitfalls to dodge when doing experimentation: 1. Failing to record every step, observation, and anomaly. Your memory is fallible, not your notebook. 2. Chasing a preordained conclusion instead of letting the data guide you. Be open to surprises! 3. Diving in without a clear plan, controls, and defined metrics. Improvisation is for jazz, not science. 4. Ignoring the power of both hypothesis-driven and data-driven approaches. Balance is key! 5. Ignoring the noise: Dismissing random errors or systematic biases as insignificant. They can mask the truth. Remember, every experiment is a learning opportunity!

Analisar os resultados do experimento em conjunto com as equipes multidisciplinares, verificando as for?as e fraquezas, bem como discutir as li??es aprendidas é fundamental!