Key to Success in Generative AI Product Development: Think Like a Researcher

After countless AI projects, one thing has become clear: the team must possess skills in research design, evaluation, NHST (Null Hypothesis Significance Testing), and other research-related areas to develop a viable and feasible solution. This is especially true for Generative AI projects. Throughout my career, including work at large tech companies such as Yahoo, eBay, and NVIDIA, I have worked with a great number of exceptional researchers and engineers. Whenever I enter a meeting with engineers, there is always a 100% confident answer for every problem, ready to be engineered. On the contrary, meetings with researchers often conclude with a plan for an experiment.

Having said that, we need both, but at the appropriate times. We cannot research our way into a working product without engineering, nor can we engineer our way out of a problem without research. The dilemma lies in the nature of the Generative AI development lifecycle. Currently, the evaluation stage of Generative AI projects lacks a defined job role. It is too research-oriented for some software engineers and business professionals, yet hiring a top-notch Ph.D. is not financially viable for many. This leaves organizations with only one option: to upskill their teams in research skills.

At its simplest, evaluating prompts, for instance, is a very straightforward task. However, it requires thorough problem definition, very detailed planning, data collection, measurement, and analysis.

I have been working on AI product development since the early 2010s and over time I had to study research approaches to be more successful in the field of AI. Approaching problems like a researcher involves a systematic, methodical, and analytical mindset. Researchers tend to be curious, critical, open-minded, and persistent, and they use a structured approach to solve problems:

1. Define the Problem Clearly: Start by understanding and defining the problem you're addressing. This involves asking questions, gathering preliminary information, and stating the problem in a clear and concise manner.
2. Review Existing Knowledge: Conduct a literature review or background research to understand what is already known about the problem. This helps in identifying gaps in knowledge, understanding different perspectives, and refining the problem statement if necessary.
3. Formulate a Hypothesis or Research Question: Based on your understanding of the problem, formulate a hypothesis (a predictive statement) or a research question (an inquiry to be answered). This gives direction to your research.
4. Design the Research Methodology: Decide on the best approach to answer your research question or test your hypothesis. This could involve designing an experiment, planning a survey, or determining the methods for data collection and analysis.
5. Collect Data: Gather the information or data needed to address the research question or test the hypothesis. Ensure that the data collection methods are systematic and that the data is reliable and valid.
6. Analyze the Data: Use appropriate statistical or qualitative methods to analyze the data. This involves looking for patterns, testing relationships, or understanding themes.
7. Interpret the Results: Assess what the results mean in the context of your research question or hypothesis. Consider the implications of the findings and whether they answer the research question or support the hypothesis.
8. Draw Conclusions and Make Recommendations: Based on the interpretation of the results, draw conclusions about the problem. Also, suggest practical implications, recommendations, or areas for further research.
9. Communicate the Findings: Share your research findings with others. This could involve writing a report, publishing a paper, or presenting the results at conferences. Effective communication also involves discussing the limitations of your study and the potential for future research.
10. Reflect and Apply Learning: Reflect on what worked well and what could be improved. Apply the lessons learned to future research problems.

As the field of AI continue to grow and evolve, the role of research-mindset becomes ever more critical. By adopting this research approach and mindset, you can tackle problems more effectively and generate insights that are well-founded and actionable. This also paves the way for more secure, reliable, and trustworthy AI development in the future.

If you have any questions or learn more about any Generative AI Product Development Lifecycle aspect, please reach out to [email protected] or visit AI Product Institute at https://aiproductinstitute.com.