Using ChatGPT in science— 1.5 yrs experience

Milton Pividori , Asst Professor and Principal Investigator at the Pividori Lab in the Department of Biomedical Informatics and the Center for Health Artificial Intelligence (AI) at the University of Colorado Anschutz Medical Campus., led a team in a comprehensive 18-month study to explore how best to incorporate ChatGPT into scientific work. Their research, supported by the Alfred P. Sloan Foundation and the Chan Zuckerberg Initiative, aimed to assess how this AI tool could be safely and effectively used to enhance scientific productivity and quality.

Prompt Engineering

One of the most crucial insights from Pividori's study is the importance of prompt engineering. A well-crafted prompt is key to getting useful and accurate results from ChatGPT. The study found that clarity and specificity in prompts significantly improve the chatbot's performance. Users should clearly define their requests, ask the model to adopt a specific role, and provide examples of the desired output. By doing so, researchers can guide the model to generate more relevant and accurate responses.

Mechanical Tasks Over Creative Work

Pividori’s team discovered that ChatGPT is particularly effective for handling more mechanical and repetitive tasks, such as summarizing texts, revising drafts, or generating code. These tasks are often time-consuming and less intellectually stimulating, making them ideal for delegation to an AI tool. However, for tasks that require creativity or nuanced understanding, such as developing new hypotheses or interpreting complex data, human input remains indispensable.

Caution with Summarization

Another significant finding was the caution required when using ChatGPT for reading and summarizing scientific texts. While the tool can generate summaries, it is prone to "hallucinations" or inaccuracies, especially when dealing with complex or specialized content. Pividori advises against relying on the chatbot for summarizing material that the user is not already familiar with, as it may introduce errors that are difficult to detect without prior knowledge of the subject.

In conclusion, Pividori’s research emphasizes that while ChatGPT can be a powerful tool for enhancing productivity in scientific work, it should be used thoughtfully, with attention to prompt engineering and a clear understanding of its limitations. By targeting appropriate tasks and maintaining human oversight, researchers can leverage AI to improve the efficiency and quality of their work.

Read the full article here from Nature

How frequently are you utilizing ChatGPT or any other LLM for your scientific research?