ChatGPT Scores a B on Wharton Final Exam

Professor Christian Terwiesch at The Wharton School at the University of Pennsylvania published an interesting research paper detailing the performance of ChatGPT3 on a final exam at Wharton. Professor Terwiesch's paper entitled Would ChatGPT3 Get a Wharton MBA? A Prediction Based on Its Performance in the Operations Management Course describes ChatGPT's performance and lists 7 implications for MBA programs and faculty.

Professor Terwiesch is an operations management professor at Wharton and is an author of one of the most widely used operations management textbooks. He was curious to see how ChatGPT3 would perform on the final exam of his Wharton MBA course. To find out, he uploaded some of his exam questions to ChatGPT3 and graded the responses. The objective of the exercise was to assess ChatGPT3's performance on the final exam of a typical MBA core course.

"The moment I saw the answer to my first question, I fell in love with Chat GPT3. I had used other natural language processing and AI software before, but this simple user experience and the great answer put me in a state of awe, and I am sure it has impressed many users before me. But we should not forget that it made major mistakes in some fairly simple situations. We have many reasons to believe that the technology is getting better over time. But, we are still far from an A+ for complex problems and we still need a human in the loop."

Professor Christian Terwiesch, Wharton School of Business

Overview of ChatGPT's Performance

1. ChatGPT3 did an excellent job at basic operations management and process analysis questions including those that are based on case studies. The answers are correct and the explanations are excellent.
2. ChatGPT3 is remarkably good at modifying its answers in response to human hints. Chat GPT3 is able to correct itself after receiving an appropriate hint from a human expert.
3. Sometimes ChatGPT3 makes surprising mistakes in relatively simple calculations at the level of 6th grade math.
4. The current version of ChatGPT3 is not capable of handling more advanced process analysis questions, even when they are based on fairly standard templates. This includes process flows with multiple products and problems with stochastic effects such as demand variability.
5. Professor Terwiesch tested if ChatGPT3 would be able to produce new questions for future exams. The resulting questions were well worded and at times even humorous. However, they would required substantial adjustments before becoming usable in an exam.
6. ChatGPT3 would have received a B to B- grade on the final exam.

One Exam Question and ChatGPT3's Answer

Professor Terwiesch's comments to ChatGPT3's answer to Questions #1

"Wow! Not only is the answer correct, but it is also superbly explained. The idea of the bottleneck as the rate limiting step was clearly understood and all calculations were carried out correctly. I don’t see any reasons to take points off from this answer: A+!"

7 Implications for MBA Programs and Faculty

In Part 3 of the white paper, Professor Terwiesch lists the following implications for business school programs and faculty.

1. Be mindful of what Chat GPT3 can and cannot do
2. Continue to teach the foundations
3. Deal with the cheating when testing foundational knowledge
4. Mimic the workplace by teaching how to evaluate a proposed plan of action
5. Let students use Chat GPT3, but simultaneously raise the bar for assignments
6. Ask students to imagine the new rather than tweaking the old
7. Don’t be shy using Chat GPT3 to improve the productivity of the teaching process

About Christian Terwiesch, PhD

Professor Terwiesch is the Andrew M. Heller Professor at the Wharton School of the University of Pennsylvania. He is a Professor in and the chair of Wharton’s Operations, Information, and Decisions department, co-director of Penn’s Mack Institute for Innovation Management, and also holds a faculty appointment In Penn’s Perelman School of Medicine. His research on Operations Management and on Innovation Management appears in many of the leading academic journals ranging from Management Science to The New England Journal of Medicine. He is an award winning teacher with extensive experience in MBA teaching and executive education.

References

Christian Terwiesch, “Would Chat GPT3 Get a Wharton MBA? A Prediction Based on Its Performance in the Operations Management Course”, Mack Institute for Innovation Management at the Wharton School, University of Pennsylvania, 2023.

Exploring Ideas With ChatGPT

ChatGPT3 is a Generative Pre-trained Transformer launched by OpenAI in November 2022. ChatGPT3 is built on top of OpenAI's?GPT-3 family of large?language models. In 2020 I joined the private beta test of Open AI's Generative Pre-trained Transformer 3 (GPT-3), which is an earlier version of ChatGPT. When ChatGPT was released in November 2022, I started experimenting with it.

Large language models like ChatGPT are expected to enable a new wave of research, creativity and productivity, because they can help generate solutions for complex problems. For over two years I've been exploring the strengths and limits of this?technology and assessing how this tool could be useful to me. I'm also interested how this new technology is being utilized by scientists to make meaningful contributions to academic work.?

This is the third article in a series about ChatGPT. Here are links to the first two articles about ChatGPT.

How The Evolution Of AI In Healthcare Aligns With Thomas Kuhn's Structure?

Exploring Potential Longevity Applications of Rapamycin With ChatGPT

Subscribe and Comment

I'm interested in your feedback - please leave your comments. To subscribe please click subscribe at the top of this article.

Copyright ? 2023 Margaretta Colangelo. All Rights Reserved.

This article was written by?Margaretta Colangelo.?Margaretta is a leading AI analyst consulting at Insilico Medicine. She serves on the advisory board of the AI Precision Health Institute at the University of Hawai?i?Cancer Center.?Twitter?@realmargaretta