Should you let GPT-3 do your programming for you?

You’ve probably read about it, but for those that don’t know – OpenAI released a chatbot a couple of weeks ago that is taking the internet by storm. Chat GPT is an AI text generator that was trained on billions of pieces texts and can respond to your queries with anything from essay explanations to poems (you can try it here).

Naturally, I wondered if it could do my homework for me - I wondered if it could actually code! So, I asked ChatGPT to “program a simple R script to analyze a large dataset of employee text comments”. At first glance, the results are impressive. The chatbot provides a well-documented chunk of code and even calls out some assumptions and suggestions for modifications

Trying the code

As the bot points out, the script requires data in the form of a CSV file called ‘employee_comments.csv’ so I downloaded a sample dataset from the web and saved it accordingly. I also noticed that it assumed that the column name for comments was ‘text’ – pretty standard practice but not something the chatbot was able to explain.

Running the code in R, I got an initial error – R cannot find the function “str_remove_punct(text)”. Fairly routine error but after doing some googling I found that this isn’t actually a real function!

Two things about this fascinate me – first, it’s surprising how confidently ChatGPT presented incorrect code, but more astonishing is that the function “str_remove_punct” looks and feels like real R syntax from the stringR package. It’s almost like ChatGPT has learned the logic of R syntax and is predicting iterations - maybe open source developers should use ChatGPT for ideas!

After removing this bit, the rest of the code worked perfectly and created a nice simple data visualization:

Takeaways

Mirroring broader questions that AI chatbots introduce, the question in my mind is: to what extent will programming be revolutionized by AI assistants? This experiment illustrates the value but also the limitations. For me, the value outweighs the limitations. The popular question & answer programming site Stack Overflow disagrees, at least for now, updating their moderation policies to not allow for answers generated by ChatGPT. This might be wholly necessary to preserve the integrity of the site, but for individual programmers, I suspect ChatGPT will be an invaluable tool for idea generation. Similar to how sites like Stack Overflow reduced the onus on individual programmers to create code chunks scripts all on their own, ChatGPT could further deemphasize individual ingenuity and place a premium on the ability to debug, troubleshoot, and tinker.

What are thoughts from the broader #peopleanalytics community? Do you envision using tools like ChatGPT to assist in your projects??