__INPUT__: Automating Comprehensible Input with AI

As soon as ChatGPT was released, I was talking to it. It quickly became my programming tutor, and boy have I learned a lot. I had been following AI progress loosely over the years but the ChatGPT-3.5 model marked a significant change in tempo. Writing this now, all I ever hear about is the latest AI tools. I wish the algorithm would remember the other dimensions of my personality, but I'm going through a phase.

When the ChatGPT API was released, allowing developers to access the model programmatically, I was travelling in Latin America with my partner. I jumped right in. Fortunately, the weather in Medellin was quite rainy, so I had time to tap away on my little Chromebook - which has been a faithful productivity companion on this trip - rather than waste time immersing myself in a rich new language and culture.

I started building a website, hit a roadblock due to an insurmountable lack of talent, and pivoted to something I felt was more achievable - an email newsletter.

A lot of what I do in my free time is designed to help me learn something new by accident. I could never sit through introductory programming courses. There's all this Lego on the floor and I need to build something. This project was a perfect fit.

The Project

There was nothing complicated about the concept. Get ChatGPT to write me some Spanish, ask it to keep it simple, slap it into an email, add a few nice pictures, job done. I had a very rough idea of what I needed to learn, so I confidently opened the door to a dark room full of Lego bricks, no shoes, and stepped in.

The OpenAI API is very easy to work with, even for a beginner programmer. I strongly recommend anyone who is thinking about it to load up a googlecolab workbook, read some documentation on the OpenAI website and dive right in. You'll be chatting away with Super Mario, or whoever else you so choose, in no time.

Using AI Art Generation

Initially I wanted to use DALLE - OpenAI's image generation model as an automated solution for adding art to the newsletter. The problem is DALLE just doesn't produce attractive content most of the time:

MidJourney, on the other hand, does a great job. Here's MidJourney's effort on the same text prompt as above:

Now that is much more likely to instill existential dread in young aspiring artists everywhere.

MidJourney currently has no API, so this part of the newsletter is done manually. I mean I say manually... just without the years of dedicated practice required to create works of art myself. It is 4 clicks and about 40 seconds of waiting though...

Once I settled on MidJourney, I created a custom "prompt style" to create the neon, impressionist/pop art style that the newsletter uses. Mum was right, I'm an artist!

Building the Model

I decided on 5 levels of reading comprehension from beginner to near-native. I wasn't sure how to define them, so naturally I asked ChatGPT to do it. It did a great job with some minor supervision. Am I... a manager now?

The secret sauce of the ChatGPT API is the "system" prompt. This acts as a "role" for the bot to take on - whether it be talking like a pirate or explaining Modern Monetary Theory to an audience of Incredulous Simon Cowell's, ChatGPT will try to do it. My system prompt explains what I want ChatGPT to do, and includes the 5 language levels.

At first this is all I did, but it became evident that the model required more structure to give consistent results, both in terms of output length and reading difficulty.

Benchmarking the Output

I knew the outputs of ChatGPT weren't doing a particularly good job of differentiating a Level 1 language learner from a Level 3 - but I wanted a way to measure that.

After some googling (and ChatGPT-ing) I had a list of metrics for reading difficulty to use:

• Type Token Ratio (TTR): The ratio of unique words to the total length of text
• "Simple Measure of Gobbledygook" (SMOG) Index: A formula focused on the number of polysyllabic (more than 3 syllables) words in the text, calibrated to estimated number of years of education required to attain comprehension
• Flesch-Kincaid Grade Level: A linear function calibrated to years of education, using syllables/word and words/sentence as the input parameters

Because these metrics are defined mathematically, they can all be calculated from a given string of text using a simple function and some regular expressions (ChatGPT is fantastic with regular expressions, by the way).

Now I could simply loop through the model, say 100 times while we go out to see a museum or something, whatever, and measure the reading difficulty each time to build a dataset. Here's a pair-plot of the metrics, showing how they relate to one another:

Based on this data, I decided to go ahead with Flesch-Kincaid Grade Level. In retrospect I should have committed to SMOG because it has a much better name, but the two metrics share information, noted by the decent correlation between them. TTR seemed too basic a measure to be very useful, so I discarded it in future data runs as well.

Here's the Flesch level (I need an abbreviation) for the first phase of data:

The hint that the model is not doing a very good job at differentiating the 5 levels is to go temporarily color-blind and see a single population blob. The variance in the populations is very high, and with similar means there is a great deal of overlap.

I wanted to do better, so I began adding more structure to the model.

Adding Structure

I added more steps to the model to wrestle some control back from the machines. I forced some differentiation by generating a simple story for levels 1 and 2, and an article for 3-5. I rewrote the API functions to generate the content in several steps. Starting with generating an outline, then looping through it to write each section sequentially. The API has another role to use here called "assistant", which is what allows ChatGPT to maintain some context of what you are talking about between messages. Building that into a loop meant that the chatbot should naturally write out each section in sequence without repeating itself too much. In reality, I am still ironing out this part - the bot seems to love saying "in conclusion" 4-5 times in a single article, as if it's dying to get it over with. In conclusion, I still have some work to do.

To further simplify Level 1 content, I asked the model to write entirely in the present tense. I'm a little torn on this idea - my personal experience with reading a new language is that past/future tense conjugations are difficult to use, but not to take in. Your brain will recognize the verb, even if the conjugation looks strange. This is part of the beauty of reading - you will naturally become comfortable with verb constructions without doing the terribly boring ground work of memorizing them. The fact we are even familiar with the word "conjugation" is an insult to the language learning process. I don't remember when I started using "estuve" and "estaba" (meaning "I was" with different forms of the Spanish past tense), I just started doing it. Sometimes I even do it correctly!

I digress. Here's a later phase of data after adding these extra steps to the model:

Looking good! The violin plot format is a lot more aesthetically pleasing and it's easier to keep track of which distribution is which. I like the subtle lighter purple in the plot area and the inter-quartile range in the middle of each violin is also a nice visual touch... Oh, the data? Yeah, it's not great but it's definitely a bit better. The Level 1 data has come down in difficulty and is more separated from the rest of the pack, while there is a gradual increase in difficulty in the other levels. A work in progress.

I've more recently added emojis to Levels 1 and 2. I actually think this is an underappreciated application of this technology. It is something that would be incredibly annoying to do manually, so it is not commonly practiced, but emojis can really help with unknown vocabulary. Consider words like "angry", "enraged", "furious" and "apoplectic" from the perspective of an English language learner. They all mean similar things, and an angry face emoji tells you that thing is negative! Furious is phonetically a lovely sounding word. I can imagine a flower named Furious!

Another very useful and very time consuming thing to do without ChatGPT is summarizing text. So, at the end of every newsletter is a summary translated to the user's first language to assist with context. Learning a bit of Portuguese, this has been very useful for me.

That is essentially where the newsletter is up to at the point of writing this. I'm not sure how much further I will take it.

Things I Accidentally Learned

A list of things I learned while working on this project, like a drunk man stumbling across a lego-covered floor in the dark:

• How to use VSCode
• Vim keyboard navigation (still learning)
• The OpenAI API for chat completion and image generation
• More generally, what an API is and how to go about using them
• How to organize a development project
• __init__.py, creating modules, importing objects between them
• Basic HTML
• Basic CSS formatting to style HTML
• Embedding HTML in an email message (still involves praying)
• Automatically sending emails using Python
• How to create classes using Python (and why)
• Basic familiarity with Object Oriented Programming vs Functional Programming
• Maintaining a database of subscribers
• Using google forms to create a sign-up sheet
• The use of Git and Github and the importance of version control
• List and dictionary comprehension
• Creating hierarchical data objects using dictionaries within dictionaries (not sure if good idea, made sense at the time)
• Use of python plotting libraries Seaborn and Plotly
• Python decorators
• Error handling (using a custom decorator)

Something very enjoyable is to be in the middle of sending emails out to a list of subscribers, encountering an API error, starting the hour-long process again after figuring out where the error occurred (obviously you built the entire program as a single, giant loop with the immensity and majesty of the outer rings of Saturn), finding who is left on the list, and writing a bunch of ad-hoc disaster functions to send out the rest. You will learn error-handling. You will code better. Trust me. Drunk man. Legos in the dark.

What Comes Next

Now that I manage an extremely successful online newsletter with literally dozens of subscribers, I must think on what to do next.

I think once I'm happy with the back-end functionality of the newsletter, I'll build that same code into a website. I'm not sure it will be worthwhile, but I can imagine frequently visiting a language learning page that gives me customized comprehensible input each day. Maybe with some fancy on-hover effects that translate sentences and things like that.

I do believe that generative AI will be extremely disruptive to the language learning industry, and the problem of finding content is in the process of being solved. It still surprises me that the comprehensible input method is not more widely known given how effective it seems to be, but maybe the ability to generate content on demand (previously not possible without considerable effort) will mainstream it. Whether I stay on this track of building things for language learning will depend on how distracted I get with my job and other coding aspirations and also my life. We will see.

I spent a little bit less time doing data science, and a little bit more time getting my background color to just the right shade of classy purple, but it was fun. I accidentally learned a lot.

In conclusion, thanks for reading!