Part 1 - Applied LLMs: How I Built My Own "ChatGPT" for (Almost) Free in Less Than 24 Hours

ChatGPT is everywhere.

Already back in December 2022, we made a podcast episode on then-brand new AI chatbot from OpenAI on a show about AI applications that I host (扩博智聊 for those who are interested and have the necessary language skills to enjoy ;) The episode was really fun to make because I got to "interview" ChatGPT.

That is, I had a natural-sounding conversation with it by first interacting with the bot in text, and then synthesizing its replies in almost natural sounding generated voice.

This weekend, that old episode blew up in the largest Chinese podcast platform Ximalaya FM because it was featured on their "hottest topics" list, and none less than #1 spot.

While I wasn't busy refreshing our number of listens every hour or so, I had another, much more interesting project going on. I had discovered that I had access to OpenAI's basic APIs already since last September. It must have been at the time I got past the Dall-E waitlist. I played around with the image generation model, but never realized I could use their API. Until now.

Another enabler was Streamlit, a sort-of semi-automated and beautiful front-end generator for Python, aimed mostly at Data Scientists for visualizations, but it is easy to use and versatile enough to bend to other use-cases too, as we shall see in this post. I had known Streamlit for several years but never really used it myself, until recently got a chance to try it out on a work project and fell in love with it instantly!

Ingredients

Okay, enough about the chit-chat. I am sure y'all came here for the beef. How did I make my own "ChatGPT" in less than a day?

Let's first round up some ingredients to this spaghetti soup:

• OpenAI API access for the responses
• Docker and Microsoft Azure for hosting and deployment of the app
• Python, Streamlit for development
• Git (GitHub) for version tracking
• A credit card (for OpenAI and Azure)

The only part in this soup that is not entirely free is the OpenAI API, although when you first get access, you do get $18 USD worth of credits to play around with. Unfortunately, I didn't even realize I had this until six months later, so they all expired. Yes, even my credit card is free of any annual fees even if I don't use it.

We use Azure App Service to host and deploy our stand-alone chatbot service once it's been developed. You can register a free Azure account (that comes with $200 USD worth of credits) but actually the App Service has an entirely free tier even after those credits expire -- that's what I am using. It has quotas on CPU, memory, bandwidth usage and such, but for small-scale testing, it's more than enough.

Docker is used to package our Python code so that it's easy to deploy in production. I probably need to write a whole another article on Docker and containerization, so for now, suffice to say it's just a neat tool to package each app separately with only the code that it needs, so we can run it not just on our own computer, but pretty much everywhere.

Finally, I already mentioned Streamlit above. It's actually just another Python pip package you can install easily. And when you write your service with Streamlit commands and run the code through it, it automagically generates both the back-end server and front-end code so everything looks modern and nice on the browser.

Tricking The Model

There are quite many examples, documentations and even a playground for you to hack away on OpenAI's documentation site. One of the toy examples on the playground is indeed a chatbot experience. Since (as of Feb 12, 2023) OpenAI has not yet released ChatGPT specific APIs, we need a hacky way of using OpenAI's existing language model APIs to enable the chat functionality on what is essentially a "text completion" task (you give the model some input text and hope it will auto-complete or at least continue writing in the same style).

Just as you can trick ChatGPT into thinking it's role-playing something entirely different than ChatGPT itself, we can trick the ordinary GPT language completion model into thinking it is a chatbot. This is done using clever prompt engineering when feeding the chat log into the model. The playground example shows nicely how it's done so I won't go into details here. Besides, playing around with the prompt yourself is the most interesting and rewarding part in my opinion.

Integration with Streamlit

The OpenAI documentations provide good sample code how to directly integrate their API, which is also available as a Python pip package openai, so when we are designing the service, we can put it all together with the Streamlit code that generates the user interface.

Here, we use a relatively simple design schema that includes the main area where the chat messages from both human and AI will appear, and a text input form that allows the human user to enter new text. Streamlit's documentation is also very sleek and well made, so I won't delve into details here, but I want to highlight one thing: persisting data throughout user interactions.

Because Streamlit is built so that every user interaction reruns the page, we cannot save any information in normal Python variables, even if we initialize them in the beginning of the code in a global manner, because after each interaction, Streamlit will rerun the entire code from top to bottom again, thus writing over anything that might have been generated in the "previous" cycle. In our case, it would cause the UI to forget any previous chat history.

To overcome this limitation, there is a special streamlit.session_state object which kind of acts like a normal Python dictionary but persists its state over multiple rounds of user interaction. It only gets reset if the user reloads the entire website manually.

import streamlit as st

if 'CHATLOG' not in st.session_state:
    st.session_state['CHATLOG'] = [
        "Your first item in the chat log"
    ]        

Another thing I love about Streamlit is the way it really saves time when doing rapid iterative cycles, because any changes you make in the code can be seen almost instantly on the browser after you save the file. No need to stop and restart the server, then reload the page.

Below is the first iteration of the chatbot after just mere hours of tweaking.

Some of you have noticed that the UI of the app is predominantly in Chinese. There's a reason for that. OpenAI has barred Chinese users from accessing ChatGPT, Dall-E and other AI services, and even from registering on their website. For example, the SMS verification will fail if the user enters a Chinese mobile number.

Since ChatGPT craze is immense right now, I am trying to build a way for the Chinese users to enjoy a ChatGPT-like experience, albeit not exactly the same. The situation here is actually so crazy that I hear some people are auctioning access to their foreign phone numbers or OpenAI accounts registered using spoofed overseas numbers because so many people in China want to try this revolutionary tech.

Containerize

When we have finished tweaking the code of the service locally, we can simply containerize or Dockerize it. It means setting the code up in a brand new mock environment that is isolated from your main computer. We build the Docker image by starting off with a minimal Python environment that's only about 40MB in size. Then we only install the necessary dependencies for our app to work, in this case, pip packages openai and streamlit.

Unfortunately, because Streamlit is versatile, it is also a bit heavy, coming with tons of dependent pip packages of its own. Therefore, in the end the total size of the Docker image ballooned up to almost 500MB when all the packages were installed.

We did our development by setting up a GitHub repository so it was easy to transfer the necessary code files into the Docker image that we set the environment up for earlier.

The Dockerfile looks like this:

FROM python:3.7-slim

ARG DEBIAN_FRONTEND="noninteractive"

ADD __app.tar.gz /app
WORKDIR /app

RUN <<EOF
set -e
python3 -m pip install --no-cache-dir --upgrade pip
python3 -m pip install --no-cache-dir -r requirements.txt
EOF

CMD streamlit run src/app.py        

The part where code gets transferred in is the "ADD" command which not only copies files into the image but for the right kind of files such as .tar.gz will automatically extract their contents. How we came up with that compressed file is by using Git:

git archive -v -o __app.tar.gz --format=tar.gz HEAD        

Be sure to add and commit all the latest files into HEAD before running the above command, it will only archive the files that are tracked by Git.

I uploaded my final image on the Docker Hub (they offer one private image repo for free). Docker Hub is also supported by Azure when doing the deployment, which we will talk about next.

Deployment

I am sure Azure isn't the only alternative when doing the App deployment. Any of the big A's (Amazon, Azure and Alphabet aka Google Cloud Platform) and probably even the Chinese equivalents like Alibaba or Tencent cloud could do. I picked Azure because I already got a personal Microsoft account, and was used to using it at my work.

Check out the tab called "Free services" once you're inside the Azure Portal, which lists all the limited-time free and permanently free services available to anyone. Azure App Service is one of those. As mentioned earlier, the free tier enforces certain usage quotas on CPU, memory and bandwidth (or rather, data flow) but for our little experimentation, it's well enough.

Here, one thing to remember is that by default, Streamlit will set its http server listening port as 8501, so it must be configured on Azure App Service as well, by adding a key-value pair of WEBSITES_PORT=8501. Hint: these App Service configuration key-value pairs act as environment variables which are fed into the app's running Docker container, so whatever sensitive information like OpenAI org IDs or API keys can be configured on the App Service's portal page and delivered to the app that way instead of hard-coding them in the script files which you should never do.

If you used Docker Hub as the place to host your app's Docker image, these credentials are also configured on the App Service page so that it can pull the actual app's code and run it on Azure.

Conclusions

So how much does it actually cost to serve a ChatGPT-like experience?

Actually, not that much (yet) on a couple day's worth of very light self-testing and sharing with a couple of friends.

Here's the OpenAI breakdown:

And here's the usage quota on Azure (which is free, remember)

Next Steps

So far, the chatbot is already running smoothly and helping me learn Italian, for example. Compared with the bare-bones version shown above, you can see the UI has already undergone pretty nice improvements and looks more like a chat interface. I managed to do all this within 24 hours, while starting off with nothing and having to figure it all out by myself. Now that I've practically revealed some key steps, why don't you try it?

We are still on the money losing side because we need to pay the OpenAI API fees (and eventually for Azure App Service hosting if traffic gets higher) to run the chatbot.

But still, there are so many Chinese users wanting to experience ChatGPT, surely there's a way to monetize all this interest, right? Remember the hockey stick from the very beginning? Yeah, that looks enticing...

Next, I am going to investigate integrating WeChat, China's own superapp, so that the users can sign on with their credentials, and use Xiaopan as distinct users with their own conversation quotas etc. I am also envisioning you could use WeChat Pay or Alipay to buy more chat tokens to keep interacting with the bot. Like OpenAI and other generative art services like Midjourney and Stability AI did.

Thanks for reading! What do you think of this project?

As a token of gratitude, here's a little easter egg for anyone who made it this far in the article: You can actually try out my bot at https://chat.xiaopan.ai but please refrain from sharing it to the wider audience just yet. Although the UI is in Chinese, you can just start typing in English or any other language. Try it out!

Edit 2023/03/19: Part 2 of the article series is out!

Edit 2023/04/03: I ended up paying about $10 for the month of February, and for March, the tally ended up at $35. The cost really started to go up after GPT4 API came out, as mentioned in Part 2.