Auto-GPT – Promise versus Reality

The Promise

The excitement when Auto-GPT was released was really high with a range of YouTube videos and Twitter posts raving about the latest AI tool. The first YouTube video I saw was ‘Auto-GPT Tutorial – Create Your Personal AI Assistant’[1] and has had 119 k views in the nine days since it was released. It sounded exactly like what I had been hoping to build myself.

Auto-GPT was open-source software released under the MIT open-source licence.[2] It has generated so much interest that it was the number one repository on GitHub in the last week.[3]

Why has Auto-GPT been so popular?

There are a number of reasons for the excitement and its hard to rank these reasons. The GitHub repository ‘Significant-Gravitas / Auto-GPT highlights the following features:

Other features that are often mentioned on YouTube include.

• Enabled anyone not taking part in the ‘alpha’ trial of OpenAI ChatGPT plugin’s – especially the chatgpt-retrieval-plugin a way to broaden your searches/prompts to look for information anywhere on the Internet if ChatGPT did not have the information available.
• Showed how AI Agents can be used to meet a user’s goals (many think AI agents are an essential component of useful AI tools)
• Easy to install and run with the only requirement that you have an OpenAI API key.
• Auto-GPT can run fully autonomously – you simply set some goals for the AI, and it will then keep going until it crashes or find the results/answers that meet these goals.

How to use Auto-GPT

It is really easy for non-programmers to use. The instructions on the website have some additional options for selecting a ‘memory’ backend which do require more than just following the install instructions, but most people won’t start with doing this.

Why was I excited with Auto-GPT?

The key limitations with GPT-4 require the functionality that OpenAI is currently trialling in its ‘alpha’ testing. They have a ChatGPT Retrieval Plugin that ‘…provides a flexible solution for semantic search and retrieval of personal or organizational documents using natural language queries.[4] This plugin will be part of a wider microsystem that OpenAI are building – likely to be the equivalent of Google Play or Apple Store but for plugins that use GPT-X as the user interface and eventually the operating system for any apps that are built on it.

There are alternatives to seeing OpenAI dominate the AI future. Microsoft surprisingly have another project on GitHub called HuggingGPT (or Jarvis) which is very exciting. This project uses ChatGPT as the entry point but then accesses all the open-source models and datasets on HuggingFace Hub.[5]

The Microsoft Jarvis repository states ‘This project is under construction, and we will have all the code ready soon.’ Assuming that you are not restricted to using the Azure platform this is a really powerful approach and, in my mind, uses ChatGPT as the user interface to the rich, open source world that already exists and avoids the lock-in to an OpenAI ecosystem that will develop once regular access to their plug-in ecosystem enters beta.

I agree with the approach advocated by Stability.Ai as stated below with the proviso that we want many open-source AI projects continuing to thrive with at least one of them reaching critical mass sufficient to give us all a real alternative to being locked into one company’s ecosystem. AI is going to change the world and no company should ‘own’ this future.

Problems with Auto-GPT

Initially, I thought the biggest problem I was having was caused by the fact that I am still waiting on access to the API for GPT-4. The major increase in functionality available in GPT-4 may fix some of the issues I was having using Auto-GPT. To test this, I used the same prompt using GPT-3.5 and GPT-4 in a web browser. I then entered this prompt as the only ‘goal’ I set in Auto-GPT:

I would like to find out which models, datasets, and spaces that use ChatGPT were the most popular in 2023 on HuggingFace. Can you tell me how I can query HuggingFace to get this information?

Both GPT-3.5 and GPT-4 responded by writing Python code for me to use to query the HuggingFace Model Hub API.

GPT-3.5 Responded:

Yes, I can help you with that.

HuggingFace provides a public API that allows you to query its model hub and obtain information about the popularity of different models, datasets, and spaces that use ChatGPT. You can use the following steps to query HuggingFace's API and obtain this information: (followed by 7 snippets of code).

GPT-4.0 Responded:

To find out the most popular models, datasets, and spaces that use ChatGPT in 2023 on Hugging Face, you can use the Hugging Face API. Here's how you can do this using Python: (with 2 snippets of code).

The code provided by the two models was quite different. What was the same though was that neither model could produce working code at the first attempt. I have found that I almost always have to ‘iterate’ code provided by GPT-4 before I get a working version.

Setting up Auto-GPT:

My prompts follow:

Name:?HuggingFace

Role:?Expert AI researcher

Goals: ['I would like to find out which models, datasets, and spaces that use ChatGPT were the most popular in 2023 on HuggingFace. Can you tell me how I can query HuggingFace to get this information.']

Continue (y/n):

Rick’s Goal

1: I would like to find out which models, datasets, and spaces that use ChatGPT were the most popular in 2023 on HuggingFace. Can you tell me how I can query HuggingFace to get this information?

Goal 2: (left blank – start Auto-GPT)

Using memory of type: LocalCache

POPULAR_HUGGING_FACE THOUGHTS:?I think I need to start by browsing the HuggingFace website to see if they have any information on the most popular models, datasets, and spaces that use ChatGPT.

REASONING:?Browsing the website is a good starting point as it will give me an idea of the information that is available on HuggingFace.

PLAN:

-?Browse the HuggingFace website to see if they have any information on the most popular models, datasets, and spaces that use ChatGPT.

CRITICISM:?I need to ensure that I am thorough in my search and take note of any relevant information that I come across.

NEXT ACTION:?COMMAND = browse_website ARGUMENTS = {'url': 'https://huggingface.co/', 'question': 'What are the most popular models, datasets, and spaces that use ChatGPT?'}

Enter 'y' to authorise command, 'y -N' to run N continuous commands, 'n' to exit program, or enter feedback for Popular_Hugging_Face...

Input: y -20”

Setting the input to y -20 sets a limit of how many steps the AI can run without coming back to the user for additional okays to continue searching.

...

POPULAR_HUGGING_FACE THOUGHTS:?Since I was not able to find the information, I need on the HuggingFace website, I will now try to query the HuggingFace API to get the information I need.

REASONING:?If the information I need is not available on the HuggingFace website, then querying the HuggingFace API is the next logical step.

PLAN:

-?Query the HuggingFace API to get information on the most popular models, datasets, and spaces that use ChatGPT.

-?Take note of any relevant information that I come across.

CRITICISM:?I need to ensure that I am thorough in my search and take note of any relevant information that I come across.

?____________End Extract from AutoGPT___________________________

The real problems at present with Auto-GPT

Failure to finish the goal(s) set for Auto-GPT.

Initially I had more success with the software. It seems as if it ‘remembers’ the last task you set it, even when you ask it to start again. I have had instances where I have run Auto-GPT for up to 150 cycles only to run into some issue that stops it finishing the goals I set it. I have tried setting only one goal at a time but with no success. What I haven’t done, and I should have, is to check from any later versions of the software to see if this helps with the problems I am having although reading the Reddit Auto-GPT forum[6] and the??Auto-GPT forum[7] on GitHub shows that many others are having the same issues.

Running Auto-GPT autonomously

Even when you run Auto-GPT with a limit to the number of steps the agent/s can run you are allowing the software a lot of power over your local system. The the agent opens webpages and searches them as it thinks are needed:

Giving Auto-GPT power over your local system

Auto-GPT will attempt to ‘fix’ issues that are stopping it trying to meet its goal without any understanding of risks to your network or PC as the example below shows:

___________________________________________

POPULAR_HUGGING_FACE THOUGHTS:?It seems that the issue with the SSL certificate still persists. I will try to resolve this issue by updating the certificate again.

REASONING:?Updating the certificate again should resolve the issue with the SSL certificate.

PLAN:

-?Open the terminal.

-?Run the command 'sudo apt-get update && sudo apt-get install openssl' to update the certificate.

-?Retry the command to query the HuggingFace API.

CRITICISM:?I need to ensure that I am able to update the certificate correctly and that the SSL certificate issue is resolved.

?At this point my 20-step prompt ended and I decided not to allow Auto-GPT to proceed any further.

?

Cost of using Auto-GPT

Even using the GPT3.5 API the costs incurred when using Auto-GPT can be expensive. The software often seems to get in loops where it keeps trying to resolve an issue using the same approach. I am not sure if this would change if it was using the GPT-4 API.??

The personalities of Generative AI

I know I am not alone in finding personalities in these AI models. When you deal with GPT-4, BingChat and Auto-GPT it is hard not to ‘humanize’ them. I find I am very polite and thankful with GPT-4 and find BingChat not helpful most of the time when it answers my queries. Auto-GPT offers so much promise and its like dealing with a really chatty, verbose human who mainly gets things wrong. Reminds me a lot of myself when I am really excited about Generative AI.

I also feel that BingChat’s ‘personality’ is more abrasive than GPT-4 and I tend to be more abrupt in my dealings when using it. Very few ‘thanks’ in my follow-on questions and ‘no’ that not what I want when trying again to get BingChat to answer what I asked. It is amazing how I treat these AI’s as ‘entities’ – not conscious or AGI’s but something that you interact with and personalise.

Conclusion

I am still impressed by Auto-GPT, and I can see that it may get to the stage I want to use it as a way of developing a ‘second brain’ as it already demonstrates some amazing functionality and like many other users I am excited by an open source project that has such lofty goals.

I really want to send my thanks and appreciation to the developers of Auto-GPT. It's an amazing start and such an inspiring project. My criticisms are one's that would apply to almost any 'alpha' project and are designed to offset some of the YouTube videos and articles that only see the positives.

Before I will spend more time using Auto-GPT though, the following conditions need to be meet:

1. I install it in a docker image – not directly on my PC as watching Auto-GPT restart my PC after trying to install some software was enough to convince me that it’s a really bad idea to give any ‘alpha’ software this much control. [8]
2. I will never run it autonomous mode – I will always set limits on my use of Auto-GPT for both cost and safety issues and to have some control over what it does when it tries to meet the goals, I have set it.
3. I will only use Auto-GPT when I have access to OpenAI GPT-4 API as I think that will improve the functionality of the AI significantly.
4. I will check out the discussions on Auto-GPT on the Reddit and Auto-GPT forums to see what issues people are still having. Forget the clickbait headings on YouTube or in articles on the web. Unless you do a deep dive, you will never have a good idea how/good bad AI software is. Seeing other people get working code for applications using Auto-GPT or GPT-4 is not something that I have seen happen without a lot of extra steps to fix issues with the code.

?

?

?

?Endnotes

[1] GitHub, Auto-GPT Repository, visited 23 April 2023

[2] Dave Ebbelaar, Auto-GPT Tutorial – Create Your Personal AI Assistant, visited 15 April 2023

[3] MIT Licence – BingChat explanation ‘The MIT License is a permissive free software license originating at the Massachusetts Institute of Technology (MIT) in the late 1980s1. It is a permissive license that puts only very limited restriction on reuse and has, therefore, high license compatibility1. The MIT license gives express permission for users to reuse code for any purpose, sometimes even if code is part of proprietary software2. As long as users include the original copy of the MIT license in their distribution, they can make changes or modifications to the code to suit their own needs2. The primary terms and conditions of the MIT license are to grant permissions and indemnify developers for future use2. Specifically, it grants any person who obtains a copy of the software and associated files the right to use, copy, modify, merge, distribute, publish, sublicense, and sell copies of the software2.’

[4] GitHub Trending page, visited 23/4/2023.

[5] OpenAI, Github repository, chatgpt-retrieval-plugin, visited 23 April 2023.

[6] HuggingFace, Build, train and deploy state of the art models powered by the reference open source in machine learning – with more than 5,000 organizations using Hugging Face, visited 23 April 2023

[7] GitHub, Microsoft / Jarvis (HuggingGPT), visited 23 April 2023

[8] Reddit, AutoGPT: Automating GPT Model for Natural Language Generation and

Significant-Gravitas/Auto-GPT, Discussions, both visited 23 April 2023

GPT-4 explanation: "Containers created from Docker images are isolated from the host system and other containers. This makes it easier to manage dependencies and avoid conflicts between different applications or versions."

Rick Molony, 23 April 2023?

Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)

Attribution Non-commercial icon?