Did OpenAI just announce the next App Store?

Yesterday, OpenAI shook up the tech sector — again.

The AI company announced ChatGPT Plug-ins, vastly extending ChatGPT’s capabilities. OpenAI’s latest strategic play will alleviate some of the many shortcomings associated with ChatGPT and its underlying technology (generally, large language models, or “LLMs”).??

ChatGPT plug-ins could cause the same industry-wide transformation as when Apple opened the App Store. It may be that impactful. We don’t know yet.

But there is a lot that we do know.?

Just as users can “shop” the iOS App Store or “shop” for Chrome or Gmail plug-ins, OpenAI’s innovation lays the foundation for a similar marketplace — one where users can browse business-specific ChatGPT plug-ins to more deeply personalize their digital experiences.?

But first, why do plug-ins matter?

Product practitioners will be familiar with the term “plug-ins.” For those newer to the space, plug-ins are software extensions that add new functionality to existing programs (a popular example would be Grammarly’s Gmail plug-in). In essence, plug-ins are to software what apps are to the iPhone.?

Plug-ins will allow ChatGPT to determine how to respond to a user’s prompts. Should ChatGPT perform the request natively (e.g., within its internally available data set)? Or should it obtain real-time information from an external source before answering? Now, ChatGPT can proactively make that choice.

Plug-ins enable ChatGPT to gather and translate real-time data.

Plug-ins address one of ChatGPT’s significant shortcomings: historically, the platform couldn’t access data made available after September 2021.?

ChatGPT’s initial “knowledge base” was built according to “training data,” or the massive amount of text available on the Internet, in books, and beyond — but it was text that existed? before September 2021. Remember: ChatGPT’s knowledge base doesn’t mean that the LLM knows anything — it just helps the platform better predict how to respond to user prompts.

When faced with a more time-sensitive request, ChatGPT can now incorporate real-time data from external sources, including the web. (To be fair, Google’s Bard can already do this — as can Bing’s GPT-4 implementation.)

Another major historical shortcoming of LLMs? In the past, they haven’t been great at computation or quantitative reasoning. Offloading that work to a plug-in like WolframAlpha has the potential to close the gap.

Soon, ChatGPT will create multimodal outputs.?

By “multimodal,” I mean a combination of text, images, and video. Users might already be familiar with text-to-image services such as OpenAI’s DALL-E 2 and Midjourney. Text-to-video is also already possible, though less publicly available.?

Soon, ChatGPT will be able to digest inputs and provide outputs in multiple formats — a vast improvement from its current state of only providing text and code outputs via its chat-like interface. One notable business application will be ChatGPT’s ability to generate data visualizations based on user input. (More on that shortly.)

This innovation drives new value for businesses in any vertical.?

Companies can now create their own plug-ins to connect their services to ChatGPT’s services and create new tools that leverage their business data in tandem with ChatGPT’s latest innovations.?

And that’s all in addition to the value businesses can unlock by using APIs from OpenAI, Google, and others to integrate LLMs into their proprietary apps and workflows.?

Take these three use cases as examples:

Case Study #1: Travel?

Imagine you want to book a vacation in a specific location and during a particular timeframe. You also want to arrive at this location via your preferred airline. ChatGPT’s KAYAK and Expedia plug-ins will allow you to do precisely that — and then, it can act as your travel agent and book the vacation for you.?

Case Study #2: Real Estate

Maybe you’re shopping for your next home. You want to know the latest mortgage interest rates and homes for sale in your zip code within a particular price point. A ChatGPT integration with a personal finance or real estate app could help answer your query. Then, you could refine your question and change parameters around your intended down payment or preferred zip code, for example.?

Case Study #3: Health and Wellness

Personal data is an emerging space in the AI sector, but we’re beginning to understand what might be possible. Let’s say you use an app or wearable that captures your sleep data. With a ChatGPT plug-in that integrates data from your wearable or app, you could inquire about your recent sleep patterns or see data visualizations of your habits directly in ChatGPT. These new capabilities will enable more personalization and more informed AI interactions.?

What announcements might come next??

The pace of change in the broader AI industry is staggering. Here are some signs alluding to what might come next:?

More multimodal.?

While OpenAI’s announcement speaks to this area, we’re seeing a lot of momentum around multimodal models and applications.?

Beyond what we’ve encountered with ChatGPT and GPT-4, DeepMind has experimented with a “Generalist Agent” called Gato that seeks to leverage a single model performing multiple tasks — including playing games, captioning images, and stacking blocks with a robotic arm.?

More specialized skillsets.?

While ChatGPT revealed the cross-domain power of LLMs, more models are in development that will align with specific domains.?

A great example is Google’s Med-Palm 2. It’s an LLM tuned for medical knowledge. Recently, Med-Palm 2 answered questions from the US Medical License Exam with an 85% success rate. More domain-specific models will likely pop up where highly specialized skill sets are needed.?

More ethical considerations, such as open-source tech.

Finally, keep an eye on open-sourcing this technology. We’re seeing praise and criticism in this area.?

On the one hand, the tech is mainly in the hands of an already powerful few. Democratizing the technology’s capabilities could prove beneficial.?

On the other hand, open-sourced technology could be leveraged for more nefarious purposes.?

Ethical considerations are already mounting for LLMs and tech like ChatGPT (read the recent TaskRabbit story, for example).?

With fewer guardrails, we increase the potential for harm.?

WillowTree is ready to help.?

With announcements and innovations like these happening nearly every week, it’s a lot to make sense of.?

Wondering how you might be able to leverage new AI technologies? We’d love to hear from you.?