AI at a Crossroads: The Open vs. Closed-Source Debate and the Future of Large Language Models

As we approach the end of 2023, a year that will undoubtedly be remembered as the 'Year of LLMs', it's fascinating to look back at the major milestones in AI. OpenAI launched their GPT-4 early in the year. This was followed by Anthropic's various releases of their model, Claude. Alongside these developments, numerous other proprietary language models emerged, considered as closed-source. Concurrently, we witnessed an influx of open-source models from different sources almost every month:

side bar: those who are not familiar with all these advancements of open-source LLMs, here are additional datapoints:

In February, Meta launched its first major open model, LLaMa, primarily for research and not for commercial use. Following this, April saw Eleuther AI release a range of 16 LLMs, varying from 70M to 12B parameters and trained on public data. This move was a game-changer, inspiring further research-based model development. MosaicML's introduction of the commercially viable MPT-7B model in May, which was developed at a significant cost but with high efficiency, caught the attention of industry giants like Databricks.

Hugging Face's collaboration with ServiceNow to create the StarCoder model and Salesforce's cost-effective X-Gen 7B were notable developments in June. TII UAE also made waves with their Falcon models, licensed under Apache 2.0 to encourage broader LLM adoption.

July heated up the competition as Meta released LLaMa 2, offering commercially viable licenses and sparking direct comparisons with TII UAE's Falcon models. The global landscape expanded with Alibaba’s Qwen model from China and Mistral's entry from France with their 7B model.

01.ai's bilingual Yi model, launched later in the year, demonstrated remarkable performance, challenging even established models like GPT-3.5 and LLaMa 2.

As the year wound down, Deci.ai prepared to launch a highly efficient 7B LLM. However, UpStage stole the show in December with their 10.7B model, topping the Hugging Face LLM leaderboard.

There is significant investment flowing towards building closed-source or proprietary LLMs, as evidenced by developments like GPT-4 from OpenAI, Anthropic's models, and others from companies such as AI21 and Cohere. At the same time, there is a notable momentum in the AI community towards developing open-source models.

So one may ask a question:

Should the future of AI be open-sourced or closed-sourced?

To answer that question, we need to look back at the evolution of our tech industry over the last 25 years. I remember using SCO-Unix on servers and DOS on IBM PCs during my college days; both were predominantly closed-source systems, reflective of the wider industry at the time. However, in the mid-90s, there was a shift towards Linux, an open-source operating system created by Linus Torvalds, primarily for server use. By the early 2000s, the LAMP stack (Linux, Apache, MySQL, and PHP) had become a staple, epitomized by its adoption by major startups like Facebook (now META), which I believe still uses some PHP in their main website infrastructure. Linux, in particular, has become a dominant force in server operating systems, a testament to the potential of open-source software. This period also witnessed significant developments in technology sectors such as databases and big data, exemplified by the Oracle vs. MySQL dynamics and the rise of Hadoop. These evolving landscapes set the stage for today's race in AI development.

Let's look at some of the pros and cons of closed-source and open-source LLMs.

Closed-source LLMs:

Pros:

• Controlled and maintained by the company who is building it
• With enough funding to do the research, they can break the barriers to achieve new highs and set a bar for others to reach - e.g. GPT-4 is still the best performing model (or maybe Gemini now?) but Claude has the most safety!
• They are in business of making money so they must advance their system to keep up in this fast paced ecosystem.
• Charged with the right intention (and the right board members!) they can do wonders and leave competition behind.

Cons:

• Controlled by the company who is building it
• Lack of transparency and potential bias in data and model (see New York Times suing Open AI for scrapping their website)
• Not much flexibility to customize or influence new features
• No intellectual property (IP) ownership rights

Open-sourced LLMs:

Pros:

• Controlled by community
• Full control and some visibility (some only open source the model weights while others also open source the data - they all should follow LLM360 framework)
• IP ownership (for permissive licensed models, such as Apache 2.0, MIT licenses)
• Easy access via Hugging Face or Amazon SageMaker Jumpstart

Cons:

• Who will maintain the code? One significant challenge for open-source LLMs is ongoing maintenance. Once a model is released to the community, the responsibility for its upkeep becomes collective. Organizations like the Apache Software Foundation or Linux Foundation face constraints, especially in funding and resources, making it challenging to support a burgeoning field like LLMs consistently. Alternatively, the originating company could take on the maintenance responsibility, but this raises concerns, particularly for startups. While well-funded, startups are inherently more volatile, as evidenced dot-com bubble. Interestingly, the success and popularity of Meta's LLaMa 2 model could be partly attributed to the stability and resources of its backing company, underlining the importance of robust support for the longevity of open-source models.
• With proliferation of so many open models, how one can decide which one to use? Not everyone is on HELM benchmark or Hugging Face LLM leaderboard. [warning: biased opinion ahead] There are model evaluation tools (and service) offered by AWS to make some of these easier but it's still an evolving field.

If history is any indicator, we might see significant consolidation in the open-source AI model landscape in the coming years, with only a few key players emerging as leaders. I am particularly optimistic about META, given their size and impact through contributions to PyTorch and the development of the LLaMa models. Similarly, TIIUAE, backed by the Abu Dhabi government's research council, with their investments in the Falcon models (7B, 40B, 180B), seems well-positioned. We might also witness the rise of a few startups, like Mosaic (well, now Databricks) and Mistral - mainly due to their novel architecture. [Disclaimer: These are my personal speculations based on current trends. I do not own stock in META nor am I affiliated with any of the mentioned entities.]

In the realm of closed-source models, we can expect continuous advancements and an increasing number of companies launching their proprietary models.

So, returning to the initial question: Should the future of AI be open-sourced or closed-sourced? In my opinion, it's too early to tell.

Let's allow the developments of the next 6-12 months to unfold, and then revisit this question in December next year.

What are your thoughts on the future direction of AI?

Wishing you all happy holidays and a joyful New Year!

Shameless Plug Alert! ??

Before you go, I can't resist sharing something I'm incredibly excited about — our brand-new AI Fundamentals Video Course! Designed for curious minds eager to delve into the world of AI, this course is a treasure trove of insights and knowledge. Whether you're starting from scratch or looking to brush up on the basics, our expert-led sessions are tailored to make AI approachable and fascinating.

Here's the kicker: You decide the price! That's right – we value your learning experience and commitment more than fixed fees. Pay what you feel is fair, and embark on a journey that demystifies AI in a practical, engaging way. Don't miss out on this unique learning opportunity – it's tailor-made for enthusiasts like you!

Check it out and jumpstart your AI adventure today!

Interest sign up page: https://bit.ly/ritesh-ai-bootcamp-signup