Perplexity.ai Overhauls Traditional SEO Approach for the Age of Large Language Models

The launch of ChatGPT on November 30, 2022, was a landmark event that garnered significant attention and critique. It highlighted a growing demand for AI-driven solutions to traditional search queries, a demand that ChatGPT struggled to meet due to issues like outdated information, lack of source citations, and inaccuracies in responses. Emily Bender, a prominent figure in computational linguistics, likened the expectations of AI search capabilities to a "Star Trek fantasy" of an omniscient computer.

Amidst these challenges, a startup named Perplexity embarked on a mission that would shift its focus from a Text-to-SQL tool to creating an AI-powered search engine. Founded in August 2022, Perplexity leveraged the unexpected potential of a chatbot developed in their Slack channels to build a search engine that updates daily and cites multiple sources for its answers. Today, it boasts over 10 million monthly users and has caught the attention of investors like Jeff Bezos.

Denis Yarats, co-founder and CTO of Perplexity.ai, initially envisioned the project as a simpler tool for developers to interact with SQL databases. However, the advent of ChatGPT and its rapid growth to 100 million users within two months revealed a broader application for their technology. Despite the original Slack bot's limitations, its popularity hinted at a significant demand for improved AI-driven search capabilities.

Perplexity's pivot to focusing on an AI-powered search engine was driven by the limitations of existing AI for search applications. Through adopting retrieval-augmented generation (RAG) technology, which pairs generative AI with a data retrieval system, Perplexity aimed to overcome these obstacles. This technology, developed by researchers from Meta, the University of London, and New York University, allows for more accurate and relevant search results by leveraging a vast database of information.

Moreover, Perplexity developed its web crawler, PerplexityBot, to maintain an up-to-date index of the internet, employing techniques such as domain-specific updating frequencies to manage the vast amount of data online. The use of Google's BERT model further enhances their ability to understand and rank web pages effectively.

Yarats acknowledges the challenges of competing with Google, particularly due to the constraints imposed by Google's ad-based business model. Unlike Google, Perplexity can focus solely on delivering accurate and relevant search results without the need to optimize for ad revenue. This allows Perplexity to experiment and innovate in ways that could redefine the future of search, emphasizing the potential of AI to transform how we find and interact with information online.