A New Paradigm for Paid Search and SEO in the Age of Generative AI

First and foremost, these opinions are solely my own and are not connected to my current job.

As I explore Generative AI and its potentially disruptive effects across various industries, my mind often focuses on Google. During strategy courses in my MBA program, the idea of the Innovator’s Dilemma frequently came up. Clayton Christensen, in his book of the same name, defines the Innovator’s Dilemma as the tendency of established companies to focus their efforts on maintaining their current customer base and revenue streams, which can lead them to overlook emerging markets and new technologies that could propel them to new frontiers and additional earnings potential. This idea couldn't be more relevant to Google with the rise of Generative AI technologies and the impact they’ll have on Paid Search and Search Engine Optimization.

Back in 2017, eight researchers working at Google Brain wrote a scientific paper titled “Attention Is All You Need.” The title was not only a play on the Beatles song “All You Need Is Love,” but it also outlined the Transformer architecture that is fueling the modern Generative AI revolution. While Google did create early Large Language Models (LLMs) like BERT (2018), LaMDA (2021), and Bard (2023), they never viewed it as a business in the same way OpenAI did when they released GPT-1 in 2018. OpenAI’s approach, along with its series of Generative Pre-trained Transformers (GPTs), set the tone for how natural language processing (NLP) and machine learning can be deployed at scale to consumers, which is now forcing Google to rethink its main lines of business in meaningful ways, especially Google Search.

In 1998, Larry Page and Sergey Brin submitted their groundbreaking Stanford paper, "The Anatomy of a Large-Scale Hypertextual Web Search Engine," which led to Google’s approach to evaluating web pages based on link structures (PageRank) rather than primarily on text-based similarity and word proximity. Google’s PageRank algorithm analyzed the quality and quantity of links, pointing to websites as ranking factors, contributing to domain authority. This led to the creation of AdWords, now Google Ads, which allows businesses to bid on keywords in a pay-per-click auction format that rewards relevance and domain authority, rather than primarily the size of the bid. Here’s where the disruption begins through the lens of Generative AI.

As it stands right now, the majority of Google searches are informational and don’t lead to revenue-generating click-throughs. People are able to get the information they need from various websites or YouTube videos that are served on the search engine results page (SERPs). Since consumers conduct research before making purchases, especially significant purchases of expensive items, many informational searches often indirectly lead to sales. If consumers—especially younger ones who rely on quick answers—are able to do much of their research and information gathering with ChatGPT or other LLMs outside of Google’s platforms, that could disrupt not only Google’s business but also businesses that rely heavily on search engine optimization (SEO) to get customers into their eCommerce buy flows and convert sales.

Google must now develop and perfect a hybrid model that blends their effectiveness in serving highly relevant web pages, based on their ranking and relevancy algorithms, with knowledge panels that provide instant answers from their Gemini LLM, which relies on Google’s Knowledge Graph. The big question is: Can Google continue to maintain the roughly $175 billion it earns from search ads in the age of Generative AI? Another question is: Are they willing to dedicate the internal talent, data science, and compute power needed to take on such a gargantuan task? This, while maintaining their market dominance in their current fields and competing with OpenAI, which is already advancing reasoning technology that allows ChatGPT to perform “deep research” autonomously on the web. I believe the short answer to both questions is yes—Google must continue their dominance. But they also need to focus on the long term and new revenue streams that Generative AI can bring. This is going to be a tall order, which is why they were slow to do it in the first place.

Moving forward, I believe that Google is likely working on novel ways to index the web based on what they’ve always done, coupled with retrieval-augmented generation (RAG), which enhances the capabilities of LLMs by incorporating complementary knowledge sources. In Google’s case, this includes all of what they know about customer intent that ultimately leads to purchasing behavior and the science behind moving customers through the buy flow to the confirmation page. This will also lead companies to rethink how they execute SEO, their overall content development strategies, and how they deploy robots.txt files that provide instructions to web crawlers on which pages of a website they should and should not access or index. A new paradigm will need to be developed across the industry to account for not only how LLMs crawl websites but also how customers ultimately engage with LLMs to make purchase decisions.

This is a simplified perspective on what could unfold. I'd love to hear your thoughts.

?

Sources:

The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail - Clayton Christensen. (2012, July 23). Clayton Christensen.?https://claytonchristensen.com/books/the-innovators-dilemma/

Levy, S. (2024, March 20). 8 Google Employees Invented Modern AI. Here’s the Inside Story. WIRED. https://www.wired.com/story/eight-google-employees-invented-modern-ai-transformers-paper/

Brin, S., & Page, L. (n.d.). The Anatomy of a Large-Scale Hypertextual Web Search Engine. In?Computer Science Department, Stanford University.?https://infolab.stanford.edu/pub/papers/google.pdf

Sha, A. (2024, August 15).?Project Strawberry Explained: Is ChatGPT Getting a Huge Upgrade??Beebom.?https://beebom.com/openai-project-strawberry-explained/

Shewale, R. (2024, August 30).?82 Google Ads Statistics 2024 (PPC, ROI & Conversion). DemandSage.?https://www.demandsage.com/google-ads-statistics/#:~:text=More%20than%2080%25%20of%20businesses,sales%20and%20reach%20more%20audiences.