RAG Systems - The ultimate AI assistants for publishers

A couple of weeks ago, FT launched its first Gen-AI product for subscribers (Disclaimer - I was not part of their project). The tool, available in beta, allows users to ask any question and receive a response using FT content published over the last two decades.?The tool is a great example of how a Retrieval augmented generation (RAG) system might be the most powerful technology for publishers. They used Claude to build this RAG system and 2 weeks later, signed a deal with Open AI. There might be a story there, but I digress.

In this edition, I will talk about what is a RAG system and how can it add value to content creation, reader engagement, monetization and creating insights

What is a RAG system?

Here’s how it works. RAG systems have 2 main components -

1. Quick information retrieval: When you need specific information or data to support a story, the RAG system acts like a super-fast librarian. It quickly sifts through a massive digital library (which could be past articles, databases, or other sources) to find the exact information you need.
2. Drafting responses: Once it retrieves the necessary information, the system acts like a writer's assistant, who uses this data to write a new draft or suggest improvements to existing content.

Now let's look at how you can use it on the day-to-day of a publisher functions. Based on the 300+ AI projects run by publishers around the world, below are the top 4 use cases of the RAG system -

1. Editor assistant for generating outlines and drafts

RAG systems excel in drafting outlines and articles by retrieving relevant information from vast archives and external data sources. This not only speeds up the writing process but also enhances the depth and accuracy of the content. For example, a sports news outlet uses RAG systems to auto-generate sports event previews by pulling historical data and recent match statistics, resulting in timely, comprehensive articles with minimal human input.

You can also use RAG systems to generate summaries from your archives for something new you might be working on.

Another example of a successful RAG system is a redaction tool that can help any content creator find inconsistencies between the written content and the editorial guidelines. Imagine being able to save multiple manual reviews where an automated RAG system can find and correct content that doesn't follow the rules!

2. Engaging visitors in different formats with the same information

RAG systems can tailor content recommendations and create reading assistants that adapt to the preferences and reading habits of individual users. These RAG systems can become a useful medium for serving the user needs of your audience in an engaging format.

Financial Times is not the only one using RAG to increase engagement and loyalty. Clarín is another big brand that uses RAG systems to create Chat GPT-like summaries and contexts within their articles. NueveCuatroUno also uses RAG systems to address the audience's user needs through summaries, timelines, facts, and quotes, among other user needs for each article

They use Bridged Media for adding such turnkey RAG systems to their content

3. Creating smarter proposals

RAG systems can be employed to create customized advertising proposals for potential advertisers. The system can generate compelling, personalised proposals by accessing a database of advertising guidelines, past campaign performances, and advertiser preferences. This not only streamlines the sales process but also increases the likelihood of securing advertising contracts.

There are many companies including 纽约时报 that are using RAG for Ad Operations.

4. Easy interaction with data for creating actionable insights for product and content

Beyond content and ads, RAG systems are invaluable in generating actionable insights from unstructured data, such as session data, social media, and customer emails. By understanding the data behind each session, publishers can answer, what type of content creates maximum value for the audience. For example, the below RAG gives insights on what type of content is working, what steps can editorial or product take and the trending topics to capitalize

RAG systems also allow you to ask your analytics service to ask questions about your content performance and user behaviour in natural language.

Addressing the risks: Mitigating challenges in RAG systems

Despite the significant advantages RAG systems offer, they are not without their challenges. One of the primary concerns is the risk of generating inaccurate or fabricated information, often referred to as "hallucinations." These inaccuracies can undermine trust and credibility, which are crucial in publishing. Everyone remembers your hallucinations - just ask Forbes or Sports Illustrated

To mitigate these risks, it's essential to implement robust validation processes. Here are practical steps publishers can take:

• Human Oversight: Maintain a system where human editors review and refine AI-generated content. This blend of human expertise and AI efficiency can significantly reduce errors and maintain editorial standards.
• Feedback Loops: Incorporate user and editor feedback into the AI system to continually train and improve the accuracy of the RAG models. This ongoing learning process helps the system recognize and correct its errors over time.
• Transparent Sourcing: When using data-driven content, clearly cite sources and provide transparency about the AI's role in content creation. This openness helps build trust and allows readers to verify the information independently.

RAG Systems might be the best thing that publishers can invest in. Some solutions can help you test these systems at almost no cost.

If you are interested to know more about the AI experiments our project does with publishers around the world, feel free to send me a shout. Same if you are thinking about a RAG system (or any AI use-case) but don't know how to execute it!

About Me

I am an experienced entrepreneur who has worked in Media and Tech consulting for many years. Apart from being a full-time parent, I am also leading a project that aims to make AI more accessible to level the playing field between tech vendors and digital publishers in the Media ecosystem. Bridged’s out-of-the-box agents eliminates the need for extensive data processing or dedicated AI resources, making AI adoption accessible and efficient.