Personalizing Content @ Edge with GenAI

Today's consumers expect relevant tailored content the moment they engage with a brand.

As personalization becomes table stakes, companies need to deliver hyper-personalized experiences in real-time to each visitor.

But todays personalization truly isn't... personalized. Often times companies maintain multiple variations of content and try their best to assign the right variation to the right user.

In this article you'll learn and ponder about...

• Art of possible for real-time personalization
• How GenAI can get you there, with limitations
• Gotchas and current tech stack problems
• It's just a fun concept.

The Power of Instant Personalization

By running at the Edge, GenAI creates new opportunities to customize the user experience across industries:

• Ecommerce sites can tailor product recommendations and descriptions for each visitor. Someone from San Francisco may see relevant local promotions.
• News platforms can automatically recommend articles based on a reader's interests and location.
• Video streaming can dynamically customize thumbnails and suggestions to individual viewing history.
• Gaming networks can adjust game interfaces, difficulty, and rewards based on player profiles.

The use cases are endless. Edge AI enables instant personalization by only using local cached context, keeping each interaction smooth, relevant, and fast.

The future is LLMs @ Edge

To create truly personalized, on demand content then companies will have to create GenAI (LLMs) architectures right at the edge to instantly generate customized content.

LLM models embedded in browsers

The clear-as-mud Edge solution is embedding GenAI mini models (mb in size) directly into browsers and on-premise servers.

Browsers often store or have access to user content such as

• User location data
• User tracking data (clicks, page views etc)
• Demographics data

This user-adjacent/on-device AI analyzes user context and generates personalized content - without needing to send requests back to a central server.

Here are a number of opensource projects working towards embeddable LLMs:

• WebLLM - Embedding models within your browser
• transformers.js - built to replicate HuggingFaces Transformers.py
• https://huggingface.co/Xenova - list of embeddable models and samples

There's no silver bullet solution or consistent reference architecture YET, but we are actively racing towards it.

Overcoming the Limitations of Edge GenAI

Batteries not included

While promising, squeezing LLMs down to the Edge introduces some constraints:

• Smaller model size - Browser-based models must be incredibly small or compressed. Assume the use of lightweight, fine-tuned foundation models.
• Latency - Edge devices often lack GPUs. GenAI architectures run fast on CPUs.
• Use case specific - Models purpose built for a specific use case can help reduce model size and performance.
• Intermittent connectivity - Models may need to work offline.
• Privacy - Only local context is used, avoiding external data privacy issues.

By creatively tackling these challenges, GenAI unlocks the potential of real-time hyper-personalization.

CDN powered Semantic-cache?

It may take years for LLM models to be small and efficient enough, so instead we could try something... different.

• Sophisticated semantic cache at the CDN (content delivery network) tier.
• Group micro-geo users into personas and pre-generating (and regenerating) cached content incrementally throughout the day.
• Lower overall cost, similar output, extremely personalized but not 1:1 personalized.
• Would fit neatly into common caching strategies for digital-first companies
• A solution such as embedded Weaviate at a colo or Amazon Web Services (AWS) Outpost deployment could help simplify the overall process.

The Future with Edge GenAI

GenAI-driven personalization is the next competitive advantage.

With LLMs, companies can leap ahead by embedding tiny machine learning models directly into browsers and on-premise servers. This edge computing approach generates customized content in the moment for each visitor - no more generic experiences.

Let's talk

If you want to discuss this topic or others in detail, feel free to reach out to me or my team at Innovative Solutions.

We've been hard at working helping dozens of companies accelerate their GenAI journey through our Tailwinds GenAI Architecture and I/we would love to help you on yours.

-Travis