AI Research in Alignment Techniques is strengthening the foundation for Personalization

In some of my last articles I talked about Natural language Processing as the driver of personalization to specific user groups and business functions - and 2023 saw perhaps the greatest jump in our ability to personalize user experiences.

2023 was the year in which decades of research and painstaking experimentation, along with billions of dollars of funding, culminated in the introduction of near-human AI capabilities into the mainstream. Large Language Models (LLMs) captured the world’s imagination and mobilized both the private sector and forward-thinking governments into diving deep into an area of engineering that was previously associated with science fiction and fantasy. Right now, it seems that the momentum created by OpenAI’s ChatGPT will only grow.

2024 will see accelerated developments in the development of AI capabilities and products. If 2023 was the year AI truly emerged into the mainstream, I believe 2024 will be the year numerous challenges around widespread and truly productive operationalization of LLMs will be resolved. The second half of 2023 saw a number of developments in the realm of academic and industrial research consistent with this theme, whether its improvements in development, training, or deployment process. One area that saw especially significant improvements was the realm of alignment techniques.

AI alignment <> Personalization

AI alignment and personalization are closely intertwined, as both aim to create a more user-centric experience for individuals.

AI alignment focuses on ensuring that the development and deployment of AI systems align with human values, goals, and ethical considerations. Personalization, on the other hand, aims to tailor services and products to meet the unique needs and preferences of each user. When these two concepts are combined, it results in a powerful approach to designing AI-driven solutions that are not only effective but also respectful of user privacy and autonomy.

By aligning AI systems with human values and providing personalized experiences, we can create a more meaningful and positive interaction between users and technology. This will create a more seamless and satisfying user experience, fostering trust and engagement between users and technology.

These approaches can then be applied in various domains, such as e-commerce, social media, entertainment, and education, among others. For example, personalized learning paths can be created based on a user's strengths and weaknesses, or personalized product recommendations can be generated based on their purchase history and preferences.

Traditional AI-driven Approaches to Personalization

We have already seen how existing approaches to AI-driven personalization can be achieved through various techniques:?

• Data-driven personalization: Gathering user data to understand their preferences, behavior, and context, allowing AI systems to make informed decisions about what content, products, or services to recommend.
• Machine learning: AI algorithms can learn from user interactions and preferences, continually refining the personalization process and improving the accuracy of their recommendations.
• Natural language processing: Understanding the nuances of user input, such as tone, sentiment, and context, to provide more personalized responses or assistance.
• Collaborative filtering: Using the collective intelligence of a user's social network or community to suggest relevant content or connections.
• User profiling: Creating a comprehensive profile of a user based on their interactions, preferences, and behavior, enabling AI systems to provide more targeted recommendations.

Note to self: It will be interesting to create a unified language model with all types of customer metadata to see what opportunities and insights open up beyond what we already know.?        

Aligning AI systems beyond traditional approaches

New accomplishments this year with accelerated research in various alignment techniques like RLHF, DPO, KTO will drive personalization in user experiences.

These alignment techniques generally aim to increase the “controllability” and robustness of otherwise opaque and unpredictable LLMs and improve alignment with business and ethical priorities.? Beyond the rule based alignments that many products have used, the first wave of methods was inspired by reinforcement learning and accordingly called Reinforcement Learning from Human Feedback - RLHF.?

Understanding RLHF, DPO, KTO in a nutshell ?

RLHF aims to produce a distribution that maximizes the ‘rewards’ produced by the model outputs while not deviating too much from the fine-tuned reference distribution. Since we only have access to a dataset of pairwise comparisons of outcomes (i.e. “human preferences”), and not the actual “human rewards”, we also need to learn a proxy reward model to be used in final reward maximization. This is accomplished by minimizing the negative-log likelihood of the preference probabilities (i.e. the probability that a particular outcome is preferred to another, given particular input x).?

Challenges with RLHF: Beyond the use of a RL algorithm for the maximization of the final objective, two features of this traditional method stand out. - use of a “preference model” to receive the “preference probabilities”, given a particular input x and the need to learn a “reward model”.? The difficulties in learning a reward model and productionizing RL optimization led to the next wave of techniques in this area, Direct Preference Optimization (DPO).?

Simplification of alignment with DPO: DPO is primarily characterized by its closed form loss-function and lack of reward modeling or RL optimization. By directly optimizing the language model to minimize the negative log-likelihood of observed human preferences (which we know from our dataset) the alignment process is greatly simplified in practice. Despite being more stable and simpler to implement, DPO frequently outperforms RLHF methods in many tasks, including in controlling the sentiment of generations.?

Further read to assess LLM alignment methods

At the end of 2023 a new report by Jurafsky et al. introduced a new way to assess LLM alignment methods. They propose a new loss function (Kahneman & Tversky Optimization - KTO) that directly optimizes for the utility of LLM outputs, instead of the negative log-likelihood of preferences.?

The differentiator with KTO is that it only requires the dataset to know whether or not an outcome is ‘desirable’ or ‘undesirable’. Thus, it does not require the collection of “pairwise comparison” type preference pair data that previous methods relied on (collection of which can provide great difficulty in real world settings). This imbues KTO with immense data efficiency. I suspect when the official, final paper is out, this method will inspire the next wave alignment methods and will spur further cross-collaboration between human centered decision making fields and AI researchers. Importantly, it will enable further democratization of AI-powered personalization.

Final thoughts

As alignment methods progress, they will increasingly incorporate theoretical and empirical insights from fields like behavioral economics, quantitative psychology and econometrics. These fields have grappled with problems and issues relating to human preferences, choice and decision making for decades in highly rigorous ways. It is no surprise that LLM and AI alignment researchers are turning to these fields for inspiration and advancement.??

Accordingly, product builders are reimagining business processes and workflows by using a combination of fine-tuning and alignment to create individual-level customization and personalization in generated text and tasks - opening the possibility to change the user journey completely.?

Question for you? ?

Which industries and products have the most potential of ROI with personalization and how are alignment techniques being adopted, which business use cases is that driving - all open questions and we would love to hear your thoughts on them!??