Context Management
"Forest feast" by OpenAI's DALL-E

Context Management

Today a two-for-one bargain: (1) an article on a very important topic about how to get the most out of programming LLMs and (2) an example of using ChatGPT to improve on something you have written.

On this second point, I wrote the article and then used ChatGPT to make improvements. I thought it would be an interesting example to show the original version next to the final output so I have appended the original at the end. I asked ChatGPT-4 to rewrite the article, improving spelling, grammar, and adding any points I might have missed. Here is the final result:


In the rapidly evolving landscape of Large Language Model (LLM) programming, mastering context management is not just an advantage, but a necessity. This article delves into the complexities of context management, shedding light on its importance, diverse applications, and the challenges faced in optimizing its utilization.

Understanding Context in LLMs

The context window in LLM programming is the segment of information that the model processes in a given request. It plays a critical role in guiding the LLM's response. Below are some key types of context:

  • Brand Guidelines: This includes preferred colors, image styles, language, and tone in communications. It encapsulates words or phrases that reflect brand values.Example: A company may insist on using a friendly and conversational tone in its customer interactions, reflecting its brand ethos.
  • Regulatory Information: This involves internal compliance policies and external legal guidelines.Case Study: A financial institution uses LLM to generate customer advice, ensuring all recommendations are in line with financial regulations.
  • Expertise: Specialized knowledge required in addressing industry-specific topics.Application: In healthcare, an LLM might be fed with medical protocols to assist in generating patient care plans.
  • Topical/Transitive Information: Dynamic information such as weather forecasts or market trends.Illustration: A travel agency using LLM to provide daily travel tips based on current weather conditions.
  • Personalization: Characteristics of the user or the audience to tailor the output.Scenario: Customizing educational content based on the learner's progress and interests.

Challenges in Context Management

Effective context management in LLM programming is fraught with challenges:

  1. Source of Information: Determining whether to access information via an API or create a new repository.
  2. Information Quality: Ensuring the information is current and accurate, possibly through automated validation tools.
  3. Context Selection: Using tools like vector databases to match content elements with contextual needs.
  4. Efficient Context Window Usage: Balancing the need for sufficient context against the cost associated with larger context windows.
  5. Sequential Request Strategy: Deciding when to use multiple requests, like generating an email with brand guidelines followed by a compliance check.

Technological Advancements and Outlook

Software companies are now introducing tools for better context management and multi-request applications. However, much of this process is still manual. Leading adopters of generative AI, with robust development teams, are pioneering in this space, providing valuable insights into scalable context management strategies.

Ethical Considerations and Bias

It's crucial to address the ethical considerations and potential biases in context management. Ensuring diverse and unbiased input data is key to maintaining the integrity of LLM outputs.

As LLMs continue to evolve, the art of context management will become increasingly sophisticated. Understanding its nuances and challenges is vital for any organization looking to leverage LLM technology effectively. The future of LLM programming hinges not just on the technology itself, but on our ability to adeptly manage the context it operates within.


And here is my original draft:

Probably the most important problem to solve in structured LLM Programming (which I recently wrote about) is in managing the context window and all of the possible information components of context. The context window is the block of information that a large language model (LLM) will process in a given request. Context can be used in a number of different ways to instruct the LLM how to perform the request, here are a few examples:

  • Brand Guidelines: This could include colors or image styles or the use of language and tone in communications that is preferred. Certain words or phrases that represent the brand values, etc.
  • Regulatory information: This could be internally created compliance information or industry/government guidance.
  • Expertise: Special information that is needed in addressing a particular topic, perhaps unique to the company or industry such as the approved way to safely perform a task in a work environment.
  • Topical/Transitive: This could be any information that is likely changing on a regular basis - the temperature expected during the day tomorrow in a particular location for example.
  • Personalization: A wide range of possible characteristics of the user or the target audience for the output of the request to make it more relevant.

You can likely imagine many more categories of context. Using these different types of context in constructing a series of prompts is the way that you develop your use of an LLM to achieve a consistent high quality result that achieves your desired outcome. But there are a number of challenges to address, here are a few examples:

  1. Where is the information coming from? Do I need to access through an API or some other integration tool? Do I need to create a new repository to store this information?
  2. How do I maintain the quality of this information? What protections do I have in place to make sure it is up to date and accurate?
  3. What tools do I use to select the right information to use in a given situation? Do I need a vector database to allow my to quickly match a content element with a given context need?
  4. How do I efficiently use the available space in a context window? Vendors have been increasing the amount of content that can be included in a request but each token used has a cost so selecting the right content to obtain the desired result must also take into consideration that expense with a goal to be as brief as possible while still providing sufficient context.
  5. When should I use multiple requests in a series to obtain results? An example might be to use generate an email to a customer using your company's brand guidelines but then do a second pass to check for regulatory compliance.

We are beginning to see software companies introduce development tools to help manage context and develop multiple request applications but right now a lot of this work has to be done by hand. The companies that are the most advanced in adopting generative AI have development teams and application frameworks in place to support this work at scale.

John Sviokla

GAI Insights Co-Founder, Executive Fellow @ Harvard Business School

1 年

Love the article. When considering Marketspace interactions in general (e.g. those interaction that occur in a digitally enabled environment), context is vital for cognitive continuity of the "user". Context was a central point of our work on Marketspace (https://hbr.org/1994/11/managing-in-the-marketspace). I think it's fascinating that in working with our silicon conversation partner(s) in the LLMs, we carbon-based cognitors need to "remind" the LLMs or our context and intent -- both to drive quality of outcome and fewer harmful answers. As Stephen Wolfram has said, we need a science of LLMs and one branch of that science, I believe, will be how to stimulate a vast probabilistic model to improve answers and/or outcomes of an interaction with that model. Firms are so used to dealing with technology as a deterministic system (with some notable exceptions including trading operations at any asset manager) that our "policies" in the IT shop will have to encompass new types of risk/return.... #genai

Amahl Williams

Go-to-Market Leader | AI Automation Strategist | Author | Driving Growth Through Intelligent Solutions

1 年

A sobering reminder of the importance of the various literacies.

回复

要查看或添加评论,请登录

Ted Shelton的更多文章

  • AI Interregnum

    AI Interregnum

    An interregnum: where one epoch is fading and another struggles to emerge. I have these wildly disparate conversations.

    11 条评论
  • Quantum-Enhanced AI?

    Quantum-Enhanced AI?

    Wednesday evening I tried to go to sleep early as I had to get up for a flight the next day and then two full two days…

    6 条评论
  • Cargo Cults and the Illusion of Openness

    Cargo Cults and the Illusion of Openness

    In the South Pacific during the 1940s, indigenous islanders witnessed military planes landing with supplies. After the…

    8 条评论
  • From WIMP to AI

    From WIMP to AI

    Evolving Interfaces and the Battle Against Cognitive Overhead The GUI Revolution and Its Growing Complexity Graphical…

    19 条评论
  • Harvesting your data

    Harvesting your data

    Much has been written this week about DeepSeek - overreaction by the markets, handwringing about China, speculation…

    5 条评论
  • Cognitive Surplus

    Cognitive Surplus

    Clay Shirky's 2010 book Cognitive Surplus: Creativity and Generosity in a Connected Age recently came to mind as I…

    17 条评论
  • Enterprise AI adoption

    Enterprise AI adoption

    I am going to go out on a limb here and just say that everyone will be wrong. Including me.

    21 条评论
  • Predictions for 2025

    Predictions for 2025

    What should we expect from AI research and development in the coming year? Will the pace of innovation that we have…

    14 条评论
  • Assessing 2024

    Assessing 2024

    At the end of each year that I have been writing this newsletter I have made a few observations about what may be…

    7 条评论
  • Change is hard

    Change is hard

    While substantial gains might come in the future from adopting new technologies such as machine intelligence, the…

    16 条评论

社区洞察

其他会员也浏览了