ChatGD: Learnings (So Far) from our Legal GenAI Experiment

In August 2023, we launched ChatGD, our homegrown internal generative AI (GenAI) chat application with retrieval-augmented generation (RAG), following a successful beta test over the summer. Now, just four months later, ChatGD continues to evolve to keep up with the blistering pace of development in AI. With the end of the first full year of mainstream GenAI upon us, we wanted to share some learnings, reflections and updates on Gunderson’s AI journey to date.

Upskilling for GenAI

As a prerequisite to using ChatGD, we required users to complete an initial training (either live or on demand). We held three live training sessions tailored specifically for our firm’s attorneys, paralegals and business professionals. We framed the rollout of ChatGD as a collaborative experiment designed to help everyone move up the learning curve and to crowdsource the most promising use cases and methods for getting the best results out of GenAI-powered tools.

The trainings focused on how large language models (LLMs) and RAG actually work beneath the hood, to provide everyone with a baseline understanding of the technology, as well as how to use ChatGD safely and ethically. We also shared ideal use cases for GenAI (e.g., text manipulation) and areas where we thought the technology was not well suited for use without further advancements (e.g., search engine replacement). More than half of the entire firm attended one of those three trainings in real time, which is a testament to the incredible level of interest in GenAI and the tool we built.

ChatGD Adoption and Cost

Since launching ChatGD firm wide in August, here are some highlights from the data we have collected on usage and cost that reflect where we are today:

●?????? Significant User Base. Nearly half of the firm has already used ChatGD.

●?????? Growing Usage and Engagement. Usage has been steadily increasing, with more than nine thousand prompts submitted and completed across several thousand conversation threads.

●?????? Pure Chat vs. RAG. We have created vector search indices for hundreds of document collections uploaded by our users for RAG, and prompts leveraging the advanced RAG feature accounted for more than 10% of the total volume of interactions with the tool.

●?????? Minimal Cost. We project the total annual out-of-pocket cost for providing ChatGD to our entire firm will be less than $10,000—a staggeringly low figure, especially when compared to the alternatives currently available in the enterprise market. We largely attribute the cost-effectiveness of ChatGD to two strategic decisions: (1) self-hosting an open-source model for RAG vector embeddings and (2) leveraging GPT 3.5 Turbo for both pure chat and RAG functionalities instead of using the most expensive models available.

Use Cases: The Good, the Bad and the Ugly

Our Gunderson colleagues are using ChatGD in a variety of ways to work smarter.? Our attorneys are using it to retrieve and manipulate or summarize language in legal agreements, draft and change the tone of emails, summarize documents and articles, and brainstorm different examples of legal language or topics for presentations. They have discovered creative ways to have the LLM help teach them to craft better prompts. Our business and technology professionals have used the tool to help them create and repurpose content for marketing, answer RFPs, prepare for meetings, structure and format data, write code and improve their written communications.

Our users have also been providing great crowdsourced feedback directly within the app to help us identify good use cases to share with the rest of the firm, as well as use cases where the technology does not perform as well. We use the latter for training users on the limitations of GenAI technology, teaching them how to write better prompts to get better results and identifying how we can improve the tool as the technology improves so that we can tackle more complex use cases in the future.

What's Next

We just released some major updates to ChatGD this week:

●?????? Using prompt-routing and open source embeddings models, we have constructed multiple indices that employ a combination of keywords, knowledge graphs, vector embeddings and autonomous retrieval to dynamically optimize the chosen fact retrieval method for a user’s specific prompt as part of our RAG workflow.

●?????? This now includes routing prompts to different LLMs for fact retrieval and summarization to perform the language generation step of the RAG process, allowing us to use larger context windows and larger models for better summarization while reserving more cost-effective models for fact retrieval.

●?????? For especially detailed summarization tasks, we route these requests to our most powerful models with the largest context windows to provide the model with full context of the source material, where possible.

In brief, we are now using three different foundational models as part of the ChatGD tech stack, deploying the best available model for each particular purpose. We have made a number of user experience and performance enhancements based on user feedback. Moreover, we are ready to upgrade our fact retrieval LLM to GPT 4 Turbo as soon as it becomes available to us for production use, thanks to our model agnostic architecture.

We are still very much at the beginning of this GenAI experiment, and we will continue to share what we learn as we all try to harness the power of this technology to help us solve real-world problems and work better.

To stay up-to-date on the latest legal developments in GenAI, bookmark Gunderson Dettmer 's AI Resources page, which includes articles, client insights, events, podcasts and webinars covering this fast-moving space.