The Accuracy Imperative in Legal AI: Why We Must Get it Right

Delving Deeper into the Challenges and Solutions for AI-powered Contract Review

In the rapidly evolving legal landscape, artificial intelligence (AI) is making significant strides, promising to transform how contracts are reviewed at work. However, the accuracy of AI in reviewing legal contracts remains a critical challenge. A recent study by Stanford University revealed that even large language models like GPT, PaLM, and LLaMA can sometimes "hallucinate," generating incorrect answers with unwarranted confidence.

Accuracy in Legal AI: A Non-Negotiable

The need for accurate AI in legal contract review is not just a matter of convenience; it's a necessity. The legal field demands precision because inaccurate analysis can lead to financial losses, missed clauses, misinterpretations of critical terms, potentially costly legal disputes and reputational damage.?

Take, for instance, the case of Tata Consultancy Services (TCS) , a prominent IT services company., where a poorly defined Master Service Agreement (MSA) with a U.S.-based client resulted in a legal battle costing TCS nearly $1 billion. This case underscores the importance of precise contract drafting and review, a task where AI can play a transformative role if accuracy can be ensured.

In our previous post, we highlighted the critical need for accuracy in AI-powered legal contract review. Now, let's delve deeper into why this is so crucial and explore the complexities involved in achieving high accuracy, particularly for Master Service Agreements (MSAs).

Why Master Service Agreements (MSAs)?

We chose to focus on MSAs because they are foundational agreements that govern long-term business relationships. A well-drafted MSA provides a clear framework for ongoing transactions, outlining the rights and responsibilities of each party. Accurate MSA review is crucial for mitigating risks, ensuring compliance, and fostering smooth business operations.

The AI Accuracy Challenge in MSAs

Our journey to develop LegalGraph AI began with a recognition of the accuracy limitations in existing AI solutions for contract review. We conducted extensive testing on 100 MSAs, each containing 40 critical clauses, to assess the AI's ability to identify and extract key legal terms with citations, AI reasoning and value of AI’s confidence for each response. While the AI demonstrated impressive speed, analyzing each contract in under 8 minutes, the accuracy varied across different clauses.

Key Challenges Faced by AI in MSA Review

• Ambiguity: AI can struggle with ambiguous clauses, especially those lacking specific or quantifiable terms. For example, in our testing, the AI sometimes reported a limitation of liability clause as "NOT FOUND" even though it was present in the agreement but expressed in general terms.
• Similar-Looking Terms: Differentiating between similar-looking terms, such as indemnification obligations by the service provider and the customer, can be challenging for AI.
• Contextual Understanding: AI may overlook the broader context of a contract, leading to incorrect conclusions. For instance, in one case, the AI incorrectly identified the presence of a software warranty based on a narrow interpretation of specific terms.
• Reading Between the Lines: AI may not always grasp implicit meanings or subtle nuances in contract language. In one example, the AI failed to identify the governing law of an MSA, even though it was implicitly stated in various provisions.

LegalGraph AI's Approach to Overcoming Challenges

To address these challenges and enhance accuracy, we adopted a multi-pronged approach:

• Collaboration with Legal Experts: We collaborated with 50+ attorneys to define essential legal terms and develop a comprehensive set of questions to guide and benchmark the results of AI's analysis.
• Contract Benchmarking: A dedicated team of legal interns analyzed over 100 contracts using LegalGraph AI, meticulously validating the AI's output against the original contracts.
• Model Optimization: We are continuously training and optimizing our Machine learning (ML) and AI algorithms to improve LegalGraph AI’s understanding of nuanced legal language and clause variations.
• Knowledge Base Development: We have created a dedicated knowledge base focused on specific clause types in Legal Contracts, such as termination clauses, Liability, indemnification, and more to provide the AI with a comprehensive reference point.

Results: Accuracy and Efficiency

Our testing on 100 MSAs revealed encouraging results:

• Key Term Extraction: LegalGraph AI achieved over 85% accuracy in extracting key legal terms and their values, including contract indemnification, termination clauses, payment terms, intellectual property rights, limitations of liability, and more.
• Numerical Data Accuracy: The AI demonstrated exceptional accuracy in processing numerical data, such as deadlines, payment amounts, and time periods.
• Time Savings: LegalGraph AI significantly reduced the time required for contract review. Compared to manual review, which can take 3-4 hours for a 50+ page complex contract, With LegalGraph AI’s multi-contract key term extraction a large complex contract can be fully reviewed in around 60 minutes, representing a time saving of 75 - 80%.

Time Savings Breakdown: Manual vs. LegalGraph AI

Where LegalGraph AI Excels

1. Numerical Data Accuracy

LegalGraph AI demonstrates exceptional accuracy in processing numerical data within contracts, including:

• Deadlines: Accurately identifies and extracts deadlines for specific obligations, such as notice periods or payment due dates.
• Payment Amounts: Precisely extracts payment amounts, ensuring accurate invoicing and payment processing.
• Time Periods: Correctly interprets and extracts time periods related to contract duration, renewal terms, or specific obligations.

Example:

In one MSA, LegalGraph AI accurately extracted the Term Period in Months as "6 months," matching the exact duration specified in the contract. Similarly, it precisely extracted the Deal Value as "$30,000 plus annual fee increase," ensuring accurate representation of the financial terms.

2. Fast and Accurate Interpretation of important clauses

LegalGraph AI automatically identifies and extracts critical clauses in an MSA within minutes including:

• Limitation of Liability: Accurately extracts clauses that limit the liability of one or both parties, ensuring clarity on financial risk mitigation.
• Intellectual Property (IP) Rights: Precisely identifies and extracts clauses related to IP ownership and usage rights, safeguarding valuable intellectual property assets.
• Warranties: Correctly extracts warranty clauses, providing insights into the quality and performance assurances related to services or products.

Example:

In one MSA, LegalGraph AI successfully extracted the Limitations of Liability clause, accurately stating that the liability was limited to the "Value of fees paid under the agreement." It also correctly extracted the Service Provider Intellectual Property Rights clause, specifying that the service provider "owns all Intellectual Property Rights in Cloud Service, <party name> Materials, and <Services>."

Our Commitment to Transparency and Collaboration:

We believe all legal AI products need not only to give a response but tell sources,? an easy way to check valid citations, and further give confidence values to ask for human help when they enter this state.?

In our commitment to transparency and collaboration, we're ready to share our evaluation methods and datasets with the legal community. Reach out to us at [email protected] to know more about the methods and data sets we used for contract accuracy testing.

Next Steps: Benchmarking and Expansion

We are committed to continuous improvement and will be conducting further benchmarking studies to compare AI's performance against other leading LLMs. We will also be expanding our analysis to include Non-Disclosure Agreements (NDAs), further enhancing LegalGraph AI's capabilities to address a wider range of contract types.

Invitation for Feedback and Free Evaluation Offer

We invite you, to share your thoughts on our approach to reviewing MSAs.

• Should we include any additional key legal terms in our evaluation?
• How can we assist you in benchmarking your legal contracts across these dimensions?

If you're considering a legal AI product, you should try out LegalGraph AI. Reviewing a contract with LegalGraph AI is now as simple as sending an email! Just send your contract to [email protected] and our AI will analyze it automatically. You'll receive a report highlighting the top risks within the contract.

P.S. Your data is encrypted with 256-bit AES & TLS encryption, its securely stored, and never shared with any third party. You have complete control and can delete the file at any time.

Special thanks to Dedipya Tawri , Shantanu Gupta , Adutiya Sengar for their invaluable contributions for reviewing the accuracy of AI in analyzing MSA contracts.

#LegalTech #AIinLaw #LegalInnovation #ContractManagement #LegalAI #LegalOps #AIforLegal #LegalGraph #LegalTechnology #LawTech