Automating Justice?

Artificial intelligence has exploded in power over the past decade, with systems like GPT-3 and GPT-4 mastering human-like language tasks. This rapid progress has fueled sky-high hopes and apprehension about AI's potential implications across knowledge professions like law. Yet, absent rigorous empirical evidence, the discourse has remained largely speculative.

That is why the recent release of a pioneering randomized control trial from University of Minnesota Law professors marks such a vital milestone. Meticulously designed and executed, this groundbreaking study provides the first solid data on if, how, and under what circumstances AI can enhance human legal work. The results offer a sobering mix of promise and caution.

On the upside, the study found that AI can dramatically enhance productivity on core legal tasks, with time savings ranging from 12% to 32%. However, quality gains were smaller on average and inconsistent across tasks and performers. For instance, AI boosted lower-performing students substantially more than top students, signaling its potential to act as an equalizing force. But on complex memos, AI offered little help to any participants.

This pattern of uneven results underscores that realizing AI's full potential in law requires judicious usage and deliberate efforts to shape its impacts. This landmark study should serve as a compass guiding various legal stakeholders, from schools and firms to clients and judges, in navigating this AI transformation proactively rather than reactively.

To contextualize the study's insights, this article proceeds as follows. Part II reviews the randomized trial's rigorous methodology and core results on efficiency, quality, and satisfaction. Part III dives into the distribution of the uneven gains across tasks, practice settings, and lawyers. Part IV lays out a comprehensive blueprint with tailored recommendations for key legal actors in response. Given AI's rapid evolution, adapting successfully requires urgency and vision. The era of AI-enhanced law is dawning; let this trailblazing study illuminate the wise path forward.

II. Core Results: Efficiency and Quality Gains, Unevenly Distributed

A) Study Methodology

- Gold standard randomized control trial design

- 59 University of Minnesota law students participated

- Students completed four practical legal tasks mirroring junior attorney work

- Federal court complaint drafting

- Plain English contract drafting

- Employee handbook section on workplace lactation rights

- Legal memo advising on product liability risks

- Students randomly assigned tasks with and without AI assistance from GPT-4

- GPT-4 access is provided through a realistic ChatGPT interface

- 2 hours of video training on effective legal AI techniques

- Blind grading by professors to prevent bias, anonymous time tracking

B) Findings on Efficiency

- Large time savings of 11.8% to 32.1%, faster completion across all four tasks

- This means that with AI, tasks were completed 11.8% to 32.1% more quickly

- Biggest gains for contract drafting - 32.1% reduction in minutes required

- Contract task took about 1/3 less time with AI versus without

- Smallest gains for legal memo - 11.8% faster completion

- Memo task took about 12% less time with AI versus without

- Clear evidence of potential for AI to substantially boost legal productivity

- Impressive, given participants' minimal experience using AI

C) Findings on Quality

- Small average gains in graded quality but significant variance by task type

- On 4 point grade scale, AI increased scores by 0.1 to 0.2 points on average

- However, gains fluctuated widely by task type

- Contract drafting improved by 0.24 points on a 4.0 grade scale

- This translates to a natural, but modest 6% grade boost

- Complaint drafting improved by 0.17 points

- This translates to approximately a 4% boost in grade

- Employee handbook and memo grades are effectively unchanged (-0.07 to 0.07)

- Much more significant gains on contracts versus memos

- Demonstrates uneven benefits to legal quality by task

D) Findings on Distribution

- AI helped weaker students substantially more than strong students

- Grades correlated 0.5 without AI but just 0.2 with AI assistance

- This means AI reduced the link between students' skill levels

- Suggests potential for AI as an equalizing force in the legal profession

- By disproportionately boosting lower-performing students

- But risks convergent work product as natural differences shrink

E) Participant Satisfaction

- Students reported increased satisfaction when using AI

- Correctly ranked tasks where AI was most/least helpful

- Indicates awareness of strengths and limitations

This rigorous study reveals tangible but uneven productivity and quality gains from legal AI. Understanding this variance is critical to optimization.

III. Implications: Uneven Impacts Across Legal Sectors

A) Quality and Efficiency Gains Will Vary by Practice Area

- Transactional lawyers will see radical workflow transformation

- 32% faster contract drafting with AI in the study

- More formulaic tasks like contract review even faster

- Litigators will see less disruption from AI

- Just 12% faster memo writing with AI in the study

- Memos involve more creativity and analysis

- Mundane legal research dramatically enhanced by AI

- Risks of over-reliance on AI likely more excellent in transactional work

B) AI Will Most Benefit the Least Skilled Lawyers

A study found weaker law students were helped much more by AI

- For average students, a 0.5 grade drop without AI but 0.2 with AI

- Substantial boost to lower-performing lawyers can expand access

- By narrowing the skill gap in legal services

- But risks convergent work product as AI diminishes differentiation

- Need for policies to preserve diversity of thinking despite AI

C) Competitive Advantage Will Hinge on AI Adoption

- Firms that proactively integrate legal AI will gain an edge

- By achieving significant efficiency gains from AI

- Laggards who resist adapting risk falling behind

- As staff get poached or clients switch to AI leaders

- Clients will scrutinize the value from law firm AI usage

- Quickly evolving into a necessary investment for competitiveness

D) Labor Market Will Shift Toward Irreplaceable Skills

- Commoditized legal work will be automated away by AI

- Greater value for creative, empathetic, and strategic skills

- Skills AI cannot replicate as well as humans can

- Demand for lawyers with high emotional intelligence

- To build trust and relationships

- Need education for next-generation skillsets

- Tech competency, data analysis, project management

Realizing AI's full potential requires understanding and adapting to its uneven impacts across the legal ecosystem. Its benefits can be maximized only by targeting integration at the highest-value tasks and users.

IV. A Comprehensive Blueprint for AI Adaptation

A) Recommendations for Law Schools

- Ban AI assistance on exams and assignments for mandatory 1L courses

- Require honor code pledges to not use AI, with strict enforcement penalties

- Utilize plagiarism detection software and technologically restrict access to generative AI on school networks

- Offer an array of focused upper-level courses on effective AI use in law

- Survey courses on the implications of AI for specific practice areas and ethical dilemmas

- Technical courses on optimizing AI usage through prompt engineering and output vetting

- Hands-on legal tech clinics allow work on real-world AI systems and policy issues

- Build a cross-disciplinary curriculum drawing on tech, ethics, and data science

- Recruit professors with computer science, data science, and ethics expertise

- Develop joint JD/PhD degree programs in law and computer science or legal informatics

- Offer certificates in legal technology, data analytics, and AI ethics

- Embed teaching assistants with tech backgrounds across the curriculum

- Create research labs dedicated to AI and law

- Track ongoing advances in legal AI capabilities and benchmark systems

- Model future implications for the profession, clients, ethics, and education

- Publish findings in academic journals and practitioner publications

- Provide CLE-qualifying seminars and course packages for alums/practitioners

- Survey graduates to identify professional education needs

- Engage alum network through free events and AI education outreach

B) Recommendations for Law Firms

- Invest in AI tools tailored to firm practice areas for optimal productivity

- Conduct extensive pilot studies to identify highest potential use cases precisely

- Work closely with legal AI vendors to deeply customize systems to unique firm needs

- Provide intensive mandatory training for all lawyers on proper AI techniques

- Dedicated legal AI teams providing live workshops in prompt engineering, vetting outputs, and optimizing workflows

- Require completion of an in-depth certification program before granting any system access

- Ongoing continuing education as capabilities advance across legal domains

- Develop rigorous quality assurance processes governing all AI usage

- Extensive protocols requiring vetting of all algorithmic work products by qualified attorneys

- Red team stress tests to proactively catch faults before any client harm

- Layered guidelines limiting AI usage for sensitive tasks or high-stakes matters

- Appoint senior AI ethics officers

- Independent, empowered role reporting directly to executive leadership

- Oversee the development of best practices and manage risks

- Provide complete transparency to clients regarding AI usage and benefits

- Communicate how AI is specifically enhancing services and value provided

- Proactively invite client scrutiny and feedback on processes and results

- Implement data privacy protections and cybersecurity measures

- Require client consent for usage of confidential data

- Invest in anti-hacking defenses and access controls

- Evaluate malpractice insurance coverage for AI issues

- Consider physical office adaptations to integrate AI smoothly

C) Recommendations for Clients

- Scrutinize law firm AI usage for maximum value and risk mitigation

- Require data on efficiency gains, risk prevention, and impact on total billing

- Aggressively pressure firms to pass efficiency gains to clients through lower fees

- Utilize client power to force cost reductions instead of merely enriching firm profits

- Consider bringing commoditized legal work in-house, given AI productivity

- Refocus high-priced outside counsel work on tasks requiring custom legal strategy

- Provide crystal clear guidelines to firms on appropriate AI usage

- Identify specific matters or tasks where AI should never be used

- Demand total transparency into law firm AI activities

- Require disclosures of AI usage time in billing records

- Audit firm protocols for vetting AI work products before deployment

- Hire dedicated legal AI specialists to liaise with outside counsel

- Evaluate AI usage in the context of overall firm relationship

- Develop internal training for legal staff on assessing outside counsel AI usage

- Create customized in-house AI tools tailored to unique client needs and data

D) Recommendations for Judges

- Broadly allow AI assistance to maximize access to the legal system

- But monitor risks closely and prohibit irresponsible usage, introducing errors

- Require disclosure when AI is substantially utilized in developing legal filings

- Enable nuanced case-by-case determinations on appropriateness

- Provide clear standards for attorneys on acceptable AI use

- Prevent factual hallucinations, legal errors, and over-reliance

- Require jury instructions explaining the proper weighting of AI-assisted arguments

- Ensure fairness and prevent over-deference to "neutral" technology

- Task court administrative bodies with issuing best practices

- Through a transparent process soliciting technologist and practitioner input

- Adapt court budgets and technology to support increasing legal AI usage

- Fund new positions, training, and systems to smooth integration

The groundbreaking University of Minnesota Law randomized control trial marks a watershed moment in understanding the monumental impact artificial intelligence will have on the legal profession. Its rigorous methodology and insights command attention.

By empirically demonstrating AI's capacity to enhance legal productivity and access significantly, this study signals the dawn of a new era. As capabilities rapidly advance, AI-enabled lawyering tools will become ubiquitous in the coming years. No segment of the legal industry will remain unaffected by this algorithmic revolution.

Yet realizing AI's full disruptive potential while mitigating risks requires thoughtful stewardship. Passively relying on market forces alone would squander opportunities and court unintended consequences. Instead, as this article states, all stakeholders—from law schools to firms, clients to courts—must act decisively to shape this transformation.

Proactive adaptation will determine whether AI promotes justice and progress or enhances inequality and stagnation. Law can lead where other fields have fallen short, writing a new narrative of technology and human collaboration for the common good.

The revelations of this landmark study provide the blueprint. I think the imperative now is to do it with urgency and vision. AI will revolutionize law practice but do so either responsibly or recklessly. The time has come to guide technology's arc toward justice.