And What Was Your Third Question?

🔷 Subscribe to get breakdowns of the most important developments in AI in your inbox every morning.

This amazing paper from New Zealand AI startup, Onit, peers into the post-lawyer era. They tried to answer:

• Do AIs outperform junior lawyers in determining and locating legal issues in contracts? Slightly, with F-scores (a balanced accuracy measure)

  • for determining legal issues => GPT-4 (0.87) vs junior lawyers (0.86)

  • for locating legal issues => GPT-4 (0.69) vs junior lawyers (0.67)

• Do AIs review contracts faster than junior lawyers?

  • Yes with GPT-4 (4.7 minutes) vs junior lawyers (56 minutes)

• Is AI contract review cheaper than junior lawyers?

  • Yes with GPT-4 ($0.25) vs junior lawyers ($74)

Thu study was limited to:

• ten procurement contracts - large volume of work with enough variability vs NDAs which have little variability

• US and New Zealand law contracts

• AI models with context windows of at least 16,000 tokens (~80 pages), as using techniques such as Retrieval Augmented Generation were found to be unstable

Notably unlike previous studies, in this study:

• the context of buyer, seller, and background to the contract was provided to the model'

• prompt engineering was performed, with the model being told it was a lawyer

• ground truth was prepared by senior lawyers reviewing the same contracts…in an effort to mimic how work is often handed off to juniors and then reviewed by seniors

• measured the setup, prompting, and fine-tuning time from a cold start, at roughly 16 hours for an AI model, comparable with "investment in time in instructing junior lawyers”

This is important as previous studies hit GPT-3/4 raw without any context or prompt engineering… causing most AI engineers to sneer at the results. The research team outlines the implications:

• Demand for junior lawyers will drop

• Legal process outsourcing business will be decimated

• Arms race for law firm adoption

While the conclusions seem like a little bit of motivated reasoning (in 2 player games, the other player adapts after you make the first move), the finding in that for specific and well-defined tasks, AIs are probably going to displace lawyers seems undeniable.

Become a subscriber for daily breakdowns of what’s happening in the AI world: