20 top lawyers were beaten by legal AI. Here are their surprising responses

In a landmark study, 20 top US corporate lawyers with decades of experience in corporate law and contract review were pitted against an AI. Their task was to spot issues in five Non-Disclosure Agreements (NDAs), which are a contractual basis for most business deals.

The study, carried out with leading legal academics and experts, saw the LawGeex AI achieve an average 94% accuracy rate, higher than the lawyers who achieved an average rate of 85%. It took the lawyers an average of 92 minutes to complete the NDA issue spotting, compared to 26 seconds for the LawGeex AI. The longest time taken by a lawyer to complete the test was 156 minutes, and the shortest time was 51 minutes. The study made waves around the world and was covered across global media.

Who were the lawyers?

Those who took on the AI are 20 US-trained corporate lawyers with legal and contract expertise with experience at companies including Goldman Sachs and Cisco, and global law firms including Alston & Bird and K&L Gates.

However, surprisingly, for these defeated lawyers there was no handwringing. There was almost no talk of the “rise of the machines” nor fear of AI taking their jobs.

Legal Crisis or Trust in Automation?

Some of these skilled lawyers did question whether their profession could ever entirely trust automation to make skilled legal decisions. For a small number, they suggested they would be sticking to “reliable” manual processes for the immediate future. However, most of the participants stressed that high-volume and low-risk contracts took up too much of their time, and felt it was incumbent on lawyers to automate work when, and where, possible. For them, the study was also a simple, practical demonstration of a not-so-scary AI future. However, lawyers also stressed that undue weight should not be put on legal AI alone. One participant, Justin Brown, stressed that humans must use new technology alongside their lawyerly instincts. He says: “Either working alone is inferior to the combination of both.”

Here, in their own words, are the responses of nine of the 20 participants in the study — many speaking for the first time.

Zakir Mir

“It is crucial to make mundane contract work more efficient, especially when there are 50–100 pages of contracts for some major deals (M&A large tenders with agreements or multinational corporations). It can really help lawyers sift through these documents, and cut down on the sometimes-deliberate verbosity of these documents which can allow one party to mask core issues.”

Zakir Mir, former regional counsel for BDP International, a $2billion global logistics firm, now at Allegiance International

Samantha Javier

“The test pointed out issues that NDA agreements generally contain and issues that lawyers look out for when drafting and reviewing NDAs. As for being automated, I think this would help clients in getting better pricing and allow lawyers to focus on more complex projects. However, I do think the test and AI technology must be very thorough to accomplish this and business clients may prefer to have a human lawyer looking at and taking care of their business concerns.”

Samantha Javier is a Lewis & Clark Law School graduate, licensed to practice law in Oregon. Her experience includes law firm, in-house, and transactional work

Grant Gulovsen

“Participating in this experiment really opened my eyes to how ridiculous it is for attorneys to spend their time (as well as their clients’ money) creating or reviewing documents like NDAs which are so fundamentally similar to one another. Having a tool that could automate this process would free up skilled attorneys to spend their time on higher-level tasks without having to hire paralegal support (thereby making the services they offer more competitive in the long run).”

Grant Gulovsen, an attorney with more than 15 years’ experience

Seun Adebiyi

“We are seeing disruption across multiple industries by increasingly sophisticated uses of Artificial Intelligence. The field of law is no exception. The correct identification of basic legal principles in contracts is the kind of routine task that may be amenable to automation. Using AI to spot routine issues in non-disclosure agreements could be a useful time- and cost-saving development for the legal industry as a whole.”

Seun Adebiyi, former corporate attorney at Goldman Sachs

Abigail Patterson

“While having the clauses categorized would make the NDA review process slightly quicker for me in my own practice, I don’t think the true legal review could ever be done by AI. However, I do see the application of the AI as a useful time-saving tool.”

Abigail is a corporate attorney at US-based medical device manufacturer DeRoyal Industries. She is licensed in both Tennessee and Wisconsin and has worked as an In-House Attorney, HR/Employment Law Attorney, and Special Projects Manager

Justin Brown

“As a chess player and attorney I will take from Grandmaster Vishy Anand and say the future of law is ‘human and computer’ versus (another) ‘human

and computer.’ Either working alone is inferior to the combination of both. I view AI and technology as exciting new tools that would allow for such drudgework to be done faster and more efficiently.”

Justin Brown, Partner at Brown Brothers Law

Hua Wang

“AI has huge potential in reducing time on standard contract reviews and making legal advice accessible and affordable for all. LawGeex asked me to review the NDA in a logical and credible manner, similar to how I reviewed documents as a former lawyer at a global law firm.”

Hua is Co-Founder of SmartBridge, and formerly a lawyer at K&L Gates and Proskauer, in-house counsel at Cisco Systems, and a Global Scholar at the Kauffman Foundation. Hua graduated from Duke University and Northwestern University School of Law

Jack West

“Something I did recently that helped keep my legal skills intact was participate in an experiment run by Lawgeex where they asked attorneys to identify issues in NDA’s to compare against their software. It was a fun, practical exercise. Did I beat the machines? Probably not.”

From Birmingham, Alabama, Jack West is a former attorney at Cabaniss, Johnston, Gardner, Dumas & O’Neal LLP, who has now founded a legal start-up called Book-It-Legal. He studied at the University of Mississippi School of Law. He focused on the areas of securities, corporate, real estate, and tax law.

Deja Colbert

“As a Contracts Administrator, I do believe AI will be useful and reduce time on contract reviews. If not done so already, there should be standard contract templates or clauses plugged into the software database and trigger discrepancies within the documents for review based on the means of the contract. Although contracts can be very similar from one another, they all have their own specific purpose, which calls for attention to detail on the use of tags. I find AI logical and credible for the purpose of reducing contract review time, thus, allowing prioritizing other time-consuming tasks at hand.”

Deja Colbert is a contracts administrator at Omega Rail Management where she drafts contractual documents and coordinates negotiation of the terms and conditions accordingly, creating abstracts of property-related agreements. She was formerly a contract specialist at Cox Automotive in Atlanta and American CyberSystems in Duluth and Experian Health.