AI Beats Law Grads on Bar Exam

When it comes to passing the bar examination, it seems Artificial Intelligence is better than Law School Intelligence. 

In a recent mano-a-machina competition, an upgraded version of ChatGPT outperformed humans on the two-day bar exam by seven percent.

“GPT-4, the upgraded AI model released by Microsoft-backed OpenAI, scored 297 on the bar exam in an experiment conducted by two law professors and two employees of legal technology company Casetext,” according to Reuters. “That places GPT-4 in the 90th percentile of actual test takers and is enough to be admitted to practice law in most states, the researchers found.” 

GPT-4 performed as well as or better than humans on the reading, comprehension and writing aspects of the bar exam.

The results show the astonishingly rapid advancement of AI technology. In late 2022, an earlier version of ChatGPT took the bar exam, and though it fell short of a passing score, it answered half the multiple-choice questions correctly. The new, improved GPT-4 got more than three-quarters of the questions right. 

GPT-4 performed best in the section on contracts law (88.1 percent correct), evidence (85.2 percent) and criminal law and procedure (81.1 percent). On the Multistate Performance Test, GPT-4 scored 4.2 out of 6 points, good enough to pass in most jurisdictions.

From Reuters: “The National Conference of Bar Examiners, which designs the multiple choice section, said in a statement that attorneys have unique skills gained through education and experience that ‘AI cannot currently match.’ Study co-author Daniel Martin Katz, a professor at Chicago-Kent College of Law, said in an interview that he was most surprised by GPT-4’s ability to produce largely relevant and coherent essay and performance test answers. ‘I heard so many people say, Well, it might get the multiple choice but it will never get the essays,’ Katz said.” 

AI has racked up similarly impressive results in taking the SAT and GRE tests.

Read “Bar Exam Score Shows AI Can Keep Up With Human Lawyers” from Reuters here.

Read more in the ABA Journal. 

Why choose Lawyers Mutual for your financial protection? One reason is experience. Lawyers Mutual has been in business since 1977 and insures more than 8000 lawyers in North Carolina. We’ve been providing continuous protection from professional liability to NC lawyers longer than any other insurance company. Another reason is stability. The company’s financial strength is absolute. Since 1999, Lawyers Mutual has paid dividends fourteen times, with more than $8 million dollars returned to policyholders since 2011. Want even more reasons? Visit our website, give us a call, or ask a colleague why Lawyers Mutual is the smart choice for liability coverage.

NC Board of Law Examiners: Laptop Requirements

The following is from the NC Board of Law Examiners website:

Beginning with the February 2023 North Carolina Bar Examination, all applicants must take the written portion of the examination (MPT and MEE) using a personal laptop computer using software designated by the Board, unless granted a testing accommodation due to a disability under the Americans with Disabilities Act.

To use your laptop for the essay portion of the bar examination you must: 

• Ensure your laptop is configured for access to the internet and meets the minimum system requirements for the ExamSoft software. This information is available on the Board’s ExamSoft homepage, www.examsoft.com/ncbar.
• Have experience using computers.
• Agree not to remove, delete or uninstall Examplify or any Examplify program files until results have been published.

• Complete the registration of your laptop during the registration period. To complete the registration of your laptop you must:
  • Complete and submit the Application to Take Written Examination on Laptop,
  • Pay the required fee,
  • Download the software, which will be sent to you by email.

• Install all components of the Examplify softwareonto the laptop you plan to use for the bar exam. Even if you currently have ExamSoft on your laptop, you will still need to download the software. The software has the testing parameters for the current bar exam.

• Complete and return two Mock Exams using Examplify prior to the published registration deadline.

• It is strongly recommended that you remove any old versions of ExamSoft from your laptop before you download a new version of the ExamSoft software. Removing older versions will help prevent delays the morning of the examination. If you have any questions about removing the older versions, please contact ExamSoft at 866-429-8889.

SOURCE: NC Board of Law Examiners

Everyone makes mistakes. When it happens to you, Lawyers Mutual has your back. For nearly half a century, Lawyers Mutual has been the smart choice for professional liability coverage for North Carolina attorneys. We cover the state from Murphy to Manteo. We insure large firms, solo practitioners and everyone in between. We help new lawyers enter the profession with confidence, and we help keep seasoned veterans safe and successful. The numbers speak for themselves. Lawyers Mutual has been in business since 1977, making us the only insurance carrier to provide continuous protection over that period. Today we insure more than 8000 lawyers in North Carolina. Most of them will stick with us until they retire. Why? Because they know we are here for them today and will be here tomorrow, bringing protection and peace of mind in turbulent times. Visit our website, give us a call, or ask a colleague why Lawyers Mutual is the smart choice for liability coverage.