GPT-4, an AI-powered modernized language model released this week by Microsoft-backed OpenAI, scored 297 on the bar exam in an experiment conducted by two professors and two employees at law tech company Casetext. The researchers said that this is enough to be allowed to practice law in most states.
The bar exam assesses knowledge and reasoning, and includes essays and tests designed to simulate legal work, as well as multiple-choice questions.
“Large language models can meet the standard applied to human lawyers in almost all U.S. jurisdictions, as they solve complex problems that require deep legal knowledge, reading comprehension, and writing skills,” the authors write.
Less than four months ago, two of the same researchers concluded that OpenAI’s earlier large language model, ChatGPT, failed to pass on the bachelor’s degree exam, highlighting how quickly the technology is improving.
GPT-4 correctly answered almost 76% of multiple-choice questions on the exam, compared to about 50% for ChatGPT, outperforming the average person who takes this exam by 7%.
Daniel Martin Katz, a professor at Chicago-Kent College of Law, said he was most surprised by the GPT-4’s ability to write compelling essays as well as pass tests.
