State Bar of California admits it used AI to develop exam questions, triggering new furor
- Share via
Nearly two months after hundreds of prospective California lawyers complained that their bar exams were plagued with technical problems and irregularities, the state’s legal licensing body has caused fresh outrage by admitting that some multiple-choice questions were developed with the aid of artificial intelligence.
The State Bar of California said in a news release Monday that it will ask the California Supreme Court to adjust test scores for those who took its February bar exam.
But it declined to acknowledge significant problems with its multiple-choice questions — even as it revealed that a subset of questions were recycled from a first-year law student exam, while others were developed with the assistance of AI by ACS Ventures, the State Bar’s independent psychometrician.
“The debacle that was the February 2025 bar exam is worse than we imagined,” said Mary Basick, assistant dean of academic skills at UC Irvine Law School. “I’m almost speechless. Having the questions drafted by non-lawyers using artificial intelligence is just unbelievable.”
‘Utterly Botched’: Glitchy rollout of new California bar exam prompts lawsuit and legislative review
New state bar exam plagued with glitches. Some test takers are considering suing the State Bar of California.
After completing the exam, Basick said, some test takers complained that some of the questions felt as if they were written by AI.
“I defended the bar,” Basick said. “‘No way! They wouldn’t do that!’”
Using AI-developed questions written by non-legally-trained psychometricians represented “an obvious conflict of interest,” Basick argued, because “these are the same psychometricians tasked with establishing that the questions are valid and reliable.”
“It’s a staggering admission,” agreed Katie Moran, an associate professor at the University of San Francisco School of Law who specializes in bar exam preparation.
“The State Bar has admitted they employed a company to have a non-lawyer use AI to draft questions that were given on the actual bar exam,” she said. “They then paid that same company to assess and ultimately approve of the questions on the exam, including the questions the company authored.”
The State Bar, which is an administrative arm of the California Supreme Court, said Monday that the majority of multiple-choice questions were developed by Kaplan Exam Services, a company it contracted with last year as it sought to save money.
According to a recent presentation by the State Bar, 100 of the 171 scored multiple-choice questions were made by Kaplan and 48 were drawn from a first-year law students exam. A smaller subset of 23 scored questions were made by ACS Ventures, the State Bar’s psychometrician, and developed with artificial intelligence.
“We have confidence in the validity of the [multiple-choice questions] to accurately and fairly assess the legal competence of test-takers,” Leah Wilson, the State Bar’s executive director, said in a statement.
On Tuesday, a spokesperson for the State Bar told The Times that all questions — including the 29 scored and unscored questions from the agency’s independent psychometrician that were developed with the assistance of AI — were reviewed by content validation panels and subject matter experts ahead of the exam for factors including legal accuracy, minimum competence and potential bias.
When measured for reliability, the State Bar told The Times, the combined scored multiple-choice questions from all sources — including AI — performed “above the psychometric target of 0.80.”
The State Bar also dismissed the idea of a conflict of interest.
“The process to validate questions and test for reliability is not a subjective one,” the State Bar said, “and the statistical parameters used by the psychometrician remain the same regardless of the source of the question.”
The California State Bar’s new exam for prospective lawyers was a disaster, with complaints from many test takers. Bar staff now recommend using the previous system.
Alex Chan, an attorney who serves as chair of the State Bar’s Committee of Bar Examiners, told The Times that only a small subset of questions used AI — and not necessarily to create the questions.
“The professors are suggesting that we used AI to draft all of the multiple choice questions, as opposed to using AI to vet them,” Chan said. “That is not my understanding.”
Chan noted that the California Supreme Court urged the State Bar in October to review “the availability of any new technologies, such as artificial intelligence, that might innovate and improve upon the reliability and cost-effectiveness of such testing.”
“The court has given its guidance to consider the use of AI, and that’s exactly what we’re going to do,” Chan said.
That process, Chan explained, would be subject to the court’s review and approval.
“The Committee was never informed about the use of AI before the exam took place,” he said, “so it could not have considered, much less endorsed, its use.”
A spokesperson for California’s highest court said Tuesday that justices found out only this week that the State Bar had utilized AI in developing exam questions.
“Until yesterday’s State Bar press release, the court was unaware that AI had been used to draft any of the multiple-choice questions,” a spokesperson said in a statement.
Last year, as the State Bar faced a $22-million deficit in its general fund, it decided to cut costs by ditching the National Conference of Bar Examiners’ Multistate Bar Examination, a system used by most states, and move to a new hybrid model of in-person and remote testing. It cut an $8.25-million deal with test prep company Kaplan Exam Services to create test questions and hired Meazure Learning to administer the exam.
There were multiple problems with the State Bar’s rollout of the new exams. Some test takers reported they were kicked off the online testing platforms or experienced screens that lagged and displayed error messages. Others complained the multiple-choice test questions had typos, consisted of nonsense questions and left out important facts.
The botched exams prompted some students to file a federal lawsuit against Meazure Learning. Meanwhile, California Senate Judiciary Chair Thomas J. Umberg (D-Santa Ana) called for an audit of the State Bar and the California Supreme Court directed the agency to revert to traditional in-person administering of July bar exams.
But the State Bar is pressing forward with its new system of multiple-choice questions — even though some academic experts have repeatedly flagged problems with the quality of the February exam questions.
“Many have expressed concern about the speed with which the Kaplan questions were drafted and the resulting quality of those questions,” Basick and Moran wrote April 16 in a public comment to the Committee of Bar Examiners. “The 50 released practice questions — which were heavily edited and re-released just weeks before the exam — still contain numerous errors. This has further eroded our confidence in the quality of the questions.”
Historically, Moran said, exam questions written by the National Conference of Bar Examiners have taken years to develop.
A resentencing hearing that could have given the brothers a shot at parole was delayed after a fight over a parole document ordered by Gov. Gavin Newsom paralyzed proceedings.
Reusing some of the questions from the first-year law exam raised red flags, Basick said. An exam to figure out if a person had learned enough in their first year of law school is different from one that determines whether a test taker is minimally competent to practice law, she argued.
“It’s a much different standard,” she said. “It’s not just, ‘Hey, do you know this rule?’ It is, ‘Do you know how to apply it in a situation where there’s ambiguity, and determine the correct course of action?’”
Also, using AI and recycling questions from a first-year law exam represented a major change to bar exam preparation, Basick said. She argued such a change required a two-year notice under California’s Business and Professions Code.
But the State Bar told The Times that the sources of the questions had not triggered that two-year notice.
“The fact there were multiple sources for the development of questions did not impact exam preparation,” the State Bar said.
Basick said she grew concerned in early March when, she said, the State Bar kicked her and other academic experts off their question-vetting panels.
She said the State Bar argued that those law professors had worked with questions drafted by the National Conference of Bar Examiners in the last six months, which could raise issues of potential copyright infringement.
“Ironically, what they did instead is have non-lawyers draft questions using artificial intelligence,” she said. “The place the artificial intelligence would have gotten their information from has to be the NCBE questions, because there’s nothing else available. What else would artificial intelligence use?”
Ever since the February exam debacle, the State Bar has underplayed the idea that there were substantial problems with the multiple-choice questions. Instead, it has focused on the problems with Meazure Learning.
“We are scrutinizing the vendor’s performance in meeting their contractual obligations,” the State Bar said in a document that listed the problems test takers experienced and highlighted the relevant performance expectations laid out in the contract.
But critics have accused the State Bar of shifting blame — and argued it has failed to acknowledge the seriousness of the problems with multiple-choice questions.
Moran called on the State Bar to release all 200 questions that were on the test for transparency and to allow future test takers a chance to get used to the different questions. She also called on the State Bar to return to the multi-state bar exam for the July exams.
“They have just shown that they cannot make a fair test,” she said.
Chan said the Committee of Bar Examiners will meet on May 5 to discuss non-scoring adjustments and remedies. But he doubted that the State Bar would release all 200 questions or revert to the National Conference of Bar Examiners exams in July.
The NCBE’s exam security would not allow any form of remote testing, he said, and the State Bar’s recent surveys showed almost 50% of California bar applicants want to keep the remote option.
“We’re not going back to NCBE — at least in the near term,” Chan said.
Sign up for Essential California
The most important California stories and recommendations in your inbox every morning.
You may occasionally receive promotional content from the Los Angeles Times.
