By Aanya Rao
A practising advocate at the Supreme Court of India has published the first benchmark of how frontier artificial intelligence models perform on real Indian legal work, and the headline number is not reassuring. Across 1,348 graded answers from twelve models, 37.7 percent relied on a case or a section of an Act that does not exist, or that says the opposite of what the answer claimed. Fewer than one answer in six met every line of the standard a competent associate would be held to.
Dewans Legal Bench, published on 27 October at bench.infeenity.org and updated on 29 October, is the work of Dewan Vakul Vinod Goyal, Advocate, who practises before the Supreme Court and the Punjab and Haryana High Court from Dewans & Associates, a chamber at the Bar since 1919. As far as can be found, it is the first legal-AI benchmark anywhere written, run and hand-graded by a practising advocate from his own case files, and the first for Indian law.
The benchmark arrives eight months after a bench of the Chief Justice of India stopped a hearing to call AI-drafted petitions with invented citations “absolutely uncalled for”, and said that a decision resting on such authorities would be misconduct, not error. The Supreme Court has since set aside an insolvency tribunal order built on judgments that do not exist.
The 160 tasks are written the way a partner hands work to an associate. Sixty-five come from the advocate’s own matters, anonymised: a cheque-dishonour complaint, a special leave petition timetable, a partition suit in Punjab, a disability pension appeal, a will contested on attestation. Ninety-five are commercial and cross-border matters of the kind that reach a firm from a client in Singapore, Dubai, London or New York. Each task carries the instruction, the facts, the documents and the dates, and nothing else. The models answered with no access to the internet or any research tool.
For each task Dewan wrote a model answer and a rubric, with the critical points marked. Flagship models applied the rubric to every answer line by line, a second model family cross-checked, and he checked a fifth of the grades by hand. Every authority flagged as invented was verified against the judgment or the Act before it was counted.
The models did not fail quietly. They failed fluently. The best of the twelve, Anthropic’s Claude Fable 5.1, scored 88.4 out of 100 and invented an authority in about one answer in sixteen. OpenAI’s GPT-6 Sol Pro cited more carefully than any model tested, with a bad citation in only one answer in twenty, yet got a critical point of law wrong in 43.8 percent of its answers. The two free open-weights models on the board, Zhipu’s GLM-5.3 and gpt-oss-120b, invented authority in most of their answers, and account for much of the pooled rate. Every model was weakest in foreign exchange law and international arbitration, the two subjects India’s lawyers most often sell abroad.
One row on the board did not come from the nine companies tested. Infee Legal 5.1, the legal model built by Infee, the AI research company Dewan founded, was graded by him by hand on the same 160 tasks and the same rubrics. It scored 99.0, passed every line in 82.4 percent of answers, and produced no invented authority in any answer. The board shows it apart from the pooled figures, marked as hand-graded, with the author’s interest in Infee printed beside it.
“The question every client now asks is whether the machine can be trusted with their matter,” Dewan said. “That question deserves an answer with a number in it, and the only people who can produce that number for Indian law are the ones who practise it.”
The task set is private and will stay so; five sample tasks are published. Vendors of closed legal-AI products are invited to be scored on the same tasks under the same terms, free, with their grades shown to them before publication. A second edition covering Singapore and the United Arab Emirates is in preparation.
The board, the method and the sample tasks are at bench.infeenity.org.
Aanya Rao is a Junior Associate with Infeenity Law.
