Fake judgments, AI ‘hallucinations’: Supreme Court asks Bar Council of India to probe risks of artificial intelligence
The Supreme Court’s Centre for Research and Planning has already prepared a white paper on the use of artificial intelligence in the judiciary, containing recommendations and guidelines.
The Supreme Court expressed concern over the absence of sovereign large language models and the possibility of “hallucinations” in AI-generated content. (Image generated using AI)
Supreme Court: The Supreme Court on Wednesday asked the Bar Council of India to set up a committee of experts, including specialists from relevant fields, to examine concerns surrounding the use of artificial intelligence (AI) in court proceedings.
The matter has now been listed for further hearing on May 26.
Justices PS Narasimha and Alok Aradhe of the Supreme Court said that the BCI committee should prepare and submit a report on the issue. (Image enhanced using AI)
Prepare report
During Wednesday’s hearing, the bench stressed the need for accountability and integrity in judicial proceedings.
The judges remarked that parties cannot rely on AI-generated material and later escape responsibility by offering an apology.
The court also noted that similar concerns arise when judges themselves rely on research that leads them to cite non-existent judgments, and questioned where accountability would lie in such situations.
The bench expressed concern over the absence of sovereign large language models (LLMs) and the possibility of “hallucinations” in AI-generated content.
At the same time, the top court clarified that it was not attempting to ban or discourage the use of artificial intelligence.
Senior Advocate Divan said the document could serve as a starting point for the court while it considers possible directions in the matter.
The direction came in a case where a trial court had relied on judgments that did not actually exist and were allegedly generated using AI.
The bench said the committee should prepare and submit a report on the issue. However, the court clarified that it was not passing any formal order at this stage.
Attorney General R Venkataramani told the Supreme Court that he would consult the Ministry of Electronics and Information Technology before presenting his views on the issue.
The court again observed that the issue is aggravated by the absence of sovereign large language models and the use of open-source algorithms, where hallucinations may become inevitable.
It also added that the intention is not to prevent the use of AI but to address the associated risks.
Background
The matter arose from a special leave petition (SLP) challenging a trial court order in a suit for an injunction.
The trial court had dismissed objections to an advocate commissioner’s report while relying on four purported Supreme Court decisions – Subramani vs M Natarajan (2013) 14 SCC 95, Ramasamy (1071) 2 SCC 68, Chidambaram Pillai vs SAL Lakshmi Devi vs K Prabha (2006) 5 SCC 551, and Gajanan vs Ramdas (2015) 6 SCC 223.
The petitioners argued that these judgments did not exist and had been generated using artificial intelligence.
The high court, after observing that the citations were AI-generated, decided the matter on the merits and dismissed the civil revision petition, while also issuing a word of caution.
Story continues below this ad
February 27 hearing
When the case reached the Supreme Court, the bench observed that reliance on fake, AI-generated judgments could not be treated as a simple judicial error and would amount to misconduct carrying legal consequences.
“At the outset, we must declare that a decision based on such non-existent and fake alleged judgments is not an error in the decision-making. It would be a misconduct, and legal consequences shall follow,” the top court had said on February 27 and issued notice to the attorney general, solicitor general, and the Bar Council of India.
The apex court also appointed Senior Advocate Shyam Divan as amicus curiae.
Vineet Upadhyay is an Assistant Editor with The Indian Express, where he leads specialized coverage of the Indian judicial system.
Expertise
Specialized Legal Authority: Vineet has spent the better part of his career analyzing the intricacies of the law. His expertise lies in "demystifying" judgments from the Supreme Court of India, various High Courts, and District Courts. His reporting covers a vast spectrum of legal issues, including:
Constitutional & Civil Rights: Reporting on landmark rulings regarding privacy, equality, and state accountability.
Criminal Justice & Enforcement: Detailed coverage of high-profile cases involving the Enforcement Directorate (ED), NIA, and POCSO matters.
Consumer Rights & Environmental Law: Authoritative pieces on medical negligence compensation, environmental protection (such as the "living person" status of rivers), and labor rights.
Over a Decade of Professional Experience: Prior to joining The Indian Express, he served as a Principal Correspondent/Legal Reporter for The Times of India and held significant roles at The New Indian Express. His tenure has seen him report from critical legal hubs, including Delhi and Uttarakhand. ... Read More
Justices PS Narasimha and Alok Aradhe of the Supreme Court said that the BCI committee should prepare and submit a report on the issue. (Image enhanced using AI)