Supreme Court sets aside NCLAT judgment over AI-generated fake citations

The Supreme Court on Thursday (July 2) set aside a judgment by the National Company Law Tribunal (NCLAT) after it found that the verdict had been delivered based on artificial intelligence (AI) generated fake and hallucinated judgments.

Zero-tolerance warning

Taking a stern view of the matter, the bench of Justices PS Narasimha and Alok Aradhe categorically stated that such unverified reliance on AI-generated hallucinated material goes against the integrity of adjudication. It warned that courts must go by a “zero tolerance” approach when it comes to dealing with fake AI-generated judicial precedents submitted by lawyers without any fact-checking and verification.

Also Read: Supreme Court draft AI rules mandate disclosure of AI use in court filings

The bench warned that submission of such hallucinated AI-generated matter would also be viewed as “misconduct” on the part of the concerned advocate.

“It is necessary for Courts to adopt a zero-tolerance mode for producing, citing or using AI-generated precedents without verification. It is a misconduct on the part of an advocate to cite such judgments without verification," stated the bench.

Judgment set aside

The top court further stated that any decision based on even the slightest amount of hallucinated material would be set aside, as it would go against the sanctity of adjudication.

Also Read: Indian courts have an AI problem, and the Supreme Court is not amused

"We have no hesitation in declaring that such a decision is no decision in the eyes of the law, irrespective of whether such material had a direct or indirect bearing on the decision-making. Such decisions are to be set aside even if an iota of fake or hallucinated material enters the decision-making process, as it would violate the sanctity of adjudication," stated the bench as quoted by the Bar and Bench.

Setting aside the related NCLT judgement and the order of the National Company Law Appellate Tribunal (NCLAT) which upheld the judgement, the apex court stated “For the reasons to follow, we have set aside the judgement of the NCLT, as well as the (NCLAT) judgement in opinion, to affirm and maintain the integrity of the adjudication and its processes.”

Human oversight

“What is significant for our decision-making is our resolve to adopt artificial intelligence technology in the aid of adjudication, while at the same time asserting, and declaring total and absolute control over adjudications, to the human in the loop, at every stage,” it added.

Also Read: CJI Surya Kant urges judiciary to use AI with caution, not fear

Warning against the pitfalls of AI, the court stated: “This is where we need to be extra cautious, as unregulated use of AI will insidiously enter legal practice and process of justice, decision making and decision making itself."

The bench said a mere declaration of prohibitory action is not sufficient, and there must be a consequential action following accountability.

“So far as the responsibility of the bar is concerned, we direct the Bar Council of India (BCI), being the apex statutory body, to constitute a committee and deliberate on this issue of members of the bar submitting such fake and hallucinated material before the Court as if they are precedents of law,” it said.

Fake precedents cited

The matter arose from an insolvency dispute involving Pooja Ramesh Singh, Jammu and Kashmir Bank Ltd, and Essel Infraprojects Ltd.

The appellant challenged an NCLT Mumbai order that had admitted a Section 7 insolvency application.

Also Read: SC flags lawyers using AI to draft petitions citing fake judgements

The top court found that several "precedents" cited by the NCLT to justify its decision simply did not exist.

These included fabricated case names and paragraphs wrongly attributed to genuine citations. For instance, the judgment cited ICICI Bank Ltd vs Urban Infrastructure Real Estate Ltd (2019) 16 SCC 528 and Sarbjit Singh vs Union Bank of India (2022) 7 SCC 464, both of which were found to be entirely non-existent citations.

Source no defence

While respondent Jammu and Kashmir Bank filed an affidavit clarifying that their counsel had not cited these cases and that the NCLT had obtained them through its "own research," the apex court held that the source of the error did not mitigate the damage to the rule of law.

"More than the inevitable consequence of setting aside such judgments, what is significant for our decision-making is our resolve to adopt AI technology in aid of adjudication, while at the same time asserting and declaring total and absolute control over adjudication, with a human in the loop at every stage," the bench observed.

(With agency inputs)