Red flags in Govt: Can GenAI models track prompts of officials, leverage citizens’ data?

Senior officials say the core issue is not only data privacy but inference risk: whether these systems can derive sensitive insights indirectly from users’ behaviour, relationships, and search patterns.

Two broad areas are under discussion. First, whether queries made by top functionaries — senior bureaucrats, policy advisers, scientists, corporate leaders and influential academics — could be mapped to identify priorities, timelines, or weaknesses.

Second, whether anonymised mass usage data from millions of Indian users could help global firms.

One issue being discussed, sources said, is whether to “protect” official systems from foreign AI services.

“We don’t know what the level of tracking is on these services, and whether they are able to identify the significance of a user’s prompts to make inferences from it. For now, the foreign LLMs are most popular, and there can be more safety in running them directly on a computer rather than on a server. But, of course, the most secure way would be to use locally made LLMs which are currently being developed. Until we have that, we are keeping a close eye on these issues,” a senior government official said.

Another top official indicated that a debate is on within the government over this issue.

The IT Ministry did not respond to a formal request for comment.

Some of these anxieties have led to specific directions from at least one major government department to not use AI services on official workstations. In February, the Finance Ministry directed its employees to “strictly” avoid use of such tools “like ChatGPT and DeepSeek” in office computers and devices over concerns pertaining to confidentiality of official documents and data.

“It has been determined that AI tools and AI apps (such as ChatGPT, DeepSeek etc.) in the office computers and devices pose risks for confidentiality of Govt data and documents,” the memo by the ministry said.

Some departments have already moved to limit exposure. In February, the Finance Ministry issued a directive prohibiting the use of GenAI platforms such as ChatGPT and DeepSeek on official workstations and devices, citing risks to “confidentiality of Government data and documents.”

The concerns come at a time when India is funding the development of indigenous large language models (LLMs) under its Rs 10,370-crore India AI Mission. At least 12 LLMs and smaller domain-specific models are being developed with government support. One of these, led by Bengaluru-based start-up Sarvam, is expected to roll out by year-end, targeting governance and public sector use cases.

The debate intersects with a larger political push.

The government has consistently urged the use of swadeshi digital tools instead of foreign platforms, a message that’s grown sharper amid strained trade ties with the United States over tariffs and H-1B visa caps. Globally dominant US-based platforms — ChatGPT, Gemini, WhatsApp, YouTube, Instagram, Gmail, X — shape much of India’s digital communication and information environment.

Last month, in a meeting with top Secretaries, the Prime Minister is understood to have flagged the need for domestic digital platforms in communication and knowledge ecosystems, not just in payments and identity. IT Minister Ashwini Vaishnaw publicly shifted to Zoho’s Indian office suite, and Home Minister Amit Shah announced that his official email would move to Zoho Mail.

Over the last six months, at least three GenAI companies have rolled out free access to user groups in India. OpenAI’s basic ChatGPT Go plan will be made available free to consumers in India for a year while Alphabet’s Gemini Pro will be offered to each of Reliance Jio’s over 500 million subscribers for 18 months. Perplexity AI will offer its Pro version to Bharti Airtel’s 350 million users.

There is some history to this. At the height of its sour relationship with X (then Twitter) in 2021, several government officials tried propping up Koo as an alternative. The app failed and shut down operations last year. After border clashes with China in 2020, India had banned several popular apps such as TikTok, citing national security concerns.

Incidentally, a subcommittee formed by the IT ministry under the IndiaAI Mission submitted its report Wednesday on governance guidelines for AI companies, which among other things, recommended the creation of an India-specific risk assessment framework that reflects real-world evidence of harm. It also advised a “whole of government approach” where ministries, sectoral regulators, and other public bodies work together to develop and implement AI governance frameworks.