Taking note of Elon Musk-owned X’s artificial intelligence service Grok being abused to generate objectionable pictures and videos of women, without their consent, while increasing the risk of exposure and harm with “grave concern,” the Indian government has sent a missive to the company saying that it was not adhering to the country’s laws, raising red flags over “serious failure” of safeguards enforcement, and has sought information within 72 hours about technical details related to Grok’s actions.
“Grok AI developed by you and integrated and made available on the X platform, is being misused by users to create fake accounts to host, generate, publish or share obscene images or videos of women in a derogatory or vulgar manner in order to indecently denigrate them,” the IT Ministry said in the notice sent to the social media company’s chief compliance officer on Friday.
The government said that the service was being used by people to target women who host or publish their images or videos, through prompts, image manipulation and synthetic outputs, and said that “such conduct reflects a serious failure of platform-level safeguards and enforcement mechanisms, and amounts to gross misuse of artificial intelligence technologies in violation of applicable laws”.
This is being viewed with “grave concern,” as it could violate the dignity, privacy and safety of women and children, “normalising sexual harassment and exploitation in digital spaces, and undermining the statutory due diligence framework applicable to intermediaries operating in India,” the IT Ministry said.
The IT Ministry has directed X to submit a detailed action taken report within 72 hours, covering specific technical and organisational measures adopted by X in relation to Grok AI, the role and oversight exercised by its India chief compliance officer, and action taken by the company against offending content, users and accounts.
It has also directed the company to undertake a comprehensive technical, procedural and governance-level review of Grok AI, including its prompt-processing, output generation, and image handling and safety guardrails, “to ensure that the application does not generate, promote or facilitate content which contains nudity, sexualisation, sexually explicit or otherwise unlawful content”.
The notice said that due to Grok’s actions, X was not adequately adhering to provisions of the Information Technology (IT) Rules, 2021, and the Bharatiya Nagarik Suraksha Sanhita, 2023, particularly in relation to obscene, indecent, vulgar, pornographic, paedophilic, or otherwise unlawful or harmful content which are potentially violative of current laws.
The ministry has asked X to “strictly desist from the hosting, displaying, uploading, publication, transmission, storage, sharing” of such content, and any failure to do so could lead to the company losing its legal immunity from hosting third-party content.
Users on X are increasingly tagging Elon Musk’s AI chatbot Grok to generate sexualised and objectionable images of women without their consent, raising fresh concerns over AI misuse and platform safeguards. In several instances, users have replied to photos of women, often taken from public posts, and prompted Grok to alter their appearance by adding revealing clothing or creating suggestive visuals, which the AI-bot has promptly done. The AI-generated images appear publicly in the same threads, exposing the women to harassment without their approval or knowledge.
Shiv Sena (UBT) MP Priyanka Chaturvedi had earlier written to Union Minister for Electronics and Information Technology Ashwini Vaishnaw raising concerns over the misuse of Grok AI to generate such images of women. “There have to be guardrails put in place by features like Grok so that they do not violate women’s dignity. Big tech firms need to take the onus,” she said.
“It is not just limited to sharing photos through fake accounts but are also targeting women who post their own photos. This is unacceptable and a gross misuse of an AI function,” she said, adding that Grok was enabling such behaviour by adhering to such requests.
Vaishnaw told reporters that the issue makes a strong case for the government’s intervention, and added that the ministry was considering a law to regulate AI content on social media. “The Parliamentary Committee has recommended a strong law for regulating social media. We are considering it,” he said.
In October, The Indian Express had reported how AI-generated clips and pictures of actors were widely available on platforms like Instagram and X, with the platforms seemingly failing in their responsibility to curb the spread of such content, jeopardising the safety and dignity of women, which could create barriers to their participation in digital spaces.
In December, the IT Ministry had issued an advisory to online platforms, including social media companies, asking them not to host obscene, or vulgar content, calling for “greater rigour” in adhering to provisions of the law. The ministry has also directed platforms to carry out an immediate review of their internal compliance frameworks.
The Centre took note of Elon Musk-owned X’s artificial intelligence service Grok being abused to generate objectionable pictures (File Photo) 
Share
