5 min readNew DelhiJan 5, 2026 09:06 AM IST
First published on: Jan 4, 2026 at 01:13 PM IST
Days after the Indian government sought technical details from Elon Musk-owned X related to the misuse of Grok, the social media platform has warned users against prompting the built-in AI chatbot to generate illegal content.
“Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content,” the official account of X’s safety team said in a post on Sunday, January 4. Musk reiterated this in his reply to a post saying that Grok cannot be held responsible for user prompts.
During the past week, several users on X have been commenting on images of people – usually women – by tagging Grok and asking the chatbot to “put her in a bikini” or “take her dress off”.
The AI chatbot, which auto-replies when tagged, complied with these user requests and generated non-consensual sexual images of both celebrities and non-celebrities, including some who appeared to be young children.
“We take action against illegal content on X, including Child Sexual Abuse Material (CSAM), by removing it, permanently suspending accounts, and working with local governments and law enforcement as necessary,” the platform’s safety team said. xAI, Musk’s AI startup that has developed Grok, also prohibits the sexualisation of children in its acceptable-use policy.
This is far from the first time that Grok has become embroiled in a controversy. The ‘anti-woke’ AI chatbot’s ‘unhinged’ mode sparked chaos in India last year, when it replied to users’ posts with expletives and misogynistic slurs in Hindi slang. It has also previously praised Adolf Hitler, re-shared conspiracy theories, and made inappropriate comments in response to users’ posts on X.
However, this appears to be the first instance of Grok’s image-generation capabilities being misused at scale. The incident extends beyond Grok, and underscores how generative AI continues to lower the barriers to create non-consensual sexually explicit imagery and child sexual abuse material (CSAM).
It also raises deeper, unresolved questions about who should be held responsible for harmful or illegal AI-generated content, whether AI companies like xAI are publishers or intermediaries, and how regulators plan to rein in the scores of shadowy ‘nudify’ platforms that could be tougher to police than mainstream AI chatbots and tools.
Govt notice to X
Taking note of Grok being abused to generate objectionable pictures and videos of women, without their consent, the IT Ministry said it has sent a directive to X which accused the social media platform of not adhering to the country’s laws and raised red flags over X’s “serious failure” to enforce safeguards.
“Grok AI developed by you and integrated and made available on the X platform, is being misused by users to create fake accounts to host, generate, publish or share obscene images or videos of women in a derogatory or vulgar manner in order to indecently denigrate them,” the IT Ministry said in the notice sent to the social media company’s chief compliance officer on Friday, January 2.
The IT Ministry directed X to submit a detailed action taken report within 72 hours, covering specific technical and organisational measures adopted by X in relation to Grok AI, the role and oversight exercised by its India chief compliance officer, and action taken by the company against offending content, users and accounts.
It also directed the company to undertake a comprehensive technical, procedural and governance-level review of Grok AI, including its prompt-processing, output generation, and image handling and safety guardrails, “to ensure that the application does not generate, promote or facilitate content which contains nudity, sexualisation, sexually explicit or otherwise unlawful content”.
The IT Ministry’s intervention came days after Shiv Sena (UBT) MP Priyanka Chaturvedi said that she had penned a letter to Union Minister for Electronics and Information Technology Ashwini Vaishnaw on the misuse of Grok to generate such images of women.
“There have to be guardrails put in place by features like Grok so that they do not violate women’s dignity. Big tech firms need to take the onus. It is not just limited to sharing photos through fake accounts but are also targeting women who post their own photos. This is unacceptable and a gross misuse of an AI function,” Chaturvedi wrote in her letter.
Many users turn to AI chatbots like Grok and Ask Perplexity to verify claims circulating on X. (File photo) 
Share
