How an AI tool by a queer engineer is making the internet a safe place, one platform at a time

The story behind Shhor AI

“I am an artist and activist from rural Tripura. Growing up, I struggled to fit in because of my marginalised identity but I couldn’t fully understand why I was treated differently. I expressed these feelings through my art,” says Barua, who is queer, indigenous and neurodivergent. Barua identifies as non-binary and their preferred pronouns are they/them. “When your lived experiences are rooted in marginalisation, your art naturally becomes political—you don’t have to choose it,” Barua adds.

Barua, who is also a computer science engineering graduate, explains, “When I started posting my art online, I connected with others who shared similar experiences. I received love and support, but I also faced an overwhelming amount of hatred and abuse. I realised that the discrimination we go through in real life has also percolated to online spaces,” Barua says.

This prompted the idea of Shhor AI. The AI solution recently won the prestigious Just AI Awards 2024 in the ‘AI for Social Good’ category at the Global AI Summit 2024 held in Hyderabad. Launched in 2024, the awards conceptualised by the Digital Empowerment Foundation and institutionalised as an annual award system in partnership with the emerging technologies wing of the Telangana government and the World Summit Awards recognise and celebrate the initiatives, achievements and efforts of tech innovators with a citizen-centric approach.

Barua pursued a career in artificial intelligence. During college, they actively studied Natural Language Processing (NLP), a branch of AI that focuses on understanding and processing human language. “My research was focused on Indian languages and transformers—the very foundation of what we now know as large language models (LLMs). This was back in 2020 before the large-scale use of LLMs reached mainstream attention,” says Barua.

With a deep understanding of AI and NLP, Barua realised they had a unique skill set that could address a pressing issue. They worked on a side project while employed as an AI engineer.

It was not an easy task as Barua had to create their own dataset. They connected with other members of the community online and realised that the idea resonated with people who started sending Barua links. That was when Barua realised that this could not be handled manually.

So an online space was created where people could send links. Barua then made a scrapper, a tool to extract data, which regularly went through the space and collected the comments. Later, Barua would give a shoutout on social media, and volunteers would contribute to flag hate comments.

How the AI model works

Shhor AI was trained on a large crowd-sourced dataset. It is a content moderation API that is not limited to any single platform but can be integrated into various online spaces to detect and combat hate speech. One such integration is as a Reddit bot, where the model was deployed to moderate specific subreddits (niche communities on the Reddit platform) in real time. This API, integrated with Reddit, got them recognised by the United Nations Population Fund (UNFPA) for building a solution to tackle technology-facilitated gender violence.

The main problem with Indian hate speech, Barua highlights, is that there are various languages but most of the keyboards are in English. Users need to type their language using the English keyboard, so there is no fixed spelling for words in Hindi. So there cannot be an exact list of slurs in the South Asian context.

This led Barua to focus on Hinglish or Hindi plus English. The model is currently trained on a unique dataset that captures and identifies hate speech in Hinglish, which is rarely addressed in existing moderation systems.

The bot was first integrated into Reddit as the platform’s user-friendly API allows for easy automation of moderation tasks, giving subreddit admins the flexibility to manage content, unlike Instagram, which follows strict community guidelines set by Meta. Just like the Shhor AI Reddit bot, the model can be integrated into various other platforms and content moderation can be customised.

“The AI bot, integrated into a particular subreddit, scans posts and comments, issues warnings, and keeps track of repeat offenders. After three warnings, if the user still posts hate speech, they will be banned from the subreddit. However, if a banned user believes the AI made an error, they can directly message the human moderator, who will review the case and make a final decision,” says Barua. The warning is publicly posted as a reply to the hate comment.

Where content moderation fails

According to Barua, centralised moderation does not understand the regional, cultural, or historical contexts of hate speech. “You can report a comment in a regional language but platforms like Instagram will not remove it because it does not violate their global community standards,” says Barua.

Shhor AI addresses this gap by taking a decentralised approach, encouraging platforms to allow third-party moderation for specific regions or communities. “We need moderation tools that are aware of local contexts—both historical and present-day. AI should be trained with that knowledge, or it will continue to fail marginalised groups,” adds Barua.

Hate speech: What the statistics say

Shhor AI identifies eight categories of hate speech: Gendered hate, racism, political hate, communalism, queerphobia, casteism, ableism (disability), and general hate.

According to Barua, most hate speech in India is directed against gender and sexual minorities. The data from Shhor AI shows that 40 per cent of online hate speech is targeted at gender and sexual minorities, followed by 23 per cent at activists and political leaders, and 11 per cent on communalism.

Shhor AI also analysed 5,703 electoral hate comments on Indian social media from October 2022 to April 2024. The intersection between election-related hate and communal hate is 78 per cent more than the next biggest intersection which is with sexist hate speech, the data revealed.

Future of content moderation

According to Barua, the future of content moderation lies in decentralisation, much like the third-party fact-checking employed by Meta. Platforms need to open their systems to external tools like Shhor AI, which are tailored to local contexts. “The goal is to create an internet that’s safe for everyone, especially those from marginalised communities,” says Barua.

As Shhor AI grows and develops, its impact is undeniable. Born from personal experience and fuelled by the collective efforts of a community, it stands as a powerful tool aimed at making the internet a safer, more inclusive space.