Tags:
Journalism of Courage
ChatGPT will be able to have kinkier conversations after OpenAI CEO Sam Altman announced the artificial intelligence company will soon allow its chatbot to engage in “erotica for verified adults.”
OpenAI won’t be the first to try to profit from sexualized AI. Sexual content was a top draw for AI tools almost as soon as the boom in AI-generated imagery and words erupted in 2022.
But the companies that were early to embrace mature AI also encountered legal and societal minefields and harmful abuse as a growing number of people have turned to the technology for companionship or titillation.
Will a sexier ChatGPT be different? After three years of largely banning mature content, Altman said Wednesday that his company is “not the elected moral police of the world” and ready to allow “more user freedom for adults” at the same time as it sets new limits for teens.
“In the same way that society differentiates other appropriate boundaries (R-rated movies, for example) we want to do a similar thing here,” Altman wrote on social media platform X, whose owner, Elon Musk, has also introduced an animated AI character that flirts with paid subscribers.
For now, unlike Musk’s Grok chatbot, paid subscriptions to ChatGPT are mostly pitched for professional use. But letting the chatbot become a friend or romantic partner could be another way for the world’s most valuable startup, which is losing more money than it makes, to turn a profit that could justify its $500 billion valuation.
There are already about 29 million active users of AI chatbots designed specifically for romantic or sexual bonding, and that’s not counting people who use conventional chatbots in that way, according to research published by Qian earlier this month.
It also doesn’t include users of Character.AI, which is fighting a lawsuit that alleges a chatbot modeled after “Game of Thrones” character Daenerys Targaryen formed a sexually abusive relationship with a 14-year-old boy and pushed him to kill himself. OpenAI is facing a lawsuit from the family of a 16-year-old ChatGPT user who died by suicide in April.
Qian said she worries about the toll on real-world relationships when mainstream chatbots, already prone to sycophancy, are primed for 24-hour availability serving sexually explicit content.
“ChatGPT has voice chat versions. I would expect that in the future, if they were to go down this way — voice, text, visual — it’s all there,” she said.
Humans who fall in love with human-like machines have long been a literary cautionary tale, from popular science fiction of the last century to the ancient Greek legend of Pygmalion, obsessed with a woman he sculpted from ivory. Creating such a machine would seem like an unusual detour for OpenAI, founded a decade ago as a nonprofit dedicated to safely building better-than-human AI.
Altman said on a podcast in August that OpenAI has tried to resist the temptation to introduce products that could “juice growth or revenue” but be “very misaligned” with its long-term mission. Asked for a specific example, he gave one: “Well, we haven’t put a sexbot avatar in ChatGPT yet.”
Idaho-based startup Civitai, a platform for AI-generated art, learned the hard way that making money off mature AI won’t be an easy path.
“When we launched the site, it was an intentional choice to allow mature content,” said Justin Maier, the company’s co-founder and CEO, in an interview last year.
Backed by the prominent venture capital firm Andreessen Horowitz, which has also invested in OpenAI, the Idaho startup was one of several that tried to capitalize on the sudden popularity of tools like Stable Diffusion and Midjourney that enabled people to type a description and conjure up almost any kind of image. Part of Stable Diffusion’s initial popularity was the ease with which it could generate a new kind of synthetic and highly customized pornography.
“What we had seen was that there was a lot of interest in mature content,” Maier said. Training these AI systems, known as models, on “mature themes actually made it so that these models were more capable of human anatomy and resulted in actually better models,” he said.
“We didn’t want to prevent the kind of growth that actually increased everything for the entire community, whether you were interested in mature content or Pixar,” Maier said. “So we allowed it early on and have always kind of had this battle of making it so that we can keep things filtered and safe, if that’s not what you’re interested in. We wanted to ultimately give the control to the user to decide what they would see on the site and what their experience would be.”
That also invited abuse. Civitai last year implemented new measures to detect and remove sexual images depicting children, but it remained a hub for AI-generated pornography, including fake images of celebrities. Confronting increasing pressure, including from credit card processors and a new law against nonconsensual images signed by President Donald Trump, Civitai earlier this year blocked users from creating deepfake images of real people. Engagement dropped.
Another company that hasn’t shied away from mature content is Baltimore-based Nomi.ai, though its founder and CEO Alex Cardinell said its companion chatbots are “strictly” for users over 18 and were never marketed to kids. They are also not designed for sex, though Cardinell said in an interview earlier this year that people who build platonic relationships with their chatbot might find it veering into a romantic one.
“It’s kind of very user-dependent for where they’re kind of missing the human gap in their life. And I think that’s different for everyone,” he said.
He declined to guess how many Nomi users are having erotic conversations with the chatbot, comparing it to real-life partners who might do “mature content things” for some part of their lives but “all sorts of other stuff together as well.”
“We’re not monitoring user conversations like that,” Cardinell said.
Altman’s announcement that erotica for adults could arrive on ChatGPT in December came a day after California Gov. Gavin Newsom vetoed legislation that would have banned companies from making AI chatbots available to anyone under 18 years old if it was “foreseeable” that they would engage in “erotic or sexually explicit interactions” with kids or encourage them to harm themselves. The tech industry lobbied heavily against the bill, which Newsom said was too broad, but OpenAI, Meta and others introduced new age restrictions and parental controls for AI-teen interactions.
