OpenAI’s ChatGPT is seen as a path-breaking chatbot for AI. But experts are not impressed

Read more: ChatGPT can be used to write phishing emails, malicious code, warn security experts

“There may be a danger here in believing and using things generated by ChatGPT. At its core, this tool (or many like it) don’t really have an understanding or knowledge of an area like a human expert would,” Chirag Shah, Professor, University of Washington, and Co-Director, Responsibility in AI Systems & Experiences (RAISE), told indianexpress.com over an email.

A review of the Xiaomi 12 Pro written by ChatGPT. Some of the information such as about the processor is wrong as are details about the camera.

This becomes evident once you start looking for context, and even accuracy in some of the responses from ChatGPT. For instance, this writer asked ChatGPT to say write a review for a phone such as the Xiaomi 12 Pro. While the essay itself reads fine, much of the information in it is wrong.

Shah pointed out that while ChatGPT “is able to leverage the latest advancements in generative models (specifically, GPT-3.5) and natural language generation,” and is a “huge improvement” but there are still several challenges. For one, he does not believe ChatGPT can replace “humans for writing jobs just yet — not as long as we question intent, authoritativeness, and provenance.” But he added that it can provide, “a starting point for many such projects.”

According to Arvind Narayanan, Professor of Computer Science, Princeton University, what users need to remember is that “just because a piece of text sounds authoritative, it isn’t necessarily so. It has always been important to check your sources, and it’s become even more important now,” given such AI tools.

Narayanan went on to call ChatGPT a ‘bulls***-generator’ in his blog post, cautioning that “you can’t tell when it’s wrong unless you already know the answer.” He gave examples of how in his testing—along with co-author Sayash Kapoor—they asked ChatGPT about “some basic information security questions,” and the answers, while they sound plausible were not reliable at all.

This ‘parroting’ of ideas is also evident when you ask the Chatbot to write on issues such as why parliamentary democracy or why a republican system is better. The essays (screenshot attached below) might appear perfect but dig deeper and you can see they lack authoritativeness. The arguments remained the same for both topics.

When asked to write about parliamentary democracy and republican systems, the arguments it chose were the same for both.

As Emily M Bender, Professor in the Linguistics department at the University of Washington points out, the system is simply designed to produce more words based on the words in the prompt. “Making these models bigger and bigger and claiming that the improvements in apparent coherence and fluency are steps towards ‘true AI’ or ‘language understanding’ is like running faster and faster and claiming that we’re getting closer to inventing teleportation,” she told indianexpress.com over email.

According to RAISE’s Shah, while incorporating context into conversations with AI might eventually happen, adding responsibility to these responses will be much harder. “Alexa told a 10-year-old to touch a live plug with a penny as a challenge. A human would have understood that’s not OK to do. A general sense of responsibility is still missing and it’s not clear if and when AI systems can have that,” he wrote.

The larger ethical concerns

There are also concerns regarding biases within the bot’s responses—issues that have plagued models before it, such as Microsoft’s Tay chatbot and Meta’s Galactica. In both cases, the companies ended up withdrawing these from public access because the models went awry. In Tay’s case, it went viral for tweeting abuses, and racist and misogynist comments.

Steven T Piantadosi, Professor, at UC Berkeley, and head of the Computation and Language Lab (Colala), gave examples of how ChatGPT was suffering from many of these biases as part of a Twitter thread. In many of the responses, the bot appeared to assign a higher worth to Caucasian males at the expense of those from other races, though OpenAI’s founder also replied to the thread encouraging users to downvote such responses.

“I think that the biases in these systems result from both their training data sets, as well as the assumptions made by people who construct the training objectives and algorithms. It’s a very difficult problem to solve, but it’s one that absolutely essential to fix before these systems are used in real applications,” Piantadosi told indianexpress.com over email.

According to Professor Bender of the University of Washington, there’s also a danger that such tools can be used by bad actors as well to spread misinformation be it “populating message boards for recruiting extremists, producing fake reviews (positive or negative, on demand), and generally polluting information sources.”

But still, OpenAI’s approach to creating the model is different compared to other companies, as some noted. For instance, Shah pointed out that the company relied on manual annotations and its own employees who acted as AI trainers for the chatbot, though he cautioned that there is no assurance of the quality or neutrality of the responses. Narayanan said while OpenAI has trained the bot to avoid saying inappropriate things, it has a filter, which “often fails.”

There’s also the question of how these models use data and the issue of ‘credit’. After all, unlike Google, ChatGPT does not link back to websites or articles, many of which are used to train it. “With generative models like ChatGPT and DALL-E, this is even more tricky because they are not delivering actual content that exists but making stuff up. Only that this ability to make stuff up is heavily dependent on the stuff that already exists,” Shah noted.