Tags:
Journalism of Courage
Since its launch last year, OpenAI’s ChatGPT has changed the AI landscape with human-like question-and-answer sessions with a chatbot. In fact, some industry insiders have predicted that ChatGPT is on pace to challenge Google’s grip on the search engine market. However, there’s a growing concern among many users about ChatGPT getting extremely slow and far less capable of providing accurate answers, raising doubts as the generative AI explosion continues to gain mainstream popularity.
Researchers at Stanford and UC Berkeley found that the performance of OpenAI’s LLMs has significantly decreased over time, thus making ChatGPT less accurate. The team of researchers analysed different versions of ChatGPT from March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on four primary tasks. They developed benchmarks to evaluate the model’s competency in solving math problems, answering sensitive questions, generating code and assessing the models on visual reasoning.
The tests revealed a significant dip in performance between versions over time. In March ChatGPT-4 was capable of identifying prime numbers with a 97.6 per cent accuracy rate. However, in June, ChatGPT only managed to get 12 questions right, plunging to 2.4 per cent accuracy.
https://platform.twitter.com/widgets.js
When it comes to generating a few lines of code, ChatGPT once again performed poorly. The study also found ChatGPT’s newer version refused to answer certain sensitive questions
“GPT-4 is getting worse over time, not better,” said AI expert Santiago Valderrama on Twitter.
“Rumors suggest they are using several smaller and specialized GPT-4 models that act similarly to a large model but are less expensive to run,” he said. Because of that, some of ChatGPT’s performance has deteriorated over the last few months.
The researchers recommended those building applications on GPT 4 should be aware that the behaviour of LLM changing could be problematic.
https://platform.twitter.com/widgets.js
Over the past few months, many users have taken to social media and forums to complain that GPT-4’s accuracy and performance are not at par with GPT 3.5, the LLM that power applications like ChatGPT. One theory is that the cost of operating the systems is so high that companies like OpenAI aren’t deploying the best versions of chatbots to the public. But the industry shows no sign of a slowdown despite pressure from a host of AI luminaries who are in favour of slowing down AI development.
After OpenAI released the ChatGPT AI chatbot, Microsoft put billions of dollars into Sam Altman’s company and released the GPT-powered Bing browser. Google’s Bard followed in quick succession. Right now, both Microsoft and Google are quickly adding AI-powered tools to their core products.
While the generative AI boom is here to stay, app downloads for ChatGPT and Microsoft Bing have slowed in recent weeks. According to a report by the Bank of America, citing Sensor Tower data, ChatGPT downloads on iPhones in the U.S. were down 38 per cent month over month in June. Bing app downloads, which includes a ChatGPT-based chatbot in the US, were also down 38 per cent in June. In the same month, traffic to the chatbot’s website globally fell 9.7 per cent from the previous month, and unique visitors dropped by 5.7 per cent, according to web analytics firm Similarweb.
