Google Gemini is no match for GPT-4: Fake demo video, shaky MMLU benchmark draw backlash

During its launch, Google enumerated how Gemini models scored or even outperformed GPT-4 in various benchmarks. This also included Massive Multitask Language Understanding (MMLU), a fundamental metric that is used to assess the AI model’s potential across a broad range of academic disciplines such as STEM, social science, math, and humanities.

However, the research paper shared by Google showed that the Ultra version surpassed both GPT-4 and GPT-3.5. Interestingly, a closer inspection would reveal a key technical detail. Based on Reddy’s tweet, Google has deployed COT@32 in place of 5-shot learning to augment the performance of Gemini. 

Digging deeper into the MMLU Gemini Beat – Gemini doesn't really Beat GPT-4 On This Key Benchmark.

The Gemini MMLU beat is specifically at CoT@32. GPT-4 still beats Gemini for the standard 5-shot – 86.4% vs. 83.7%

5-shot is the standard way to evaluate this benchmark. You… pic.twitter.com/2OIzF8tL1a

— Bindu Reddy (@bindureddy) December 6, 2023

“The Gemini MMLU beat is specifically at CoT@32. GPT-4 still beats Gemini for the standard 5-shot – 86.4% vs. 83.7 per cent” Reddy wrote in her tweet. According to the CEO, 5-shot is the standard measure to evaluate this benchmark, and one prepends five examples in the prompt.

CoT stands for Chain of Thought prompting which involves offering a series of steps like a chain of thought to help it generate rationale to solve problems. CoT is aimed at improving the model’s multi-step reasoning capabilities. Meanwhile, 5-shot learning is when an AI model is trained using five examples of each class. This limited set of examples is training data from which an AI model is expected to recognise patterns.  

Reddy, in her tweet, claimed that Google invented a different methodology around CoT@32 to claim that it is way better than GPT-4. The former AWS/Google staff said that Cot@32 only beats when you add in for “uncertainty routing”. Below are some more tweets from users who have flagged similar concerns with benchmarking.

Google is trying to fool us with Gemini Ultra’s numbers.

The benchmark they’re most proud of is Gemini’s score of 90.0% on the MMLU.

This beats GPT-4’s score of 86.4%, and it even tops human test results of 89.8%.

But they’re not comparing the models using the same prompting… pic.twitter.com/oAOWDtkz5U

— Igor Pogany (@TheAIAdvantage) December 8, 2023

Gemini MMLU results

TL;DR: Gemini does not really outperform GPT4 on MMLU, and it will likely show significantly in the product. However, this does not necessarily impact the other metrics, like HumanEval for code etc.

There's been a lot of discussions around the coherence… pic.twitter.com/yZj3yRT3y6

— Hadi Azzouni (@hadiazouni) December 8, 2023

The Gemini video isn’t all that real

While unveiling Gemini, Google demonstrated a video of its multimodal and reasoning capabilities. Ever since the launch, there have been numerous reports that the video has not been relayed in real-time. Clint Ehrilch, who is an attorney and computer scientist, as per his X bio, shared a detailed tweet where he claimed that the video demo of Gemini was fake. 

According to Ehrlich, three things about the video were exciting to the viewers such as that Gemini processed video and not just still images, it conferred context without being spoon-fed prompts, and it seamlessly spoke and understood conversational audio. Ehrlich said that none of these three facets were real. 

Yesterday, Google shocked the world with its new AI, "Gemini."

But it turns out the video was fake: the A.I. *cannot* do what Google showed.

It's my opinion, as a lawyer and computer scientist, that (1) Google lied and (2) it broke the law. 🧵 pic.twitter.com/VDfvcwfQeO

— Clint Ehrlich (@ClintEhrlich) December 7, 2023

Ehrlich in his thread said that Gemini did not process video instead it processed images, it required detailed prompting, and that it communicates best with written prompts and not audio. “You won’t get any of this from the viral video, but it’s spelled out in Google’s documentation for developers,” Ehrlich said. He went on to debunk sleight-of-hand tricks performed with a coin, the geography quiz, ball-and-cup shuffling game shown in the video.

In his post, Ehrlich asked if Google broke the law by showing a fake video. He went on to say that under Federal Trade Commission (FTC) standards, a disclaimer is necessary to prevent an ad from being misleading. “Under FTC standards, if a disclaimer is necessary to prevent an ad from being misleading, it must appear *in the ad.* A separate blog post doesn’t cut it,” he said in his tweet. 

Claims of deceptions

According to a report in Bloomberg Opinion, the output of Gemini is much slower compared to the demo. It said that although there was a disclaimer in the video saying that the responses had been sped up, however, that was not the biggest deception. According to the report, Gemini wasn’t even watching the video and all the responses heard in the video were its replies to still frames from the video and text prompts. 

Following the incident, a Google spokesperson told Bloomberg Opinion that the video was made ‘using still image frames from the footage, and prompting via text’. Oriol Vinyals, who co-leads Gemini, took to X to claim that the video was intended to ‘inspire developers”. 

“All the user prompts and outputs in the video are real, shortened for brevity. The video illustrates what the multimodal user experiences built with Gemini could look like. We made it to inspire developers.” he said in his tweet. 

Really happy to see the interest around our “Hands-on with Gemini” video. In our developer blog yesterday, we broke down how Gemini was used to create it. https://t.co/50gjMkaVc0

We gave Gemini sequences of different modalities — image and text in this case — and had it respond… pic.twitter.com/Beba5M5dHP

— Oriol Vinyals (@OriolVinyalsML) December 7, 2023

From a consumer perspective, the true assessment of Google Gemini will happen when the model becomes accessible on a wider scale. While Gemini AI have outperformed other AI models on many benchmarks, it’s important to acknowledge that its creators have asserted from the start that it is not perfect and that it is still evolving.