None of the big AI models are transparent finds Stanford study

The highest scores ranged from 47 to 54, which can be described as average at best and the lowest score bottomed out at 12. “This is a pretty clear indication of how these companies compare to their competitors, and we hope will motivate them to improve their transparency,” said Rishi Bommasani, Society Lead at the Center for Research on Foundation Models (CRFM), within Stanford HAI, in a press statement.

A lack of transparency is not a new phenomenon in the digital technology industry. From deceptive ads to unclear wage practices in aggregator apps to opaque content moderation systems on social media platforms, transparency issues have remained a mainstay of the silicon valley.

“As AI technologies rapidly evolve and are rapidly adopted across industries, it is particularly important for journalists and scientists to understand their designs, and in particular the raw ingredients, or data, that powers them,” said Shayne Longpre, a PhD candidate at MIT, in a press statement.

Transparent is also important for policymakers to make better decisions. Foundation models can raise significant questions surrounding energy use, labour practises, intellectual property, bias and more. “If you don’t have transparency, regulators can’t even pose the right questions, let alone take action in these areas,” added Bommasani.

Among the top 10 companies building foundational models, Meta scored the highest for its Llama 2 model with 54. BigScience, the company behind BLOOMZ came at second with 53 with OpenAI and ChatGPT at third with 48. But none of these are good marks according to the researchers.

Bommasani asserts that this does not mean Meta should be seen as a goalpost for other companies to move towards. Rather, everyone should be trying to get to 80, 90, or possibly 100.