What is SIMA, Google Deepmind’s new AI that can play video games with you

With an AI robot to help seniors beat loneliness and a fully autonomous AI software engineer released recently, the week has been a busy one for AI enthusiasts. Here’s what you need to know about SIMA.

What is SIMA?

AI research lab Google Deepmind describes SIMA as an AI Agent, which is different from AI models such as OpenAI’s ChatGPT or Google Gemini. AI models are trained on a vast data set and are limited when it comes to working on their own. On the other hand, an AI Agent can process data and take action themselves.

SIMA can be called a generalist AI Agent that is capable of doing different kinds of tasks. It is like a virtual buddy who can understand and follow instructions in all sorts of virtual environments – from exploring mysterious dungeons to building lavish castles. It can accomplish tasks or solve challenges assigned to it.

It is essentially a super-smart computer programme that can be thought of as a digital explorer, having the ability to understand what you want and help create it in the virtual world.

How does SIMA work?

SIMA “understands” your commands as it has been trained to process human language. So when you ask it to build a castle or find the treasure chest, it understands exactly what these commands mean.

One distinct feature of this AI Agent is that it is capable of learning and adapting. SIMA does this through the interactions it has with the user. The more you interact with SIMA, the smarter it gets by learning from its experiences and improves over time. This makes it better at understanding and fulfilling user requests.

Based on the current stage of AI development, it is a big feat for an AI system to be able to play even one game. However, SIMA goes beyond that and can follow instructions in a variety of game settings. This could potentially introduce more helpful AI agents for other environments.

The latest research from Google DeepMind shows that it is possible to translate the capabilities of advanced AI models into “useful, real-world actions through a language interface.” With this, Google is hoping that SIMA and other AI agents will be able to use video games as sandboxes to understand how AI systems can be more helpful.

How was SIMA trained?

On the company’s official blog, its description states: “We partnered with game developers to train SIMA on a variety of video games. This research marks the first time an agent has demonstrated it can understand a broad range of gaming worlds and follow natural-language instructions to carry out tasks within them, as a human might.”

To expose the AI agent to different environments, Google Deepmind collaborated with eight game studios to test SIMA on nine different video games, including Teardown by Tuxedo Labs and No Man’s Sky by Hello Games.

With every game, SIMA’s portfolio opened a new interactive world along with a range of skills for it to learn, such as simple navigation, menu use, mining resources, flying spaceships, etc.

They also used four research environments, including one built with the cross-platform game engine Unity. This environment, called Construction Lab, required AI agents to build sculptures from building blocks. This exercise was done to test the agent’s object manipulation abilities and intuitive understanding of the physical world.

According to the researchers, by learning from different gaming worlds SIMA comprehends how language ties in with game-play behaviour. “Our first approach was to record pairs of human players across the games in our portfolio, with one player watching and instructing the other. We also had players play freely, then rewatch what they did and record instructions that would have led to their game actions,” read the official blog.