Tags:
Journalism of Courage
Two prominent former Google researchers, one of whom was a co-inventor of the “transformer” artificial intelligence architecture that paved the way for the generative AI boom, announced on Thursday they had started a new AI company based in Tokyo.
Sakana AI, founded by David Ha and Llion Jones, declined to disclose funding details.
Jones is the fifth author on Google’s 2017 research paper “Attention Is All You Need”, which introduced the “transformer” deep learning architecture that went on to become the basis for viral chatbot ChatGPT and the current race to develop products powered by generative AI. Ha was previously the head of research at Stability AI and a Google Brain researcher.
After the famous paper came out, advances in generative AI foundation models have centered around making the “transformer”-based models larger and larger. Instead of doing that, Sakana AI will focus on creating new architectures for foundation models, Jones told Reuters.
“Rather than building one huge model that sucks all this data, our approach could be using a large number of smaller models, each with their own unique advantage and smaller data set, and having these models communicate and work with each other to solve a problem,” said Ha, though he clarified this was just an idea.
All the authors on the “Attention Is All You Need” paper have now left Google. The authors’ new ventures have attracted millions in funding from venture investors, including Noam Shazeer, who is running AI chatbot startup Character.AI, and Aidan Gomez, who founded large language model startup Cohere.
