Tuesday, Sep 27, 2022

Explained: ‘Virtual human’ NEONs — how do they work

India-born scientist Pranav Mistry's Star Labs showcased the concept at CES last week. NEON's creators believe the virtual humans can ultimately be fully autonomous 'friends, collaborators, and companions.'

NEON, Pranav Mistry, Virtual Humans, CES, Pranav Mistry Neon. Consumer Electronics Show, Consumer Electronics Show Las Vegas, CES 2020, CES 2020 Las Vegas, Star Labs head Pranav Mistry at CES 2020 in Las Vegas last week. (Express Photo by Nandagopal Rajan)

Among the most-discussed new concepts at the annual Consumer Electronics Show (CES) in Las Vegas this year was NEON. The first project of Samsung’s Star Labs, NEONs are being called the world’s first artificial humans. They look and behave like real humans, and could one day develop memories and emotions — though from behind a 4K display. It won’t be wrong to call them life-size human avatars, or maybe a human interface for whatever you want to do with technology.

Star Labs is headed by India-born scientist Pranav Mistry who underlines that what was showcased at CES was the product of just four months’ work. Mistry, who is president and CEO of Star Labs, was earlier behind a lot of products released by Samsung, including the Galaxy Gear smartwatches. Star Labs is Samsung Technology & Advanced Research Labs (not to be confused with the fictional labs of DC Comics), an independent project funded by the Korean tech giant.

So what are NEONs?

The company says NEONs are computationally created virtual humans — the word derives from NEO (new) + humaN. For now the virtual humans can show emotions when manually controlled by their creators. But the idea is for NEONs to become intelligent enough to be fully autonomous, showing emotions, learning skills, creating memories, and being intelligent on their own. Star Labs thinks they can be “friends, collaborators, and companions”, but all that is a few years away. At CES, the idea was to showcase the concept to the world.

How do the NEONs work?

Mistry started work on NEONs by trying to replicate a friend. Initially, the models were trained on his face, and there were significant errors. But then, they started getting better, almost indistinguishable from the original.

Subscriber Only Stories
Kurmi club: On national path, Nitish looks east, at UP’s Sonelal Pa...Premium
Interview: MD-CEO, Central bank of India | ‘During PCA years, none of our...Premium
The Kurmis: a political historyPremium
Deepti Sharma and the question of law vs spirit in cricketPremium

Mistry says there are two core technologies behind his virtual humans. First, there is the proprietary CORE R3 technology that drives the “reality, real time and responsiveness” behind NEONs. The company claims CORE R3 “leapfrogs in the domains of Behavioral Neural Networks, Evolutionary Generative Intelligence and Computational Reality”, and is “extensively trained” on how humans look, behave and interact.

But in the end, it is like a rendition engine, converting the mathematical models to look like actual humans. At the moment the latency for response is less than a few milliseconds. CORE R3 can also connect to other domain-specific and value-added services like language kits.

ces 2020, pranav mistry ces 2020, neon life, Consumer Electronics Show (CES), NEONs, NEONs ces, Star Labs samsung, ces news, latest news, tech news, indian express Star Labs CEO Pranav Mistry with his parents in Las Vegas before his presentation. (Express photo/Nandagopal Rajan)

The next stage will be SPECTRA, which will complement CORE R3 with the “spectrum of intelligence, learning, emotions and memory”. But SPECTRA is still in development, and is not expected before NEONWORLD 2020 later this year.


“CORE R3 is the front-end reality engine that is able to give you that real expression. My NEON, right now, does not know when to smile. When you come tomorrow and talk to a NEON, they don’t know that you were here yesterday,” Mistry explained. The spectrum of emotions and knowledge will come only when the NEONs are “actually in the field”; Mistry says this is the layer SPECTRA will enable. Mistry thinks with initial focus on B2B scenarios, he will get more time to work on tech that helps solve for complex memory.

How could NEONs be used?

Mistry sees a world in which NEONs are the interface for technologies and services. They will answer your queries at a bank, welcome you at a restaurant, or read out the breaking news on television at an unearthly hour. Mistry says this form of virtual assistance would be more effective, for example, while teaching languages, as NEONs will be capable of understanding and sympathising.

However, Mistry is clear that a physical form for his NEONs is not possible in the near future. “I don’t think that we are anywhere close to having the physical embodiment of the NEONs in the next 25 or 30 years.” Also, he does not want to enable NEONs on existing robots — but would not mind collaborating with companies like Google, Facebook, and Baidu that have done work in similar fields.

ces 2020, pranav mistry ces 2020, neon life, Consumer Electronics Show (CES), NEONs, NEONs ces, Star Labs samsung, ces news, latest news, tech news, indian express NEONs are powered by a proprietory technology called CORE R3 — “reality, real time, and responsiveness”

How are NEONs different from Virtual Assistants?

Virtual Assistants now learn from all the data they are plugged into. NEONs will be limited to what they know and learn. Their leaning could potentially be limited to the person they are catering to, and maybe her friends — but not the entire Internet. They will not be an interface for you to request a song, rather they will be a friend to speak to and share experiences with, says Star Labs.

Unlike deep fakes, Star Labs says, CORE R3 does not manipulate any scene, videos, or sequence, and instead creates unique behaviours and interactions in real time. “CORE R3 creates new realities,” it says.

What about personal data with NEON?

Mistry does not want his NEONs to have collective memory, or to share data among themselves. So what is known to one NEON cannot be useful for another. “My network is a small network that can live independently,” Mistry said, underlining the difference between his plan and what has been seen so far among other Internet-based companies. Also, Star Labs says “no one except you and your NEON can ever have access to your interactions”, and that private data will never be shared without your permission.

So, what next for NEON?

Star Labs is talking of launching a beta later this year, and claims it is flooded with queries from potential partners. You could find NEONs trying to chat you up on airport screens or bank lobbies soon. Mistry says he does not envision a physical for his NEONs, and anyway that technology is not in the realm of the possible in his lifetime.

First published on: 14-01-2020 at 03:14:52 am
Next Story

France says closely following situation in Kashmir

Latest Comment
Post Comment
Read Comments