The inside story behind Apple’s Face ID, and all your doubts cleared

Anyone who has used a face authentication system after a pattern unlock or even a fingerprint reader, realises how this is a more intuitive and natural way to let the device know that it is being accessed by its owner. Apple users, on the other hand, have experienced how Face ID works even when the ambient light is low and you are directly not in front of the device. With the latest devices and iOS 13, Face ID has also become faster and has a wider field of view than before.

Apple Face ID can be used to unlock the iPhone, authorise purchase on iTunes and App Store and for authenticating passwords via Apple KeyChain. (Image source: Nandagopal Rajan/Indian Express)

Drance acknowledged that we are in a world with a lot of facial recognition solutions, “but not all of them are created equal”. “With Face ID, we focus on usability, but also security and privacy. One thing that many of our customers have noticed is that Face ID is really fast and dependable. So you can simply glance at your phone and in unlock,” she added.

While unlocking is the most common use case for Face ID, it is also used to authenticate purchases using Apple Pay and to authenticate passwords using Apple’s Keychain, where users have saved their user name and password. Drance said one big feedback from users was they did not need an update when they were already using Touch ID in an app. “Any app that has worked with Touch ID, including banking apps would automatically be able to work with face ID,” she explained.

In India, for instance, financial services apps like iMobile app by ICICI, Paytm money, HDFC Bank, Kotak Bank, Citibank India, Axis Bank, FreeCharge, Cred and PhonePe use it along with travel apps from Ixigo and Cleartrip. A lot of new health and shopping apps are also adding this feature now for authentication.

Apple has over time trained multiple neural networks to see if there are changes every time it looks at the user and then make that adaptive change. (Image source: Nandagopal Rajan/Indian Express)

How does Face ID work?

The TrueDepth camera system looks for the user’s face when you wake the iPhone by raising it or simply tapping the screen.

Also Read: Apple FaceID tips and tricks you did not know about

When a face is detected, Face ID confirms the user’s attention and intent to unlock it by detecting if the eyes are open and directed towards the camera.

What is the technology behind Face ID?

The camera system will map the geometry of the face along with 2D and depth information before the phone unlocks it. This is done only after the attention aware requirement is met. The depth information is an additional level of security as most other facial unlock technologies use only 2D information.

The TrueDepth camera system has an inbuilt dot projector that puts 30,000 invisible dots on the user’s face to build out a unique facial map. This works along with the flood illuminator to identity a face even in the dark. Facial authentication technologies that don’t have this layer don’t work well in the dark and can potentially be spoofed by a photograph as it does not have depth information or a facial map to match against.

Lastly, the infrared camera reads out the dot pattern in the infrared image of the face and sends it along with the depth map to the secure enclave on the chip. Here, this information is converted it to a mathematical representation of the face data that has been used to match against the data from your original enrollment.

How does Face ID understand changes over time?

Drance explained that the original enrollment data of your face has to be able to adapt to natural changes over time. “This is something that was very important to us because we know our lives are fluid, our looks change. So we use advanced machine learning that’s used to adapt to these changes in your appearance. So that means you can be recognised even after you change your haircut, put on glasses or take them off or you gradually grow a beard,” she added.

Apple has over time trained multiple neural networks to see if there are changes every time it looks at the user and then make that adaptive change.  “All of this amazing technology happens seamlessly. But there was a lot of engineering behind getting it to be this seamless and flexible over time,” Drance underlined.

“We trained this network with over 2 billion images including infrared and depth images collected in extensive studies. And of course, they were all done with users informed consent,” Drance said, adding how they ensure that the images came from across the world so that it accounted for gender, for age, ethnicity and other factors that might better represent our customers. And this is an ongoing process with the studies continuing and the neural networks getting better at what they do.

Can Face ID be spoofed?

Drance said Apple has neural networks that are dedicated to protecting against spoofing mechanisms using photos or with masks, even very sophisticated masks. “We worked extensively to provide very realistic masks to train our neural networks to protect against this type of thing. So we really went very far with the security and privacy of your facial data in development of the solution.”

To prevent against someone trying to still hack the system, the phone will lock after five unsuccessful match attempts after which passwords have to be entered for additional security.

But what are the chances of someone looking like you being able to unlock your phone? Drance said this probability was again one in a million. And remember this has to be done within five attempts.

What sort of data does Apple have?

For most of the data that it collects from the user, Apple has a policy of storing the details on the device itself without sending it up to the cloud. With Face ID, Drance said, Apple does not store photos anyway and has just the “mathematical representations of your face”.

“Even that is encrypted and protected by the secure enclave,” she explained, elaborating on the separate section in Apple’s A series chips. “So we are providing not only a software mechanism for additional security through algorithms, but actually a hardware architectural design decision that we made to make this walled-off part of the chip that wasn’t even accessible to the rest of your system on a chip. And that was really important to us for protecting your facial data.” And since this data is on device and never backed up, “Apple doesn’t have access to your facial data in the server”.