[Expert Talks] Machine Learning role in the Frame Removal experience
23 February 2023 • 2 min read
With our "Expert Talks" series, we give the floor to Fittingbox's talents!
They'll share exclusive insights, the challenges they face and talk about their domain of expertise. From Augmented Reality (AR) & Diminished Reality (DR) to Machine Learning and Computer Vision, get glimpses of what's behind Fittingbox's innovations!
We give the floor to Xavier Naturel, Machine Learning Researcher at Fittingbox. He has significantly contributed to the technological research for the design of our innovation: Frame Removal. What role does Machine Learning play in this digital experience? Xavier explains it to us.
About Xavier Naturel, Machine Learning Researcher at Fittingbox
After joining Fittingbox in 2019 as a Machine Learning Researcher, Xavier and his team have been working on several Machine Learning methods that detect landmarks on faces, and image segmentation. This research is the basis of the functioning of the Diminished Reality experience "Frame Removal", which, unlike Augmented Reality, virtually removes an element from the real world.
Frame Removal, a truly immersive experience!
Launched in 2021, Frame Removal is a world first. It is an exclusive add-on of the Fittingbox Virtual Try-On, which makes life easier for glasses wearers. As soon as the user launches a Virtual Try-On session, this innovative technology detects in real time the marks corresponding to the physical pairs of glasses worn on users’ faces, to erase them on the screen and replace it with the 3D digitized frame, previously chosen by the users in question. The goal: to be able to see clearly when choosing the next pair of glasses, and to be able to keep the corrective lenses on your nose while trying new frames on.
To ensure that Frame Removal works, Xavier and his team exploit Artificial Intelligence and its sub-fields, in particular Machine Learning and Deep Learning. The latter is essential for the detection of each pixel belonging to the glasses worn by the user. This detection process is better known as “semantic segmentation”.
In short, each pixel is detected by a Deep Learning algorithm (the DeepLabv3+) and then classified into 3 categories: background, lenses and frame. Once the pixels are labelled in one of these categories, the fitting stage can be carried out as if the user was not wearing glasses.
In a nutshell, how does Machine Learning play a fundamental role in how Frame Removal works?
Machine Learning, as explained at the beginning of this article, is a sub-field of Artificial Intelligence. It is an algorithm that learns to interpret images content from a set of known images.
The data set from which the Frame Removal algorithm is taken, draws information from our Virtual Try-On, a semi-synthetic data set. Fittingbox Virtual Try-On technology is used to generate digitized glasses on a real face, as well as masks, landmarks and visibility information, the so-called “ground truth.”
It is precisely by having learned to discern and analyse these 3 elements - background, lenses and frame - separately, that our Frame Removal is able to recognise the physical frame when the user starts a Virtual Try-On experience with real glasses on the face. At that point, the deep neural network provides the location of the glasses so that they can be erased and replaced by the 3D digitised pair to be tried on.