Secret's out on our 3D virtual try-on

We are constantly getting asked about the nuts and bolts of virtual try-on technology. Today, I'll be breaking down what goes on when you create your DITTO, the video of your face from your webcam used to virtually try on glasses to see if they fit.  

There are 3 key processes: 1) creating the 3D face model of your head with accurate scale 2) digitizing the glasses photo-realistically, and 3) rendering the glasses onto your DITTO with proper scale and with matching lighting and shadows so you can see which pairs fit and flatter your unique face. 

Process 1: Creating your DITTO

Step 1: Capture webcam video of your face turning from side to side

We record 4-8 seconds of video from your webcam of you slowly turning from side to side like you are looking into a mirror.  Once your DITTO is ready, our technology works such that you will essentially forward and rewind this video with glasses augmented into the video, so don't forget to smile!  If you make a goofy face when making your DITTO, you'll see that face over and over again when you're trying on products later.  

We are constantly improving the user interface (UI) to make the recording process easy and intuitive, so send us any tips you might have to

Step 2: Detect the facial features in each frame of the video

We use our patent-pending computer vision algorithms to detect ~30 points on your face in each frame of the video you recorded. There are usually about 150 frames per video. This is particularly tricky given all the various camera parameters and noise from the webcam. Lighting can also make this very difficult which is why it helps (a lot!) to have even lighting on your entire face.   

Step 3: Reconstruct the face in 3D 

We use these 30 detected points in the 150 frames to reconstruct your head paying extra attention to your nose and ears, so we can decide where the glasses sit on the face. Behind the scenes, we use these points to reconstruct your face in 3D.  

Step 4: Take a picture holding up any plastic card in your wallet against your forehead

Since plastic cards like library cards, gym membership cards, or the back of your license are all sized exactly the same, we use this standardized object to understand the scale in your video.  

Step 5: Automatically detect the edges of the card to understand the scale of the face

We use those card dimensions and the 3d model of your face described above to calculate the exact dimensions of your face.  It's important to have a 3D model as opposed to just using a 2D picture so we can understand the plane of your eyes relative to your forehead.  This comes into play later when we are using this technology to detect important measurements on your prescription like your pupillary distance and your segment height.

Process 2: Digitizing the Glasses

Step 1: Create a “mesh” of the glasses

We take a series of pictures of each pair of glasses as well as detailed digital caliper measurements of each frame. We use this information to create a mesh for each pair of glasses that looks similar to a CAD file. 

Step 2: Add materials to each mesh

We add detailed, photo-realistic “materials” to each “mesh”. We have a specialized team of designers who make each pair of glasses look exactly like the real thing by overlaying sections of real photography onto the mesh.  We are looking for ways to automate parts of this process (email if you've got any ideas), but to-date we've found that manually tweaking each pair of glasses leads to the most photo realistic results.  We run tests on Facebook and the results have proved that we are getting so good customers can't tell the difference!  We have even decided to use these 3D models to replace our photography. So when you are viewing a pair of glasses on our site in 360, you are looking at the digital representation of them, not a set of pictures.

Step 3: Size the glasses properly relative to the face model

Compare the Glasses Measurements taken in Process 2, Step 1 to the Face Measurement taken in Process 1, Step 5 to size the glasses properly relative the face size.  We have enough data to scale the glasses to your face with millimetric precision.

Process 3: Rendering the Glasses Back onto the DITTO

Step 1: Position the glasses onto the 3D face model and figure out the occlusions

Position the properly scaled glasses onto the 3D face model taking into account the specific nose bridge, face structure and ears of the customer. This shows how the glasses would actually sit on your face in real-life and are tilted at the proper angle.  

We use the word occlusions to mean the parts of the glasses that need to be hidden from certain views. For example, as you turn your head, your face starts to hide sections of the glasses that are now behind it from the frontal view.  Your nose bridge also requires special occlusions since it blocks the bottom of the far lens as you turn.  We figure out which sections need to be hidden at each angle.

Step 2:  Rendering the glasses back onto the video of your face (aka your DITTO)

Rendering is the process of generating a 2-D image from a 3-D model by means of computer programs. Once we understand your 3D scene and your occlusions, we render the glasses back into your 2D video according to those parameters.  As you move your DITTO back and forth, it now appears you are wearing glasses!   

Step 3: Matching the lighting in your scene and rendering reflections and shadows

Once you have a photo realistic pair of glasses scaled and fit to your face, we still have the challenge of making those glasses look photo realistic on your DITTO.  Lighting makes this much harder.  For example, if you are sitting in a dark room and you have a lot of shadows on your face, adding a brightly lit pair of glasses onto your DITTO will glow and look cartoony.  Our engineers have built special tools that detect the light sources in your scene and model your environment in a 3D scene.  They use this scene to change the lighting on the glasses. 

Then we add reflections in the lenses and shadows onto your face to up the ante. It's amazing how much the details matter!  

Step 4: We calculate key measurements for your prescription lenses like pupillary distance (aka PD) and segment height

We can very accurately detect the two most critical measurements needed to create an accurate prescription lenses:  pupillary distance and segment height.  Pupillary distance is the distance between your two pupils.  Segment height is the distance between your pupil and the bottom of the frame of your glasses.  DITTO's try-on technology is the only existing online technology that accurately takes both these measurements.  Other sites make you go back into your doctor to have them measure both your PD and your segment height, but we have tested our tool and found it has an error margin less than 1mm, which is well within the accepted range.  

Step 5: We use your information to manufacture better product.

Not only does adding scale help you pick out glasses that actually fit, we use your DITTO and your measurements to more accurately create your prescription lenses. When our team of highly-trained optical laboratory technicians create your prescription lens, we look at your DITTO to make sure we put the focal center of your lens in the exact same spot your pupil is relative to your eyeglass frames. This can be really important if you have a strong prescription.  Similarly, seeing the frames on your face enables us to measure your segment height, which is where the "no-line" progressive lens will transition from one prescription to another in a bifocal lens.

We hope this description was helpful to understand what goes on beyond the scenes when you virtually try on glasses on DITTO.  Our goal is perfection and we will continue making the tiny tweaks necessary so that you think you're really looking into a mirror. Any feedback and suggestions are greatly appreciated! 

If you'd like to create your DITTO, you can do that here.