The bird-eye view of the process of facial recognition is as follows:
- Face detection and face representation : Find the face(s) in the photo, and create a representation (also known as face signature).
- Model training : convert several examples of a person's face into a single model)
- Recognition : compare a face representation of a new photo to the models of a list of candidate identities.
Face detection
Face detection involves passing a sliding window of various sizes across the image, each time trying to answer the question "is there a face at this size at this position". This is an expensive operation, and is typically dominating the CPU cost of facial recognition systems. This step may involve an initial quick filtering stage, that will result in a fairly large number of candidate faces, and a following step that will run a more expensive test on each candidate face to confirm it's a face. Face detection is considered a solved problem, there are very good face detectors available. Once a face is detected, we need to extract a face signature. The best facial recognition systems are aligning the faces before extracting the face signature as it makes the face signature much better. Aligning is the process of rotating and scaling the face in order to bring all of the key features into the same places (eyes, nose, mouth etc). This is important because it allows the the function generating the face signature to work on much cleaner input, and makes recognition perform much better. Representing the face is probably the most important step in facial recognition. A good representation will allow us to distinguish between faces is different people even if they are similar. Typically the face is represented as a vector of floating point numbers, where the meaning of each number depends on the representation itself. In reality - classic facial recognition systems does nothing like this instead of trying to engineer the features, they run a low level descriptor such as LBP, which encodes the gradient direction around each pixel. There are other descriptors, but LBP have been one of the more successful ones for a while. Other techniques involved using several such descriptors and combining them into one big representation. About a year ago, with the publication of the Deep Face paper it became clear that the best representation can be generated using a neural network.
Model training
Simply put, model training is the process of taking several face representation belonging to the same person, and creating a classifier that can recognize more examples of face representation from the same person. The most popular approach is to use a linear SVM.
Recognition:
Given a new face representation and a list of candidate identities, recognition will 'compare' the face models of the identities with the new face representation and will generate a score for each one. The score if passing a certain threshold is an indication of a recognition.
This is very fast operation, and depending on the size of the representation - modern systems can obtain scores for millions of candidates per second.
How good the scores are for recognition depends on the quality of all the preceding steps, and especially on the quality of the representation function.
