Computer Vision
• Face Detector

Due: October 16

Code

1. Detecting and computing symmetries of 3D objects has many applications in shape analysis, shape matching, shape representation, object detection, object recognition, etc. Assume you have been given an oriented point cloud: a dense set of 3D points (x,y,z) and corresponding surface normals (n_x,n_y,n_z), evenly distributed over the surface of some object. Using the Hough transform, develop an algorithm for detecting the planes of symmetry for this object.
   • First, to think about this in 2D, consider the case when there are only two points. The plane of symmetry for these two points is just the plane that is the direction perpendicular to the vector connecting the points as shown here:
   
   
   
   
   • Now consider a 2D shape that has N points. Connect each of the N points with a line, and then construct a line perpendicular to this one. For each possible splitting, we need only to record the angle θ that the plane needs to be rotated by in order to make it perpendicular to the line connecting the points. The position need not be stored as explained later. If we plot along a line each value of θ, we can determine the most likely splitting planes where there is a high concentartion of votes, just as in the case of line detectors.
   • Extending this this to 3D, we can now take a point cloud. If we connect each of the N points in the cloud with a vector, we can use this vector as the normal of a plane. Normally planes need a vetor in the plane and a normal. Here, we don't need the vector since any plane whose normal is the vector connecting any two points effectively splits them. If we parametrize this normal vector with (φ,θ), we can encode all planes with only 2 coordinates. Once again we don't need to specify the position. This is because any plane of symmetry MUST be centered at the center of mass of the point cloud. Therefore, once we have determined the (φ,θ), we can compute the point for the plane as the center of mass of the point cloud. The vector needed to form the actual plane can be arbitrary as long as it is perpendicular to the normal. Therefore we can generate planes of symmetry simply by finding the vectors between every pair of points in the cloud, finding the corresponding (φ,θ) pair, plotting this in an image, and aggregating the results. We can then extract the likely planes of symmetry by selecting points in that image that have a high number of (φ,θ) votes.


2. Imagine you are given two vectors, a "signal" S and a "template" T. Assume T is shorter than S. Now, you want to find the position within S at which T is the best match according to the sum of squared differences (SSD) criterion. That is, given an offset k at which you are looking for T within S, you want to find the k that minimizes:

   

   Show how you can do this without an explicit loop over k by computing two convolutions:
   1. S convolved with some variant of T
      
   2. The vector consisting of the square of each element of S, convolved with a vector the same length as T but consisting of all ones.
   You will need this result for the face detection portion of the assignment below.
   • First let's remind ourselves of what the discrete convolution looks like:

     

   • We can use this as a framework to construct the convolution analog of the SSD algorithm. If we multiply out the terms of the SSD algorithm we are left with:

   

   • The first term in that sum can be written in terms of a convolution as the convolution of the square of S with a vector of ones. This is because, for each k we want the value of, the value of that first term is the sum over the indices of T of the squared elements of S offset by k. If we let F = S^2 and G by the array of ones, then (FxG) = the first term of the sum.
   • The last term in the sum can actually be neglected. This is because it is a constant term that is added to each value of k we wish to compute. If we minimize the SSD of the first two terms, we are still minimizing the total sum because T^2 must be positive. It effectively just offsets the SSD.
   • The middle term is the trickier one. First, notice that FxG = GxF. If we were to let k+i become k-i, then clearly the second term becomes the convolution of T and S multiplied by negative 2. But we just said that we can also write this as the convolution of T indexed by k-i and S by i. Evidently we should come up with a way to make T(k+i) become T(k-i). This is done by a simple reflection about the first dimension in the 1D case, or the two dimensions in the 2D case.
   • So we can compute this total SSD sum by a convolution of S^2 with a vector of ones then subtracted by 2 times the convolution of reflection(T) and S. To really do this efficiently on a computer, the theory of Fourier Analysis says that a convolution of any two vectors A and B is the multiplication of the Fourier Transform of A and B. That is:

   

   • Where a^* is the fourier transform of A and b^* is the fourier transform of B. Using the Fast Fourier Transform algorithm the SSD of the two vectors S and T can be computed extremely efficiently.


3. Suppose you only need to determine the most significant eigenvector (i.e., that whose corresponding eigenvalue has the largest magnitude) of an arbitrary NxN matrix A. Although computing the full SVD of A will certainly do the job, this approach is rather inefficient since it requires computing a complete decomposition of the matrix. This question asks you to devise a more efficient strategy.
   Assume A is not singular and has eigenvalues and corresponding eigenvectors . First, convince yourself that we can write any arbitrary N-element vector as a linear combination of these eigenvectors:

   

   Next, note what happens when we multiply v by A:

   

   Use this result to devise a simple iterative algorithm for approximating the most significant eigenvector of A. Discuss the expected running time of your algorithm as it relates to the eigenvalues of A. There is a situation (property of A) in which you will not be able to identify the eigenvector with the largest eigenvalue using this trick. What is it? What is a real-world application of this algorithm? Lastly, describe how to use this same property to find the second most significant eigenvector, the third, etc.
   • First note what happens when we multiply the vector v by A^N where N is arbitrary. This produces the following sum:

   

   • If we were to let N get arbitrarily large, only the largest eignevalue would survive. This will effectively produce the following equation which can be solved for e_k:

   

   • This of course assumes that the e_n's are all orthonormal. The running time of this requires n^2 multiplies per iteration if A is nxn. If there are eigen values close to the largest eigenvalue, then this may need to run quite a while until the answer converges to a suitable solution. The actual number of eigenvalues doesn't matter as much as the distribution of them.
   • Notice that if A's rows are orthogonal with its column's A^N = 0 for all N != 1.
   • This method can be used for the face decomposition of this assignment. We could just choose to project onto the first eigenvector (eigenface) efficiently using this method.
   • Once we have recovered e_k, we can recover e_k-1 by letting one more term out of the Ω(0) term and solving in a similar manner. This will increase the running time obviously because e_k must already have been computed. This process can be continued up to any i for e_k-i.

• The first step of the face detector is to construct a suitable average face that will serve as our template for comparison. This template will be utilized by computing the SSD between it and a target image. The average face is constructed by taking a series of photographs of test subjects with two poses: happy and neutral. These images must be aligned so that the average face that is computed from them is reliable. The alignment goes as follows:
  
  
  1. Load in 2N images where N is the number of subjects
  2. Have a user select control points (features) in each image
  3. Map these control points to the same points in an output image
  4. Crop each output image to a standard size (200x300 in this case)
  
  
• This will produce a set of 2N images of the same size that have features at the same pixel locations. These can then be averaged to produce the template for comparison. This process is shown below for 3 cases:

Grayscale (2 control points - eyes)	Color (2 control points - eyes)	Color (3 control points - eyes and mouth)

• From initial experiments it was determined that the colored 3-point average face resulted in more accurate face detection in some circumstances. This is reasonable to assume considering we were considering three times as much information to match faces in an input image and the alignment is better. The image below attempts to convey this. Notice that in the color generated image, we have more blue areas around where we expect a face. In these visualizations, the darker the blue, the more likely we have detected a face.

Input Image	Sum-Squared Difference (B&W average face)	Sum-Squared Difference (Color average face)

• Notice that in the first image the color template does a better job of recognizing the entire face and in the second it does a better job of not return false-positives in the case of JD's face, however it turns up more false positives. In general it seems that neither does a clearly better job than the other. The color template was used from here on out.


• The actual face detection algorithm uses this average face as a template to compute the SSD between it and an input image. This is performed efficiently using the convolution strategy answered above. Initially Matlab's conv2 method was used, but it apparently is implemented as a sum over areas, and so to improve efficiency, an FFT approach was used. This reduced the SSD running time from about 2500s to 9s in the larger images.


• When an input image is fed to the detector, the detector computes the SSD for the entire image. The output of this can be seen below in the first column. The next step is to perform non-maximum suppression to only retain local minima. This is done by sorting the list of possible face pixels as taken from the SSD image above some threshold. This list is then visited in descending order. At each possible face pixel, the neighborhood is considered. If there are any pixels in the neighborhood above the current face pixel, they are set to zero and the neighborhood is flag as visited. In this way, we can visit all local neighborhoods and only retain local minima in an efficient way. The final step is to draw boxes at the face pixel locations.


• Here are the results of the detector:

SSD Image Output	Face Detector Output



• In the first two cases pictures, the face detector performed perfectly, accurately matching JD and Dan's faces with no false positives, however in the subsequent images there were some imperfections. In the third image, Colleen's face is accurately identified, but there are a number of false positives whose SSD were actually smaller than the correct choice. In most of the false-positives some type of beige coloring was exhibited. This is due to the fact that the average face was close to a beige color with slight color variations.


• Two other pictures of note are the third to last, and the last ones. The third to last is a picture from my own personal pictures. No ones face in that image was included in the average face yet the detector was able to pick out my girfriend and my face to a reasonable degree. The last images were a complete failure. The likely reason for this is the scale issue. Dan's face is too small to be picked up by the detector and Colleen is skewed too much for the faces to accuratey line-up. To attempt to correct this, the template was made to shrink to 100%, 75%, 50%, and 25% its original sixe, but this failed to in the last picture.


• In general the detector worked well. If it came to a comparison between other detectors, it would make sense to use these same test images and have a human annotate them as well. The discrepancy between this detector and a human detector could be measured against another method's detector and the same human detector. This is probably the best measure since there is little quantitative data that can be extracted and used as a measure of success.


• Although the alignment control points used in this method were hand-selected, an algorithm could be implemented to perform this automatically. In almost all of the input face images there was a large amount of "red-eye". Using this fact, an algorithm could be developed that looks for these reddish regions with a bright spot in the center of them around the middle of the image for the two eye points. If other points are desired, a corner detector could find the corners in all images and then find corners that lie in the same region across images. This last attempt might prove less fruitful, but certainly if the red-eye method were used, there would be at least two strong control points.

• If we inroll each aligned face image into a column of the matrix A, and then compute its SVD, we can recover the basis of the face-space. The vectors that form this basis can be thought of as eigenfaces. Below is a plot of the singular values (eigenvalues) of the SVD of the cropped and aligned images used above.



• Below is a visualization of the first five eigenfaces of the aligned and cropped images:



• The eigenfaces encode a sense of how the subjects faces in the source images are broken down. They encode in them the type of features that have the most variance across the subjects. The first one, for example, encodes how a person smiles and whether or not he/she wears glasses. The second one seems to encode how Jason-like somone is with JD's hair. The third one seems dominated by Dan. Apparently if you have a large eigenvalue corresponding to eigenface three, you are very Dan.


• Now, we can project any face back onto this set of eigenfaces. The projection of each input face's projection onto the first two of these is shown in the scatter plot below where the 'x' axis is the projection onto the first principle component and the 'y' axis is for the second:



• Next we can use these eigenfaces to more efficiently store faces. Take this input face for instance:



• Not the prettiest subject, but nevertheless, here are the projections of this face onto the first N eigenfaces where N increases from left to right from 1 to 8:



• After 7 projections, the image converges to the same thing as ground truth. Here is another example:



• Despite the rather scary firs ttwo images, after about the sixth image there is little discernable difference (after 3 it's even pretty good!).


• Settling on 7 as the number onto which to project, here are pictures of the error associated with the method with respect to ground truth:



• Notice there is very little error associated with projecting down to 7 basis vectors. Using this as a compression scheme, only 7 images at a size of 201x301 each need to be stored, along with 7 numbers for each of the eight images. If we were to store all 8 201x301 images, this would be 201 * 301 * 8 = 484008 bytes. Storing only seven images at the same size, and 56 numbers for the coefficients to be multiplied by each of the seven images requires 201 * 301 * 7 + 7 * 8 = 423563 bytes. That's a compression ratio of 1.14:1. For an arbitray set of N images each of which is (x,y) in size, if they can be accurately represented by Z of the N total eigenfaces, the ratio of compression is:




---

© 2007 Sean M. Arietta
University of Virginia