Assignment 3: Optical Flow

Overview

In this assignment you will implement, test, and analyze a differential algorithm for computing optical flow. A major component of this assignment is related to visualization. The ability to create compelling and informative visualizations is an essential skill for computer scientists and engineers. In this assignment, effective visualizations will help you understand and evalauate the performance and accuracy of your solution.

1. Questions (30 points)

1. Generalize the line fitting technique covered in class, which allows fitting a single line to a set of points, to allow fitting multiple lines to a set of points. Your solution should use the Expectation-Maximization (EM) framework and answer the following questions:
   • What is the generative process you assume underlies the distribution of the set of points?
   • What are the parameters of your model that you aim to estimate?
   • What is the hidden or missing data in this problem?
   • What is the E-step?
   • What is the M-step?
   • How do you initialize your algorithm?
   • How would you evaluate your solution?

2. Imagine that you are the lead engineer in the Image Search Group at Google. Outline an image retrieval system that takes an image as input and returns a rank-ordered list of images from the Internet that are most similar. Your solution should make use of the image segmentation and texture analysis methods we covered in class. Clearly specify the way you define distance between two images (this must be a scalar value).

2. Colorspaces and false-color visualizations (10 points)

For this part of the assignment you need to devise a method for mapping a scalar value p that lies in the range [0...1] to a unique color. This helper function will be used in future parts of the assignment to visualize scalar-valued functions (e.g., error images, stability measures, etc.). The HSV colorspace is well suited for this task. One idea is to map p to a unique hue value and either set saturation and value to 1 or map them to functions of p as well. Play around with different choices, be creative!! You may use images from previous assignments or things you find on-line (topographic images are good for this) to test your function by first converting it to grayscale (i.e., each pixel will have a value in the range [0,1]) and then generating a unique color at each pixel using your mapping function. Be sure that your final "false color" image is in RGB format and displayable within MATLAB (consult help hsv2rgb). Include some false color images obtained with your mapping function in your write-up and explain what you did.

3. Optical Flow (60 points)

Your next task is to implement the differential algorithm for computing optical flow that we covered in class. Recall that there are two main underlying assumptions: object brightness is constant and motion is constant and purely translational within small image regions (details can be found in the class notes).

Implement the following (30 points):

• Read in each frame of a video sequence in turn (see datasets below). You can read frame number i using a call such as imread(sprintf('%08d.jpg',i));.
• Compute the spatial gradient of each video frame just like you did in Assignment 1, by convolving it with the partial derivatives of a Gaussian in the x- and y-dimensions. (You may find that using the FFT is necessary to speed up the processing of videos.) For the temporal gradient, dI/dt (consult slide #16 in the relevant lecture notes), use simple finite differencing: the rate of change of the intensity at one pixel as a function of time is simply the difference between its value at time t+1 (the next frame) and its value at time t (the current frame). Note that this definition won't let you compute dI/dt at the last frame. That is OK, just ignore those frames.
• At each pixel and for an NxN window, assemble the "A" matrix from the spatial components of the gradient and the "b" vector from the temporal component.
• Solve the resulting linear system to obtain an estimate of the motion at each pixel. Again, the details are all spelled out in the lecture notes.

By superimposing colored line segments over images and using your color mapping function from part one, create the following visualizations (20 points):

• Visualize the optical flow at several frames in each video sequence. You needn't draw the entire dense motion field (i.e., play around with what density of motion vectors clearly shows the trend in the field without resulting in too much visual clutter). The plot command can be used to draw simple line segments (on top of images displayed with the imshow command) and fancier arrows can be made with this .m file. You are encouraged to search the web for inspiration and more inspiration.
• Visualize the error in your optical flow calculation. There are no hard rules on how this visualization should be formed, but it should be proportional to the residual error in the motion vector you estimated at each pixel.
• Visualize the stability in your optical flow calculation as something proportional to the second largest eigenvalue in the C matrix (similar in spirit to what we did in the first assignment). Again, there are no hard rules, but your result should clearly show which areas provide more/less stable motion flow estimates.
• Visualize the optical flow across at least one entire video sequence. Be creative!!

Next, choose one (or two) of the above visualizations and a single pair of images to analyze the impact of adjusting the image neighborhood size N. Based on these visualizations, answer the following questions (10 points):

• Why did you choose this/these visualization(s)?
• What happens when you increase/decrease N?
• What elements in the scene cause this?

Submitting

This assignment is due Monday, March 28, 2011 at 11:55 PM. Please see the general notes on submitting your assignments, as well as the late policy and the collaboration policy.

Please submit as one single ZIP file:

• Your false-color mapping function and optical flow code (as one or more .m files).
• The various visualizations.
• A WRITEUP.html file that contains all of the images and visualizations you produce, along with a description of your experiments with different parameters, and any relevant implementation notes. This file should also include your answers to the various questions posed above. This is extremely important. Submissions that do not include a WRITEUP.html file will receive a grade of ZERO.

Note that programming in Matlab is not an excuse to write unreadable code. You are expected to use good programming style, including meaningful variable names, a comment or three describing what the code is doing, etc. Also, all images you submit should have been saved with the "imwrite" function - do not submit screen captures of the image window.

Data

• book.zip
• hand.zip
• mug.zip
• lawn.zip
• traffic.zip
• bottles.zip
• flowers.zip