Assignment 3: Optical Flow and Image Warping

Overview

In this assignment you will implement, test, and analyze a standard algorithm for computing optical flow between images. A major component of this assignment is related to visualization. The ability to create compelling and informative visualizations is an essential skill for all computer scientists and engineers. In this assignment, effective visualizations will help you understand and evalauate the performance of your solution.

1. Colorspaces and false-color visualizations (10 points)

For this part of the assignment you need to devise a method for mapping a scalar value p that lies in the range [0...1] to a unique color. This helper function will be used repeatedly in future parts of the assignment to visualize scalar-valued functions (e.g., error images, stability measures, etc.). The HSV colorspace is well suited for this task. One idea is to map p to a unique hue value and either set saturation and value to 1 or map them to functions of p as well. Play around with different choices, be creative!! You may use images from previous assignments or things you find on-line (topographic images are good for this) to test your function by first converting it to grayscale (i.e., each pixel will have a value in the range [0,1]) and then generating a unique color at each pixel using your mapping function. Be sure that your final "false color" image is in RGB format and displayable within MATLAB (consult help hsv2rgb). Include some false color images obtained with your mapping function in your write-up and explain what you did.

2. Optical Flow (55 points)

Your next task is to implement the "differential" algorithm for computing optical flow we covered in class. Recall that there are two main underlying assumptions: the image brightness is constant and that motion is constant and purely translational within small image regions (details can be found in the class notes):

• Read in each frame of a video sequence in turn (see datasets below). You can read frame number i using a call such as imread(sprintf('%08d.jpg',i));. (5 points)
• Compute the spatial gradient of each image by convolving it with the partial derivatives of a 2D Gaussian just like you did in the first assignment. To make things fast you may also want to use the FFT to perform this convolution like in the second assignment. (5 points)
• Approximate the partial derivative with respect to time at each frame by computing the difference between the intensity at each pixel and its intensity in the following frame (note that you will not have time derivative information in the last frame). (5 points)
• Assemble the "A" matrix discussed in class from the spatial gradients within NxN neighborhoods and the "b" vector by collecting the corresponding time derivatives. (5 points)
• Solve the resulting linear system using MATLAB's left matrix divide '\' (consult help mldivide) or the pinv function for computing the pseudoinverse (based on SVD as discussed in class). Solving this linear system at each pixel will result in a dense motion field (i.e., a 2-D displacement vector at each pixel). (10 points)

• Visualize the optical flow at a few frames in each video sequence. You needn't draw the entire dense motion field (i.e., play around with what density of motion vectors clearly shows the trend in the field without leaving too much visual clutter). The plot command can be used to draw simple line segments (on top of images displayed with the imshow command) and fancier arrows can be made with this .m file. You are encouraged to search the web for inspiration and more inspiration. (5 points)
• Visualize the error in your optical flow calculation as something proportional to the residual error in the overconstrained linear system you end up solving at each pixel. There are no hard rules on how this visualization should be formed, but you should try to use the function you created in part one. Examples include showing a false-color visualization superimposed over the original image, or some modification to the previous visualization of the motion field, etc. Be creative!!! (5 points)
• Visualize the stability in your optical flow calculation using the second largest eigenvalue in the C matrix (similar in spirit to what we did in the first assignment). Again, there are no hard rules about what you need to do here, but your final result will be judged based on its utility and creativity. (5 points)
• Visualize the divergence and curl at each point in your optical flow field (you may need to Google these terms from vector calculus and remind yourself how they are computed). Discuss how (if at all) these quantities could be used in the design of an algorithm for automatic object detection/avoidance in autonomous navigation systems. (5 points)
• Extra credit: visualize the optical flow across the entire video in a single image. This is worth a maximum of 2 points and will be based on creativity and clarity.

• Why did you choose this/these visualization(s)?
• What happens when you increase/decrease N?
• What elements in the scene cause this?

3. Image warping (35 points)

• Create a function that accepts as input two images and their corresponding dense motion field (you should have plenty of examples from the first part).
• Initialize space for the output (warped) image.
• Visit each pixel in the output image and consider the forward motion vector at that position [dx,dy]. Note that [dx,dy] tells you to where in I_2 this pixel in I_1 was mapped. This motion field can be used to warp I_2 in a way that tries to "undo" the result of the motion between these two frames. In particular, sample I_2 at this displaced location and store the result in your output image. Because these motion vectors will never have integral components you will need to interpolate the discrete values in the image to form a continuous approximation that can be evaluated everywhere. Use the interp2 built-in MATLAB function to perform this interpolation. Play around with the reconstruction methods (i.e., 'nearest', 'linear', 'cubic', etc.) and see how choice of reconstruction method influences your final output (25 points).
• Perform this warping on I_2 for each R,G,B channel to obtain color images (5 points).

• Warp each image in a video sequence according to its motion field (note you will not be able to do this for the very first image in the sequence).
• Use your warping function to create more interesting visualizations of the optical flow field by applying it to an image of a regular grid or some other structured image.
• We would like to compare these warped images to the actual motion in the video sequence (which is revealed in the frames themselves). Compute the SSD between the color values stored at each pixel in your warped image and those in the corresponding frame in the sequence (using the notation above: compare the warped version of I_2 with I_1). Visualize this SSD image using your color mapping function from part one. Create a movie from these visualizations across an entire sequence. See comments below regarding making movies.
• Superimpose these SSD images over each corresponding frame in the original sequence by simply computing a weighted combination of the respective colors at each pixel location. Again, make a movie from this image collection.

Submitting

This assignment is due Tuesday, March 20, 2007 at 11:59 PM. Please see the general notes on submitting your assignments, as well as the late policy and the collaboration policy.

Please submit:

• Your write-up as an HTML file with embedded images/videos and links to your code. Please include all of your beautiful visualizations along with a discussion for each and your answers to any questions posed in the assignment.

Data

• book.zip
• hand.zip
• mug.zip
• lawn.zip
• traffic.zip
• bottles.zip
• flowers.zip