Computer Vision
• Optical Flow

Due: November 6

Code

1. Generalize the line fitting technique covered in class, which allows fitting a single line to a set of points, to allow fitting multiple lines to a set of points. Your solution should use the Expectation-Maximization (EM) framework and answer the following questions:
   • What is the generative process you assume underlies the distribution of the set of points?
   • Points ought to be generated by some random, yet small, perturbation of a main line to which they are associated. If we have a set of points that we would like to fit a series of lines to, then we must have assumed that these points were generated in this way, or they would not map to lines at all.
   • What are the parameters of your model that you aim to estimate?
   • We aim to estimate the slope and y-intercept of the set of lines we wish to find. There may be a user parameter controlling this amount, or possible an interative approach. Assuming we had a user, then the number of parameters would be 2N where N is the number of lines. If we were to iterate over the problem to determine which number of lines yields the lowest error, at each iteration we would have 2(i+1), where 'i' refers to the ith iteration starting from iteration 0. We don't have estimate slope and y-intercept. We can just as easily use r and θ instead, and in practice we would want to do this to allow vertical lines.
   • What is the hidden or missing data in this problem?
   • The r and θ coordinates for each line and possibly the number of lines if we do not wish to have a user set this number.
   • What is the E-step?
   • The expectation value should be associated with the deviation of the points from their closest line segment. Thus the E-step should consist of the aggregating all points to their closest line.
   • What is the M-step?
   • In the maximization step, we seek to construct a new set of r and θ that maximizes the probability that points belong to this new line. This can be done by moving the lines or rotating the lines so that the probability or likelihood increases. These new points can then be fed into the E step for iteration.
   • One extra set of steps may be to add a new line in the maximization step. Since the added line may increase the probability that the new set of lines is the correct set, it can also be considered as part of the maximization step.
   • How do you initialize your algorithm?
   • The algorithm could be initialized with a random set of lines with random r and θ values but a better approach would use the output of a Hough Transform line detector. This output would already be close to the correct solution, but the EM approach would allow the lines to be translated and rotated slightly to find the true solution.
   • How would you evaluate your solution?
   • The best way to evaluate a solution such as this is to find the deviation of all points along a given line. This deviation should be minimized when the EM process has completed on a per line basis.


2. Imagine that you are the lead researcher for the Image Search group at Google. Outline an image retrieval system that takes an image as input and returns a rank-ordered list of images from the Internet that are most similar. Your solution should make use of the image segmentation and texture analysis methods we covered in class. Clearly specify the way you define distance between two images (this must be a scalar value).

The first step would be to separate the input image into some set of usable images. This would be achieved by computing the Laplacian of the input image scaled by powers of two. The so-called Laplacian pyramid could then be convolved on each level with a subset of the textons proposed by Olhausen and Field. This would produce N number of Laplacian pyramids, where N would be the number of textons. N is hard to determine without knowledge of the results of this operation, but it should be related to the amount of variance in "texture" space the input image has. The textons can be thought of as eigentextures and so depending on the input image, fewer eigentextures may be needed to decompose it. The resulting images can be thought of as vectors in texture space. These vectors can then be searched against a database of precomputed texture vectors. The "engine" under the hood here is this: assuming that we can recover the eigentextures present in the world, we can project any input texture into this space and the Euclidean distance in texture space should be small when a match is present. Therefore, we would look to minimize the Euclidean distance in texture space between the input convolved Laplacian pyramids and the database's convolved Laplacian pyramids. The Laplacian pyramid approach is rather secondary to the mechanism behind the matching. It ensures that scaled forms of the same texture can be accurately found too. Unfortunately this approach would require a good deal of texture present in all images so that each vector in texture space would be unique.

• A false colorization is useful for visualizing optical flow in a more meaningful manner. By mapping scalar values in [0,1] to color channels, more information can be extracted from a single optical flow picture. Since optical flow is the motivating factor for creating a false colorization scheme, the choices for how values were mapped was dependent on visualizing error. At first a simple mapping was performed that converted the scalar value to a hue value and color was assigned in HSV space as (value, 1, 1). Although this does show a unique mapping, it can be extended further to allow better visualization of error.

V = 1 :: Relatively no error is shown	γ = 0.7
γ = 0.5	γ = 0.2 :: Low error is accurately visualized and the overall scaling is adequate

• A second method was developed that attempted to exploit the needs of the colorization. The idea is such: areas that are light should correspond to low error or high confidence, and dark areas the opposite. The requires that the V parameter in HSV space be used. However, if a linear relationship is deployed, most areas become too dark. Mixing this two ideas gives a hint that the function for the V coordinate must rapidly rise, but have enough variance after that rise to provide discernable darkness. Many factors of the variety x^(γ) where γ < 1. The value that produced the most desirable result was γ=0.2.
• Here are some fun examples using this derived visualization presenting the women of Food Network© and a boring topo-map:

Original Image	False-Colored Image

• The optical flow algorithm uses the assumption that brightness is constant per image and movement is only translational. It can understood as solving a linear system of equations resulting from these assumptions:

• Here the optical flow field is defined as v=(dx/dt,dy/dt)
• Although this looks as though it would be a simple task, unfortunately that is not the case. The gradients noted above are 3D by definition and so any gradient that is to be computed for the purposes of optical flow must be done as such in 3D, rather than the standard 2D. Luckily, the gradient can be thought of as a convolution with a 3D gaussian distribution function. Also lucky is the fact that a convolution in 3D can be performed as multiplication in fourier space using the standard fourier transform trick.
• In practice this would work just fine, though with a rather hefty running time. However, there was an incredible struggle with memory in performing this task. MATLAB would not allow the convolutions to be performed on stacks much deeper than about 25 images at first. In order to remedy this, the images were first converted to single-precision. This allowed more images to be in the stack, but there were more hang-ups. Since the fourier trick requires that the fourier transform be performed on two matrices which are then subsequently multiplied, the amount of memory needed is 3 times the amount needed for one matrix. On top of all of this, other matrices had to be extracted from this product to form the appropriate gradients. The workaround was to compute each fourier transform separately and immediately save to disk. Then, when they were needed, they were loaded only as needed and then immediately deleted. This method allowed a stack depth of about 175.
• Here are a series of movies that show the resulting gradients for the book dataset:

• Once this gradient has been computed, it can be used to construct matrices that will solve for the optival flow field v:

• Where the columns of A are the x and y components of the gradient of I and b is a column vector of the t gradient component of I. Notice that when written in this form, the covariance matrix C falls out of the equation. This will provide a fast way to compute the necessary inverse, since C is 2x2.
• This calculation must be performed at each (x,y) in the image with the columns of A and b extracted around a specified neighborhood of size NxN.
• Visualizing the optical flow becomes the challenge now. For each supplied dataset the table below contains the optical flow, the stability of the solution, and the error associated with each pixel. The optical flow is drawn as arrows pointing in the direction of v and with magnitude proportional to v. The color of the arrows is generated by the false colorization from above. Since we would like encode both the stability of the solution and the error, the color is generated as (stability) * (1-error). This means that when error is low and stability high (aka a good point) we expect to have a hue value near 1 (blue). Stability is calculated as the second largest eigenvalue of the covariance matrix at each point and the error is the magnitude of deviation from 0 that results in Av+b at each pixel. For ease of visualization, pictures of the continuous error and stability are provided for each image. The red portions correspond to low error and low stability, while blue areas correspond to high error and high stability.

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

Optical Flow
Stability	Error

• Notice that the optical flow estimates are better when there is a good amount of texture present. This is because in these regions, the gradients of the image are differing from pixel to pixel. When these gradients are accumulated into the A and b matrices, they will contain few values of 0 where there is a lot of texture. This means that the system will be well-constrained to be solved and will therefore produce a good estimate of the optical flow.
• Something interesting that happens in some of the scenes (the mug scene most notably) is that areas with strong specular highlights throw the calculation way off. This is most likely to the fact that the values at these pixels are constantly changing over time, because the viewing angle is constantly changing. In more diffuse surfaces (such as the hand), the optical flow behaves as expected more often.
• The aperture problem na also be seen in the image sets. Take the lawn sequence (the final three images). In the second image, the man is obviously walking towards the camera and the algorithm accurately portrays this motion. However, in the third image, he is too small for the detector to accurately gauge his flow. Since his pixels are being considered with other pixels that are moving in other directions, his optical flow calculation produces an incorrect result. In this case, a smaller neighborhood would be needed to accurately recover his optical flow.
• Here is a video sequence showing the optical flow of the bottles sequence as it evolves over 110 frames.

• Although it may be hard to see from the first run, if the movie played over and over it does provide a relatively good measure of the optical flow in the scene subject to the problems mentioned above.
• All images above were created with a neighborhood size of 20 pixels. This size was used because it exhibited the most optimal behaviour in early test scenes. A size of 10 pixels often falied to capture enough motion in the scene because the overall motion of a 10x10 window was unclear. Increasing this to 20 solved this problem in many regions, but it also minimized the number of false identifications due to more than one motion direction. If the size were to be increased to 40x40, too many pixels, many of which are moving in different directions, are being considered and therefore give bad results.

• To further inspect this idea, a pair of images was evaluated at different neighborhood sizes. The first, was an image from the hand sequence and the second was from the flowers sequence. In both cases, neighborhood sizes of 10, 20, and 40 pixels were used, and both the error and stability were used to perform the analysis. The optical was not used because the information it provides is less quantitative in nature. The images that are produced by the error and stability calculations are derived from real numbers that result from solving the linear system of equations.

	Hand Image	Flowers Image
Image
N=10 Error
N=10 Stability
N=20 Error
N=20 Stability
N=40 Error
N=40 Stability

• From this table we see that the effect of increasing N is associated with the "blur" in the error and stability. This is why a middle choice of N works well in practice; it gives necessary attention to high error and low stability, but is more forgiving in neighoring areas where the optical flow might have been poorly calculated. It essentially defacts to its neighbors more in determining its own optical flow.
• One important note regarding the stability: the stability is calculated as the second largest eigenvalue of the covariance matrix at each pixel. This means it is a measure of how strong of an edge appears at that pixel. The problem here is that the edge that is referred to is only in the spatial domain. Edges in the temporal domain do not recieve a high stability measurement. This is a slight flaw in the design ofthe system, but it is acceptable because the error should be low at those points anyway. This is one of the main reasons both the stabiltiy and the error are used to color the arrows.
• The error images are somewhat disturbing at first. Apparently error is very low in regions that aren't moving much. Well this is unfortunately a biproduct of design. Since the error is related to how well Av = -b, the error will be low when this system is well-constrained. But in areas where these is motion, the A and b matrices are both behaving poorly. In addition, any area that has uniform texture will have very high error associated with it. This is because the columns of A are the spatial gradients of the image and will be almost 0 everywhere for uniform textures. This will not allow the system to be solved to good accuracy.
  

---

© 2007 Sean M. Arietta
University of Virginia