Introduction With interactive ray tracing becoming more of a possibility, frameless rendering becomes more important. This project takes an existing OpenGL application and simulates a frameless system. This system can then output samples which can be reconstructed by a completely separate system. This provides the framework for us to experiment with different reconstruction algorithms. Implementation We use Chromium to intercept OpenGL calls and perform our own operations on them. Namely, we wrote a SPU that intercepts the SwapBuffers command and reads the back frame buffer at that point. The frame is read back with a CopyTexSubImage call and bound as a texture that is drawn back into the back buffer with a stencil applied to it. The stencil determines which pixels to keep as samples to be passed to the reconstructor. Finally, the buffer is swapped into the front buffer for visualization of the sampling if the user desires. Since we are using Chromium, we can theoretically convert any OpenGL application into a framelessly rendered application, provided it uses SwapBuffers to determine frame boundaries. Uniform Sampling To achieve uniform sampling, we allocate N stencil buffers and allocate 1/N of the total pixels per stencil buffer. This is done with a series of calls to rand() to determine which pixel to choose, and then checking the current set of buffers already made to see if that pixel has been chosen already. If the pixel has already been chosen, we choose another random pixel. If it hasn't been chosen, then we mark that pixel as a desired pixel to be sampled. The final result is a set of frames with 1/N samples per frame. One of the problems with doing sampling this way is that it is hard to guide sampling in light of information such as edges in the image (see below). The stencil buffers would need to be remade every frame, which would be extremely inefficient. In order to adaptively sample the framebuffer a different way to define the stencil buffers would have to be determined. One possibility for this is overwriting the stencil buffer based on the stencil test. The problem with the method below is we have little control over the number of samples being taken in the edge detection step. The alternative might be to use the edge detection information as a priority map and use this to guide sampling. Adaptive Sampling Because uniform sampling is not always ideal for a reconstruction technique, we can also change our system to accomodate for adaptive sampling. In an ideal world, the reconstruction system would want to guide the sampling with something along the lines of a priority map. The sampling system would then take this priority map and take random samples that were over a certain threshold that could be determined on the fly. Since our framework currently does not allow for crosstalk between the reconstructor and sampler, we implemented a simple Sobel edge detector that finds areas of high luminance difference between adjacent pixels. This serves as one possible technique the reconstructor could have used to determine the priority map. Once given the priority map, the sampler can use any threshold value to determine which samples to pass back to the reconstructor. Obviously, the adaptively sampled image looks much closer to the original image. This is partly because of the greatly increased number of samples (for the city demo, 25000 samples instead of 2500 uniform samples). However, for an application such as the atlantis demo, the adaptively sampled frame appears much closer to the real image than the uniformly sampled frame, with about the same number of total samples taken. The adaptively sampled frames show much higher details around the dolphins, and we are using approximately the same number of samples. Results To be determined... this mostly depends on the reconstruction people The edge detection pass plus the original uniform sampling create the following stencil buffer: Depending on the application, the edge detection may create many more samples than the uniform sampling. For the city demo, edge detection creates almost an order of magnitude more samples than simple uniform sampling, but for the atlantis demo, not that many more samples are created. Speed is also an issue. For uniform sampling, we keep the entire frame as a texture in video memory, so the pixels are never shipped over the bus, so we can maintain a high framerate (about 200-300fps for the city demo). However, for adaptive sampling, it is necessary in this framework to read the pixels off the card before we can run any filters over them. Simply doing a ReadPixels slows down the framerate to about 15fps. Any passes over the framebuffer are almost free relative to the ReadPixels call. In the future, this could all be solved when we put everything in a fragment program. More Background This project was the final assignment in a seminar on Interactive Ray Tracing run by David Luebke in Spring 2003. The reconstructor used along with this can be found here.