CS851: Data-Driven Models in Computer Graphics
Fall 2006
Jason Lawrence

Course Summary

Lectures: F 1200-1500, Room: 228E Olsson

Instructor: Jason Lawrence - 212 Olsson

Announcements

We will meet each Friday from 1200-1500 in 228E Olsson.

A course mailing list has been setup.

Course Description

Computer graphics abounds with hard modeling and simulation problems. A designer must specify the 3d geometry of the objects in a virtual scene, along with their material properties, the sources of illumination and the scene's temporal behavior. These specifications become input to rendering algorithms that simulate the flow of light energy from the sources, through the scene and into a virtual camera to produce a synthetic image.

Geometry, material properties, illumination and animation have traditionally been modeled and represented with "analytic" techniques. For example, the way a plastic material scatters light is often modeled with the popular Phong BRDF. Although expert designers can use these simple models to produce surprisingly compelling imagery, specifying their parameters is tedious and even careful tuning often falls short of photo-realistic results.

One way to address this lack of realism is to incorporate measurements of real-world phenomena into the production pipeline. For example, an alternative to adjusting the parameters of the Phong BRDF is to directly measure the optical properties of the material in question. Efficient acquisition techniques and compact representations of these datasets can aid novice and expert designers in specifying complex real-world scenes.

In this seminar, we will study the ways empirical data is incorporated into the modern graphics production pipeline. These so called data-driven models rely on vision and learning techniques to make them accessible to a wide audience of users in many different contexts. We will study the following topics:

In each case, we will review seminal and current research papers, with the goal of better understanding the capabilities and limitations of current state-of-the-art methods.

Prerequisites

Students should have completed CS445 or an equivalent undergraduate course in computer graphics.

Although not a requirement, undergraduate-level coursework in probability theory, along with exposure to computer vision and machine learning will be very helpful.

Course requirements

There are three course requirements:

Syllabus

August 25
  • Introduction and overview [slides]
September 1
  • Image formation (optics, CCDs, dynamic range, etc.) [slides]
  • Student presentations:
    • [Debevec and Malik 1997] Debevec, P. and Malik, J. (1997). Recovering High Dynamic Range Radiance Maps from Photographs. In Proceedings of ACM SIGGRAPH 1997. [slides]
  • Supplemental material:
    • Paul Debevec's website (hi-res light probes).
    • [Golub 89] Golub, G., van Loan, C.. (1989). Matrix Computations. Johns Hopkins University Press, Baltimore, 1989.
September 8
  • Texture acquisition; polynomial texture maps (PTMs); bi-directional texture function (BTF) [slides]
  • Markovian vs. parametric techniques for texture synthesis [slides]
  • Student presentations:
    • [Heeger and Bergen 1995] Heeger, D. and Bergen J. (1995). Pyramid-Based Texture Analysis/Synthesis. Proceedings of ACM SIGGRAPH 1995, pages 229-238.
    • [Efros and Leung 1999] Efros, A. and Leung, T. (1999). Texture Synthesis by Non-Parametric Sampling. IEEE International Conference on Computer Vision (ICCV). [slides]
    • [Hertzmann et al., 2001] Hertzmann, A., Jacobs, C., Oliver, N., Curless, B., Salesin, D. (2001). Image Analogies. Proceedings of ACM SIGGRAPH 2001, pages 327-340. [slides]
  • Required reading:
    • [Dana et al., 1999] Dana, K., van Ginneken, B., Nayar S., and Koenderink. (1999). Reflectance and Texture of Real World Surfaces. ACM Transactions on Graphics, Vol. 18, No. 1, 1-34.
    • [Malzbender et al., 2001] Malzbender, T., Gelb, D., Wolters, H. (2001). Polynomial Texture Maps. In Proceedings of ACM SIGGRAPH 2001.
  • Supplemental material:
    • [Portilla and Simoncelli 2000] Portilla, J. and Simoncelli, E. (2000). A Parametric Texture Model based on Joint Statistics of Complex Wavelet Coefficients. International Journal of Computer Vision 40(1), 49-70.
    • [Grinstead and Snell] Grinstead, C. and Snell, J. Introduction to Probability. American Mathematical Society. Note: Chapter 11 covers Markov Chains.
    • [Zhu et al., 1997] Zhu, S., Wu, Y., Mumford, D. (1997). Filters, Random Fields and Maximum Entropy (FRAME): Towards a Unified Theory for Texture Modeling. International Journal of Computer Vision, 27(2), 107-126.
    • [De Bonet 1997] De Bonet, J. (1997). Multiresolution Sampling Procedure for Analysis and Synthesis of Texture Images. Proceedings of ACM SIGGRAPH 1997, pages 361-368.
September 15
  • Image matting; compositing [slides]
  • Student presentations:
    • [Chuang et al., 2001] Chuang, Y., Curless, B., Salesin, D., Szeliski, R. (2001). A Bayesian Approach to Digital Matting. In Proceedings of IEEE Computer Vision and Pattern Recognition (CVPR 2001), Vol. 11, 264-271. [slides]
    • [Sun et al., 2004] Sun, J., Jia, J., Tang, C.K., Shum, H.Y. (2004). Poisson Matting. In Proceedings of ACM SIGGRAPH 2004.
  • Required reading:
  • Supplemental material:
    • [NR, Ch. 19.4] Press et al. (1992). Numerical Recipes in C.
    • [NR, Ch. 19.5] Press et al. (1992). Numerical Recipes in C.
    • [NR, Ch. 19.6] Press et al. (1992). Numerical Recipes in C.
    • [Smith and Blinn 1996] Smith, A.R. and Blinn, J. (1996). Blue Screen Matting. In Proceedings of ACM SIGGRAPH 1996.
    • [Ruzon and Tomasi 2000] Ruzon, M.A. and Tomasi, C. (2000). Alpha Estimation in Natural Imges. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2000).
    • [Hillman et al., 2001] Hillman, P., Hannah, J., Renshaw, D. (2001). Alpha Channel Estimation in High Resolution Images and Image Sequences. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2001).
    • [Wang and Cohen 2005] Wang, J. and Cohen, M. (2005). An Iterative Optimization Approach for Unified Image Segmentation and Matting. In Proceedings of IEEE International Conference on Computer Vision (ICCV 2005).
    • [Levin et al., 2006] Levin, A., Lischinski, D., Weiss, Y. (2006). A Closed Form Solution to Natural Image Matting. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2006).
    • [Chuang et al., 2002] Chuang, Y.Y., Agarwala, A., Curless, B., Salesin, D., Szeliski, R. (2002). Video Matting of Complex Scenes. In Proceedings of ACM SIGGRAPH 2002.
September 22
  • Plenoptic function; description and taxonomy
  • Light field capture and rendering [slides]
  • Student presentations:
    • [Levoy and Hanrahan 1996] Levoy, M. and Hanrahan, P. (1996). Light Field Rendering. In Proceedings of ACM SIGGRAPH 1996.
    • [Gortler et al. 1996] Gortler, S. J., Grzeszczuk, R., Szeliski R., and Cohen, M. F. (1996). The Lumigraph. In Proceedings of ACM SIGGRAPH 1996.
  • Supplemental material:
    • [Buehler et al. 2001] Buehler, C., Bosse, M., McMillan, L., Gortler, S., Cohen, M. (2001). Unstructured Lumigraph Rendering. In Proceedings of ACM SIGGRAPH 2001.
    • [Shum and He 1999] Shum, H. Y., He, L. W. (1999). Rendering with concentric mosaics. In Proceedings of ACM SIGGRAPH 1999.
    • [Wood et al. 2000] Wood, D., Azuma, D., Aldinger, K., Curless, B., Duchamp, T., Salesin, D., and Stuetzle, W. (2000). Surface Light Fields for 3D Photography. In Proceedings of ACM SIGGRAPH 2000.
    • [Aliaga et al. 2002] Aliaga, D., Funkhouser, T., Yanovsky, D., Carlbon, I. (2002). Sea of Images. In Proceedings of IEEE Visualization.
    • [Ng et al. 2005] Ng, R., Levoy, M., Bredif, M., Duval, G., Horowitz, M., Hanrahan, P. (2005). Light Field Photography with a Hand-held Plenoptic Camera. Stanford Technical Report CTSR 2005-02.
September 29
  • Image-based modeling; structure-from-motion. [slides]
  • Student presentations:
    • [Debevec et al. 1996] Debevec, P., Taylor, C.J., Malik, J. (1996). Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach. In Proceedings of ACM SIGGRAPH 1996.
    • [Snavely et al. 2006] Snavely, N., Seitz, S.M., Szeliski, R., (2006). Photo Tourism: Exploring Photo Collections in 3D. In Proceedings of ACM SIGGRAPH 2006. [slides]
  • Supplemental material:
    • [Brown and Lowe 2005] Brown, M., Lowe, D.G. (2005). Unsupervised 3D Object Recognition and Reconstruction in Unordered Datasets. In Proceedings of International Conference on 3D Digital Imaging and Modelling.
    • [Pollefeys et al. 1999] Pollefeys, M., Koch, R., Vergauwen, M., van Gool, L. (1999). Hand-held Acquisition of 3D Models with a Video Camera. In Proceedings of 3D Imaging and Modeling.
October 6
  • Projects (webpage):
    • Written project proposals due.
    • Project proposal presentations.
October 13
  • Reflectance fields; acquisition and representation [slides]
  • Student presentations:
    • [Debevec et al. 2000] Debevec, P., Hawkins, T., Tchou, C., Duiker, H.P., Sarokin, W., Sagar, M. (2000). Acquiring the Reflectance Field of a Human Face. In Proceedings of ACM SIGGRAPH 2000.
    • [Masselus et al. 2003] Masselus, V., Peers, P., Dutre, P., Willems, Y.D. (2003). Relighting with 4D Incident Light Fields. In Proceedings of ACM SIGGRAPH 2003.
  • Supplemental reading material:
    • [Chen et al. 2002] Chen, W.C., Bouguet, J.Y., Chu, M.H., Grzeszczuk, R. (2002). Light Field Mapping: Efficient Representation and Hardware Rendering of Surface Light Fields. In Proceedings of ACM SIGGRAPH 2002.
October 20
  • Appearance acquisition and representation [slides]
  • Student presentations:
    • [Ward 1992] Ward, G. (1992). Measuring and Modeling Anisotropic Reflection. In Proceedings of ACM SIGGRAPH 1992.
    • [Marschner et al. 2000] Marschner, S., Westin, S., Lafortune, E., and Torrance, K. (2000). Image-Based Bidirectional Reflectance Distribution Function Measurement. Applied Optics, Vol. 39, Issue 16.
  • Required reading:
    • [Nicodemus et al. 1977] Nicodemus, F. E., Richmond, J. C., Hsia, J. J., Ginsberg, I. W., and Limperis, T. (1977). Geometric Considerations and Nomenclature for Reflectance. Monograph 160, National Bureau of Standards (US).
    • [White et al. 1998] White, D. R., Saunders, P., Bonsey S. J., van de Ven, J., and Edgar, H. (1998). Reflectometer for Measuring the Bidirectional Reflectance of Rough Surfaces. Applied Optics, Vol. 37, Issue 16.
  • Supplemental material:
    • [Nadal and Thompson 2001] Nadal, M. E., and Thompson, E. A. (2001). NIST Reference Goniophotometer for Specular Gloss Measurements. Journal of Coatings Technology, 73 (73-80).
    • [Marschner et al. 1999] Marschner, S., Westin, S., Lafortune, E., Torrance, K., and Greenberg, D. (1999). Image-Based BRDF Measurement Including Human Skin. In Proceedings of the Eurographics Workshop on Rendering.
    • [Leonard et al. 1993] Leonard, T. A., Rudolph, P. (1993) BRDF Round-Robin Test of ASTM E1392. In Optical Scattering (SPIE).
    • [Early et al. 2000] Early E. A, et al. (2000). Bidirectional Reflectance Round-Robin in Support of the Earth Observing System Program. Journal of Atmospheric and Oceanic Technology Vol. 17, No. 8.
    • [Lu et al. 1998] Lu R., Koenderink, J. J., and Kappers A. M. L. (1998). Optical Properties (Bidirectional Reflectance Distribution Functions) of Velvet. In Applied Optics, Vol. 37, Issue 15.
    • [Stavridi et al. 1997] Stavridi, M., van Ginneken, B., Koenderink, J. J. (1997). Surface Bidirectional Reflection Distribution Function and the Texture of Bricks and Tiles. In Applied Optics, Vol. 36, Issue 15.
October 27
  • Perception- and data-driven reflectance models
  • Student presentations:
    • [Pellacini et al. 2000] Pellacini, F., Ferwerda, J. A., Greenberg, D. P. (2000). Toward a Psychophysically-based reflection model for computer graphics. In Proceedings of ACM SIGGRAPH 2000. [slides]
    • [Matusik et al. 2003] Matusik, W., Pfister, H., Brand, M., McMillan, L. (2003). A Data-Driven Reflectance Model. In Proceedings of ACM SIGGRAPH 2003. [slides]
  • Supplemental reading:
    • [Rusinkiewicz 1998] Rusinkiewicz, S. (1998). A New Change of Variables for Efficient BRDF Representation. In Proceedings of Eurographics Workshop on Rendering.
November 3
  • Acquisition and representation of spatially-varying reflectance [slides]
  • Student presentations:
    • [Lensch et al. 2003] Lensch, H. P. A., Kautz, J., Goesele, M., Heidrich, W., and Seidel, H.P. (2003). Image-Based Reconstruction of Spatial Appearance and Geometric Detail. In ACM Transactions on Graphics, 22(2).
    • [Lawrence et al. 2006] Lawrence, J. Ben-Artzi, A., DeCoro, C., Matusik, W., Pfister, H., Ramamoorthi, R., Rusinkiewicz, S. (2006). Inverse Shade Trees for Non-Parametric Material Representation and Editing. In Proceedings of ACM SIGGRAPH 2006.
  • Required reading:
    • [Wang et al. 2006] Wang, J., Tong, X., Lin, S., Wang, C., Pan, M., Bao, H., Guo, B., Shum, H.Y. (2006). Appearance Manifolds for Modeling Time-Variant Appearance of Materials. In Proceedings of ACM SIGGRAPH 2006.
November 10
  • Projects (webpage):
    • Project progress presentations.
  • Human motion capture and data analysis
  • Student presentations:
November 17
  • Procedural modeling and performance-driven facial animation
  • Student presentations:
    • [Quan et al. 2006] Quan, L., Tan, P., Zeng, G., Yuan, L., Wang, J., Kang, S.B. (2006). Image-based Plant Modeling. In Proceedings of ACM SIGGRAPH 2006.
    • [Vlasic et al. 2005] Vlasic, D., Brand, M., Pfister, H., Popovic, J. (2005). Face Transfer with Multilinear Models. In Proceedings of ACM SIGGRAPH 2005.
December 1
  • Miscellaneous topics and course summary [slides]
  • Required reading:
    • [Nayar et al. 2006] Nayar, S.K., Krishnan, G., Grossberg, M.D., Raskar, R. (2006). Fast Separation of Direct and Global Components of a Scene Using High Frequency Illumination. In Proceedings of ACM SIGGRAPH 2006.
    • [Lischinski et al. 2006] Lischinski, D., Farbman, Z., Uyttendaele, M., Szeliski, R. (2006). Interactive Local Adjustment of Tonal Values. In Proceedings of ACM SIGGRAPH 2006.
December 14
  • Projects (webpage):
    • Final project presentations and write-ups due.