CS851: Data Mining Algorithms
Nina Mishra

Course Overview

Many organizations have compiled a diverse collection of massively large and dynamic datasets over the years.  Data mining is a tool that has been actively used to discover interesting and surprising patterns in these datasets.  The technology has been successfully utilized by organizations that collect web click streams, financial transactions, observational science data, etc.  This course will cover major algorithmic advances in data mining with an eye towards both the theoretical underpinnings of these problems as well as successful practical deployments.  Topics covered include clustering, association rules, machine learning, web link analysis, data streams, and privacy-preserving algorithms.

This course should be of interest to graduate students in computer science and many other related disciplines. This course is especially tuned to those with an interest in understanding the fundamentals of data mining. Familiarity with basic material in algorithms, databases and probability at the level of the core undergraduate courses is useful.

Grading:  Those students interested in taking the course for a letter grade must complete a class project, and scribe two lectures.  Those signing up for the pass/fail option will need to scribe one lecture or complete a class mini-project.

Class Project: Deadlines:

• Feb 15, 2006: Proposal: at most 1 page  (5%)
• Mar 15, 2006: Progress Report: at most 5 pages (20%)
• Apr 17, 2006: Project Presentations.  (25%)
• May 8, 2006: Final Project Report  (30%)

Scribe:  Each registered student will sign up as the official scribe for two lectures.  This involves taking detailed notes, reading the background papers, and preparing a set of lecture notes that will be distributed on the web. (20%)

Scribe Schedule

Presentation Schedule

Homework Assignment: Due May 8th, for students who scribed only one lecture.

Time and Location:

Office Hours:

Overview of Lectures (Tentative)

 

Date	Topics	Lectures	Scribes
Jan 18, 2006	Introduction, Overview of the Class, Preliminaries	Lecture 1	Scribe 1
Jan 23/25, 2006	Clustering: k-Center, k-Median, k-Median-squared	Lecture 2, Lecture 3	Scribe 2/3
Jan 30/Feb 1, 2006	Clustering: Hierarchical, Correlation	Lecture 4, Lecture 5	Scribe 4
Feb 6/8, 2006	Correlation Clustering, Association Rules	Lecture 6	Scribe 5/6
Feb 13/15, 2006	Frequent Itemsets and CNF/DNF Dualization	Lecture 7/8, Lecture 9	Scribe 7/8, Scribe 9
Feb 20/22, 2006	Winnow, PAC-learning, Consistency, Learning Conjunctions, k-DL	Lecture 10, Lecture 11	Scribe 10/11
Feb 27/Mar 1, 2006	Boosting	Lecture 12, Lecture 13	Scribe 13
Mar 6/8, 2006	Spring Break
Mar 13/15, 2006	Pagerank, Hubs & Authorities, Preferential Attachment Random Graph Model	Lecture 14, Lecture 15	Scribe 14, Scribe 15
Mar 20/22, 2006	Viral Marketing/Spreading Epidemics	Lecture 16/17	Scribe 16, Scribe 17
Mar 27/29, 2006	Data Streams	Lecture 18/19	Scribe 18, Scribe 19
Apr 3/5, 2006	Data Privacy	Lecture 20/21	Scribe 20/21
Apr 10/12, 2006	Data Privacy	Lecture 22	Scribe 22, Scribe 23
Apr 17/19, 2006	Project Presentations
April 24/26, 2006	Project Presentations
May 1, 2006	Project Presentations

 

Reading List

k-Center

• Clustering to minimize the maximum intercluster distance, T. F. Gonzalez. Theoretical Computer Science, 38: 293-306, (1985).
• A unified approach to approximation algorithms for bottleneck problems.  D. Hochbaum  and D. Shmoys.  Journal of the ACM (JACM).  Volume 33 ,  Issue 3  (July 1986). Pages: 533 - 550 
• Optimal algorithms for approximate clustering.  T. Feder, D. Greene.  STOC'88.  pages 434-444.
• Clustering Motion.  S. Har-Peled.  FOCS, 2001.

k-Median/k-Median-squared/Facility Location

• Local Search Heuristics for k-median and Facility Location Problems, V. Arya, N. Garg, R. Khandekar, A.Meyerson, K. Munagala and V. Pandit. STOC 2001. Rohit Khandekar's slides
• Improved Combinatorial Algorithms for the Facility Location and k-Median Problems, M. Charikar and S. Guha.  FOCS 1999.
• Clustering in large graphs and matrices, P. Drineas, R. Kannan, A. Frieze, S. Vempala, and V. Vinay. SODA 1999.
• A generalized convergence theorem and characterization of local optimality.  S. Selim and M. Ismail.  IEEE Transansactions on Pattern Analysis and Machine Intelligence.  6:(1):81-86.  1984.
• A simple linear time (1+eps)-approximation algorithm for k-Means clustering in any dimensions A. Kumar, Y. Sabharwal, S. Sen. FOCS 2004.
• On coresets for k-means and k-median clustering S. Har-Peled and S. Mazumdar. STOC 2004.
• How fast is the k-means Method?. B. Sadri and S. Har-Peled. In Algorithmica, 41(3):185--202, 2005. Also in SODA 05.
• A Local Search Approximation Algorithm for k-Means Clustering. T. Kanungo, D. M. Mount, N. Netanyahu, C. Piatko, R. Silverman, and A. Y. Wu, Computational Geometry: Theory and Applications, 28 (2004), 89-112.

Hierarchical Clustering

• Criteria for Polynomial-Time (Conceptual) Clustering,  L. Pitt and Reinke. Machine Learning 2:371-396 (1988).
• Performance guarantees for hierarchical clustering, S. Dasgupta, COLT, 2002.

Clustering Large Data Sets

• Testing of Clustering.  N. Alon, S. Dar, M. Parnas, D. Ron.  FOCS 2000.
• Sublinear Time Algorithms for Metric Space Problems. P. Indyk, STOC 1999.
• CURE: An Efficient Clustering Algorithm for Large Databases, S. Guha, R. Rastogi, and K. Shim. SIGMOD 1998.  Note: this PDF file requires a huge amount of temp space (over 200Mb).
• Clustering Large Datasets in Arbitrary Metric Spaces, V. Ganti, R. Ramakrishnan, J. Gehrke, A. L. Powell, and J.C. French. ICDE 1999.
• BIRCH: an efficient data clustering method for very large databases, T. Zhang, R. Ramakrishnan, and M. Livny.  SIGMOD 1996.
• Sublinear Time Approximate Clustering, N. Mishra, D. Oblinger, and L. Pitt. SODA 2001.
• A Framework for Statistical Clustering with a Constant Time Approximation Algorithms for K-Median Clustering.  S. Ben-David.  p. 415-426. COLT'04.

Clustering Data Streams

• Incremental Clustering and Dynamic Information Retrieval, M. Charikar, C. Chekuri, T. Feder and R. Motwani. STOC 1997.
• Clustering Data Streams, S. Guha, N. Mishra, R. Motwani and L. O'Callaghan. FOCS 2000.
• Clustering Data Streams: Theory and Practice.  S. Guha, A. Meyerson, N. Mishra, R. Motwani and L. O'Callaghan.  IEEE TKDE, 15(3): 515-528, 2003.
• Maintaining Variance and k-Medians over Data Stream Windows,  B. Babcock, M. Datar, R. Motwani, and L. O'Callaghan, PODS 2003.
• Better Streaming Algorithms for Clustering Problems, M. Charikar, L. O'Callaghan, and R. Panigrahy, STOC 2003.

Correlation Clustering

• Correlation Clustering. A. Blum, N. Bansal, S. Chawla.  FOCS'02.
• Clustering with Qualitative Information. M. Charikar, V. Guruswami, A. Wirth: FOCS 2003: 524-533.  Slides.
• Correlation Clustering with Partial Information.  E. Demaine and N. Immorlica. in APPROX 2003.

Association Rule Mining and Generalizations

• Mining Associations between Sets of Items in Massive Databases, R. Agrawal, T. Imielinski, and A. Swami. SIGMOD 1993.
• Fast Algorithms for Mining Association Rules, R. Agrawal and R. Srikant. VLDB 1994.
• An Effective Hash-Based Algorithm for Mining Association Rules, J. S. Park, M.-S. Chen, and P. S. Yu. SIGMOD 1995.
• An Efficient Algorithm for Mining Association Rules in Large Databases , A. Savasere, E. Omiecinski, and S. Navathe. The VLDB Journal 1995.
• Sampling Large Databases for Association Rules, H. Toivonen. VLDB 1996.
• Dynamic Itemset Counting and Implication Rules for Market Basket Data, S. Brin, R. Motwani, S. Tsur, and J.D. Ullman. SIGMOD 1997.
• Query Flocks: A Generalization of Association-Rule Mining, D. Tsur, J.D. Ullman, S. Abiteboul, C. Clifton, R. Motwani, S. Nestorov and A. Rosenthal. SIGMOD 1998.
• Finding Interesting Associations without Support Pruning, E. Cohen, M. Datar, S. Fujiwara, A. Gionis, P. Indyk, R. Motwani, J.D. Ullman, and C. Yang. ICDE 2000.
• Dynamic Miss-Counting Algorithms: Finding Implication and Similarity Rules with Confidence Pruning, S. Fujiwara, R. Motwani, and J.D. Ullman. ICDE 2000.

Combinatorics of Association Rules

• New results on monotone dualization and generating hypergraph transversals, T. Eiter, G. Gottlob, and K. Makino. STOC 2002.
• Efficient Read-Restricted Monotone CNF/DNF Dualization by Learning with Membership Queries, C. Domingo, N. Mishra, and L. Pitt. Machine Learning 37(1):89-110 (1999).
• On the Complexity of Dualization of Monotone Disjunctive Normal Forms, M.L. Fredman and L. Khachiyan.
  Journal of Algorithms 21(3): 618-628 (1996).
• Identifying the Minimal Transversals of a Hypergraph and Related Problems, T. Eiter, and G. Gottlob. SIAM Journal on Computing 24(6): 1278-1304 (1995).

Frequency Counting

• Approximate Frequency Counts over Data Streams, G. Manku, and R. Motwani. VLDB 2002.  
• Computing iceberg queries efficiently, M. Fang, N. Shivakumar, H. Garcia-Molina, R. Motwani and J.D. Ullman. VLDB 1998.

Machine Learning

• An Introduction to Computational Learning Theory, M. Kearns and U. Vazirani. MIT Press, 1994.
• Learning Decision Lists, Ronald Rivest, Machine Learning, 1987.
• Learning quickly when Irrelevant attributes abound. Nick Littlestone, Machine Learning, 1988
• Boosting, Rob Schapire.
• Machine Learning, T. Mitchell. McGraw Hill, 1997. (The website for the book has additional materials such as slides).

 

Random Graph Models

• A.-L. Barabasi, Reka Albert, and Hawoong Jeong. Mean-field theory for scale-free random networks. Physica A 272 173-187 (1999).
• Bernardo A. Huberman, Lada A. Adamic. Growth dynamics of the World-Wide Web. Nature, 399 (1999) 130.
• R. Kumar, P. Raghavan, S. Rajagopalan, D. Sivakumar, A. Tomkins, and E. Upfal. Stochastic models for the Web graph. 41th IEEE Symp. on Foundations of Computer Science, 2000, pp. 57-65.
• W. Aiello, F. Chung, L. Lu. Random evolution of massive graphs. Handbook of Massive Data Sets, (Eds. James Abello et al.), Kluwer, 2002, pages 97-122.

Viral Marketing/Spreading Epidemics

• Pedro Domingos, Matt Richardson. Mining Knowledge-Sharing Sites for Viral Marketing. Eighth International Conference on Knowledge Discovery and Data Mining, KDD-2002.
• D. Kempe, J. Kleinberg, E. Tardos. Maximizing the Spread of Influence through a Social Network. Proc. 9th ACM SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining, 2003.

Privacy: Query Restriction/Auditing

• Simulatable Auditing.  K. Kenthapadi, N. Mishra, K. Nissim.  PODS, 2005.
• Auditing boolean attributes, Kleinberg, Papadimitriou, Raghavan, Journal of Computer and System Sciences, Volume 66 ,  Issue 1  (February 2003), Special issue on PODS 2000, Pages: 244 - 253, 2003.
• Security problems on inference control for SUM, MAX, and MIN queries.  F. Chin.  Journal of the ACM (JACM).  Volume 33 ,  Issue 3.   Pages: 451-464.  1986.
• Auditing for secure statistical databases. F. Chin and G. Ozsoyoglu. ACM 81 Conference (1981).
• Security in Databases: A Combinatorial Study.  S. Reiss.  Journal of the ACM (JACM),  Volume 26 ,  Issue 1. Pages: 45 - 57.  1979.
• Secure databases: protection against user influence, Dobkin, Jones, Lipton, ACM Transactions on Database Systems (TODS), Volume 4 ,  Issue 1  (March 1979), 97 - 106.
• Security in statistical databases for queries with small counts.  Chin.  ACM Transactions on Database Systems (TODS).  Volume 3 ,  Issue 1.  pages 92-104. 1978

Privacy: Cell Suppression

• A graph theoretic approach to statistical data security. D. Gusfield. SIAM J. Comput., 75:552--571, 1989.
• Data security equals graph connectivity, Ming-Yang Kao. SIAM Journal on Discrete Mathematics, Volume 9, Number 1,pp. 87-100

Privacy: Input Perturbation

• Limiting Privacy Breaches in Privacy Preserving Data Mining. A. Evfimievski, J. Gehrke, and R. Srikant. PODS 2003.
• Towards Privacy in Public Databases. S. Chawla, C. Dwork, F. McSherry, A. Smith, H. Wee.
• Privacy-Preserving OLAP.  Agarwal, Srikant, Thomas. SIGMOD 2005.

Privacy: Output Perturbation

• Practical Privacy: The SuLQ Framework.  Avrim Blum, Cynthia Dwork, Frank McSherry, Kobbi Nissim: . PODS 2005.

• Revealing Information while Preserving Privacy. I. Dinur and K. Nissim, PODS 2003.  (abstract) (ppt) (html) (pdf)

• Privacy Preserving Data Mining on Vertically Partitioned Databases.  C. Dwork and K. Nissim.  Manuscript.  2004.

Privacy: K-Anonymity