The course project is to help you get hands-on experience on developing reinforcement learning algorithms to solve interesting and practical problems. The project appreciates either research-oriented problems or “deliverables.” It is preferred that the outcome of your project could be directly publishable, e.g., your (unique) solution to some (interesting/important/new) problems, or tangible, e.g., some kind of prototype system that can be demonstrated. Bonus points will be given to the groups meet either one of above criteria. Group work (2-3 students) is required.

General Steps

• Pick a topic and form a team
• Survey related work and prepare a proposal
• Work on the project
• Write a report and present your project

Your project will be graded based on the following required components:

• Project proposal (25%)
  • Everyone posts on Piazza to describe your project idea and find your teammates (by Sept. 22th);
  • Every team writes a maximum four-page proposal to deliver your motivation and management plan for your project;
  • The report is due by the end of 7th week (Oct. 7th, 11:59pm).
• Project presentation (40%)
  • 15 minutes presentation (including Q&A) about what you have done for this course project. Format could be tailored according to the nature of the project, e.g., slides presentation and/or system demo;
  • Performance will be graded by both instructor and other students;
  • Presentation will be given in the exam week (Dec. 12th).
• Project report (35%)
  • Detail written report of your project with the required templates;
  • Quality requirement is the same as research papers, i.e., in formal written English and rigorous paper format;
  • Report is due on the last day of this semester (Dec. 15th).

The official rubric for the final report and rubric for the project presentation is provided here.

Note that you are required to use the provided templates for your project proposal and final report. See the Materials page for the template and example file.

What to Work on?

When picking a topic, try to ask yourself the following questions:

• What is exactly the (research) problem that you want to solve? Will it matter if nobody realizes this problem?
• What kind of changes could your project make to the others?
• Is there any existing alternative? If so, why do you still want to do it? How is your idea different from theirs? Would people appreciate about the difference?
• What would be the major challenge(s) in this problem? Any specific background or resource you have to solve the identified problem?
• What is the minimum goal to be achieved during this semester? (Try to drop everything non-essential and only keep the part that is truly novel.)
• How do you plan to demonstrate that method to be developed is indeed solving the pain? Empirical experimentation and/or demo are required, unless you are doing a purely theoretic work.

Before making your decision, it is very important to be aware of whether the problem you would like to tackle has already been solved. If so, you may want to figure out where exactly your novelty is and whether novelty leads to any benefit to others. Your goal is to go beyond, rather than simply duplicate, the existing work.

Leveraging existing resources is especially encouraged as it allows you to minimize the amount of work that you have to do and focus on developing truly your ideas.

Tell us about it!

At the end of the semester, each project team is expected to present their project in class. The purpose of this presentation is

• Let you know about others’ projects.
• Give you some opportunity to practice presentation skills, which are very important for a successful career in both academia and industry.
• Obtain feedback from others about your project.

In general, the structure of your presentation should be prepared like a conference presentation. So it should touch all the following aspects (text in parenthesis states the instructor’s expectation):

• What is the background/motivation of your work? What research question will you address? (Learn how to attract public attention.)
• Why is this problem important? (Learn how to persuade others.)
• Is there any existing work? How novel is yours? (Learn how to sell your ideas.)
• How did you solve this problem? (Learn how to deliver your solution.)
• How good was your method? (Learn how to quantitatively/qualitatively evaluate your work.)
• Any ideas for further improvement? (Learn how to look ahead.)

To make it even more concrete, here how we are going to evaluate your presentation.

We will follow the schedule below to carry out project presentations. The presentation is scheduled on December 12th, Monday, 10:30am-1:00pm EST, 2022, at Rice 340.

Tell me your Story

You should write your report as if you were writing a regular research paper. You should address the same questions as those you have addressed in the proposal and presentation, only with more details. Pay special attention to the challenges that you have solved and your detailed solutions. Basic sections to be included in the report should be the same as those in a conference paper, e.g., abstract, introduction, related work, method, experiment and conclusion. If you are developing a demo system or toolkit, your report can follow the format of a demo paper.

The project report must be at most eight pages with that format (no minimal requirement, as long as you feel it is sufficient to prove the merit of your work).

And this is how I am going to read it,