Fall 2018 — Syllabus

The true subject matter of the tester is not testing, but the design of test cases —Jeff Offutt

Contact information

Instructor: Upsorn Praphamontripong
Office: Rice Hall 206
Office hours: Wednesday 10:00am-12:00pm, Thursday 2:00pm-3:00pm

TA: DJ Anderson, Office hours: Monday 6:30pm-8:00pm Rice 504, Wednesday 6:30pm-8:00pm Olsson 018
TA: Yiding Wen, Office hours: Monday and Wednesday 2:30pm-4:00pm Olsson 018

Class hours

Tuesday, Thursday 12:30PM - 1:45PM, Thornton Hall E303

Overview

Software is everywhere. Software defines behavior of services such as financial engines, transportation and control systems, and medical services. Software is an essential component of embedded applications such as airplanes, spaceships, cars, household appliances, and mobile phones. Society's reliance on software applications places a high demand on their reliability; knowledge of software testing is becoming necessary for all software developers and software engineers. Although many factors affect the engineering of reliable software, including careful design and sound process management, testing is the primary way industry evaluates software during development.

Testing is the most time consuming and expensive part of software development. Not testing is even more expensive. Inappropriate or inadequate testing can cause unexpected behavior, catastrophic software failure, monetary loss, and loss of life.

This course will help you see how you can easily apply the concepts to any software testing situation and how testing is more efficient and effective by using a classical engineering approach. This course presents the concepts and techniques for testing software and assuring its quality. Topics cover software testing at the unit, module, subsystem, and system levels, automatic and manual techniques for generating and validating test data, the testing process, static vs. dynamic analysis, functional testing, inspections, web application testing, and reliability assessment.

Over the semester, you will learn when, where, and how to test software appropriately and effectively and have hands-on experience with test generation and test automation.

Everyone can be successful in this course and my hope is that developing an understanding of the field of software testing will have positive impact on your software-related profession.

Topics to be covered in Fall 2018

CS 4501 vs CS 6501: Homework assignments and exams in this course will be given as problem solving questions drawn from the Introduction to Software Testing textbook (by Ammann and Offutt) and other assigned reading materials. Each problem is defined by its level of difficulty - fundamental, intermediate, and advanced. Graduate students will be required to solve advanced problems. Undergraduates will be expected to solve intermediate problems and those who also choose to solve advanced problems will be given extra credit opportunities. Extra credit opportunities will be offered to those who choose to complete a short research paper (this is not a showcase option #2); please make an appointment with me to discuss your research interests.

Learning outcomes

  • Knowledge of quantitative, technical, practical methods that software engineers and developers can use to test their software
  • Testing techniques and criteria for all phases of software development - unit (developer) testing, integration testing, system testing, etc.
  • Theoretical and practical knowledge of how to apply test criteria to improve the quality of software
  • Understanding of best quantitative programming and design practices for ensuring software can be efficiently and effectively modified and tested
  • Understanding that maintainability and testability are more important than efficiency for almost all modern software projects

Content

This course has two closely related themes. First, more than half the effort in software development is devoted to activities related to testing, including test design, execution and evaluation. In this course, you will learn quantitative, technical, practical methods that software engineers and developers can use to test their software, both during and at the end of development. Second, more than half of software development effort is not new development, but maintenance activities such as adding new features, correcting problems, migrating to new platforms, and integrating third-party components into new projects. These two themes are intertwined because much of the effort during maintenance is testing the changes, and much of the effort in testing is about evaluating changes.

This course covers these two themes quantitatively, with a solid basis in theory and with practical applications. These topics will be useful to strong programmers in the Computer Science program, as well as engineers, physical scientists, and mathematicians who regularly integrate software components as part of their work. The topic of this course is of interest to and accessible to students in a wide variety of specializations.

Prerequisite

We will generate tests from mathematical models of the software. You will need knowledge of discrete math (sets, graphs, logic, and grammars), programming, data representation, and general knowledge of software engineering. You will need an understanding of the syntax and semantics of multiple programming languages. Most examples will be in Java and some assignments will require automated tests. Programming skills are required and you are expected to know Java or learn on your own.

Text

Required: Introduction to Software Testing (2nd edition), Paul Ammann and Jeff Offutt, Cambridge University Press, 2016. Book website (solution manual)

Where appropriate, additional references that can be relevant for further reading will be provided.

Reading

You are expected to read the relevant material (from the text, various sources on the web, and transparencies that will be made available on the website). You will understand the lectures much better if you read the material before the lectures. The lectures may not cover everything in the readings and will often include material not found in the readings.

Learning activities

To facilitate your learning process, the course structure consists of five main activities: in-class exercises, homework assignments, showcase, quizzes, and final exam.

1. In-class exercises

Three main purposes of this activity are to help you to (i) understand the underlying concepts and apply them to test software, (ii) introduce specific concepts, technologies, tools and frameworks, and (iii) get ready to work on homework assignments.

You are encouraged to work in small groups, allowing peer learning and imitating industrial software development and quality assurance scenarios. Each exercise is designed to get you to think about some specific aspect of the material we are learning that day.

Throughout the course, you will have multiple opportunities to explore test generation tools and test automation frameworks and share your opinion through discussion and writing.

At the end of some classes, we will have a "five-minute paper." You will be asked to respond in written form to a simple question related to the day's discussion. This is to help you to assess your understanding and allow me to clarify any doubts you might have.

We will do in-class exercises during most Thursdays and some Tuesdays. Most will be done in small groups and a few may be individual exercises. They will be graded on a "done / not done" basis. Credit can only be received if done in class, although if you miss class, you should do the posted exercise on your own to learn the material. You are allowed to miss up to two (out of twelve) in-class exercises. The 10 in-class exercises will be used to calculate the final grade.

2. Homework assignments

Homework assignments are the main mechanism to help you learn the fundamentals, apply the concepts, and experience the software testing process. Some homework assignments will be written and some will require programming. Most will allow collaboration with one partner. Assignments will be posted on the class website before class and discussed in class. Some homework assignments will be submitted on paper in class. Others must be submitted to UVa Collab. More details on submission guidelines will be provided on the assignment pages.

Because each homework helps you develop competency in basic and advanced concepts, it is important that you manage your time wisely. All homework assignments must be submitted before class on the day they are due. Any submission received after this time will be considered to be late. If you finish an assignment late, you must inform us (instructor and TAs) by email when it is ready to be graded. Changing an assignment after the due date without prior permission will be treated as a late submission.

You are expected to have a backup disk (or storage somewhere) for every assignment you turn in. In this way, if there is any problem with the copy that is handed in the backup can be used for grading purposes.

3. Showcase

A showcase will provide you the opportunity to explore and experience software testing in practice. There are three options, please choose only one option: (1) test generation project — to provide you the opportunity to design and automate tests for a software project, (2) position paper — to expose you to advanced topics and challenges in software testing and give you the opportunity to learn more about an area of your interest, and (3) digital media — to explore aspects of software testing that interest you and then to produce a 5-7 minute movie, photo story, or animation that creatively teaches something about software testing. You may work on the showcase individually or with one partner. Descriptions will be provided on the showcase page.

4. Quizzes

We will have weekly quizzes and no midterm exam. The goals of the quizzes are to reinforce the concepts in recent classes and assess your understanding of recent topics/skills. This also gives me an opportunity to work with you on any particular topics/skills that can be improved.

Quizzes will generally cover material from previous classes. They will be given during the first 10-15 minutes of class on most Tuesdays and some Thursdays and no make-up or late quizzes will be given. You are allowed to miss up to two (out of twelve) quizzes; the lowest two quiz grades will be dropped. The 10 quizzes with the highest scores will be used to calculate the final grade.

5. Final Exam

Final exam is comprehensive, closed book/notes/web, in class. The only exception is a single sheet of 8 1/2 x 11 paper (single/double sided, single/double spaces, your choice) with hand-written or typed notes by the person taking the exam. Appropriate documentation (as determined by instructor) and requisite permissions (approved by the Dean's Office, in particular) are required for make-up exam requests. Picture IDs are required to take all exams.

If you know in advance that you are unable to make an exam for a valid and unavoidable reason (such as a scheduled surgery, etc), you must notify me at least one week before the scheduled exam date to make arrangements for a make-up, and bring documentation with you when you take the make-up. If you miss an exam due to a university-accepted excused absence (such as an illness or car accident the day and time of the exam), you must notify me within 24 hours of your absence to make arrangements for a make-up, and bring approved documentation with you when you take the make-up exam. Failure to follow either of these policies will result in a zero grade on the exam.


Assessment

The grade you will earn for this course will be based on how well you demonstrate your learning.

Assessment and evaluation

  • In-class exercises: 20% (lowest 2 will be dropped, no make-ups)
  • Homework assignments: 20%
  • Showcase: 10% (choose one of the three options)
  • Quizzes: 30% (lowest 2 will be dropped, no make-ups)
  • Final exam: 20% (comprehensive)
Your overall score will be mapped to a letter grade as follows:
LetterMin
A+ >= 98
A   >= 93
A-  >= 90
B+ >= 87
B   >= 83
B-  >= 80
C+ >= 77
C   >= 73
C-  >= 70
D+ >= 67
D   >= 63
D-  >= 60
F    < 60

By default, grades will not be rounded in this course

Discussion board

We will use Piazza for a discussion board. In addition to in-person assistance, this is the place where you can seek help, offer help, share your thoughts and discoveries, or discuss technical difficulties and potential troubleshooting. Information for accessing our class Piazza site will be provided on the first day of class. Ask all technical questions about the material or the assignments on Piazza.

You may post any reasonable class-related topics.
  • Real-life software failures: Start a discussion about a real-life software failure. Tell us what happened and how it affected users.
  • Application: Describe how one of the ideas from our testing class could have found a real-life software failure before the software was released.
  • Start a thread about an error in the handouts or slides, or about a topic that goes beyond what we discuss in class.
  • Post questions to an appropriate thread and they will be answered by instructor, TAs, or classmates.
  • Give an answer to a classmate's question
Updates to the course web page will generally be noted on the Piazza page.

Collaboration

You are allowed to work on homeworks individually or collaboratively using one of two models:
  • Collaborate while analyzing the problem and developing an answer or solution, then develop the final solution independently. In this model, each person will turn in a separate document. The submissions must include a section that lists everybody you worked with and what each person contributed. You can work with as many classmates as you like with this model, but only other students in this course this semester.
  • Collaborate from start to finish with at most two other students in this course. You must submit one solution and each person will get the same grade. The submission must outline what each person contributed.
Note: You are NOT ALLOWED to include "guest names." Every person listed as a collaborator must contribute. If someone is listed as a collaborator but did not contribute, all will receive a zero grade.

Research

Your class work might be used for research purposes. For example, we may use anonymized student assignments to design algorithms or build tools to help programmers; use data collected from course support tools to better understand and seek to improve student engagement and learning. Any student who wishes to opt out can contact the instructor or TAs to do so. This has no impact on your grade in any manner.

Academic Integrity

The School of Engineering and Applied Science relies upon and cherishes its community of trust. We firmly endorse, uphold, and embrace the University’s Honor principle that students will not lie, cheat, or steal, nor shall they tolerate those who do. We recognize that even one honor infraction can destroy an exemplary reputation that has taken years to build. Acting in a manner consistent with the principles of honor will benefit every member of the community both while enrolled in the Engineering School and in the future.

Students are expected to be familiar with the university honor code, including the section on academic fraud. Each assignment will describe allowed collaborations, and deviations from these will be considered Honor violations. If you are in doubt regarding the requirements, please consult with me before you complete any requirement of this course. Unless otherwise noted, exams and individual assignments will be considered pledged that you have neither given nor received help. (Among other things, this means that you are not allowed to describe problems on an exam to a student who has not taken it yet. You are not allowed to show exam papers to another student or view another student’s exam papers while working on an exam.) Send, receiving or otherwise copying electronic files that are part of course assignments are not allowed collaborations (except for those explicitly allowed in assignment instructions).

Assignments or exams where honor infractions or prohibited collaborations occur will receive a zero grade for that entire assignment or exam, as well as a full letter-grade penalty on the course grade. Such infractions will also be submitted to the Honor Committee if that is appropriate. Students who have had prohibited collaborations may not be allowed to work with partners on remaining homeworks.

SDAC and Other Special Circumstances

The University of Virginia strives to provide accessibility to all students. If you require an accommodation to fully access this course, please contact the Student Disability Access Center (SDAC) at 434-243-5180 or sdac@virginia.edu. If you are unsure if you require an accommodation, or to learn more about their services, you may contact the SDAC at the number above or by visiting their website at http://studenthealth.virginia.edu/student-disability-access-center/faculty-staff.

If you have been identified as an SDAC student, please let the Center know you are taking this class. If you suspect you should be an SDAC student, please schedule an appointment with them for an evaluation. Students who need academic accommodations should see me and contact the SDAC. All academic accommodations must be arranged through the SDAC.

If you have other special circumstances (athletics, other university-related activities, etc.) please contact instructor and/or TA as soon as you know these may affect you in class.

Religious Accommodations

It is the University's long-standing policy and practice to reasonably accommodate students so that they do not experience an adverse academic consequence when sincerely held religious beliefs or observances conflict with academic requirements. Students who wish to request academic accommodation for a religious observance should submit their request in writing directly to me by email as far in advance as possible. Students and instructors who have questions or concerns about academic accommodations for religious observance or religious beliefs may contact the University's Office for Equal Opportunity and Civil Rights (EOCR) at UVAEOCR@virginia.edu or 434-924-3200.

Accommodations do not relieve you of the responsibility for completion of any part of the coursework missed as the result of a religious observance.

Statement on Violence

The University of Virginia is dedicated to providing a safe and equitable learning environment for all students. To that end, it is vital that you know two values that I and the University hold as critically important:

  1. Power-based personal violence will not be tolerated.
  2. Everyone has a responsibility to do their part to maintain a safe community on Grounds.
If you or someone you know has been affected by power-based personal violence, more information can be found on the UVA Sexual Violence website that describes reporting options and resources available - www.virginia.edu/sexualviolence.

As your professor and as a person, know that I care about you and your well-being and stand ready to provide support and resources as I can. As a faculty member, I am a responsible employee, which means that I am required by University policy and federal law to report what you tell me to the University's Title IX Coordinator. The Title IX Coordinator's job is to ensure that the reporting student receives the resources and support that they need, while also reviewing the information presented to determine whether further action is necessary to ensure survivor safety and the safety of the University community. If you would rather keep this information confidential, there are Confidential Employees you can talk to on Grounds (See http://www.virginia.edu/justreportit/confidential_resources.pdf). The worst possible situation would be for you or your friend to remain silent when there are so many here willing and able to help.

This syllabus

This syllabus is to be considered a reference document that can and will be adjusted through the course of the semester to address changing needs. This syllabus can be changed at any time without notification. It is up to the student to monitor this page for any changes. Final authority on any decision in this course rests with the professor, not with this document.


Top Thanks to Professor Jeff Offutt and Professor Paul Ammann (GMU), Professor Mark Sherriff and Professor Sebastian Elbaum (UVA), and Mr. Corey Vaudo and Mr. Conrad Rybka (APT) for the input and material provided for this course.