Software Testing

Activity: Program mutation (CountWords)

(no submission)

Purpose: Understand and apply mutation testing concept to program source code; get ready to work on homework assignment, and prepare for quiz 5 and the final exam.

You may make a copy of a worksheet and complete this activity or type your answers in any text editor.

You may work alone or with at most two other students in this course.

Consider the following count() method of the muCountWords class.

public class muCountWords 
{
     /**
      * Counts the number of words in a string that end with either "r" or "t"	
      * @param str -- an incoming sentence
      * @return word_count -- the number of words
      */
      
line  1:   public static int count(String str) 
line  2:   {
line  3:      int word_count = 0;    
line  4:      char last = ' ';      
line  5:    
line  6:      for (int i=0; i<str.length(); i++)
line  7:      {
line  8:         if ( !Character.isLetter(str.charAt(i)) && (last == 'r' || last == 't') )
line  9:            word_count++;
line 10:
line 11:         last = str.charAt(i);
line 12:      }
line 13:      
line 14:      if ( last == 'r' || last == 't' )
line 15:         word_count++;
line 16:      
line 17:      return word_count;       
line 18:   }

}

Create at least one mutation operator that is applicable (and meaningful) to the count() method.
- Write down the name of your mutation operator(s)
- For each operator, briefly describe what it does or how it mutates the program source code
Assume we are considering first-order mutation testing; i.e., only one change per mutant.
```
Sample solution:

Conditional Operator Replacement (COR) replaces each occurrence of
each logical operator (&&, ||, falseOp – replace the entire condition by false, 
and trueOp – replace the entire condition by true) by each of the other operators. 
```

Apply your mutation operator(s) to the count() method. How many mutants does your operator create? For each of your mutation operator(s), Write down all mutants generated by the operator.

Sample solution:

Applying COR to this method will generate 9 mutants:
                 
original: line 8: if ( !Character.isLetter(str.charAt(i)) && (last == 'r' || last == 't') )
  mutant# 1: line 8: if ( !Character.isLetter(str.charAt(i)) || (last == 'r' || last == 't') )
  mutant# 2: line 8: if ( false )
  mutant# 3: line 8: if ( true )
  mutant# 4: line 8: if ( !Character.isLetter(str.charAt(i)) && (last == 'r' && last == 't') )
  mutant# 5: line 8: if ( !Character.isLetter(str.charAt(i)) && false )
  mutant# 6: line 8: if ( !Character.isLetter(str.charAt(i)) && true ) 

original: line 14: if ( last == 'r' || last == 't' )
  mutant# 7: line 14: if ( last == 'r' && last == 't' )
  mutant# 8: line 14: if ( false )
  mutant# 9: line 14: if ( true )

Are any of your mutants "equivalent"?

Sample solution: 

For these COR mutants, none of them are equivalent. 
There exist test cases that kill them.

Are any of your mutants redundant?

Sample solution:

Mutants 2, 4, 5 are redundant. 
Mutants 7 and 8 are redundant.

Design a set of tests to kill the mutants. Try to satisfy Strong Mutation Coverage.

Sample solution:

input: str = "car was cat"
original: output = 2
   mutant 1: output = 3
   mutant 2: output = 1 (and mutant 4, mutant 5)
   mutant 3: output = 12
   mutant 6: output = 3
   mutant 7: output = 1 (and mutant 8)

input: str = "car was"    
original: output = 1
   mutant 2: output = 0 (and mutant 4, mutant 5)
   mutant 3: output = 7
   mutant 9: output = 2

Download the subject under test (muCountWords.java) and create mutants using your operator(s). For simplicity (no additional software installation / configuration), let's manually mutate the program source code (i.e., inject the fault(s)).

Transform your abstract tests into JUnit test code.

The following contains JUnit test methods. Be sure to properly set up your JUnit test file.  
Note: in this sample solution, count_org() is the original code, count() is the mutated code. 

@Test
void test_1R1T()     // killed mutants 1,2,3,4,5,6,7,8
{
   int words_count = new muCountWords().count("car was cat");
   int words_count_org = new muCountWords().count_org("car was cat");
   assertTrue(words_count==words_count_org, "compare mutant_output and org_program_output");
}   
   
@Test
void test_1R0T()    // killed mutants 2,3,4,5,9  -- need this test to kill mutant 9
{
   int words_count = new muCountWords().count("car was");
   int words_count_org = new muCountWords().count_org("car was");
   assertTrue(words_count==words_count_org, "compare mutant_output and org_program_output");
}

Execute your JUnit tests on the original program. Record the results.
Execute your JUnit tests on the mutants. Compare the results of running the tests on the mutants and the results of running the tests on the original program. Record the number of mutants killed by your set of tests.
Compute the mutation score of your set of tests. If your set of tests does not achieve a 100% mutation score, design more tests. If it is impossible to achieve a 100% mutation score, analyze and discuss why not.
```
Mutation score = #killed mutants / #non-equivalent mutants 
              = 9/9 = 1.0    (thus, 100%)  
```

For more hands-on experience, you may explore mutation testing tools such as PIT, Major, or a traditional mutation testing tool, muJava. You will need to set up the mutation testing framework, upload the subject under test, apply the pre-defined mutation operators, upload and execute your JUnit tests, and analyze the results/reports.