Mutation Testing vs. Coverage

Given: Pelican's DELETE_OUTPUT_DIRECTORY is set to True (which it is by default)
When: we generate several experiments using the commands:
    AB_EXPERIMENT="control" make regenerate
    AB_EXPERIMENT="123" make regenerate
    AB_EXPERIMENT="xy" make regenerate
    make publish
Actual result: only the "xy" experiment (the last one) would be published online.
And: all of the other contents will be deleted.

Expected result: content from all experiments will be available under the output directory.

$ cosmic-ray run --baseline=10 --test-runner=unittest example.json pelican_ab -- tests/
$ cosmic-ray report example.json 
job ID 29:Outcome.SURVIVED:pelican_ab
command: cosmic-ray worker pelican_ab mutate_comparison_operator 3 unittest -- tests/
--- mutation diff ---
--- a/home/senko/pelican-ab/pelican_ab/__init__.py
+++ b/home/senko/pelican-ab/pelican_ab/__init__.py
@@ -14,7 +14,7 @@
     def __init__(self, output_path, settings=None):
         super(self.__class__, self).__init__(output_path, settings)
         experiment = os.environ.get(jinja_ab._ENV, jinja_ab._ENV_DEFAULT)
-        if (experiment != jinja_ab._ENV_DEFAULT):
+        if (experiment > jinja_ab._ENV_DEFAULT):
             self.output_path = os.path.join(self.output_path, experiment)
             Content.url = property((lambda s: ((experiment + '/') + _orig_content_url.fget(s))))
             URLWrapper.url = property((lambda s: ((experiment + '/') + _orig_urlwrapper_url.fget(s))))

job ID 33:Outcome.SURVIVED:pelican_ab
command: cosmic-ray worker pelican_ab mutate_comparison_operator 7 unittest -- tests/
--- mutation diff ---
--- a/home/senko/pelican-ab/pelican_ab/__init__.py
+++ b/home/senko/pelican-ab/pelican_ab/__init__.py
@@ -14,7 +14,7 @@
     def __init__(self, output_path, settings=None):
         super(self.__class__, self).__init__(output_path, settings)
         experiment = os.environ.get(jinja_ab._ENV, jinja_ab._ENV_DEFAULT)
-        if (experiment != jinja_ab._ENV_DEFAULT):
+        if (experiment not in jinja_ab._ENV_DEFAULT):
             self.output_path = os.path.join(self.output_path, experiment)
             Content.url = property((lambda s: ((experiment + '/') + _orig_content_url.fget(s))))
             URLWrapper.url = property((lambda s: ((experiment + '/') + _orig_urlwrapper_url.fget(s))))

total jobs: 33
complete: 33 (100.00%)
survival rate: 6.06%

     def __init__(self, writePriority=0, *args, **kwargs):
-        KickstartCommand.__init__(self, *args, **kwargs)
+        KickstartCommand.__init__(self, writePriority, *args, **kwargs)
         self.authconfig = kwargs.get("authconfig", "")

-    def __init__(self, writePriority=0, *args, **kwargs):
-        BaseData.__init__(self, writePriority, *args, **kwargs)
+    def __init__(self, *args, **kwargs):
+        BaseData.__init__(self, *args, **kwargs)

atodorov 2:28 PM
Hello everyone, I'd like to kick-off a discussion / interested in what you think about
Rahul Gopinath's talk at GTAC this year. What he argues is that test coverage is still
the best metric for how good a test suite is and that mutation coverage doesn't add much
additional value. His talk is basically the opposite of what @lminozem presented last year
at GTAC. Obviously the community here and especially tools authors will have an opinion on
these two presentations.

tjchambers 12:37 AM
@atodorov I have had the "pleasure" of working on a couple projects lately that illustrate
why LOC test coverage is a misnomer. I am a **strong** proponent of mutation testing so will
declare my bias.

The projects I have worked on have had a mix of test coverage - one about 50% and
another > 90%.

In both cases however there was a significant difference IMO relative to mutation coverage
(which I have more faith in as representative of true tested code).

Critical factors I see when I look at the difference:

- Line length: in both projects the line lengths FAR exceeded visible line lengths that are
"acceptable". Many LONGER lines had inline conditionals at the end, or had ternary operators
and therefore were in fact only 50% or not at all covered, but were "traversed"

- Code Conviction (my term): Most of the code in these projects (Rails applications) had
significant Hash references all of which were declared in "traditional" format hhh[:symbol].
So it was nearly impossible for the code in execution to confirm the expectation of the
existence of a hash entry as would be the case with stronger code such as "hhh.fetch(:symbol)"

- Instance variables abound: As with most of Rails code the number of instance variables
in a controller are extreme. This pattern of reference leaked into all other code as well,
making it nearly impossible with the complex code flow to ascertain proper reference
patterns that ensured the use of the instance variables, so there were numerous cases
of instance variable typos that went unnoticed for years. (edited)

- .save and .update: yes again a Rails issue, but use of these "weak" operations showed again
that although they were traversed, in many cases those method references could be removed
during mutation and the tests would still pass - a clear indication that save or update was
silently failing.

I could go on and on, but the mere traversal of a line of code in Ruby is far from an indication
of anything more than it may be "typed in correctly".

@atodorov Hope that helps.

LOC test coverage is a place to begin - NOT a place to end.

atodorov 1:01 AM
@tjchambers: thanks for your answer. It's too late for me here to read it carefully but
I'll do it tomorrow and ping you back

dkubb 1:13 AM
As a practice mutation testing is less widely used. The tooling is still maturing. Depending on your
language and environment you might have widely different experiences with mutation testing

I have not watched the video, but it is conceivable that someone could try out mutation testing tools
for their language and conclude it doesn’t add very much

mbj 1:14 AM
Yeah, I recall talking with @lminozem here and we identified that the tools she used likely
show high rates of false positives / false coverage (as the tools likely do not protect against
certain types of integration errors)

dkubb 1:15 AM
IME, having done TDD for about 15+ years or so, and mutation testing for about 6 years, I think
when it is done well it can be far superior to using line coverage as a measurement of test quality

mbj 1:16 AM
Any talk pro/against mutation testing must, as the tool basis is not very homogeneous, show a non consistent result.

dkubb 1:16 AM
Like @tjchambers says though, if you have really poor line coverage you’re not going to
get as much of a benefit from mutation testing, since it’s going to be telling you what
you already know — that your project is poorly tested and lots of code is never exercised

mbj 1:19 AM
Thats a good and likely the core point. I consider that mutation testing only makes sense
when aiming for 100% (and this is to my experience not impractical).

tjchambers 1:20 AM
I don't discount the fact that tool quality in any endeavor can bring pro/con judgements
based on particular outcomes

dkubb 1:20 AM
What is really interesting for people is to get to 100% line coverage, and then try mutation
testing. You think you’ve done a good job, but I guarantee mutation testing will find dozens
if not hundreds of untested cases .. even in something with 100% line coverage

To properly evaluate mutation testing, I think this process is required, because you can’t
truly understand how little line coverage gives you in comparison

tjchambers 1:22 AM
But I don't need a tool to tell me that a 250 character line of conditional code that by
itself would be an oversized method AND counts more because there are fewer lines in the
overall percentage contributes to a very foggy sense of coverage.

dkubb 1:22 AM
It would not be unusual for something with 100% line coverage to undergo mutation testing
and actually find out that the tests only kill 60-70% of possible mutations

tjchambers 1:22 AM
@dkubb or less

dkubb 1:23 AM
usually much less :stuck_out_tongue:

it can be really humbling

mbj 1:23 AM
In this discussion you miss that many test suites (unless you have noop detection):
Will show false coverage.

tjchambers 1:23 AM
When I started with mutant on my own project which I developed I had 95% LOC coverage

mbj 1:23 AM
Test suites need to be fixed to comply to mutation testing invariants.

tjchambers 1:23 AM
I had 34% mutation coverage

And that was ignoring the 5% that wasn't covered at all

mbj 1:24 AM
Also if the tool you compare MT with line coverage on: Is not very strong,
the improvement may not be visible.

dkubb 1:24 AM
another nice benefit is that you will become much better at enumerating all
the things you need to do when writing tests

tjchambers 1:24 AM
@dkubb or better yet - when writing code.

The way I look at it - the fewer the alive mutations the better the test,
the fewer the mutations the better the code.

dkubb 1:29 AM
yeah, you can infer a kind of cyclomatic complexity by looking at how many mutations there are

tjchambers 1:31 AM
Even without tests (not recommended) you can judge a lot from the mutations themselves.

I still am an advocate for mutations/LOC metric