Kill All Mutants! (Intro to Mutation Testing)

By Dave Aronson

Elevator Pitch

Is all your code meaningful, or is some redundant or unreachable? Are your tests strict, or do they let code get away with undesired behavior? Mutation Testing helps root out these problems. This talk explains what Mutation Testing is, how it works, how to use it, and what to do about the problems.

Description

How good are your tests? Would they still pass if the tested code was changed? If so, there may be problems with your code, your tests, or both!

Mutation Testing helps reveal these cases. It your tests against lots of slightly altered versions, called “mutants”, of your code. If any lets all the tests pass, that implies certain flaws. Your code might not be meaningful enough that a slight mutation would change the behavior, your tests might not be strict enough to catch the difference that the mutation made, or both!

This talk will tell you what mutation testing is, its benefits and drawbacks, how to use it, and how it works under the hood. There will be examples, and a list of tools for many popular languages and platforms.

You will come away equipped with a powerful new technique for making sure your tests are strict and your code is meaningful.

Notes

I’ve picked Advanced level, but might also be suitable for some intermediates. Juniors might not grok the concepts, or even the underlying stuff about testing in the first place. Mainly, they need to be already comfortable with unit testing. Ideally they should also be familiar with more advanced concepts like an Abstract Syntax Tree. If you mean the level of their knowledge within the topic, then it’s more introductory. The advanced-ness is more about what to do about the problems that mutation testing finds, and that is mainly a matter of experience.

I have done this talk before. Videos include https://www.youtube.com/watch?v=9GId6mFL0_c (from RubyConf 2019), https://www.youtube.com/watch?v=yNMBOj7JUPs (from JSConf Hawai’i 2020), and https://www.youtube.com/watch?v=vQrnVD6CbxU (from NDC Sydney 2020).

I’m self-employed, so having my company cover it is a moot point, it would still be out of my own pocket.

Rough agenda: - Benefits and Drawbacks - Why Test Coverage is Not Enough - General Concept - How it Works (High Level) - How it Works (Low Level) - Example: Finding Bad Test and Recommended Fix (Simple) - Example: Finding Bad Test and Recommended Fix (Complex) - Example: Finding Redundant Code and Recommended Fix (Complex) - FAQs: History, Why so CPU-Intensive, and Why Change Mutants in Only One Way Each - List of Tools - Conclusion