Mutation testing for JavaScript
Your team is developing amazing software solutions using one of the most ubiquitous programming languages — JavaScript. The team consists…
Your team is developing amazing software solutions using one of the most ubiquitous programming languages — JavaScript. The team consists of seasoned professionals who practice Test Driven Development (TDD), clean architecture, clean design, clean code. Everything looks great. So, the question is — can we find some room for improvement?
Oftentimes when I get hired to lead/coach teams of software developers, I like to check how do their solutions fare when subjected to stresses introduced by mutation testing. In this article I will briefly discuss how to add mutation testing to your JavaScript/node.js projects.
Who let the cat out?
I will use the same example I’ve used in the original article about mutation testing and will replace the C# solution with the JavaScript solution. To quickly recap, the challenge is to develop an automated system that will control when is the cat trapdoor enabled and when is it locked. The logic guiding that automation depends on the system being able to decide what part of the day constitutes nighttime, and what part of the day constitutes daylight (we don’t want the cat to sneak out during nighttime).
Develop the solution
Start by making a new directory and navigating to it on using CLI. Now do (I’m assuming you have node and npm installed):
npm initAnswer the prompts by specifying that you’ll be using jest as your testing framework, and you got yourself a nifty little package.json file.
Now, in case you don’t have jest installed, do the following on the CLI:
npm install --save-dev jest(you can replace save-dev with global, if you wish)
Now create jest.config.js file (while still on the command line):
touch jest.config.jsOpen the newly created jest.config.js file and add configuration values. Bare bones would be:
module.exports = {
collectCoverage: true,
coverageDirectory: "coverage",
coverageProvider: "v8",
};OK, we’re now ready to start cracking. Our discipline is TDD, so the first thing we do is write a failing test. I like to keep tests separate from the shipping code, so I always create separate directory for the actual code; I tend to call that directory app:
mkdir appNow create the test file (call it whatever you want, but keep in mind that by convention, test file name must end with .test.js).
Add the first failing test; we want to see if the hour of the day denotes daylight:
describe(“When is it daylight?”, () => {
test("'Daylight' denotes hours between 7 am and 6 pm (18 hours)", () => {
let expectedValue = "Daylight";
let actualValue = "Undetermined"
expect(actualValue).toBe(expectedValue);
})
});Now run the test:
npm test
All right, now we know that we have a sentinel in our midst, a sentinel that will always blow the whistle in case our code does not deliver to the specified expectation.
Great, now let’s work on making the test pass. Since our failing test is driving our development, we let the test decide how will it interact with the nascent system (i.e., a system that hasn’t been developed yet). And because the system hasn’t been firmed up yet (i.e., we have no ‘blueprint’ delivered to us), we are free to choose whatever makes sense from the client side (client in this case being any part of the system that is going to collaborate with our yet-to-be-firmed-up API).
Let’s imagine that we have a JavaScript program located in the app directory. We can now christen that JavaScript program by calling it day-or-night-utility.js. We imagine this utility being capable of automatically deciding if it’s daylight or nighttime. Let’s go ahead and in our tests file declare that day-or-night-utility.js file as if it already exists in the app directory (remember, in TDD we always make the first move in the test, never in the shipping app).
const { someFunction } = require(‘./app/day-or-night-utility’);We haven’t decided yet how to call the function that will decide if it’s daylight or nighttime, so for now we simply declare it as someFunction. That function will be found in the ./app/day-or-night-utility file.
If we save our test file and run the test again, we will get this error:
Cannot find module ‘./app/day-or-night-utility’ from ‘tests/dayornight.test.js’Let’s create that file in the app directory now. Going back to our failing test, let’s think of a name that is more descriptive than ‘someFunction’. Perhaps ‘whatPartOfDay’? OK, that’s sound pretty descriptive. refactor the test to include the newly named function:
const { whatPartOfDay } = require(‘./app/day-or-night-utility’);Now add that function to the day-or-night-utility.js file:
function whatPartOfDay() {
return "No idea";
}
module.exports = {
whatPartOfDay
};Now replace the hard-coded value “Undetermined” in our test with the actual call to the whatPartOfDay function in the day-or-night-utility.js file. The test now looks like this:
const { whatPartOfDay } = require('./app/day-or-night-utility');
describe("When is it daylight?", () => {
test("'Daylight' denotes hours between 7 am and 6 pm (18 hours)", () => {
let expectedValue = "Daylight";
let actualValue = whatPartOfDay();
expect(actualValue).toBe(expectedValue);
})
});Run the test and it fails, this time with a different error message (progress!):
Instead of the earlier “Undermined” ( a hard-coded actual value), we now get “No idea” (that’s the hard-coded value returned from our function.
OK, time to make our test pass. Following Kent Beck’s software development model, we are now going to fake-it-till-we-make-it. Meaning, we’re going to make the failing test pass by hard-coding the value returned from the function by making sure the returned hard-coded value matches the expected value in the test. Go back to the function and replace “No idea” with “Daylight”. Save the change and run the test — voila! the test passes now. Congratulations! We’ve moved from the red to the green phase.
What now? Well, as TDD teaches, we now move into the so-called blue phase — refactoring. We need to refactor our crude, Mickey Mouse cheat by replacing the hard-coded return value with some code that’s doing the actual processing!
Let’s get rid of the hard-coded value in the function and add this code instead:
function whatPartOfDay() {
const hour = new Date().getHours();
if (hour >= 7 && hour < 18) {
return "Daylight";
}
return "Nighttime";
}
module.exports = {
whatPartOfDay
};The function now checks for the hour of the day and looks to see if the hour is between 7 am and 6 pm (or, 18 hours military time). If yes, the function returns the value “Daylight”. Otherwise, the function returns the value “Nighttime”.
Looks reasonably logical, no? Run the test, and the test passes.
But check this out — in the “Uncovered Line #s” column in the report, it says that lines 6 and 7 are not covered by the test! What does that mean?
Check the code in the function and we see that line 6 contains the:
return “Nighttime”;statement. What that means is that we don’t have a test to check if it’s nighttime. OK, let’s add that test to our test file:
describe("When is it nighttime?", () => {
test("Expect to get 'Nighttime' for hours between 6 pm and 7 am", () => {
let expectedValue = "Nighttime";
let actualValue = "Undetermined";
expect(actualValue).toBe(expectedValue);
})
});Notice how we cheated again and have enforced the hard-coded value “Undetermined” as the actualValue in this new test? Why are we doing that? The reason is simple — whenever we create a new test, we are always rushing to see it fail. And the quickest way to see it fail is to add incorrect value to the actualValue before the assertion.
And why are we rushing to see a new test fail? We want to make sure we never end up in the false-positive situation.
OK, if we now run tests we’ll see that the second test fails, since it was expecting “Nighttime” but received “Undetermined”. Let’s now replace the hard-coded value in the test with the actual call to the whatPartOfDay function. We run the tests and — our new tests still fails! Why?
Looking at the code in the function we see that it has two problems:
First problem — it is responsible for more than one type of processing (first it must obtain the hour of day, then it must decide if that hour of day falls in the daylight or nighttime). A function should never have more than one responsibility.
Secondly, this function depends on the hidden input; in this case, a system clock. Any time our code depends on the hidden input, it becomes untestable.
Depending on what hour of the day we are running these tests, one of them will pass and the other fail. It is impossible to make both tests pass at the same time. That situation renders our system nondeterministic. Which is super bad.
How to fix nondeterministic system?
The only way to fix it is to refactor our function. We need to slim it down. Let’s extract one of its responsibilities — obtaining the hour of the day, and keep only one responsibility — deciding if it’s daylight or nighttime. How do we do that?
We inject the hour of day as a value into the function. We do that by specifying the argument that the function will accept. Let’s call it something intuitive — how about ‘hour’? Makes sense?
So, the refactored function now accepts the hour:
function whatPartOfDay(hour) {
if (hour >= 7 && hour < 18) {
return "Daylight";
}
return "Nighttime";
}
module.exports = {
whatPartOfDay
};Notice how we’ve removed the code that was responsible for obtaining the hour of the day:
const hour = new Date().getHours();The tests are now injecting the hours as values; for example, a daylightHour would be, say, 8 am, and a nighttimeHour would be, say, 20 hours (military time, which is 8 pm):
const { whatPartOfDay } = require('./app/day-or-night-utility');
describe("When is it daylight?", () => {
test("Expect to get 'Daylight' for hours between 7 am and 6 pm (18 hours)", () => {
let expectedValue = "Daylight";
let daylightHour = 8;
let actualValue = whatPartOfDay(daylightHour);
expect(actualValue).toBe(expectedValue);
})
});
describe("When is it nighttime?", () => {
test("Expect to get 'Nighttime' for hours between 6 pm and 7 am", () => {
let expectedValue = "Nighttime";
let nighttimeHour = 20;
let actualValue = whatPartOfDay(nighttimeHour);
expect(actualValue).toBe(expectedValue);
})
});Run the tests now, everything passes (2 passed), and there are no more “Uncovered Line #s”. Yay!
We’re now rock solid. Or, are we?
It would appear that our solution is solid, because all tests pass and the system is fully covered. Still, there could be some room for improvement, as I always like to say. Yes, but what else could we do to improve this solution?
I always like to try mutation testing once we get to the milestone where all tests are passing and the code is fully covered. Let’s do it now!
We will be using community-based mutation tool called Stryker. Head over to their site and install the mutation testing tool:
npm i --save-dev @stryker-mutator/jest-runnerMake sure to place stryker.conf.js file in the root directory.
The simplest bare bones configuration for Stryker could be:
module.exports = {
mutate: ['./app/*.js'],
coverageAnalysis: 'off'
};You’re now ready to run mutation testing:
npx stryker runStryker with mutate all statements found in files with the extension .js in the app directory. Each time Stryker mutates one statement, it runs all tests to check if any tests complain. If at least one test complains, that means our tests have eliminated that mutant. If all tests remain silent, the mutant has survived. Which is not a good thing.
Here is what Stryker found after mutating our function:
Stryker successfully killed 11 mutants, however, we see that 3 mutants have survived. What happened?
Let’s look at one mutant that survived:
The statement on line 2 in the day-or-night-utility.js was mutated from:
if (hour >= 7 && hour < 18) {to
if (hour > 7 && hour < 18) {After that change in the conditional logic, Stryker ran all tests and no test complained. But at least one test should complain, because hey, there was a change in the conditional logic.
What that means is that our code is not actually fully covered. We are missing one test! Our system can deal with situations when the hour is greater than 7 and less than 18; however, our system doesn’t know what to do when the hour is exactly 7.
Let’s add the new test now (for brevity sake, I will skip the regular red phase to observe the test fail; I will leave that exercise to the reader):
describe("It's already Daylight when the hour is 7 am", () => {
test('expect to get "Daylight" when the hour is 7 am', () => {
let expectedValue = "Daylight";
let daylightHour = 7;
let actualValue = whatPartOfDay(daylightHour);
expect(actualValue).toBe(expectedValue);
})Run Stryker again, and lo ad behold, the mutant is now killed!
We see from the report that our tests have now killed 12 mutants and only 2 have survived. The other equality operator mutant that survived is:
Stryker mutated the same statement as previously, only this time it modified the latter part of the conditional logic. It changed it from:
if (hour >= 7 && hour < 18) {to
if (hour >= 7 && hour <= 18) {Again, we see that our tests know how to deal with situations when the hour is exactly 7 o’clock, and also with any hour between 7 o’clock and less than 18 hours (6 pm). But it does not know what to decide when it’s exactly 18 hours. Time for new test:
describe("It's already Nighttime at 6 pm", () => {
test('expect to get "Nighttime" when the hour is 18 hours (6 pm)', () => {
let expectedValue = "Nighttime";
let nighttimeHour = 18;
let actualValue = whatPartOfDay(nighttimeHour);
expect(actualValue).toBe(expectedValue);
})
});Bam! That new test killed another surviving mutant!
I’ll leave it to the reader to continue growing the system, but by now it should be clear what value mutation testing brings to the table.
Thanks for joining me on this journey!