Property-based testing (with a sprinkle of JavaScript)

If you come from OO and have ever tested any piece of software chances are you’ve employed example-based testing. In other words, the function under test is called with some predefined input and the return value is checked against some predefined expectation. For instance, the function is called with and and the expected result is . sum 1 2 3 There are problems with this approach though. Firstly, only a limited amount of examples can be checked. Secondly, it’s a human being deciding what input-output to verify. This, unfortunately, reflects the developer’s bias. Therefore there’s a big chance some critical input is left unchecked. But how is it possible to have expectations on the return value of a function if the input is not chosen by the developer? Turns out, it’s possible to derive invariants (properties) out of the code that are always true, no matter what the input is. If you mix that with random values generators you get property-based testing. In the case of , we would use a number generator to produce the input. Then, we would derive some properties to test such as commutativity (i.e. ) and associativity (i.e. ). Lastly, the property-based testing framework would generate some sets of , and and check multiple times if the properties hold. sum x + y == y + x (x + y) + z == x + (y + z) x y z In presence of a failure, we would get back the first set of , and for which the property didn’t hold. x y z The one is a simple example. But we could imagine something more complex that involved other function calls and less trivial input. In that case, just getting the input set that failed could be not enough. sum That’s why most property-based testing frameworks provide a feature called shrinking. In other words, after a failure, the framework tries to shrink the input that made the property fail to its simplest form by removing or simplifying input data. Show me the code Let’s apply property-based testing to one of those stupid examples you are never going to see in real life. I’m going to use JavaScript and . JSVerify Let’s say we have the following code which needs tests: const div = (dividend, divisor) => dividend / divisor Easy, right? The following example-based tests are green so we can call it a day! describe('div', () => {it('with natural numbers', () => {const expected = 2 const actual = div(6, 3) assert.strictEqual(actual, expected) }) it('with decimal numbers', () => {const expected = 2 const actual = div(6.3, 3.15) assert.strictEqual(actual, expected) })}) Well, not really. Let’s see what happens with property-based tests. Firstly, as the generator we can use which returns natural numbers. jsc.nat Secondly, as the properties let’s just check the “right-distributive” one (i.e. ). (n1 + n2) / n3 == (n1 / n3) + (n2 / n3) describe('div', () => {const naturalNumber = jsc.nat jsc.property('is right-distributive',naturalNumber, naturalNumber, naturalNumber,(n1, n2, n3) => div(n1 + n2, n3) === div(n1, n3) + div(n2, n3))}) BOOM! Error: Failed after 4 tests and 4 shrinks. rngState: 08b51479f83d6a20ec; Counterexample: 0; 0; 0; With , and something went wrong. From the node REPL n1 = 0 n2 = 0 n3 = 0 div(0 + 0, 0) === div(0, 0) + div(0, 0)// false div(0 + 0, 0)// NaN div(0, 0) + div(0, 0)// NaN NaN === NaN// false JavaScript, right? If we run the property-based test again Error: Failed after 4 tests and 4 shrinks. rngState: 08b51479f83d6a20ec; Counterexample: 2; 32; 3; BOOM AGAIN. But this time it’s a different failure. From the node REPL div(2 + 32, 3) === div(2, 3) + div(32, 3)// false div(2 + 32, 3)// 11.333333333333334 div(2, 3) + div(32, 3)// 11.333333333333332 JavaScript and floating point arithmetic, right? Now, if property-based testing found out 2 bugs in 2 runs out of const div = (dividend, divisor) => dividend / divisor imagine what else it can find out of more complex code. Outro Property-based testing takes away bias. That way it enables discovering bugs the developer didn’t think to test for. Also, it forces to consider the code from yet another point of view, which is a good thing. At the same time, properties are somewhat more abstract than examples (e.g. commutativity in sum vs ). That’s why mixing the two styles is prolly the best idea. sum(1, 2) == 3 Hungry for more? Check out I use property-based TDD in the . follow up post Pointers [video] by John Hughes Don’t Write Tests [video] by Scott Wlaschin The lazy programmer’s guide to writing 1000’s of tests: An introduction to property based testing [paper] by David Saff and Marat Boshernitsan The Practice of Theories: Adding “For-all”Statements to “There-Exists” Tests