Tools/libraries such as FsCheck (for .NET), fast-check (for JavaScript) and ofcourse QuickCheck (for Haskell) are aimed at allowing us to write property based tests. In truth we can write property based test with existing tools, what these tend to offer is frameworks with the ability to execute multiple tests and generate data for our tests. So with this in mind, what is property based testing?
What is property based testing?
When we use the word property, we’re not using it like you might in C#, F#, Delphi etc. where it’s a way of accessing fields on an object. Instead we’re really saying is “what are the properties that make this function what it is or do what it does”, when I say “do what it does” I don’t mean from an implementation point.
I’m going to use a canonical example (and I suggest you watch the youtube video The lazy programmer’s guide to writing thousands of tests – Scott Wlaschin for a far better explanation of property based testing using similar examples and a far more complete description of property based testing than I’m going to list here.
Let’s take a simple function, the Add function. Now we want to test our implementation of the add function so how do we do this? Well, we might write something like
[Test]
public void Add_TwoAndFour()
{
var result = MyMath.Add(2, 4);
Assert.That(result, Is.EqualTo(6));
}
This is fine, but what we write out Add implementation top allow this test to pass, theoretically (I know nobody, or I hope nobody would do this) we might have an implementation which looks like this
public class MyMath
{
public static int Add(int a, int b) => 6;
}
Now our unit test will pass. Ofcourse this is a ridiculous implementation, but we’ve just demonstrated that the method passes the supplied unit test but certainly isn’t a valid implementation of an Add function. I mean what if we might add another unit test
[Test]
public void Add_FourAndFour()
{
var result = MyMath.Add(4, 4);
Assert.That(result, Is.EqualTo(8));
}
Our test now fails, but what if our mad implementation simply changes to look like this
public static int Add(int a, int b) => (a, b) switch
{
(2, 4) => 6,
(4, 4) => 8,
_ => 0
};
You get the idea, our Add method is passing the test each time by us adding code to pass each test.
If we change our test to supply multiple random values, this will stop our implementation passing our tests by adding a case for every possible test scenario, but now we hit another interesting question. How do we define success?
If we assume that x and y in the following code are supplied via some random number generator. Then we could simply write the implementation, as per the Add function, within our test, like this
[TestRandomData(Max = 100)]
public void Add_XAndY(int x, int y)
{
var result = MyMath.Add(x, y);
var expected = x + y;
Assert.That(result, Is.EqualTo(expected));
}
This would work (assuming we had a TestRandomDataAttribute which took a Max value to supply 100 iterations of random pairs of data.
But there’s a problem, we’re basically having to write the implementation of the method in the unit test, to prove the method worked. Ofcourse it’ll work, the unit test implementation matches what would be the implementation of Add, i.e.
public static int Add(int a, int b) => a + b;
This is not really helping us much!
Note: It’s not unusual to use alternate implementations of code to prove a function works, for example we define a distributed sorting algorithm, so we could test it against a standard built-in sort, but this has limitations of alternate and fully tested implementations do not exist.
What we really need to be doing is testing the properties of the Add method. In other words, what can we use to determine whether our Add method worked apart from using the same implementation code. What are the properties of addition?
What are the properties of addition?
What can we say about properties of addition, let’s look at the Khan Academy definitions for properties of addition.
One of the obvious features of addition is that we can swap the inputs and the output should remain the same, this is known as the Commutative property. Hence, we could now write a test like this
[TestRandomData(Max = 100)]
public void Add_Commutative(int x, int y)
{
var a = MyMath.Add(x, y);
var b = MyMath.Add(y, x);
Assert.That(a, Is.EqualTo(b));
}
Now that we’re testing whether the method is commutative we also no longer care about the numbers being used or the results of those numbers used, we simply care that x + y = y + x.
Ofcourse you may say, “well that’s great but x * y = y * x also holds true and is therefore commutative”. You’d be correct just using this single property based test would also suggest a function Multiply is the same as Add. Hence we need to add more tests to prove more properties. In fact, it’s likely that property based testing will need multiple property based tests to truly define any uniqueness to a function in scenarios like this.
The next property of addition is the Associative property which says that (x + y) + z = x + (y + z). We can therefore write a property test that looks like
[TestRandomData(Max = 100)]
public void Add_Associative(int x, int y, int z)
{
var a = MyMath.Add(add(x, y), z);
var b = MyMath.Add(x, add(y, z));
Assert.That(a, Is.EqualTo(b));
}
and finally from the Properties of addition we look at an identity property based test, i.e. 0 + x = x and we could write something like this
[TestRandomData(Max = 100)]
public void Add_Associative(int x)
{
var y = MyMath.Add(0, x);
Assert.That(x, Is.EqualTo(y));
}
Now these three property based tests tell us the Add method adheres to the expected properties of addition. If we compare the properties to multiplication, we know that the identity test fails on Multiply as 0 * x = 0, however the other two tests pass – so again you see why multiple property based tests are required.
What’s next?
Ofcourse this demonstrates property based testing on a pretty simple function, more complicated functions require the developer to try to view the function, not so much about its inputs and outputs and values or the likes that are expected, but instead think about what properties make that function unique. This is not always easy.
Let’s look at some other examples of how we might use property based testing…
If we have code that reverses a string we can actually reverse a string then reverse again and see if it results in the original list. Now you’d be correct in thinking that this is not a great test as, if the reverse function does nothing, then reversing the reversed string will be the same as just reversing the string – so property based testing does not exclude the need for standard unit tests with known and expected values.
We could also use the Test oracle whereby we have a different implementation to an existing method that we then check that both output the same value.