Category Archives: Property based testing

Property based testing

Tools/libraries such as FsCheck (for .NET), fast-check (for JavaScript) and ofcourse QuickCheck (for Haskell) are aimed at allowing us to write property based tests. In truth we can write property based test with existing tools, what these tend to offer is frameworks with the ability to execute multiple tests and generate data for our tests. So with this in mind, what is property based testing?

What is property based testing?

When we use the word property, we’re not using it like you might in C#, F#, Delphi etc. where it’s a way of accessing fields on an object. Instead we’re really saying is “what are the properties that make this function what it is or do what it does”, when I say “do what it does” I don’t mean from an implementation point.

I’m going to use a canonical example (and I suggest you watch the youtube video The lazy programmer’s guide to writing thousands of tests – Scott Wlaschin for a far better explanation of property based testing using similar examples and a far more complete description of property based testing than I’m going to list here.

Let’s take a simple function, the Add function. Now we want to test our implementation of the add function so how do we do this? Well, we might write something like

[Test]
public void Add_TwoAndFour()
{
   var result = MyMath.Add(2, 4);
   Assert.That(result, Is.EqualTo(6));
}

This is fine, but what we write out Add implementation top allow this test to pass, theoretically (I know nobody, or I hope nobody would do this) we might have an implementation which looks like this

public class MyMath
{
   public static int Add(int a, int b) => 6;
}

Now our unit test will pass. Ofcourse this is a ridiculous implementation, but we’ve just demonstrated that the method passes the supplied unit test but certainly isn’t a valid implementation of an Add function. I mean what if we might add another unit test

[Test]
public void Add_FourAndFour()
{
   var result = MyMath.Add(4, 4);
   Assert.That(result, Is.EqualTo(8));
}

Our test now fails, but what if our mad implementation simply changes to look like this

public static int Add(int a, int b) => (a, b) switch
{
   (2, 4) => 6,
   (4, 4) => 8,
   _ => 0
};

You get the idea, our Add method is passing the test each time by us adding code to pass each test.

If we change our test to supply multiple random values, this will stop our implementation passing our tests by adding a case for every possible test scenario, but now we hit another interesting question. How do we define success?

If we assume that x and y in the following code are supplied via some random number generator. Then we could simply write the implementation, as per the Add function, within our test, like this

[TestRandomData(Max = 100)]
public void Add_XAndY(int x, int y)
{
   var result = MyMath.Add(x, y);
   var expected = x + y;
   Assert.That(result, Is.EqualTo(expected));
}

This would work (assuming we had a TestRandomDataAttribute which took a Max value to supply 100 iterations of random pairs of data.

But there’s a problem, we’re basically having to write the implementation of the method in the unit test, to prove the method worked. Ofcourse it’ll work, the unit test implementation matches what would be the implementation of Add, i.e. 

public static int Add(int a, int b) => a + b;

This is not really helping us much!

Note: It’s not unusual to use alternate implementations of code to prove a function works, for example we define a distributed sorting algorithm, so we could test it against a standard built-in sort, but this has limitations of alternate and fully tested implementations do not exist.

What we really need to be doing is testing the properties of the Add method. In other words, what can we use to determine whether our Add method worked apart from using the same implementation code. What are the properties of addition?

What are the properties of addition?

What can we say about properties of addition, let’s look at the Khan Academy definitions for properties of addition.

One of the obvious features of addition is that we can swap the inputs and the output should remain the same, this is known as the Commutative property. Hence, we could now write a test like this

[TestRandomData(Max = 100)]
public void Add_Commutative(int x, int y)
{
   var a = MyMath.Add(x, y);
   var b = MyMath.Add(y, x);
   Assert.That(a, Is.EqualTo(b));
}

Now that we’re testing whether the method is commutative we also no longer care about the numbers being used or the results of those numbers used, we simply care that x + y = y + x.

Ofcourse you may say, “well that’s great but x * y = y * x also holds true and is therefore commutative”. You’d be correct just using this single property based test would also suggest a function Multiply is the same as Add. Hence we need to add more tests to prove more properties. In fact, it’s likely that property based testing will need multiple property based tests to truly define any uniqueness to a function in scenarios like this.

The next property of addition is the Associative property which says that (x + y) + z = x + (y + z). We can therefore write a property test that looks like

[TestRandomData(Max = 100)]
public void Add_Associative(int x, int y, int z)
{
   var a = MyMath.Add(add(x, y), z);
   var b = MyMath.Add(x, add(y, z));
   Assert.That(a, Is.EqualTo(b));
}

and finally from the Properties of addition we look at an identity property based test, i.e. 0 + x = x and we could write something like this

[TestRandomData(Max = 100)]
public void Add_Associative(int x)
{
   var y = MyMath.Add(0, x);
   Assert.That(x, Is.EqualTo(y));
}

Now these three property based tests tell us the Add method adheres to the expected properties of addition. If we compare the properties to multiplication, we know that the identity test fails on Multiply as 0 * x = 0, however the other two tests pass – so again you see why multiple property based tests are required.

What’s next?

Ofcourse this demonstrates property based testing on a pretty simple function, more complicated functions require the developer to try to view the function, not so much about its inputs and outputs and values or the likes that are expected, but instead think about what properties make that function unique. This is not always easy.

Let’s look at some other examples of how we might use property based testing…

If we have code that reverses a string we can actually reverse a string then reverse again and see if it results in the original list. Now you’d be correct in thinking that this is not a great test as, if the reverse function does nothing, then reversing the reversed string will be the same as just reversing the string – so property based testing does not exclude the need for standard unit tests with known and expected values.

We could also use the Test oracle whereby we have a different implementation to an existing method that we then check that both output the same value.

Property based testing in Java with kqwik

There are a couple (maybe more) libraries for property based testing, I found that jqwik worked well for me.

I’m not going to go too deep into what property based testing is, except to state that for my purposes, property based testing allows me to generate random values to pass into my tests. I can generate multiple “tries” or items of data and even restrict ranges of values.

Specifically I’m using property based testing to generate values for some unit conversion code, the values get converted to a different unit and back – if the resultant value is the same as the input value we can assume that the conversion back and forth works correctly.

Anyway there are plenty of resources to read on the subject, so let’s get stuck into some code.

First off we need the following dependency to our pom.xml (obviously change the version to suit)

<dependency>
  <groupId>net.jqwik</groupId>
  <artifactId>jqwik</artifactId>
  <version>1.6.2</version>
  <scope>test</scope>
</dependency>

Let’s look a an example unit test

@Property(tries = 100)
public void testFromDegreesToGradiansAndBack(@ForAll @DoubleRange(min = -1E12, max = 1E12) double value) {
  final double convertTo = Angle.Degrees.toGradians(value);
  final double convertBack = Angle.Gradians.toDegrees(convertTo);
  assertEquals(value, convertBack, 0.01);
}

The @Property annotation tells the jqik runner to supply data for the method, creating 100 items of data (or tries) and @ForAll is used to assign data to the double value. Finally for this test I wanted to constrain the range of double values to [-1E12, 1E12].

The output (upon success) looks similar to this

timestamp = 2022-01-05T20:00:08.038391600, AngleTests:testFromDegreesToGradiansAndBack = 
                              |--------------------jqwik--------------------
tries = 100                   | # of calls to property
checks = 100                  | # of not rejected calls
generation = RANDOMIZED       | parameters are randomly generated
after-failure = PREVIOUS_SEED | use the previous seed
when-fixed-seed = ALLOW       | fixing the random seed is allowed
edge-cases#mode = MIXIN       | edge cases are mixed in
edge-cases#total = 7          | # of all combined edge cases
edge-cases#tried = 6          | # of edge cases tried in current run
seed = -3195058119397656120   | random seed to reproduce generated values

Errors will cause the test to fail and output the failure for you to review.

Property based testing in JavaScript with fast-check

Property testing is a way to test functionality by automatically generating many different inputs.

We’re going to use the fast-check library

yarn add fast-check -D

Let’s create a rather contrived example function to test

export const divide = (a: number, b: number): number => {
  if(b === 0) {
    throw new Error("Denominator cannot be 0");
  }
  return a / b;
}

As you can see, this function simply divides a by b but will fails in the denominator is 0. This was done on purpose to ensure that there is a failing test available for a specific input value (I did say it was contrived).

Now to test this we might implement the following test

import * as fc from 'fast-check';

test("Divide property test", () => {
   fc.assert(
      fc.property(fc.nat(), fc.nat(), (a, b) => {
         return divide(a, b) === a / b; 
      })
  );
});

In this instance we create the assert which wraps the property generation code, i.e. the call to property, which itself generates two natural numbers properties as input to the lambda/fat arrow function.

We then call the code that we wish to test, passing in the generated values and then returning whether the expectation was true or false.

We can use Jest or the likes in place of the return divide(a, b) === a / b, i.e.
replacing it with if we prefer and just use fast-check to generate the properties

expect(divide(a, b)).toEqual(a / b);

Now the only problem with the property based testing like this is that it will randomly generate values as opposed to inspecting the flow of your code to generate values (like tools such as Pex for .NET). Hence these values may not always include the edge cases that we might not have coded for.

For example, executing this test in Jest I find that sometimes it will pass a 0 as a denominator and other times it will not, therefore we do not always get a clear idea of all possible inputs, but it’s a great starting point for proving your functions.

In the test we generated natural numbers, we can also generate restricted ranges of values, along with integers, floating point, doubles, strings and more, see Arbitraries.