More adventures with F#…

Operator overloading is documented perfectly well in Operator Overloading (F#) but to summarize I’ve created this post…

Operators in F# can be overloaded at the class, record type or global level.

Overloading an operator on a class or record type

Let’s look at the syntax for overloading an operator on a class or record type.

Note: Reproduced from the Operator Overloading (F#) page

static member (operator-symbols) (parameter-list> =
    method-body

Overloading an operator which is globally accessible

Let’s look at the syntax for overloading an operator at a global level.

Note: Reproduced from the Operator Overloading (F#) post

let [inline] (operator-symbols) parameter-list = 
    function-body

More…

You can overload the “standard” operators but you can also create your own operators. For example FAKE creates it’s own global operator to combine two file paths using the @@ operator

The below is taken from the FAKE source code available in github

let inline (@@) path1 path2 = combinePaths path1 path2

As it’s pretty obvious, this is a global level overload of a newly created operator, the @@ which takes two parameters.

Where an operator may be used for binary or unary (infix or prefix), such as + or – (+., -., &, &&, %, and %% may also be used for infix or prefix operators) we need to prefix the operator with a tilde ~, hence to overload the + and use as a unary operator we’d write something like

let inline (~+) x = [ x ]

// in use we'd have

let v = + "hello"

Whereas using the + as a binary operator we’d write something like

let inline (+) x y = [ x, y ]

// in use we'd have

let v = "hello" + "world"

If you’re coming from C# to F# you’ll have already noticed the “funky” set of operators that F# supports. As already shown with the @@ operator sample (taken from FAKE) you can combine operators to produce all sorts of new operators, for example

let (<==>) x  y = [ x, y]

So this is my first attempt at implementing a computational expression in F#. I’m not going to go into definitions or the likes as to what computational expressions are as there are far better posts out there on this subject than I could probably write, at this time. I’ll list some in a “further reading” section at the end of the post.

What I’m going to present here is a builder which really just creates a list of names – nothing particularly clever – but it gives a basic idea on getting started with computational expressions (I hope).

So first off we need to create a builder type, mine’s called CurveBuilder

type Items = Items of string list

type CurveBuilder() =
    member this.Yield (()) = Items []

    [<CustomOperation ("create", MaintainsVariableSpace = true)>]
    member this.Create (Items sources, name: string) = 
        Items [ yield! sources
                yield name ]

As this is just a simple demo, the only thing I’m doing is creating a list of strings which you would be correct in thinking that I could do easier using a collection class, but I’m really just interested in seeing the builder and it’s interactions without too much clutter, so go with me on this one…

Before we discuss the builder code, let’s take a look at how we’d use a builder in our application

let builder = CurveBuilder()

let curves = 
   builder {
      create "risky_curve1.gbp"
      create "risky_curve2.usd"
   }

In this code we first create an instance of a CurveBuilder, then when we use the builder to create a list of the curves. The Yield method is first called on the CurveBuilder, returning an empty Items value. Subsequent calls to the create method then call the Create method of the CurveBuilder and, as can be seen, create a new Items value containing the previous Items plus our new curve name.

Simple enough but surely there’s more to this than meets the eye

The example above is a minimal implementation, of course, of a builder. You’ll notice that we have an attribute on one method (the Create method) and not on the Yield method.

So the attribute CustomOperation is used on a member of a builder to create new “query operators”. Basically it extends the builder functionality with new operators named whatever you want to name them.

On the other hand the Yield method is a “standard” builder method. There are several other methods which F# knows about implicitly within a builder class, including Return, Zero, Bind, For, YieldFrom, ReturnFrom, Delay, Combine, Run and more, see Computation Expressions (F#)
for a full list of “built-in workflows”. With these we can obviously produce something more complex than the example presented in this post – which is in essence a glorified “list” builder with snazzy syntax.

Discriminated Union gotcha

One thing I got caught out with from the above code is that I wanted to simply list the curves I’d created but couldn’t figure out how to “deconstruct” the discriminated union

type Items = Items of string list

to a simple string list – at this point I claim ignorance as I’m not using F# as much as I’d like so am still fairly inexperienced with it.

My aim was to produce something like this

for curve in curves do
   printfn "Curve: %s" curve

but this failed with the error The type ‘Items’ is not a type whose values can be enumerated with this syntax, i.e. is not compatible with either seq<_>, IEnumerable<_> or IEnumerable and does not have a GetEnumerator method

I need to in essence disconnect the Items from the string list, to achieve this I found the excellent post Discriminated Unions.

The solution is as follows

let (Items items) = curves

for curve in items do
    printfn "Curve: %s" curve

Note: the let (Items items) = curves

Further Reading
Implementing a builder: Zero and Yield
Implementing a builder: Combine
Implementing a builder: Delay and Run
Implementing a builder: Overloading
Implementing a builder: Adding laziness
Implementing a builder: The rest of the standard methods

Computation Expressions (F#)

The FSharp.Data library introduces several TypeProviders. At the time of writing the general purpose TypeProviders include one for XML, JSON and CVS along with TypeProviders specific to Freebase and Worldbank.

Either add a reference to FSharp.Data or use NuGet to install FSharp.Data to work through these examples.

CsvTypeProvider

The CsvTypeProvider (as the name suggests) allows us to interact with Csv data. Here’s some code based upon the Titanic.csv Csv file supplied with FSharp.Data’s source via GitHub.

open FSharp.Data

type TitanicData = CsvProvider<"c:\dev\Titanic.csv", SafeMode=true, PreferOptionals=true>

[<EntryPoint>]
let main argv =     

    let ds = TitanicData.Load(@"c:\dev\Titanic.csv")

    for i in ds.Data do
        printfn "%s" i.Name.Value 

    0 // return an integer exit code

Firstly we open the FSharp.Data namespace, then create a type abbreviation, to the CsvProvider. We pass in the sample data file so that the provider can generate it’s schema.

We can ignore the main function and return value and instead jump straight to the let ds = … where we load the actual data that we want to parse. The type abbreviation used this same file to generate the schema but in reality we could now load any file that matches the expected titanic file schema.

We can now enumerate over the file data using ds.Data and best still is that within Visual Studio we can now use intellisense to look at the headings generated by the CsvProvider, which now appear as properties of the type. i.e. i.Name is an Option type supplied by the schema, hence if we’d had a heading in the Csv named “Passenger Id” we’d now have Passenger showing as a property on the instance I within the loop and hence be able to write like printfn “%i” i.“Passenger Id“.Value.

We’re talking class properties…

Automatic Properties

Let’s start with the one I seem to have most trouble remembering, especially as it’s a different syntax to the other properties. We’re talking automatic properties.

type Rectangle()= 
   member val length = 0 with get, set
   member val width = 0 with get, set

The key differences to the other property syntax, i.e. properties with backing fields (for example) are, the addition of the val keyword, the lack of a self-identifier, the initialization expression, i.e. the length = 0 and the get, set, i.e. no and.

So the above creates a type Rectangle and two auto properties, length and width. Both are initialized to 0.

Non-automatic properties

Let’s start with the syntax for a property with both a getter and setter. Let’s start with the version of the above automatic property example but with backing fields

Note: the use of this and value as the identifier and parameter respectively is just to use C# style naming and obviously can be changed to suit

type Rectangle() =
   let mutable l = 0
   let mutable w = 0
   member this.length 
      with get() = l
      and set value = l <- value
   member this.width
      with get() = w
      and set value = w <- value

The alternative syntax for the above is as follows

type Rectangle() =
   let mutable l = 0
   let mutable w = 0
   member this.length = l
   member this.length with set value = l <- value
   member this.width = w
   member this.width with set value = w <- value

Using properties

And just for completeness, let’s look at how we use properties on a class. As in C# we can simply use the following

let r = new Rectangle()
r.length <- 100

or using the F# equivalent of the C# initializer

let r = new Rectangle(length = 100)

A far more comprehensive and frankly much better post on the subject of records can be found here.

However this post is an attempt to distil things down to some of the basics.

Declaring a record type

Records can be thought of as a cross between a tuple with named fields and a specialised class. Basically the syntax is more in keeping with a class type but the specialisations are that, all properties are automatically exposed, you cannot define constructors as per classes and they have structural equality semantics as opposed to a classes reference equality semantics.

Let’s declare a record type

type Person = { Name : string; Age : int }

Now to use this we simple write the following

let p = { Name = "Bob"; Age = 43 }

So as you see we do not need to actually use the type name Person in the declaration (hence it appears similar to a tuple but with named properties and ofcourse using {…} instead of (…)).

What if we declared the following

type Person = { Name : string; Age : int }
type Employee = { Name : string; Age : int }

let p = { Name = "Bob"; Age = 43 }

What type is p ?

The value p will actually be assigned the type Employee as this was the last record declared. So we need a way to explicitly declare the type we want p to be (in the example above). This is done in the following way

let p = { Person.Name = "Bob"; Age = 43 }

Now p is of type Person.

You will need to assign each of the named properties when creating a value of a record.

To access the properties, we simply use dot notation and the property name, thus

let p = { Person.Name = "Bob"; Age = 43 }

let n = p.Name

Deconstructing a record

We can also deconstruct a record to it’s constituent value as follows

let p = { Person.Name = "Bob"; Age = 43 }

let { Name = n; Age = a } = p

The above let looks slightly odd in that we appear to be assigning the value n to Name whereas it’s actually the other way around in essence.

Again, if you have more than one type which looks the same (as Person and Employee) you can again prefix the deconstruction to ensure the type is matched for example

let { Person.Name = n; Age = a } = p

Tuples are used a lot in F#. Let’s start with a definition

A tuple is a grouping of unnamed but ordered values, possibly of different type

This was take from Tuples (F#) and I’d highly recommend reading this. I will undoubtedly recreate some of the information from this document here, but hopefully add something of use.

Declaring tuples

So we’ve got a definition (above) for what a tuple is, now let’s have a look at some

let tupleInts = (1, 2, 3, 4)
let tupleStrings = ("one", "two", "three")
let tupleValues = (a, b, c, d)
let tupleMixed = ("one", 2, c)
let tupleExpressions = (a + 1, b + 2, c + 3)

The above should be pretty self explanatory.

If we take the first tuple, name tupleInts, and run this in the F# interactive window we’ll see that this translates to val tupleInts : int * int * int * int = (1, 2, 3, 4). The * denoting a tuple. In this case you can see that each item/element in the tuple is of type int.

Getting values from a tuple

Using let bindings

let (a, b) = (1, 2)

Using let bindings but ignore one or more elements (i.e. using the _ wildcard character)

let (a, b, _) = (1, 2, 3)

// or

let a, b, _ = (1, 2, 3)

Using pattern matching

let tuple = (1, 2, 3)

let pattern t =
    match t with
    | (a, b, c) -> printfn "%d" a

pattern tuple

We can use the fst and snd functions which get the first or second elements from the tuple, i.e.

let a = fst (1, 2)
let b = snd (1, 2)

Obviously a will have the value 1 and bReturning multiple values

We can obviously pass multiple values into a function using a tuple, but we can also return a tuple and hence return multiple values as per

let inc a b = (a + 1, b + 1)

where the return is a tuple of the values (a + 1, b + 1)

Cross language

In C# (.NET 4.0) we have the Tuple generic class which we can create in one of two ways

Tuple<int, int> tuple = mew Tuple<int, int>(1, 2);

// or using the helper method

var tuple = Tuple.Create(1, 2);

F# uses the same Tuple class internally (we can confirm this by checking the IL using ILSpy or the likes) to represent it’s tuples. However, as seen, it’s more of a language feature in F#. Hence we have the syntactic sugar to allow us to create the tuple just using (item1, item2).

Obviously if we can create a Tuple in C# and the underlying representation of a tuple in F# is the same, we can pass data to and from each language using this mechanism. So for example if we had this rather contrived C# code

public static class Data
{
   public static Tuple<int, string> Get()
   {
      return Tuple.Create(123, "One Two Three");
   }

   public static void Set(Tuple<int, string> tuple)
   {
      Console.WriteLine(tuple.Item1 + " - " + tuple.Item2);
   }
}

and in F# we can interact with these methods as follows

let (num, value) = CSharpLib.Data.Get()
CSharpLib.Data.Set(321, "Three Two One");

Publishing events

Let’s look at the subject of event handling by starting with some code

type Application() =
    let state = new Event<_>()
    
    member this.State = state.Publish
    member this.Run() =
        state.Trigger(this, "Starting")
        state.Trigger(this, "Running")
        state.Trigger(this, "Finished")

So we’ve declared an Application object which has a private state value, of type Event<_>. Basically this is declaring a value as an Event, the <_> leaves it to the compiler to infer the usage, ofcourse you can explicitly state the type if you prefer.

An Event is an implementation of IEvent, which is itself derived from an IDelegateEvent and IObservable.

The IObservable allows an event to be composable using Reactive Extensions (Rx) or using F#’s subset of Rx – this means we can filter, merge, split and more on events.

Going back to the code sample, we declare a member property using this.State = state.Publish which basically binds to the Publish property which itself allows us to use the Add method to add event handlers. Had we wished to make this property CLI compliant we would write

[<CLIEvent>]
member this.State = state.Publish

The next piece of code in our example is a Run function and this simply triggers events in a way analogous to state changing during the function.

Let’s now look at some code for handling these “state” change events…

Subscribing to events

Assuming we’ve created an instance of the application object, as follows

let app = new Application()

we can now subscribe to events on the object by adding an event handler

app.State.Add(fun (sender, obj) -> printfn "%s" obj)

in this example we use the Add function to subscribe to the State event and in this example we supply an anonymous function which is called each time a trigger is fired on the Application object’s State event.

Now the Add function doesn’t have an equivalent remove, but as an alternative we could use the IDelegateEvent’s AddHandler and RemoveHandler as per the following

let handler = new Handler<_>(fun s o -> printfn "%s" (snd o))
app.State.AddHandler(handler)

// ...

app.State.RemoveHandler(handler)

We can also use the more functional style Event module methods to both add and filter (for example) to only handle certain messages, as per

app.State 
   |> Event.filter(fun (sender, obj) -> obj = "Running")
   |> Event.add(fun (sender, obj) -> printfn "%s" obj)

In this example we’re filtering the events to only handle the case when the message is “Running” and the add function simply subscribes to the resultant events and outputs the message.

Observable

We can also use the IObservable subscription based model for events using

app.State 
   |> Observable.map(fun (sender, obj) -> obj)
   |> Observable.filter(fun obj -> obj = "Running")
   |> Observable.add(fun obj -> printfn "%s" obj)

In this sample code we map the events to extract the message string (using the map function), then we filter it and finally we add our event handler.

And here’s an example using the subscribe function

app.State 
   |> Observable.map(fun (sender, obj) -> obj)
   |> Observable.filter(fun obj -> obj = "Running")
   |> Observable.subscribe(fun obj -> printfn "%s" obj)
   |> ignore

F# has the concept of units of measure. In essence we can declare a unit of measure on a numerical type which associates the numeric with a specific “measurement type” of data. For example, we might define a value as one of the imperial unit of measure, such as lb. Such as

As you can see, declare a unit of measure we use the [<Measure>] attribute as follows

[<Measure>]
type lb

We can now use this unit on a type as follows

let weightInPounds = 100<lb>

The inferred type here is val weightInPounds : int = 100, so as you can see the unit of measure is tied to the type (in this case an int).

If we wish to add or subtract to/from the weightInPounds we might be tempted to write

let newWeight = weightInPounds - 2

But we’ll get the following error The type ‘int’ does not match the type ‘int‘. This is because the 2 is simply an int and we’d need convert to the same unit of measurements, as per

let newWeight = weightInPounds - 2<lb>

However using multiplication or division will work without the need for the unit of measure added, so the following is perfectly fine

let newWeight = weightInPounds * 2

Deriving units of measurement from existing units of measurement

We can also derived new units of measurement from existing ones. So assuming we have the following

[<Measure>]
type lb

[<Measure>]
type inch

We might create a psi (pounds per square inch) as follows

[<Measure>]
type psi = lb inch^2

We could also apply multiple units of measurement as part of the let binding, for example

let p = 12<lb inch^2>

Removing a unit of measure

In some cases we might want to remove the unit of measure from a value, for example maybe we have a method that takes an int. We will get a compile time error stating “The expression was expected to have type in but here has type int“. We can simply cast

The code below is trying to set the value i (which we’ve setup to be an int) with a unit of measure value (where weight is shown as an int).

let weight = 12<kg>

let i : int = int weight

Writing to the console

To output a value with a unit of measure, using printn, we don’t use the following

let weight = 12<kg>

printfn "%A" weight

The Stack

In most languages, when a function is called, the function arguments and the return address of the calling function (possibly including other things like the frame pointer – which can be turned off in the likes of VC++) are pushed onto the stack. When the function call ends, these are popped from the stack and the function returns to the calling method.

The stack is finite, and per thread is by default 1MB in size (see Thread Stack Size). So obviously it’s possible to get to point where we get a StackOverflow exception. This is especially a problem with recursive functions.

Another issue with having deep stacks in .NET is when the garbage collector kicks in as it will have to traverse the entire stack each time garbage collection is required.

Tail Recursion/Tail Call

C# has no built in tail call mechanism, unlike F#. A tail call allows the .NET CLR to drop the current stack frame before executing the target function.

Let’s look at a really simple recursive function, which simply increments a value and outputs the result.

let rec increment i =
    if i < 100000 then 
        printfn "%d" i
        increment (i + 1)
        printfn "next"

The above will be terminated by a StackOverflowException very quickly. But if we remove the last printfn line to give

let rec increment i =
    if i < 100000 then 
        printfn "%d" i
        increment (i + 1)

we will find the function continues until completion.

So basically a tail call is where the recursive call is the last (or tail) function call. This doesn’t strictly mean the recursive call is the last line of the function, it’s that it’s the last called within the function. For example

let rec increment i =
    if i < 100000 then 
        printfn "%d" i
        increment (i + 1)
    else
        printfn "next"

shows that the recursive call is the last function call and only when the recursive calls are completed (and stack unwound) does the last line get executed. But beware, the recursive call should be the last call executed but if we alter the code slightly, to have it return an int we might have something like the following code

let rec increment i =
    if i < 100000 then 
        printfn "%d" i
        i + increment (i + 1)
    else
        printfn "%d" i
        0

This is contrived example, to force an int return (so don’t worry about what it actually does). Now the line i + increment (i + 1) is the last executed as the code goes recursive and it appears that increment (i + 1) is still the last line executed, however the inclusion of the i + code has mean’t this code is no longer tail recursive.

Further Reading

Tail call
Tail calls in F#

The real title of this post should be The mutable variable ‘count’ is used in an invalid way. Mutable variables may not be captured by closures. Consider eliminating this use of mutation or using a heap-allocated mutable reference cell via ‘ref’ and ‘!’ but I thought it was a little long for a title.

Setting the scene

I was writing a simple C# extension method for the System.String class which has the following method signature

public static void Split(this string s, char[] seperator, StringSplitOptions options, Action<string, int, int> action)

The idea being that instead of creating an array of strings for each delimited string, I would instead pass the original string to an Action delegate with the start of a delimited field and the length of the field. This would help save on potentially wasteful creation of memory when handling large amounts of data.

I decided it’d be cool to write the unit tests for this method using F# (to help increase my knowledge of F#) – review Unit testing in F# to see how I set up the tests etc.

So the intention was to create a simple unit test, which just checked the number of times the Action delegate was called, which should be the same number of times as the number of delimited words within the string. So here’s a sample of what I would have written in C#

[Fact]
public void ExpectTheSameNumberOfCallsToTheActionDelegateAsDelimitedWords()
{
   string test = "The, quick, brown, fox";

   int count = 0;
   test.Split(new []{','}, StringSplitOptions.None, (s, i, j) => count++);

   Assert.Equal(4, count);
}

So I implemented the following F# text, very much along the lines of the above

[<Fact>]
let ``Expect the same number of calls to the action delegate as delimited words`` () =
   let mutable count = 0
   "The, quick, brown, fox".Split([|','|], StringSplitOptions.None, 
            fun s a b -> count <- count + 1)
   count |> should equal 4

at this point Visual Studio shows a squiggly red line under the code fun s a b -> count <- count + 1 with the message The mutable variable ‘x’ is used in an invalid way. Mutable variables may not be captured by closures. Consider eliminating this use of mutation or using a heap-allocated mutable reference cell via ‘ref’ and ‘!’. What’s going on ?

Searching for answers

Credit where credit is due, I searched around and found the web and found the following StackOverflow question “The mutable variable ‘i’ is used in an invalid way.?” which describes a solution to this problem.

Basically you cannot use mutable values inside a closure and thus we need to change to a ref type. So the following code now works

[<Fact>]
let ``Expect the same number of calls to the action delegate as delimited words`` () =
   let count = ref 0
   "The, quick, brown, fox".Split([|','|], StringSplitOptions.None, 
                fun s a b -> count := !count + 1)
   !count |> should equal 4

The changes here are that the mutable value count is now a ref type, the code to increment the count changes to use the ref assignment operator := and we also use the reference cell dereferencing operator !. The last line also uses the same. Alternatively we could have written fun s a b -> count := count.Value + 1 and the last line as count.Value |> should equal 4.

So why can’t we use the mutable type within a closure, after all it works fine in C#. The StackOverflow question (listed above) has a telling comment. Mutable values are stored on the stack not the heap. Therefore when the anonymous function exits, values stored on the stack are discarded during the stack unwind.

So how can C# handle this code ?

In C# we can do this because the Common Language Runtime (CLR) automatically places variables on the heap if they’re used as part of a closure as detailed in the post Capturing Mutables in F#.

References

The following are some references I located either whilst trying to solve this problem or as part of my research into the reasons for this after fixing the code.

The mutable variable ‘i’ is used in an invalid way.?
Mutable variables cannot be captured by closures” is giving me a bad day
Capturing Mutables in F#
The Truth About Value Types