Category Archives: Programming

Hosting IronPython in a C# application

Let’s use IronPython as a scripting language within our C# application.

Note: an application I work on uses an earlier version of IronPython and they’ve significantly changed the way you use it, so this information is only relevant for version 2.7.x of Python

What are we trying to do here ?

So, the premise is this. I have a C# application and I want to allow a user/ops or whoever to write scripts which can interact directly with the running instance of the application and/or the types within an assembly. Standard scripting fair I’m sure we’d all agree.

Let’s start off by downloading the package IronPython via NuGet. The version installed is 2.7.4.

To create the Python scripting engine we use the following

ScriptEngine engine = Python.CreateEngine();

We could alternatively create a runtime and then get the engine from it, as per

ScriptRuntime runtime = Python.CreateRuntime();
ScriptEngine engine = runtime.GetEngine("Python");

Now, unlike earlier versions of the scripting engine, it appears (unless I find information that differs from this view) that we no longer really have a global scope for writing our scripts but instead create a scope ourselves (i.e. in earlier versions we could add variables and execute code on what appeared to be a global level).

So we need to create a script scope using

ScriptScope scope = engine.CreateScope();

Now we can interact with this scope, importing modules, executing code and adding variables.

Allowing our scripts to use our assembly types

Let’s assume our application adds a bunch of types/classes that we might instantiate from IronPython. To make these available to the Python code we need to use the CLR from Python to add references to our assemblies, but in this instance we don’t want the script writer to have to do this every time, so we write

scope.ImportModule("clr");

engine.Execute("import clr", scope);
engine.Execute("clr.AddReference(\"MyAssembly\")", scope);
engine.Execute("from MyAssembly import *", scope);

In the above we import the CLR module into the scope object and then execute some Python code using the engine, against the scope we created. As eluded to, we could have written these commands in every Python script as per the code below

import clr
cl.AddReference("MyAssembly")
from MyAssembly import *

but I think you’d agree it’s much better to let the host setup this code instead, especially if we wanted to add lots of assemblies.

Now from Python we can instantiate classes from MyAssembly. For example, if my host application included an assembly called MyAssembly, with the following class definition

public class MyLibrary
{
   public void Run()
   {
      Console.WriteLine("MyLibrary run called");
   }
}

then, from Python we could write the following

r = MyLibrary()
r.Run()

Putting this altogether to show how we can reference an assembly and create the types within the said assembly, just create a console application project called ConsoleApplication and enter the following

public class Application
{
   public string Name { get { return "MyApp"; } }
}

class Program
{
   static void Main(string[] args)
   {
      string code = "app = Application()\n" +
                    "print app.Name";

      ScriptEngine engine = Python.CreateEngine();
      ScriptScope scope = engine.CreateScope();

      scope.ImportModule("clr");

      engine.Execute("import clr", scope);
      engine.Execute("clr.AddReference(\"ConsoleApplication\")", scope);
      engine.Execute("from ConsoleApplication import *", scope);

      ScriptSource source = engine.CreateScriptSourceFromString(code, 
                     SourceCodeKind.Statements);
      source.Execute(scope);
   }
}


What if we already have an instance of an object in our hosting application that we want to make available to IronPython ?

So in this instance we use the following, from the hosting C# application

scope.SetVariable("host", this);

where, in this example, we pass in an instance of the hosting application. Now from Python we can call methods on the variable host, i.e. assuming the hosting application has a method named ShowMessage, we could write the following

host.ShowMessage("Hello from Python")

Finally, let’s say we have everything setup and we have a text editor embedded in the hosting application with a run button, when run is pressed we want to run the code entered into the text editor, thus

var source = engine.CreateScriptSourceFromString(code, SourceCodeKind.Statements);
source.Execute(scope);

the above code takes the source code (the code variable), then executes it in the given scope.

We could load the script from a file if we wished using

engine.CreateScriptSourceFromFile(filename);

where filename is string representing the location and filename of the Python script file.

Putting this all together into a simple example, just create a console application project and enter the following code

public class Application
{
   public string Name { get { return "MyApp"; } }
}

class Program
{
   static void Main(string[] args)
   {
      string code = "print application.Name";

      ScriptEngine engine = Python.CreateEngine();
      ScriptScope scope = engine.CreateScope();

      scope.SetVariable("application", new Application());

      ScriptSource source = engine.CreateScriptSourceFromString(code, 
                     SourceCodeKind.Statements);
      source.Execute(scope);
   }
}

Using .NET reflection to dynamically create methods with generic parameters

Following on from my post Dynamically creating C# code using the CodeDomProvider I’ve created a new type dynamically using the CodeDomProvider and I now need to use the type within a generic method call.

So to start with, I have a DataSerializer class (the basics are shown below – I’ve just distilled it down to the method signature that we’re interested in for this post)

public class DataSerializer<T> where T : new()
{
   public static IList<T> Deserialize(IDelimiterSeperatedReader dsr, 
              string data, DelimitedDeserializeOptions options)
   {
      // method implementaion
   }
   // other methods
}

From the previous post I have a dynamically created type (so I just have a Type object, the variable generatedType shown in the code below). I need to create a call to the static method (Deserialize) with this type as a type generic parameter.

Type generatedType = cr.CompiledAssembly.GetType("CsvGeneratedData");

Type[] arguments = { typeof(IDelimiterSeperatedReader), 
                  typeof(string), typeof(DelimitedDeserializeOptions) };

Type dataserializer = typeof (DataSerializer<>);
Type typedSerializer = dataserializer.MakeGenericType(generatedType);
MethodInfo mi = typedSerializer.GetMethod("Deserialize", 
               BindingFlags.Public | BindingFlags.Static, 
               null, arguments, null);

In the above the arguments variable is an array of Type’s in the order of the arguments the method Deserialize takes. We then use

This code creates the MethodInfo object but we now need to invoke the method itself, so we use the following Type dataserializer = typeof (DataSerializer<>) to create a DataSerializer but at this point it does not have the generic parameter assigned to it, so we then use MakeGenericType to assign the generic parameter to the dataserializer. Finally we get the method info. for the static method we want to call.

Now we need to invoke the method using the following

mi.Invoke(null, parameters);

where parameters is an object array with the relevant method arguments.

Dynamically creating C# code using the CodeDomProvider

I’ve been working on some code which generates C# classes. Basically it creates POCO classes to allow me to serialize/deserialize data from a data file into the POCO classes. The data file format can change and so this code allows me to easily regenerate the classes I need. This works fine, but I thought it might be better if I could create the code on the fly and use in without having to recompile the application (that would otherwise include the compiled source). In other words I wanted to create the C# classes and then generate the types and use them all dynamically at runtime.

.NET includes the CodeDomProvider class which can easily be used to create an assembly with our types created on the fly.

Here’s the code

CodeDomProvider provider = CodeDomProvider.CreateProvider("CSharp");

CompilerParameters cp = new CompilerParameters();
cp.GenerateInMemory = true;
cp.TreatWarningsAsErrors = false;
cp.ReferencedAssemblies.Add("putridparrot.Data.dll");

CompilerResults cr = provider.CompileAssemblyFromSource(cp, source);

The above is pretty self explanatory, but let’s go through it anyway.

First off, we create a CodeDomeProvider specifying the provider we require for C# code. We can (currently) create code dom’s for the following languages

  1. CSharp – C#
  2. VisualBasic – VB.NET
  3. JScript – JavaScript
  4. Cpp – C++

Next we create the compiler parameters. In this instance we want the code generated in memory, as opposed to using a file and I don’t want “warnings as error’s” in this instance. I do however need to add any of my referenced assemblies. In this case the code uses one external assembly, the putridparrot.Data.dll (as the generated code uses attributes from this assembly to mark the properties of the POCO for serialization/deserialization).

Finally we compile the code into the assembly from the source code supplied (in this case the string source contains the C# class to be compiled). The CompilerResults may include errors, so we can check this by using the following

cr.Errors.Count

Assuming we’ve no errors, then we can now access a compiled type by interacting with the assembly created by the CodeDomProvider, for example

Type type = cr.CompiledAssembly.GetType("AutoGeneratedDataType");

The CompiledAssembly is just a standard Assembly at this point, so obviously you can do all the standard things you can do on any other Assembly object.

Side note

As a side note we can iterate over the languages supported via the CodeDomProvider using the following

CompilerInfo[] compilerInfo = CodeDomProvider.GetAllCompilerInfo();
foreach (var ci in compilerInfo)
{
   string[] languages = ci.GetLanguages();
   foreach(var language in languages)
   {
      Console.WriteLine(language);
   }
}

Properties in F#

We’re talking class properties…

Automatic Properties

Let’s start with the one I seem to have most trouble remembering, especially as it’s a different syntax to the other properties. We’re talking automatic properties.

type Rectangle()= 
   member val length = 0 with get, set
   member val width = 0 with get, set

The key differences to the other property syntax, i.e. properties with backing fields (for example) are, the addition of the val keyword, the lack of a self-identifier, the initialization expression, i.e. the length = 0 and the get, set, i.e. no and.

So the above creates a type Rectangle and two auto properties, length and width. Both are initialized to 0.

Non-automatic properties

Let’s start with the syntax for a property with both a getter and setter. Let’s start with the version of the above automatic property example but with backing fields

Note: the use of this and value as the identifier and parameter respectively is just to use C# style naming and obviously can be changed to suit

type Rectangle() =
   let mutable l = 0
   let mutable w = 0
   member this.length 
      with get() = l
      and set value = l <- value
   member this.width
      with get() = w
      and set value = w <- value

The alternative syntax for the above is as follows

type Rectangle() =
   let mutable l = 0
   let mutable w = 0
   member this.length = l
   member this.length with set value = l <- value
   member this.width = w
   member this.width with set value = w <- value

Using properties

And just for completeness, let’s look at how we use properties on a class. As in C# we can simply use the following

let r = new Rectangle()
r.length <- 100

or using the F# equivalent of the C# initializer

let r = new Rectangle(length = 100)

Records in F#

A far more comprehensive and frankly much better post on the subject of records can be found here.

However this post is an attempt to distil things down to some of the basics.

Declaring a record type

Records can be thought of as a cross between a tuple with named fields and a specialised class. Basically the syntax is more in keeping with a class type but the specialisations are that, all properties are automatically exposed, you cannot define constructors as per classes and they have structural equality semantics as opposed to a classes reference equality semantics.

Let’s declare a record type

type Person = { Name : string; Age : int }

Now to use this we simple write the following

let p = { Name = "Bob"; Age = 43 }

So as you see we do not need to actually use the type name Person in the declaration (hence it appears similar to a tuple but with named properties and ofcourse using {…} instead of (…)).

What if we declared the following

type Person = { Name : string; Age : int }
type Employee = { Name : string; Age : int }

let p = { Name = "Bob"; Age = 43 }

What type is p ?

The value p will actually be assigned the type Employee as this was the last record declared. So we need a way to explicitly declare the type we want p to be (in the example above). This is done in the following way

let p = { Person.Name = "Bob"; Age = 43 }

Now p is of type Person.

You will need to assign each of the named properties when creating a value of a record.

To access the properties, we simply use dot notation and the property name, thus

let p = { Person.Name = "Bob"; Age = 43 }

let n = p.Name

Deconstructing a record

We can also deconstruct a record to it’s constituent value as follows

let p = { Person.Name = "Bob"; Age = 43 }

let { Name = n; Age = a } = p

The above let looks slightly odd in that we appear to be assigning the value n to Name whereas it’s actually the other way around in essence.

Again, if you have more than one type which looks the same (as Person and Employee) you can again prefix the deconstruction to ensure the type is matched for example

let { Person.Name = n; Age = a } = p

Tuples, tuples and more tuples

Tuples are used a lot in F#. Let’s start with a definition

A tuple is a grouping of unnamed but ordered values, possibly of different type

This was take from Tuples (F#) and I’d highly recommend reading this. I will undoubtedly recreate some of the information from this document here, but hopefully add something of use.

Declaring tuples

So we’ve got a definition (above) for what a tuple is, now let’s have a look at some

let tupleInts = (1, 2, 3, 4)
let tupleStrings = ("one", "two", "three")
let tupleValues = (a, b, c, d)
let tupleMixed = ("one", 2, c)
let tupleExpressions = (a + 1, b + 2, c + 3)

The above should be pretty self explanatory.

If we take the first tuple, name tupleInts, and run this in the F# interactive window we’ll see that this translates to val tupleInts : int * int * int * int = (1, 2, 3, 4). The * denoting a tuple. In this case you can see that each item/element in the tuple is of type int.

Getting values from a tuple

Using let bindings

let (a, b) = (1, 2)

Using let bindings but ignore one or more elements (i.e. using the _ wildcard character)

let (a, b, _) = (1, 2, 3)

// or

let a, b, _ = (1, 2, 3)

Using pattern matching

let tuple = (1, 2, 3)

let pattern t =
    match t with
    | (a, b, c) -> printfn "%d" a

pattern tuple

We can use the fst and snd functions which get the first or second elements from the tuple, i.e.

let a = fst (1, 2)
let b = snd (1, 2)

Obviously a will have the value 1 and bReturning multiple values

We can obviously pass multiple values into a function using a tuple, but we can also return a tuple and hence return multiple values as per

let inc a b = (a + 1, b + 1)

where the return is a tuple of the values (a + 1, b + 1)

Cross language

In C# (.NET 4.0) we have the Tuple generic class which we can create in one of two ways

Tuple<int, int> tuple = mew Tuple<int, int>(1, 2);

// or using the helper method

var tuple = Tuple.Create(1, 2);

F# uses the same Tuple class internally (we can confirm this by checking the IL using ILSpy or the likes) to represent it’s tuples. However, as seen, it’s more of a language feature in F#. Hence we have the syntactic sugar to allow us to create the tuple just using (item1, item2).

Obviously if we can create a Tuple in C# and the underlying representation of a tuple in F# is the same, we can pass data to and from each language using this mechanism. So for example if we had this rather contrived C# code

public static class Data
{
   public static Tuple<int, string> Get()
   {
      return Tuple.Create(123, "One Two Three");
   }

   public static void Set(Tuple<int, string> tuple)
   {
      Console.WriteLine(tuple.Item1 + " - " + tuple.Item2);
   }
}

and in F# we can interact with these methods as follows

let (num, value) = CSharpLib.Data.Get()
CSharpLib.Data.Set(321, "Three Two One");

Event and event handling in F#

Publishing events

Let’s look at the subject of event handling by starting with some code

type Application() =
    let state = new Event<_>()
    
    member this.State = state.Publish
    member this.Run() =
        state.Trigger(this, "Starting")
        state.Trigger(this, "Running")
        state.Trigger(this, "Finished")

So we’ve declared an Application object which has a private state value, of type Event<_>. Basically this is declaring a value as an Event, the <_> leaves it to the compiler to infer the usage, ofcourse you can explicitly state the type if you prefer.

An Event is an implementation of IEvent, which is itself derived from an IDelegateEvent and IObservable.

The IDelegateEvent has a special use when you mark properties with the attribute [<CLIEvent>]. The properties are now compatible with other languages which use CLI compatible events.

The IObservable allows an event to be composable using Reactive Extensions (Rx) or using F#’s subset of Rx – this means we can filter, merge, split and more on events.

Going back to the code sample, we declare a member property using this.State = state.Publish which basically binds to the Publish property which itself allows us to use the Add method to add event handlers. Had we wished to make this property CLI compliant we would write

[<CLIEvent>]
member this.State = state.Publish

The next piece of code in our example is a Run function and this simply triggers events in a way analogous to state changing during the function.

Let’s now look at some code for handling these “state” change events…

Subscribing to events

Assuming we’ve created an instance of the application object, as follows

let app = new Application()

we can now subscribe to events on the object by adding an event handler

app.State.Add(fun (sender, obj) -> printfn "%s" obj)

in this example we use the Add function to subscribe to the State event and in this example we supply an anonymous function which is called each time a trigger is fired on the Application object’s State event.

Now the Add function doesn’t have an equivalent remove, but as an alternative we could use the IDelegateEvent’s AddHandler and RemoveHandler as per the following

let handler = new Handler<_>(fun s o -> printfn "%s" (snd o))
app.State.AddHandler(handler)

// ...

app.State.RemoveHandler(handler)

We can also use the more functional style Event module methods to both add and filter (for example) to only handle certain messages, as per

app.State 
   |> Event.filter(fun (sender, obj) -> obj = "Running")
   |> Event.add(fun (sender, obj) -> printfn "%s" obj)

In this example we’re filtering the events to only handle the case when the message is “Running” and the add function simply subscribes to the resultant events and outputs the message.

Observable

We can also use the IObservable subscription based model for events using

app.State 
   |> Observable.map(fun (sender, obj) -> obj)
   |> Observable.filter(fun obj -> obj = "Running")
   |> Observable.add(fun obj -> printfn "%s" obj)

In this sample code we map the events to extract the message string (using the map function), then we filter it and finally we add our event handler.

And here’s an example using the subscribe function

app.State 
   |> Observable.map(fun (sender, obj) -> obj)
   |> Observable.filter(fun obj -> obj = "Running")
   |> Observable.subscribe(fun obj -> printfn "%s" obj)
   |> ignore

Units of Measure

F# has the concept of units of measure. In essence we can declare a unit of measure on a numerical type which associates the numeric with a specific “measurement type” of data. For example, we might define a value as one of the imperial unit of measure, such as lb. Such as

As you can see, declare a unit of measure we use the [<Measure>] attribute as follows

[<Measure>]
type lb

We can now use this unit on a type as follows

let weightInPounds = 100<lb>

The inferred type here is val weightInPounds : int = 100, so as you can see the unit of measure is tied to the type (in this case an int).

If we wish to add or subtract to/from the weightInPounds we might be tempted to write

let newWeight = weightInPounds - 2

But we’ll get the following error The type ‘int’ does not match the type ‘int. This is because the 2 is simply an int and we’d need convert to the same unit of measurements, as per

let newWeight = weightInPounds - 2<lb>

However using multiplication or division will work without the need for the unit of measure added, so the following is perfectly fine

let newWeight = weightInPounds * 2

Deriving units of measurement from existing units of measurement

We can also derived new units of measurement from existing ones. So assuming we have the following

[<Measure>]
type lb

[<Measure>]
type inch

We might create a psi (pounds per square inch) as follows

[<Measure>]
type psi = lb inch^2

We could also apply multiple units of measurement as part of the let binding, for example

let p = 12<lb inch^2>

Removing a unit of measure

In some cases we might want to remove the unit of measure from a value, for example maybe we have a method that takes an int. We will get a compile time error stating “The expression was expected to have type in but here has type int“. We can simply cast

The code below is trying to set the value i (which we’ve setup to be an int) with a unit of measure value (where weight is shown as an int).

let weight = 12<kg>

let i : int = int weight

Writing to the console

To output a value with a unit of measure, using printn, we don’t use the following

let weight = 12<kg>

printfn "%A" weight

Tail Recursion in F#

The Stack

In most languages, when a function is called, the function arguments and the return address of the calling function (possibly including other things like the frame pointer – which can be turned off in the likes of VC++) are pushed onto the stack. When the function call ends, these are popped from the stack and the function returns to the calling method.

The stack is finite, and per thread is by default 1MB in size (see Thread Stack Size). So obviously it’s possible to get to point where we get a StackOverflow exception. This is especially a problem with recursive functions.

Another issue with having deep stacks in .NET is when the garbage collector kicks in as it will have to traverse the entire stack each time garbage collection is required.

Tail Recursion/Tail Call

C# has no built in tail call mechanism, unlike F#. A tail call allows the .NET CLR to drop the current stack frame before executing the target function.

Let’s look at a really simple recursive function, which simply increments a value and outputs the result.

let rec increment i =
    if i < 100000 then 
        printfn "%d" i
        increment (i + 1)
        printfn "next"

The above will be terminated by a StackOverflowException very quickly. But if we remove the last printfn line to give

let rec increment i =
    if i < 100000 then 
        printfn "%d" i
        increment (i + 1)

we will find the function continues until completion.

So basically a tail call is where the recursive call is the last (or tail) function call. This doesn’t strictly mean the recursive call is the last line of the function, it’s that it’s the last called within the function. For example

let rec increment i =
    if i < 100000 then 
        printfn "%d" i
        increment (i + 1)
    else
        printfn "next"

shows that the recursive call is the last function call and only when the recursive calls are completed (and stack unwound) does the last line get executed. But beware, the recursive call should be the last call executed but if we alter the code slightly, to have it return an int we might have something like the following code

let rec increment i =
    if i < 100000 then 
        printfn "%d" i
        i + increment (i + 1)
    else
        printfn "%d" i
        0

This is contrived example, to force an int return (so don’t worry about what it actually does). Now the line i + increment (i + 1) is the last executed as the code goes recursive and it appears that increment (i + 1) is still the last line executed, however the inclusion of the i + code has mean’t this code is no longer tail recursive.

Further Reading

Tail call
Tail calls in F#

Mutable values and closures

The real title of this post should be The mutable variable ‘count’ is used in an invalid way. Mutable variables may not be captured by closures. Consider eliminating this use of mutation or using a heap-allocated mutable reference cell via ‘ref’ and ‘!’ but I thought it was a little long for a title.

Setting the scene

I was writing a simple C# extension method for the System.String class which has the following method signature

public static void Split(this string s, char[] seperator, StringSplitOptions options, Action<string, int, int> action)

The idea being that instead of creating an array of strings for each delimited string, I would instead pass the original string to an Action delegate with the start of a delimited field and the length of the field. This would help save on potentially wasteful creation of memory when handling large amounts of data.

I decided it’d be cool to write the unit tests for this method using F# (to help increase my knowledge of F#) – review Unit testing in F# to see how I set up the tests etc.

So the intention was to create a simple unit test, which just checked the number of times the Action delegate was called, which should be the same number of times as the number of delimited words within the string. So here’s a sample of what I would have written in C#

[Fact]
public void ExpectTheSameNumberOfCallsToTheActionDelegateAsDelimitedWords()
{
   string test = "The, quick, brown, fox";

   int count = 0;
   test.Split(new []{','}, StringSplitOptions.None, (s, i, j) => count++);

   Assert.Equal(4, count);
}

So I implemented the following F# text, very much along the lines of the above

[<Fact>]
let ``Expect the same number of calls to the action delegate as delimited words`` () =
   let mutable count = 0
   "The, quick, brown, fox".Split([|','|], StringSplitOptions.None, 
            fun s a b -> count <- count + 1)
   count |> should equal 4

at this point Visual Studio shows a squiggly red line under the code fun s a b -> count <- count + 1 with the message The mutable variable ‘x’ is used in an invalid way. Mutable variables may not be captured by closures. Consider eliminating this use of mutation or using a heap-allocated mutable reference cell via ‘ref’ and ‘!’. What’s going on ?

Searching for answers

Credit where credit is due, I searched around and found the web and found the following StackOverflow question “The mutable variable ‘i’ is used in an invalid way.?” which describes a solution to this problem.

Basically you cannot use mutable values inside a closure and thus we need to change to a ref type. So the following code now works

[<Fact>]
let ``Expect the same number of calls to the action delegate as delimited words`` () =
   let count = ref 0
   "The, quick, brown, fox".Split([|','|], StringSplitOptions.None, 
                fun s a b -> count := !count + 1)
   !count |> should equal 4

The changes here are that the mutable value count is now a ref type, the code to increment the count changes to use the ref assignment operator := and we also use the reference cell dereferencing operator !. The last line also uses the same. Alternatively we could have written fun s a b -> count := count.Value + 1 and the last line as count.Value |> should equal 4.

So why can’t we use the mutable type within a closure, after all it works fine in C#. The StackOverflow question (listed above) has a telling comment. Mutable values are stored on the stack not the heap. Therefore when the anonymous function exits, values stored on the stack are discarded during the stack unwind.

So how can C# handle this code ?

In C# we can do this because the Common Language Runtime (CLR) automatically places variables on the heap if they’re used as part of a closure as detailed in the post Capturing Mutables in F#.

References

The following are some references I located either whilst trying to solve this problem or as part of my research into the reasons for this after fixing the code.

The mutable variable ‘i’ is used in an invalid way.?
Mutable variables cannot be captured by closures” is giving me a bad day
Capturing Mutables in F#
The Truth About Value Types