Category Archives: .NET

PowerArgs, command line parser

I cannot tell you how many times I forgot the name of this project when looking for a command line parser, so I thought the best way to remember it is by writing a blog post on the subject.

The github repository for PowerArgs has excellent documentation, so I will simply cover a few of the basics here, just to get things started.

PowerArgs is available via Nuget using Install-Package PowerArgs.

With PowerArgs we can define a class for our command line arguments and using PowerArgs attribute we define required arguments, optional arguments, argument descriptions and many other options. One very useful options is ArgExistingFile which tells PowerArgs the argument is a filename and it should exist.

Let’s look at some simple code. This class acts as my command line arguments for a simple Csv to Xml file application

public class Arguments
{
   [ArgRequired]
   [ArgExistingFile]
   [ArgDescription("The mapping file")]
   public string MappingFile { get; set; }

   [ArgRequired]
   [ArgExistingFile]
   [ArgDescription("The CSV file to convert")]
   public string CsvFile { get; set; }

   [ArgRequired]
   [ArgDescription("The output XML file")]
   public string XmlFile { get; set; }
}

In our Main method we’d then having something like this

try
{
   var arguments = Args.Parse<Arguments>(args);
   // use the arguments
}
catch (ArgException e)
{
   Console.WriteLine(ArgUsage.GenerateUsageFromTemplate<Arguments>());
}

In the above, we parse the args using the Parse method which will ensure the ArgRequired properties are supplied and the files exist via ArgExistingFile. If any required arguments are missing an ArgException occurs and we use ArgUsage.GenerateUsageFromTemplate to output a list of the command line arguments expect, as well as description of the arguments and we can also list examples.

Go look at the github repository PowerArgs for further documentation.

.NET CLR Version Tool

I’d not come across this tool before, but whilst checking the location of the sn.exe tool I spotted it and thought I’d see what it did.

So the clrver.exe can be run from the Visual Studio command prompt or found at a location where your SDK exists, for example

"%ProgramFiles%\\Microsoft SDKs\Windows\v8.0A\bin\NETFX 4.0 Tools\clrver.exe"

Just running clrver.exe without any arguments will tell you which versions of the CLR are installed on your machine, whilst using the switch -all will result in a list of applications/processes running using the .NET CLR and tells us the version they’re using. If you already know the process id (pid) you can use clrver 123 to list the .NET CLR version being used by pid 123.

WeakReferences in .NET

By default when we create an object in .NET we get, what’s known as, a strong reference to that object (in truth we usually simply refer to this as a reference, and omit the word strong). Only when the object is no longer required, i.e. it’s out of scope or no longer referenced can that object be garbage collected.

So for example

// object o can be garbage collected any time after leaving this method call
public void Run()
{
   var o = new MyObject();

   o.DoSomething();
}

// object o can be garbage collection only after the SomeObject is no longer used, 
//i.e. no longer any strong references to it exist
public class SomeObject
{
   private MyObject o = new MyObject();
}

This is fine in many cases, but the canonical example of WeakReference use is, what if MyObject is a large object and an instance of it is held for a long time. It might be that whilst it’s a large object it’s actually rarely used in which case it would be more efficient from a memory point of view if it’s memory was reclaimed if no strong references to it existed and then recreated when required.

If we’re happy for our object to be garbage collected and regenerated/recreated at a later time then we could instead create them as WeakReferences. Obviously from the examples above the method call is not a good fit for using a WeakReference as (assuming the method exists) this instance of MyObject will be garbage collected after the method exits and assuming no further strong references to the object exist. But let’s see how we might create a WeakReference stored as part of an object that might hang around in memory a while.

public class SomeObject
{
   private WeakReference o = new WeakReference();

   public void Run()
   {
      var myObject = GetMyObject();
      // now use myObject
   }

   private MyObject GetMyObject()
   {
      if(o.Target == null)
      {
         o.Target = new MyObject();
      }
      return o.Target;
   }
}

So in the above example we create a WeakReference, it’s best that we go through a helper method to get the instance of the object held within the weak reference, because we can then check whether the object still exists and if it doesn’t recreate it. Hence the GetMyObject method call.

So in GetMyObject, we check whether the weak reference’s Target property is null. In other words, either the data stored within the WeakReference has not been created or it has been garbage collected and we now need to recreate it. So assuming it’s Target is null we create the theoretically large object and assign it to the Target property. Otherwise we simply return the Target property.

At this point it appears we’re just creating something like the Lazy type. But the key difference is that unlike a Lazy type which creates an object when needed then holds a strong reference, the WeakReference not only creates the object when needed but also allows the garbage collection process to free the memory if it appears to be no longer needed. So obviously don’t store something in the WeakReference that tracks current state in your application unless you are also persisting such data to a more permanent store.

References

https://msdn.microsoft.com/en-us/library/system.weakreference(v=vs.110).aspx
https://msdn.microsoft.com/en-us/library/system.weakreference.target(v=vs.110).aspx
https://msdn.microsoft.com/en-us/library/ee787088%28v=vs.110%29.aspx

Console.ReadKey and Console.WriteLine deadlock

So, let’s assume we have the following code

static void Main(string[] args)
{
   var interval = Observable.Interval(TimeSpan.FromSeconds(1));

   interval.Subscribe(v => Console.WriteLine("*"));

   Console.ReadKey();
}

There’s nothing too exciting here, We’re using Reactive Extensions to periodically (every second) produce a message which we subscribe to and where we output an asterix to the console when the message arrives at the subscriber. The message is produced on it’s own thread, so we have the main thread running the Main method and the second thread should be outputting an asterix every second.

However if you run this you may well find that there’s no asterixes written to the console.

So what’s the deal ?

Whilst searching around for an answer I came across the reply to the StackOverflow question Strange behaviour of Console.ReadKey() with multithreading. Although my code is very different to that shown by the person asking the question, the reason for the failure to output an asterix to the console is the same.

Basically there’s a race condition which then causes a deadlock.

To verify this I hooked up Visual Studio to the .NET source using the information contained at Configure Visual Studio 2013 for debugging .NET framework, ran the code and then took a look at the code to verify the previously mentioned StackOverlow post…

What happens is that in a situation when the Console.ReadKey is called before Console.WriteLine is…

  • Console.ReadKey takes a lock on the Console InternalSyncObject and basically loops until it gets input
  • Next Console.WriteLine runs in it’s own thread and Console.WriteLine calls the Console.Out property which in turn calls Console.InitializeStdOutError (private static method) if Console.Out hasn’t yet been created/initialized
  • Console.InitializeStdOutError then attempts to obtain a lock on InternalSyncObject which, as we know has already been taken by Console.ReadKey, causing a deadlock until the user presses a key

To get around this we just need to stop Console.ReadKey being hit before Console.Out is initialized, which means we could call Console.WriteLine, Console.Write or Console.Out prior to Console.ReadKey and everything will work correctly as the Console.Out will have been created (and thus InternalSyncObject locked and released) prior to Console.ReadKey going into it’s loop.

So with the following code everything should work fine

static void Main(string[] args)
{
   var interval = Observable.Interval(TimeSpan.FromSeconds(1));

   Console.Write(String.Empty); // write nothing, but causes Console.Out to be initialized

   interval.Subscribe(v => Console.WriteLine("*"));

   Console.ReadKey();
}

Pulse and Wait

The Monitor class contains the static methods Pulse and Wait. Well it has more than those two ofcourse, but these are the two this post is interested in.

Both methods must be enclosed in a lock or more explicitly the object that we pass to Pulse and Wait must have had a lock acquired against it. For example

private readonly object sync = new object();

// thread A
lock(sync)
{
   Monitor.Wait(sync);
}

// thread B
lock(sync)
{
   Monitor.Pulse(sync);
}

In the above code, we have two threads. Thread A acquires a lock on sync and then enters a Wait, at which point the lock is released but the thread now blocks until it reacquires the lock. Meanwhile, thread B acquires the lock on the sync object. It calls Monitor.Pulse which basically moved the waiting thread (thread A) to the ready queue and when thread B releases the lock (i.e. exits the lock block) then the next ready thread (in this case thread A) reacquires the lock and any code after the Monitor.Wait would be executed until it exits the lock block and releases the lock.

Producer-consumer pattern

Okay, I’ll admit to a lack of imagination – the producer consumer pattern also known as the producer consumer queue, is a standard sample for showing Pulse and Wait in use and the excellent Threading C# posts by Joseph Albahari are probably the best place to look for all C# threading information, but we’re going to walk through the producer consumer queue here anyway.

So the producer consumer queue is simply put, a queue whereby multiple threads may add to the queue and as data is added a thread within the queue does something with the data, see Producer–consumer problem.

This example creates one or more threads (as specified in the threadCount) which will be used to process of items from the queue.

The Enqueue method locks then adds the action to the queue and then pulses. The “processing” threads wait for the lock on the sync object being released and makes sure there are still items in the queue (otherwise the threads goes into a wait again). Assuming there is an item in the queue we get the item within the lock to ensure no other thread will have removed it then we release the lock and invoke the action before checking for more items to be processed.

public class ProducerConsumerQueue
{
   private readonly Task[] tasks;
   private readonly object sync = new object();
   private readonly Queue<Action> queue;

   public ProducerConsumerQueue(int threadCount, CancellationToken cancellationToken)
   {
      Contract.Requires(threadCount > 0);

      queue = new Queue<Action>();
      cancellationToken.Register(() => Close(false));

      tasks = new Task[threadCount];
      for (int i = 0; i < threadCount; i++)
      {
         tasks[i] = Task.Factory.StartNew(Process, TaskCreationOptions.LongRunning);
      }
   }

   public void Enqueue(Action action)
   {
      lock (sync)
      {
         queue.Enqueue(action);
         Monitor.Pulse(sync);
      }
   }

   public void Close(bool waitOnCompletion = true)
   {
      for (int i = 0; i < tasks.Length; i++)
      {
         Enqueue(null);
      }
      if (waitOnCompletion)
      {
         Task.WaitAll(tasks);
      }
   }

   private void Process()
   {
      while (true)
      {
         Action action;
         lock (sync)
         {
            while(queue.Count == 0)
            {
                Monitor.Wait(sync);
            }
            action = queue.Dequeue();
         }
         if (action == null)
         {
            break;
         }
         action();
      }
   }
}

Here’s a simple sample of code to interact with the ProducerConsumerQueue

CancellationTokenSource cts = new CancellationTokenSource();

ProducerConsumerQueue pc = new ProducerConsumerQueue(1, cts.Token);
pc.Enqueue(() => Console.WriteLine("1"));
pc.Enqueue(() => Console.WriteLine("2"));
pc.Enqueue(() => Console.WriteLine("3"));
pc.Enqueue(() => Console.WriteLine("4"));

// various ways to exit the queue in an orderly manner
cts.Cancel();
//pc.Enqueue(null);
//pc.Close();

So in this code we create a ProducerConsumerQueue with a single thread, actions are added to the queue via Enqueue and as the ProducerConsumerQueue has only a single thread and as all the items were added from a single thread, each item is simply invoked in the order they were added to the queue. However we could have been adding from multiple threads as the ProducerConsumerQueue is thread safe. Had we created the ProducerConsumerQueue with multiple threads then the order of processing may also be different.

Waiting for a specified number of threads using the CountdownEvent Class

In a previous post I discussed the Barrier class. The CountdownEvent class is closely related to the Barrier in that it can be used to block until a set number of signals have been received.

One thing you’ll notice is that the signal on the Barrier is tied to a wait, in other words you
SignalAndWait on a thread, whereas the CountdownEvent has a Signal and a separate Wait method, hence threads could signal they’ve reach a point then continue anyway or call Wait after they call Signal equally a thread could signal multiple times.

The biggest difference with the Barrier class is that the CountdownEvent does not automatically reset once the specified number of signals have been received. So, once the CountdownEvent counter reaches zero any attempt to Signal it will result in an InvalidOperationException.

You can increment the participants (or in this case we say that we’re incrementing the count) at runtime, but only before the counter reaches zero.

Time for some sample code

class Program
{
   static CountdownEvent ev = new CountdownEvent(5);
   static Random random = new Random();

   static void Main()
   {
      Task horse1 = Task.Run(() => 
            GetReadyToRace("Horse1", random.Next(1, 1000)));
      Task horse2 = Task.Run(() => 
            GetReadyToRace("Horse2", random.Next(1, 1000)));
      Task horse3 = Task.Run(() => 
            GetReadyToRace("Horse3", random.Next(1, 1000)));
      Task horse4 = Task.Run(() => 
            GetReadyToRace("Horse4", random.Next(1, 1000)));
      Task horse5 = Task.Run(() => 
            GetReadyToRace("Horse5", random.Next(1, 1000)));

      Task.WaitAll(horse1, horse2, horse3, horse4, horse5);
   }

   static void GetReadyToRace(string horse, int speed)
   {
      Console.WriteLine(horse + " arrives at the race course");
      ev.Signal();

      // wait a random amount of time before the horse reaches the starting gate
      Task.Delay(speed);
      Console.WriteLine(horse + " arrive as the start gate");
      ev.Wait();
  }
}

In the above code, each thread signals when it “arrives at the race course”, then we have a delay for effect and then we wait for all threads to signal. The CountdownEvent counter is decremented for each signal and threads wait until the final thread signals the CountdownEvent at which time the threads are unblocked and able to complete.

As stated previously unlike the Barrier once all signals have been received there’s no reset of the CountDownEvent, it has simply finished it’s job.

Code Contracts

Elevator pitch

Code contracts allow us to define preconditions, post conditions and the likes in our code. These are constructs I first read about when doing a (very) little bit of Eiffel development.

Visual Studio additions

Before we get into Code Contracts let’s just note that they are available in .NET 4.5 but we need to enable their usage and here we need to install a couple of additions to Visual Studio.

So if you’re running Visual Studio 2012 goto Tools | Extensions and Updates and search online for Code Contract Tools – install this. Then find Code Contracts Editor Extensions VS2012 and install this. You’ll have to restart Visual Studio to see the new installs working. So go ahead and do that.

If Visual Studio 2013 I just install Code Contract Tools.

This will install some addition static analysis tools etc. which we’ll look at later, for now let’s code…

Using Code Contracts

Predconditions

Often in our code we’ll have methods which check the arguments passed to a method or possibly check class wide variables have been set etc. Usually the code might take the following form

public void Analyze(string data)
{
   if(data == null)
      throw new ArgumentNullException("data");

   // do something
}

Well first off we can now replace this with a Code Contract precondition. The precondition ofcourse states that for this method to run, this or these preconditions must be met.

public void Analyze(string data)
{
   Contract.Requires<ArgumentNullException>(line != null)
}

We could have specified the same precondition without the ArgumentNullException and a ContractException would be thrown instead.

Postconditions

A postcondition specifies the state of a method when it ends. For example, we can declare that we’ll never return a null by declaring the following contract

public static string Analyze(string line)
{
   Contract.Requires<ArgumentNullException>(line != null);
   Contract.Ensures(Contract.Result<string>() != null);
   
   // do something
   
   return result;
}

Now if we actually did attempt to return null we’d get a ContractException.

Assumptions

So, we can define pre and postconditions, but also we can define assumptions. Basically let’s assume we have our Analyze method (above) but without the pre or postconditions and we want to declare an assumption that the result from Analyze will not be null, we could write

public string Analyze(string data)
{
   // do something

   return result;
}

static void Main(string[] args)
{
   string result = Analyze("Some Text");

   Contract.Assume(result != null);

   // do something with result
}

Obviously if result is set to null, our contract is violated and the ContractException will be thrown.

Object Invariants

An object invariant basically defines the state of an object. So for example, let’s assume we have a Person class as below

public class Person
{
   public Person(string firstName, string lastName)
   {
      FirstName = firstName;
      LastName = lastName;
   }

   public string FirstName { get; set; }
   public string LastName { get; set; }
}

It may be that we want to define a contract which states that FirstName and LastName are not null. We could add preconditions to the constructor – but what if we also wanted to define a contract that says FirstName and LastName are never null, not just within the contructor’s scope.

Now we can add a ContractInvariantMethod as below

public class Person
{
   public Person(string firstName, string lastName)
   {
      FirstName = firstName;
      LastName = lastName;
   }

   public string FirstName { get; set; }
   public string LastName { get; set; }

   [ContractInvariantMethod]
   private void ContractInvariant()
   {
      Contract.Invariant(FirstName != null && LastName != null);
   }
}

Note: We could have had the left and right parts of the && in different Contract.Invariant declarations if we wished, so basically we can have multiple Contract.Invariant in this method.

The method name can be anything you like and can only contain Contract.Invariant statements. A compile time error will occur if you have other code in this method.

Now when we create a Person object if either FirstName or LastName are set to null a ContractException will occur, but better still let’s assume we add a new method to the class which sets one of these variables to null. The ContractInvariantMethod will be called and the contract will be violated leading to a ContactException.

Contracts on interfaces

An interesting use of contracts is with interfaces. Basically if we define an interface it would be nice if we could declare the contract definitions against the interface. Then any implementation could “inherit” the contract definitions.

So let’s look at our Person class again but now add an interface and define some contracts on the interface

[ContractClass(typeof(IPersonContract))]
public interface IPerson
{
   string FirstName { get; set; }
   string LastName { get; set; }			
}

[ContractClassFor(typeof(IPerson))]
internal abstract class IPersonContract : IPerson
{
   public string FirstName
   {
      get 
      { 
         Contract.Ensures(Contract.Result<string>() != null);
         return default(string); 
      }
      set { Contract.Requires(value != null); }
   }

   public string LastName
   {
      get 
      { 
         Contract.Ensures(Contract.Result<string>() != null);
         return default(string); 
      }
      set { Contract.Requires(value != null); }
   }
}

public class Person : IPerson
{
   public Person(string firstName, string lastName)
   {
      FirstName = firstName;
      LastName = lastName;
   }

   public string FirstName { get; set; }
   public string LastName { get; set; }
}

So the interface is a standard interface except that we are declaring (using the ContractClass attribute) that the IPersonContract class includes contract definitions for the interfaces. Likewise on the IPersonContract abstract class. Yes, it’s mean’t to be abstract and implement the interface. Now when we return values, just use default(<type>) so all will compile and add your pre or postconditions. Unfortunately it doesn’t look like we can add a ContractInvariantMethod method to this abstract class (or at least we can but it doesn’t appear to do anything in the current release of .NET).

ContractAbbreviator

The ContractAbbreviatorAttribute is used on a method to denote it contains contract data which, when the method is called from another method in essence runs the contract checks in that method – eh ? It’s a bit like creating a C style macro and having it inlined in a method – eh ?

It’s a was to reuse contracts.

Okay let’s see an example

public class Person
{
   public string FirstName { get; set; }
   public string LastName { get; set; }

   [ContractAbbreviator]
   private void Validate()
   {
      Contract.Ensures(FirstName != null);
      Contract.Ensures(LastName != null);
   }

   public void Amend()
   {
      Validate();
   }
}

Now when we run Amend (with FirstName and/or LastName set to null) the exception occurs from the Amend method, as if the Contract.Ensure’s replaced the Validate call.

Based upon the above information it would come as no surprise to find out that when the ContractAbbreviator attribute is used on a method (and ofcourse the method is called from another method), the compiler does indeed place the contract code into the calling method and replaces the instruments within the Validate method (in the above example) with a nop. When the attribute is not included we really are calling another method.

Obviously this means we can reuse the Amend method now in more than one method (or more specifically the contracts contained within the method), so we’ve created a reusable set of contracts. If you remove the ContractAbbreviator attribute then run as previously suggested, the validation failure will occur in the Validate method.

Visual Studio Plugin

At the start of this post we installed a couple of code contract specific plugins (in VS2012 and only one in VS2013). These added the property editor into the project properties for enabling etc. Code Contracts. But they also installed editor warnings.

If you check the “Perform Static Contract Checking” option in the Code Contract property page, then rebuild your project. Static analysis will take place and may display warnings. For example if your code looked like this 

public void Analyze(string data)
{
   Contract.Requires<ArgumentNullException>(line != null)
}

static void Main(string[] args)
{
   Analyze(null);
}

We would get warnings stating CodeContracts: requires is false: line != null. Along with a red squiggly underline of the code Analyze(null) and a tooltip popup showing the warning message.

So giving us an indication early on of possible contract violations.

References

Code Contracts User Manual
Code Contracts

Some Linq Patterns

What follows is a list of a few patterns for specific tasks using LINQ

Combining two lists

We can combine two lists, where duplicates are removed using

var a = new List {0, 2, 3, 4, 5, 6};
var b = new List { 0, 1, 3, 4, 6, 6 };

var union = a.Union(b);
[/code]

Find the common items in both lists

var a = new List<int> {0, 2, 3, 4, 5, 6};
var b = new List<int> { 0, 1, 3, 4, 6, 6 };

var intersection = a.Intersect(b);

we can also use the lambda expression join

var l1 = new List<int> {1, 2, 4, 5};
var l2 = new List<int> { 0, 2, 4, 6 };

var r = from a in l1 join b in l2 on a equals b select a;

Find the differences between two lists

This will find the items in the first list which are not in the second list

var listA = new List<int> {0, 2, 3, 4, 5, 6};
var listB = new List<int> { 0, 1, 3, 4, 6, 6 };

var differences = from a in listA where listB.All(i => i != a) select a;

Combining every item in one list with every item in another list

Here we’re going to do a cross join, where the output enumerable is a made up of each element from the first list, combined with each element of the second list

var l1 = new List<int> {1, 2};
var l2 = new List<int> { 0, 2 };

var crossjoin = from a in l1
       from b in l2
       select new {a, b};

The output would be

1 0
1 2
2 0
2 2

I’ll add more as and when I remember to add them.

Creating a custom Linq Provider

To create a custom LINQ provider we need to implement the interfaces, IQueryable or IOrderedQueryable and IQueryProvider.

This is a summary of several posts/blogs I’ve read on the subject and source code available on github etc. See References below.

IQueryable<T>

To implement a minimal LINQ provider we need to implement IQueryable<>. This interface inherits from IEnumerable. The actual IQueryable methods are minimal

  1. ElementType
  2. Expression
  3. Provider

IOrderedQueryable<T>

If we want to support the sorting query operators, then we need to implement IOrderedQueryable<T>.

IOrderedQueryable inherits from IEnumerable, IEnumerable<T>, IOrderedQueryable, IQueryable, and IQueryable<T>.

We can implement an IOrderedQueryable that’s reusable for most situations, as follows

public class Queryable<T> : IOrderedQueryable<T>
{
   public Queryable(IQueryContext queryContext)
   {
      Initialize(new QueryProvider(queryContext), null);
   }

   public Queryable(IQueryProvider provider)
   {
     Initialize(provider, null);
   }

   internal Queryable(IQueryProvider provider, Expression expression)
   {
      Initialize(provider, expression);
   }

   private void Initialize(IQueryProvider provider, Expression expression)
   {
      if (provider == null)
         throw new ArgumentNullException("provider");
      if (expression != null && !typeof(IQueryable<T>).
             IsAssignableFrom(expression.Type))
         throw new ArgumentException(
              String.Format("Not assignable from {0}", expression.Type), "expression");

      Provider = provider;
      Expression = expression ?? Expression.Constant(this);
   }

   public IEnumerator<T> GetEnumerator()
   {
      return (Provider.Execute<IEnumerable<T>>(Expression)).GetEnumerator();
   }

   System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
   {
      return (Provider.Execute<System.Collections.IEnumerable>(Expression)).GetEnumerator();
   }

   public Type ElementType
   {
      get { return typeof(T); }
   }

   public Expression Expression { get; private set; }
   public IQueryProvider Provider { get; private set; }
}

Note: The IQueryContext is not part of the LINQ interfaces but one I’ve created to allow me to abstract the actual “custom” part of the code and allow maximum reuse of default functionality.

IQueryProvider

The IQueryProvider basically requires two methods (a generic and non-generic version of both) to be implemented. Obviously the generic versions are for strongly typed and the non-generic are loosely typed.

CreateQuery

The CreateQuery method is used to create a new instance of an IQueryable based upon the supplied expression tree.

Execute

The Execute method is used to actually execute a query expression, i.e. in a custom provider we will get the data from the datasource, for example a webservice or database etc. and it’s in the Execute method that we both get the data and do any conversion to, say SQL or the likes (if the LINQ query ultimately queries some other system.

An example of an implementation for an IQueryProvider might look like this…

public class QueryProvider : IQueryProvider
{
   private readonly IQueryContext queryContext;

   public QueryProvider(IQueryContext queryContext)
   {
      this.queryContext = queryContext;
   }

   public virtual IQueryable CreateQuery(Expression expression)
   {
      Type elementType = TypeSystem.GetElementType(expression.Type);
      try
      {
         return                
            (IQueryable)Activator.CreateInstance(typeof(Queryable<>).
                   MakeGenericType(elementType), new object[] { this, expression });
      }
      catch (TargetInvocationException e)
      {
         throw e.InnerException;
      }
   }

   public virtual IQueryable<T> CreateQuery<T>(Expression expression)
   {
      return new Queryable<T>(this, expression);
   }

   object IQueryProvider.Execute(Expression expression)
   {
      return queryContext.Execute(expression, false);
   }

   T IQueryProvider.Execute<T>(Expression expression)
   {
      return (T)queryContext.Execute(expression, 
                 (typeof(T).Name == "IEnumerable`1"));
   }
}

Note: Again, the IQueryContext is not part of the LINQ interfaces but one I’ve created to allow me to abstract the actual “custom” part of the code and allow maximum reuse of default functionality.

The TypeSystem class is taken from the post Walkthrough: Creating an IQueryable LINQ Provider

IQueryContext

The above shows some LINQ interfaces and some sample implementations. I’ve pointed out that IQueryContext is not part of LINQ but is instead something I’ve created (based upon reading various other implementations) to allow me to abstract the actual LINQ provider code specific to my provider’s implementation. Ofcourse we could have derived from QueryProvider, but for now we “plug-in” the data context instead. To change the implementation to derive from QueryProvider simple remove the IQueryContext (also from the Queryable implementation) and override the Execute methods.

For now I’ll continue this post using the IQueryContext, so here’s the interface

public interface IQueryContext
{
   object Execute(Expression expression, bool isEnumerable);
}

Implementing the IQueryContext

Whether the implementation of the actual Execute code (on the IQueryProvider) is abstracted into the IQueryContext or implemented within an alternate implementation of IQueryProvider. This is where the fun of actually “running” the LINQ query against your custom provider takes place.

When writing something like this…

var ctx = new Queryable<string>(new CustomContext())

var query = from s in ctx where s.StartsWith("T") select s;

what we are really doing is creating the query. Hence the CreateQuery methods on the IQueryProvider are called, but the actual data source or whatever supplies the data for your custom LINQ provider should not be called until we reach the execution phase, this is known as deferred execution. The execution phase takes place when we enumerate over the query or call methods, such as Count() etc. against a query.

foreach (var q in query)
{
   Console.WriteLine(q);
}

So at the point that we GetEnumerator implicitly via the foreach loop, that’s when the Execute method is called.

Executing the query

So, the Execute method is called and we will have an expression tree defined by LINQ supplied as an argument. We now need to translate that query into something useful, i.e. turn it into an SQL query for a custom database LINQ provider and then get the data for this query, or get data from a webservice and allow the query to be executed against the returned data etc.

As the actual decoding of the Expression is a fairly large subject in itself, I’ll leave that for another post, but suffice to say, there’s a lot we need to implement to duplicate some if not all of the “standard” LINQ query operators etc.

References

Walkthrough: Creating an IQueryable LINQ Provider
LINQ: Building an IQueryable provider series
LINQ and Deferred Execution

Separating out configuration files in .NET

Occasionally we might wish to de-clutter our App.config by moving parts of the configuration into separate files. Ofcourse this process is also useful where we might wish to simply reuse existing config parts.

This capability is built into .NET, so for example we can have

<appSettings configSource="MyAppSettings.config" />

We can also use the appSettings attribute file, for example

<appSettings file="MyAppSettings.config" />

The MyAppSettings.config file might look something like

<?xml version="1.0" encoding="utf-8" ?>
<appSettings>
  <add key="configuration" value="..\..\..\..\Samples\SpaceMonitor.xml"/>
  <add key="artifacts" value="..\..\..\..\..\Samples\artifacts"/>
</appSettings>

Note: the appSettings section starts the file (after the xml declaration), i.e. we do not have a configuration section within this file.

So this is cool, appSettings shows the configSource attribute in intellisense in Visual Studio, but we can in fact use the configSource on any section. Maybe we’ve created a section called CustomSection, then it implicitly has a configSource attribute built in.

It’s important though to note that sectionGroups do not have an implicit configSource. Say we have something like

<configSections>
  <sectionGroup name="orchestrator">
    <section name="aliases" type="Orchestrator.Configuration.AliasesSectionHandler,   
                                  Orchestrator.Configuration"/>
    <section name="plugins" type="Orchestrator.Configuration.PluginsSectionHandler, 
                                  Orchestrator.Configuration"/>
  </sectionGroup>
</configSections>

Now you might think you can write <orchestrator configSource=”Orchestrator.config”/> but I’m afraid not.

Only sections can use the configSource

Instead what we’d need to do is something like this

<orchestrator>
  <aliases configSource="Alias.config" />
  <plugins configSource="Plugins.config"/>
</orchestrator>

and the actual files would look something like

<?xml version="1.0" encoding="utf-8" ?>
<aliases>
  <alias key="cron" value="StandardTriggers.CronTrigger, StandardTriggers"/>
</aliases>

and

<?xml version="1.0" encoding="utf-8" ?>
<plugins>
  <plugin type="StandardPlugins.WCFMonitorPlugin, StandardPlugins"/>
</plugins>

So with the above information you can see that if you wanted to separate out a system.serviceModel, for example, you would only be able to separate the behaviors, services etc. as the system.serviceModel is a sectionGroup.