Monthly Archives: August 2014

Using protobuf-net in C#

protobuf-net is a .NET library around the Google Protocol Buffers.

For information on the actual Google Protocol Buffers you can checkout the Google documentation.

To get the library via NuGet you can use Install-Package protobuf-net from the Package Manager Console or locate the same from the NuGet UI.

To quote Implementing Google Protocol Buffers using C#, “Protocol Buffers are not designed to handle large messages. If you are dealing in messages larger than a megabyte each, it may be time to consider an alternate strategy. Protocol Buffers are great for handling individual messages within a large data set. Usually, large data sets are really just a collection of small pieces, where each small piece may be a structured piece of data.”

So Protocol Buffers are best used on small sets of of data, but let’s start coding and see how to use Protocol Buffers using protobuf-net.

Time to code

There are a couple of ways of making your classes compliant with the protobuf-net library, the first is to use attributes and the second without attributes, instead setting up the meta data yourself.

Let’s look at an example from the protobuf-net website, a Person class which contains an Address class and other properties.

[ProtoContract]
public class Person 
{
   [ProtoMember(1)]
   public int Id { get; set; }
   [ProtoMember(2)]
   public string Name { get; set; }
   [ProtoMember(3)]
   public Address Address { get; set;}
}

[ProtoContract]
public class Address 
{
   [ProtoMember(1)]
   public string Line1 {get;set;}
   [ProtoMember(2)]
   public string Line2 {get;set;}
}

As can be seen the classes to be serialized are marked with the ProtoContractAttribute. By default protobuf-net expects you to mark your objects with attributes but as already mentioned, you can also use classes without the attributes as we’ll see soon.

The ProtoMemberAttribute marks each property to be serialized and should contain a unique, positive integer. These identifiers are serialized as opposed to the property name itself (for example) and thus you can change the property name but not the ProtoMemberAttribute number. Obviously, as this value is serialized the smaller the number the better, i.e. a large number will take up unnecessary space.

Serialize/Deserialize

Once we’ve defined the objects we want to serialize with information to tell the serializer the id’s and data then we can actually start serializing and deserializing some data. So here goes. Assuming that we’ve created a Person object and assigned it to the variable person then we can do the following to serialize this instance of the Person object

using (var fs = File.Create("test.bin"))
{
   Serializer.Serialize(fs, person);
}

and to deserialize we can do the following

Person serlizedPerson;
using (var fs = File.OpenRead("test.bin"))
{
   Person person = Serializer.Deserialize<Person>(fs);
   // do something with person
}

Note: Protocol Buffers is a binary serialization protocol

Now without attributes

As mentioned previously, we might not be able to alter the class definition or simply prefer to not use attributes, in which case we need to setup the meta data programmatically. So let’s redefine Person and Address just to be perfectly clear

public class Person 
{
   public int Id { get; set; }
   public string Name { get; set; }
   public Address Address { get; set;}
}
    
public class Address 
{
   public string Line1 {get;set;}
   public string Line2 {get;set;}
}

Prior to serialization/deserialization we would write something like

var personMetaType = RuntimeTypeModel.Default.Add(typeof (Person), false);
personMetaType.Add(1, "Id");
personMetaType.Add(2, "Name");
personMetaType.Add(3, "Address");

var addressMetaType = RuntimeTypeModel.Default.Add(typeof(Address), false);
addressMetaType.Add(1, "Line1");
addressMetaType.Add(2, "Line2");

as you can see we supply the identifier integer and then the property name.

RuntimeTypeModel.Default is used to setup the configuration details for our types and their properties.

Inheritance

Like the SOAP serialization and the likes, when we derive a new class from a type we need to mark the base type with an attribute telling the serializer what types it might expect. So for example if we added the following derived types

[ProtoContract]
public class Male : Person
{		
}

[ProtoContract]
public class Female : Person
{	
}

we’d need to update our Person class to look something like

[ProtoContract]
[ProtoInclude(10, typeof(Male))]
[ProtoInclude(11, typeof(Female))]
public class Person 
{
   // properties
}

Note: the identifiers 10 and 11 are again unique positive integers but must be unique to the class, so for example no other ProtoIncludeAttribute or ProtoMemberAttribute within the class should have the same identifier.

Without attributes we simply AddSubType to the personMetaType defined previous, so for example we would add the following code to our previous example of setting up the metadata

// previous metadata configuration
personMetaType.AddSubType(10, typeof (Male));
personMetaType.AddSubType(11, typeof(Female));

// and now add the new types
RuntimeTypeModel.Default.Add(typeof(Male), false);
RuntimeTypeModel.Default.Add(typeof(Female), false);

Alternative Implementations of Protocol Buffers for .NET

protobuf-csharp-port is written by Jon Skeet and appears to be closer related to the Google code using .proto files to describe the messages.

Exploring Akavache

Akavache is a key-value store/cache. It’s compatible with flavours of Windows including the desktop, WinRT and Windows Phone 8 as well as iOS, Mac and Android via Xamarin. It supports async await as well as RX observable.

Let’s code

We interact with Akavache via the BlobCache static class but before we do anything with Akavache we need to setup the application name. So at the first opportunity, such as when your applications starts you should supply Akavache with your application name.

BlobCache.ApplicationName = "MyApplicationName";

There are several different caches that we can use on the BlobCache object. Each implements the IBlobCache interface.

  • The InMemory property is used for caching anything which is to be stored (as the name suggests) in memory and thus lost when the application shuts down.
  • The LocalMachine is used to store data that is not related to the user’s account.
  • The Secure is encrypted and is used to anything you want to store in a more secure manner.
  • Finally the UserAccount is used to store data on a per user basis and can be part of a roaming profile.

As stated on the Akavache readme.md, “on the desktop, your application’s data will be stored in the %AppData%\[ApplicationName] and %LocalAppData%\[ApplicationName]”.

Inserting data into the cache

There are several insertion methods (instance and extension methods) and overloads for each. I’m not going to go through each method and their overloads, instead let’s look at the insertion method you’ll most likely start with – inserting a single object.

BlobCache.UserAccount.InsertObject("someObject", new SomeObject());

We supply a key, in this case the string someObject followed by an object. We can optionally supply a DateTimeOffset or TimeSpan to use as an expiration time, i.e. when the data is ejected from the cache.

Ofcourse, if we insert more than one item with the same key, we’re simply going to overwrite the existing item, therefore updating the cache with the last inserted value.

Getting data from the cache

Once we’ve inserted items in the cache we’ll need a way to retrieve them. As can be seen from the InsertObject method we supply a key for our cached item. So we obviously use the same key to get the value from the cache. Akavache uses an asynchronous model for getting items from a cache so uses either an IObservable (the RX way) or getting values using async await. Here’s an example using the RX method

SomeObject result = null;
BlobCache.UserAccount.GetObject<SomeObject>("someObject")
   .Subscribe(i => result = i);

and here’s the same thing using async await

SomeObject result = await BlobCache.UserAccount.GetObject<SomeObject>("someObject")

Getting data which is currently not cached

Obviously we don’t always start an application with the items already in the cache. So a common requirement is to query the cache for a value, if it doesn’t exist then get it from somewhere and then place it in the cache. Akavache offers a GetOrFetchObject method which allows the developer to to do just this. First we try to get an item from the cache and if it does not exist in the cache we can supply a Func<Task<T>> or Func<IObservable<T>> to get the item from somewhere else and it’s then automatically added to the cache for us.

SomeObject result = null;
BlobCache.UserAccount.GetOrFetchObject(
   "someObject", 
   () => Task.Factory.StartNew(() => new SomeObject()))
   .Subscribe(o => result = o);

// or using async await

SomeObject result = await BlobCache.UserAccount
   .GetOrFetchObject("someObject", 
   () => Task.Factory.StartNew(() => new SomeObject()));

We can also pass an expiration DateTimeOffset to the above method.

Note: GetOrFetchObject and GetOrCreateObject appear to do the same thing, both calling GetOrFetchObject which takes a Func<IObservable>. The former simply converts a Task<T> to an and IObservable<T> and the latter converts a Func<T> to an IObservable<T> also.

An interesting other “get” method to look at is GetAndFetchLatest. This will attempt to get a value from the cache and at the same time will use the supplied Func to get the value from some alternate place. So for example, let’s say you’re going to get a value from a webservice. This will return the current cached version (if one exists) and at the same time try to get the current value from the webservice. Hence two items may be returned.

In use…

BlobCache.UserAccount.GetAndFetchLatest(
   "someObject", () => Task.Factory.StartNew(() => new SomeObject()))
   .Subscribe(i => result = i);

In the above code the result = i will be called twice (assuming we’ve already inserted an object with the key “someObject” into the cache). As you’d expect from the previous description of this method, the first call would contain cached “someObject” value and the second would be the new object we’ve created in the task.

Removing items from the cache

Apart from setting an expiration for items as they’re added to the cache we can also use the method Invalidate to remove specific key values from the cache or InvalidateAll to remove/clear all items from the cache.

For example

BlobCache.UserAccount.Invalidate("someObject");