Microsoft’s Data Programmability team listening to the community

While Microsoft’s Entity Framework continues to add new features with release of SP1.  It still doesn’t go anyway to supporting persistent ignorant domain models in the way the NHibernate does.  As i blogged about previously, this has been so unpopular that a PostSharp plugin has been developed to inject the nesseacary bits to support the EF at compile time.

However the good news is that a delegation of great minds including Eric Evans and Jimmy Nilsson (both with great DDD books) will be brain storming on ways to improve the next release – especially in relation to Domain Drive Design practices.

Do you think in LINQ?

If not you should download LINQPad from Joseph Albahari’s (of C#3 in a nutshell fame).  It’s a great tool to practicing LINQ expressions and has syntax highlighting (but unfortunatly no intellisense).  Alternatively if you really miss the old Query Analyzer from SQL 2000 days, then Query Express is a good option – super fast 100k stand-alone executable!

i4o (Indexed LINQ) adds support for POCO

Aaron Erickson has released a new version of his i4o (Indexed LINQ).  This great open source library dynamically hooks into the statically typed Expression Trees generated when querying with LINQ eg:

  var nonIndexedQuery = from item in list
      where item.DemoStringNonIndexed == nonIndexedString
      select item;
  var resultNonIndexed = nonIndexedQuery.ToList();

The library now supports indexing Plain Old C# objects (POCOs) such as PocoClass in the following example.  Attributes can still be applied to properties, but the new AddIndex method allows indexing objects that have no knowledge of i4o. 

var sampleSize = 1000000;
var list = new IndexableCollection();
for (int i = 0; i < sampleSize; i++) list.Add(new PocoClass() { DemoStringNonIndexed = Guid.NewGuid().ToString(), DemoStringIndexed = Guid.NewGuid().ToString() } ); list.AddIndex("DemoStringIndexed"); [/sourcecode] The biggest hit when using i4o is loading the data into the collection, then applying the index.  After that querying is markedly improved, in this example 85x faster then traditional select.

load 3725ms
index 3754ms
nonIndexedStopwatch 175ms
indexedStopwatch 2ms

This makes this an ideal library to be used in caching scenarios.  I wonder if it could be plugged into Microsoft’s new velocity distributed cache!

High performance Pub/Sub .NET libraries

In my search for a high-performance publish/subscribe library i again stumbled across the MS robotics CCR (Concurrency and Coordination Runtime).  The April CTP was just released an includes a number of new features including LINQ support.  The CCR remains stable and relatively unchanged, but new C# language features of lambdas and inline iterators open up new possibilities of writing streamlined async code and reduces problems with handling exceptions across threads due to the notion of Ports for success & failure scenarios.

Developers have typically steered clear of CCR given it is only bundled with MS Robotics studio (opting for alternatives such as the new TPL concurrency library), however the 200k CCR library can be shipped with your software at worst case costing US$2 per unit.

Some great examples include using CCR with ASP.NET to handle async requests showcasing iterators with yield keyword.  Another being a web crawler by Angle ‘Java’ Lopez which also leverages the DSS (decentralised software services).

An alternatives to this in-memory concurrent processing is publish/subscribe messaging over a reliable transport such as MSMQ some libraries include:

Async or concurrent programming is getting easier, and importantly more reliable.  Expect more posts and samples in the future.

Book Review: C# in depth

If your one of those people who dreads cramming for exams but loves understanding the nitty-gritty of a problem, then C# in depth is a book you should get your hands on.

C# in Depth

Jon Skeet an MVP for many years now working for Google has a great writing style the provokes questions to answers you thought you already knew.  As Eric Lippert puts it

Jon gets all that. This book is ideal for professional developers who have a need to understand the “what” and “how” of the latest revision to C#. But it is also for those developers whose understanding is enriched by exploring the “why” of the language’s design principles.  

And if you need a teaser, Jon has a number of articles on line on the subject matter covered in his book.  Highly recommended, 4/5 from me.

The art and science of performant Web 2.0 AJAX apps

With so many frameworks, libraries and open source AJAX widgets it’s easy for a well performing Web 1.0 site to be ‘upgraded’ into a poor performing web 2.0 site. Microsoft made it very easy to Ajaxify a web app with ASP.NET AJAX offering both 

  1. the update panel for plug & play AJAX including smarts for injecting and executing script blocks,
  2. support for writing script control behaviour and
  3. ability return JSON from script web services.

A good web developer will always keep performance in mind when wiring up a page (using tools such as FireBug and YSlow to pinpoint bottlenecks).  But with so many helpful high-level JS libraries such as Prototype, JQuery, DOJO etc. it can be easy to forget how expensive repeated DOM calls are. 

If nothing else, performance is a good reason to embrace the GWT and Script# toolkits – as they enable you to write compile checked code while outputting performant JS that doesn’t necessary have any reliance on an interpreted JS framework.

However perhaps more important then all of the above is how to make your application ‘feel’ like it loads quickly and is responsive.  This really comes down to understanding what information needs to be immediately available, what can be lazy-loaded/rendered, including scripts.  Joseph Smarr from plaxo has a good presentation on ‘Why Everything You’ve Been Taught is Wrong’.

Finally if that all sounds like to much hard-work, then look out for DOLOTO – a Microsoft research initiative which attempts to automagically lazy-load your app for you!

Concurrent programming lesson with Erlang

I recently watched a very good presentation by Joe Armstrong – the inventor of Erlang, the functionality language originally developped for Ericsson telecom infrastructure products.  He is a very eloquent speaker, taking through the history of computing, explaining that moore’s law trajectory may in fact be restored if Network On Chip architectures take off in a big way.

I really like messaging framework approach to parallelism and immutable types are at the heart of this approach – a native construct in functional language such as F# on the .net platform.  Joe Duffy has written an article on archiving a degree of immutability in C#.

I dug a little deeper and came across a great open source library ‘Retlang‘ written by Mike Rettig (it’s name borrowing heavily from Erlang).  It currently uses native .NET 2 constructs, but I’m sure would benefit from running on top of Microsoft’s new parallel initiatives, especially as they mature in the future.

Multi-Threading, Silverlight Sockets & Visualisation Part 3

Part 2 of this Silverlight visualisation instalment discussed how data could be streamed over a socket for rendering on the client.

To recap, Part 1 discussed how the Training Set data was loaded from file and stored in memory.  Well the data structure is as follows:

    private void SetImage(Stream stream)
        BitmapImage bi = new BitmapImage();
        this.Image1.Source = null;
        this.Image1.Source = bi;
    public struct TrianglePoints
        public int P1 { get; set; }
        public int P2 { get; set; }
        public int P3 { get; set; }

    public struct TrianglePoint
        public float X { get; set; }
        public float Y { get; set; }

    /// <summary>
    /// A container for all the data that needs to be sent over the wire to render the image
    /// </summary>
    public class TrainingSetData
        public int Index { get; set; }
        public TrianglePoint[] Points { get; set; }
        public TrianglePoints[] Triangles { get; set; }
        public int Height { get; set; }
        public int Width { get; set; }
        public byte[] Image { get; set; }
        public string ImageUrl { get; set; }
        public string ImageContentType { get; set; }

Notice these classes have been attributed so that can be XML serialized, but that the Image blob is ignored given it is sent over as a separate block.  In addition the Socket Server that broadcasts the training set data has a configuration flag to disable streaming the image at all.  In which case the client can download the image via the specified Url.

Silverlight supports loading an image in a number of ways

1.       Relative or Absoulte Uri’s.

2.       Set directly via resource stream (loaded from memory, file etc).

3.       Downloading on demand using the asynchronous web client.

Option 1 is not appropriate given my images are dynamic and change on each request, so I have implemented Option 2 (when image is streamed over socket), and Option3 (when downloading) as the following code illustrates:

    private void RenderImage(TrainingSetData data, byte[] image)
            if (image == null || image.Length == 0)
                // Fire off a request to the web service to download the image
                if (!String.IsNullOrEmpty(data.ImageUrl))
                    WebClient wc = new WebClient();
                    wc.OpenReadCompleted += new OpenReadCompletedEventHandler(Image_OpenReadCompleted);
                    wc.OpenReadAsync(new Uri(data.ImageUrl, UriKind.Absolute), data.Index);
                    System.Diagnostics.Debug.WriteLine("Image Requested");
                    System.Diagnostics.Debug.WriteLine("Empty Image Url");
                // Load the image setting the content type, so it knows how to read the file
                using (MemoryStream ms = new MemoryStream(image))
                    StreamResourceInfo resource = new StreamResourceInfo(ms, null);
                System.Diagnostics.Debug.WriteLine("Image Streamed");
        catch (Exception ex)
            System.Diagnostics.Debug.WriteLine("Error rendering image: " + ex);
    void Image_OpenReadCompleted(object sender, OpenReadCompletedEventArgs e)
        if (e.Error == null)
                StreamResourceInfo sri = new StreamResourceInfo(e.Result as Stream, null);
                System.Diagnostics.Debug.WriteLine("Image downloaded");
            catch (Exception ex)
                System.Diagnostics.Debug.WriteLine("Error downloading image: " + ex);

The ImageUrl in this case is served up by the WCF service as configured by the Socket Service:


  <add key="ModelFilename" value="..\..\..\..\data\face\models\tim_face.smd" />

  <add key="Loader.VisualisationServiceUri" value="http://localhost:1236/VisualisationService.svc" />


This WCF service uses the .NET 3.5 extensions to serve up the images in a RESTful manner

    public interface IVisualisationService
        [WebGet(UriTemplate = "image/{index}")]
        Stream GetImage(string index);

And the following implementation:

    public class VisualisationService : IVisualisationService
        /// <summary>
        /// Gets the image.
        /// </summary>
        /// <param name="index">The index.</param>
        /// <returns></returns>
        public Stream GetImage(string index)
            // Pull the model filename from config
            string modelFilename = Path.GetFullPath(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, 
            int i = Int32.Parse(index);

            // Load the training data
            VisualisationLoader loader = new VisualisationLoader();
            TrainingSetData d = loader.Read(modelFilename)[i];
            MemoryStream ms = new MemoryStream(d.Image, false);

            // Set the content type and return stream
            WebOperationContext.Current.OutgoingResponse.ContentType = d.ImageContentType;
            return ms;

After applying the appropriate webHttpBinding configuration, the only other consider is to include a crossdomain.xml file in the web server root, otherwise Silverlight will through a protocol error.

<?xml version="1.0"?>



  <allow-access-from domain="*" />


It is also worth pointing out that you must be running under a web project to test download of Images.  See this blog post for this and more helpful tips.

From my testing send the image stream directly over the socket resulted in better performance, but the delay in ‘downloading’ the image once the process fired up was not significant.