Fluent.Interface


Multi-Threading, Silverlight Sockets & Visualisation Part 3

Part 2 of this Silverlight visualisation instalment discussed how data could be streamed over a socket for rendering on the client.

To recap, Part 1 discussed how the Training Set data was loaded from file and stored in memory.  Well the data structure is as follows:

    private void SetImage(Stream stream)
    {
        BitmapImage bi = new BitmapImage();
        bi.SetSource(stream);
        this.Image1.Source = null;
        this.Image1.Source = bi;
    }
    [XmlType("T")]
    public struct TrianglePoints
    {
        public int P1 { get; set; }
        public int P2 { get; set; }
        public int P3 { get; set; }
    }

    [XmlType("P")]
    public struct TrianglePoint
    {
        public float X { get; set; }
        public float Y { get; set; }
    }

    /// <summary>
    /// A container for all the data that needs to be sent over the wire to render the image
    /// </summary>
    [XmlType("D")]
    public class TrainingSetData
    {
        public int Index { get; set; }
        public TrianglePoint[] Points { get; set; }
        public TrianglePoints[] Triangles { get; set; }
        public int Height { get; set; }
        public int Width { get; set; }
        [XmlIgnore]
        public byte[] Image { get; set; }
        public string ImageUrl { get; set; }
        public string ImageContentType { get; set; }
    }

Notice these classes have been attributed so that can be XML serialized, but that the Image blob is ignored given it is sent over as a separate block.  In addition the Socket Server that broadcasts the training set data has a configuration flag to disable streaming the image at all.  In which case the client can download the image via the specified Url.

Silverlight supports loading an image in a number of ways

1.       Relative or Absoulte Uri’s.

2.       Set directly via resource stream (loaded from memory, file etc).

3.       Downloading on demand using the asynchronous web client.

Option 1 is not appropriate given my images are dynamic and change on each request, so I have implemented Option 2 (when image is streamed over socket), and Option3 (when downloading) as the following code illustrates:

    private void RenderImage(TrainingSetData data, byte[] image)
    {
        try
        {
            if (image == null || image.Length == 0)
            {
                // Fire off a request to the web service to download the image
                if (!String.IsNullOrEmpty(data.ImageUrl))
                {
                    WebClient wc = new WebClient();
                    wc.OpenReadCompleted += new OpenReadCompletedEventHandler(Image_OpenReadCompleted);
                    wc.OpenReadAsync(new Uri(data.ImageUrl, UriKind.Absolute), data.Index);
                    System.Diagnostics.Debug.WriteLine("Image Requested");
                }
                else
                {
                    System.Diagnostics.Debug.WriteLine("Empty Image Url");
                }
            }
            else
            {
                // Load the image setting the content type, so it knows how to read the file
                using (MemoryStream ms = new MemoryStream(image))
                {
                    StreamResourceInfo resource = new StreamResourceInfo(ms, null);
                    SetImage(resource.Stream);
                }
                System.Diagnostics.Debug.WriteLine("Image Streamed");
            }
        }
        catch (Exception ex)
        {
            System.Diagnostics.Debug.WriteLine("Error rendering image: " + ex);
        }
    }
    void Image_OpenReadCompleted(object sender, OpenReadCompletedEventArgs e)
    {
        if (e.Error == null)
        {
            try
            {
                StreamResourceInfo sri = new StreamResourceInfo(e.Result as Stream, null);
                SetImage(sri.Stream);
                System.Diagnostics.Debug.WriteLine("Image downloaded");
            }
            catch (Exception ex)
            {
                System.Diagnostics.Debug.WriteLine("Error downloading image: " + ex);
            }
        }
    }

The ImageUrl in this case is served up by the WCF service as configured by the Socket Service:

<appSettings>

  <add key="ModelFilename" value="..\..\..\..\data\face\models\tim_face.smd" />

  <add key="Loader.VisualisationServiceUri" value="http://localhost:1236/VisualisationService.svc" />

</appSettings>

This WCF service uses the .NET 3.5 extensions to serve up the images in a RESTful manner

    [ServiceContract]
    public interface IVisualisationService
    {
        [OperationContract]
        [WebGet(UriTemplate = "image/{index}")]
        Stream GetImage(string index);
    }

And the following implementation:

    public class VisualisationService : IVisualisationService
    {
        /// <summary>
        /// Gets the image.
        /// </summary>
        /// <param name="index">The index.</param>
        /// <returns></returns>
        public Stream GetImage(string index)
        {
            // Pull the model filename from config
            string modelFilename = Path.GetFullPath(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, 
                ConfigurationManager.AppSettings["ModelFilename"]));
            int i = Int32.Parse(index);

            // Load the training data
            VisualisationLoader loader = new VisualisationLoader();
            TrainingSetData d = loader.Read(modelFilename)[i];
            MemoryStream ms = new MemoryStream(d.Image, false);

            // Set the content type and return stream
            WebOperationContext.Current.OutgoingResponse.ContentType = d.ImageContentType;
            return ms;
        }
    }

After applying the appropriate webHttpBinding configuration, the only other consider is to include a crossdomain.xml file in the web server root, otherwise Silverlight will through a protocol error.

<?xml version="1.0"?>

<!–
http://www.mysite.com/crossdomain.xml
–>

<cross-domain-policy>

  <allow-access-from domain="*" />

</cross-domain-policy>

It is also worth pointing out that you must be running under a web project to test download of Images.  See this blog post for this and more helpful tips.

From my testing send the image stream directly over the socket resulted in better performance, but the delay in ‘downloading’ the image once the process fired up was not significant.

Advertisements

Multi-Threading, Silverlight Sockets & Visualisation Part 1

When looking for ideas to build a socket-aware sample silverlight application, I came across the statistical shape modelling of research of Newcastle professor Tim Coote’s.  He has available on his website some tools for marking up a model with points, and mapping related images.

 

I used Tim’s sample face dataset, and created a Silverlight application that receives a stream of socket information containing the image data and polygons for rendering the ‘mask’

Silverlight Visualisation

Before getting to the visualisation however, I will cover off loading the dataset, and the multi-threading performance tests I did along the way.

The face dataset consists of three directories

·         images

·         points

·         models

A model is defined by a series of images, each of which map to a points file which contain a fixed array of X,Y co-ordinates.  These points map to triangles that form the polygons that are drawn on top of the image for each Timestep in the dataset.

Model contains the training set:

training_set:

{

  107_0764.pts : 107_0764.jpg

  107_0766.pts : 107_0766.jpg

  107_0791.pts : 107_0791.jpg

  107_0792.pts : 107_0792.jpg

}

Points for for each time step contains 68 points at slightly different co-ordinates.

version: 1

n_points: 68

{

249.599 153.934

246.122 205.482

384.186 323.087

394.303 235.871

}

The triangles files indicates which points are joined to form the pologyons.

version: 1

n_triangles: 113

{

 { v1: 0 v2: 21 v3: 27 }

 { v1: 27 v2: 0 v3: 1 }

 { v1: 58 v2: 6 v3: 5 }

 { v1: 61 v2: 58 v3: 60 }

}

My sample contains a Loader responsible for parsing these files as and putting the object model into an in-memory cache.  Given the training data set contains a sequential list of files; loading of this information is inherently parallelisable.

When dealing with multi-threading scenarios, resource locking also becomes an issue, so I constructed a unit test which compares the following thread models:

1.       Single Threaded

2.       Pooled: Multi-Threaded using ThreadPool

3.       Parallel: Multi-Threaded using Microsoft’s new Task Parallel Library (TPL)

With the following locking models

·         Read/Write Lock

·         Write Lock

Following is the sample code for the Pooled method

 
private void RunPooled(Action threadProc)
{
    // Set the stop date
    stop = DateTime.Now.AddSeconds(runFor);

    // Use an event to wait for all work items to complete
    int count = threadCount;
    using (var mre = new ManualResetEvent(false))
    {
        for (int i = 0; i < threadCount; i++)
        {
            ThreadPool.QueueUserWorkItem((state) =>
            {
                // Execute the method
                Thread.CurrentThread.Name = String.Format("Thread#{0:00}", state);
                threadProc();
                if (Interlocked.Decrement(ref count) == 0)
                {
                    mre.Set();
                }
            }, i);
        }

        // Wait for all work to complete
        mre.WaitOne();
    }
}

Following is an example using Parallel.Do method (Note the Parallel.For method seems to perform some internal ‘optimisations’ which results in fewer worker thread’s being spun up).

private void RunParalell(Action threadProc)
{
// Set the stop date
stop = DateTime.Now.AddSeconds(runFor);

// Create the list of actions
List actions = new List();
for (int i = 0; i < threadCount; i++) { actions.Add(threadProc); } // Run the series of actions Parallel.Do(actions.ToArray()); } [/sourcecode]

The test spins up a series of threads to load the dataset which is will randomly read or write to the in-memory cache which is also subject to time outs.  The following configure was used:

Cache Expiry

200ms

Read Lock

20ms

Write Lock

100ms

Thread Sleep

50ms

Read/Write Ratio

9:1 (9 Reads for every 1 Write)

My machine configuration is Del MultiPlex 745, Intel Core 2, 2,394MHz, 4GM RAM running Windows 2003 SP 2.  The average execution time (ms) for each load request is summarised in the chart below:

Silverlight threading performance

Microsoft’s new parallel library wins out on a 2:1 ratio over the single treaded equivalent, and the Read/Write lock proves more efficient as it locks the resource less often given Read’s are more frequent the Writes as configured by the test ratio.

For complete test results, please see the complete spreadsheet or download the sample to play for yourself.

Stay tuned for Part 2 where I will discuss the Silverlight socket implementation, including the complexity of sending multi-part chunks of data for the large images.