Fluent.Interface


Category Archive

The following is a list of all entries from the LINQ category.

Cloudera building a business around open source Map Reduce

The heavy hitting ex-executives behind start up CloudEra are banking on a business based around Hadoop, the open source Map Reduce implementation with a distribution capable of running on Amazon’s EC2.   Google is credited with popularising (inventing) Map Reduce and has been tuning its own implementation for many years.  It gave insights into the origin and future research direction in a round table video last year.

Increasingly companies need to make sense of Terabytes or even Petabytes of data.  This information is stored across many machines on many disks, and needs distributed algorithms for sifting through the data in any reasonable time.  This is where Map-Reduce comes in.

Interestingly Microsoft has taken a step back from this direction when with deciding that its SDS offering should support standard ‘relational’ features, in effect turning the product into a hosted SQL Server cloud.

It has however been active in this research field.  It released its functional programming language F# and it runs its ad serving on Dryad – a distributed execution software engine.  DryadLINQ combines the power of this engine, with the simplicity of LINQ by creating a SQL-like execution plan for distributed processing, very cool! 

Large scale distributed processing software typically runs on many low grade Linux servers running open source software so that licensing costs are kept low.  However with the army of MS developers out there, there are companies springing up to provide software to make the most out of idle cycles on Windows boxes around the network.  Manjrasoft a recent graduate from Melbourne University’s GridBus laboratory have released an Alpha of their Aneka software – a .NET Map Reduce implementation.


Open source .NET CMS, Oxite

There are numerous open source CMS platforms out there, wordpress being one of the most recognizable (hosting this blog).  But when it comes to .NET the choice is more limited.  Microsoft’s channel 9 community platform started its life from the telligent Community Server codebase.  Since then that team has built a new Blog/CMS platform Oxite which powers MIX online.

The project targets ASP.NET MVC developers and leverages LINQ to SQL for persistence, but hides the implementation behind a clean repository interface.  There is not end-user installation so without VSTS you will need to attach the database and config a login.  Boasting ‘standards compliant’ features such as RSS and Trackback capabilities, it makes good use of modern features such as extension methods to keep the front end clean.  Although still not out of beta, the ASP.NET MVC is proving popular, and powers some high traffic sites such as stackoverflow.com.

Microsoft’s other CMS platform sharepoint is getting behind CMIS. The content management integration services (CMIS) specificiation leverages SOAP, REST and Atom to enable communication with and between ECM systems. Although Roy Fielding isn’t impressed by the initiative, in a Web 2.0 environment a languague neutral API is almost as important as the website – so it should bring for some good vendor competition.  Alfresco in particular has recently released beta support for CMIS on its java-based platform.

Also worth a look the .NET CMS Umbraco and the popular Radiant CMS built on rails.


The future is parallel

Although most of us are only running dual core computers, if you have a spare £2000 you can pickup up this G71 quad core notebook from Ausus, and if you belive the hype we will be hitting 80 cores by 2014.

While hardware plays catch up up with moore’s law, Microsoft’s next generation .NET 4 platform  will have a native concurrency run-time.  This is already available in the form of VS 2010 CTP and can be configured run with Windows 2008 Hyper-V to test out the running on virtualized multiple cores.

Parallel programming brings a host or problems for developers when it comes to debugging and diagnosing problems.  It is great to see Microsoft helping to ease this with initiatives such as PLINQ, and the interesting (albeit experimental) Transactional Memory group which looks at how code might be modelled to run as efficiently as database logic for example. 

F# will ship with VS 2010 and is a powerful functional language that is naturally immutable, compiles to .NET IL and supports erlang style message passing.


Do you think in LINQ?

If not you should download LINQPad from Joseph Albahari’s (of C#3 in a nutshell fame).  It’s a great tool to practicing LINQ expressions and has syntax highlighting (but unfortunatly no intellisense).  Alternatively if you really miss the old Query Analyzer from SQL 2000 days, then Query Express is a good option – super fast 100k stand-alone executable!