Microsoft Big Data Solutions - Hadoop / Hive and SQL Server

How to create Big Data solutions in Microsoft Framework

For quite a while Microsoft developers were baited by Open source folks on the advancement of Big Data solutions in the open source world.

Hadoop + Hive + Pig + R gave a very good platform for Big Data solutions in Open Source platform. Slowly there are many licensed versions that are coming out of the same stack - Cloudera,  Revolutionary R etc.

Microsoft had started Big Data based solutions long back in the Labs have released different tools like PowerPivot etc. Now Microsoft has its own Big Data Technology Stack

Microsoft's Big Data Technology Stack


HDInsight is Microsoft’s Hadoop-based distribution that is available on Windows Azure

The platform can be used for storing large chunks of data (as Blobs, Tables, Columnar Database etc)


SQL Server 2012 is used for Analysis and Integration (ETL)
the SQL Server instance and the Hadoop/Hive data warehouse are

configurable to establish connectivity between them



Real-Time Example of Big Data Solution using Microsoft Technology Stack



(Big Data Solution Courtesy: Ayad Shammout's SQL & BI Blog)

Ayad Shammout's SQL & BI Blog explains how the components are used in various stages effectively for analysing the Audit Logs











 Microsoft's Statistical Component / Solution

Big Data is not data alone - it's more to do with Analyzing the Data. Microsoft has SQL Server 2012 Analysis services. However, certain analysis require custom coding / statistical analysis

Microsoft's Cloud Numerics (now in Azure labs) does exactly the same