Managing the Data from the Internet of Things – Sidney Andrews

Our web services and applications can now be connected to millions of devices simultaneously and are still expected to perform admirably. For line-of-business data, this is an exercise where we scale our database. What about our other smaller, more frequent data? How do we handle billions of telemetry transactions from millions of devices? In this talk we will explore the Lambda Architecture design and how we can split the processing of our data so that we have both immediate feedback from our telemetry and long-term analysis. We will first explore the Event Hubs service as an endpoint to capture our telemetry. For speed, we will look at how we can use Stream Analytics to create a slice of our telemetry averaged over time and then dump that data into SQL Database. For completeness, we will look at options such as HDInsight and SQL Data Warehouse and how they can allow us to perform advanced analytics over our complete, raw data.