How to run HDP on Azure

There are few options if you are looking at Azure to host Hadoop

1. HDInsight - which is Microsoft's flavor of Hadoop (built on top of HDP).  This does provide good separation of storage Azure Blobstore + compute on hardware.
2. HDP on Azure is new option, where you can get real Hortonworks distribution spun up as VM's.   Each data node is serving data and is managing compute. However in this model you cannot use Azure Blob storage easily.  If you shutdown the cluster, data is gone, unless you script the storage back to Azure blob storage and have the reverse in case you want to bring data back to Hadoop.
3. Azure is also supporting another model that is useful for screnarios like "OnPremp" Hadoop cluster for day to day operations, but backup to Azure Blob storage.  This utilizes HDFS abstraction.  The demo below will show you how to setup a feed with 2 data paths... primary and secondary.  The secondary is on Azure blobstorage.  

 

Microsoft expands Azure Data Lake

Key addition... The store in Azure Data Lake is HDFS compatible so Hadoop distributions like Cloudera, Hortonworks®, and MapR can readily access the data for processing and analytics. http://blogs.technet.com/b/dataplatforminsider/archive/2015/09/28/microsoft-expands-azure-data-lake-to-unleash-big-data-productivity.aspx

Managing CDC using Sqoop and Hive

http://hortonworks.com/blog/four-step-strategy-incremental-updates-hive/

Full SQL on Hadoop? Splice machine

http://www.zdnet.com/article/full-sql-on-hadoop-splice-machine-opens-up-its-database-for-trials/

Hive Authorization Model

Hive 13 onwards has an Oracle style User - Role - Privileges model for authorization https://hadooptutorial.info/hive-authorization-models-and-hive-security/

Hadoop Summit 2015 San Jose Key Notes

Excellent kick off and very informative key notes. Key takeaways

• Hadoop data platform is next generation "Data Operating System"
• Atlas from Hortonworks is the “metadata
• Real-time was last year, today it is about “predictive”.  Can we anticipate violations and prevent it.
• Data Scientist = Explore data and build model

Hadoop Data Lake for Healthcare

Good article from Hortonworks on Data Lake and Healthcare http://hortonworks.com/blog/hdp-for-healthcare-providers-common-data-challenges/?mkt_tok=3RkMMJWWfF9wsRonuazKdu%2FhmjTEU5z17%2BwoUKe0hIkz2EFye%2BLIHETpodcMTcNlMr7YDBceEJhqyQJxPr3AKNkNy9RxRhHqDg%3D%3D