Scaling Hive based datawarehouse in FB

https://code.facebook.com/posts/229861827208629/scaling-the-facebook-data-warehouse-to-300-pb/

Hortonworks HDP - Hive and Pig interface

Hortonworks Hcatalog interface is pretty useful for quick ingest and query. Also check out quick start slides on how to start the sandbox vm.

http://hortonworks.com/hadoop-tutorial/hello-world-an-introduction-to-hadoop-hcatalog-hive-and-pig/#loading

Hortonworks Sandbox Startup Guide for VirtualBox from Hortonworks

Good reference articles on Hadoop based solution architectures

http://hortonworks.com/blog/modern-financial-services-architectures-built-hadoop/

How to run a simple mapreduce job to verify the cluster health?

Many times you want to check if cluster is in good shape... run this command that does not require much setup...

$ hadoop jar /usr/lib/gphd/hadoop-mapreduce/hadoop-mapreduce-examples-*.jar pi 10 1000

HDFS Data Federation

My client has this requirement to restrict certain Hadoop directories based on user/group.  This seems to be a common issue where you want to isolate portions of HDFS to say a department or users.

Federated NameNode sounds a perfect solution for this, however this has issues and limitations.  It does not work well with shared network storage.  Additionally I would like to understand who is utilizing this in production.

One alternative is to use the Posix permissions owner-group-others  and restrict certain directories to groups.  This provides basic security but is harder to manage and does not provide a clean isolation like the federated namenode.


Some links to read

http://www.slideshare.net/huguk/hdfs-federation-hadoop-summit2011

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/Federation.html  

http://www.slideshare.net/AdamKawa/apache-hadoop-yarn-namenode-ha-hdfs-federation

  

Yarn Command Line

Starting from Hadoop 2.0, we can use Yarn to run mapreduce as an application.

• Running a Wordcount using Yarn

         $yarn jar $HADOOP_HOME/HadoopSamples.jar \
          mr.wordcount  -libjars $MYLIBS/custom-libs.jar

• To get list of all applications/mapreduce that is running
  
  $yarn application -list
• To get information on specific mapreduce job
  
  $yarn application -status application_133690873084_3466
• Reference:  http://www.slideshare.net/martyhall/hadoop-tutorial-mapreduce-on-yarn-part-1-overview-and-installation 
• http://www.coreservlets.com/hadoop-tutorial/#Tutorial-Intro
• 
  

Impala Presentation

Cloudera Impala from Scott Leberknight