HDFS Data Federation

My client has this requirement to restrict certain Hadoop directories based on user/group.  This seems to be a common issue where you want to isolate portions of HDFS to say a department or users.

Federated NameNode sounds a perfect solution for this, however this has issues and limitations.  It does not work well with shared network storage.  Additionally I would like to understand who is utilizing this in production.

One alternative is to use the Posix permissions owner-group-others  and restrict certain directories to groups.  This provides basic security but is harder to manage and does not provide a clean isolation like the federated namenode.


Some links to read

http://www.slideshare.net/huguk/hdfs-federation-hadoop-summit2011

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/Federation.html  

http://www.slideshare.net/AdamKawa/apache-hadoop-yarn-namenode-ha-hdfs-federation

  

Yarn Command Line

Starting from Hadoop 2.0, we can use Yarn to run mapreduce as an application.

• Running a Wordcount using Yarn

         $yarn jar $HADOOP_HOME/HadoopSamples.jar \
          mr.wordcount  -libjars $MYLIBS/custom-libs.jar

• To get list of all applications/mapreduce that is running
  
  $yarn application -list
• To get information on specific mapreduce job
  
  $yarn application -status application_133690873084_3466
• Reference:  http://www.slideshare.net/martyhall/hadoop-tutorial-mapreduce-on-yarn-part-1-overview-and-installation 
• http://www.coreservlets.com/hadoop-tutorial/#Tutorial-Intro
• 
  

Impala Presentation

Cloudera Impala from Scott Leberknight

Avro DataSerialization in Hadoop

• Avro is a data serialization system 
• http://avro.apache.org/
• Key advantage is that it supports schema evolution.
• Schema is stored along with data.
• Schema is expressed in JSON format.
• Both writer and reader have to define a schema to access avro files.
• This allows a good way to handle schema evolution.
• Youtube link: http://www.youtube.com/watch?v=EBV4C-P3G94
• IBM article: http://www.ibm.com/developerworks/library/bd-avrohadoop/ 
• MapReduce and Avro example: http://avro.apache.org/docs/current/mr.html#Example%3A+ColorCount 
• Other links:  
• http://www.kiji.org/2013/12/02/defining-schema-compatibility-and-more-on-schema-evolution/
• http://docs.oracle.com/cd/E26161_02/html/GettingStartedGuide/avroschemas.html
• Hive has a Serde, so it can query data that is in Avro format http://www.michael-noll.com/blog/2013/07/04/using-avro-in-mapreduce-jobs-with-hadoop-pig-hive/ 
• MSDN article on how to use this with C#: http://code.msdn.microsoft.com/Schema-Evolution-In-Avro-240f0a7a