HDFS Data Federation

My client has this requirement to restrict certain Hadoop directories based on user/group.  This seems to be a common issue where you want to isolate portions of HDFS to say a department or users.

Federated NameNode sounds a perfect solution for this, however this has issues and limitations.  It does not work well with shared network storage.  Additionally I would like to understand who is utilizing this in production.

One alternative is to use the Posix permissions owner-group-others  and restrict certain directories to groups.  This provides basic security but is harder to manage and does not provide a clean isolation like the federated namenode.


Some links to read

http://www.slideshare.net/huguk/hdfs-federation-hadoop-summit2011

https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/Federation.html  

http://www.slideshare.net/AdamKawa/apache-hadoop-yarn-namenode-ha-hdfs-federation

  

Yarn Command Line

Starting from Hadoop 2.0, we can use Yarn to run mapreduce as an application.

• Running a Wordcount using Yarn

         $yarn jar $HADOOP_HOME/HadoopSamples.jar \
          mr.wordcount  -libjars $MYLIBS/custom-libs.jar

• To get list of all applications/mapreduce that is running
  
  $yarn application -list
• To get information on specific mapreduce job
  
  $yarn application -status application_133690873084_3466
• Reference:  http://www.slideshare.net/martyhall/hadoop-tutorial-mapreduce-on-yarn-part-1-overview-and-installation 
• http://www.coreservlets.com/hadoop-tutorial/#Tutorial-Intro
• 
  

Impala Presentation

Cloudera Impala from Scott Leberknight

Avro DataSerialization in Hadoop

• Avro is a data serialization system 
• http://avro.apache.org/
• Key advantage is that it supports schema evolution.
• Schema is stored along with data.
• Schema is expressed in JSON format.
• Both writer and reader have to define a schema to access avro files.
• This allows a good way to handle schema evolution.
• Youtube link: http://www.youtube.com/watch?v=EBV4C-P3G94
• IBM article: http://www.ibm.com/developerworks/library/bd-avrohadoop/ 
• MapReduce and Avro example: http://avro.apache.org/docs/current/mr.html#Example%3A+ColorCount 
• Other links:  
• http://www.kiji.org/2013/12/02/defining-schema-compatibility-and-more-on-schema-evolution/
• http://docs.oracle.com/cd/E26161_02/html/GettingStartedGuide/avroschemas.html
• Hive has a Serde, so it can query data that is in Avro format http://www.michael-noll.com/blog/2013/07/04/using-avro-in-mapreduce-jobs-with-hadoop-pig-hive/ 
• MSDN article on how to use this with C#: http://code.msdn.microsoft.com/Schema-Evolution-In-Avro-240f0a7a 

How To get Job Status?

There are several ways to get status of your Hadoop Jobs.  I will discuss the Hadoop 2.0 options here

1. Hadoop Cluster via Yarn Application Manager.  Example -  http://serverhostingyarn:8088/cluster/app/application_1390931417866_0020
   
2. hadoop fs -ls /user/history/done/ | grep "1390931417866_0020"
   
3. yarn command line to get status

Appending Files in Hadoop

Can I really append a file in HDFS OR do i need to always replace the file? http://hadoop4mapreduce.blogspot.com/2012/08/two-methods-to-append-content-to-file.html We had a client that wanted to merge data coming from same source into a single large file. The challenge was how to handle concurrency. The append process is not thread safe. The solution we used was to batch all the small files into a larger file periodically.

Compression Techniques for Hadoop

I am currently researching various options to do compression in Hadoop and found the following articles useful:

• http://comphadoop.weebly.com/index.html
  
• Slide share = http://www.slideshare.net/Hadoop_Summit/kamat-singh-june27425pmroom210cv2
  
• Splitable LZO = http://blog.cloudera.com/blog/2009/11/hadoop-at-twitter-part-1-splittable-lzo-compression/
  
• Read this Microsoft article: http://download.microsoft.com/download/1/C/6/1C66D134-1FD5-4493-90BD-98F94A881626/Compression%20in%20Hadoop%20(Microsoft%20IT%20white%20paper).docx

My project needs are = should have minimum performance impact + should give decent compression gains. We looked at Spilatble LZO. However this requires

• A custom codec to be compiled on your environment (Linux)
  
• The library files need to be copied to all nodes
  
• Update the Hadoop site config file
  
• Additionally you will need to refer to your codec in your mapreduce
  
• If Hive is used, your table creation command has to refer to LZO compression codec in both READ and WRITE
  

With all these requirements, my client decided to use Bzip2 as it is native to Hadoop and does not require additional libraries to be installed. I have attached a quick comparison of the 2 solutions: