We have already seen MapReduce. Now, lets dig deep into the new data processing model of Hadoop. Hadoop comes with a advanced resource management tool called YARN, and packaged as Hadoop-2.x.
What is YARN?
YARN stands for Yet Another Resource Navigator. YARN is also known as Hadoop Data Operating System — YARN enables data processing models beyond traditional mapreduce, such as Storm (real time streaming), Solr (searching) and interactive systems (Apache Tez).
Why YARN?
Hadoop committers decided to split up resource management and job scheduling into separate daemons to overcome some deficiencies in the original MapReduce. In traditional MR terms, this is basically splitting up jobtracker to provide flexibility and improve performance.
YARN splits the functionality of a JobTracker into two separate daemons:
A global Resource Manager (RM) that consists of a Scheduler and an Applications Manager.
An Application Master (AM) that provides support for a specific application. It runs on every node and controls execution of applications. (Here application could mean a single MR job or a Directed Acyclic Graph (DAG) of jobs).
Caveat:
YARN, also referred to as MapReduce 2 is not an improvment over traditional MapReduce data processing model. It merely provides a resource management model that executes MapReduce jobs.
How to deploy Hadoop-2.x YARN?
We will look at how to deploy YARN in a single node cluster setup. The following are system requirements:
Java 6 or above
SSH
Download Hadoop-2.2.0:
wget http://your download link here/hadoop-2.2.0.tar.gz
unpack the tarball:
tar -zxvf hadoop-2.2.0.tar.gz
Create HDFS directory inside Hadoop-2.2.0 folder:
mkdir -p /data/namenode
mkdir -p /data/datanode
Rename hadoop-2.2.0 to hadoop but I wouldn’t suggest that.
Go to Hadoop-2.2.0/etc/hadoop/hadoop-env.sh and copy/paste the following:
export JAVA_HOME= < your java path here >
<your path for lib folder in Hadoop-2.2.0> type pwd from lib and paste it below.
export HADOOP_COMMON_LIB_NATIVE_DIR="/home/hduser/hadoop-2.2.0/lib"
export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/home/hduser/hadoop-2.2.0/lib"
Similar to Hadoop 1 (traditional hadoop), make changes in core-site, mapred-site, yarn-site, hdfs-site:
In core-site.xml, copy/paste:
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
In hdfs-site.xml, copy/paste:
<property>
<name>dfs.replication</name> <value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value> <path to your hadoop> /hadoop/data/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value> <path to your hadoop> /hadoop/data/datanode</value>
</property>
In mapred-site.xml, copy/paste: (create mapred-site.xml by vim mapred-site.xml , if file is not available)
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
In yarn-site.xml, copy/paste:
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
Either update your bashrc file or create yarnconfig.sh and paste the following:
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk <your java path here>
export HADOOP_HOME=/home/hduser/hadoop-2.2.0 <your hadoop path here>
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
To run,
source ~/.bashrc or . yarnconfig.sh
Before you run YARN, go to Hadoop-2.2.0 folder and type to format the namenode
bin/hdfs namenode -format
To start Hadoop:
sbin/start-dfs.sh
sbin/start-yarn.sh
You can check whether everthing went well by checking jps. Ideally you should have the following:
8144 ResourceManager
7722 DataNode
7960 SecondaryNameNode
8333 NodeManager
9636 Jps
7539 NameNode
Note: The PID might be different but the processes are the same.
Access the UI for:
Try some examples in your new YARN:
bin/hadoop jar /home/hduser/hadoop-2.2.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 4 15
