Upload
pavan2711
View
27
Download
1
Embed Size (px)
DESCRIPTION
Hadoop
Citation preview
Managing Hadoop Cluster
Topology of a typical Hadoop cluster .
Installation Steps
Installed java
ssh and sshd
gunzip hadoop-0.18.0.tar.gz
Or tar vxf hadoop-0.18.0.tar Or tar vxf hadoop-0.18.0.tar
Set JAVA_HOME in conf/hadoop-env.sh
Modified hadoop-site.xml
Hadoop Installation Flavors
Standalone
Pseudo-distributed
Hadoop clusters of multiple nodes
Additional Configuration
conf/masters
contains the hostname of the SecondaryNameNode
It should be fully-qualified domain name.
conf/slaves conf/slaves
the hostname of every machine in the cluster which
should start TaskTracker and DataNode daemons
Ex:slave01
slave02
slave03
Advance Configuration
enable passwordless ssh
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
The ~/.ssh/id_dsa.pub and authorized_keys
files should be replicated on all machines in
the cluster.
Advance Configuration
Various directories should be created on each
node
The NameNode requires the NameNode metadata
directorydirectory
$ mkdir -p /home/hadoop/dfs/name
Every node needs the Hadoop tmp directory and
DataNode directory created
Advance Configuration..
bin/slaves.sh allows a command to be
executed on all nodes in the slaves file. $ mkdir -p /tmp/hadoop
$ export HADOOP_CONF_DIR=${HADOOP_HOME}/conf
$ export HADOOP_SLAVES=${HADOOP_CONF_DIR}/slaves
$ ${HADOOP_HOME}/bin/slaves.sh "mkdir -p /tmp/hadoop"$ ${HADOOP_HOME}/bin/slaves.sh "mkdir -p /tmp/hadoop"
$ ${HADOOP_HOME}/bin/slaves.sh "mkdir -p /home/hadoop/dfs/data
Format HDFS
$ bin/hadoop namenode -format
start the cluster:
$ bin/start-all.sh
Selecting Machines
Hadoop is designed to take advantage of
whatever hardware is available
Hadoop jobs written in Java can consume
between 1 and 2 GB of RAM per corebetween 1 and 2 GB of RAM per core
If you use HadoopStreaming to write your jobs
in a scripting language such as Python, more
memory may be advisable.
Cluster Configurations
Small Clusters: 2-10 Nodes
Medium Clusters: 10-40 Nodes
Large Clusters: Multiple Racks
Small Clusters: 2-10 Nodes
In two nodes,
one node: NameNode/JobTracker and a
DataNode/TaskTracker;
the other node: DataNode/TaskTracker. the other node: DataNode/TaskTracker.
Clusters of three or more machines typically
use a dedicated NameNode/JobTracker, and
all other nodes are workers.
configuration in conf/hadoop-site.xml
mapred.job.trackerhead.server.node.com:9001
fs.default.name
hdfs://head.server.node.com:9000
hadoop.tmp.dir/tmp/hadooptrue
mapred.system.dir/hadoop/mapred/systemtrue
ue>
dfs.data.dir/home/hadoop/dfs/datatrue
dfs.name.dir/home/hadoop/dfs/nametrue
dfs.replication2
Medium Clusters: 10-40 Nodes
The single point of failure in a Hadoop cluster
is the NameNode
Hence, back up the NameNode metadata.
One machine in the cluster should be designated One machine in the cluster should be designated
as the NameNode's backup
It does not run the normal Hadoop daemons
it exposes a directory via NFS which is only
mounted on the NameNode
NameNodes backup
The cluster's hadoop-site.xml file should then
instruct the NameNode to write to this
directory as well:
dfs.name.dir
/home/hadoop/dfs/name,/mnt/namenode-backup
true
conf/hadoop-site.xml
Nodes must be decommissioned on a schedule that permits replication of blocks being decommissioned.
conf/hadoop-site.xml
dfs.hosts.exclude/home/hadoop/excludes/home/hadoop/excludestrue
mapred.hosts.exclude/home/hadoop/excludestrue
create an empty file with this name: $ touch /home/hadoop/excludes
Replication Setting
dfs.replication
3
Tutorial
Configure Hadoop Cluster in two nodes.
Tutorial-Installed Hadoop in Cluster.docx
Performance Monitoring
Ganglia
Nagios
Ganglia
performance monitoring framework for
distributed systems
collects metrics on individual machines and
forwards them to an aggregatorforwards them to an aggregator
designed to be integrated into other
applications
Ganglia
Installed and configured Ganglia
create a file named hadoop-metrics.propertiesin the $HADOOP_HOME/conf directorydfs.class=org.apache.hadoop.metrics.ganglia.GangliaContextdfs.period=10dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContextdfs.period=10dfs.servers=localhost:8649
mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContextmapred.period=10mapred.servers=localhost:8649
Nagios
a machine and service monitoring system
designed for large clusters
provide useful diagnostic information for
tuning your cluster, including network, disk, tuning your cluster, including network, disk,
and CPU utilization across machines.
Tutorial
Installed Ganglia /Nagios and monitor Hadoop
Tutorial-MonitorHadoopWithGanglia.docx