温馨提示×

温馨提示×

您好,登录后才能下订单哦!

密码登录×
登录注册×
其他方式登录
点击 登录注册 即表示同意《亿速云用户服务条款》

Hadoop2 namenode联邦实验分析

发布时间:2021-12-09 17:30:28 来源:亿速云 阅读:136 作者:iii 栏目:云计算

本篇内容主要讲解“Hadoop2 namenode联邦实验分析”,感兴趣的朋友不妨来看看。本文介绍的方法操作简单快捷,实用性强。下面就让小编来带大家学习“Hadoop2 namenode联邦实验分析”吧!

实验的Hadoop版本为2.5.2,硬件环境是5台虚拟机,使用的均是CentOS6.6操作系统,虚拟机IP和hostname分别为: 
192.168.63.171    node1.zhch
192.168.63.172    node2.zhch
192.168.63.173    node3.zhch
192.168.63.174    node4.zhch
192.168.63.175    node5.zhch

ssh免密码、防火墙、JDK这里就不在赘述了。虚拟机的角色分配是 node1、2是namendoe节点,node3、4、5为datanode节点。 

步骤和 搭建普通hadoop集群 基本相同 ,主要的不同在于 hdfs-site.xml这个配置文件,其余配置和hadoop的安装配置基本一致。

一、配置Hadoop

## 解压
[yyl@node1 program]$ tar -zxf hadoop-2.5.2.tar.gz 
## 创建文件夹
[yyl@node1 program]$ mkdir hadoop-2.5.2/name
[yyl@node1 program]$ mkdir hadoop-2.5.2/data
[yyl@node1 program]$ mkdir hadoop-2.5.2/tmp
## 配置hadoop-env.sh
[yyl@node1 program]$ cd hadoop-2.5.2/etc/hadoop/
[yyl@node1 hadoop]$ vim hadoop-env.sh
export JAVA_HOME=/usr/lib/java/jdk1.7.0_80
## 配置yarn-env.sh
[yyl@node1 hadoop]$ vim yarn-env.sh
export JAVA_HOME=/usr/lib/java/jdk1.7.0_80
## 配置slaves
[yyl@node1 hadoop]$ vim slaves 
node3.zhch
node4.zhch
node5.zhch
## 配置core-site.xml
[yyl@node1 program]$ cd hadoop-2.5.2/etc/hadoop/
[yyl@node1 hadoop]$ vim core-site.xml
<configuration>
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://node1.zhch:9000</value>
</property>
<property>
  <name>io.file.buffer.size</name>
  <value>131072</value>
</property>
<property>
  <name>hadoop.tmp.dir</name>
  <value>file:/home/yyl/program/hadoop-2.5.2/tmp</value>
</property>
<property>
  <name>hadoop.proxyuser.hduser.hosts</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.hduser.groups</name>
  <value>*</value>
</property>
</configuration>
## 配置hdfs-site.xml
[yyl@node1 hadoop]$ vim hdfs-site.xml
<configuration>
<property>
  <name>dfs.namenode.name.dir</name>
  <value>file:/home/yyl/program/hadoop-2.5.2/name</value>
</property>
<property>
  <name>dfs.datanode.data.dir</name>
  <value>file:/home/yyl/program/hadoop-2.5.2/data</value>
</property>
<property>
  <name>dfs.replication</name>
  <value>1</value>
</property>
<property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
</property>
<property>
  <name>dfs.permissions</name>
  <value>false</value>
</property>
<property>
  <name>dfs.nameservices</name>
  <value>ns1,ns2</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.ns1</name>
  <value>node1.zhch:9000</value>
</property>
<property>
  <name>dfs.namenode.http-address.ns1</name>
  <value>node1.zhch:50070</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.ns2</name>
  <value>node2.zhch:9000</value>
</property>
<property>
  <name>dfs.namenode.http-address.ns2</name>
  <value>node2.zhch:50070</value>
</property>
</configuration>
##配置 mapred-site.xml
[yyl@node1 hadoop]$ cp mapred-site.xml.template mapred-site.xml
[yyl@node1 hadoop]$ vim mapred-site.xml
<configuration>
<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>
<property>
  <name>mapreduce.jobhistory.address</name>
  <value>node1.zhch:10020</value>
</property>
<property>
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>node1.zhch:19888</value>
</property>
</configuration>
##配置 yarn-site.xml
[yyl@node1 hadoop]$ vim yarn-site.xml
<configuration>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
<property>
  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
  <name>yarn.resourcemanager.address</name>
  <value>node1.zhch:8032</value>
</property>
<property>
  <name>yarn.resourcemanager.scheduler.address</name>
  <value>node1.zhch:8030</value>
</property>
<property>
  <name>yarn.resourcemanager.resource-tracker.address</name>
  <value>node1.zhch:8031</value>
</property>
<property>
  <name>yarn.resourcemanager.admin.address</name>
  <value>node1.zhch:8033</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address</name>
  <value>node1.zhch:8088</value>
</property>
</configuration>
## 分发到各个节点
[yyl@node1 hadoop]$ cd /home/yyl/program/
[yyl@node1 program]$ scp -rp hadoop-2.5.2 yyl@node2.zhch:/home/yyl/program/
[yyl@node1 program]$ scp -rp hadoop-2.5.2 yyl@node3.zhch:/home/yyl/program/
[yyl@node1 program]$ scp -rp hadoop-2.5.2 yyl@node4.zhch:/home/yyl/program/
[yyl@node1 program]$ scp -rp hadoop-2.5.2 yyl@node5.zhch:/home/yyl/program/
## 在各个节点上设置hadoop环境变量
[yyl@node1 ~]$ vim .bash_profile 
export HADOOP_PREFIX=/home/yyl/program/hadoop-2.5.2
export HADOOP_COMMON_HOME=$HADOOP_PREFIX
export HADOOP_HDFS_HOME=$HADOOP_PREFIX
export HADOOP_MAPRED_HOME=$HADOOP_PREFIX
export HADOOP_YARN_HOME=$HADOOP_PREFIX
export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
export PATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin


二、NameNode 

## 在namenode1上执行格式化
[yyl@node1 ~]$ hdfs namenode -format -clusterId c1
## 在namenode2上执行格式化
[yyl@node2 ~]$ hdfs namenode -format -clusterId c1
## 在namenode1启动namenode
[yyl@node1 ~]$ hadoop-daemon.sh start namenode
starting namenode, logging to /home/yyl/program/hadoop-2.5.2/logs/hadoop-yyl-namenode-node1.zhch.out
[yyl@node1 ~]$ jps
1177 NameNode
1240 Jps
## 在namenode2启动namenode
[yyl@node2 ~]$ hadoop-daemon.sh start namenode
starting namenode, logging to /home/yyl/program/hadoop-2.5.2/logs/hadoop-yyl-namenode-node2.zhch.out
[yyl@node2 ~]$ jps
1508 Jps
1445 NameNode


三、HDFS联邦检查 

http://node1.zhch:50070/
Hadoop2 namenode联邦实验分析
http://node2.zhch:50070/
Hadoop2 namenode联邦实验分析
 

四、启动DataNode和yarn 

[yyl@node1 ~]$ hadoop-daemons.sh start datanode
node4.zhch: starting datanode, logging to /home/yyl/program/hadoop-2.5.2/logs/hadoop-yyl-datanode-node4.zhch.out
node5.zhch: starting datanode, logging to /home/yyl/program/hadoop-2.5.2/logs/hadoop-yyl-datanode-node5.zhch.out
node3.zhch: starting datanode, logging to /home/yyl/program/hadoop-2.5.2/logs/hadoop-yyl-datanode-node3.zhch.out
[yyl@node1 ~]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/yyl/program/hadoop-2.5.2/logs/yarn-yyl-resourcemanager-node1.zhch.out
node5.zhch: starting nodemanager, logging to /home/yyl/program/hadoop-2.5.2/logs/yarn-yyl-nodemanager-node5.zhch.out
node3.zhch: starting nodemanager, logging to /home/yyl/program/hadoop-2.5.2/logs/yarn-yyl-nodemanager-node3.zhch.out
node4.zhch: starting nodemanager, logging to /home/yyl/program/hadoop-2.5.2/logs/yarn-yyl-nodemanager-node4.zhch.out
[yyl@node1 ~]$ jps
1402 Jps
1177 NameNode
1333 ResourceManager
[yyl@node2 ~]$ jps
1445 NameNode
1539 Jps
[yyl@node3 ~]$ jps
1214 NodeManager
1166 DataNode
1256 Jps

下次启动不需要重复上面的步骤,可以直接使用下面的命令启动集群:
sh $HADOOP_HOME/sbin/start-dfs.sh
sh $HADOOP_HOME/sbin/start-yarn.sh

到此,相信大家对“Hadoop2 namenode联邦实验分析”有了更深的了解,不妨来实际操作一番吧!这里是亿速云网站,更多相关内容可以进入相关频道进行查询,关注我们,继续学习!

向AI问一下细节

免责声明:本站发布的内容(图片、视频和文字)以原创、转载和分享为主,文章观点不代表本网站立场,如果涉及侵权请联系站长邮箱:is@yisu.com进行举报,并提供相关证据,一经查实,将立刻删除涉嫌侵权内容。

AI