温馨提示×

温馨提示×

您好,登录后才能下订单哦!

密码登录×
登录注册×
其他方式登录
点击 登录注册 即表示同意《亿速云用户服务条款》

大数据之---hadoop伪分布式部署(HDFS)全网终极篇

发布时间:2020-06-14 00:40:17 来源:网络 阅读:373 作者:ycwyong 栏目:大数据

 

1、软件环境

RHEL6 jdk-8u45
hadoop-2.8.1.tar.gz ssh
xx.xx.xx.xx ip地址 hadoop1
xx.xx.xx.xx ip地址 hadoop2
xx.xx.xx.xx ip地址 hadoop3
xx.xx.xx.xx ip地址 hadoop4
xx.xx.xx.xx ip地址 hadoop5

本次部署只涉及伪分布式部署只是要主机hadoop1

 

2.伪分布式部署

伪分布式部署文档参考官方网站hadoop
Apache > Hadoop > Apache Hadoop Project Dist POM > Apache Hadoop 2.8.3 > Hadoop: Setting up a Single Node Cluster
添加hadoop用户

[root@hadoop1 ~]# useradd hadoop
[root@hadoop1 ~]# vi /etc/sudoers
# 找到root     ALL=(ALL)     ALL,添加
hadoop     ALL=(ALL)       NOPASSWD:ALL
上传并解压
[root@hadoop1 software]# rz #上传hadoop-2.8.1.tar.gz
[root@hadoop1 software]# tar -xzvf hadoop-2.8.1.tar.gz
软连接
[root@hadoop1 software]# ln -s /opt/software/hadoop-2.8.1 /opt/software/hadoop
设置环境变量
[root@hadoop1 software]# vi /etc/profile
export HADOOP_HOME=/opt/software/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
[root@hadoop1 software]# source /etc/profile
设置用户、用户组
[root@hadoop1 software]# chown -R hadoop:hadoop hadoop
[root@hadoop1 software]# chown -R hadoop:hadoop hadoop/*
[root@hadoop1 software]# chown -R hadoop:hadoop hadoop-2.8.1       
[root@hadoop1 software]# cd hadoop
[root@hadoop1 hadoop]# rm -f *.txt
[root@hadoop1 software]# su - hadoop
[root@hadoop1 hadoop]# ll
total 32
drwxrwxr-x. 2 hadoop hadoop 4096 Jun  2 14:24 bin
drwxrwxr-x. 3 hadoop hadoop 4096 Jun  2 14:24 etc
drwxrwxr-x. 2 hadoop hadoop 4096 Jun  2 14:24 include
drwxrwxr-x. 3 hadoop hadoop 4096 Jun  2 14:24 lib
drwxrwxr-x. 2 hadoop hadoop 4096 Aug 20 13:59 libexec
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 13:59 logs
drwxrwxr-x. 2 hadoop hadoop 4096 Jun  2 14:24 sbin
drwxrwxr-x. 4 hadoop hadoop 4096 Jun  2 14:24 share   
       
# bin:        可执行文件
# etc:         配置文件
# sbin:        shell脚本,启动关闭hdfs,yarn等配置文件
[hadoop@hadoop1 ~]# cd /opt/software/hadoop
[hadoop@hadoop1 hadoop]# vi etc/hadoop/core-site.xml
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://172.168.18.133:9000</value>    # 配置自己机器的IP
    </property>

</configuration>
       
[hadoop@hadoop1 hadoop]# vi etc/hadoop/hdfs-site.xml
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>

</configuration>

---红色部分新增
配置hadoop用户的ssh信任关系
# 公钥/密钥   配置无密码登录
[hadoop@hadoop1 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
[hadoop@hadoop1 ~]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@hadoop1 ~]# chmod 0600 ~/.ssh/authorized_keys       
# 查看日期,看是否配置成功
[root@hadoop1 ~]# ssh hadoop1 date
Wed Apr 25 15:22:58 CST 2018

[root@hadoop1 ~]#  ssh localhost date
Wed Apr 25 15:23:25 CST 2018

# 第一次执行命令有交互选择输入yes
格式化和启动
[hadoop@hadoop1 hadoop]# bin/hdfs namenode -format
[hadoop@hadoop1 hadoop]# sbin/start-dfs.sh
ERROR:
    hadoop1: Error: JAVA_HOME is not set and could not be found.
    localhost: Error: JAVA_HOME is not set and could not be found.
解决方法:添加环境变量
[hadoop@hadoop1 hadoop]#  vi etc/hadoop/hadoop-env.sh
# 将export JAVA_HOME=${JAVA_HOME}改为
export JAVA_HOME=/usr/java/jdk1.8.0_45
# 继续启动
[hadoop@hadoop1 hadoop]# sbin/start-dfs.sh
检查是否成功
[hadoop@hadoop1 hadoop]# jps
8026 Jps
7485 NameNode
7597 DataNode
7807 SecondaryNameNode
访问: http://192.168.137.130:50070

修改dfs启动的进程,以hadoop1启动  启动的三个进程:
namenode: hadoop1    bin/hdfs getconf -namenodes
datanode: localhost    datanodes (using default slaves file)   etc/hadoop/slaves
secondarynamenode: 0.0.0.0

[hadoop@hadoop1 ~]# cd /opt/software/hadoop
[hadoop@hadoop1 hadoop]# echo  "hadoop1" > ./etc/hadoop/slaves
[hadoop@hadoop1 hadoop]# cat ./etc/hadoop/slaves
hadoop1
[hadoop@hadoop1 hadoop]# vi ./etc/hadoop/hdfs-site.xml
<property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>hadoop1:50090</value>
</property>
<property>
    <name>dfs.namenode.secondary.https-address</name>
    <value>hadoop1:50091</value>
</property>

[hadoop@hadoop1 hadoop]# sbin/stop-dfs.sh
[hadoop@hadoop1 hadoop]# sbin/start-dfs.sh

访问: http://192.168.137.130:50070界面

向AI问一下细节

免责声明:本站发布的内容(图片、视频和文字)以原创、转载和分享为主,文章观点不代表本网站立场,如果涉及侵权请联系站长邮箱:is@yisu.com进行举报,并提供相关证据,一经查实,将立刻删除涉嫌侵权内容。

AI