HBase系列之-Hadoop环境安装

1.创建一个单独HBase用户

[root@instance-n4r06itt ~]# useradd hadoop
[root@instance-n4r06itt ~]# passwd hadoop
Changing password for user hadoop.
New password: 
BAD PASSWORD: The password is shorter than 8 characters
Retype new password: 
passwd: all authentication tokens updated successfully.

2.SSH设置和密钥生成

SSH设置需要在集群上执行不同的操作,如启动,停止和分布式守护shell操作。进行身份验证不同的Hadoop用户,需要一种用于Hadoop的用户提供的公钥/私钥对,并用不同的用户共享。

以下的命令被用于生成使用SSH密钥值对。复制公钥从id_rsa.pub为authorized_keys,并提供所有者,读写权限到authorized_keys文件。

密钥对生成方式参考:自动化构建系列之-GItlap添加SSH keys

[root@instance-n4r06itt ~]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[root@instance-n4r06itt ~]# cd ~/.ssh
[root@instance-n4r06itt .ssh]# ll
total 16
-rw------- 1 root root  398 May 19 11:00 authorized_keys
-rw------- 1 root root 1679 Apr 30 16:17 id_rsa
-rw-r--r-- 1 root root  398 Apr 30 16:17 id_rsa.pub
-rw-r--r-- 1 root root  175 May 13 18:17 known_hosts
[root@instance-n4r06itt .ssh]# chmod 0600 ~/.ssh/authorized_keys

[root@instance-n4r06itt ~]# ssh 106.12.175.83
Last login: Tue May 19 11:01:40 2020 from ::1

3.Java Jdk环境变量配置

参考:linux系列之-jdk环境变量配置

4.Hadoop下载以及服务安装

# 下载hadoop
wget http://mirrors.advancedhosters.com/apache/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
# 解压
tar zxvf hadoop-3.2.1.tar.gz

修改四个配置文件:

1)修改hadoop-env.sh

vi hadoop-env.sh

添加JDK配置

export JAVA_HOME=/opt/middleware/jdk1.8.0_171

2)修改yarn-env.sh

vi yarn-env.sh

添加JDK配置

export JAVA_HOME=/opt/middleware/jdk1.8.0_171

3)修改core-site.xml

      <configuration>
      <!-- 指定HDFS老大(namenode)的通信地址 -->
            <property>
                    <name>fs.defaultFS</name>
                    <value>hdfs://0.0.0.0:9000</value>
            </property>
            <!-- 指定hadoop运行时产生文件的存储路径 -->
            <property>
                    <name>hadoop.tmp.dir</name>
                    <value>/usr/src/hadoop-3.2.1/tmp</value>
            </property>
    </configuration>

4)修改mapred-site.xml

       <configuration>
      <!-- 通知框架MR使用YARN -->
            <property>
                    <name>mapreduce.framework.name</name>
                    <value>yarn</value>
            </property>
    </configuration>

5)修改hdfs-site.xml

      <configuration>
      <property>
        <name>dfs.replication</name>
        <value>1</value>
      </property>
      <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/usr/src/hadoop-3.2.1/tmp_dfs/name</value>
      </property>
      <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/usr/src/hadoop-3.2.1/tmp_dfs/dfs/data</value>
      </property>
           <property>
               <name>dfs.http.address</name>
               <value>0.0.0.0:9870</value>
           </property>
    </configuration>

6)修改yarn-site.xml

      <configuration>
            <!-- reducer取数据的方式是mapreduce_shuffle -->
            <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
            </property>
      </configuration>

7)将hadoop添加到环境变量中

vim /etc/profile
export JAVA_HOME=/opt/middleware/jdk1.8.0_171
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/jre/lib:${JRE_HOME}/lib
export PATH=$PATH:${JAVA_HOME}/bin:${HADOOP_HOME}/bin
export HADOOP_HOME=/usr/src/hadoop-3.2.1
vim /etc/bashrc
export HADOOP_HOME=/usr/src/hadoop-3.2.1
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_INSTALL=$HADOOP_HOME
source /etc/profile
source /etc/bashrc

8)格式化HDFS(namenode)第一次使用时要格式化

[root@instance-n4r06itt bin]# ./hdfs namenode -format

 5.Hadoop验证

启动:./start-dfs.sh 出现错误

解决办法,start-dfs.sh以及stop-dfs.sh添加如下配置:

#!/usr/bin/env bash

HDFS_DATANODE_USER=root
#HADOOP_SECURE_DN_USER=hdfs
HDFS_DATANODE_SECURE_USER=hdfs  
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

start-yarn.sh以及stop-yarn.sh添加如下配置:

#!/usr/bin/env bash

YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

再次启动服务

[root@instance-n4r06itt sbin]# ./start-dfs.sh 
Starting namenodes on [0.0.0.0]
Last login: Wed May 20 15:44:24 CST 2020 on pts/1
Starting datanodes
Last login: Wed May 20 15:46:54 CST 2020 on pts/1
Starting secondary namenodes [instance-n4r06itt]
Last login: Wed May 20 15:46:57 CST 2020 on pts/1
2020-05-20 15:47:24,789 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@instance-n4r06itt sbin]# jps
25408 DataNode
25799 Jps
25657 SecondaryNameNode
1291 jenkins.war
25262 NameNode
[root@instance-n4r06itt sbin]# ./start-yarn.sh 
Starting resourcemanager
Last login: Wed May 20 15:47:10 CST 2020 on pts/1
Starting nodemanagers
Last login: Wed May 20 15:47:41 CST 2020 on pts/1

服务启动成功!需要注意的是Hadoop每次启动关闭后要重新格式化HDFS(namenode)

[root@instance-n4r06itt bin]# ./hdfs namenode -format

我们来访问两个页面来看一看,能正常访问:HDFS管理界面(http://www.qy-bb.club:9870) 以及 MR管理界面(http://www.qy-bb.club:8088

 

还需要主义的是3.0以前HDFS管理界面 端口为50070端口,3.0以后版本默认为9870端口,我就是在端口方面走了弯路!!!

 

至此,Hadoop在Linux系统下的安装就完毕了!!!谢谢!!!

已标记关键词 清除标记
相关推荐
©️2020 CSDN 皮肤主题: 酷酷鲨 设计师:CSDN官方博客 返回首页