hjm | gyt | |
---|---|---|
HDFS | NameNode\SecondaryNameNode\DataNode | DataNode |
YARN | ResourceManager\NodeManager | NodeManager |
useradd hujinming passwd hujinming配置用户sudo权限
vim /etc/sudoers在
## Allow root to run any commands anywhere root ALL=(ALL) ALL ## Allows people in group wheel to run all commands %wheel ALL=(ALL) ALL下面新增一行
hujinming ALL=(ALL) NOPASSWD:ALL创建目录并更改权限
mkdir /opt/module mkdir /opt/software切换到root用户下,修改 module、software 文件夹的所有者和所属组均为hujinming用户
chown hujinming:hujinming /opt/module chown hujinming:hujinming /opt/software查看 module、software 文件夹的所有者和所属组
ll
tar -zxvf jdk-8u212-linux.x64.tar.gz -C /opt/module/3.配置jdk环境变量
sudo vim /etc/profile.d/my_env.sh3.2 添加如下内容
#JAVA_HOME export JAVA_HOME=/opt/module/jdk1.8.0_212 export PATH=$PATH:$JAVA_HOME/bin3.3 保存后退出,source 一下/etc/profile 文件,让新的环境变量 PATH 生效
source /etc/profile3.4 测试jdk是否安装成功
java -version
pwd3.2 打开/etc/profile.d/my_env.sh 文件
sudo vim /etc/profile.d/my_env.sh3.3 在 my_env.sh 文件末尾添加如下内容:
#HADOOP_HOME export HADOOP_HOME=/opt/module/hadoop-3.1.3 export PATH=$PATH:$HADOOP_HOME/bin export PATH=$PATH:$HADOOP_HOME/sbin4. 保存退出,让修改后的文件生效
vim /etc/hostname分别把两台服务器的名字改成hjm和gyt,这里以gyt举例,直接在hostname文件上输入
gyt2.改映射文件
vim /etc/hosts在linux中键入ifconfig命令可以查看内网ip。在两台服务器中,填写自己的私网,访问别人的填写公网,这里以gyt为例,gyt的公网IP是175.178.236.48,内网IP是10.0.12.1。这里要注意一点,阿里云在hosts文件中已经将本地IP映射成了一串英文,把这行信息删掉再进行上面的操作
47.115.207.108 hjm 10.0.12.1 gyt3.在客户端电脑(默认windows)配置映射
175.178.236.48 gyt 47.115.207.108 hjm
hjm -> gyt gyt -> hjm hjm -> hjm gyt -> gyt注意切换对应用户操作。先cd到~/.ssh,生成公钥和私钥
ssh-keygen -t rsa这里以gyt -> hjm或hjm -> hjm为例:
ssh-copy-id hjm
<configuration> <!-- 指定 NameNode 的地址 --> <property> <name>fs.defaultFS</name> <value>hdfs://hjm:8020</value> </property> <!-- 指定 hadoop 数据的存储目录 --> <property> <name>hadoop.tmp.dir</name> <value>/opt/module/hadoop-3.1.3/data</value> </property> <!-- 配置 HDFS 网页登录使用的静态用户为 root --> <property> <name>hadoop.http.staticuser.user</name> <value>hujinming</value> </property> </configuration>hdfs-site.xml
<configuration> <!-- nn web 端访问地址--> <property> <name>dfs.namenode.http-address</name> <value>hjm:9870</value> </property> <!-- 2nn web 端访问地址--> <property> <name>dfs.namenode.secondary.http-address</name> <value>hjm:9868</value> </property> </configuration>yarn-site.xml
<configuration> <!-- 指定 MR 走 shuffle --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <!-- 指定 ResourceManager 的地址--> <property> <name>yarn.resourcemanager.hostname</name> <value>hjm</value> </property> <!-- 环境变量的继承 --> <property> <name>yarn.nodemanager.env-whitelist</name> <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CO NF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAP RED_HOME</value> </property> </configuration>mapred-site.xml
<configuration> <!-- 指定 MapReduce 程序运行在 Yarn 上 --> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
hjm gyt
.同时,对公网暴露9864、9866、9870、9868端口
Starting namenodes on [hjm] ERROR: Attempting to operate on hdfs namenode as root ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation. Starting datanodes ERROR: Attempting to operate on hdfs datanode as root ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation. Starting secondary namenodes [hjm] ERROR: Attempting to operate on hdfs secondarynamenode as root ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation.解决方法:
HDFS_DATANODE_USER=hujinming HADOOP_SECURE_SECURE_USER=hdfs HDFS_NAMENODE_USER=hujinming HDFS_SECONDARYNAMENODE_USER=hujinming在hjm上的start-yarn.sh和stop-yarn.sh上增加如下几行:
YARN_RESOURCEMANAGER_USER=hujinming HADOOP_SECURE_DN_USER=yarn YARN_NODEMANAGER_USER=hujinming2.报错hjm: ERROR: Cannot set priority of namenode process 23214,没有启动NameNode和SecondaryNameNode
<!--web console cors settings--> <property> <name>hadoop.http.filter.initializers</name> <value>org.apache.hadoop.security.HttpCrossOriginFilterInitializer</value> </property> <property> <name>hadoop.http.cross-origin.enabled</name> <value>true</value> </property> <property> <name>hadoop.http.cross-origin.allowed-origins</name> <value>*</value> </property> <property> <name>hadoop.http.cross-origin.allowed-methods</name> <value>*</value> </property> <property> <name>hadoop.http.cross-origin.allowed-headers</name> <value>X-Requested-With,Content-Type,Accept,Origin</value> </property> <property> <name>hadoop.http.cross-origin.max-age</name> <value>1800</value> </property>6.没有足够的副本数量?
解决方法:在master主机的slaves文件中删除localhost即可。