hadoop搭建之hadoop安装

落日映苍穹つ 2022-06-07 13:43 279阅读 0赞

> 1.下载hadoop ，推荐去官网下载：请点击    [官网地址][Link 1]（我下载的是2.6.5版本的）
> 
> 
> 
> 
> 2，把它进行拷贝到linux中;
> 
> 
> 
> 
> 3，在usr下解压其压缩文件
> 
>  命令【tar -zxvf +文件名】
> 
> 
> 
> 
> 4，删除压缩包
> 
> 命令【rm -rf +文件名】
> 
> 
> 
> 
> 5，将解压后的文件改名为“hadoop”
> 
> 命令【mv +旧文件名+hadoop】
> 
> 
> 
> 
> 6，根据hadoop文件的位置编辑它的环境变量
> 
> 命令【vim /etc/bashrc】
> 
> 添加以下内容：
> 
> <table> 
> <tbody> 
> <tr> 
> <td>#set hadoop env export HADOOP_HOME=/usr/hadoop export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH </td> 
> </tr> 
> </tbody> 
> </table>
> 
> 
> 
> 
> 补充：关于sbin的目录的配置可以，在任何位置启动hadoop
> 
> 7，使其上面配置的文件生效
> 
> 命令【source /etc/bashrc】
> 
> 8,hadoop是否能够启动的重点来了，那就是文件的配置（/usr/hadoop/etc/hadoop/下的文件）

*   *  hadoop-env.sh :主要添加jdk的环境变量

>  
> 
> <table> 
> <tbody> 
> <tr> 
> <td>export JAVA_HOME=/usr/java/jdk1.7.0_80 </td> 
> </tr> 
> </tbody> 
> </table>

*   *  core-site.xml : 添加内容如下(小括号的内容不必填写，但所描述的目录需要自己建立)

> <table> 
> <tbody> 
> <tr> 
> <td> &nbsp;&lt;configuration&gt;&nbsp; &nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;property&gt;&nbsp;&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; &lt;name&gt;fs.defaultFS&lt;/name&gt;&nbsp; &nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; &lt;value&gt;hdfs://172.21.3.73:9000&lt;/value&gt;&nbsp; &nbsp; (默认的HDFS路径) &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&lt;/property&gt; &nbsp; &nbsp; &lt;property&gt;&nbsp; &nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; &nbsp; &lt;name&gt;hadoop.tmp.dir&lt;/name&gt;&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&nbsp; &nbsp;&lt;value&gt;/home/hadoopdir/tmp&lt;/value&gt;&nbsp; (默认是NameNode、DataNode、JournalNode等存放数据的公共目录) &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&lt;/property&gt; &lt;configuration&gt;</td> 
> </tr> 
> </tbody> 
> </table>

*   *   hdfs-site.xml : 添加内容如下

> <table> 
> <tbody> 
> <tr> 
> <td> &lt;configuration&gt; &nbsp;&lt;property&gt; &nbsp; &nbsp; &nbsp;&lt;name&gt;dfs.namenode.name.dir&lt;/name&gt; &nbsp; &nbsp; &nbsp;&lt;value&gt;file:///home/hadoopdir/dfs/name&lt;/value&gt; &nbsp; &lt;/property&gt; &lt;property&gt; &nbsp; &nbsp; &nbsp;&lt;name&gt;dfs.namenode.data.dir&lt;/name&gt; &nbsp; &nbsp; &nbsp;&lt;value&gt;file:///home/hadoopdir/dfs/data&lt;/value&gt; &nbsp; &lt;/property&gt; &nbsp; &lt;property&gt; &nbsp; &nbsp; &nbsp;&lt;name&gt;dfs.replication&lt;/name&gt; &nbsp; &nbsp; &nbsp;&lt;value&gt;3&lt;/value&gt;（指定DataNode存储block的副本数量。默认值是3个，我们现在有4个DataNode，该值不大于4即可。） &nbsp; &lt;/property&gt; &lt;property&gt; &nbsp; &nbsp; &nbsp;&lt;name&gt;dfs.namenode.secondary.http-address&lt;/name&gt; &nbsp; &nbsp; &nbsp;&lt;value&gt;172.21.3.73:50090&lt;/value&gt; &nbsp; &lt;/property&gt; &lt;property&gt; &nbsp; &nbsp; &nbsp;&lt;name&gt;dfs.webhdfs.enabled&lt;/name&gt; &nbsp; &nbsp; &nbsp;&lt;value&gt;true&lt;/value&gt; &nbsp; &lt;/property&gt; &lt;/configuration&gt; </td> 
> </tr> 
> </tbody> 
> </table>
> 
> 
> 
> 
> 
>

*   *  yarn-site.xml : 添加内容如下

> <table> 
> <tbody> 
> <tr> 
> <td>&lt;configuration&gt; &lt;!-- Site specific YARN configuration properties --&gt; &lt;property&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;name&gt;yarn.resourcemanager.hostname&lt;/name&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;value&gt;master.hadoop&lt;/value&gt; &nbsp; &nbsp;&lt;/property&gt; &lt;property&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;name&gt;yarn.nodemanager.aux-services&lt;/name&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;value&gt;mapreduce_shuffle&lt;/value&gt; &nbsp; &nbsp;&lt;/property&gt; &nbsp; &lt;property&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;name&gt;yarn.resourcemanager.address&lt;/name&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;value&gt;172.21.3.73:8032&lt;/value&gt; &nbsp; &nbsp;&lt;/property&gt; &nbsp; &nbsp;&lt;property&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;name&gt;yarn.resourcemanager.scheduler.address&lt;/name&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;value&gt;172.21.3.73:8030&lt;/value&gt; &nbsp; &nbsp;&lt;/property&gt; &nbsp; &nbsp;&lt;property&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;name&gt;yarn.resourcemanager.resource-tracker.address&lt;/name&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;value&gt;172.21.3.73:8031&lt;/value&gt; &nbsp; &nbsp;&lt;/property&gt; &nbsp; &nbsp;&lt;property&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;name&gt;yarn.resourcemanager.admin.address&lt;/name&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;value&gt;172.21.3.73:8033&lt;/value&gt; &nbsp; &nbsp;&lt;/property&gt; &nbsp; &nbsp;&lt;property&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;name&gt;yarn.resourcemanager.webapp.address&lt;/name&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;value&gt;172.21.3.73:8088&lt;/value&gt; &nbsp; &nbsp;&lt;/property&gt; &lt;/configuration </td> 
> </tr> 
> </tbody> 
> </table>
> 
> 
>

*   *  mapred-site.xml  (这个文件本身是没有的，cp   mapre-site.xml.template  mapre-site.xml)

> <table> 
> <tbody> 
> <tr> 
> <td>&lt;configuration&gt; &nbsp; &nbsp; &lt;property&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;name&gt;mapreduce.framework.name&lt;/name&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;value&gt;yarn&lt;/value&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;final&gt;true&lt;/final&gt; &nbsp; &nbsp;&lt;/property&gt; &nbsp; &nbsp; &lt;property&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;name&gt;mapreduce.jobtracker.http.address&lt;/name&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;value&gt;172.21.3.73:50030&lt;/value&gt; &nbsp; &nbsp; &lt;/property&gt; &nbsp; &nbsp;&lt;property&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;name&gt;mapreduce.jobhistory.address&lt;/name&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;value&gt;http://172.21.3.73:10020&lt;/value&gt; &nbsp; &nbsp;&lt;/property&gt; &nbsp; &nbsp; &lt;property&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;name&gt;mapreduce.jobhistory.address&lt;/name&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;value&gt;172.21.3.73:19888&lt;/value&gt; &nbsp; &nbsp; &lt;/property&gt; &nbsp; &nbsp; &nbsp; &nbsp; &lt;property&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;name&gt;mapred.job.tracker&lt;/name&gt; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &lt;value&gt;http://172.21.3.73:9001&lt;/value&gt; &nbsp; &nbsp; &nbsp; &nbsp; &lt;/property&gt; &lt;/configuration&gt; </td> 
> </tr> 
> </tbody> 
> </table>
> 
> 9，在Slaves 里配置节点信息(只配置从节点就可以了)
> 
> <table> 
> <tbody> 
> <tr> 
> <td>Slave1.Hadoop Slave2.Hadoop Slave3.Hadoop</td> 
> </tr> 
> </tbody> 
> </table>
> 
> 
> 
> 
> 10,接下来就可以进行大胆的验证了，集群启动时需要格式化
> 
> 命令【hadoop namenode -format】
> 
> 
> 
> 
> 11,启动集群
> 
> 命令【start-all.sh】
> 
> 
> 
> 
> 12，可以在网页上看出集群的状态（当然 也可以直接：jps 查询）
> 
> 两个网页【主节点ip：50070】，【主节点ip; 9000】
> 
> 
> 
> 
> 12,当然中间会出现很多问题，接下来我会慢慢补充的。。。。。！

[Link 1]: http://hadoop.apache.org/releases.html