《MY大数据开发笔记》----Hadoop默认端口应用

深藏阁楼爱情的钟 2022-03-29 11:58 349阅读 0赞

Hadoop集群的各部分一般都会使用到多个端口，有些是daemon之间进行交互之用，有些是用于RPC访问以及HTTP访问。而随着Hadoop周边组件的增多，完全记不住哪个端口对应哪个应用，特收集记录如此，以便查询。

这里包含我们使用到的组件：HDFS, YARN, HBase, Hive, ZooKeeper。

<table> 
 <tbody> 
  <tr> 
   <td>组件</td> 
   <td>Daemon</td> 
   <td>端口</td> 
   <td>配置</td> 
   <td>说明</td> 
  </tr> 
  <tr> 
   <td>HDFS</td> 
   <td>DataNode</td> 
   <td>50010</td> 
   <td>dfs.datanode.address</td> 
   <td>datanode服务端口，用于数据传输</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>50075</td> 
   <td>dfs.datanode.http.address</td> 
   <td>http服务的端口</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>50475</td> 
   <td>dfs.datanode.https.address</td> 
   <td>https服务的端口</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>50020</td> 
   <td>dfs.datanode.ipc.address</td> 
   <td>ipc服务的端口</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>NameNode</td> 
   <td>50070</td> 
   <td>dfs.namenode.http-address</td> 
   <td>http服务的端口</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>50470</td> 
   <td>dfs.namenode.https-address</td> 
   <td>https服务的端口</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>8020</td> 
   <td>fs.defaultFS</td> 
   <td>接收Client连接的RPC端口，用于获取文件系统metadata信息。</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>journalnode</td> 
   <td>8485</td> 
   <td>dfs.journalnode.rpc-address</td> 
   <td>RPC服务</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>8480</td> 
   <td>dfs.journalnode.http-address</td> 
   <td>HTTP服务</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>ZKFC</td> 
   <td>8019</td> 
   <td>dfs.ha.zkfc.port</td> 
   <td>ZooKeeper FailoverController，用于NN HA</td> 
  </tr> 
  <tr> 
   <td>YARN</td> 
   <td>ResourceManager</td> 
   <td>8032</td> 
   <td>yarn.resourcemanager.address</td> 
   <td>RM的applications manager(ASM)端口</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>8030</td> 
   <td>yarn.resourcemanager.scheduler.address</td> 
   <td>scheduler组件的IPC端口</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>8031</td> 
   <td>yarn.resourcemanager.resource-tracker.address</td> 
   <td>IPC</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>8033</td> 
   <td>yarn.resourcemanager.admin.address</td> 
   <td>IPC</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>8088</td> 
   <td>yarn.resourcemanager.webapp.address</td> 
   <td>http服务端口</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>NodeManager</td> 
   <td>8040</td> 
   <td>yarn.nodemanager.localizer.address</td> 
   <td>localizer IPC</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>8042</td> 
   <td>yarn.nodemanager.webapp.address</td> 
   <td>http服务端口</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>8041</td> 
   <td>yarn.nodemanager.address</td> 
   <td>NM中container manager的端口</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>JobHistory Server</td> 
   <td>10020</td> 
   <td>mapreduce.jobhistory.address</td> 
   <td>IPC</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>19888</td> 
   <td>mapreduce.jobhistory.webapp.address</td> 
   <td>http服务端口</td> 
  </tr> 
  <tr> 
   <td>HBase</td> 
   <td>Master</td> 
   <td>60000</td> 
   <td>hbase.master.port</td> 
   <td>IPC</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>60010</td> 
   <td>hbase.master.info.port</td> 
   <td>http服务端口</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>RegionServer</td> 
   <td>60020</td> 
   <td>hbase.regionserver.port</td> 
   <td>IPC</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>60030</td> 
   <td>hbase.regionserver.info.port</td> 
   <td>http服务端口</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>HQuorumPeer</td> 
   <td>2181</td> 
   <td>hbase.zookeeper.property.clientPort</td> 
   <td>HBase-managed ZK mode，使用独立的ZooKeeper集群则不会启用该端口。</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>2888</td> 
   <td>hbase.zookeeper.peerport</td> 
   <td>HBase-managed ZK mode，使用独立的ZooKeeper集群则不会启用该端口。</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>3888</td> 
   <td>hbase.zookeeper.leaderport</td> 
   <td>HBase-managed ZK mode，使用独立的ZooKeeper集群则不会启用该端口。</td> 
  </tr> 
  <tr> 
   <td>Hive</td> 
   <td>Metastore</td> 
   <td>9083</td> 
   <td>/etc/default/hive-metastore中export PORT=&lt;port&gt;来更新默认端口</td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>HiveServer</td> 
   <td>10000</td> 
   <td>/etc/hive/conf/hive-env.sh中export HIVE_SERVER2_THRIFT_PORT=&lt;port&gt;来更新默认端口</td> 
   <td>&nbsp;</td> 
  </tr> 
  <tr> 
   <td>ZooKeeper</td> 
   <td>Server</td> 
   <td>2181</td> 
   <td>/etc/zookeeper/conf/zoo.cfg中clientPort=&lt;port&gt;</td> 
   <td>对客户端提供服务的端口</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>2888</td> 
   <td>/etc/zookeeper/conf/zoo.cfg中server.x=[hostname]:nnnnn[:nnnnn]，标蓝部分</td> 
   <td>follower用来连接到leader，只在leader上监听该端口。</td> 
  </tr> 
  <tr> 
   <td>&nbsp;</td> 
   <td>&nbsp;</td> 
   <td>3888</td> 
   <td>/etc/zookeeper/conf/zoo.cfg中server.x=[hostname]:nnnnn[:nnnnn]，标蓝部分</td> 
   <td>用于leader选举的。只在electionAlg是1,2或3(默认)时需要。</td> 
  </tr> 
 </tbody> 
</table>

所有端口协议均基于TCP。

对于存在Web UI（HTTP服务）的所有hadoop daemon，有如下url：

**/logs**  
日志文件列表，用于下载和查看

**/logLevel**  
允许你设定log4j的日志记录级别，类似于hadoop daemonlog

**/stacks**  
所有线程的stack trace，对于debug很有帮助

**/jmx**  
服务端的Metrics，以JSON格式输出。

/jmx?qry=Hadoop:\*会返回所有hadoop相关指标。  
/jmx?get=MXBeanName::AttributeName 查询指定bean指定属性的值，例如/jmx?get=Hadoop:service=NameNode,name=NameNodeInfo::ClusterId会返回ClusterId。  
这个请求的处理类：org.apache.hadoop.jmx.JMXJsonServlet

而特定的Daemon又有特定的URL路径特定相应信息。

**NameNode**:http://:50070/

**/dfshealth.jsp**  
HDFS信息页面，其中有链接可以查看文件系统

/dfsnodelist.jsp?whatNodes=(DEAD|LIVE)  
显示DEAD或LIVE状态的datanode

**/fsck**  
运行fsck命令，不推荐在集群繁忙时使用！

**DataNode**:http://:50075/

**/blockScannerReport**  
每个datanode都会指定间隔验证块信息

参考：  
[http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml][http_hadoop.apache.org_docs_current_hadoop-project-dist_hadoop-hdfs_hdfs-default.xml]  
[http://blog.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/][http_blog.cloudera.com_blog_2009_08_hadoop-default-ports-quick-reference]

[http_hadoop.apache.org_docs_current_hadoop-project-dist_hadoop-hdfs_hdfs-default.xml]: http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
[http_blog.cloudera.com_blog_2009_08_hadoop-default-ports-quick-reference]: http://blog.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/