大数据学习各技术参考文档及虚拟机配置

小咪咪 2022-04-15 05:47 25阅读 0赞

环境搭建的文章对于简单的不再更新了，稍微复杂点的再写博客记录吧

### 参考文档地址（持续新增） ###

<table> 
 <thead> 
  <tr> 
   <th>标题</th> 
   <th>地址</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>Hadoop Yarn Container 资源分配</td> 
   <td><a href="https://blog.csdn.net/szh1124/article/details/76178699?tdsourcetag=s_pcqq_aiomsg" rel="nofollow">https://blog.csdn.net/szh1124/article/details/76178699?tdsourcetag=s_pcqq_aiomsg</a></td> 
  </tr> 
  <tr> 
   <td>map数量如何控制</td> 
   <td><a href="https://www.cnblogs.com/junneyang/p/5850440.html" rel="nofollow">https://www.cnblogs.com/junneyang/p/5850440.html</a></td> 
  </tr> 
  <tr> 
   <td>MapReduce 程序内部数据处理流程全解析</td> 
   <td><a href="http://www.aboutyun.com/thread-15494-1-2.html" rel="nofollow">http://www.aboutyun.com/thread-15494-1-2.html</a><br><a href="https://blog.csdn.net/qq_17776287/article/details/78176515" rel="nofollow">https://blog.csdn.net/qq_17776287/article/details/78176515</a><br><a href="https://blog.csdn.net/aa518189/article/details/80020857" rel="nofollow">https://blog.csdn.net/aa518189/article/details/80020857</a></td> 
  </tr> 
  <tr> 
   <td>maptask 并行度决定机制</td> 
   <td><a href="https://blog.csdn.net/tototuzuoquan/article/details/72851603" rel="nofollow">https://blog.csdn.net/tototuzuoquan/article/details/72851603</a> <br><a href="https://blog.csdn.net/wu_cai/article/details/77869610" rel="nofollow">https://blog.csdn.net/wu_cai/article/details/77869610</a></td> 
  </tr> 
  <tr> 
   <td>Shuffle过程介绍</td> 
   <td><a href="https://blog.csdn.net/txbsw/article/details/80760285" rel="nofollow">https://blog.csdn.net/txbsw/article/details/80760285</a></td> 
  </tr> 
  <tr> 
   <td>详解 WordCount 运行后历史日志记录</td> 
   <td><a href="https://blog.csdn.net/u011414200/article/details/50372331" rel="nofollow">https://blog.csdn.net/u011414200/article/details/50372331</a></td> 
  </tr> 
  <tr> 
   <td>Hadoop HDFS高可用（HA）</td> 
   <td><a href="https://blog.csdn.net/bingduanlbd/article/details/51946540" rel="nofollow">https://blog.csdn.net/bingduanlbd/article/details/51946540</a></td> 
  </tr> 
  <tr> 
   <td>理解Hadoop YARN架构</td> 
   <td><a href="https://blog.csdn.net/bingduanlbd/article/details/51880019" rel="nofollow">https://blog.csdn.net/bingduanlbd/article/details/51880019</a></td> 
  </tr> 
  <tr> 
   <td>MapReduce shuffle过程剖析及调优</td> 
   <td><a href="https://blog.csdn.net/bingduanlbd/article/details/51933914" rel="nofollow">https://blog.csdn.net/bingduanlbd/article/details/51933914</a></td> 
  </tr> 
  <tr> 
   <td>MapReduce shuffle过程详解</td> 
   <td><a href="https://blog.csdn.net/u014374284/article/details/49205885" rel="nofollow">https://blog.csdn.net/u014374284/article/details/49205885</a></td> 
  </tr> 
  <tr> 
   <td>Hadoop MapReduce原理及实例</td> 
   <td><a href="https://blog.csdn.net/bingduanlbd/article/details/51924398" rel="nofollow">https://blog.csdn.net/bingduanlbd/article/details/51924398</a></td> 
  </tr> 
  <tr> 
   <td>LCS</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>HMM隐马尔科夫</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>协同过滤</td> 
   <td></td> 
  </tr> 
  <tr> 
   <td>贝叶斯</td> 
   <td><a href="https://baike.baidu.com/tashuo/browse/content?id=bbf9b41cf33c09f9d4856750&amp;lemmaId=&amp;fromLemmaModule=pcBottom" rel="nofollow">https://baike.baidu.com/tashuo/browse/content?id=bbf9b41cf33c09f9d4856750&amp;lemmaId=&amp;fromLemmaModule=pcBottom</a></td> 
  </tr> 
  <tr> 
   <td>朴素贝叶斯</td> 
   <td><a href="https://blog.csdn.net/fisherming/article/details/79509025" rel="nofollow">https://blog.csdn.net/fisherming/article/details/79509025</a><br><a href="https://blog.csdn.net/li8zi8fa/article/details/76176597" rel="nofollow">https://blog.csdn.net/li8zi8fa/article/details/76176597</a></td> 
  </tr> 
  <tr> 
   <td>逻辑回归</td> 
   <td><a href="https://blog.csdn.net/saltriver/article/details/63681339" rel="nofollow">https://blog.csdn.net/saltriver/article/details/63681339</a></td> 
  </tr> 
  <tr> 
   <td>逻辑回归——梯度下降法</td> 
   <td><a href="https://www.jianshu.com/p/c7e642877b0e" rel="nofollow">https://www.jianshu.com/p/c7e642877b0e</a></td> 
  </tr> 
 </tbody> 
</table>

--------------------

### 虚拟机配置 ###

**虚拟机：** Vmware12

--------------------

**操作系统：** CentOS7.5

--------------------

**系统内常用组件：** lrzsz、vim、yum、wget

--------------------

**主从机地址：**  
Master —> 192.168.202.10  
Slave1 —> 192.168.202.11  
Slave2 —> 192.168.202.12

--------------------

**软件版本（及时更新）**  
Anaconda3 用于切换python2和3版本和管理python所需依赖  
Maria DB  
Hadoop2.7.7  
HBase1.3.1  
Zookeeper 3.4.11  
Scala 2.11.12  
Spark 2.0.2  
Hive 1.2.2  
Flume 1.9.0  
Kafka 2.11-2.1.0

--------------------