【HBase】HBase系列之HBase概述及环境搭建

短命女 2023-10-05 11:26 74阅读 0赞

#### HBase ####

*  Apache HBase
 *   *  HBase简介
     *  特性
     *  环境构建
     *   *  架构草图
         *  单机搭建
     *  技巧

## Apache HBase ##

### HBase简介 ###

**Apache HBase**是一个**分布式、面向列**的开源数据库，该技术来源于 Fay Chang 所撰写的Google论文“Bigtable：一个结构化数据的分布式存储系统”。就像Bigtable利用了Google文件系统（File System）所提供的分布式数据存储一样，HBase在Hadoop的HDFS之上提供了类似于Bigtable的能力。

**HDFS和HBase之间的关系**

> HDFS is a distributed file system that is well suited for the storage of large files. Its documentation states that it is not, however, a general purpose file system, and does not provide fast individual record lookups in files. HBase, on the other hand, is built on top of HDFS and provides fast record lookups (and updates) for large tables. This can sometimes be a point of conceptual confusion. HBase internally puts your data in indexed “StoreFiles” that exist on HDFS for high-speed lookups.  
> \----**摘自官网**
> 
> HDFS是一种分布式文件系统，非常适合存储大型文件。然而，它的文档声明它不是一个通用的文件系统，并且不提供快速的单个记录在文件中的查找。另一方面，HBase构建在HDFS之上，为大型表提供快速的记录查找(和更新)。这有时是概念上的混乱。HBase内部将数据放在索引的**存储文件**中，这些文件存在于HDFS中用于高速查找。

**自译：HBase的全称Hadoop Database**，HBase是构建在HDFS之上的一款**数据存储服务**，所有的物理数据都是存储在HDFS之上，HBase仅仅是提供了对HDFS上数据的索引能力，继而实现对海量数据的随机读写。相比较于HDFS文件系统仅仅只是提供了海量数据的存储和下载，并不能实现海量数据的交互，例如：用户想修改HDFS中一条文本记录。

![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-oNuCiLlb-1602292051761)(assets/image-20201009100631144.png)\]][img-oNuCiLlb-1602292051761_assets_image-20201009100631144.png]  
**什么时候使用HBase**

> First, make sure you have enough data. If you have hundreds of millions or billions of rows, then HBase is a good candidate. If you only have a few thousand/million rows, then using a traditional RDBMS might be a better choice due to the fact that all of your data might wind up on a single node (or two) and the rest of the cluster may be sitting idle.
> 
> Second, make sure you can live without all the extra features that an RDBMS provides (e.g., typed columns, secondary indexes, transactions, advanced query languages, etc.) An application built against an RDBMS cannot be “ported” to HBase by simply changing a JDBC driver, for example. Consider moving from an RDBMS to HBase as a complete redesign as opposed to a port.
> 
> Third, make sure you have enough hardware. Even HDFS doesn’t do well with anything less than 5 DataNodes (due to things such as HDFS block replication which has a default of 3), plus a NameNode.
> 
> HBase can run quite well stand-alone on a laptop - but this should be considered a development configuration only.  
>   
> **译文：** 首先，确保你有足够的数据。如果您有数亿或数十亿行，那么HBase是一个不错的选择。如果您只有几千/几百万行，那么使用传统的RDBMS可能是更好的选择，因为您的所有数据可能都在一个(或两个)节点上，而集群的其余部分可能处于空闲状态。
> 
> 其次，确保没有RDBMS提供的所有额外特性(例如，类型化列、二级索引、事务、高级查询语言等)，基于RDBMS构建的应用程序不能通过简单地更改JDBC驱动程序来“移植”到HBase。把从RDBMS迁移到HBase看作是一次彻底的重新设计，而不是一个端口。
> 
> 第三，确保您有足够的硬件。甚至HDFS也不能很好地处理少于5个datanode(因为HDFS块复制的默认值是3)，再加上一个NameNode。
> 
> HBase独立运行在笔记本上也可以很好——但这只能作为开发配置。

### 特性 ###

HBase是NoSQL数据库中面向列存储的代表，在NoSQL设计中遵循**CP**设计原则（[CAP定理][CAP]）,其中HBase的面向列存储是HBASE之所以能够高性能的一个非常关键因素。面向列存储旨在提升系统磁盘利用率和IO利用率，其中所有NoSQL产品一般都能力很好提升磁盘利用率，因为所有的NoSQL产品都支持稀疏存储（null值不占用存储空间）。

![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-HPNjVuKL-1602292051766)(assets/image-20201009114413568.png)\]][img-HPNjVuKL-1602292051766_assets_image-20201009114413568.png]

### 环境构建 ###

#### 架构草图 ####

![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-NPVOcJtW-1602292051769)(assets/image-20201009115447048.png)\]][img-NPVOcJtW-1602292051769_assets_image-20201009115447048.png]

#### 单机搭建 ####

准备一台CentOS7 x64虚拟机，需提前配置好jdk环境、Hadoop·HDFS环境、关闭防火墙等操作，[详见此文][Link 1]  
1、安装配置**Zookeeper**，确保Zookeeper正常运行

*  上传zookeeper的安装包,并解压在`/usr`目录下

[root@CentOS ~]# tar -zxf zookeeper-3.4.6.tar.gz -C /usr/

*  配置Zookepeer的`zoo.cfg`

[root@CentOS ~]# cd /usr/zookeeper-3.4.6/
    [root@CentOS zookeeper-3.4.6]# cp conf/zoo_sample.cfg conf/zoo.cfg
    [root@CentOS zookeeper-3.4.6]# vi conf/zoo.cfg

# The number of milliseconds of each tick
    tickTime=2000
    # The number of ticks that the initial
    # synchronization phase can take
    initLimit=10
    # The number of ticks that can pass between
    # sending a request and getting an acknowledgement
    syncLimit=5
    # the directory where the snapshot is stored.
    # do not use /tmp for storage, /tmp here is just
    # example sakes.
    dataDir=/root/zkdata
    # the port at which the clients will connect
    clientPort=2181
    # the maximum number of client connections.
    # increase this if you need to handle more clients
    #maxClientCnxns=60
    #
    # Be sure to read the maintenance section of the
    # administrator guide before turning on autopurge.
    #
    # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
    #
    # The number of snapshots to retain in dataDir
    #autopurge.snapRetainCount=3
    # Purge task interval in hours
    # Set to "0" to disable auto purge feature
    #autopurge.purgeInterval=1

*  创建zookeeper的数据目录`zkdata`

[root@CentOS zookeeper-3.4.6]# mkdir /root/zkdata

*  启动zookeeper服务

[root@CentOS zookeeper-3.4.6]# ./bin/zkServer.sh start zoo.cfg
    JMX enabled by default
    Using config: /usr/zookeeper-3.4.6/bin/../conf/zoo.cfg
    Starting zookeeper ... STARTED

*  查看zookeeper服务是否正常

[root@CentOS zookeeper-3.4.6]# jps
    7121 Jps
    6934 QuorumPeerMain
    [root@CentOS zookeeper-3.4.6]# ./bin/zkServer.sh status zoo.cfg
    JMX enabled by default
    Using config: /usr/zookeeper-3.4.6/bin/../conf/zoo.cfg
    Mode: standalone

2、启动HDFS，[详见此文][Link 1]

[root@CentOS ~]# start-dfs.sh

3、安装配置HBase服务

*  上传Hbase安装包,并解压到`/usr`目录下

[root@CentOS ~]# tar -zxf hbase-1.2.4-bin.tar.gz -C /usr/

*  配置Hbase环境变量`HBASE_HOME`

[root@CentOS ~]# vi .bashrc
    JAVA_HOME=/usr/java/latest
    HADOOP_HOME=/usr/hadoop-2.9.2/
    HBASE_HOME=/usr/hbase-1.2.4/
    PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin
    CLASSPATH=.
    export JAVA_HOME
    export PATH
    export CLASSPATH
    export HADOOP_HOME
    HADOOP_CLASSPATH=(hadoop classpath):/root/mysql-connector-java-5.1.49.jar
    export HADOOP_CLASSPATH
    export HBASE_HOME
    [root@CentOS ~]# source .bashrc

*  配置`hbase-site.xml`

[root@CentOS ~]# cd /usr/hbase-1.2.4/
    [root@CentOS hbase-1.2.4]# vi conf/hbase-site.xml

<property>
        <name>hbase.rootdir</name>
        <value>hdfs://CentOS:9000/hbase</value>
    </property>
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>CentOS</value>
    </property>
    <property>
        <name>hbase.zookeeper.property.clientPort</name>
        <value>2181</value>
    </property>

*  修改hbase-env.sh,将`HBASE_MANAGES_ZK`修改为**false**

[root@CentOS hbase-1.2.4]# grep -i HBASE_MANAGES_ZK conf/hbase-env.sh
    # export HBASE_MANAGES_ZK=true

将128行的注释去掉，并且将true修改为**false**，大家可以在选择模式下使用`set nu`显示行号

[root@CentOS hbase-1.2.4]# grep -i HBASE_MANAGES_ZK conf/hbase-env.sh
    export HBASE_MANAGES_ZK=false

*  修改`regionservers`配置文件

[root@CentOS hbase-1.2.4]# vi conf/regionservers
    CentOS

*  启动Hbase

[root@CentOS ~]# start-hbase.sh
    starting master, logging to /usr/hbase-1.2.4//logs/hbase-root-master-CentOS.out
    Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
    Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
    CentOS: starting regionserver, logging to /usr/hbase-1.2.4//logs/hbase-root-regionserver-CentOS.out

*  检验是否安装

[root@CentOS ~]# jps
    13328 Jps
    12979 HRegionServer
    6934 QuorumPeerMain
    8105 NameNode
    12825 HMaster
    8253 DataNode
    8509 SecondaryNameNode

然后可以访问:`http://主机:16010`访问HBase主页

![\[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-rECKlrOn-1602292051774)(assets/image-20201009121418104.png)\]][img-rECKlrOn-1602292051774_assets_image-20201009121418104.png]

### 技巧 ###

一般HBase数据存储在HDFS上和Zookeeper上 ，由于用户的非法操作导致Zookeeper数据和HDFS中的数据不一致，这可能会导致无法正常使用HBase的服务，因此大家可以考虑：

*  停掉HBase服务

[root@CentOS ~]# stop-hbase.sh
    stopping hbase...........

*  清理HDFS和Zookeeper的残留数据

[root@CentOS ~]# hbase clean
    Usage: hbase clean (--cleanZk|--cleanHdfs|--cleanAll)
    Options:
            --cleanZk   cleans hbase related data from zookeeper.  #清除来自zookeeper的hbase相关数据
            --cleanHdfs cleans hbase related data from hdfs.   #从hdfs中清除hbase相关数据
            --cleanAll  cleans hbase related data from both zookeeper and hdfs.  #清除来自zookeeper和hdfs的hbase相关数据

*  重新启动HBase服务即可

> 如果用户希望排查具体启动失败的原因，可以使用tail -f指令查看HBase安装目录下的`logs/`目录下文件

--------------------

[img-oNuCiLlb-1602292051761_assets_image-20201009100631144.png]: https://img-blog.csdnimg.cn/20201010092023395.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ0NjY0MjMx,size_16,color_FFFFFF,t_70#pic_center
[CAP]: https://baike.baidu.com/item/CAP%E5%8E%9F%E5%88%99/5712863
[img-HPNjVuKL-1602292051766_assets_image-20201009114413568.png]: https://img-blog.csdnimg.cn/20201010092343180.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ0NjY0MjMx,size_16,color_FFFFFF,t_70#pic_center
[img-NPVOcJtW-1602292051769_assets_image-20201009115447048.png]: https://img-blog.csdnimg.cn/20201010092423603.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ0NjY0MjMx,size_16,color_FFFFFF,t_70#pic_center
[Link 1]: https://blog.csdn.net/qq_44664231/article/details/108804561
[img-rECKlrOn-1602292051774_assets_image-20201009121418104.png]: https://img-blog.csdnimg.cn/2020101009344618.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ0NjY0MjMx,size_16,color_FFFFFF,t_70#pic_center