Phoenix批量加载数据到HBase

落日映苍穹つ 2022-01-26 07:04 460阅读 0赞

1.  **通过PSQL加载**
    
    **用法:**
    
        bin/psql.py [-t table-name] [-h comma-separated-column-names | in-line]
                    [-d field-delimiter-char quote-char escape-char]<zookeeper>
                    <path-to-sql-or-csv-file>...
    
    <table> 
     <thead> 
      <tr> 
       <th>参数</th> 
       <th>描述</th> 
      </tr> 
     </thead> 
     <tbody> 
      <tr> 
       <td>-a,–array-separator </td> 
       <td>数组元素分隔符,默认 ‘:’</td> 
      </tr> 
      <tr> 
       <td>-b,–binaryEncoding </td> 
       <td>指定二进制编码</td> 
      </tr> 
      <tr> 
       <td>-d,–delimiter </td> 
       <td>提供自定义分隔符解析 CSV 文件</td> 
      </tr> 
      <tr> 
       <td>-e,–escape-character </td> 
       <td>提供自定义转义字符，默认为反斜杠</td> 
      </tr> 
      <tr> 
       <td>-h,–header </td> 
       <td>覆盖CSV数据映射到的列名称，区分大小写。内联的特殊值，表示CSV文件的第一行确定数据映射到的列</td> 
      </tr> 
      <tr> 
       <td>-l,–local-index-upgrade</td> 
       <td>用于通过将索引数据从单独的表移动到同一表中单独的列族来升级本地索引数据。</td> 
      </tr> 
      <tr> 
       <td>-m,–map-namespace </td> 
       <td>用于将表映射到与schema匹配的名称空间，要求phoenix.schema。isNamespaceMappingEnabled被启用</td> 
      </tr> 
      <tr> 
       <td>-q,–quote-character </td> 
       <td>自定义短语分隔符，默认为双引号</td> 
      </tr> 
      <tr> 
       <td>-s,–strict</td> 
       <td>以严格模式运行，在CSV解析错误时抛出异常</td> 
      </tr> 
      <tr> 
       <td>-t,–table </td> 
       <td>加载数据的表的名称。默认情况下，表的名称取自CSV文件的名称，此参数区分大小写。</td> 
      </tr> 
     </tbody> 
    </table>
    
    **示例：**
    
        [root@cdh01 ~]# cd /opt/cloudera/parcels/APACHE_PHOENIX/bin
        [root@cdh01 bin]# ./phoenix-psql.py -t MY_SCHEMA.PERSONAS -d '\t' localhost /root/part-m-00017.csv 
        SLF4J: Class path contains multiple SLF4J bindings.
        SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/APACHE_PHOENIX-4.14.0-cdh5.12.2.p0.3/lib/phoenix/phoenix-4.14.0-cdh5.12.2-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
        SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.12.1-1.cdh5.12.1.p0.3/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
        SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
        SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
        19/05/28 14:00:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
        csv columns from database.
        CSV Upsert complete. 1 rows upserted
        Time: 0.351 sec(s)
2.  **通过MapReduce加载**
    
    **用法：**
    
        HADOOP_CLASSPATH=$(hbase mapredcp):/path/to/hbase/conf \
        hadoop jar phoenix-<version>-client.jar \
        org.apache.phoenix.mapreduce.CsvBulkLoadTool \
        --table EXAMPLE \
        --input /data/example.csv
    
    **参数列表:**
    
    <table> 
     <thead> 
      <tr> 
       <th>参数</th> 
       <th>描述</th> 
      </tr> 
     </thead> 
     <tbody> 
      <tr> 
       <td>-a,–array-delimiter </td> 
       <td>数组元素分隔符 (可选)</td> 
      </tr> 
      <tr> 
       <td>-b,–binaryEncoding </td> 
       <td>二进制编码格式</td> 
      </tr> 
      <tr> 
       <td>-c,–import-columns </td> 
       <td>要导入的列列表,以逗号分隔</td> 
      </tr> 
      <tr> 
       <td>-d,–delimiter </td> 
       <td>输入分隔符，默认为逗号</td> 
      </tr> 
      <tr> 
       <td>-e,–escape </td> 
       <td>提供自定义转义字符，默认为反斜杠</td> 
      </tr> 
      <tr> 
       <td>-g,–ignore-errors</td> 
       <td>忽略输入错误</td> 
      </tr> 
      <tr> 
       <td>-h,–help</td> 
       <td>打印帮助说明</td> 
      </tr> 
      <tr> 
       <td>-i,–input </td> 
       <td>输入CSV路径,为hdfs路径 (必填)</td> 
      </tr> 
      <tr> 
       <td>-it,–index-table </td> 
       <td>需要加载的索引表</td> 
      </tr> 
      <tr> 
       <td>-o,–output </td> 
       <td>临时HFiles的输出路径 (可选)</td> 
      </tr> 
      <tr> 
       <td>-q,–quote </td> 
       <td>自定义短语分隔符，默认为双引号</td> 
      </tr> 
      <tr> 
       <td>-s,–schema </td> 
       <td>schema 名称 (可选)</td> 
      </tr> 
      <tr> 
       <td>-t,–table </td> 
       <td>Phoenix表名 (必填)</td> 
      </tr> 
      <tr> 
       <td>-z,–zookeeper </td> 
       <td>Zookeeper连接地址 (可选)</td> 
      </tr> 
     </tbody> 
    </table>
    
    **示例：**
    
        HADOOP_CLASSPATH=/opt/cloudera/parcels/CDH/lib/hbase/hbase-protocol.jar:/opt/cloudera/parcels/CDH/lib/hbase/conf \
        hadoop jar /opt/cloudera/parcels/APACHE_PHOENIX/lib/phoenix/phoenix-4.14.0-cdh5.12.2-client.jar \
        org.apache.phoenix.mapreduce.CsvBulkLoadTool \
        --table MY_SCHEMA.PERSONAS \
        --delimiter '\t' \
        --input /tmp/hbase/test/part-m-00017
    
    **注意：**  
    phoenix-4.14.0-cdh5.12.2-client.jar需添加集群的`hbase-site.xml`配置文件。  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70]  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 1]  
    参考 ：[https://phoenix.apache.org/bulk\_dataload.html][https_phoenix.apache.org_bulk_dataload.html]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70]: /images/20220126/37b4dcdc26964ff6b2469e7625a757fb.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzIxNTI1MA_size_16_color_FFFFFF_t_70 1]: /images/20220126/fcbf489c4a5e4994b8b3309d95715712.png
[https_phoenix.apache.org_bulk_dataload.html]: https://phoenix.apache.org/bulk_dataload.html