Spark和Alluxio测试

Dear 丶 2021-09-28 23:52 437阅读 0赞

### 使用anisble配置好的脚本安装环境： ###

zookeeper

hadoop

spark

alluxio

### 先修改一下配置文件： ###

修改alluxio的配置文件/opt/alluxio/roles/install-alluxio/templates/alluxio-site.properties.j2：

alluxio.zookeeper.enabled=false
    
    #true
    
    #alluxio.zookeeper.address={
       { groups['zookeeper'][0] }}:2181,{
       { groups['zookeeper'][1] }}:2181,{
       { groups['zookeeper'][2] }}:2181

修改spark的配置文件vars.yml:

增加alluxio配置

#alluxio
    alluxio_path: "/opt/alluxio-1.8.1"
    spark_driver_path: "{
       { alluxio_path }}/client/alluxio-1.8.1-client.jar"
    spark_executor_path: "{
       { alluxio_path }}/client/alluxio-1.8.1-client.jar"

spark\_properties部分增加：

- {
          "name":"spark.driver.extraClassPath",
          "value":"{
       { spark_driver_path }}"
      }
      - {
          "name":"spark.executor.extraClassPath",
          "value":"{
       { spark_executor_path }}"
      }

### 浏览器访问 ###

[http://192.168.6.12:19999/home][http_192.168.6.12_19999_home]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70][]

[http://192.168.6.12:18080/][http_192.168.6.12_18080]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 1][]

### 检验Alluxio环境安装： ###

./bin/alluxio validateEnv all
    或
    ./bin/alluxio runTests

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 2][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 3][]

### 检查Spark是否设置正确： ###

integration/checker/bin/alluxio-checker.sh spark spark://192.168.6.12:17077

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 4][]

### 测试Alluxio作为输入输出： ###

1、访问Alluxio中的数据：

上传文件：

cd /opt/alluxio-1.8.1
    ./bin/alluxio fs copyFromLocal LICENSE /LICENSE
    ./bin/alluxio fs ls /
    ./bin/alluxio fs cat /LICENSE

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 5][]

spark-shell测试读写alluxio数据：

val s = sc.textFile("alluxio://192.168.6.12:19998/LICENSE")
    val double = s.map(line => line + line)
    double.saveAsTextFile("alluxio://192.168.6.12:19998/Output")

![20190612091257334.png][]

spark-shell测试写alluxio数据：

val data = Array(1, 2, 3, 4, 5)
    val distData = sc.parallelize(data)
    distData.saveAsTextFile("alluxio://192.168.6.12:19998/distData")

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 6][]

2、访问底层存储数据：

hdfs dfs -put -f LICENSE hdfs://192.168.6.12:9000/alluxio/backend/Input_HDFS
    hdfs dfs -ls /alluxio/backend
    ./bin/alluxio fs ls /

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 7][]

val s = sc.textFile("alluxio://192.168.6.12:19998/Input_HDFS")
     val double = s.map(line => line + line)
     double.saveAsTextFile("alluxio://192.168.6.12:19998/Output_HDFS")

![20190612091416625.png][]

查看结果：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 8][]

### 高级部分：配置zookeeper ###

修改alluxio的配置文件/opt/alluxio/roles/install-alluxio/templates/alluxio-site.properties.j2：

alluxio.zookeeper.enabled=true
    alluxio.zookeeper.address={
       { groups['zookeeper'][0] }}:2181,{
       { groups['zookeeper'][1] }}:2181,{
       { groups['zookeeper'][2] }}:2181

修改spark的配置文件vars.yml:

alluxio_zookeeper_address: "192.168.6.12:2181,192.168.6.13:2181,192.168.6.14:2181"
    spark_properties部分增加：
    - {
          "name":"spark.driver.extraJavaOptions",
          "value":"-Dalluxio.zookeeper.address=192.168.6.12:2181,192.168.6.13:2181,192.168.6.14:2181 -Dalluxio.zookeeper.enabled=true"
      }
      - {
          "name":"spark.executor.extraJavaOptions",
          "value":"-Dalluxio.zookeeper.address=192.168.6.12:2181,192.168.6.13:2181,192.168.6.14:2181 -Dalluxio.zookeeper.enabled=true"
      }

测试1：

＃加上zookeeper配置信息

val s = sc.textFile("alluxio://192.168.6.12:2181;192.168.6.13:2181;192.168.6.14:2181/Input_HDFS")
     val double = s.map(line => line + line)
     double.saveAsTextFile("alluxio://192.168.6.12:2181;192.168.6.13:2181;192.168.6.14:2181/Output_HDFS")

![20190612091556705.png][]

测试2:

＃省略zookeeper配置信息

val s = sc.textFile("alluxio:///Input_HDFS")
     val double = s.map(line => line + line)
     double.saveAsTextFile("alluxio:///Output_HDFS1")

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 9][]

[http_192.168.6.12_19999_home]: http://192.168.6.12:19999/home
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70]: /images/20210923/a2651ed53eb24863869db8db4cf8686b.png
[http_192.168.6.12_18080]: http://sdw-12:18080/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 1]: /images/20210923/a01526b0164c47d2bc82bfcaa47b2835.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 2]: /images/20210923/5442a4cd129845e788c7dbf0091ce691.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 3]: /images/20210923/09a9d2de3478405998b9316b0e931af8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 4]: /images/20210923/2b9ab7592b7e4793b27fa4a9145721df.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 5]: /images/20210923/a9ef072043d2463d8aaa6712d6e4aefe.png
[20190612091257334.png]: /images/20210923/55a9fd4f15294532ba88c384eafcacda.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 6]: /images/20210923/f6c705362acb40ea9672868b6d92cd6a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 7]: /images/20210923/2a1d9c0d921344bda2458ab71bca7324.png
[20190612091416625.png]: /images/20210923/53256c7abded47908a6aea3252697b46.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 8]: /images/20210923/473f6c6c9eb54ba7815ec0751b7d7996.png
[20190612091556705.png]: /images/20210923/23ad17cbb6a345a98190c6d4eb6e79d8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3p3YWh1dA_size_16_color_FFFFFF_t_70 9]: /images/20210923/9d9948efb7194ebca23cbd7c08724722.png