ES reindex 实战

浅浅的花香味﹌ 2024-03-17 09:18 47阅读 0赞

### 机器配置 ###

Data nodes：i3.4xlarge.search * 3     16C 122G
    master nodes：c5.2xlarge.search * 3    8C 16G

### 案例1-kafka ###

kafka consumer tps = 18k qps  
bulk request batch size = 100 （程序单线程等待100个消息到达后提交，因此系统的吞吐量与batch无关）  
数据量（cat命令）：

green  open   admin_operate_log_v1_2021       1   0        432            0    183.9kb        183.9kb
    green  open   admin_operate_log_v1_2022       1   0   38855695            0     10.9gb         10.9gb
    green  open   admin_operate_log_v1_2023       1   0   15996192            0      4.5gb          4.5gb

#### 索引配置 ####

"settings": {
            "index": {
                "refresh_interval": "-1",
                "number_of_shards": "1",
                "number_of_replicas": "0"
            }
        }

#### ES负载 ####

cpu： 写入期间<=49%，写入完成后短暂峰值：68-72%  
内存：73%左右  
![截屏2023-05-11 22.33.15.png][2023-05-11 22.33.15.png]

#### 性能 ####

耗时：55-60分钟左右

### 案例2-reindex ###

数据量

<table> 
 <thead> 
  <tr> 
   <th>health</th> 
   <th>status</th> 
   <th>index</th> 
   <th>uuid</th> 
   <th>pri</th> 
   <th>rep</th> 
   <th>docs.count</th> 
   <th>docs.deleted</th> 
   <th>store.size</th> 
   <th>pri.store.size</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202211</td> 
   <td>F7qtS9A3QxCFlIiZ8vCB0Q</td> 
   <td>2</td> 
   <td>1</td> 
   <td>6471447</td> 
   <td>0</td> 
   <td>5.3gb</td> 
   <td>2.6gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202210</td> 
   <td>9MWtw90pTLes7ZzsDo5zxw</td> 
   <td>2</td> 
   <td>1</td> 
   <td>5479698</td> 
   <td>0</td> 
   <td>4.3gb</td> 
   <td>2.1gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>admin_operate_log_v1_2021</td> 
   <td>cxmLz54CR2C2eQkWcftGdw</td> 
   <td>1</td> 
   <td>0</td> 
   <td>432</td> 
   <td>0</td> 
   <td>183.9kb</td> 
   <td>183.9kb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>admin_operate_log_v1_2022</td> 
   <td>mD9B3fY3T5moY2iMoSJ6rA</td> 
   <td>1</td> 
   <td>0</td> 
   <td>38855695</td> 
   <td>0</td> 
   <td>10.9gb</td> 
   <td>10.9gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>admin_operate_log_v1_2023</td> 
   <td>Q8umgdzcR1idQncdPnUCbA</td> 
   <td>1</td> 
   <td>0</td> 
   <td>15996192</td> 
   <td>0</td> 
   <td>4.5gb</td> 
   <td>4.5gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202212</td> 
   <td>9Bo4HkB1T_G8oiOiJVcDow</td> 
   <td>2</td> 
   <td>1</td> 
   <td>5714707</td> 
   <td>0</td> 
   <td>4.6gb</td> 
   <td>2.3gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202208</td> 
   <td>8jMbqQPIQT-GI67lo72Xbg</td> 
   <td>2</td> 
   <td>1</td> 
   <td>5070471</td> 
   <td>0</td> 
   <td>5gb</td> 
   <td>2.5gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202207</td> 
   <td>ppYBaM3NQkaPBkZowXqbdA</td> 
   <td>2</td> 
   <td>1</td> 
   <td>5253290</td> 
   <td>0</td> 
   <td>5.1gb</td> 
   <td>2.5gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202206</td> 
   <td>A12fRvkhTOCcI12QPCzbxw</td> 
   <td>2</td> 
   <td>1</td> 
   <td>5499146</td> 
   <td>0</td> 
   <td>5.6gb</td> 
   <td>2.8gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202305</td> 
   <td>ltM-SbzaQkaEmqVmV84iEQ</td> 
   <td>2</td> 
   <td>1</td> 
   <td>1659596</td> 
   <td>50</td> 
   <td>1.1gb</td> 
   <td>583mb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202304</td> 
   <td>0W9kvl-jTlWizYltRq4yGQ</td> 
   <td>2</td> 
   <td>1</td> 
   <td>7589405</td> 
   <td>44</td> 
   <td>4.7gb</td> 
   <td>2.3gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202205</td> 
   <td>KzLscm-gRSq-v7aps96M4Q</td> 
   <td>2</td> 
   <td>1</td> 
   <td>7343020</td> 
   <td>0</td> 
   <td>7.5gb</td> 
   <td>3.7gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202209</td> 
   <td>DDMTU6mVTn-uF1NZPJ-x4Q</td> 
   <td>2</td> 
   <td>1</td> 
   <td>4699022</td> 
   <td>0</td> 
   <td>4.3gb</td> 
   <td>2.1gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202303</td> 
   <td>f13sJ9wyRxagWYOCK0t07w</td> 
   <td>2</td> 
   <td>1</td> 
   <td>7698296</td> 
   <td>0</td> 
   <td>5gb</td> 
   <td>2.5gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>token_sale_v1</td> 
   <td>K7FFK24FSWi1oKlXrRuBug</td> 
   <td>3</td> 
   <td>1</td> 
   <td>1157</td> 
   <td>22</td> 
   <td>550.2kb</td> 
   <td>275.1kb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202204</td> 
   <td>grbxxC5iQie3ADnnyE5Erw</td> 
   <td>2</td> 
   <td>1</td> 
   <td>8288019</td> 
   <td>0</td> 
   <td>8.3gb</td> 
   <td>4.1gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202302</td> 
   <td>oDOUznF5QKGgTmetRv2uRA</td> 
   <td>2</td> 
   <td>1</td> 
   <td>6797596</td> 
   <td>0</td> 
   <td>4.4gb</td> 
   <td>2.2gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202203</td> 
   <td>aJLU9fhzT3uCNf2wVdb3Gw</td> 
   <td>2</td> 
   <td>1</td> 
   <td>9291234</td> 
   <td>0</td> 
   <td>11.3gb</td> 
   <td>5.6gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>.kibana_1</td> 
   <td>ErpcRNskT_-GLNVpeQd-1A</td> 
   <td>1</td> 
   <td>1</td> 
   <td>1</td> 
   <td>0</td> 
   <td>10.1kb</td> 
   <td>5kb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202301</td> 
   <td>BqpOEoDTTO2ue-uDfjUoCA</td> 
   <td>2</td> 
   <td>1</td> 
   <td>6103653</td> 
   <td>152925</td> 
   <td>4.2gb</td> 
   <td>2.1gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202202</td> 
   <td>d9VaDToWSs2qbdyxJyDvdg</td> 
   <td>2</td> 
   <td>1</td> 
   <td>9415513</td> 
   <td>0</td> 
   <td>12gb</td> 
   <td>6gb</td> 
  </tr> 
  <tr> 
   <td>green</td> 
   <td>open</td> 
   <td>user_operation_log_v1_202201</td> 
   <td>FahUyKbeTYK_zYYc_UhofQ</td> 
   <td>2</td> 
   <td>1</td> 
   <td>11594662</td> 
   <td>0</td> 
   <td>15.4gb</td> 
   <td>7.7gb</td> 
  </tr> 
 </tbody> 
</table>

#### 索引配置 ####

es_host='https://my_es_host'
    indices=(user_operation_log_v1_202301 user_operation_log_v1_202302)
    
    curl --location --request POST "${es_host}/_template/user_operation_log_template_v2" \
    --header 'Content-Type: application/json' \
    --data-raw '{
      "order": 0,
      "index_patterns": [
        "user_operation_log_v2*"
      ],
      "mappings": {
        "dynamic": "false",
        "properties": {
          "user_id": {
            "type": "long"
          },
          "operation": {
            "type": "keyword",
            "normalizer": "caseSensitive"
          },
          "real_ip": {
            "type": "keyword"
          },
          "full_ip": {
            "type": "keyword"
          },
          "client_type": {
            "type": "keyword",
            "normalizer": "caseSensitive"
          },
          "version_code": {
            "type": "keyword",
            "index": false
          },
          "apikey": {
            "type": "keyword",
            "index": false
          },
          "user_agent": {
            "type": "keyword",
            "index": false
          },
          "request_time": {
            "type": "long"
          },
          "response_time": {
            "type": "long",
            "index": false
          },
          "request": {
            "type": "text"
          },
          "response": {
            "type": "text"
          },
          "response_status": {
            "type": "keyword",
            "normalizer": "caseSensitive"
          }
        }
      },
      "settings": {
        "index": {
          "refresh_interval" : "-1",
          "number_of_shards": "2",
          "number_of_replicas": "0"
        },
        "analysis": {
          "normalizer": {
            "caseSensitive": {
              "filter": "lowercase",
              "type": "custom"
            }
          }
        }
      },
      "aliases": {
        "user_operation_log_v2": {}
      }
    }'
    
    for((i=0;i<${#indices[*]};i++));do
      target_index=${indices[i]/v1/v2};
      message="source = ${indices[i]}, target = ${target_index}";
      echo "${message}"
      start=$(date +"%s")
      curl -HContent-Type:application/json -XPOST "${es_host}/_reindex?slices=2&requests_per_second=-1&wait_for_completion=false&pretty" -d'{
        "source": {
          "index": "'${indices[i]}'"
          ,"size": 5000
        },
        "dest": {
          "index": "'${target_index}'"
        }
      }'
      end=$(date +"%s")
      echo "${message} cost `expr $end - $start` seconds"
    done

#### ES负载 ####

![截屏2023-05-11 14.26.56.png][2023-05-11 14.26.56.png]

#### 性能 ####

耗时：10分钟左右

##### 第一次提交2个索引reindex #####

{
        
      "completed": true,
      "task": {
        
        "node": "b8mpzb7ARLau_iJVU4fkmg",
        "id": 246335488,
        "type": "transport",
        "action": "indices:data/write/reindex",
        "status": {
        
          "total": 9415513,
          "updated": 0,
          "created": 9415513,
          "deleted": 0,
          "batches": 1884,
          "version_conflicts": 0,
          "noops": 0,
          "retries": {
        
            "bulk": 0,
            "search": 0
          },
          "throttled_millis": 0,
          "requests_per_second": -1.0,
          "throttled_until_millis": 0,
          "slices": [
            {
        
              "slice_id": 0,
              "total": 4708452,
              "updated": 0,
              "created": 4708452,
              "deleted": 0,
              "batches": 942,
              "version_conflicts": 0,
              "noops": 0,
              "retries": {
        
                "bulk": 0,
                "search": 0
              },
              "throttled_millis": 0,
              "requests_per_second": -1.0,
              "throttled_until_millis": 0
            },
            {
        
              "slice_id": 1,
              "total": 4707061,
              "updated": 0,
              "created": 4707061,
              "deleted": 0,
              "batches": 942,
              "version_conflicts": 0,
              "noops": 0,
              "retries": {
        
                "bulk": 0,
                "search": 0
              },
              "throttled_millis": 0,
              "requests_per_second": -1.0,
              "throttled_until_millis": 0
            }
          ]
        },
        "description": "reindex from [user_operation_log_v1_202202] to [user_operation_log_v2_202202][_doc]",
        "start_time_in_millis": 1683784030867,
        "running_time_in_nanos": 447756932234,
        "cancellable": true,
        "headers": {
        
          
        }
      },
      "response": {
        
        "took": 447740,
        "timed_out": false,
        "total": 9415513,
        "updated": 0,
        "created": 9415513,
        "deleted": 0,
        "batches": 1884,
        "version_conflicts": 0,
        "noops": 0,
        "retries": {
        
          "bulk": 0,
          "search": 0
        },
        "throttled": "0s",
        "throttled_millis": 0,
        "requests_per_second": -1.0,
        "throttled_until": "0s",
        "throttled_until_millis": 0,
        "slices": [
          {
        
            "slice_id": 0,
            "total": 4708452,
            "updated": 0,
            "created": 4708452,
            "deleted": 0,
            "batches": 942,
            "version_conflicts": 0,
            "noops": 0,
            "retries": {
        
              "bulk": 0,
              "search": 0
            },
            "throttled": "0s",
            "throttled_millis": 0,
            "requests_per_second": -1.0,
            "throttled_until": "0s",
            "throttled_until_millis": 0
          },
          {
        
            "slice_id": 1,
            "total": 4707061,
            "updated": 0,
            "created": 4707061,
            "deleted": 0,
            "batches": 942,
            "version_conflicts": 0,
            "noops": 0,
            "retries": {
        
              "bulk": 0,
              "search": 0
            },
            "throttled": "0s",
            "throttled_millis": 0,
            "requests_per_second": -1.0,
            "throttled_until": "0s",
            "throttled_until_millis": 0
          }
        ],
        "failures": [
          
        ]
      }
    }
    -
    {
        
      "completed": true,
      "task": {
        
        "node": "BrXZu22XQfi94P1l5ErOwA",
        "id": 168840526,
        "type": "transport",
        "action": "indices:data/write/reindex",
        "status": {
        
          "total": 9291234,
          "updated": 0,
          "created": 9291234,
          "deleted": 0,
          "batches": 1859,
          "version_conflicts": 0,
          "noops": 0,
          "retries": {
        
            "bulk": 0,
            "search": 0
          },
          "throttled_millis": 0,
          "requests_per_second": -1.0,
          "throttled_until_millis": 0,
          "slices": [
            {
        
              "slice_id": 0,
              "total": 4642972,
              "updated": 0,
              "created": 4642972,
              "deleted": 0,
              "batches": 929,
              "version_conflicts": 0,
              "noops": 0,
              "retries": {
        
                "bulk": 0,
                "search": 0
              },
              "throttled_millis": 0,
              "requests_per_second": -1.0,
              "throttled_until_millis": 0
            },
            {
        
              "slice_id": 1,
              "total": 4648262,
              "updated": 0,
              "created": 4648262,
              "deleted": 0,
              "batches": 930,
              "version_conflicts": 0,
              "noops": 0,
              "retries": {
        
                "bulk": 0,
                "search": 0
              },
              "throttled_millis": 0,
              "requests_per_second": -1.0,
              "throttled_until_millis": 0
            }
          ]
        },
        "description": "reindex from [user_operation_log_v1_202203] to [user_operation_log_v2_202203][_doc]",
        "start_time_in_millis": 1683784030916,
        "running_time_in_nanos": 437625719457,
        "cancellable": true,
        "headers": {
        
          
        }
      },
      "response": {
        
        "took": 437607,
        "timed_out": false,
        "total": 9291234,
        "updated": 0,
        "created": 9291234,
        "deleted": 0,
        "batches": 1859,
        "version_conflicts": 0,
        "noops": 0,
        "retries": {
        
          "bulk": 0,
          "search": 0
        },
        "throttled": "0s",
        "throttled_millis": 0,
        "requests_per_second": -1.0,
        "throttled_until": "0s",
        "throttled_until_millis": 0,
        "slices": [
          {
        
            "slice_id": 0,
            "total": 4642972,
            "updated": 0,
            "created": 4642972,
            "deleted": 0,
            "batches": 929,
            "version_conflicts": 0,
            "noops": 0,
            "retries": {
        
              "bulk": 0,
              "search": 0
            },
            "throttled": "0s",
            "throttled_millis": 0,
            "requests_per_second": -1.0,
            "throttled_until": "0s",
            "throttled_until_millis": 0
          },
          {
        
            "slice_id": 1,
            "total": 4648262,
            "updated": 0,
            "created": 4648262,
            "deleted": 0,
            "batches": 930,
            "version_conflicts": 0,
            "noops": 0,
            "retries": {
        
              "bulk": 0,
              "search": 0
            },
            "throttled": "0s",
            "throttled_millis": 0,
            "requests_per_second": -1.0,
            "throttled_until": "0s",
            "throttled_until_millis": 0
          }
        ],
        "failures": [
          
        ]
      }
    }

##### 第二次提交14个索引reindex #####

202204-202305  
日志过多省略，与第一次提交差别不大

### 总结 ###

#### 案例1 ####

kafka：tps=18k  
文档量：54851887  
文档大小：15.4GB  
ES节点：单个  
耗时：55-60分钟  
负载：cpu=50%左右，内存=70%左右

#### 案例2 ####

根据第二次提交的reindex分析  
文档量：113968775  
文档大小：53.6GB  
ES节点：3个  
耗时：10分钟  
负载：cpu=80%左右，内存=80%左右

正如开头描述的，系统单线程工作瓶颈明显，并发与吞吐量低，优点是限制了单个业务索引（多个物理索引按月分片）资源占用，避免拖垮系统

#### reindex风险 ####

14个索引reindex的异步任务启动时间如下，可以看到几乎是同时启动的，因此批量操作是要把控并行的数量，slices参数以及size参数的配置，都要事先把控，避免将服务器跑垮

1683785201349 2023-05-11 14:06:41
    1683785201371 2023-05-11 14:06:41
    1683785201395
    1683785201418
    1683785201440
    1683785201463
    1683785201485
    1683785201508
    1683785201531
    1683785201551
    1683785201572
    1683785201598
    1683785201623
    1683785201645

[2023-05-11 22.33.15.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/771db65bf6f9499d895a69a5e9cf1ff4.png
[2023-05-11 14.26.56.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/17/a2575f68b95f448d97f90d7b21c1f33a.png