【Kafka】记录一次K8环境下不同存储的Kafka benchmark测试

清疚 2023-10-05 16:41 4阅读 0赞

**本次主要针对k8s环境下不同的存储openebs和ceph进行benchmark测试**

## 集群环境 ##

两个集群环境出了存储不同，broker、topic信息均相同

**kafka Info：**5个broker

**Topic Info：**副本数3；分区30

**kafka server version：**2.6.0

## 生产者测试 ##

消息数：100w；每条消息大小：10byte

<table> 
 <thead> 
 <tr> 
 <th> 存储类型 </th> 
 <th> 每秒发送消息数 &nbsp;(records/sec) </th> 
 <th> 发送速率 (MB/sec) </th> 
 <th> 平均延迟 (ms) </th> 
 <th> 最大延迟 (ms) </th> 
 <th> 50% (ms) </th> 
 <th> 95% (ms) </th> 
 <th> 99% (ms) </th> 
 <th> 99.9% (ms) </th> 
 </tr> 
 </thead> 
 <tbody> 
 <tr> 
 <td>openebs</td> 
 <td>956022</td> 
 <td>9.12</td> 
 <td>4.26</td> 
 <td>161.00</td> 
 <td>3</td> 
 <td>9</td> 
 <td>23</td> 
 <td>31</td> 
 </tr> 
 <tr> 
 <td>ceph</td> 
 <td>862812</td> 
 <td>8.23</td> 
 <td>4.83</td> 
 <td>155.00</td> 
 <td>4</td> 
 <td>10</td> 
 <td>35</td> 
 <td>67</td> 
 </tr> 
 </tbody> 
</table>

消息数：100w；每条消息大小：100byte

<table> 
 <thead> 
 <tr> 
 <th> 存储类型 </th> 
 <th> 每秒发送消息数 &nbsp;(records/sec) </th> 
 <th> 发送速率 (MB/sec) </th> 
 <th> 平均延迟 (ms) </th> 
 <th> 最大延迟 (ms) </th> 
 <th> 50% (ms) </th> 
 <th> 95% (ms) </th> 
 <th> 99% (ms) </th> 
 <th> 99.9% (ms) </th> 
 </tr> 
 </thead> 
 <tbody> 
 <tr> 
 <td>openebs</td> 
 <td>822368</td> 
 <td>78.43</td> 
 <td>3.31</td> 
 <td>171.00</td> 
 <td>3</td> 
 <td>6</td> 
 <td>19</td> 
 <td>37</td> 
 </tr> 
 <tr> 
 <td>ceph</td> 
 <td>915750</td> 
 <td>87.33</td> 
 <td>2.48</td> 
 <td>164.00</td> 
 <td>2</td> 
 <td>4</td> 
 <td>9</td> 
 <td>22</td> 
 </tr> 
 </tbody> 
</table>

消息数：100w；每条消息大小：1kb

<table> 
 <thead> 
 <tr> 
 <th> 存储类型 </th> 
 <th> 每秒发送消息数 &nbsp;(records/sec) </th> 
 <th> 发送速率 (MB/sec) </th> 
 <th> 平均延迟 (ms) </th> 
 <th> 最大延迟 (ms) </th> 
 <th> 50% (ms) </th> 
 <th> 95% (ms) </th> 
 <th> 99% (ms) </th> 
 <th> 99.9% (ms) </th> 
 </tr> 
 </thead> 
 <tbody> 
 <tr> 
 <td>openebs</td> 
 <td>366703</td> 
 <td>358.11</td> 
 <td>74.59</td> 
 <td>161.00</td> 
 <td>71</td> 
 <td>115</td> 
 <td>126</td> 
 <td>142</td> 
 </tr> 
 <tr> 
 <td>ceph</td> 
 <td>250000</td> 
 <td>244.14</td> 
 <td>98.46</td> 
 <td>1481.00</td> 
 <td>71</td> 
 <td>115</td> 
 <td>1440</td> 
 <td>1476</td> 
 </tr> 
 </tbody> 
</table>

消息数：100w；每条消息大小：2kb

<table> 
 <thead> 
 <tr> 
 <th> 存储类型 </th> 
 <th> 每秒发送消息数 &nbsp;(records/sec) </th> 
 <th> 发送速率 (MB/sec) </th> 
 <th> 平均延迟 (ms) </th> 
 <th> 最大延迟 (ms) </th> 
 <th> 50% (ms) </th> 
 <th> 95% (ms) </th> 
 <th> 99% (ms) </th> 
 <th> 99.9% (ms) </th> 
 </tr> 
 </thead> 
 <tbody> 
 <tr> 
 <td>openebs</td> 
 <td>225479</td> 
 <td>440.39</td> 
 <td>59.57</td> 
 <td>170.00</td> 
 <td>59</td> 
 <td>83</td> 
 <td>97</td> 
 <td>112</td> 
 </tr> 
 <tr> 
 <td>ceph</td> 
 <td>140904</td> 
 <td>275.20</td> 
 <td>96.59</td> 
 <td>1274.00</td> 
 <td>12</td> 
 <td>439</td> 
 <td>1111</td> 
 <td>1269</td> 
 </tr> 
 </tbody> 
</table>

消息数：100w；每条消息大小：4kb

<table> 
 <thead> 
 <tr> 
 <th> 存储类型 </th> 
 <th> 每秒发送消息数 &nbsp;(records/sec) </th> 
 <th> 发送速率 (MB/sec) </th> 
 <th> 平均延迟 (ms) </th> 
 <th> 最大延迟 (ms) </th> 
 <th> 50% (ms) </th> 
 <th> 95% (ms) </th> 
 <th> 99% (ms) </th> 
 <th> 99.9% (ms) </th> 
 </tr> 
 </thead> 
 <tbody> 
 <tr> 
 <td>openebs</td> 
 <td>125062</td> 
 <td>488.53</td> 
 <td>47.45</td> 
 <td>181.00</td> 
 <td>46</td> 
 <td>73</td> 
 <td>99</td> 
 <td>115</td> 
 </tr> 
 <tr> 
 <td>ceph</td> 
 <td>92276</td> 
 <td>360.45</td> 
 <td>64.75</td> 
 <td>1411.00</td> 
 <td>6</td> 
 <td>226</td> 
 <td>1023</td> 
 <td>1397</td> 
 </tr> 
 </tbody> 
</table>

消息数：100w；每条消息大小：8kb

<table> 
 <thead> 
 <tr> 
 <th> 存储类型 </th> 
 <th> 每秒发送消息数 &nbsp;(records/sec) </th> 
 <th> 发送速率 (MB/sec) </th> 
 <th> 平均延迟 (ms) </th> 
 <th> 最大延迟 (ms) </th> 
 <th> 50% (ms) </th> 
 <th> 95% (ms) </th> 
 <th> 99% (ms) </th> 
 <th> 99.9% (ms) </th> 
 </tr> 
 </thead> 
 <tbody> 
 <tr> 
 <td>openebs</td> 
 <td>62309</td> 
 <td>486.79</td> 
 <td>32.40</td> 
 <td>218.00</td> 
 <td>23</td> 
 <td>131</td> 
 <td>154</td> 
 <td>206</td> 
 </tr> 
 <tr> 
 <td>ceph</td> 
 <td>32344</td> 
 <td>252.69</td> 
 <td>62.84</td> 
 <td>4519.00</td> 
 <td>6</td> 
 <td>138</td> 
 <td>1702</td> 
 <td>4512</td> 
 </tr> 
 </tbody> 
</table>

### 消费者测试 ###

**消费者参数均为：线程数：10；每次拉取消息大小：1mb**

每条消息大小：10byte

<table> 
 <tbody> 
 <tr> 
 <th>存储类型</th> 
 <th> 消息总大小(MB) data.consumed.in.MB </th> 
 <th> 每秒传输数据大小(MB) MB.sec </th> 
 <th> 每秒消费消息数 nMsg.sec </th> 
 <th> 均衡耗时(ms) rebalance.time.ms </th> 
 <th> 拉取耗时(ms) fetch.time.ms </th> 
 <th> 每秒拉取数据(MB) fetch.MB.sec </th> 
 <th> 每秒拉取消息数 fetch.nMsg.sec </th> 
 </tr> 
 <tr> 
 <td>openebs</td> 
 <td>9.5367</td> 
 <td>2.6308</td> 
 <td>275862.0690</td> 
 <td>3012</td> 
 <td>613</td> 
 <td>15.5575</td> 
 <td>1631321.3703</td> 
 </tr> 
 <tr> 
 <td>ceph</td> 
 <td>9.5367</td> 
 <td>2.6200</td> 
 <td>274725.2747</td> 
 <td>3013</td> 
 <td>627</td> 
 <td>15.2101</td> 
 <td>1594896.3317</td> 
 </tr> 
 </tbody> 
</table>

每条消息大小：100byte

<table> 
 <tbody> 
 <tr> 
 <th>存储类型</th> 
 <th> 消息总大小(MB) data.consumed.in.MB </th> 
 <th> 每秒传输数据大小(MB) MB.sec </th> 
 <th> 每秒消费消息数 nMsg.sec </th> 
 <th> 均衡耗时(ms) rebalance.time.ms </th> 
 <th> 拉取耗时(ms) fetch.time.ms </th> 
 <th> 每秒拉取数据(MB) fetch.MB.sec </th> 
 <th> 每秒拉取消息数 fetch.nMsg.sec </th> 
 </tr> 
 <tr> 
 <td>openebs</td> 
 <td>95.3674</td> 
 <td>25.4993</td> 
 <td>267379.6791</td> 
 <td>3010</td> 
 <td>730</td> 
 <td>130.6403</td> 
 <td>1369863.0137</td> 
 </tr> 
 <tr> 
 <td>ceph</td> 
 <td>95.3674</td> 
 <td>25.5130</td> 
 <td>267522.7394</td> 
 <td>3012</td> 
 <td>726</td> 
 <td>131.3601</td> 
 <td>1377410.4683</td> 
 </tr> 
 </tbody> 
</table>

每条消息大小：1kb

<table> 
 <tbody> 
 <tr> 
 <th>存储类型</th> 
 <th> 消息总大小(MB) data.consumed.in.MB </th> 
 <th> 每秒传输数据大小(MB) MB.sec </th> 
 <th> 每秒消费消息数 nMsg.sec </th> 
 <th> 均衡耗时(ms) rebalance.time.ms </th> 
 <th> 拉取耗时(ms) fetch.time.ms </th> 
 <th> 每秒拉取数据(MB) fetch.MB.sec </th> 
 <th> 每秒拉取消息数 fetch.nMsg.sec </th> 
 </tr> 
 <tr> 
 <td>openebs</td> 
 <td>976.5625</td> 
 <td>197.2057</td> 
 <td>201938.6107</td> 
 <td>3010</td> 
 <td>1942</td> 
 <td>502.8643</td> 
 <td>514933.0587</td> 
 </tr> 
 <tr> 
 <td>ceph</td> 
 <td>976.5625</td> 
 <td>185.8703</td> 
 <td>190331.1762</td> 
 <td>3010</td> 
 <td>2244</td> 
 <td>435.1883</td> 
 <td>445632.7986</td> 
 </tr> 
 </tbody> 
</table>

每条消息大小：2kb

<table> 
 <tbody> 
 <tr> 
 <th>存储类型</th> 
 <th> 消息总大小(MB) data.consumed.in.MB </th> 
 <th> 每秒传输数据大小(MB) MB.sec </th> 
 <th> 每秒消费消息数 nMsg.sec </th> 
 <th> 均衡耗时(ms) rebalance.time.ms </th> 
 <th> 拉取耗时(ms) fetch.time.ms </th> 
 <th> 每秒拉取数据(MB) fetch.MB.sec </th> 
 <th> 每秒拉取消息数 fetch.nMsg.sec </th> 
 </tr> 
 <tr> 
 <td>openebs</td> 
 <td>1953.1250</td> 
 <td>326.8281</td> 
 <td>167336.0107</td> 
 <td>3009</td> 
 <td>2967</td> 
 <td>658.2828</td> 
 <td>337040.7819</td> 
 </tr> 
 <tr> 
 <td>ceph</td> 
 <td>1953.1250</td> 
 <td>315.8863</td> 
 <td>161733.7862</td> 
 <td>3010</td> 
 <td>3173</td> 
 <td>615.5452</td> 
 <td>315159.1554</td> 
 </tr> 
 </tbody> 
</table>

每条消息大小：4kb

<table> 
 <tbody> 
 <tr> 
 <th>存储类型</th> 
 <th> 消息总大小(MB) data.consumed.in.MB </th> 
 <th> 每秒传输数据大小(MB) MB.sec </th> 
 <th> 每秒消费消息数 nMsg.sec </th> 
 <th> 均衡耗时(ms) rebalance.time.ms </th> 
 <th> 拉取耗时(ms) fetch.time.ms </th> 
 <th> 每秒拉取数据(MB) fetch.MB.sec </th> 
 <th> 每秒拉取消息数 fetch.nMsg.sec </th> 
 </tr> 
 <tr> 
 <td>openebs</td> 
 <td>3906.2500</td> 
 <td>467.0313</td> 
 <td>119560.0191</td> 
 <td>3010</td> 
 <td>5354</td> 
 <td>729.5947</td> 
 <td>186776.2421</td> 
 </tr> 
 <tr> 
 <td>ceph</td> 
 <td>3906.2500</td> 
 <td>469.8965</td> 
 <td>120293.5162</td> 
 <td>3011</td> 
 <td>5302</td> 
 <td>736.7503</td> 
 <td>188608.0724</td> 
 </tr> 
 </tbody> 
</table>

每条消息大小：8kb

<table> 
 <tbody> 
 <tr> 
 <th>存储类型</th> 
 <th> 消息总大小(MB) data.consumed.in.MB </th> 
 <th> 每秒传输数据大小(MB) MB.sec </th> 
 <th> 每秒消费消息数 nMsg.sec </th> 
 <th> 均衡耗时(ms) rebalance.time.ms </th> 
 <th> 拉取耗时(ms) fetch.time.ms </th> 
 <th> 每秒拉取数据(MB) fetch.MB.sec </th> 
 <th> 每秒拉取消息数 fetch.nMsg.sec </th> 
 </tr> 
 <tr> 
 <td>openebs</td> 
 <td>7812.5000</td> 
 <td>587.2735</td> 
 <td>75171.0141</td> 
 <td>3011</td> 
 <td>10292</td> 
 <td>759.0847</td> 
 <td>97162.8449</td> 
 </tr> 
 <tr> 
 <td>ceph</td> 
 <td>7812.5000</td> 
 <td>517.3498</td> 
 <td>66220.7801</td> 
 <td>3011</td> 
 <td>12090</td> 
 <td>646.1952</td> 
 <td>82712.9859</td> 
 </tr> 
 </tbody> 
</table>