剖析生产系统的I/O模式

╰半橙微兮° 2021-12-15 12:21 110阅读 0赞

# [剖析生产系统的I/O模式][I_O] #

[2019/02/13][I_O] [vmunix][]

了解I/O的特点对于优化系统性能非常重要，I/O是顺序的还是随机的，是读操作还是写操作，读写的比例是多少，I/O数据块的大小，这些都是影响性能的关键因素。很多存储设备都基于特定的I/O模式做过调校，通用的测试工具跑分都相当漂亮，然而一到实际环境区别就来了，同样的应用环境下，不同的设备表现可能天差地别。我就见过不同厂商的设备，档次差不多，测试跑分高的那个在生产环境下的IO响应速度却慢了十倍。所以跑分高的设备真的不一定适合你的应用。

如果能够模拟出应用的I/O模式，那么在问题复现、乃至设备选型等方面都会有很大帮助。在此之前，了解I/O模式是第一步，这并不容易，像iostat之类的工具只能看到平均值，然而应用系统的I/O请求有可能是波浪式的，一秒之内也可以时高时低，I/O延迟可能平均值不高但是波动很大，而且I/O块大小也可以是变化的，尤其现在大数据应用的块大小可能在很大的范围内变化，与过去常见的交易型数据库有所不同，它们的块大小基本是固定的。

要剖析生产系统的I/O模式，好像没有现成的工具。但是我们可以利用blktrace自己做一个，blktrace在内核的block layer记录每一个I/O，提供了分析的素材。它记录的格式如下：![blktrace-event-output.jpeg][]

下面是一个简化版的示例，主要利用了Event “Q”和”C”，分别表示IO开始和IO完成，两者之间的耗时就相当于iostat看到的await，但blktrace可以精确到单个IO：

\#!/bin/bash if \[ $\# -ne 1 \]; then echo "Usage: $0 <block\_device\_name>" exit fi if \[ ! -b $1 \]; then echo "could not find block device $1" exit fi duration=10 echo "running blktrace for $duration seconds to collect data..." timeout $duration blktrace -d $1 >/dev/null 2>&1 DEVNAME=\`basename $1\` echo "parsing blktrace data..." blkparse -i $DEVNAME |sort -g -k8 -k10 -k4 |awk ' BEGIN \{ total\_read=0; total\_write=0; maxwait\_read=0; maxwait\_write=0; \} \{ if ($6=="Q") \{ queue\_ts=$4; block=$8; nblock=$10; rw=$7; \}; if ($6=="C" && $8==block && $10==nblock && $7==rw) \{ await=$4-queue\_ts; if (rw=="R") \{ if (await>maxwait\_read) maxwait\_read=await; total\_read++; read\_count\_block\[nblock\]++; if (await>0.001) read\_count1++; if (await>0.01) read\_count10++; if (await>0.02) read\_count20++; if (await>0.03) read\_count30++; \} if (rw=="W") \{ if (await>maxwait\_write) maxwait\_write=await; total\_write++; write\_count\_block\[nblock\]++; if (await>0.001) write\_count1++; if (await>0.01) write\_count10++; if (await>0.02) write\_count20++; if (await>0.03) write\_count30++; \} \} \} END \{ printf("========\\nsummary:\\n========\\n"); printf("total number of reads: %d\\n", total\_read); printf("total number of writes: %d\\n", total\_write); printf("slowest read : %.6f second\\n", maxwait\_read); printf("slowest write: %.6f second\\n", maxwait\_write); printf("reads\\n> 1ms: %d\\n>10ms: %d\\n>20ms: %d\\n>30ms: %d\\n", read\_count1, read\_count10, read\_count20, read\_count30); printf("writes\\n> 1ms: %d\\n>10ms: %d\\n>20ms: %d\\n>30ms: %d\\n", write\_count1, write\_count10, write\_count20, write\_count30); printf("\\nblock size:%16s\\n","Read Count"); for (i in read\_count\_block) printf("%10d:%16d\\n", i, read\_count\_block\[i\]); printf("\\nblock size:%16s\\n","Write Count"); for (i in write\_count\_block) printf("%10d:%16d\\n", i, write\_count\_block\[i\]); \}'

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
     <div>
       6 
     </div> 
     <div>
       7 
     </div> 
     <div>
       8 
     </div> 
     <div>
       9 
     </div> 
     <div>
       10 
     </div> 
     <div>
       11 
     </div> 
     <div>
       12 
     </div> 
     <div>
       13 
     </div> 
     <div>
       14 
     </div> 
     <div>
       15 
     </div> 
     <div>
       16 
     </div> 
     <div>
       17 
     </div> 
     <div>
       18 
     </div> 
     <div>
       19 
     </div> 
     <div>
       20 
     </div> 
     <div>
       21 
     </div> 
     <div>
       22 
     </div> 
     <div>
       23 
     </div> 
     <div>
       24 
     </div> 
     <div>
       25 
     </div> 
     <div>
       26 
     </div> 
     <div>
       27 
     </div> 
     <div>
       28 
     </div> 
     <div>
       29 
     </div> 
     <div>
       30 
     </div> 
     <div>
       31 
     </div> 
     <div>
       32 
     </div> 
     <div>
       33 
     </div> 
     <div>
       34 
     </div> 
     <div>
       35 
     </div> 
     <div>
       36 
     </div> 
     <div>
       37 
     </div> 
     <div>
       38 
     </div> 
     <div>
       39 
     </div> 
     <div>
       40 
     </div> 
     <div>
       41 
     </div> 
     <div>
       42 
     </div> 
     <div>
       43 
     </div> 
     <div>
       44 
     </div> 
     <div>
       45 
     </div> 
     <div>
       46 
     </div> 
     <div>
       47 
     </div> 
     <div>
       48 
     </div> 
     <div>
       49 
     </div> 
     <div>
       50 
     </div> 
     <div>
       51 
     </div> 
     <div>
       52 
     </div> 
     <div>
       53 
     </div> 
     <div>
       54 
     </div> 
     <div>
       55 
     </div> 
     <div>
       56 
     </div> 
     <div>
       57 
     </div> 
     <div>
       58 
     </div> 
     <div>
       59 
     </div> 
     <div>
       60 
     </div> 
     <div>
       61 
     </div> 
     <div>
       62 
     </div> 
     <div>
       63 
     </div> 
     <div>
       64 
     </div> 
     <div>
       65 
     </div> 
     <div>
       66 
     </div> 
     <div>
       67 
     </div> 
     <div>
       68 
     </div> 
    </div> </td> 
   <td> 
    <div> 
     <div> 
      <span>#!/bin/bash</span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>if<span> <span>[<span> <span>$<span># -ne 1 ]; then</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>echo<span> <span>"Usage: $0 &lt;block_device_name&gt;"</span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>exit</span></span> 
     </div> 
     <div> 
      <span>fi</span> 
     </div> 
     <div> 
      <span>if<span> <span>[<span> <span>!<span> <span>-<span>b<span> <span>$<span>1<span> <span>]<span>;<span> <span>then</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>echo<span> <span>"could not find block device $1"</span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>exit</span></span> 
     </div> 
     <div> 
      <span>fi</span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>duration<span>=<span>10</span></span></span> 
     </div> 
     <div> 
      <span>echo<span> <span>"running blktrace for $duration seconds to collect data..."</span></span></span> 
     </div> 
     <div> 
      <span>timeout<span> <span>$duration<span> <span>blktrace<span> <span>-<span>d<span> <span>$<span>1<span> <span>&gt;<span>/<span>dev<span>/<span>null<span> <span>2<span>&gt;<span>&amp;<span>1</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>DEVNAME<span>=<span>`<span>basename<span> <span>$<span>1<span>`</span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>echo<span> <span>"parsing blktrace data..."</span></span></span> 
     </div> 
     <div> 
      <span>blkparse<span> <span>-<span>i<span> <span>$DEVNAME<span> <span>|<span>sort<span> <span>-<span>g<span> <span>-<span>k8<span> <span>-<span>k10<span> <span>-<span>k4<span> <span>|<span>awk<span> '</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>BEGIN<span>&nbsp;&nbsp; <span>{ </span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>total_read<span>=<span>0<span>;</span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>total_write<span>=<span>0<span>;</span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>maxwait_read<span>=<span>0<span>;</span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>maxwait_write<span>=<span>0<span>;</span></span></span></span></span> 
     </div> 
     <div> 
      <span>}</span> 
     </div> 
     <div> 
      <span>{ </span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>if<span> <span>(<span>$<span>6<span>==<span>"Q"<span>)<span> <span>{ </span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>queue_ts<span>=<span>$<span>4<span>;</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>block<span>=<span>$<span>8<span>;</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>nblock<span>=<span>$<span>10<span>;</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>rw<span>=<span>$<span>7<span>;</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>}<span>;</span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>if<span> <span>(<span>$<span>6<span>==<span>"C"<span> <span>&amp;&amp;<span> <span>$<span>8<span>==<span>block<span> <span>&amp;&amp;<span> <span>$<span>10<span>==<span>nblock<span> <span>&amp;&amp;<span> <span>$<span>7<span>==<span>rw<span>)<span> <span>{ </span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>await<span>=<span>$<span>4<span>-<span>queue_ts<span>;</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>if<span> <span>(<span>rw<span>==<span>"R"<span>)<span> <span>{ </span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>if<span> <span>(<span>await<span>&gt;<span>maxwait_read<span>)<span> <span>maxwait_read<span>=<span>await<span>;</span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>total_read<span>++<span>;</span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>read_count_block<span>[<span>nblock<span>]<span>++<span>;</span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>if<span> <span>(<span>await<span>&gt;<span>0.001<span>)<span> <span>read_count1<span>++<span>;</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>if<span> <span>(<span>await<span>&gt;<span>0.01<span>)<span> <span>read_count10<span>++<span>;</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>if<span> <span>(<span>await<span>&gt;<span>0.02<span>)<span> <span>read_count20<span>++<span>;</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>if<span> <span>(<span>await<span>&gt;<span>0.03<span>)<span> <span>read_count30<span>++<span>;</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>}</span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>if<span> <span>(<span>rw<span>==<span>"W"<span>)<span> <span>{ </span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>if<span> <span>(<span>await<span>&gt;<span>maxwait_write<span>)<span> <span>maxwait_write<span>=<span>await<span>;</span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>total_write<span>++<span>;</span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>write_count_block<span>[<span>nblock<span>]<span>++<span>;</span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>if<span> <span>(<span>await<span>&gt;<span>0.001<span>)<span> <span>write_count1<span>++<span>;</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>if<span> <span>(<span>await<span>&gt;<span>0.01<span>)<span> <span>write_count10<span>++<span>;</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>if<span> <span>(<span>await<span>&gt;<span>0.02<span>)<span> <span>write_count20<span>++<span>;</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>if<span> <span>(<span>await<span>&gt;<span>0.03<span>)<span> <span>write_count30<span>++<span>;</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>}</span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>}</span></span> 
     </div> 
     <div> 
      <span>}<span> <span>END<span>&nbsp;&nbsp; <span>{ </span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>printf<span>(<span>"========\nsummary:\n========\n"<span>)<span>;</span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>printf<span>(<span>"total number of reads: %d\n"<span>,<span> <span>total_read<span>)<span>;</span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>printf<span>(<span>"total number of writes: %d\n"<span>,<span> <span>total_write<span>)<span>;</span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>printf<span>(<span>"slowest read : %.6f second\n"<span>,<span> <span>maxwait_read<span>)<span>;</span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>printf<span>(<span>"slowest write: %.6f second\n"<span>,<span> <span>maxwait_write<span>)<span>;</span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>printf<span>(<span>"reads\n&gt; 1ms: %d\n&gt;10ms: %d\n&gt;20ms: %d\n&gt;30ms: %d\n"<span>,<span> <span>read_count1<span>,<span> <span>read_count10<span>,<span> <span>read_count20<span>,<span> <span>read_count30<span>)<span>;</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>printf<span>(<span>"writes\n&gt; 1ms: %d\n&gt;10ms: %d\n&gt;20ms: %d\n&gt;30ms: %d\n"<span>,<span> <span>write_count1<span>,<span> <span>write_count10<span>,<span> <span>write_count20<span>,<span> <span>write_count30<span>)<span>;</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>printf<span>(<span>"\nblock size:%16s\n"<span>,<span>"Read Count"<span>)<span>;</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>for<span> <span>(<span>i<span> <span>in<span> <span>read_count_block<span>)</span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>printf<span>(<span>"%10d:%16d\n"<span>,<span> <span>i<span>,<span> <span>read_count_block<span>[<span>i<span>]<span>)<span>;</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>printf<span>(<span>"\nblock size:%16s\n"<span>,<span>"Write Count"<span>)<span>;</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>for<span> <span>(<span>i<span> <span>in<span> <span>write_count_block<span>)</span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>printf<span>(<span>"%10d:%16d\n"<span>,<span> <span>i<span>,<span> <span>write_count_block<span>[<span>i<span>]<span>)<span>;</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>}'</span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

输出示例：

======== summary: ======== total number of reads: 1081513 total number of writes: 0 slowest read : 0.032560 second slowest write: 0.000000 second reads > 1ms: 18253 >10ms: 17058 >20ms: 17045 >30ms: 780 writes > 1ms: 0 >10ms: 0 >20ms: 0 >30ms: 0 block size: Read Count 256: 93756 248: 1538 64: 98084 56: 7475 8: 101218 48: 15889 240: 1637 232: 1651 224: 1942 40: 21693 216: 1811 32: 197893 208: 1907 24: 37787 128: 97382 16: 399850

这个例子统计了IO的读/写数量、最大延迟、延迟的分布情况、块大小及数量，这些信息比iostat要具体得多，有助于进一步了解系统的IO模式。blktrace数据还有更多的利用空间等待你去发掘，譬如还可以根据时间戳去统计每个毫秒内的IO数，有助于更微观地了解IO请求数量的波动。

参考：  
[利用BLKTRACE分析IO性能][BLKTRACE_IO]

转载自：

http://linuxperf.com/?cat=11

转载于:https://www.cnblogs.com/xibuhaohao/p/11096503.html

[I_O]: http://linuxperf.com/?p=227
[vmunix]: http://linuxperf.com/?author=1
[blktrace-event-output.jpeg]: /images/20211214/6967360850f24c0eb5a41ae20a498fd2.png
[BLKTRACE_IO]: http://linuxperf.com/?p=161