先日初めてCephとProxmoxを触ったのですが、SSDなら速いだろと思ってたのですがなかなか遅くて分散ストレージって難しいなと思ったのですが、もうちょっとどうにかならんかなということでOpane M10を使ってみた結果の話です。
書き込み速度の遅さは同期書き込みでSSDのDRAMキャッシュが無視された結果だとのことで、電力保護機能の付いたエンプラSSDやOptaneならそれなりに速度が出そうという情報にたどり着き、一番お手軽に試せる値段だった後者のOptane M10をebayから6つ買ってみました。
CephもProxmoxも初学者なので、理解の誤り等があるかもしれない点はご了承ください。
3台のちょい仕様違いの同型ホストでproxmox7.0を立てて、PacificのCephをセットアップしました。
(1)(2)ML110 Gen9
CPU: E5-2680 v4 @ 2.40GHz
RAM: 64GB
WAL/DB: Optane M10 16GB x2(PCIE x4接続)
OSD: Crucial BX500 960GB(B140i AHCI接続)
(3)ML110 Gen9
E5-2603 v4 @ 1.70GHz
RAM 64GB
WAL/DB: Optane M10 16GB x2(PCIE x4接続)
OSD: SANDISK SSD PLUS 960GB(B140i AHCI接続)
1,2台目はほぼ同じ仕様ですが3台目だけ寄せ集めです。
WALとDBにそれぞれOptane M10を1つずつ割り当ててます。合計3OSDの構成です。ベンチ取ったプールのPGは64だったと思います（パフォーマンスに関係あるかわからない）
書き込みパフォーマンスから。
どんどん落ち込みはしますが最初の方はちゃんとOptane M10の数値が出てますね
レイテンシ見ると高くなったり低くなったりしてるのでOptaneの容量不足かもしれないです。容量が大きいものであれば、安定した速度が出そうです
root@pve02:~# rados bench -p ceph-pool 600 write -b 4M -t 16 --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 600 seconds or 0 objects Object prefix: benchmark_data_pve02_588203 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 59 43 171.965 172 0.302155 0.279233 2 16 96 80 159.971 148 0.352309 0.310451 3 16 140 124 165.304 176 0.250117 0.300122 4 16 181 165 164.973 164 0.46753 0.363316 5 16 223 207 165.572 168 0.335612 0.374626 6 16 269 253 168.638 184 0.398286 0.366386 7 16 309 293 167.4 160 0.320529 0.37062 8 16 350 334 166.971 164 0.371455 0.365106 9 16 376 360 159.973 104 0.563815 0.375611 10 16 411 395 157.972 140 0.285516 0.382177 11 16 447 431 156.7 144 0.378483 0.379005 12 16 477 461 153.64 120 0.669639 0.404245 13 16 505 489 150.436 112 0.508691 0.415475 14 16 542 526 150.26 148 0.24359 0.41644 15 16 583 567 151.174 164 0.376658 0.417551 16 16 625 609 152.224 168 0.372797 0.4154 17 16 670 654 153.856 180 0.431735 0.410147 18 16 710 694 154.196 160 0.378545 0.410815 19 16 758 742 156.184 192 0.380034 0.405211 2021-11-02T00:29:47.784583+0900 min lat: 0.0640217 max lat: 5.23761 avg lat: 0.405829 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 797 781 156.174 156 0.416006 0.405829 21 16 846 830 158.068 196 0.277449 0.401871 22 16 883 867 157.609 148 0.368873 0.40145 23 16 923 907 157.712 160 0.344536 0.400207 24 16 967 951 158.473 176 0.374476 0.396294 25 16 1006 990 158.373 156 0.264756 0.393914 26 16 1055 1039 159.819 196 0.376732 0.397414 27 16 1097 1081 160.121 168 0.3682 0.39709 28 16 1140 1124 160.544 172 0.342407 0.395681 29 16 1176 1160 159.973 144 0.410897 0.395529 30 16 1218 1202 160.239 168 0.29346 0.392192 31 16 1259 1243 160.36 164 0.303161 0.388606 32 16 1302 1286 160.723 172 0.360005 0.396088 33 16 1344 1328 160.942 168 0.386347 0.394348 34 16 1383 1367 160.796 156 0.338867 0.394641 35 16 1427 1411 161.23 176 0.298365 0.393229 36 16 1464 1448 160.862 148 0.444889 0.395196 37 16 1472 1456 157.379 32 0.696538 0.396167 38 16 1473 1457 153.342 4 2.1451 0.397367 39 16 1474 1458 149.513 4 2.94094 0.399111 2021-11-02T00:30:07.787968+0900 min lat: 0.0640217 max lat: 5.23761 avg lat: 0.40619 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 1477 1461 146.075 12 4.05631 0.40619 41 16 1478 1462 142.61 4 4.74574 0.409159 42 16 1480 1464 139.405 8 5.85113 0.416241 43 16 1481 1465 136.256 4 6.622 0.420477 44 16 1483 1467 133.341 8 7.7848 0.430122 45 16 1484 1468 130.467 4 8.37164 0.435531 46 16 1486 1470 127.805 8 9.63381 0.447604 47 16 1486 1470 125.085 0 - 0.447604 48 16 1488 1472 122.646 4 9.88628 0.460903 49 16 1490 1474 120.306 8 9.69968 0.474998 50 16 1492 1476 118.06 8 9.89903 0.48764 51 16 1493 1477 115.823 4 10.4348 0.494375 52 16 1495 1479 113.75 8 10.4265 0.5078 53 16 1496 1480 111.679 4 9.90457 0.514149 54 16 1498 1482 109.759 8 11.9163 0.5281 55 16 1500 1484 107.909 8 10.4864 0.541495 56 16 1501 1485 106.054 4 10.714 0.548346 57 16 1502 1486 104.263 4 9.31513 0.554245 58 16 1504 1488 102.603 8 11.7939 0.567557 59 16 1506 1490 101 8 10.3446 0.580455 2021-11-02T00:30:27.791340+0900 min lat: 0.0640217 max lat: 11.9979 avg lat: 0.586981 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 1507 1491 99.3832 4 10.3109 0.586981 61 16 1509 1493 97.885 8 10.3009 0.60009 62 16 1510 1494 96.3708 4 10.2754 0.606567 63 16 1512 1496 94.9681 8 10.0276 0.61933 64 16 1514 1498 93.6092 8 10.2144 0.632206 65 16 1515 1499 92.2306 4 10.2215 0.638603 66 16 1517 1501 90.9544 8 9.91051 0.651122 67 16 1518 1502 89.6565 4 10.1498 0.657446 68 16 1520 1504 88.4557 8 10.1929 0.670109 69 16 1522 1506 87.2896 8 10.1279 0.682729 70 16 1523 1507 86.0997 4 10.3508 0.689144 71 16 1524 1508 84.9434 4 10.3412 0.695545 72 16 1526 1510 83.8747 8 10.1384 0.708229 73 16 1528 1512 82.8354 8 9.93558 0.720519 74 16 1530 1514 81.824 8 9.64409 0.732287 75 16 1532 1516 80.8397 8 9.4892 0.74386 76 16 1534 1518 79.8813 8 8.99299 0.754968 77 16 1536 1520 78.9477 8 8.71985 0.765542 78 16 1540 1524 78.1407 16 7.39825 0.785011 79 16 1542 1526 77.2528 8 7.4422 0.793634 2021-11-02T00:30:47.794733+0900 min lat: 0.0640217 max lat: 11.9979 avg lat: 0.802233 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 80 16 1544 1528 76.3871 8 7.37682 0.802233 81 16 1546 1530 75.5427 8 7.31428 0.810888 82 16 1548 1532 74.719 8 7.28205 0.819397 83 16 1550 1534 73.9152 8 7.43494 0.827921 84 16 1551 1535 73.0828 4 7.42987 0.832222 85 16 1552 1536 72.2701 4 8.03438 0.836911 86 16 1553 1537 71.4762 4 8.64075 0.841989 87 16 1554 1538 70.7006 4 9.25482 0.847458 88 16 1555 1539 69.9427 4 9.9848 0.853396 89 16 1556 1540 69.2017 4 10.4288 0.859613 90 16 1558 1542 68.5217 8 11.0398 0.872595 91 16 1574 1558 68.4719 64 0.621977 0.928939 92 16 1574 1558 67.7276 0 - 0.928939 93 16 1576 1560 67.0854 4 2.09755 0.930296 94 16 1578 1562 66.4568 8 3.55953 0.933119 95 16 1581 1565 65.8836 12 4.57355 0.939258 96 16 1582 1566 65.239 4 5.12222 0.941929 97 16 1584 1568 64.6489 8 6.5302 0.948654 98 16 1585 1569 64.03 4 7.06041 0.952549 99 16 1587 1571 63.464 8 8.16233 0.961368 2021-11-02T00:31:07.798115+0900 min lat: 0.0640217 max lat: 11.9979 avg lat: 0.971841 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 100 16 1589 1573 62.9094 8 9.47527 0.971841 101 16 1590 1574 62.3261 4 9.91033 0.97752 102 16 1592 1576 61.7934 8 9.3174 0.987856 103 16 1593 1577 61.2323 4 9.30294 0.993128 104 16 1595 1579 60.7204 8 9.523 1.0037 105 16 1596 1580 60.1803 4 9.91178 1.00934 106 16 1598 1582 59.688 8 10.0962 1.02078 107 16 1599 1583 59.1675 4 9.87688 1.02637 108 16 1601 1585 58.6937 8 10.1326 1.03785 109 16 1603 1587 58.2286 8 10.2073 1.04942 110 16 1604 1588 57.7357 4 10.0489 1.05508 111 16 1606 1590 57.2876 8 10.0885 1.06642 112 16 1608 1592 56.8475 8 10.1625 1.07787 113 16 1609 1593 56.3798 4 10.1912 1.08359 114 16 1611 1595 55.9554 8 10.0061 1.0948 115 16 1613 1597 55.5384 8 9.96083 1.10592 116 16 1614 1598 55.0941 4 9.91723 1.11144 117 16 1616 1600 54.6916 8 9.87934 1.12241 118 16 1617 1601 54.262 4 9.86232 1.12786 119 16 1619 1603 53.8732 8 9.83002 1.1387 2021-11-02T00:31:27.801585+0900 min lat: 0.0640217 max lat: 11.9979 avg lat: 1.14425 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 120 16 1620 1604 53.4576 4 10.041 1.14425 121 16 1622 1606 53.0819 8 10.0715 1.15534 122 16 1623 1607 52.6796 4 9.88141 1.16077 123 16 1625 1609 52.3163 8 9.69102 1.17135 124 16 1628 1612 51.9912 12 9.5053 1.18787 125 16 1629 1613 51.6072 4 9.24868 1.19286 126 16 1632 1616 51.2929 12 10.2475 1.20845 127 16 1634 1618 50.952 8 8.6179 1.21771 128 16 1635 1619 50.5852 4 8.99225 1.22251 129 16 1635 1619 50.193 0 - 1.22251 130 16 1636 1620 49.8377 2 9.62542 1.2277 131 16 1638 1622 49.5183 8 10.0392 1.23854 132 16 1644 1628 49.325 24 7.77527 1.26736 133 16 1700 1684 50.638 224 0.339871 1.26216 134 16 1746 1730 51.633 184 0.388686 1.23771 135 16 1786 1770 52.4356 160 0.304148 1.21825 136 16 1834 1818 53.4615 192 0.349844 1.19551 137 16 1861 1845 53.8595 108 0.55876 1.18562 138 16 1892 1876 54.3676 124 0.366159 1.17489 139 16 1942 1926 55.4151 200 0.312913 1.15351 2021-11-02T00:31:47.804926+0900 min lat: 0.0640217 max lat: 11.9979 avg lat: 1.13736 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 140 16 1981 1965 56.1333 156 0.359962 1.13736 141 16 2026 2010 57.0116 180 0.236976 1.11861 142 16 2068 2052 57.793 168 0.281214 1.10122 143 16 2105 2089 58.4237 148 0.27386 1.08671 144 16 2159 2143 59.5177 216 0.332117 1.07367 145 16 2201 2185 60.2656 168 0.345887 1.06064 146 16 2247 2231 61.1129 184 0.29659 1.0459 147 16 2289 2273 61.8399 168 0.329595 1.03289 148 16 2330 2314 62.5299 164 0.299823 1.02017 149 16 2374 2358 63.2913 176 0.234298 1.00646 150 16 2415 2399 63.9625 164 0.244259 0.993809 151 16 2456 2440 64.6248 164 0.28723 0.988988 152 16 2494 2478 65.1995 152 0.35544 0.979948 153 16 2534 2518 65.8189 160 0.321084 0.969907 154 16 2575 2559 66.4563 164 0.227402 0.961406 155 16 2617 2601 67.1112 168 0.3444 0.951042 156 16 2651 2635 67.5527 136 0.376077 0.945464 157 16 2695 2679 68.2433 176 0.373967 0.936417 158 16 2734 2718 68.7985 156 0.335418 0.927923 159 16 2773 2757 69.3468 156 0.266545 0.919493 2021-11-02T00:32:07.808152+0900 min lat: 0.0640217 max lat: 11.9979 avg lat: 0.910416 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 160 16 2813 2797 69.9132 160 0.343592 0.910416 161 16 2848 2832 70.3484 140 0.339608 0.902829 162 16 2895 2879 71.0744 188 0.309845 0.898894 163 16 2935 2919 71.6198 160 0.342288 0.892871 164 16 2972 2956 72.0854 148 0.39933 0.886409 165 16 3017 3001 72.7391 180 0.301469 0.878767 166 16 3054 3038 73.1923 148 0.494849 0.873161 167 16 3094 3078 73.712 160 0.263186 0.865683 168 16 3140 3124 74.3683 184 0.369167 0.859682 169 16 3179 3163 74.8511 156 0.282716 0.853776 170 16 3220 3204 75.3754 164 0.353765 0.846574 171 16 3227 3211 75.0983 28 0.973748 0.847581 172 16 3228 3212 74.6849 4 2.00731 0.847942 173 16 3229 3213 74.2763 4 2.77641 0.848543 174 16 3232 3216 73.9184 12 3.93364 0.851222 175 16 3233 3217 73.5189 4 4.60761 0.85239 176 16 3235 3219 73.1466 8 5.65276 0.855217 177 16 3237 3221 72.7785 8 6.73342 0.858695 178 16 3238 3222 72.3921 4 7.18939 0.86066 179 16 3240 3224 72.0324 8 8.26629 0.865081 2021-11-02T00:32:27.811908+0900 min lat: 0.0640217 max lat: 11.9979 avg lat: 0.870189 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 180 16 3242 3226 71.6767 8 9.38804 0.870189 181 16 3243 3227 71.3027 4 9.69878 0.872925 182 16 3245 3229 70.9549 8 9.11045 0.878048 183 16 3247 3231 70.6109 8 9.07728 0.883158 184 16 3248 3232 70.2489 4 9.69497 0.885885 185 16 3250 3234 69.9124 8 9.60291 0.89125 186 16 3252 3236 69.5795 8 9.69868 0.896681 187 16 3253 3237 69.2288 4 9.67823 0.899394 188 16 3255 3239 68.9031 8 9.77029 0.904853 189 16 3257 3241 68.5809 8 9.80992 0.910337 190 16 3258 3242 68.241 4 9.79834 0.913079 191 16 3260 3244 67.9256 8 9.83463 0.918541 192 16 3261 3245 67.5926 4 9.87954 0.921302 193 16 3263 3247 67.2839 8 9.831 0.926759 194 16 3265 3249 66.9783 8 9.87373 0.932256 195 16 3266 3250 66.6553 4 9.90832 0.935018 196 16 3268 3252 66.356 8 9.95929 0.940558 197 16 3269 3253 66.0395 4 9.97457 0.943335 198 16 3271 3255 65.7464 8 9.95345 0.948882 199 16 3273 3257 65.4562 8 9.99643 0.954434 2021-11-02T00:32:47.815306+0900 min lat: 0.0640217 max lat: 11.9979 avg lat: 0.957213 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 200 16 3274 3258 65.1489 4 10.0093 0.957213 201 16 3276 3260 64.8645 8 9.94971 0.962756 202 16 3278 3262 64.583 8 9.97085 0.96826 203 16 3279 3263 64.2846 4 10.1448 0.971073 204 16 3281 3265 64.0087 8 9.98265 0.976596 205 16 3282 3266 63.7159 4 10.0742 0.979381 206 16 3284 3268 63.4455 8 9.96779 0.984903 207 16 3286 3270 63.1776 8 9.71664 0.99031 208 16 3287 3271 62.8931 4 9.94546 0.993047 209 16 3289 3273 62.6304 8 9.91851 0.998498 210 16 3290 3274 62.3513 4 9.97145 1.00124 211 16 3292 3276 62.0937 8 9.76105 1.0066 212 16 3294 3278 61.8385 8 9.75099 1.01194 213 16 3296 3280 61.5857 8 9.63892 1.01723 214 16 3298 3282 61.3353 8 9.207 1.02231 215 16 3300 3284 61.0872 8 9.16258 1.02728 216 16 3302 3286 60.8415 8 8.98837 1.03216 217 16 3304 3288 60.598 8 8.71579 1.03681 218 16 3306 3290 60.3567 8 8.41163 1.04134 219 16 3307 3291 60.0993 4 8.33324 1.04356 2021-11-02T00:33:07.818703+0900 min lat: 0.0640217 max lat: 11.9979 avg lat: 1.0459 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 220 16 3308 3292 59.8443 4 8.76755 1.0459 221 16 3309 3293 59.5916 4 9.65903 1.04852 222 16 3310 3294 59.3412 4 9.97074 1.05123 223 16 3312 3296 59.111 8 10.0572 1.05667 224 16 3328 3312 59.1328 64 0.725485 1.07935 225 16 3328 3312 58.87 0 - 1.07935 226 16 3330 3314 58.6449 4 2.65718 1.0803 227 16 3331 3315 58.4042 4 3.66891 1.08108 228 16 3334 3318 58.2006 12 4.44532 1.08372 229 16 3336 3320 57.9814 8 5.32195 1.08613 230 16 3338 3322 57.7641 8 6.58769 1.08924 231 16 3339 3323 57.5313 4 7.15763 1.09106 232 16 3340 3324 57.3006 4 7.69962 1.09305 233 16 3342 3326 57.089 8 9.51529 1.09793 234 16 3342 3326 56.845 0 - 1.09793 235 16 3344 3328 56.6372 4 8.85943 1.10243 236 16 3348 3332 56.465 16 8.92811 1.11362 237 16 3349 3333 56.2436 4 9.37411 1.1161 238 16 3351 3335 56.0409 8 9.68224 1.12117 239 16 3352 3336 55.8231 4 9.82465 1.12378 2021-11-02T00:33:27.821989+0900 min lat: 0.0640217 max lat: 12.308 avg lat: 1.12901 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 240 16 3354 3338 55.6239 8 9.8443 1.12901 241 16 3355 3339 55.4097 4 10.0374 1.13168 242 16 3357 3341 55.2138 8 8.87386 1.1363 243 16 3357 3341 54.9865 0 - 1.1363 244 16 3359 3343 54.794 4 12.5823 1.1418 245 16 3360 3344 54.5866 4 10.0654 1.14447 246 16 3362 3346 54.3972 8 9.9066 1.14956 247 16 3365 3349 54.2256 12 10.3078 1.15807 248 16 3366 3350 54.0231 4 9.66187 1.1606 249 16 3368 3352 53.8382 8 9.76205 1.16614 250 16 3369 3353 53.6389 4 9.71855 1.16869 251 16 3370 3354 53.4411 4 11.5747 1.17179 252 16 3372 3356 53.2608 8 11.3585 1.17731 253 16 3373 3357 53.0661 4 9.15935 1.17969 254 16 3375 3359 52.8887 8 12.2333 1.18546 255 16 3377 3361 52.7126 8 8.94819 1.1903 256 16 3379 3363 52.5379 8 9.34305 1.19509 257 16 3381 3365 52.3646 8 9.94418 1.20068 258 16 3383 3367 52.1927 8 9.52103 1.20571 259 16 3385 3369 52.022 8 9.24619 1.21053 2021-11-02T00:33:47.825357+0900 min lat: 0.0640217 max lat: 12.5823 avg lat: 1.21507 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 260 16 3387 3371 51.8527 8 8.46317 1.21507 261 16 3389 3373 51.6847 8 8.21026 1.21928 262 16 3391 3375 51.518 8 8.40689 1.22347 263 16 3392 3376 51.3373 4 8.5792 1.22565 264 16 3393 3377 51.158 4 9.2051 1.22801 265 16 3393 3377 50.9649 0 - 1.22801 266 16 3395 3379 50.8034 4 10.1447 1.23322 267 16 3405 3389 50.7629 40 6.1451 1.25406 268 16 3458 3442 51.3644 212 0.384336 1.24504 269 16 3505 3489 51.8722 188 0.302666 1.23293 270 16 3546 3530 52.2874 164 0.376482 1.22272 271 16 3582 3566 52.6258 144 0.463934 1.21439 272 16 3610 3594 52.844 112 0.550364 1.20886 273 16 3651 3635 53.2511 164 0.372393 1.20096 274 16 3697 3681 53.7281 184 0.278212 1.19013 275 16 3738 3722 54.129 164 0.277646 1.1806 276 16 3782 3766 54.5705 176 0.400907 1.17188 277 16 3825 3809 54.9943 172 0.398939 1.16292 278 16 3871 3855 55.4582 184 0.358813 1.15332 279 16 3912 3896 55.8472 164 0.31207 1.14504 2021-11-02T00:34:07.828578+0900 min lat: 0.0640217 max lat: 12.5823 avg lat: 1.13682 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 280 16 3953 3937 56.2333 164 0.336942 1.13682 281 16 3992 3976 56.5883 156 0.384379 1.12933 282 16 4040 4024 57.0684 192 0.370553 1.12076 283 16 4077 4061 57.3896 148 0.419191 1.11403 284 16 4124 4108 57.8494 188 1.12227 1.10603 285 16 4168 4152 58.2638 176 0.374688 1.09771 286 16 4209 4193 58.6334 164 0.397875 1.09072 287 16 4252 4236 59.0283 172 0.419613 1.08335 288 16 4295 4279 59.4205 172 0.347448 1.07642 289 16 4337 4321 59.7961 168 0.337534 1.06944 290 16 4378 4362 60.1553 164 0.357567 1.06279 291 16 4424 4408 60.5808 184 0.368637 1.0557 292 16 4467 4451 60.9623 172 0.411104 1.04911 293 16 4497 4481 61.1637 120 0.377708 1.04452 294 16 4500 4484 60.9965 12 1.44547 1.04454 295 16 4501 4485 60.8033 4 1.99899 1.04476 296 16 4502 4486 60.6114 4 3.19286 1.04524 297 16 4506 4490 60.4612 16 4.279 1.04787 298 16 4508 4492 60.2851 8 5.1706 1.04958 299 16 4510 4494 60.1103 8 6.34651 1.05182 2021-11-02T00:34:27.831828+0900 min lat: 0.0640217 max lat: 12.5823 avg lat: 1.05311 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 300 16 4511 4495 59.9232 4 6.871 1.05311 301 16 4513 4497 59.7507 8 8.02703 1.05607 302 16 4515 4499 59.5793 8 8.84771 1.05939 303 16 4516 4500 59.3959 4 8.66909 1.06108 304 16 4518 4502 59.2268 8 8.19454 1.06436 305 16 4519 4503 59.0458 4 8.27253 1.06596 306 16 4521 4505 58.879 8 9.08013 1.0694 307 16 4523 4507 58.7132 8 9.62723 1.07315 308 16 4524 4508 58.5356 4 9.61335 1.07504 309 16 4526 4510 58.372 8 9.62929 1.07884 310 16 4528 4512 58.2096 8 9.78547 1.08267 311 16 4529 4513 58.0352 4 9.82058 1.0846 312 16 4531 4515 57.8749 8 9.79913 1.08848 313 16 4533 4517 57.7155 8 9.72175 1.0923 314 16 4534 4518 57.5445 4 9.75373 1.09422 315 16 4536 4520 57.3872 8 9.72593 1.09805 316 16 4538 4522 57.2309 8 9.72673 1.10189 317 16 4539 4523 57.0629 4 9.72411 1.10379 318 16 4541 4525 56.9087 8 9.90585 1.10766 319 16 4542 4526 56.7428 4 9.93077 1.10961 2021-11-02T00:34:47.835075+0900 min lat: 0.0640217 max lat: 12.5823 avg lat: 1.11349 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 320 16 4544 4528 56.5905 8 9.79593 1.11349 321 16 4546 4530 56.4391 8 9.86196 1.11735 322 16 4547 4531 56.2762 4 9.8707 1.11928 323 16 4549 4533 56.1268 8 9.91403 1.12316 324 16 4550 4534 55.9659 4 10.0152 1.12512 325 16 4552 4536 55.8183 8 9.94202 1.12902 326 16 4554 4538 55.6716 8 9.90435 1.13288 327 16 4555 4539 55.5136 4 10.1028 1.13486 328 16 4557 4541 55.3687 8 9.82176 1.13873 329 16 4559 4543 55.2247 8 9.77883 1.14254 330 16 4560 4544 55.0695 4 10.0149 1.14449 331 16 4562 4546 54.9273 8 9.77599 1.1483 332 16 4563 4547 54.7739 4 9.96825 1.15024 333 16 4565 4549 54.6334 8 9.92098 1.15411 334 16 4567 4551 54.4938 8 9.89196 1.1579 335 16 4568 4552 54.3431 4 9.89553 1.15982 336 16 4571 4555 54.2171 12 9.32557 1.16539 337 16 4572 4556 54.068 4 9.40835 1.1672 338 16 4574 4558 53.9317 8 9.16821 1.17075 339 16 4577 4561 53.808 12 8.74463 1.17585 2021-11-02T00:35:07.838449+0900 min lat: 0.0640217 max lat: 12.5823 avg lat: 1.17753 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 340 16 4578 4562 53.6616 4 8.83401 1.17753 341 16 4580 4564 53.5276 8 8.4178 1.18073 342 16 4581 4565 53.3828 4 8.46948 1.18233 343 16 4582 4566 53.2388 4 9.08982 1.18406 344 16 4583 4567 53.0957 4 9.72954 1.18593 345 16 4584 4568 52.9534 4 9.95019 1.18785 346 16 4601 4585 52.9969 68 0.767586 1.20652 347 16 4602 4586 52.8556 4 1.00207 1.20648 348 16 4603 4587 52.7153 4 2.25084 1.20671 349 16 4605 4589 52.5871 8 3.21273 1.20746 350 16 4606 4590 52.4483 4 3.49988 1.20795 351 16 4607 4591 52.3103 4 4.50975 1.20867 352 16 4608 4592 52.173 4 5.92964 1.2097 353 16 4611 4595 52.0592 12 7.14488 1.21333 354 16 4612 4596 51.9235 4 7.74693 1.21475 355 16 4613 4597 51.7885 4 8.31133 1.2163 356 16 4614 4598 51.6542 4 9.77945 1.21816 357 16 4614 4598 51.5095 0 - 1.21816 358 16 4619 4603 51.4215 10 11.0884 1.2288 359 16 4622 4606 51.3117 12 9.75634 1.23438 2021-11-02T00:35:27.841751+0900 min lat: 0.0640217 max lat: 12.5823 avg lat: 1.23438 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 360 16 4622 4606 51.1692 0 - 1.23438 361 16 4623 4607 51.0385 2 7.97962 1.23584 362 16 4624 4608 50.9086 4 8.74746 1.23747 363 16 4626 4610 50.7904 8 8.93807 1.24079 364 16 4628 4612 50.6728 8 7.98293 1.2445 365 16 4629 4613 50.5449 4 7.3346 1.24582 366 16 4632 4616 50.4396 12 8.94766 1.25174 367 16 4633 4617 50.3131 4 8.92668 1.2534 368 16 4634 4618 50.1872 4 9.57812 1.2552 369 16 4635 4619 50.062 4 10.1061 1.25712 370 16 4636 4620 49.9375 4 10.5854 1.25914 371 16 4640 4624 49.8461 16 9.27175 1.26901 372 16 4641 4625 49.7228 4 9.03154 1.27069 373 16 4643 4627 49.611 8 9.52497 1.27418 374 16 4644 4628 49.489 4 9.51631 1.27596 375 16 4645 4629 49.3677 4 8.834 1.27759 376 16 4647 4631 49.2577 8 9.87738 1.28113 377 16 4649 4633 49.1482 8 9.67575 1.28479 378 16 4651 4635 49.0394 8 14.2303 1.28932 379 16 4653 4637 48.9311 8 8.91605 1.29259 2021-11-02T00:35:47.845182+0900 min lat: 0.0640217 max lat: 19.9062 avg lat: 1.29616 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 380 16 4655 4639 48.8234 8 9.80491 1.29616 381 16 4657 4641 48.7162 8 9.45697 1.29967 382 16 4659 4643 48.6096 8 9.16591 1.30309 383 16 4661 4645 48.5036 8 8.41869 1.30617 384 16 4663 4647 48.3981 8 7.98085 1.3091 385 16 4664 4648 48.2828 4 7.98872 1.31054 386 16 4665 4649 48.1681 4 8.47518 1.31208 387 16 4666 4650 48.0539 4 8.8793 1.31371 388 16 4667 4651 47.9404 4 9.59712 1.31549 389 16 4668 4652 47.8274 4 10.0041 1.31736 390 16 4714 4698 48.1765 184 0.244404 1.32799 391 16 4768 4752 48.6056 216 0.243521 1.31617 392 16 4813 4797 48.9407 180 0.340054 1.30695 393 16 4858 4842 49.2741 180 0.362724 1.29823 394 16 4906 4890 49.6363 192 0.242189 1.28893 395 16 4942 4926 49.8752 144 0.392419 1.2824 396 16 4989 4973 50.2239 188 0.377182 1.27372 397 16 5027 5011 50.4802 152 0.380128 1.26705 398 16 5070 5054 50.7854 172 0.35552 1.25963 399 16 5110 5094 51.0591 160 0.420795 1.25264 2021-11-02T00:36:07.848414+0900 min lat: 0.0640217 max lat: 19.9062 avg lat: 1.2489 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 400 16 5138 5122 51.2114 112 0.58793 1.2489 401 16 5172 5156 51.4228 136 0.38158 1.244 402 16 5214 5198 51.7127 168 0.46313 1.23694 403 16 5254 5238 51.9813 160 0.376296 1.23054 404 16 5299 5283 52.2981 180 0.35501 1.22322 405 16 5336 5320 52.5344 148 0.417457 1.21745 406 16 5363 5347 52.6709 108 0.523305 1.21346 407 16 5364 5348 52.5514 4 1.06465 1.21343 408 16 5366 5350 52.4422 8 2.22445 1.2137 409 16 5367 5351 52.3237 4 3.75777 1.21417 410 16 5369 5353 52.2156 8 4.54188 1.21537 411 16 5371 5355 52.108 8 5.28122 1.21678 412 16 5373 5357 52.001 8 6.39076 1.21861 413 16 5374 5358 51.8847 4 6.95757 1.21969 414 16 5376 5360 51.7787 8 8.24044 1.22221 415 16 5377 5361 51.6636 4 8.7814 1.22362 416 16 5379 5363 51.5586 8 10.3368 1.22688 417 16 5381 5365 51.4542 8 9.25091 1.22987 418 16 5382 5366 51.3406 4 8.50127 1.23122 419 16 5384 5368 51.2372 8 8.8314 1.23397 2021-11-02T00:36:27.851944+0900 min lat: 0.0640217 max lat: 19.9062 avg lat: 1.23695 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 420 16 5386 5370 51.1342 8 9.23982 1.23695 421 16 5387 5371 51.0223 4 8.67552 1.23833 422 16 5389 5373 50.9203 8 9.39956 1.24241 423 16 5390 5374 50.8094 4 11.1502 1.24426 424 16 5392 5376 50.7084 8 9.80784 1.24744 425 16 5394 5378 50.6079 8 9.43897 1.25056 426 16 5395 5379 50.4985 4 9.55698 1.2521 427 16 5397 5381 50.399 8 9.76339 1.25527 428 16 5399 5383 50.2999 8 9.81747 1.25846 429 16 5400 5384 50.192 4 9.96301 1.26007 430 16 5402 5386 50.0939 8 9.79607 1.26325 431 16 5404 5388 49.9962 8 9.50989 1.26637 432 16 5405 5389 49.8897 4 9.92944 1.26798 433 16 5407 5391 49.793 8 9.91391 1.27118 434 16 5408 5392 49.6875 4 10.022 1.27281 435 16 5410 5394 49.5917 8 8.92991 1.27585 436 16 5411 5395 49.4871 4 8.89268 1.27726 437 16 5411 5395 49.3738 0 - 1.27726 438 16 5411 5395 49.2611 0 - 1.27726 439 16 5413 5397 49.1671 2.66667 10.1271 1.28056 2021-11-02T00:36:47.855271+0900 min lat: 0.0640217 max lat: 19.9062 avg lat: 1.29056 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 440 16 5418 5402 49.1008 20 10.1186 1.29056 441 16 5419 5403 48.9985 4 10.0999 1.29219 442 16 5421 5405 48.9058 8 10.0122 1.29546 443 16 5423 5407 48.8134 8 10.0647 1.29869 444 16 5424 5408 48.7125 4 10.1949 1.30034 445 16 5426 5410 48.621 8 9.94377 1.30357 446 16 5427 5411 48.521 4 7.56506 1.30473 447 16 5429 5413 48.4303 8 7.63533 1.30704 448 16 5430 5414 48.3311 4 8.67312 1.3084 449 16 5432 5416 48.2413 8 9.59667 1.31137 450 16 5434 5418 48.1519 8 8.7712 1.31418 451 16 5436 5420 48.0628 8 8.57038 1.31687 452 16 5438 5422 47.9742 8 8.19249 1.31944 453 16 5441 5425 47.8948 12 13.0528 1.32562 454 16 5442 5426 47.7981 4 9.02291 1.32703 455 16 5443 5427 47.7018 4 8.84645 1.32842 456 16 5444 5428 47.606 4 9.32977 1.32989 457 16 5445 5429 47.5106 4 9.59775 1.33142 458 16 5446 5430 47.4156 4 8.95467 1.33282 459 16 5493 5477 47.7218 188 0.298506 1.3405 2021-11-02T00:37:07.858706+0900 min lat: 0.0640217 max lat: 19.9062 avg lat: 1.33251 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 460 16 5538 5522 48.0093 180 0.385662 1.33251 461 16 5579 5563 48.2608 164 0.339389 1.32563 462 16 5619 5603 48.5026 160 0.424287 1.31894 463 16 5662 5646 48.7693 172 0.321716 1.31179 464 16 5703 5687 49.0176 164 0.378395 1.30507 465 16 5747 5731 49.2906 176 0.344555 1.29785 466 16 5790 5774 49.5539 172 0.401751 1.29098 467 16 5832 5816 49.8075 168 0.361983 1.28444 468 16 5877 5861 50.0856 180 0.361287 1.27717 469 16 5908 5892 50.2431 124 0.472755 1.27248 470 16 5919 5903 50.2298 44 1.21603 1.27186 471 16 5920 5904 50.1317 4 1.72273 1.27193 472 16 5921 5905 50.0339 4 2.48236 1.27214 473 16 5923 5907 49.9451 8 3.6848 1.27296 474 16 5924 5908 49.8481 4 4.23176 1.27346 475 16 5927 5911 49.7685 12 5.7886 1.2756 476 16 5928 5912 49.6723 4 6.37402 1.27646 477 16 5930 5914 49.5849 8 7.8111 1.27853 478 16 5931 5915 49.4896 4 8.10548 1.27968 479 16 5933 5917 49.403 8 9.31731 1.28227 2021-11-02T00:37:27.862050+0900 min lat: 0.0640217 max lat: 19.9062 avg lat: 1.28373 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 480 16 5934 5918 49.3084 4 9.94045 1.28373 481 16 5936 5920 49.2225 8 10.343 1.28683 482 16 5936 5920 49.1204 0 - 1.28683 483 16 5939 5923 49.0435 6 10.174 1.29139 484 16 5940 5924 48.9504 4 10.1221 1.29288 485 16 5942 5926 48.866 8 10.3508 1.29587 486 16 5943 5927 48.7737 4 10.5872 1.29743 487 16 5945 5929 48.6899 8 10.7907 1.30061 488 16 5946 5930 48.5984 4 10.4949 1.30216 489 16 5948 5932 48.5153 8 10.6333 1.30528 490 16 5950 5934 48.4327 8 10.4461 1.30837 491 16 5951 5935 48.3422 4 10.5284 1.30992 492 16 5953 5937 48.2602 8 9.78197 1.31286 493 16 5954 5938 48.1704 4 9.90781 1.31431 494 16 5956 5940 48.0891 8 10.4199 1.31738 495 16 5958 5942 48.0081 8 10.1977 1.32037 496 16 5959 5943 47.9193 4 10.1342 1.32185 497 16 5961 5945 47.839 8 9.89717 1.32479 498 16 5962 5946 47.751 4 10.1222 1.32627 499 16 5964 5948 47.6713 8 10.0534 1.32921 2021-11-02T00:37:47.865501+0900 min lat: 0.0640217 max lat: 19.9062 avg lat: 1.33067 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 500 16 5965 5949 47.584 4 10.0505 1.33067 501 16 5967 5951 47.505 8 10.0619 1.33362 502 16 5969 5953 47.4263 8 10.034 1.33656 503 16 5970 5954 47.3399 4 10.1536 1.33804 504 16 5972 5956 47.2619 8 10.1983 1.34101 505 16 5973 5957 47.1762 4 10.3302 1.34252 506 16 5975 5959 47.0988 8 10.044 1.34546 507 16 5977 5961 47.0217 8 9.82304 1.3483 508 16 5979 5963 46.9448 8 9.53198 1.35104 509 16 5979 5963 46.8526 0 - 1.35104 510 16 5981 5965 46.7764 4 8.97063 1.35362 511 16 5989 5973 46.7475 32 6.28404 1.36369 512 16 6046 6030 47.1014 228 0.262084 1.35821 513 16 6088 6072 47.337 168 0.232471 1.35151 514 16 6125 6109 47.5328 148 0.354651 1.34572 515 16 6127 6111 47.4561 8 1.3347 1.34563 516 16 6128 6112 47.3719 4 2.34577 1.3458 517 16 6129 6113 47.288 4 2.92107 1.34605 518 16 6131 6115 47.2121 8 4.05308 1.34689 519 16 6133 6117 47.1366 8 5.01625 1.348 2021-11-02T00:38:07.868948+0900 min lat: 0.0640217 max lat: 19.9062 avg lat: 1.3495 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 520 16 6135 6119 47.0613 8 6.20936 1.3495 521 16 6136 6120 46.9786 4 6.74999 1.35038 522 16 6138 6122 46.904 8 7.96995 1.35246 523 16 6140 6124 46.8296 8 9.11892 1.3549 524 16 6141 6125 46.7478 4 9.70277 1.35626 525 16 6142 6126 46.6664 4 9.81679 1.35764 526 16 6144 6128 46.5929 8 9.65011 1.36042 527 16 6146 6130 46.5197 8 9.39339 1.36308 528 16 6147 6131 46.4391 4 9.72732 1.36444 529 16 6149 6133 46.3665 8 9.9636 1.36724 530 16 6150 6134 46.2865 4 10.1308 1.36867 531 16 6152 6136 46.2144 8 10.1654 1.37153 532 16 6153 6137 46.1351 4 10.1231 1.37296 533 16 6155 6139 46.0635 8 10.2461 1.37584 534 16 6156 6140 45.9847 4 10.3025 1.3773 535 16 6156 6140 45.8988 0 - 1.3773 536 16 6159 6143 45.8355 6 9.91725 1.38162 537 16 6162 6146 45.7725 12 10.165 1.38628 538 16 6163 6147 45.6949 4 10.0593 1.38769 539 16 6165 6149 45.6249 8 10.2158 1.39052 2021-11-02T00:38:27.872497+0900 min lat: 0.0640217 max lat: 19.9062 avg lat: 1.39192 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 540 16 6166 6150 45.5479 4 10.0064 1.39192 541 16 6168 6152 45.4784 8 10.1189 1.39475 542 16 6169 6153 45.4019 4 10.0209 1.39616 543 16 6171 6155 45.333 8 10.1796 1.39901 544 16 6172 6156 45.257 4 10.1378 1.40043 545 16 6174 6158 45.1887 8 9.73089 1.40303 546 16 6175 6159 45.1132 4 9.78517 1.40439 547 16 6177 6161 45.0454 8 10.3103 1.40722 548 16 6179 6163 44.9778 8 10.3455 1.41011 549 16 6180 6164 44.9031 4 10.382 1.41156 550 16 6182 6166 44.836 8 10.2435 1.41442 551 16 6183 6167 44.7619 4 10.1672 1.41583 552 16 6185 6169 44.6953 8 10.252 1.4187 553 16 6187 6171 44.629 8 10.0591 1.4215 554 16 6188 6172 44.5556 4 10.093 1.42291 555 16 6190 6174 44.4898 8 10.0162 1.4257 556 16 6191 6175 44.4169 4 10.1093 1.42711 557 16 6193 6177 44.3516 8 9.89932 1.42987 558 16 6195 6179 44.2864 8 9.67348 1.43257 559 16 6197 6181 44.2215 8 9.41629 1.43519 2021-11-02T00:38:47.876015+0900 min lat: 0.0640217 max lat: 19.9062 avg lat: 1.43771 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 560 16 6199 6183 44.1568 8 9.16272 1.43771 561 16 6201 6185 44.0924 8 8.89765 1.44015 562 16 6203 6187 44.0281 8 8.63767 1.4425 563 16 6204 6188 43.957 4 8.50954 1.44364 564 16 6205 6189 43.8862 4 8.92195 1.44485 565 16 6206 6190 43.8156 4 9.4448 1.44614 566 16 6207 6191 43.7452 4 9.8194 1.44749 567 16 6222 6206 43.7739 60 2.79029 1.46135 568 16 6277 6261 44.0841 220 0.333012 1.4513 569 16 6314 6298 44.2667 148 0.394224 1.44509 570 16 6353 6337 44.4626 156 0.252101 1.43793 571 16 6397 6381 44.693 176 0.37699 1.43139 572 16 6444 6428 44.9434 188 0.248251 1.42368 573 16 6477 6461 45.0953 132 0.352523 1.41844 574 16 6524 6508 45.3442 188 0.165233 1.41114 575 16 6563 6547 45.5366 156 0.42851 1.40486 576 16 6607 6591 45.7631 176 0.332452 1.39812 577 16 6647 6631 45.961 160 0.415296 1.3919 578 16 6687 6671 46.1583 160 0.347753 1.38585 579 16 6727 6711 46.3548 160 0.340362 1.37944 2021-11-02T00:39:07.879406+0900 min lat: 0.0640217 max lat: 19.9062 avg lat: 1.37353 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 580 16 6765 6749 46.5369 152 0.309735 1.37353 581 16 6814 6798 46.7941 196 0.358006 1.36729 582 16 6846 6830 46.9336 128 0.302637 1.36272 583 16 6892 6876 47.1687 184 0.381831 1.3564 584 16 6931 6915 47.355 156 0.405319 1.35066 585 16 6974 6958 47.568 172 0.237625 1.34525 586 16 7017 7001 47.7803 172 0.381257 1.339 587 16 7045 7029 47.8897 112 0.409451 1.33527 588 16 7047 7031 47.8218 8 1.15934 1.33517 589 16 7048 7032 47.7474 4 1.82851 1.33524 590 16 7050 7034 47.6801 8 3.06966 1.33569 591 16 7052 7036 47.6129 8 4.33976 1.33644 592 16 7054 7038 47.546 8 5.22699 1.33747 593 16 7056 7040 47.4793 8 6.27541 1.3388 594 16 7057 7041 47.4061 4 7.03521 1.33961 595 16 7059 7043 47.3399 8 8.03992 1.34144 596 16 7061 7045 47.2739 8 9.22756 1.34358 597 16 7062 7046 47.2014 4 9.36703 1.34472 598 16 7064 7048 47.1358 8 9.49112 1.34704 599 16 7066 7050 47.0705 8 9.45729 1.3493 2021-11-02T00:39:27.882891+0900 min lat: 0.0640217 max lat: 19.9062 avg lat: 1.35044 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 600 16 7067 7051 46.9987 4 9.41901 1.35044 601 14 7067 7053 46.9338 8 9.65188 1.35277 602 12 7067 7055 46.8691 8 9.68742 1.35513 603 11 7067 7056 46.798 4 9.69127 1.35631 604 9 7067 7058 46.7338 8 9.63021 1.35863 605 8 7067 7059 46.6632 4 9.82748 1.35983 606 6 7067 7061 46.5994 8 9.62935 1.3622 607 4 7067 7063 46.5358 8 9.78338 1.36456 608 3 7067 7064 46.4658 4 9.82826 1.36576 609 1 7067 7066 46.4026 8 9.61322 1.36813 Total time run: 609.254 Total writes made: 7067 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 46.3977 Stddev Bandwidth: 69.4197 Max bandwidth (MB/sec): 228 Min bandwidth (MB/sec): 0 Average IOPS: 11 Stddev IOPS: 17.3581 Max IOPS: 57 Min IOPS: 0 Average Latency(s): 1.3693 Stddev Latency(s): 2.79673 Max latency(s): 19.9062 Min latency(s): 0.0640217
読み込み
読み取りはちゃんとOptaneの数値出てますね。10Gネットワークのボトルネックもあると思います
root@pve02:~# rados -p ceph-pool bench 600 seq -t 16
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 266 250 999.062 1000 0.0778427 0.0599475
2 16 551 535 1069.41 1140 0.0583746 0.0577366
3 16 787 771 1027.56 944 0.0159722 0.0592135
4 15 1039 1024 1023.6 1012 0.0216412 0.060204
5 16 1285 1269 1014.85 980 0.147305 0.0607561
6 16 1482 1466 977.024 788 0.0229682 0.0632294
7 16 1740 1724 984.853 1032 0.0217979 0.0636235
8 16 2018 2002 1000.68 1112 0.0176852 0.0627416
9 16 2299 2283 1014.36 1124 0.033461 0.0619284
10 16 2568 2552 1020.51 1076 0.197787 0.0614825
11 16 2767 2751 1000.09 796 0.0443909 0.0628538
12 16 3011 2995 998.07 976 0.129787 0.0628454
13 16 3221 3205 985.901 840 0.196669 0.0637594
14 15 3517 3502 1000.32 1188 0.0356204 0.0629529
15 16 3788 3772 1005.62 1080 0.0187274 0.0625688
16 16 4051 4035 1008.51 1052 0.0702988 0.0623702
17 16 4301 4285 1008 1000 0.082052 0.0624482
18 16 4580 4564 1013.99 1116 0.0229403 0.0620656
19 16 4859 4843 1019.35 1116 0.103958 0.0617847
2021-11-02T00:42:36.401183+0900 min lat: 0.0129594 max lat: 0.4413 avg lat: 0.0615946
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 5130 5114 1022.57 1084 0.0321774 0.0615946
21 15 5412 5397 1027.73 1132 0.020704 0.0612288
22 16 5664 5648 1026.64 1004 0.0189896 0.0612815
23 16 5881 5865 1019.74 868 0.080352 0.0617704
24 16 6133 6117 1019.24 1008 0.184109 0.0617158
25 16 6333 6317 1010.47 800 0.0428111 0.0623864
26 16 6528 6512 1001.6 780 0.0944985 0.0627697
27 16 6768 6752 1000.05 960 0.0146449 0.0630317
28 15 6961 6946 992.033 776 0.0155821 0.063433
Total time run: 28.656
Total reads made: 7067
Read size: 4194304
Object size: 4194304
Bandwidth (MB/sec): 986.459
Average IOPS: 246
Stddev IOPS: 31.2765
Max IOPS: 297
Min IOPS: 194
Average Latency(s): 0.0638671
Max latency(s): 0.451412
Min latency(s): 0.0129135
root@pve02:~#
よ
Proxmoxに立てたWindows10からATTOをかけてみました。VirtIOドライバを当ててあります。
最初ディスクをIDEで作ってしまいひどい結果になってヘコみました。ちゃんとSCSIで立てましょう。ディスクのオプションは何もつけてません。
８MBと12MBはこの箇所だけベンチ取り直したら妥当な数値が出たのと、通しでやると何度やってもこの結果なので連続書き込みにおいて何かボトルネックがあるんでしょう。
安価に小規模にやろうとすると激遅なCephですがこのくらいならまだマシかな？
でも物足りない気がする。vSANに続く
