- simple program (adapted from the Internet) which calculates fibonacci sequence from multiple threads
fib-fast-1thread only runs fib iterationsfib-slow-{2,1000}threads run fun small number of fib iterations and then sleep for 10ms- on exit, program reports total number of iterations/work done in all threads (throughput)
- program is executed in a container with 1 cpu limit on a 32-core machine (unpinned vs pinned)
- results are recorder using linux
perf sched record
(ins)milos@blackbox:~/Projects/cgroups$ sudo perf sched record -o unpinned.perf -- bash ./fib.sh 1 1000 5
Iterations Completed(M): 9555
[ perf record: Woken up 73 times to write data ]
[ perf record: Captured and wrote 190,689 MB unpinned.perf (1757059 samples) ]
(ins)milos@blackbox:~/Projects/cgroups$ sudo perf sched record -o pinned.perf -- bash ./fib.sh 1 1000 5 p
Iterations Completed(M): 19437
[ perf record: Woken up 109 times to write data ]
[ perf record: Captured and wrote 228,440 MB pinned.perf (2118203 samples) ]Remarks: pinned version has ~2x throughput
(ins)milos@blackbox:~/Projects/cgroups$ sudo perf sched latency -i unpinned.perf
-------------------------------------------------------------------------------------------------------------------------------------------
Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Max delay start | Max delay end |
-------------------------------------------------------------------------------------------------------------------------------------------
fib-fast-1:69590 | 2319.276 ms | 489 | avg: 5.483 ms | max: 59.185 ms | max start: 7797.286383 s | max end: 7797.345568 s
...(ins)milos@blackbox:~/Projects/cgroups$ sudo perf sched latency -i pinned.perf
-------------------------------------------------------------------------------------------------------------------------------------------
Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Max delay start | Max delay end |
-------------------------------------------------------------------------------------------------------------------------------------------
fib-fast-1:70932 | 4199.866 ms | 20136 | avg: 0.040 ms | max: 17.618 ms | max start: 7813.101741 s | max end: 7813.119359 s
...(ins)milos@blackbox:~/Projects/cgroups$ sudo perf sched timehist -s -t 69590 -i unpinned.perf
Samples do not have callchains.
Runtime summary
comm parent sched-in run-time min-run avg-run max-run stddev migrations
(count) (msec) (msec) (msec) (msec) %
---------------------------------------------------------------------------------------------------------------------
<no still running tasks>
Terminated tasks:
fib-fast-1[69590/69534] 69534 488 2313.004 0.003 4.739 12.001 4.08 0(ins)milos@blackbox:~/Projects/cgroups$ sudo perf sched timehist -s -t 70932 -i pinned.perf
Samples do not have callchains.
Runtime summary
comm parent sched-in run-time min-run avg-run max-run stddev migrations
(count) (msec) (msec) (msec) (msec) %
---------------------------------------------------------------------------------------------------------------------
<no still running tasks>
Terminated tasks:
fib-fast-1[70932/70876] 70876 20138 4204.286 0.001 0.208 4.915 1.62 0(ins)milos@blackbox:~/Projects/cgroups$ sudo perf sched timehist -t 69590 -i unpinned.perf
time cpu task name wait time sch delay run time
[tid/pid] (msec) (msec) (msec)
--------------- ------ ------------------------------ --------- --------- ---------
7797.090383 [0028] fib-fast-1[69590/69534] 0.000 0.005 8.502
7797.102378 [0028] fib-fast-1[69590/69534] 0.004 0.000 11.991
7797.114378 [0028] fib-fast-1[69590/69534] 0.001 0.000 11.997
7797.122379 [0028] fib-fast-1[69590/69534] 0.001 0.000 8.000
7797.146379 [0028] fib-fast-1[69590/69534] 22.841 0.000 1.158
7797.158378 [0028] fib-fast-1[69590/69534] 0.001 0.000 11.997
7797.166379 [0028] fib-fast-1[69590/69534] 0.001 0.000 7.999
7797.174378 [0028] fib-fast-1[69590/69534] 0.001 0.000 7.998
7797.182380 [0028] fib-fast-1[69590/69534] 0.001 0.000 8.000
7797.194378 [0028] fib-fast-1[69590/69534] 0.001 0.000 11.996
7797.202379 [0028] fib-fast-1[69590/69534] 0.001 0.000 7.999
7797.206380 [0028] fib-fast-1[69590/69534] 0.001 0.000 3.999
7797.246384 [0028] fib-fast-1[69590/69534] 38.886 0.000 1.117
7797.255319 [0028] fib-fast-1[69590/69534] 0.002 0.000 8.932
7797.265367 [0012] fib-fast-1[69590/69534] 0.022 0.000 10.025
7797.274379 [0012] fib-fast-1[69590/69534] 0.009 0.000 9.002
7797.282381 [0012] fib-fast-1[69590/69534] 0.001 0.000 8.001
7797.282397 [0012] fib-fast-1[69590/69534] 0.001 0.000 0.015
7797.286383 [0012] fib-fast-1[69590/69534] 0.003 0.000 3.982
7797.354387 [0026] fib-fast-1[69590/69534] 59.185 0.000 8.818
7797.355327 [0026] fib-fast-1[69590/69534] 0.007 0.000 0.932
7797.355426 [0026] fib-fast-1[69590/69534] 0.041 0.000 0.056
7797.365417 [0026] fib-fast-1[69590/69534] 0.028 0.000 9.963
7797.374381 [0026] fib-fast-1[69590/69534] 0.005 0.000 8.957
7797.375476 [0026] fib-fast-1[69590/69534] 0.001 0.000 1.094
7797.382380 [0026] fib-fast-1[69590/69534] 0.002 0.000 6.901
7797.390380 [0026] fib-fast-1[69590/69534] 0.001 0.000 7.998
7797.446387 [0026] fib-fast-1[69590/69534] 55.047 0.000 0.959
7797.455330 [0026] fib-fast-1[69590/69534] 0.003 0.000 8.938
7797.458382 [0010] fib-fast-1[69590/69534] 0.044 0.000 3.008
7797.470381 [0010] fib-fast-1[69590/69534] 0.001 0.000 11.996
7797.478382 [0010] fib-fast-1[69590/69534] 0.001 0.000 8.000
7797.490382 [0010] fib-fast-1[69590/69534] 0.001 0.000 11.998
7797.554387 [0011] fib-fast-1[69590/69534] 55.027 0.000 8.977
7797.566387 [0027] fib-fast-1[69590/69534] 0.008 0.000 11.991
7797.578380 [0027] fib-fast-1[69590/69534] 0.004 0.000 11.988
7797.586382 [0027] fib-fast-1[69590/69534] 0.001 0.000 8.000
7797.590379 [0027] fib-fast-1[69590/69534] 0.001 0.000 3.996
7797.646389 [0027] fib-fast-1[69590/69534] 55.066 0.000 0.944
7797.655628 [0027] fib-fast-1[69590/69534] 0.004 0.000 9.233
7797.665428 [0011] fib-fast-1[69590/69534] 0.012 0.000 9.787
7797.674380 [0027] fib-fast-1[69590/69534] 0.007 0.000 8.944
7797.675427 [0027] fib-fast-1[69590/69534] 0.001 0.000 1.045
7797.682381 [0027] fib-fast-1[69590/69534] 0.003 0.000 6.949
7797.685483 [0027] fib-fast-1[69590/69534] 0.001 0.000 3.100
7797.694381 [0027] fib-fast-1[69590/69534] 0.002 0.000 8.895
7797.746392 [0027] fib-fast-1[69590/69534] 51.042 0.000 0.968
7797.754383 [0011] fib-fast-1[69590/69534] 0.010 0.000 7.980
7797.766384 [0011] fib-fast-1[69590/69534] 0.003 0.000 11.997
7797.774381 [0011] fib-fast-1[69590/69534] 0.001 0.000 7.996
7797.786384 [0011] fib-fast-1[69590/69534] 0.001 0.000 12.001
7797.790381 [0011] fib-fast-1[69590/69534] 0.001 0.000 3.995
7797.854385 [0029] fib-fast-1[69590/69534] 55.106 0.000 8.898
7797.855340 [0029] fib-fast-1[69590/69534] 0.005 0.000 0.949(ins)milos@blackbox:~/Projects/cgroups$ sudo perf sched timehist -t 70932 -i pinned.perf
time cpu task name wait time sch delay run time
[tid/pid] (msec) (msec) (msec)
--------------- ------ ------------------------------ --------- --------- ---------
7813.120578 [0001] fib-fast-1[70932/70876] 0.000 17.617 1.219
7813.120634 [0001] fib-fast-1[70932/70876] 0.015 0.000 0.040
7813.120684 [0001] fib-fast-1[70932/70876] 0.025 0.000 0.024
7813.120735 [0001] fib-fast-1[70932/70876] 0.029 0.000 0.021
7813.120788 [0001] fib-fast-1[70932/70876] 0.030 0.000 0.021
7813.120840 [0001] fib-fast-1[70932/70876] 0.032 0.000 0.019
7813.120895 [0001] fib-fast-1[70932/70876] 0.032 0.000 0.022
7813.120949 [0001] fib-fast-1[70932/70876] 0.031 0.000 0.022
7813.121003 [0001] fib-fast-1[70932/70876] 0.033 0.000 0.020
7813.121056 [0001] fib-fast-1[70932/70876] 0.032 0.000 0.021
7813.121110 [0001] fib-fast-1[70932/70876] 0.029 0.000 0.024
7813.121164 [0001] fib-fast-1[70932/70876] 0.031 0.000 0.022
7813.121217 [0001] fib-fast-1[70932/70876] 0.034 0.000 0.018
7813.121267 [0001] fib-fast-1[70932/70876] 0.034 0.000 0.014
7813.121321 [0001] fib-fast-1[70932/70876] 0.026 0.000 0.027
7813.121373 [0001] fib-fast-1[70932/70876] 0.033 0.000 0.018
7813.121424 [0001] fib-fast-1[70932/70876] 0.029 0.000 0.021
7813.121479 [0001] fib-fast-1[70932/70876] 0.033 0.000 0.020
7813.121529 [0001] fib-fast-1[70932/70876] 0.031 0.000 0.018
7813.121580 [0001] fib-fast-1[70932/70876] 0.029 0.000 0.021
7813.121633 [0001] fib-fast-1[70932/70876] 0.029 0.000 0.023
7813.121682 [0001] fib-fast-1[70932/70876] 0.031 0.000 0.018
7813.121721 [0001] fib-fast-1[70932/70876] 0.022 0.000 0.016
7813.126511 [0001] fib-fast-1[70932/70876] 0.024 0.000 4.765
7813.136579 [0001] fib-fast-1[70932/70876] 5.686 0.000 4.381
7813.139385 [0001] fib-fast-1[70932/70876] 2.364 0.000 0.441
7813.139441 [0001] fib-fast-1[70932/70876] 0.034 0.000 0.021
7813.140587 [0001] fib-fast-1[70932/70876] 0.045 0.000 1.100
7813.140697 [0001] fib-fast-1[70932/70876] 0.002 0.000 0.106
7813.140757 [0001] fib-fast-1[70932/70876] 0.014 0.000 0.046
7813.140811 [0001] fib-fast-1[70932/70876] 0.023 0.000 0.029
7813.140863 [0001] fib-fast-1[70932/70876] 0.027 0.000 0.024
7813.140915 [0001] fib-fast-1[70932/70876] 0.027 0.000 0.024
7813.140965 [0001] fib-fast-1[70932/70876] 0.029 0.000 0.021
7813.141021 [0001] fib-fast-1[70932/70876] 0.026 0.000 0.029
7813.141074 [0001] fib-fast-1[70932/70876] 0.029 0.000 0.022
7813.141129 [0001] fib-fast-1[70932/70876] 0.029 0.000 0.025
7813.141182 [0001] fib-fast-1[70932/70876] 0.029 0.000 0.022
7813.141237 [0001] fib-fast-1[70932/70876] 0.027 0.000 0.028
7813.141295 [0001] fib-fast-1[70932/70876] 0.028 0.000 0.029
7813.141349 [0001] fib-fast-1[70932/70876] 0.028 0.000 0.024
7813.141393 [0001] fib-fast-1[70932/70876] 0.033 0.000 0.010
7813.141451 [0001] fib-fast-1[70932/70876] 0.018 0.000 0.039
7813.141500 [0001] fib-fast-1[70932/70876] 0.032 0.000 0.016
7813.141551 [0001] fib-fast-1[70932/70876] 0.024 0.000 0.027
7813.141607 [0001] fib-fast-1[70932/70876] 0.028 0.000 0.027
7813.141656 [0001] fib-fast-1[70932/70876] 0.033 0.000 0.015
7813.141708 [0001] fib-fast-1[70932/70876] 0.025 0.000 0.026
7813.141763 [0001] fib-fast-1[70932/70876] 0.026 0.000 0.028
7813.141819 [0001] fib-fast-1[70932/70876] 0.030 0.000 0.024
7813.146511 [0001] fib-fast-1[70932/70876] 0.040 0.000 4.652
7813.146646 [0001] fib-fast-1[70932/70876] 0.001 0.000 0.133
7813.146698 [0001] fib-fast-1[70932/70876] 0.037 0.000 0.015
7813.146750 [0001] fib-fast-1[70932/70876] 0.034 0.000 0.017
7813.146802 [0001] fib-fast-1[70932/70876] 0.034 0.000 0.017
7813.146854 [0001] fib-fast-1[70932/70876] 0.033 0.000 0.018
7813.146906 [0001] fib-fast-1[70932/70876] 0.035 0.000 0.015
7813.146958 [0001] fib-fast-1[70932/70876] 0.036 0.000 0.015
7813.147009 [0001] fib-fast-1[70932/70876] 0.034 0.000 0.016
7813.147059 [0001] fib-fast-1[70932/70876] 0.032 0.000 0.018
7813.147115 [0001] fib-fast-1[70932/70876] 0.035 0.000 0.020
7813.147169 [0001] fib-fast-1[70932/70876] 0.038 0.000 0.016
7813.147225 [0001] fib-fast-1[70932/70876] 0.037 0.000 0.017
7813.147276 [0001] fib-fast-1[70932/70876] 0.035 0.000 0.016
7813.147325 [0001] fib-fast-1[70932/70876] 0.035 0.000 0.013
7813.147377 [0001] fib-fast-1[70932/70876] 0.032 0.000 0.018
7813.147430 [0001] fib-fast-1[70932/70876] 0.033 0.000 0.019
7813.147483 [0001] fib-fast-1[70932/70876] 0.032 0.000 0.020
7813.147533 [0001] fib-fast-1[70932/70876] 0.034 0.000 0.015
7813.147588 [0001] fib-fast-1[70932/70876] 0.037 0.000 0.017
7813.147639 [0001] fib-fast-1[70932/70876] 0.033 0.000 0.018
7813.147691 [0001] fib-fast-1[70932/70876] 0.031 0.000 0.019