Skip to content

Instantly share code, notes, and snippets.

@bobrik

bobrik/README.md

Last active November 2, 2024 08:59
Show Gist options
  • Save bobrik/2030ff040fad360327a5fab7a09c4ff1 to your computer and use it in GitHub Desktop.
Save bobrik/2030ff040fad360327a5fab7a09c4ff1 to your computer and use it in GitHub Desktop.
Download ZIP
CFS hiccups
Raw
README.md

Overly aggressive CFS

We're looking at CPU bandwidth control via CFS:

Reproduction program

Program does number of iterations, in each iteration we burn CPU in small chunks until we get 5ms of real time spent. On each iteration we also print how much time we spent burning CPU, how much real time passed and how much CPU time.

If we don't go over CFS quota, we should always get very close to 5ms.

If we go over quota, it's fair to expect to have higher real timings, while on-CPU time should remain around 5ms.

To make tests easier, we run code in a docker container. Code is in cfs.go.

We tested:

  • Kernel 4.14.4, 4.9.51 and 4.4.88 on bare metal with NUMA
  • Xubuntu 17.04 with kernel 4.10.0-19-generic on VirtualBox
  • Ubuntu 16.04 with kernel 4.4.0-101-generic on DigitalOcean
  • Fedora 27 with kernel 4.13.9-300.fc27.x86_64 on DigitalOcean

All with the same results.

Running without throttling

100ms sleep between iterations

As expected, nothing is throttled.

$ docker run --rm -it -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 100 -sleep 100ms
2017/12/08 01:39:54 [0] burn took 5ms, real time so far: 5ms, cpu time so far: 6ms
2017/12/08 01:39:54 [1] burn took 5ms, real time so far: 110ms, cpu time so far: 11ms
2017/12/08 01:39:55 [2] burn took 5ms, real time so far: 215ms, cpu time so far: 17ms
2017/12/08 01:39:55 [3] burn took 5ms, real time so far: 320ms, cpu time so far: 23ms
2017/12/08 01:39:55 [4] burn took 5ms, real time so far: 425ms, cpu time so far: 29ms
2017/12/08 01:39:55 [5] burn took 5ms, real time so far: 530ms, cpu time so far: 35ms
2017/12/08 01:39:55 [6] burn took 5ms, real time so far: 635ms, cpu time so far: 40ms
2017/12/08 01:39:55 [7] burn took 5ms, real time so far: 741ms, cpu time so far: 46ms
2017/12/08 01:39:55 [8] burn took 5ms, real time so far: 846ms, cpu time so far: 52ms
2017/12/08 01:39:55 [9] burn took 5ms, real time so far: 951ms, cpu time so far: 57ms
2017/12/08 01:39:55 [10] burn took 5ms, real time so far: 1056ms, cpu time so far: 63ms
2017/12/08 01:39:55 [11] burn took 5ms, real time so far: 1161ms, cpu time so far: 68ms
2017/12/08 01:39:56 [12] burn took 5ms, real time so far: 1266ms, cpu time so far: 75ms
2017/12/08 01:39:56 [13] burn took 5ms, real time so far: 1371ms, cpu time so far: 80ms
2017/12/08 01:39:56 [14] burn took 5ms, real time so far: 1476ms, cpu time so far: 86ms
2017/12/08 01:39:56 [15] burn took 5ms, real time so far: 1582ms, cpu time so far: 92ms
2017/12/08 01:39:56 [16] burn took 5ms, real time so far: 1687ms, cpu time so far: 98ms
2017/12/08 01:39:56 [17] burn took 5ms, real time so far: 1792ms, cpu time so far: 104ms
2017/12/08 01:39:56 [18] burn took 5ms, real time so far: 1897ms, cpu time so far: 109ms
2017/12/08 01:39:56 [19] burn took 5ms, real time so far: 2002ms, cpu time so far: 116ms
2017/12/08 01:39:56 [20] burn took 5ms, real time so far: 2107ms, cpu time so far: 121ms
2017/12/08 01:39:57 [21] burn took 5ms, real time so far: 2212ms, cpu time so far: 127ms
2017/12/08 01:39:57 [22] burn took 5ms, real time so far: 2317ms, cpu time so far: 134ms
2017/12/08 01:39:57 [23] burn took 5ms, real time so far: 2423ms, cpu time so far: 139ms
2017/12/08 01:39:57 [24] burn took 5ms, real time so far: 2528ms, cpu time so far: 145ms
2017/12/08 01:39:57 [25] burn took 5ms, real time so far: 2633ms, cpu time so far: 150ms
2017/12/08 01:39:57 [26] burn took 5ms, real time so far: 2738ms, cpu time so far: 157ms
2017/12/08 01:39:57 [27] burn took 5ms, real time so far: 2843ms, cpu time so far: 162ms
2017/12/08 01:39:57 [28] burn took 5ms, real time so far: 2948ms, cpu time so far: 169ms
2017/12/08 01:39:57 [29] burn took 5ms, real time so far: 3053ms, cpu time so far: 175ms
2017/12/08 01:39:57 [30] burn took 5ms, real time so far: 3158ms, cpu time so far: 180ms
2017/12/08 01:39:58 [31] burn took 5ms, real time so far: 3264ms, cpu time so far: 187ms
2017/12/08 01:39:58 [32] burn took 5ms, real time so far: 3369ms, cpu time so far: 192ms
2017/12/08 01:39:58 [33] burn took 5ms, real time so far: 3474ms, cpu time so far: 198ms
2017/12/08 01:39:58 [34] burn took 5ms, real time so far: 3579ms, cpu time so far: 203ms
2017/12/08 01:39:58 [35] burn took 5ms, real time so far: 3684ms, cpu time so far: 210ms
2017/12/08 01:39:58 [36] burn took 5ms, real time so far: 3789ms, cpu time so far: 215ms
2017/12/08 01:39:58 [37] burn took 5ms, real time so far: 3894ms, cpu time so far: 221ms
2017/12/08 01:39:58 [38] burn took 5ms, real time so far: 3999ms, cpu time so far: 227ms
2017/12/08 01:39:58 [39] burn took 5ms, real time so far: 4104ms, cpu time so far: 233ms
2017/12/08 01:39:59 [40] burn took 5ms, real time so far: 4209ms, cpu time so far: 239ms
2017/12/08 01:39:59 [41] burn took 5ms, real time so far: 4315ms, cpu time so far: 244ms
2017/12/08 01:39:59 [42] burn took 5ms, real time so far: 4420ms, cpu time so far: 250ms
2017/12/08 01:39:59 [43] burn took 5ms, real time so far: 4525ms, cpu time so far: 255ms
2017/12/08 01:39:59 [44] burn took 5ms, real time so far: 4630ms, cpu time so far: 262ms
2017/12/08 01:39:59 [45] burn took 5ms, real time so far: 4735ms, cpu time so far: 268ms
2017/12/08 01:39:59 [46] burn took 5ms, real time so far: 4840ms, cpu time so far: 273ms
2017/12/08 01:39:59 [47] burn took 5ms, real time so far: 4945ms, cpu time so far: 279ms
2017/12/08 01:39:59 [48] burn took 5ms, real time so far: 5050ms, cpu time so far: 285ms
2017/12/08 01:39:59 [49] burn took 5ms, real time so far: 5155ms, cpu time so far: 291ms
2017/12/08 01:40:00 [50] burn took 5ms, real time so far: 5261ms, cpu time so far: 296ms
2017/12/08 01:40:00 [51] burn took 5ms, real time so far: 5366ms, cpu time so far: 302ms
2017/12/08 01:40:00 [52] burn took 5ms, real time so far: 5471ms, cpu time so far: 307ms
2017/12/08 01:40:00 [53] burn took 5ms, real time so far: 5576ms, cpu time so far: 314ms
2017/12/08 01:40:00 [54] burn took 5ms, real time so far: 5681ms, cpu time so far: 320ms
2017/12/08 01:40:00 [55] burn took 5ms, real time so far: 5786ms, cpu time so far: 325ms
2017/12/08 01:40:00 [56] burn took 5ms, real time so far: 5891ms, cpu time so far: 332ms
2017/12/08 01:40:00 [57] burn took 5ms, real time so far: 5996ms, cpu time so far: 337ms
2017/12/08 01:40:00 [58] burn took 5ms, real time so far: 6101ms, cpu time so far: 343ms
2017/12/08 01:40:01 [59] burn took 5ms, real time so far: 6207ms, cpu time so far: 349ms
2017/12/08 01:40:01 [60] burn took 5ms, real time so far: 6312ms, cpu time so far: 355ms
2017/12/08 01:40:01 [61] burn took 5ms, real time so far: 6417ms, cpu time so far: 361ms
2017/12/08 01:40:01 [62] burn took 5ms, real time so far: 6522ms, cpu time so far: 366ms
2017/12/08 01:40:01 [63] burn took 5ms, real time so far: 6627ms, cpu time so far: 372ms
2017/12/08 01:40:01 [64] burn took 5ms, real time so far: 6732ms, cpu time so far: 377ms
2017/12/08 01:40:01 [65] burn took 5ms, real time so far: 6837ms, cpu time so far: 384ms
2017/12/08 01:40:01 [66] burn took 5ms, real time so far: 6942ms, cpu time so far: 389ms
2017/12/08 01:40:01 [67] burn took 5ms, real time so far: 7048ms, cpu time so far: 396ms
2017/12/08 01:40:01 [68] burn took 5ms, real time so far: 7153ms, cpu time so far: 401ms
2017/12/08 01:40:02 [69] burn took 5ms, real time so far: 7258ms, cpu time so far: 408ms
2017/12/08 01:40:02 [70] burn took 5ms, real time so far: 7363ms, cpu time so far: 414ms
2017/12/08 01:40:02 [71] burn took 5ms, real time so far: 7468ms, cpu time so far: 419ms
2017/12/08 01:40:02 [72] burn took 5ms, real time so far: 7573ms, cpu time so far: 426ms
2017/12/08 01:40:02 [73] burn took 5ms, real time so far: 7678ms, cpu time so far: 431ms
2017/12/08 01:40:02 [74] burn took 5ms, real time so far: 7783ms, cpu time so far: 437ms
2017/12/08 01:40:02 [75] burn took 5ms, real time so far: 7889ms, cpu time so far: 442ms
2017/12/08 01:40:02 [76] burn took 5ms, real time so far: 7994ms, cpu time so far: 449ms
2017/12/08 01:40:02 [77] burn took 5ms, real time so far: 8099ms, cpu time so far: 455ms
2017/12/08 01:40:03 [78] burn took 5ms, real time so far: 8204ms, cpu time so far: 460ms
2017/12/08 01:40:03 [79] burn took 5ms, real time so far: 8309ms, cpu time so far: 466ms
2017/12/08 01:40:03 [80] burn took 5ms, real time so far: 8414ms, cpu time so far: 471ms
2017/12/08 01:40:03 [81] burn took 5ms, real time so far: 8519ms, cpu time so far: 478ms
2017/12/08 01:40:03 [82] burn took 5ms, real time so far: 8624ms, cpu time so far: 483ms
2017/12/08 01:40:03 [83] burn took 5ms, real time so far: 8730ms, cpu time so far: 489ms
2017/12/08 01:40:03 [84] burn took 5ms, real time so far: 8835ms, cpu time so far: 495ms
2017/12/08 01:40:03 [85] burn took 5ms, real time so far: 8940ms, cpu time so far: 501ms
2017/12/08 01:40:03 [86] burn took 5ms, real time so far: 9045ms, cpu time so far: 507ms
2017/12/08 01:40:03 [87] burn took 5ms, real time so far: 9150ms, cpu time so far: 512ms
2017/12/08 01:40:04 [88] burn took 5ms, real time so far: 9255ms, cpu time so far: 518ms
2017/12/08 01:40:04 [89] burn took 5ms, real time so far: 9360ms, cpu time so far: 523ms
2017/12/08 01:40:04 [90] burn took 5ms, real time so far: 9465ms, cpu time so far: 529ms
2017/12/08 01:40:04 [91] burn took 5ms, real time so far: 9570ms, cpu time so far: 535ms
2017/12/08 01:40:04 [92] burn took 5ms, real time so far: 9675ms, cpu time so far: 541ms
2017/12/08 01:40:04 [93] burn took 5ms, real time so far: 9781ms, cpu time so far: 547ms
2017/12/08 01:40:04 [94] burn took 5ms, real time so far: 9886ms, cpu time so far: 552ms
2017/12/08 01:40:04 [95] burn took 5ms, real time so far: 9991ms, cpu time so far: 559ms
2017/12/08 01:40:04 [96] burn took 5ms, real time so far: 10096ms, cpu time so far: 564ms
2017/12/08 01:40:05 [97] burn took 5ms, real time so far: 10201ms, cpu time so far: 570ms
2017/12/08 01:40:05 [98] burn took 5ms, real time so far: 10306ms, cpu time so far: 575ms
2017/12/08 01:40:05 [99] burn took 5ms, real time so far: 10411ms, cpu time so far: 582ms

1000ms sleep between iterations

Same, no throttling here.

$ docker run --rm -it -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 100 -sleep 1000ms
2017/12/08 01:40:42 [0] burn took 5ms, real time so far: 5ms, cpu time so far: 6ms
2017/12/08 01:40:43 [1] burn took 5ms, real time so far: 1010ms, cpu time so far: 12ms
2017/12/08 01:40:44 [2] burn took 5ms, real time so far: 2015ms, cpu time so far: 18ms
2017/12/08 01:40:45 [3] burn took 5ms, real time so far: 3020ms, cpu time so far: 24ms
2017/12/08 01:40:46 [4] burn took 5ms, real time so far: 4026ms, cpu time so far: 30ms
2017/12/08 01:40:47 [5] burn took 5ms, real time so far: 5031ms, cpu time so far: 36ms
2017/12/08 01:40:48 [6] burn took 5ms, real time so far: 6036ms, cpu time so far: 41ms
2017/12/08 01:40:49 [7] burn took 5ms, real time so far: 7041ms, cpu time so far: 47ms
2017/12/08 01:40:50 [8] burn took 5ms, real time so far: 8046ms, cpu time so far: 53ms
2017/12/08 01:40:51 [9] burn took 5ms, real time so far: 9052ms, cpu time so far: 59ms
2017/12/08 01:40:52 [10] burn took 5ms, real time so far: 10057ms, cpu time so far: 65ms
2017/12/08 01:40:53 [11] burn took 5ms, real time so far: 11062ms, cpu time so far: 71ms
2017/12/08 01:40:54 [12] burn took 5ms, real time so far: 12067ms, cpu time so far: 78ms
2017/12/08 01:40:55 [13] burn took 5ms, real time so far: 13072ms, cpu time so far: 83ms
2017/12/08 01:40:56 [14] burn took 5ms, real time so far: 14077ms, cpu time so far: 90ms
2017/12/08 01:40:57 [15] burn took 5ms, real time so far: 15083ms, cpu time so far: 95ms
2017/12/08 01:40:58 [16] burn took 5ms, real time so far: 16088ms, cpu time so far: 101ms
2017/12/08 01:40:59 [17] burn took 5ms, real time so far: 17093ms, cpu time so far: 107ms
2017/12/08 01:41:00 [18] burn took 5ms, real time so far: 18098ms, cpu time so far: 113ms
2017/12/08 01:41:01 [19] burn took 5ms, real time so far: 19103ms, cpu time so far: 119ms
2017/12/08 01:41:02 [20] burn took 5ms, real time so far: 20108ms, cpu time so far: 125ms
2017/12/08 01:41:03 [21] burn took 5ms, real time so far: 21113ms, cpu time so far: 130ms
2017/12/08 01:41:04 [22] burn took 5ms, real time so far: 22119ms, cpu time so far: 137ms
2017/12/08 01:41:05 [23] burn took 5ms, real time so far: 23124ms, cpu time so far: 143ms
2017/12/08 01:41:06 [24] burn took 5ms, real time so far: 24129ms, cpu time so far: 149ms
2017/12/08 01:41:07 [25] burn took 5ms, real time so far: 25134ms, cpu time so far: 156ms
2017/12/08 01:41:08 [26] burn took 5ms, real time so far: 26139ms, cpu time so far: 161ms
2017/12/08 01:41:09 [27] burn took 5ms, real time so far: 27144ms, cpu time so far: 167ms
2017/12/08 01:41:10 [28] burn took 5ms, real time so far: 28150ms, cpu time so far: 172ms
2017/12/08 01:41:11 [29] burn took 5ms, real time so far: 29155ms, cpu time so far: 179ms
2017/12/08 01:41:12 [30] burn took 5ms, real time so far: 30160ms, cpu time so far: 184ms
2017/12/08 01:41:13 [31] burn took 5ms, real time so far: 31165ms, cpu time so far: 191ms
2017/12/08 01:41:14 [32] burn took 5ms, real time so far: 32170ms, cpu time so far: 197ms
2017/12/08 01:41:15 [33] burn took 5ms, real time so far: 33175ms, cpu time so far: 203ms
2017/12/08 01:41:16 [34] burn took 5ms, real time so far: 34181ms, cpu time so far: 209ms
2017/12/08 01:41:17 [35] burn took 5ms, real time so far: 35186ms, cpu time so far: 214ms
2017/12/08 01:41:18 [36] burn took 5ms, real time so far: 36191ms, cpu time so far: 221ms
2017/12/08 01:41:19 [37] burn took 5ms, real time so far: 37196ms, cpu time so far: 226ms
2017/12/08 01:41:20 [38] burn took 5ms, real time so far: 38201ms, cpu time so far: 233ms
2017/12/08 01:41:21 [39] burn took 5ms, real time so far: 39206ms, cpu time so far: 238ms
2017/12/08 01:41:22 [40] burn took 5ms, real time so far: 40212ms, cpu time so far: 244ms
2017/12/08 01:41:23 [41] burn took 5ms, real time so far: 41217ms, cpu time so far: 251ms
2017/12/08 01:41:24 [42] burn took 5ms, real time so far: 42222ms, cpu time so far: 256ms
2017/12/08 01:41:25 [43] burn took 5ms, real time so far: 43227ms, cpu time so far: 263ms
2017/12/08 01:41:26 [44] burn took 5ms, real time so far: 44232ms, cpu time so far: 268ms
2017/12/08 01:41:27 [45] burn took 5ms, real time so far: 45237ms, cpu time so far: 274ms
2017/12/08 01:41:28 [46] burn took 5ms, real time so far: 46242ms, cpu time so far: 281ms
2017/12/08 01:41:29 [47] burn took 5ms, real time so far: 47248ms, cpu time so far: 286ms
2017/12/08 01:41:30 [48] burn took 5ms, real time so far: 48253ms, cpu time so far: 293ms
2017/12/08 01:41:31 [49] burn took 5ms, real time so far: 49258ms, cpu time so far: 298ms
2017/12/08 01:41:32 [50] burn took 5ms, real time so far: 50263ms, cpu time so far: 305ms
2017/12/08 01:41:33 [51] burn took 5ms, real time so far: 51268ms, cpu time so far: 310ms
2017/12/08 01:41:34 [52] burn took 5ms, real time so far: 52274ms, cpu time so far: 317ms
2017/12/08 01:41:35 [53] burn took 5ms, real time so far: 53279ms, cpu time so far: 324ms
2017/12/08 01:41:36 [54] burn took 5ms, real time so far: 54284ms, cpu time so far: 329ms
2017/12/08 01:41:37 [55] burn took 5ms, real time so far: 55289ms, cpu time so far: 336ms
2017/12/08 01:41:38 [56] burn took 5ms, real time so far: 56294ms, cpu time so far: 341ms
2017/12/08 01:41:39 [57] burn took 5ms, real time so far: 57299ms, cpu time so far: 347ms
2017/12/08 01:41:40 [58] burn took 5ms, real time so far: 58304ms, cpu time so far: 353ms
2017/12/08 01:41:41 [59] burn took 5ms, real time so far: 59310ms, cpu time so far: 359ms
2017/12/08 01:41:42 [60] burn took 5ms, real time so far: 60315ms, cpu time so far: 365ms
2017/12/08 01:41:43 [61] burn took 5ms, real time so far: 61320ms, cpu time so far: 371ms
2017/12/08 01:41:44 [62] burn took 5ms, real time so far: 62325ms, cpu time so far: 377ms
2017/12/08 01:41:45 [63] burn took 5ms, real time so far: 63330ms, cpu time so far: 382ms
2017/12/08 01:41:46 [64] burn took 5ms, real time so far: 64335ms, cpu time so far: 389ms
2017/12/08 01:41:47 [65] burn took 5ms, real time so far: 65341ms, cpu time so far: 394ms
2017/12/08 01:41:48 [66] burn took 5ms, real time so far: 66346ms, cpu time so far: 401ms
2017/12/08 01:41:49 [67] burn took 5ms, real time so far: 67351ms, cpu time so far: 408ms
2017/12/08 01:41:50 [68] burn took 5ms, real time so far: 68356ms, cpu time so far: 413ms
2017/12/08 01:41:51 [69] burn took 5ms, real time so far: 69361ms, cpu time so far: 420ms
2017/12/08 01:41:52 [70] burn took 5ms, real time so far: 70366ms, cpu time so far: 425ms
2017/12/08 01:41:53 [71] burn took 5ms, real time so far: 71371ms, cpu time so far: 432ms
2017/12/08 01:41:54 [72] burn took 5ms, real time so far: 72377ms, cpu time so far: 438ms
2017/12/08 01:41:55 [73] burn took 5ms, real time so far: 73382ms, cpu time so far: 444ms
2017/12/08 01:41:56 [74] burn took 5ms, real time so far: 74387ms, cpu time so far: 450ms
2017/12/08 01:41:57 [75] burn took 5ms, real time so far: 75392ms, cpu time so far: 456ms
2017/12/08 01:41:58 [76] burn took 5ms, real time so far: 76397ms, cpu time so far: 462ms
2017/12/08 01:41:59 [77] burn took 5ms, real time so far: 77403ms, cpu time so far: 467ms
2017/12/08 01:42:00 [78] burn took 5ms, real time so far: 78408ms, cpu time so far: 474ms
2017/12/08 01:42:01 [79] burn took 5ms, real time so far: 79413ms, cpu time so far: 479ms
2017/12/08 01:42:02 [80] burn took 5ms, real time so far: 80418ms, cpu time so far: 486ms
2017/12/08 01:42:03 [81] burn took 5ms, real time so far: 81423ms, cpu time so far: 492ms
2017/12/08 01:42:04 [82] burn took 5ms, real time so far: 82428ms, cpu time so far: 498ms
2017/12/08 01:42:05 [83] burn took 5ms, real time so far: 83434ms, cpu time so far: 505ms
2017/12/08 01:42:06 [84] burn took 5ms, real time so far: 84439ms, cpu time so far: 510ms
2017/12/08 01:42:07 [85] burn took 5ms, real time so far: 85444ms, cpu time so far: 516ms
2017/12/08 01:42:08 [86] burn took 5ms, real time so far: 86449ms, cpu time so far: 522ms
2017/12/08 01:42:09 [87] burn took 5ms, real time so far: 87454ms, cpu time so far: 528ms
2017/12/08 01:42:10 [88] burn took 5ms, real time so far: 88460ms, cpu time so far: 535ms
2017/12/08 01:42:11 [89] burn took 5ms, real time so far: 89465ms, cpu time so far: 540ms
2017/12/08 01:42:12 [90] burn took 5ms, real time so far: 90470ms, cpu time so far: 547ms
2017/12/08 01:42:13 [91] burn took 5ms, real time so far: 91475ms, cpu time so far: 552ms
2017/12/08 01:42:14 [92] burn took 5ms, real time so far: 92480ms, cpu time so far: 559ms
2017/12/08 01:42:15 [93] burn took 5ms, real time so far: 93485ms, cpu time so far: 564ms
2017/12/08 01:42:16 [94] burn took 5ms, real time so far: 94490ms, cpu time so far: 570ms
2017/12/08 01:42:17 [95] burn took 5ms, real time so far: 95496ms, cpu time so far: 576ms
2017/12/08 01:42:18 [96] burn took 5ms, real time so far: 96501ms, cpu time so far: 582ms
2017/12/08 01:42:19 [97] burn took 5ms, real time so far: 97506ms, cpu time so far: 588ms
2017/12/08 01:42:20 [98] burn took 5ms, real time so far: 98511ms, cpu time so far: 594ms
2017/12/08 01:42:21 [99] burn took 5ms, real time so far: 99516ms, cpu time so far: 600ms

Running with throttling

This is where things get interesting. We set cfs quota to 20ms and cfs period to 100ms, so we can use at most 20% of cpu during any 100ms period. If we ever go over 20ms of cpu time, we'll be throttled for the remaining of 100ms period.

100ms sleep between iterations

We burn CPU for 5ms and then we sleep for 100ms, that sums up to 105ms, so in theory we should never go over quota. In practice, we see throttles from time to time.

$ docker run --rm -it --cpu-quota 20000 --cpu-period 100000 -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 100 -sleep 100ms
2017/12/08 01:42:50 [0] burn took 5ms, real time so far: 5ms, cpu time so far: 6ms
2017/12/08 01:42:50 [1] burn took 5ms, real time so far: 194ms, cpu time so far: 12ms
2017/12/08 01:42:50 [2] burn took 5ms, real time so far: 299ms, cpu time so far: 18ms
2017/12/08 01:42:50 [3] burn took 5ms, real time so far: 404ms, cpu time so far: 23ms
2017/12/08 01:42:51 [4] burn took 5ms, real time so far: 509ms, cpu time so far: 29ms
2017/12/08 01:42:51 [5] burn took 5ms, real time so far: 614ms, cpu time so far: 35ms
2017/12/08 01:42:51 [6] burn took 5ms, real time so far: 719ms, cpu time so far: 40ms
2017/12/08 01:42:51 [7] burn took 5ms, real time so far: 824ms, cpu time so far: 46ms
2017/12/08 01:42:51 [8] burn took 5ms, real time so far: 930ms, cpu time so far: 51ms
2017/12/08 01:42:51 [9] burn took 5ms, real time so far: 1035ms, cpu time so far: 58ms
2017/12/08 01:42:51 [10] burn took 5ms, real time so far: 1140ms, cpu time so far: 64ms
2017/12/08 01:42:51 [11] burn took 5ms, real time so far: 1245ms, cpu time so far: 69ms
2017/12/08 01:42:51 [12] burn took 5ms, real time so far: 1350ms, cpu time so far: 75ms
2017/12/08 01:42:51 [13] burn took 5ms, real time so far: 1455ms, cpu time so far: 81ms
2017/12/08 01:42:52 [14] burn took 5ms, real time so far: 1560ms, cpu time so far: 87ms
2017/12/08 01:42:52 [15] burn took 5ms, real time so far: 1665ms, cpu time so far: 92ms
2017/12/08 01:42:52 [16] burn took 5ms, real time so far: 1770ms, cpu time so far: 98ms
2017/12/08 01:42:52 [17] burn took 5ms, real time so far: 1876ms, cpu time so far: 105ms
2017/12/08 01:42:52 [18] burn took 5ms, real time so far: 1981ms, cpu time so far: 110ms
2017/12/08 01:42:52 [19] burn took 5ms, real time so far: 2086ms, cpu time so far: 117ms
2017/12/08 01:42:52 [20] burn took 5ms, real time so far: 2191ms, cpu time so far: 122ms
2017/12/08 01:42:52 [21] burn took 97ms, real time so far: 2389ms, cpu time so far: 127ms
2017/12/08 01:42:53 [22] burn took 5ms, real time so far: 2494ms, cpu time so far: 133ms
2017/12/08 01:42:53 [23] burn took 5ms, real time so far: 2599ms, cpu time so far: 140ms
2017/12/08 01:42:53 [24] burn took 5ms, real time so far: 2704ms, cpu time so far: 145ms
2017/12/08 01:42:53 [25] burn took 5ms, real time so far: 2809ms, cpu time so far: 152ms
2017/12/08 01:42:53 [26] burn took 5ms, real time so far: 2914ms, cpu time so far: 157ms
2017/12/08 01:42:53 [27] burn took 5ms, real time so far: 3019ms, cpu time so far: 164ms
2017/12/08 01:42:53 [28] burn took 5ms, real time so far: 3125ms, cpu time so far: 170ms
2017/12/08 01:42:53 [29] burn took 5ms, real time so far: 3230ms, cpu time so far: 175ms
2017/12/08 01:42:53 [30] burn took 5ms, real time so far: 3335ms, cpu time so far: 182ms
2017/12/08 01:42:53 [31] burn took 5ms, real time so far: 3440ms, cpu time so far: 187ms
2017/12/08 01:42:54 [32] burn took 5ms, real time so far: 3545ms, cpu time so far: 193ms
2017/12/08 01:42:54 [33] burn took 5ms, real time so far: 3650ms, cpu time so far: 198ms
2017/12/08 01:42:54 [34] burn took 5ms, real time so far: 3755ms, cpu time so far: 205ms
2017/12/08 01:42:54 [35] burn took 5ms, real time so far: 3860ms, cpu time so far: 211ms
2017/12/08 01:42:54 [36] burn took 5ms, real time so far: 3965ms, cpu time so far: 216ms
2017/12/08 01:42:54 [37] burn took 5ms, real time so far: 4071ms, cpu time so far: 222ms
2017/12/08 01:42:54 [38] burn took 5ms, real time so far: 4176ms, cpu time so far: 228ms
2017/12/08 01:42:54 [39] burn took 5ms, real time so far: 4281ms, cpu time so far: 234ms
2017/12/08 01:42:54 [40] burn took 5ms, real time so far: 4386ms, cpu time so far: 239ms
2017/12/08 01:42:55 [41] burn took 5ms, real time so far: 4491ms, cpu time so far: 246ms
2017/12/08 01:42:55 [42] burn took 97ms, real time so far: 4689ms, cpu time so far: 252ms
2017/12/08 01:42:55 [43] burn took 5ms, real time so far: 4794ms, cpu time so far: 257ms
2017/12/08 01:42:55 [44] burn took 5ms, real time so far: 4899ms, cpu time so far: 264ms
2017/12/08 01:42:55 [45] burn took 5ms, real time so far: 5004ms, cpu time so far: 269ms
2017/12/08 01:42:55 [46] burn took 5ms, real time so far: 5109ms, cpu time so far: 275ms
2017/12/08 01:42:55 [47] burn took 5ms, real time so far: 5214ms, cpu time so far: 281ms
2017/12/08 01:42:55 [48] burn took 5ms, real time so far: 5320ms, cpu time so far: 287ms
2017/12/08 01:42:55 [49] burn took 5ms, real time so far: 5425ms, cpu time so far: 292ms
2017/12/08 01:42:56 [50] burn took 5ms, real time so far: 5530ms, cpu time so far: 298ms
2017/12/08 01:42:56 [51] burn took 5ms, real time so far: 5635ms, cpu time so far: 305ms
2017/12/08 01:42:56 [52] burn took 5ms, real time so far: 5740ms, cpu time so far: 310ms
2017/12/08 01:42:56 [53] burn took 5ms, real time so far: 5845ms, cpu time so far: 316ms
2017/12/08 01:42:56 [54] burn took 5ms, real time so far: 5950ms, cpu time so far: 321ms
2017/12/08 01:42:56 [55] burn took 5ms, real time so far: 6055ms, cpu time so far: 328ms
2017/12/08 01:42:56 [56] burn took 5ms, real time so far: 6161ms, cpu time so far: 333ms
2017/12/08 01:42:56 [57] burn took 5ms, real time so far: 6266ms, cpu time so far: 339ms
2017/12/08 01:42:56 [58] burn took 5ms, real time so far: 6371ms, cpu time so far: 345ms
2017/12/08 01:42:56 [59] burn took 5ms, real time so far: 6476ms, cpu time so far: 351ms
2017/12/08 01:42:57 [60] burn took 5ms, real time so far: 6581ms, cpu time so far: 357ms
2017/12/08 01:42:57 [61] burn took 5ms, real time so far: 6686ms, cpu time so far: 362ms
2017/12/08 01:42:57 [62] burn took 5ms, real time so far: 6791ms, cpu time so far: 368ms
2017/12/08 01:42:57 [63] burn took 5ms, real time so far: 6896ms, cpu time so far: 374ms
2017/12/08 01:42:57 [64] burn took 5ms, real time so far: 7001ms, cpu time so far: 380ms
2017/12/08 01:42:57 [65] burn took 5ms, real time so far: 7107ms, cpu time so far: 386ms
2017/12/08 01:42:57 [66] burn took 5ms, real time so far: 7212ms, cpu time so far: 392ms
2017/12/08 01:42:57 [67] burn took 5ms, real time so far: 7317ms, cpu time so far: 398ms
2017/12/08 01:42:57 [68] burn took 5ms, real time so far: 7422ms, cpu time so far: 403ms
2017/12/08 01:42:58 [69] burn took 5ms, real time so far: 7527ms, cpu time so far: 410ms
2017/12/08 01:42:58 [70] burn took 5ms, real time so far: 7632ms, cpu time so far: 415ms
2017/12/08 01:42:58 [71] burn took 5ms, real time so far: 7737ms, cpu time so far: 421ms
2017/12/08 01:42:58 [72] burn took 5ms, real time so far: 7842ms, cpu time so far: 427ms
2017/12/08 01:42:58 [73] burn took 5ms, real time so far: 7947ms, cpu time so far: 432ms
2017/12/08 01:42:58 [74] burn took 5ms, real time so far: 8053ms, cpu time so far: 439ms
2017/12/08 01:42:58 [75] burn took 5ms, real time so far: 8158ms, cpu time so far: 444ms
2017/12/08 01:42:58 [76] burn took 5ms, real time so far: 8263ms, cpu time so far: 450ms
2017/12/08 01:42:58 [77] burn took 5ms, real time so far: 8368ms, cpu time so far: 455ms
2017/12/08 01:42:58 [78] burn took 5ms, real time so far: 8473ms, cpu time so far: 461ms
2017/12/08 01:42:59 [79] burn took 5ms, real time so far: 8578ms, cpu time so far: 468ms
2017/12/08 01:42:59 [80] burn took 5ms, real time so far: 8683ms, cpu time so far: 473ms
2017/12/08 01:42:59 [81] burn took 5ms, real time so far: 8788ms, cpu time so far: 479ms
2017/12/08 01:42:59 [82] burn took 5ms, real time so far: 8894ms, cpu time so far: 484ms
2017/12/08 01:42:59 [83] burn took 5ms, real time so far: 9094ms, cpu time so far: 491ms
2017/12/08 01:42:59 [84] burn took 5ms, real time so far: 9199ms, cpu time so far: 497ms
2017/12/08 01:42:59 [85] burn took 5ms, real time so far: 9304ms, cpu time so far: 502ms
2017/12/08 01:42:59 [86] burn took 5ms, real time so far: 9409ms, cpu time so far: 508ms
2017/12/08 01:43:00 [87] burn took 5ms, real time so far: 9514ms, cpu time so far: 514ms
2017/12/08 01:43:00 [88] burn took 5ms, real time so far: 9619ms, cpu time so far: 520ms
2017/12/08 01:43:00 [89] burn took 5ms, real time so far: 9724ms, cpu time so far: 525ms
2017/12/08 01:43:00 [90] burn took 5ms, real time so far: 9829ms, cpu time so far: 531ms
2017/12/08 01:43:00 [91] burn took 5ms, real time so far: 9935ms, cpu time so far: 538ms
2017/12/08 01:43:00 [92] burn took 5ms, real time so far: 10040ms, cpu time so far: 543ms
2017/12/08 01:43:00 [93] burn took 5ms, real time so far: 10145ms, cpu time so far: 550ms
2017/12/08 01:43:00 [94] burn took 5ms, real time so far: 10250ms, cpu time so far: 555ms
2017/12/08 01:43:00 [95] burn took 5ms, real time so far: 10355ms, cpu time so far: 561ms
2017/12/08 01:43:00 [96] burn took 5ms, real time so far: 10460ms, cpu time so far: 566ms
2017/12/08 01:43:01 [97] burn took 5ms, real time so far: 10565ms, cpu time so far: 573ms
2017/12/08 01:43:01 [98] burn took 5ms, real time so far: 10670ms, cpu time so far: 578ms
2017/12/08 01:43:01 [99] burn took 5ms, real time so far: 10776ms, cpu time so far: 585ms

1000ms sleep between iterations

With 5ms burns and 1000ms sleeps between them there are no 100ms intervals during which we can possibly see 20ms burned on CPU to get throttled. However, we see lots of throttling here. Almost every burn is throttled.

$ docker run --rm -it --cpu-quota 20000 --cpu-period 100000 -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 100 -sleep 1000ms
2017/12/08 01:44:27 [0] burn took 5ms, real time so far: 5ms, cpu time so far: 6ms
2017/12/08 01:44:28 [1] burn took 100ms, real time so far: 1187ms, cpu time so far: 12ms
2017/12/08 01:44:30 [2] burn took 5ms, real time so far: 2192ms, cpu time so far: 18ms
2017/12/08 01:44:31 [3] burn took 99ms, real time so far: 3386ms, cpu time so far: 25ms
2017/12/08 01:44:32 [4] burn took 5ms, real time so far: 4391ms, cpu time so far: 30ms
2017/12/08 01:44:33 [5] burn took 100ms, real time so far: 5586ms, cpu time so far: 35ms
2017/12/08 01:44:34 [6] burn took 99ms, real time so far: 6686ms, cpu time so far: 40ms
2017/12/08 01:44:35 [7] burn took 5ms, real time so far: 7691ms, cpu time so far: 45ms
2017/12/08 01:44:36 [8] burn took 99ms, real time so far: 8886ms, cpu time so far: 52ms
2017/12/08 01:44:37 [9] burn took 99ms, real time so far: 9986ms, cpu time so far: 58ms
2017/12/08 01:44:38 [10] burn took 5ms, real time so far: 10991ms, cpu time so far: 64ms
2017/12/08 01:44:39 [11] burn took 99ms, real time so far: 12186ms, cpu time so far: 69ms
2017/12/08 01:44:41 [12] burn took 5ms, real time so far: 13191ms, cpu time so far: 75ms
2017/12/08 01:44:42 [13] burn took 99ms, real time so far: 14386ms, cpu time so far: 80ms
2017/12/08 01:44:43 [14] burn took 99ms, real time so far: 15486ms, cpu time so far: 86ms
2017/12/08 01:44:44 [15] burn took 100ms, real time so far: 16586ms, cpu time so far: 93ms
2017/12/08 01:44:45 [16] burn took 5ms, real time so far: 17591ms, cpu time so far: 99ms
2017/12/08 01:44:46 [17] burn took 99ms, real time so far: 18786ms, cpu time so far: 101ms
2017/12/08 01:44:47 [18] burn took 99ms, real time so far: 19886ms, cpu time so far: 104ms
2017/12/08 01:44:48 [19] burn took 5ms, real time so far: 20891ms, cpu time so far: 109ms
2017/12/08 01:44:49 [20] burn took 100ms, real time so far: 22086ms, cpu time so far: 115ms
2017/12/08 01:44:50 [21] burn took 99ms, real time so far: 23186ms, cpu time so far: 120ms
2017/12/08 01:44:52 [22] burn took 99ms, real time so far: 24286ms, cpu time so far: 126ms
2017/12/08 01:44:53 [23] burn took 5ms, real time so far: 25291ms, cpu time so far: 132ms
2017/12/08 01:44:54 [24] burn took 99ms, real time so far: 26486ms, cpu time so far: 137ms
2017/12/08 01:44:55 [25] burn took 5ms, real time so far: 27491ms, cpu time so far: 143ms
2017/12/08 01:44:56 [26] burn took 99ms, real time so far: 28686ms, cpu time so far: 150ms
2017/12/08 01:44:57 [27] burn took 5ms, real time so far: 29691ms, cpu time so far: 155ms
2017/12/08 01:44:58 [28] burn took 100ms, real time so far: 30886ms, cpu time so far: 160ms
2017/12/08 01:44:59 [29] burn took 5ms, real time so far: 31891ms, cpu time so far: 166ms
2017/12/08 01:45:00 [30] burn took 99ms, real time so far: 33086ms, cpu time so far: 170ms
2017/12/08 01:45:01 [31] burn took 99ms, real time so far: 34186ms, cpu time so far: 176ms
2017/12/08 01:45:03 [32] burn took 99ms, real time so far: 35286ms, cpu time so far: 182ms
2017/12/08 01:45:04 [33] burn took 5ms, real time so far: 36291ms, cpu time so far: 187ms
2017/12/08 01:45:05 [34] burn took 99ms, real time so far: 37486ms, cpu time so far: 192ms
2017/12/08 01:45:06 [35] burn took 99ms, real time so far: 38586ms, cpu time so far: 197ms
2017/12/08 01:45:07 [36] burn took 99ms, real time so far: 39686ms, cpu time so far: 202ms
2017/12/08 01:45:08 [37] burn took 5ms, real time so far: 40691ms, cpu time so far: 208ms
2017/12/08 01:45:09 [38] burn took 99ms, real time so far: 41886ms, cpu time so far: 213ms
2017/12/08 01:45:10 [39] burn took 99ms, real time so far: 42986ms, cpu time so far: 219ms
2017/12/08 01:45:11 [40] burn took 99ms, real time so far: 44086ms, cpu time so far: 224ms
2017/12/08 01:45:12 [41] burn took 99ms, real time so far: 45186ms, cpu time so far: 229ms
2017/12/08 01:45:14 [42] burn took 99ms, real time so far: 46286ms, cpu time so far: 235ms
2017/12/08 01:45:15 [43] burn took 99ms, real time so far: 47386ms, cpu time so far: 241ms
2017/12/08 01:45:16 [44] burn took 100ms, real time so far: 48486ms, cpu time so far: 247ms
2017/12/08 01:45:17 [45] burn took 99ms, real time so far: 49586ms, cpu time so far: 253ms
2017/12/08 01:45:18 [46] burn took 99ms, real time so far: 50686ms, cpu time so far: 259ms
2017/12/08 01:45:19 [47] burn took 5ms, real time so far: 51691ms, cpu time so far: 265ms
2017/12/08 01:45:20 [48] burn took 99ms, real time so far: 52886ms, cpu time so far: 270ms
2017/12/08 01:45:21 [49] burn took 5ms, real time so far: 53891ms, cpu time so far: 276ms
2017/12/08 01:45:22 [50] burn took 99ms, real time so far: 55086ms, cpu time so far: 281ms
2017/12/08 01:45:23 [51] burn took 99ms, real time so far: 56186ms, cpu time so far: 286ms
2017/12/08 01:45:25 [52] burn took 99ms, real time so far: 57286ms, cpu time so far: 292ms
2017/12/08 01:45:26 [53] burn took 5ms, real time so far: 58291ms, cpu time so far: 298ms
2017/12/08 01:45:27 [54] burn took 99ms, real time so far: 59486ms, cpu time so far: 303ms
2017/12/08 01:45:28 [55] burn took 5ms, real time so far: 60491ms, cpu time so far: 309ms
2017/12/08 01:45:29 [56] burn took 99ms, real time so far: 61686ms, cpu time so far: 314ms
2017/12/08 01:45:30 [57] burn took 5ms, real time so far: 62691ms, cpu time so far: 320ms
2017/12/08 01:45:31 [58] burn took 100ms, real time so far: 63886ms, cpu time so far: 326ms
2017/12/08 01:45:32 [59] burn took 99ms, real time so far: 64986ms, cpu time so far: 328ms
2017/12/08 01:45:33 [60] burn took 5ms, real time so far: 65991ms, cpu time so far: 333ms
2017/12/08 01:45:34 [61] burn took 99ms, real time so far: 67186ms, cpu time so far: 337ms
2017/12/08 01:45:36 [62] burn took 5ms, real time so far: 68191ms, cpu time so far: 343ms
2017/12/08 01:45:37 [63] burn took 100ms, real time so far: 69386ms, cpu time so far: 350ms
2017/12/08 01:45:38 [64] burn took 99ms, real time so far: 70486ms, cpu time so far: 355ms
2017/12/08 01:45:39 [65] burn took 100ms, real time so far: 71586ms, cpu time so far: 361ms
2017/12/08 01:45:40 [66] burn took 5ms, real time so far: 72591ms, cpu time so far: 367ms
2017/12/08 01:45:41 [67] burn took 5ms, real time so far: 73691ms, cpu time so far: 374ms
2017/12/08 01:45:42 [68] burn took 94ms, real time so far: 74786ms, cpu time so far: 379ms
2017/12/08 01:45:43 [69] burn took 99ms, real time so far: 75886ms, cpu time so far: 385ms
2017/12/08 01:45:44 [70] burn took 5ms, real time so far: 76891ms, cpu time so far: 390ms
2017/12/08 01:45:45 [71] burn took 99ms, real time so far: 78086ms, cpu time so far: 395ms
2017/12/08 01:45:46 [72] burn took 99ms, real time so far: 79186ms, cpu time so far: 396ms
2017/12/08 01:45:48 [73] burn took 5ms, real time so far: 80191ms, cpu time so far: 401ms
2017/12/08 01:45:49 [74] burn took 99ms, real time so far: 81386ms, cpu time so far: 406ms
2017/12/08 01:45:50 [75] burn took 5ms, real time so far: 82391ms, cpu time so far: 412ms
2017/12/08 01:45:51 [76] burn took 99ms, real time so far: 83586ms, cpu time so far: 417ms
2017/12/08 01:45:52 [77] burn took 5ms, real time so far: 84591ms, cpu time so far: 423ms
2017/12/08 01:45:53 [78] burn took 99ms, real time so far: 85786ms, cpu time so far: 429ms
2017/12/08 01:45:54 [79] burn took 5ms, real time so far: 86791ms, cpu time so far: 434ms
2017/12/08 01:45:55 [80] burn took 100ms, real time so far: 87986ms, cpu time so far: 440ms
2017/12/08 01:45:56 [81] burn took 99ms, real time so far: 89086ms, cpu time so far: 446ms
2017/12/08 01:45:57 [82] burn took 99ms, real time so far: 90186ms, cpu time so far: 448ms
2017/12/08 01:45:59 [83] burn took 99ms, real time so far: 91286ms, cpu time so far: 452ms
2017/12/08 01:46:00 [84] burn took 99ms, real time so far: 92386ms, cpu time so far: 457ms
2017/12/08 01:46:01 [85] burn took 99ms, real time so far: 93486ms, cpu time so far: 463ms
2017/12/08 01:46:02 [86] burn took 5ms, real time so far: 94491ms, cpu time so far: 468ms
2017/12/08 01:46:03 [87] burn took 99ms, real time so far: 95686ms, cpu time so far: 474ms
2017/12/08 01:46:04 [88] burn took 5ms, real time so far: 96691ms, cpu time so far: 480ms
2017/12/08 01:46:05 [89] burn took 99ms, real time so far: 97886ms, cpu time so far: 486ms
2017/12/08 01:46:06 [90] burn took 5ms, real time so far: 98891ms, cpu time so far: 492ms
2017/12/08 01:46:07 [91] burn took 100ms, real time so far: 100086ms, cpu time so far: 497ms
2017/12/08 01:46:08 [92] burn took 5ms, real time so far: 101091ms, cpu time so far: 503ms
2017/12/08 01:46:10 [93] burn took 99ms, real time so far: 102286ms, cpu time so far: 508ms
2017/12/08 01:46:11 [94] burn took 5ms, real time so far: 103291ms, cpu time so far: 514ms
2017/12/08 01:46:12 [95] burn took 99ms, real time so far: 104486ms, cpu time so far: 518ms
2017/12/08 01:46:13 [96] burn took 99ms, real time so far: 105586ms, cpu time so far: 525ms
2017/12/08 01:46:14 [97] burn took 99ms, real time so far: 106686ms, cpu time so far: 529ms
2017/12/08 01:46:15 [98] burn took 99ms, real time so far: 107786ms, cpu time so far: 532ms
2017/12/08 01:46:16 [99] burn took 5ms, real time so far: 108791ms, cpu time so far: 538ms

5000ms sleep between iterations

Again, lots of unexpected throttling.

$ docker run --rm -it --cpu-quota 20000 --cpu-period 100000 -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 10 -sleep 5000ms
2017/12/08 01:46:45 [0] burn took 5ms, real time so far: 5ms, cpu time so far: 6ms
2017/12/08 01:46:50 [1] burn took 100ms, real time so far: 5199ms, cpu time so far: 12ms
2017/12/08 01:46:55 [2] burn took 98ms, real time so far: 10298ms, cpu time so far: 15ms
2017/12/08 01:47:00 [3] burn took 5ms, real time so far: 15303ms, cpu time so far: 20ms
2017/12/08 01:47:05 [4] burn took 99ms, real time so far: 20498ms, cpu time so far: 26ms
2017/12/08 01:47:10 [5] burn took 99ms, real time so far: 25598ms, cpu time so far: 31ms
2017/12/08 01:47:15 [6] burn took 99ms, real time so far: 30698ms, cpu time so far: 38ms
2017/12/08 01:47:20 [7] burn took 99ms, real time so far: 35798ms, cpu time so far: 44ms
2017/12/08 01:47:25 [8] burn took 99ms, real time so far: 40898ms, cpu time so far: 49ms
2017/12/08 01:47:30 [9] burn took 99ms, real time so far: 45998ms, cpu time so far: 55ms
Raw
cfs.go
package main
import (
"crypto/sha512"
"flag"
"log"
"syscall"
"time"
)
func main() {
sleep := flag.Duration("sleep", time.Second, "sleep between iterations")
interations := flag.Int("iterations", 100, "number of iterations")
flag.Parse()
time.Sleep(time.Second)
b := time.Now()
for i := 0; i < *interations; i++ {
s := time.Now()
burn(time.Millisecond * 5)
e := time.Now().Sub(s)
log.Printf("[%d] burn took %dms, real time so far: %dms, cpu time so far: %dms", i, ms(e), ms(time.Since(b)), ms(usage()))
time.Sleep(*sleep)
}
}
func ms(duration time.Duration) int {
return int(duration.Nanoseconds() / 1000 / 1000)
}
func burn(duration time.Duration) {
s := time.Now()
for {
sum := sha512.New()
sum.Write([]byte("banana"))
sum.Sum([]byte{})
if time.Since(s) > duration {
break
}
}
}
func usage() time.Duration {
r := syscall.Rusage{}
syscall.Getrusage(syscall.RUSAGE_SELF, &r)
return time.Duration(r.Stime.Nano() + r.Utime.Nano())
}
@pracucci
Copy link

Thanks for sharing. Do you have any pointer to share on the root cause of such throttling? I'm aware as well of it (we've experienced pretty much the same in production) but the root cause is still not very clear to me.

@Matt-Esch
Copy link

I'm working with a theory that syscall time is added to the CFS throttling counters for the cgroup. I would be curious to see if you could use kernelshark to profile the scheduler to add or subtract any weight from that theory: https://lwn.net/Articles/425583/

@stefanschneider
Copy link

Thx for the write-up. This patch merged in Linux 4.18 is most likely related to this described behavior: torvalds/linux@512ac99#diff-1c5364196d98130348bddabaad0a701f
Tested cfs.go + docker on Ubuntu Bionic with 4.17 kernel and the scheduling jitter is still there, but with the 4.18 kernel it works well.

@nrobert13
Copy link

nrobert13 commented Jan 11, 2019 •
edited
Loading

I'm also trying to understand this issue, and not sure how is it possible to run this cfs.go with cfs-quota 4000, and get Throttling in less than 50% of the iterations. If I understand it right cfs-quota 4000 is 4 ms, so basically trying to execute this 5ms burn in 4ms, it should always Throttle.... what am I missing here?

@realdimas
Copy link

The issue seems to be fixed in the recent kernels.
Cannot reproduce on CoreOS Container Linux Stable 2023.4.0 running Kernel 4.19.23:

Docker Desktop for Mac Stable 2.0.0.3 running Linux Kernel 4.9.125 - Not OK 
docker run --rm -it --cpu-quota 20000 --cpu-period 100000 -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 100 -sleep 100ms
2019/03/08 15:57:48 [0] burn took 5ms, real time so far: 5ms, cpu time so far: 0ms
2019/03/08 15:57:48 [1] burn took 5ms, real time so far: 186ms, cpu time so far: 10ms
2019/03/08 15:57:48 [2] burn took 5ms, real time so far: 292ms, cpu time so far: 20ms
2019/03/08 15:57:48 [3] burn took 5ms, real time so far: 398ms, cpu time so far: 30ms
2019/03/08 15:57:48 [4] burn took 5ms, real time so far: 505ms, cpu time so far: 30ms
2019/03/08 15:57:48 [5] burn took 5ms, real time so far: 612ms, cpu time so far: 40ms
2019/03/08 15:57:48 [6] burn took 5ms, real time so far: 722ms, cpu time so far: 40ms
2019/03/08 15:57:48 [7] burn took 5ms, real time so far: 833ms, cpu time so far: 50ms
2019/03/08 15:57:49 [8] burn took 5ms, real time so far: 940ms, cpu time so far: 50ms
2019/03/08 15:57:49 [9] burn took 5ms, real time so far: 1048ms, cpu time so far: 60ms
2019/03/08 15:57:49 [10] burn took 5ms, real time so far: 1155ms, cpu time so far: 70ms
2019/03/08 15:57:49 [11] burn took 5ms, real time so far: 1261ms, cpu time so far: 70ms
2019/03/08 15:57:49 [12] burn took 5ms, real time so far: 1366ms, cpu time so far: 80ms
2019/03/08 15:57:49 [13] burn took 5ms, real time so far: 1475ms, cpu time so far: 80ms
2019/03/08 15:57:49 [14] burn took 5ms, real time so far: 1583ms, cpu time so far: 90ms
2019/03/08 15:57:49 [15] burn took 5ms, real time so far: 1689ms, cpu time so far: 100ms
2019/03/08 15:57:49 [16] burn took 5ms, real time so far: 1796ms, cpu time so far: 100ms
2019/03/08 15:57:50 [17] burn took 5ms, real time so far: 1904ms, cpu time so far: 110ms
2019/03/08 15:57:50 [18] burn took 5ms, real time so far: 2011ms, cpu time so far: 120ms
2019/03/08 15:57:50 [19] burn took 5ms, real time so far: 2117ms, cpu time so far: 120ms
2019/03/08 15:57:50 [20] burn took 5ms, real time so far: 2223ms, cpu time so far: 130ms
2019/03/08 15:57:50 [21] burn took 5ms, real time so far: 2328ms, cpu time so far: 140ms
2019/03/08 15:57:50 [22] burn took 5ms, real time so far: 2434ms, cpu time so far: 150ms
2019/03/08 15:57:50 [23] burn took 5ms, real time so far: 2540ms, cpu time so far: 150ms
2019/03/08 15:57:50 [24] burn took 5ms, real time so far: 2647ms, cpu time so far: 160ms
2019/03/08 15:57:50 [25] burn took 5ms, real time so far: 2757ms, cpu time so far: 170ms
2019/03/08 15:57:50 [26] burn took 5ms, real time so far: 2864ms, cpu time so far: 170ms
2019/03/08 15:57:51 [27] burn took 5ms, real time so far: 2974ms, cpu time so far: 180ms
2019/03/08 15:57:51 [28] burn took 5ms, real time so far: 3081ms, cpu time so far: 190ms
2019/03/08 15:57:51 [29] burn took 5ms, real time so far: 3187ms, cpu time so far: 200ms
2019/03/08 15:57:51 [30] burn took 5ms, real time so far: 3293ms, cpu time so far: 200ms
2019/03/08 15:57:51 [31] burn took 5ms, real time so far: 3400ms, cpu time so far: 210ms
2019/03/08 15:57:51 [32] burn took 5ms, real time so far: 3509ms, cpu time so far: 220ms
2019/03/08 15:57:51 [33] burn took 5ms, real time so far: 3616ms, cpu time so far: 220ms
2019/03/08 15:57:51 [34] burn took 5ms, real time so far: 3725ms, cpu time so far: 230ms
2019/03/08 15:57:51 [35] burn took 5ms, real time so far: 3832ms, cpu time so far: 240ms
2019/03/08 15:57:52 [36] burn took 5ms, real time so far: 3942ms, cpu time so far: 250ms
2019/03/08 15:57:52 [37] burn took 5ms, real time so far: 4050ms, cpu time so far: 250ms
2019/03/08 15:57:52 [38] burn took 5ms, real time so far: 4158ms, cpu time so far: 260ms
2019/03/08 15:57:52 [39] burn took 5ms, real time so far: 4266ms, cpu time so far: 270ms
2019/03/08 15:57:52 [40] burn took 5ms, real time so far: 4375ms, cpu time so far: 280ms
2019/03/08 15:57:52 [41] burn took 97ms, real time so far: 4578ms, cpu time so far: 280ms
2019/03/08 15:57:52 [42] burn took 5ms, real time so far: 4684ms, cpu time so far: 290ms
2019/03/08 15:57:52 [43] burn took 5ms, real time so far: 4790ms, cpu time so far: 300ms
2019/03/08 15:57:53 [44] burn took 5ms, real time so far: 4896ms, cpu time so far: 300ms
2019/03/08 15:57:53 [45] burn took 5ms, real time so far: 5004ms, cpu time so far: 310ms
2019/03/08 15:57:53 [46] burn took 5ms, real time so far: 5111ms, cpu time so far: 320ms
2019/03/08 15:57:53 [47] burn took 5ms, real time so far: 5217ms, cpu time so far: 330ms
2019/03/08 15:57:53 [48] burn took 5ms, real time so far: 5324ms, cpu time so far: 330ms
2019/03/08 15:57:53 [49] burn took 5ms, real time so far: 5431ms, cpu time so far: 340ms
2019/03/08 15:57:53 [50] burn took 5ms, real time so far: 5537ms, cpu time so far: 350ms
2019/03/08 15:57:53 [51] burn took 5ms, real time so far: 5645ms, cpu time so far: 360ms
2019/03/08 15:57:53 [52] burn took 5ms, real time so far: 5751ms, cpu time so far: 360ms
2019/03/08 15:57:53 [53] burn took 5ms, real time so far: 5856ms, cpu time so far: 370ms
2019/03/08 15:57:54 [54] burn took 5ms, real time so far: 5964ms, cpu time so far: 380ms
2019/03/08 15:57:54 [55] burn took 5ms, real time so far: 6070ms, cpu time so far: 390ms
2019/03/08 15:57:54 [56] burn took 5ms, real time so far: 6180ms, cpu time so far: 390ms
2019/03/08 15:57:54 [57] burn took 5ms, real time so far: 6286ms, cpu time so far: 400ms
2019/03/08 15:57:54 [58] burn took 5ms, real time so far: 6393ms, cpu time so far: 410ms
2019/03/08 15:57:54 [59] burn took 5ms, real time so far: 6499ms, cpu time so far: 420ms
2019/03/08 15:57:54 [60] burn took 5ms, real time so far: 6605ms, cpu time so far: 420ms
2019/03/08 15:57:54 [61] burn took 5ms, real time so far: 6714ms, cpu time so far: 430ms
2019/03/08 15:57:54 [62] burn took 5ms, real time so far: 6822ms, cpu time so far: 440ms
2019/03/08 15:57:55 [63] burn took 5ms, real time so far: 6930ms, cpu time so far: 450ms
2019/03/08 15:57:55 [64] burn took 5ms, real time so far: 7037ms, cpu time so far: 450ms
2019/03/08 15:57:55 [65] burn took 5ms, real time so far: 7147ms, cpu time so far: 460ms
2019/03/08 15:57:55 [66] burn took 5ms, real time so far: 7253ms, cpu time so far: 470ms
2019/03/08 15:57:55 [67] burn took 5ms, real time so far: 7359ms, cpu time so far: 480ms
2019/03/08 15:57:55 [68] burn took 5ms, real time so far: 7469ms, cpu time so far: 480ms
2019/03/08 15:57:55 [69] burn took 5ms, real time so far: 7579ms, cpu time so far: 490ms
2019/03/08 15:57:55 [70] burn took 5ms, real time so far: 7685ms, cpu time so far: 500ms
2019/03/08 15:57:55 [71] burn took 5ms, real time so far: 7792ms, cpu time so far: 510ms
2019/03/08 15:57:56 [72] burn took 5ms, real time so far: 7898ms, cpu time so far: 510ms
2019/03/08 15:57:56 [73] burn took 5ms, real time so far: 8006ms, cpu time so far: 520ms
2019/03/08 15:57:56 [74] burn took 5ms, real time so far: 8115ms, cpu time so far: 530ms
2019/03/08 15:57:56 [75] burn took 5ms, real time so far: 8220ms, cpu time so far: 540ms
2019/03/08 15:57:56 [76] burn took 5ms, real time so far: 8331ms, cpu time so far: 540ms
2019/03/08 15:57:56 [77] burn took 5ms, real time so far: 8441ms, cpu time so far: 550ms
2019/03/08 15:57:56 [78] burn took 5ms, real time so far: 8548ms, cpu time so far: 560ms
2019/03/08 15:57:56 [79] burn took 5ms, real time so far: 8655ms, cpu time so far: 560ms
2019/03/08 15:57:56 [80] burn took 5ms, real time so far: 8761ms, cpu time so far: 570ms
2019/03/08 15:57:56 [81] burn took 5ms, real time so far: 8867ms, cpu time so far: 580ms
2019/03/08 15:57:57 [82] burn took 5ms, real time so far: 8974ms, cpu time so far: 580ms
2019/03/08 15:57:57 [83] burn took 5ms, real time so far: 9082ms, cpu time so far: 590ms
2019/03/08 15:57:57 [84] burn took 5ms, real time so far: 9189ms, cpu time so far: 600ms
2019/03/08 15:57:57 [85] burn took 5ms, real time so far: 9295ms, cpu time so far: 610ms
2019/03/08 15:57:57 [86] burn took 5ms, real time so far: 9403ms, cpu time so far: 610ms
2019/03/08 15:57:57 [87] burn took 5ms, real time so far: 9510ms, cpu time so far: 620ms
2019/03/08 15:57:57 [88] burn took 5ms, real time so far: 9622ms, cpu time so far: 630ms
2019/03/08 15:57:57 [89] burn took 5ms, real time so far: 9731ms, cpu time so far: 640ms
2019/03/08 15:57:57 [90] burn took 5ms, real time so far: 9837ms, cpu time so far: 640ms
2019/03/08 15:57:58 [91] burn took 5ms, real time so far: 9946ms, cpu time so far: 650ms
2019/03/08 15:57:58 [92] burn took 5ms, real time so far: 10054ms, cpu time so far: 660ms
2019/03/08 15:57:58 [93] burn took 5ms, real time so far: 10163ms, cpu time so far: 660ms
2019/03/08 15:57:58 [94] burn took 5ms, real time so far: 10273ms, cpu time so far: 670ms
2019/03/08 15:57:58 [95] burn took 5ms, real time so far: 10380ms, cpu time so far: 680ms
2019/03/08 15:57:58 [96] burn took 5ms, real time so far: 10485ms, cpu time so far: 690ms
2019/03/08 15:57:58 [97] burn took 5ms, real time so far: 10591ms, cpu time so far: 690ms
2019/03/08 15:57:58 [98] burn took 5ms, real time so far: 10698ms, cpu time so far: 700ms
2019/03/08 15:57:58 [99] burn took 5ms, real time so far: 10804ms, cpu time so far: 710ms

docker run --rm -it --cpu-quota 20000 --cpu-period 100000 -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 100 -sleep 1000ms
2019/03/08 15:58:13 [0] burn took 5ms, real time so far: 5ms, cpu time so far: 10ms
2019/03/08 15:58:14 [1] burn took 5ms, real time so far: 1012ms, cpu time so far: 10ms
2019/03/08 15:58:15 [2] burn took 5ms, real time so far: 2100ms, cpu time so far: 20ms
2019/03/08 15:58:16 [3] burn took 89ms, real time so far: 3194ms, cpu time so far: 30ms
2019/03/08 15:58:17 [4] burn took 5ms, real time so far: 4201ms, cpu time so far: 40ms
2019/03/08 15:58:18 [5] burn took 5ms, real time so far: 5206ms, cpu time so far: 40ms
2019/03/08 15:58:19 [6] burn took 5ms, real time so far: 6298ms, cpu time so far: 50ms
2019/03/08 15:58:20 [7] burn took 90ms, real time so far: 7389ms, cpu time so far: 50ms
2019/03/08 15:58:21 [8] burn took 5ms, real time so far: 8395ms, cpu time so far: 60ms
2019/03/08 15:58:22 [9] burn took 5ms, real time so far: 9499ms, cpu time so far: 70ms
2019/03/08 15:58:23 [10] burn took 5ms, real time so far: 10600ms, cpu time so far: 70ms
2019/03/08 15:58:25 [11] burn took 5ms, real time so far: 11699ms, cpu time so far: 80ms
2019/03/08 15:58:26 [12] burn took 5ms, real time so far: 12801ms, cpu time so far: 90ms
2019/03/08 15:58:27 [13] burn took 5ms, real time so far: 13903ms, cpu time so far: 100ms
2019/03/08 15:58:28 [14] burn took 5ms, real time so far: 14999ms, cpu time so far: 100ms
2019/03/08 15:58:29 [15] burn took 5ms, real time so far: 16097ms, cpu time so far: 110ms
2019/03/08 15:58:30 [16] burn took 91ms, real time so far: 17192ms, cpu time so far: 120ms
2019/03/08 15:58:31 [17] burn took 5ms, real time so far: 18198ms, cpu time so far: 120ms
2019/03/08 15:58:32 [18] burn took 5ms, real time so far: 19296ms, cpu time so far: 130ms
2019/03/08 15:58:33 [19] burn took 88ms, real time so far: 20390ms, cpu time so far: 140ms
2019/03/08 15:58:34 [20] burn took 5ms, real time so far: 21397ms, cpu time so far: 150ms
2019/03/08 15:58:35 [21] burn took 5ms, real time so far: 22499ms, cpu time so far: 150ms
2019/03/08 15:58:37 [22] burn took 5ms, real time so far: 23598ms, cpu time so far: 160ms
2019/03/08 15:58:38 [23] burn took 5ms, real time so far: 24698ms, cpu time so far: 170ms
2019/03/08 15:58:39 [24] burn took 5ms, real time so far: 25705ms, cpu time so far: 180ms
2019/03/08 15:58:40 [25] burn took 98ms, real time so far: 26894ms, cpu time so far: 180ms
2019/03/08 15:58:41 [26] burn took 5ms, real time so far: 27904ms, cpu time so far: 190ms
2019/03/08 15:58:42 [27] burn took 5ms, real time so far: 28999ms, cpu time so far: 200ms
2019/03/08 15:58:43 [28] burn took 90ms, real time so far: 30092ms, cpu time so far: 200ms
2019/03/08 15:58:44 [29] burn took 5ms, real time so far: 31097ms, cpu time so far: 210ms
2019/03/08 15:58:45 [30] burn took 5ms, real time so far: 32195ms, cpu time so far: 220ms
2019/03/08 15:58:46 [31] burn took 93ms, real time so far: 33290ms, cpu time so far: 220ms
2019/03/08 15:58:47 [32] burn took 5ms, real time so far: 34301ms, cpu time so far: 230ms
2019/03/08 15:58:48 [33] burn took 5ms, real time so far: 35397ms, cpu time so far: 230ms
2019/03/08 15:58:49 [34] burn took 92ms, real time so far: 36490ms, cpu time so far: 240ms
2019/03/08 15:58:50 [35] burn took 5ms, real time so far: 37496ms, cpu time so far: 250ms
2019/03/08 15:58:51 [36] burn took 5ms, real time so far: 38597ms, cpu time so far: 250ms
2019/03/08 15:58:53 [37] burn took 5ms, real time so far: 39698ms, cpu time so far: 260ms
2019/03/08 15:58:54 [38] burn took 5ms, real time so far: 40704ms, cpu time so far: 270ms
2019/03/08 15:58:55 [39] burn took 5ms, real time so far: 41796ms, cpu time so far: 270ms
2019/03/08 15:58:56 [40] burn took 5ms, real time so far: 42804ms, cpu time so far: 280ms
2019/03/08 15:58:57 [41] burn took 5ms, real time so far: 43901ms, cpu time so far: 290ms
2019/03/08 15:58:58 [42] burn took 5ms, real time so far: 44907ms, cpu time so far: 290ms
2019/03/08 15:58:59 [43] burn took 5ms, real time so far: 46001ms, cpu time so far: 300ms
2019/03/08 15:59:00 [44] burn took 5ms, real time so far: 47098ms, cpu time so far: 310ms
2019/03/08 15:59:01 [45] burn took 5ms, real time so far: 48108ms, cpu time so far: 310ms
2019/03/08 15:59:02 [46] burn took 79ms, real time so far: 49190ms, cpu time so far: 310ms
2019/03/08 15:59:03 [47] burn took 5ms, real time so far: 50196ms, cpu time so far: 320ms
2019/03/08 15:59:04 [48] burn took 5ms, real time so far: 51304ms, cpu time so far: 330ms
2019/03/08 15:59:05 [49] burn took 97ms, real time so far: 52494ms, cpu time so far: 340ms
2019/03/08 15:59:06 [50] burn took 5ms, real time so far: 53500ms, cpu time so far: 340ms
2019/03/08 15:59:07 [51] burn took 5ms, real time so far: 54597ms, cpu time so far: 350ms
2019/03/08 15:59:08 [52] burn took 5ms, real time so far: 55607ms, cpu time so far: 360ms
2019/03/08 15:59:10 [53] burn took 82ms, real time so far: 56691ms, cpu time so far: 360ms
2019/03/08 15:59:11 [54] burn took 5ms, real time so far: 57696ms, cpu time so far: 360ms
2019/03/08 15:59:12 [55] burn took 5ms, real time so far: 58702ms, cpu time so far: 370ms
2019/03/08 15:59:13 [56] burn took 87ms, real time so far: 59792ms, cpu time so far: 380ms
2019/03/08 15:59:14 [57] burn took 96ms, real time so far: 60890ms, cpu time so far: 380ms
2019/03/08 15:59:15 [58] burn took 5ms, real time so far: 61897ms, cpu time so far: 390ms
2019/03/08 15:59:16 [59] burn took 5ms, real time so far: 62904ms, cpu time so far: 400ms
2019/03/08 15:59:17 [60] burn took 5ms, real time so far: 63995ms, cpu time so far: 400ms
2019/03/08 15:59:18 [61] burn took 5ms, real time so far: 65004ms, cpu time so far: 410ms
2019/03/08 15:59:19 [62] burn took 5ms, real time so far: 66013ms, cpu time so far: 410ms
2019/03/08 15:59:20 [63] burn took 5ms, real time so far: 67099ms, cpu time so far: 420ms
2019/03/08 15:59:21 [64] burn took 91ms, real time so far: 68192ms, cpu time so far: 420ms
2019/03/08 15:59:22 [65] burn took 5ms, real time so far: 69199ms, cpu time so far: 430ms
2019/03/08 15:59:23 [66] burn took 5ms, real time so far: 70297ms, cpu time so far: 440ms
2019/03/08 15:59:24 [67] burn took 5ms, real time so far: 71402ms, cpu time so far: 450ms
2019/03/08 15:59:25 [68] burn took 5ms, real time so far: 72500ms, cpu time so far: 460ms
2019/03/08 15:59:26 [69] burn took 5ms, real time so far: 73594ms, cpu time so far: 460ms
2019/03/08 15:59:27 [70] burn took 5ms, real time so far: 74600ms, cpu time so far: 470ms
2019/03/08 15:59:29 [71] burn took 90ms, real time so far: 75691ms, cpu time so far: 470ms
2019/03/08 15:59:30 [72] burn took 5ms, real time so far: 76697ms, cpu time so far: 470ms
2019/03/08 15:59:31 [73] burn took 91ms, real time so far: 77789ms, cpu time so far: 480ms
2019/03/08 15:59:32 [74] burn took 5ms, real time so far: 78796ms, cpu time so far: 480ms
2019/03/08 15:59:33 [75] burn took 89ms, real time so far: 79890ms, cpu time so far: 480ms
2019/03/08 15:59:34 [76] burn took 5ms, real time so far: 80896ms, cpu time so far: 490ms
2019/03/08 15:59:35 [77] burn took 92ms, real time so far: 81989ms, cpu time so far: 490ms
2019/03/08 15:59:36 [78] burn took 5ms, real time so far: 82995ms, cpu time so far: 500ms
2019/03/08 15:59:37 [79] burn took 93ms, real time so far: 84091ms, cpu time so far: 500ms
2019/03/08 15:59:38 [80] burn took 5ms, real time so far: 85102ms, cpu time so far: 500ms
2019/03/08 15:59:39 [81] burn took 86ms, real time so far: 86189ms, cpu time so far: 500ms
2019/03/08 15:59:40 [82] burn took 5ms, real time so far: 87195ms, cpu time so far: 510ms
2019/03/08 15:59:41 [83] burn took 5ms, real time so far: 88202ms, cpu time so far: 520ms
2019/03/08 15:59:42 [84] burn took 5ms, real time so far: 89297ms, cpu time so far: 530ms
2019/03/08 15:59:43 [85] burn took 5ms, real time so far: 90394ms, cpu time so far: 540ms
2019/03/08 15:59:44 [86] burn took 5ms, real time so far: 91496ms, cpu time so far: 540ms
2019/03/08 15:59:45 [87] burn took 5ms, real time so far: 92505ms, cpu time so far: 550ms
2019/03/08 15:59:47 [88] burn took 98ms, real time so far: 93694ms, cpu time so far: 560ms
2019/03/08 15:59:48 [89] burn took 96ms, real time so far: 94791ms, cpu time so far: 560ms
2019/03/08 15:59:49 [90] burn took 5ms, real time so far: 95797ms, cpu time so far: 570ms
2019/03/08 15:59:50 [91] burn took 5ms, real time so far: 96896ms, cpu time so far: 580ms
2019/03/08 15:59:51 [92] burn took 5ms, real time so far: 97999ms, cpu time so far: 590ms
2019/03/08 15:59:52 [93] burn took 5ms, real time so far: 99099ms, cpu time so far: 590ms
2019/03/08 15:59:53 [94] burn took 86ms, real time so far: 100191ms, cpu time so far: 600ms
2019/03/08 15:59:54 [95] burn took 5ms, real time so far: 101197ms, cpu time so far: 610ms
2019/03/08 15:59:55 [96] burn took 5ms, real time so far: 102304ms, cpu time so far: 610ms
2019/03/08 15:59:56 [97] burn took 85ms, real time so far: 103392ms, cpu time so far: 620ms
2019/03/08 15:59:57 [98] burn took 5ms, real time so far: 104402ms, cpu time so far: 620ms
2019/03/08 15:59:58 [99] burn took 87ms, real time so far: 105491ms, cpu time so far: 620ms

docker run --rm -it --cpu-quota 20000 --cpu-period 100000 -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 10 -sleep 5000ms
2019/03/08 16:00:14 [0] burn took 95ms, real time so far: 95ms, cpu time so far: 0ms
2019/03/08 16:00:19 [1] burn took 5ms, real time so far: 5103ms, cpu time so far: 10ms
2019/03/08 16:00:24 [2] burn took 5ms, real time so far: 10110ms, cpu time so far: 20ms
2019/03/08 16:00:29 [3] burn took 100ms, real time so far: 15295ms, cpu time so far: 20ms
2019/03/08 16:00:34 [4] burn took 5ms, real time so far: 20302ms, cpu time so far: 30ms
2019/03/08 16:00:39 [5] burn took 5ms, real time so far: 25403ms, cpu time so far: 40ms
2019/03/08 16:00:44 [6] burn took 5ms, real time so far: 30408ms, cpu time so far: 50ms
2019/03/08 16:00:49 [7] burn took 5ms, real time so far: 35414ms, cpu time so far: 60ms
2019/03/08 16:00:54 [8] burn took 5ms, real time so far: 40420ms, cpu time so far: 60ms
2019/03/08 16:00:59 [9] burn took 100ms, real time so far: 45597ms, cpu time so far: 70ms
Minikube 0.35.0 on VirtualBox on a Mac running Linux Kernel 4.15.0 – Not OK 
docker run --rm -it --cpu-quota 20000 --cpu-period 100000 -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 100 -sleep 100ms
2019/03/08 16:41:57 [0] burn took 84ms, real time so far: 84ms, cpu time so far: 4ms
2019/03/08 16:41:57 [1] burn took 5ms, real time so far: 190ms, cpu time so far: 10ms
2019/03/08 16:41:57 [2] burn took 5ms, real time so far: 295ms, cpu time so far: 16ms
2019/03/08 16:41:58 [3] burn took 5ms, real time so far: 403ms, cpu time so far: 21ms
2019/03/08 16:41:58 [4] burn took 5ms, real time so far: 508ms, cpu time so far: 27ms
2019/03/08 16:41:58 [5] burn took 5ms, real time so far: 614ms, cpu time so far: 32ms
2019/03/08 16:41:58 [6] burn took 5ms, real time so far: 720ms, cpu time so far: 37ms
2019/03/08 16:41:58 [7] burn took 5ms, real time so far: 826ms, cpu time so far: 43ms
2019/03/08 16:41:58 [8] burn took 5ms, real time so far: 933ms, cpu time so far: 49ms
2019/03/08 16:41:58 [9] burn took 5ms, real time so far: 1038ms, cpu time so far: 55ms
2019/03/08 16:41:58 [10] burn took 5ms, real time so far: 1144ms, cpu time so far: 60ms
2019/03/08 16:41:58 [11] burn took 5ms, real time so far: 1249ms, cpu time so far: 65ms
2019/03/08 16:41:58 [12] burn took 5ms, real time so far: 1355ms, cpu time so far: 72ms
2019/03/08 16:41:59 [13] burn took 5ms, real time so far: 1461ms, cpu time so far: 78ms
2019/03/08 16:41:59 [14] burn took 5ms, real time so far: 1566ms, cpu time so far: 84ms
2019/03/08 16:41:59 [15] burn took 5ms, real time so far: 1672ms, cpu time so far: 90ms
2019/03/08 16:41:59 [16] burn took 5ms, real time so far: 1777ms, cpu time so far: 97ms
2019/03/08 16:41:59 [17] burn took 5ms, real time so far: 1882ms, cpu time so far: 103ms
2019/03/08 16:41:59 [18] burn took 99ms, real time so far: 2082ms, cpu time so far: 106ms
2019/03/08 16:41:59 [19] burn took 5ms, real time so far: 2188ms, cpu time so far: 112ms
2019/03/08 16:41:59 [20] burn took 5ms, real time so far: 2294ms, cpu time so far: 118ms
2019/03/08 16:42:00 [21] burn took 5ms, real time so far: 2399ms, cpu time so far: 123ms
2019/03/08 16:42:00 [22] burn took 5ms, real time so far: 2506ms, cpu time so far: 130ms
2019/03/08 16:42:00 [23] burn took 5ms, real time so far: 2612ms, cpu time so far: 137ms
2019/03/08 16:42:00 [24] burn took 5ms, real time so far: 2720ms, cpu time so far: 143ms
2019/03/08 16:42:00 [25] burn took 5ms, real time so far: 2826ms, cpu time so far: 149ms
2019/03/08 16:42:00 [26] burn took 5ms, real time so far: 2933ms, cpu time so far: 155ms
2019/03/08 16:42:00 [27] burn took 5ms, real time so far: 3039ms, cpu time so far: 161ms
2019/03/08 16:42:00 [28] burn took 5ms, real time so far: 3144ms, cpu time so far: 166ms
2019/03/08 16:42:00 [29] burn took 5ms, real time so far: 3250ms, cpu time so far: 172ms
2019/03/08 16:42:00 [30] burn took 5ms, real time so far: 3357ms, cpu time so far: 177ms
2019/03/08 16:42:01 [31] burn took 5ms, real time so far: 3463ms, cpu time so far: 183ms
2019/03/08 16:42:01 [32] burn took 5ms, real time so far: 3569ms, cpu time so far: 189ms
2019/03/08 16:42:01 [33] burn took 5ms, real time so far: 3675ms, cpu time so far: 195ms
2019/03/08 16:42:01 [34] burn took 5ms, real time so far: 3781ms, cpu time so far: 201ms
2019/03/08 16:42:01 [35] burn took 5ms, real time so far: 3887ms, cpu time so far: 207ms
2019/03/08 16:42:01 [36] burn took 5ms, real time so far: 3993ms, cpu time so far: 214ms
2019/03/08 16:42:01 [37] burn took 5ms, real time so far: 4098ms, cpu time so far: 220ms
2019/03/08 16:42:01 [38] burn took 5ms, real time so far: 4204ms, cpu time so far: 227ms
2019/03/08 16:42:01 [39] burn took 5ms, real time so far: 4309ms, cpu time so far: 233ms
2019/03/08 16:42:02 [40] burn took 5ms, real time so far: 4414ms, cpu time so far: 239ms
2019/03/08 16:42:02 [41] burn took 5ms, real time so far: 4520ms, cpu time so far: 246ms
2019/03/08 16:42:02 [42] burn took 5ms, real time so far: 4626ms, cpu time so far: 251ms
2019/03/08 16:42:02 [43] burn took 7ms, real time so far: 4734ms, cpu time so far: 258ms
2019/03/08 16:42:02 [44] burn took 5ms, real time so far: 4839ms, cpu time so far: 265ms
2019/03/08 16:42:02 [45] burn took 5ms, real time so far: 4945ms, cpu time so far: 270ms
2019/03/08 16:42:02 [46] burn took 5ms, real time so far: 5050ms, cpu time so far: 276ms
2019/03/08 16:42:02 [47] burn took 5ms, real time so far: 5156ms, cpu time so far: 282ms
2019/03/08 16:42:02 [48] burn took 5ms, real time so far: 5262ms, cpu time so far: 288ms
2019/03/08 16:42:02 [49] burn took 5ms, real time so far: 5368ms, cpu time so far: 294ms
2019/03/08 16:42:03 [50] burn took 5ms, real time so far: 5473ms, cpu time so far: 300ms
2019/03/08 16:42:03 [51] burn took 5ms, real time so far: 5579ms, cpu time so far: 305ms
2019/03/08 16:42:03 [52] burn took 5ms, real time so far: 5684ms, cpu time so far: 310ms
2019/03/08 16:42:03 [53] burn took 97ms, real time so far: 5883ms, cpu time so far: 312ms
2019/03/08 16:42:03 [54] burn took 5ms, real time so far: 5988ms, cpu time so far: 317ms
2019/03/08 16:42:03 [55] burn took 5ms, real time so far: 6094ms, cpu time so far: 323ms
2019/03/08 16:42:03 [56] burn took 5ms, real time so far: 6199ms, cpu time so far: 328ms
2019/03/08 16:42:03 [57] burn took 5ms, real time so far: 6305ms, cpu time so far: 333ms
2019/03/08 16:42:04 [58] burn took 5ms, real time so far: 6411ms, cpu time so far: 340ms
2019/03/08 16:42:04 [59] burn took 5ms, real time so far: 6516ms, cpu time so far: 346ms
2019/03/08 16:42:04 [60] burn took 5ms, real time so far: 6623ms, cpu time so far: 352ms
2019/03/08 16:42:04 [61] burn took 5ms, real time so far: 6729ms, cpu time so far: 358ms
2019/03/08 16:42:04 [62] burn took 5ms, real time so far: 6836ms, cpu time so far: 366ms
2019/03/08 16:42:04 [63] burn took 5ms, real time so far: 6941ms, cpu time so far: 372ms
2019/03/08 16:42:04 [64] burn took 5ms, real time so far: 7047ms, cpu time so far: 377ms
2019/03/08 16:42:04 [65] burn took 5ms, real time so far: 7154ms, cpu time so far: 383ms
2019/03/08 16:42:04 [66] burn took 5ms, real time so far: 7259ms, cpu time so far: 388ms
2019/03/08 16:42:04 [67] burn took 5ms, real time so far: 7366ms, cpu time so far: 394ms
2019/03/08 16:42:05 [68] burn took 5ms, real time so far: 7472ms, cpu time so far: 399ms
2019/03/08 16:42:05 [69] burn took 6ms, real time so far: 7579ms, cpu time so far: 405ms
2019/03/08 16:42:05 [70] burn took 5ms, real time so far: 7686ms, cpu time so far: 411ms
2019/03/08 16:42:05 [71] burn took 5ms, real time so far: 7791ms, cpu time so far: 416ms
2019/03/08 16:42:05 [72] burn took 5ms, real time so far: 7897ms, cpu time so far: 421ms
2019/03/08 16:42:05 [73] burn took 5ms, real time so far: 8002ms, cpu time so far: 427ms
2019/03/08 16:42:05 [74] burn took 5ms, real time so far: 8108ms, cpu time so far: 432ms
2019/03/08 16:42:05 [75] burn took 5ms, real time so far: 8214ms, cpu time so far: 438ms
2019/03/08 16:42:05 [76] burn took 5ms, real time so far: 8319ms, cpu time so far: 444ms
2019/03/08 16:42:06 [77] burn took 5ms, real time so far: 8425ms, cpu time so far: 450ms
2019/03/08 16:42:06 [78] burn took 5ms, real time so far: 8531ms, cpu time so far: 455ms
2019/03/08 16:42:06 [79] burn took 5ms, real time so far: 8638ms, cpu time so far: 461ms
2019/03/08 16:42:06 [80] burn took 5ms, real time so far: 8743ms, cpu time so far: 466ms
2019/03/08 16:42:06 [81] burn took 5ms, real time so far: 8849ms, cpu time so far: 472ms
2019/03/08 16:42:06 [82] burn took 5ms, real time so far: 8955ms, cpu time so far: 477ms
2019/03/08 16:42:06 [83] burn took 5ms, real time so far: 9060ms, cpu time so far: 482ms
2019/03/08 16:42:06 [84] burn took 5ms, real time so far: 9166ms, cpu time so far: 488ms
2019/03/08 16:42:06 [85] burn took 5ms, real time so far: 9272ms, cpu time so far: 493ms
2019/03/08 16:42:07 [86] burn took 5ms, real time so far: 9378ms, cpu time so far: 498ms
2019/03/08 16:42:07 [87] burn took 7ms, real time so far: 9487ms, cpu time so far: 506ms
2019/03/08 16:42:07 [88] burn took 5ms, real time so far: 9592ms, cpu time so far: 512ms
2019/03/08 16:42:07 [89] burn took 5ms, real time so far: 9698ms, cpu time so far: 518ms
2019/03/08 16:42:07 [90] burn took 5ms, real time so far: 9804ms, cpu time so far: 524ms
2019/03/08 16:42:07 [91] burn took 5ms, real time so far: 9911ms, cpu time so far: 529ms
2019/03/08 16:42:07 [92] burn took 5ms, real time so far: 10016ms, cpu time so far: 535ms
2019/03/08 16:42:07 [93] burn took 5ms, real time so far: 10122ms, cpu time so far: 540ms
2019/03/08 16:42:07 [94] burn took 5ms, real time so far: 10228ms, cpu time so far: 545ms
2019/03/08 16:42:07 [95] burn took 5ms, real time so far: 10333ms, cpu time so far: 550ms
2019/03/08 16:42:08 [96] burn took 5ms, real time so far: 10439ms, cpu time so far: 556ms
2019/03/08 16:42:08 [97] burn took 5ms, real time so far: 10545ms, cpu time so far: 562ms
2019/03/08 16:42:08 [98] burn took 5ms, real time so far: 10650ms, cpu time so far: 568ms
2019/03/08 16:42:08 [99] burn took 5ms, real time so far: 10755ms, cpu time so far: 574ms

docker run --rm -it --cpu-quota 20000 --cpu-period 100000 -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 100 -sleep 1000ms
2019/03/08 16:42:21 [0] burn took 98ms, real time so far: 98ms, cpu time so far: 8ms
2019/03/08 16:42:22 [1] burn took 5ms, real time so far: 1104ms, cpu time so far: 14ms
2019/03/08 16:42:24 [2] burn took 99ms, real time so far: 2298ms, cpu time so far: 20ms
2019/03/08 16:42:25 [3] burn took 101ms, real time so far: 3399ms, cpu time so far: 24ms
2019/03/08 16:42:26 [4] burn took 98ms, real time so far: 4498ms, cpu time so far: 30ms
2019/03/08 16:42:27 [5] burn took 100ms, real time so far: 5599ms, cpu time so far: 36ms
2019/03/08 16:42:28 [6] burn took 5ms, real time so far: 6605ms, cpu time so far: 42ms
2019/03/08 16:42:29 [7] burn took 99ms, real time so far: 7798ms, cpu time so far: 45ms
2019/03/08 16:42:30 [8] burn took 100ms, real time so far: 8898ms, cpu time so far: 47ms
2019/03/08 16:42:31 [9] burn took 100ms, real time so far: 9999ms, cpu time so far: 48ms
2019/03/08 16:42:32 [10] burn took 98ms, real time so far: 11098ms, cpu time so far: 53ms
2019/03/08 16:42:34 [11] burn took 99ms, real time so far: 12198ms, cpu time so far: 55ms
2019/03/08 16:42:35 [12] burn took 100ms, real time so far: 13298ms, cpu time so far: 58ms
2019/03/08 16:42:36 [13] burn took 98ms, real time so far: 14398ms, cpu time so far: 60ms
2019/03/08 16:42:37 [14] burn took 99ms, real time so far: 15498ms, cpu time so far: 62ms
2019/03/08 16:42:38 [15] burn took 99ms, real time so far: 16598ms, cpu time so far: 64ms
2019/03/08 16:42:39 [16] burn took 99ms, real time so far: 17798ms, cpu time so far: 70ms
2019/03/08 16:42:40 [17] burn took 98ms, real time so far: 18898ms, cpu time so far: 72ms
2019/03/08 16:42:41 [18] burn took 100ms, real time so far: 19998ms, cpu time so far: 76ms
2019/03/08 16:42:42 [19] burn took 98ms, real time so far: 21098ms, cpu time so far: 83ms
2019/03/08 16:42:44 [20] burn took 99ms, real time so far: 22198ms, cpu time so far: 88ms
2019/03/08 16:42:45 [21] burn took 100ms, real time so far: 23299ms, cpu time so far: 93ms
2019/03/08 16:42:46 [22] burn took 98ms, real time so far: 24398ms, cpu time so far: 95ms
2019/03/08 16:42:47 [23] burn took 99ms, real time so far: 25498ms, cpu time so far: 101ms
2019/03/08 16:42:48 [24] burn took 98ms, real time so far: 26598ms, cpu time so far: 106ms
2019/03/08 16:42:49 [25] burn took 99ms, real time so far: 27698ms, cpu time so far: 112ms
2019/03/08 16:42:50 [26] burn took 99ms, real time so far: 28798ms, cpu time so far: 118ms
2019/03/08 16:42:51 [27] burn took 7ms, real time so far: 29806ms, cpu time so far: 126ms
2019/03/08 16:42:52 [28] burn took 97ms, real time so far: 30998ms, cpu time so far: 131ms
2019/03/08 16:42:53 [29] burn took 99ms, real time so far: 32098ms, cpu time so far: 136ms
2019/03/08 16:42:55 [30] burn took 98ms, real time so far: 33198ms, cpu time so far: 139ms
2019/03/08 16:42:56 [31] burn took 99ms, real time so far: 34298ms, cpu time so far: 145ms
2019/03/08 16:42:57 [32] burn took 5ms, real time so far: 35304ms, cpu time so far: 151ms
2019/03/08 16:42:58 [33] burn took 98ms, real time so far: 36498ms, cpu time so far: 157ms
2019/03/08 16:42:59 [34] burn took 5ms, real time so far: 37503ms, cpu time so far: 163ms
2019/03/08 16:43:00 [35] burn took 98ms, real time so far: 38698ms, cpu time so far: 171ms
2019/03/08 16:43:01 [36] burn took 98ms, real time so far: 39798ms, cpu time so far: 177ms
2019/03/08 16:43:02 [37] burn took 99ms, real time so far: 40898ms, cpu time so far: 183ms
2019/03/08 16:43:03 [38] burn took 99ms, real time so far: 41998ms, cpu time so far: 187ms
2019/03/08 16:43:04 [39] burn took 99ms, real time so far: 43098ms, cpu time so far: 193ms
2019/03/08 16:43:06 [40] burn took 98ms, real time so far: 44198ms, cpu time so far: 197ms
2019/03/08 16:43:07 [41] burn took 5ms, real time so far: 45203ms, cpu time so far: 202ms
2019/03/08 16:43:08 [42] burn took 99ms, real time so far: 46398ms, cpu time so far: 209ms
2019/03/08 16:43:09 [43] burn took 99ms, real time so far: 47498ms, cpu time so far: 214ms
2019/03/08 16:43:10 [44] burn took 98ms, real time so far: 48598ms, cpu time so far: 218ms
2019/03/08 16:43:11 [45] burn took 98ms, real time so far: 49698ms, cpu time so far: 223ms
2019/03/08 16:43:12 [46] burn took 99ms, real time so far: 50798ms, cpu time so far: 229ms
2019/03/08 16:43:13 [47] burn took 99ms, real time so far: 51898ms, cpu time so far: 235ms
2019/03/08 16:43:14 [48] burn took 99ms, real time so far: 52998ms, cpu time so far: 237ms
2019/03/08 16:43:15 [49] burn took 99ms, real time so far: 54098ms, cpu time so far: 238ms
2019/03/08 16:43:17 [50] burn took 99ms, real time so far: 55198ms, cpu time so far: 241ms
2019/03/08 16:43:18 [51] burn took 99ms, real time so far: 56298ms, cpu time so far: 242ms
2019/03/08 16:43:19 [52] burn took 99ms, real time so far: 57398ms, cpu time so far: 248ms
2019/03/08 16:43:20 [53] burn took 99ms, real time so far: 58498ms, cpu time so far: 253ms
2019/03/08 16:43:21 [54] burn took 99ms, real time so far: 59598ms, cpu time so far: 257ms
2019/03/08 16:43:22 [55] burn took 99ms, real time so far: 60698ms, cpu time so far: 262ms
2019/03/08 16:43:23 [56] burn took 99ms, real time so far: 61798ms, cpu time so far: 263ms
2019/03/08 16:43:24 [57] burn took 99ms, real time so far: 62898ms, cpu time so far: 266ms
2019/03/08 16:43:25 [58] burn took 100ms, real time so far: 63998ms, cpu time so far: 268ms
2019/03/08 16:43:26 [59] burn took 98ms, real time so far: 65098ms, cpu time so far: 275ms
2019/03/08 16:43:28 [60] burn took 99ms, real time so far: 66198ms, cpu time so far: 281ms
2019/03/08 16:43:29 [61] burn took 98ms, real time so far: 67298ms, cpu time so far: 286ms
2019/03/08 16:43:30 [62] burn took 98ms, real time so far: 68398ms, cpu time so far: 288ms
2019/03/08 16:43:31 [63] burn took 100ms, real time so far: 69498ms, cpu time so far: 291ms
2019/03/08 16:43:32 [64] burn took 98ms, real time so far: 70598ms, cpu time so far: 293ms
2019/03/08 16:43:33 [65] burn took 99ms, real time so far: 71698ms, cpu time so far: 295ms
2019/03/08 16:43:34 [66] burn took 99ms, real time so far: 72798ms, cpu time so far: 297ms
2019/03/08 16:43:35 [67] burn took 99ms, real time so far: 73898ms, cpu time so far: 299ms
2019/03/08 16:43:36 [68] burn took 99ms, real time so far: 74998ms, cpu time so far: 301ms
2019/03/08 16:43:37 [69] burn took 100ms, real time so far: 76099ms, cpu time so far: 306ms
2019/03/08 16:43:39 [70] burn took 98ms, real time so far: 77198ms, cpu time so far: 310ms
2019/03/08 16:43:40 [71] burn took 99ms, real time so far: 78298ms, cpu time so far: 312ms
2019/03/08 16:43:41 [72] burn took 100ms, real time so far: 79398ms, cpu time so far: 314ms
2019/03/08 16:43:42 [73] burn took 99ms, real time so far: 80498ms, cpu time so far: 316ms
2019/03/08 16:43:43 [74] burn took 99ms, real time so far: 81598ms, cpu time so far: 317ms
2019/03/08 16:43:44 [75] burn took 99ms, real time so far: 82698ms, cpu time so far: 319ms
2019/03/08 16:43:45 [76] burn took 99ms, real time so far: 83798ms, cpu time so far: 320ms
2019/03/08 16:43:46 [77] burn took 99ms, real time so far: 84898ms, cpu time so far: 322ms
2019/03/08 16:43:47 [78] burn took 98ms, real time so far: 85998ms, cpu time so far: 326ms
2019/03/08 16:43:48 [79] burn took 99ms, real time so far: 87098ms, cpu time so far: 328ms
2019/03/08 16:43:50 [80] burn took 100ms, real time so far: 88199ms, cpu time so far: 332ms
2019/03/08 16:43:51 [81] burn took 98ms, real time so far: 89298ms, cpu time so far: 334ms
2019/03/08 16:43:52 [82] burn took 98ms, real time so far: 90398ms, cpu time so far: 336ms
2019/03/08 16:43:53 [83] burn took 98ms, real time so far: 91498ms, cpu time so far: 341ms
2019/03/08 16:43:54 [84] burn took 99ms, real time so far: 92598ms, cpu time so far: 347ms
2019/03/08 16:43:55 [85] burn took 99ms, real time so far: 93698ms, cpu time so far: 353ms
2019/03/08 16:43:56 [86] burn took 99ms, real time so far: 94798ms, cpu time so far: 359ms
2019/03/08 16:43:57 [87] burn took 98ms, real time so far: 95898ms, cpu time so far: 367ms
2019/03/08 16:43:58 [88] burn took 99ms, real time so far: 96998ms, cpu time so far: 373ms
2019/03/08 16:43:59 [89] burn took 98ms, real time so far: 98098ms, cpu time so far: 379ms
2019/03/08 16:44:01 [90] burn took 100ms, real time so far: 99198ms, cpu time so far: 384ms
2019/03/08 16:44:02 [91] burn took 5ms, real time so far: 100204ms, cpu time so far: 390ms
2019/03/08 16:44:03 [92] burn took 99ms, real time so far: 101398ms, cpu time so far: 396ms
2019/03/08 16:44:04 [93] burn took 5ms, real time so far: 102404ms, cpu time so far: 402ms
2019/03/08 16:44:05 [94] burn took 5ms, real time so far: 103504ms, cpu time so far: 408ms
2019/03/08 16:44:06 [95] burn took 99ms, real time so far: 104698ms, cpu time so far: 413ms
2019/03/08 16:44:07 [96] burn took 99ms, real time so far: 105798ms, cpu time so far: 418ms
2019/03/08 16:44:08 [97] burn took 99ms, real time so far: 106898ms, cpu time so far: 425ms
2019/03/08 16:44:09 [98] burn took 102ms, real time so far: 108002ms, cpu time so far: 430ms
2019/03/08 16:44:10 [99] burn took 94ms, real time so far: 109098ms, cpu time so far: 436ms

docker run --rm -it --cpu-quota 20000 --cpu-period 100000 -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 10 -sleep 5000ms
2019/03/08 16:44:25 [0] burn took 90ms, real time so far: 90ms, cpu time so far: 6ms
2019/03/08 16:44:30 [1] burn took 98ms, real time so far: 5190ms, cpu time so far: 13ms
2019/03/08 16:44:35 [2] burn took 99ms, real time so far: 10291ms, cpu time so far: 19ms
2019/03/08 16:44:40 [3] burn took 99ms, real time so far: 15390ms, cpu time so far: 23ms
2019/03/08 16:44:45 [4] burn took 98ms, real time so far: 20491ms, cpu time so far: 31ms
2019/03/08 16:44:50 [5] burn took 98ms, real time so far: 25590ms, cpu time so far: 37ms
2019/03/08 16:44:56 [6] burn took 100ms, real time so far: 30691ms, cpu time so far: 43ms
2019/03/08 16:45:01 [7] burn took 99ms, real time so far: 35791ms, cpu time so far: 47ms
2019/03/08 16:45:06 [8] burn took 99ms, real time so far: 40891ms, cpu time so far: 50ms
2019/03/08 16:45:11 [9] burn took 98ms, real time so far: 45990ms, cpu time so far: 55ms
CoreOS Container Linux Stable 2023.4.0 on AWS EC2 running Linux Kernel 4.19.23 – OK 
docker run --rm -it --cpu-quota 20000 --cpu-period 100000 -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 100 -sleep 100ms
2019/03/08 15:59:16 [0] burn took 5ms, real time so far: 5ms, cpu time so far: 6ms
2019/03/08 15:59:16 [1] burn took 5ms, real time so far: 110ms, cpu time so far: 12ms
2019/03/08 15:59:16 [2] burn took 5ms, real time so far: 215ms, cpu time so far: 18ms
2019/03/08 15:59:16 [3] burn took 5ms, real time so far: 320ms, cpu time so far: 23ms
2019/03/08 15:59:17 [4] burn took 5ms, real time so far: 426ms, cpu time so far: 29ms
2019/03/08 15:59:17 [5] burn took 5ms, real time so far: 531ms, cpu time so far: 35ms
2019/03/08 15:59:17 [6] burn took 5ms, real time so far: 636ms, cpu time so far: 40ms
2019/03/08 15:59:17 [7] burn took 5ms, real time so far: 741ms, cpu time so far: 46ms
2019/03/08 15:59:17 [8] burn took 5ms, real time so far: 847ms, cpu time so far: 51ms
2019/03/08 15:59:17 [9] burn took 5ms, real time so far: 952ms, cpu time so far: 57ms
2019/03/08 15:59:17 [10] burn took 5ms, real time so far: 1057ms, cpu time so far: 63ms
2019/03/08 15:59:17 [11] burn took 5ms, real time so far: 1162ms, cpu time so far: 68ms
2019/03/08 15:59:17 [12] burn took 5ms, real time so far: 1268ms, cpu time so far: 74ms
2019/03/08 15:59:18 [13] burn took 5ms, real time so far: 1373ms, cpu time so far: 79ms
2019/03/08 15:59:18 [14] burn took 5ms, real time so far: 1478ms, cpu time so far: 85ms
2019/03/08 15:59:18 [15] burn took 5ms, real time so far: 1583ms, cpu time so far: 90ms
2019/03/08 15:59:18 [16] burn took 5ms, real time so far: 1689ms, cpu time so far: 96ms
2019/03/08 15:59:18 [17] burn took 5ms, real time so far: 1794ms, cpu time so far: 102ms
2019/03/08 15:59:18 [18] burn took 5ms, real time so far: 1899ms, cpu time so far: 107ms
2019/03/08 15:59:18 [19] burn took 5ms, real time so far: 2004ms, cpu time so far: 113ms
2019/03/08 15:59:18 [20] burn took 5ms, real time so far: 2110ms, cpu time so far: 118ms
2019/03/08 15:59:18 [21] burn took 5ms, real time so far: 2215ms, cpu time so far: 127ms
2019/03/08 15:59:18 [22] burn took 5ms, real time so far: 2320ms, cpu time so far: 133ms
2019/03/08 15:59:19 [23] burn took 5ms, real time so far: 2425ms, cpu time so far: 138ms
2019/03/08 15:59:19 [24] burn took 5ms, real time so far: 2531ms, cpu time so far: 144ms
2019/03/08 15:59:19 [25] burn took 5ms, real time so far: 2636ms, cpu time so far: 149ms
2019/03/08 15:59:19 [26] burn took 5ms, real time so far: 2741ms, cpu time so far: 155ms
2019/03/08 15:59:19 [27] burn took 5ms, real time so far: 2846ms, cpu time so far: 161ms
2019/03/08 15:59:19 [28] burn took 5ms, real time so far: 2952ms, cpu time so far: 166ms
2019/03/08 15:59:19 [29] burn took 5ms, real time so far: 3057ms, cpu time so far: 172ms
2019/03/08 15:59:19 [30] burn took 5ms, real time so far: 3162ms, cpu time so far: 177ms
2019/03/08 15:59:19 [31] burn took 5ms, real time so far: 3268ms, cpu time so far: 183ms
2019/03/08 15:59:20 [32] burn took 5ms, real time so far: 3373ms, cpu time so far: 189ms
2019/03/08 15:59:20 [33] burn took 5ms, real time so far: 3478ms, cpu time so far: 194ms
2019/03/08 15:59:20 [34] burn took 5ms, real time so far: 3583ms, cpu time so far: 200ms
2019/03/08 15:59:20 [35] burn took 5ms, real time so far: 3689ms, cpu time so far: 205ms
2019/03/08 15:59:20 [36] burn took 5ms, real time so far: 3794ms, cpu time so far: 211ms
2019/03/08 15:59:20 [37] burn took 5ms, real time so far: 3899ms, cpu time so far: 216ms
2019/03/08 15:59:20 [38] burn took 5ms, real time so far: 4004ms, cpu time so far: 222ms
2019/03/08 15:59:20 [39] burn took 5ms, real time so far: 4110ms, cpu time so far: 228ms
2019/03/08 15:59:20 [40] burn took 5ms, real time so far: 4215ms, cpu time so far: 233ms
2019/03/08 15:59:20 [41] burn took 5ms, real time so far: 4320ms, cpu time so far: 239ms
2019/03/08 15:59:21 [42] burn took 5ms, real time so far: 4425ms, cpu time so far: 244ms
2019/03/08 15:59:21 [43] burn took 5ms, real time so far: 4531ms, cpu time so far: 251ms
2019/03/08 15:59:21 [44] burn took 5ms, real time so far: 4636ms, cpu time so far: 259ms
2019/03/08 15:59:21 [45] burn took 5ms, real time so far: 4741ms, cpu time so far: 265ms
2019/03/08 15:59:21 [46] burn took 5ms, real time so far: 4846ms, cpu time so far: 270ms
2019/03/08 15:59:21 [47] burn took 5ms, real time so far: 4952ms, cpu time so far: 276ms
2019/03/08 15:59:21 [48] burn took 5ms, real time so far: 5057ms, cpu time so far: 281ms
2019/03/08 15:59:21 [49] burn took 5ms, real time so far: 5162ms, cpu time so far: 287ms
2019/03/08 15:59:21 [50] burn took 5ms, real time so far: 5267ms, cpu time so far: 292ms
2019/03/08 15:59:22 [51] burn took 5ms, real time so far: 5373ms, cpu time so far: 298ms
2019/03/08 15:59:22 [52] burn took 5ms, real time so far: 5478ms, cpu time so far: 304ms
2019/03/08 15:59:22 [53] burn took 5ms, real time so far: 5583ms, cpu time so far: 309ms
2019/03/08 15:59:22 [54] burn took 5ms, real time so far: 5688ms, cpu time so far: 315ms
2019/03/08 15:59:22 [55] burn took 5ms, real time so far: 5794ms, cpu time so far: 320ms
2019/03/08 15:59:22 [56] burn took 5ms, real time so far: 5899ms, cpu time so far: 326ms
2019/03/08 15:59:22 [57] burn took 5ms, real time so far: 6004ms, cpu time so far: 332ms
2019/03/08 15:59:22 [58] burn took 5ms, real time so far: 6109ms, cpu time so far: 337ms
2019/03/08 15:59:22 [59] burn took 5ms, real time so far: 6215ms, cpu time so far: 343ms
2019/03/08 15:59:22 [60] burn took 5ms, real time so far: 6320ms, cpu time so far: 349ms
2019/03/08 15:59:23 [61] burn took 5ms, real time so far: 6425ms, cpu time so far: 354ms
2019/03/08 15:59:23 [62] burn took 5ms, real time so far: 6530ms, cpu time so far: 360ms
2019/03/08 15:59:23 [63] burn took 5ms, real time so far: 6636ms, cpu time so far: 366ms
2019/03/08 15:59:23 [64] burn took 5ms, real time so far: 6741ms, cpu time so far: 371ms
2019/03/08 15:59:23 [65] burn took 5ms, real time so far: 6846ms, cpu time so far: 377ms
2019/03/08 15:59:23 [66] burn took 5ms, real time so far: 6951ms, cpu time so far: 385ms
2019/03/08 15:59:23 [67] burn took 5ms, real time so far: 7057ms, cpu time so far: 391ms
2019/03/08 15:59:23 [68] burn took 5ms, real time so far: 7162ms, cpu time so far: 397ms
2019/03/08 15:59:23 [69] burn took 5ms, real time so far: 7267ms, cpu time so far: 402ms
2019/03/08 15:59:24 [70] burn took 5ms, real time so far: 7372ms, cpu time so far: 408ms
2019/03/08 15:59:24 [71] burn took 5ms, real time so far: 7477ms, cpu time so far: 413ms
2019/03/08 15:59:24 [72] burn took 5ms, real time so far: 7583ms, cpu time so far: 419ms
2019/03/08 15:59:24 [73] burn took 5ms, real time so far: 7688ms, cpu time so far: 425ms
2019/03/08 15:59:24 [74] burn took 5ms, real time so far: 7793ms, cpu time so far: 430ms
2019/03/08 15:59:24 [75] burn took 5ms, real time so far: 7898ms, cpu time so far: 436ms
2019/03/08 15:59:24 [76] burn took 5ms, real time so far: 8004ms, cpu time so far: 441ms
2019/03/08 15:59:24 [77] burn took 5ms, real time so far: 8109ms, cpu time so far: 447ms
2019/03/08 15:59:24 [78] burn took 5ms, real time so far: 8214ms, cpu time so far: 452ms
2019/03/08 15:59:24 [79] burn took 5ms, real time so far: 8319ms, cpu time so far: 458ms
2019/03/08 15:59:25 [80] burn took 5ms, real time so far: 8425ms, cpu time so far: 464ms
2019/03/08 15:59:25 [81] burn took 5ms, real time so far: 8530ms, cpu time so far: 468ms
2019/03/08 15:59:25 [82] burn took 5ms, real time so far: 8635ms, cpu time so far: 473ms
2019/03/08 15:59:25 [83] burn took 5ms, real time so far: 8741ms, cpu time so far: 478ms
2019/03/08 15:59:25 [84] burn took 5ms, real time so far: 8846ms, cpu time so far: 483ms
2019/03/08 15:59:25 [85] burn took 5ms, real time so far: 8951ms, cpu time so far: 487ms
2019/03/08 15:59:25 [86] burn took 5ms, real time so far: 9057ms, cpu time so far: 492ms
2019/03/08 15:59:25 [87] burn took 5ms, real time so far: 9174ms, cpu time so far: 496ms
2019/03/08 15:59:25 [88] burn took 5ms, real time so far: 9279ms, cpu time so far: 501ms
2019/03/08 15:59:26 [89] burn took 5ms, real time so far: 9385ms, cpu time so far: 506ms
2019/03/08 15:59:26 [90] burn took 5ms, real time so far: 9490ms, cpu time so far: 510ms
2019/03/08 15:59:26 [91] burn took 5ms, real time so far: 9595ms, cpu time so far: 517ms
2019/03/08 15:59:26 [92] burn took 5ms, real time so far: 9701ms, cpu time so far: 523ms
2019/03/08 15:59:26 [93] burn took 5ms, real time so far: 9806ms, cpu time so far: 529ms
2019/03/08 15:59:26 [94] burn took 5ms, real time so far: 9911ms, cpu time so far: 534ms
2019/03/08 15:59:26 [95] burn took 5ms, real time so far: 10017ms, cpu time so far: 539ms
2019/03/08 15:59:26 [96] burn took 5ms, real time so far: 10122ms, cpu time so far: 544ms
2019/03/08 15:59:26 [97] burn took 5ms, real time so far: 10227ms, cpu time so far: 549ms
2019/03/08 15:59:26 [98] burn took 5ms, real time so far: 10333ms, cpu time so far: 554ms
2019/03/08 15:59:27 [99] burn took 5ms, real time so far: 10438ms, cpu time so far: 560ms

docker run --rm -it --cpu-quota 20000 --cpu-period 100000 -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 100 -sleep 1000ms
2019/03/08 15:59:41 [0] burn took 5ms, real time so far: 5ms, cpu time so far: 7ms
2019/03/08 15:59:42 [1] burn took 5ms, real time so far: 1010ms, cpu time so far: 12ms
2019/03/08 15:59:43 [2] burn took 5ms, real time so far: 2015ms, cpu time so far: 17ms
2019/03/08 15:59:44 [3] burn took 5ms, real time so far: 3021ms, cpu time so far: 23ms
2019/03/08 15:59:45 [4] burn took 5ms, real time so far: 4026ms, cpu time so far: 29ms
2019/03/08 15:59:46 [5] burn took 5ms, real time so far: 5031ms, cpu time so far: 34ms
2019/03/08 15:59:47 [6] burn took 5ms, real time so far: 6036ms, cpu time so far: 40ms
2019/03/08 15:59:48 [7] burn took 5ms, real time so far: 7042ms, cpu time so far: 45ms
2019/03/08 15:59:49 [8] burn took 5ms, real time so far: 8047ms, cpu time so far: 51ms
2019/03/08 15:59:50 [9] burn took 5ms, real time so far: 9052ms, cpu time so far: 57ms
2019/03/08 15:59:51 [10] burn took 5ms, real time so far: 10057ms, cpu time so far: 62ms
2019/03/08 15:59:52 [11] burn took 5ms, real time so far: 11063ms, cpu time so far: 68ms
2019/03/08 15:59:53 [12] burn took 5ms, real time so far: 12068ms, cpu time so far: 73ms
2019/03/08 15:59:54 [13] burn took 5ms, real time so far: 13073ms, cpu time so far: 79ms
2019/03/08 15:59:55 [14] burn took 5ms, real time so far: 14079ms, cpu time so far: 84ms
2019/03/08 15:59:56 [15] burn took 5ms, real time so far: 15084ms, cpu time so far: 90ms
2019/03/08 15:59:57 [16] burn took 5ms, real time so far: 16089ms, cpu time so far: 95ms
2019/03/08 15:59:58 [17] burn took 5ms, real time so far: 17094ms, cpu time so far: 101ms
2019/03/08 15:59:59 [18] burn took 5ms, real time so far: 18100ms, cpu time so far: 107ms
2019/03/08 16:00:00 [19] burn took 5ms, real time so far: 19105ms, cpu time so far: 112ms
2019/03/08 16:00:01 [20] burn took 5ms, real time so far: 20110ms, cpu time so far: 118ms
2019/03/08 16:00:02 [21] burn took 5ms, real time so far: 21115ms, cpu time so far: 123ms
2019/03/08 16:00:03 [22] burn took 5ms, real time so far: 22120ms, cpu time so far: 132ms
2019/03/08 16:00:04 [23] burn took 5ms, real time so far: 23126ms, cpu time so far: 137ms
2019/03/08 16:00:05 [24] burn took 5ms, real time so far: 24131ms, cpu time so far: 143ms
2019/03/08 16:00:06 [25] burn took 5ms, real time so far: 25136ms, cpu time so far: 149ms
2019/03/08 16:00:07 [26] burn took 5ms, real time so far: 26141ms, cpu time so far: 154ms
2019/03/08 16:00:08 [27] burn took 5ms, real time so far: 27147ms, cpu time so far: 160ms
2019/03/08 16:00:09 [28] burn took 5ms, real time so far: 28152ms, cpu time so far: 165ms
2019/03/08 16:00:10 [29] burn took 5ms, real time so far: 29157ms, cpu time so far: 170ms
2019/03/08 16:00:11 [30] burn took 5ms, real time so far: 30162ms, cpu time so far: 175ms
2019/03/08 16:00:12 [31] burn took 5ms, real time so far: 31168ms, cpu time so far: 180ms
2019/03/08 16:00:13 [32] burn took 5ms, real time so far: 32173ms, cpu time so far: 185ms
2019/03/08 16:00:14 [33] burn took 5ms, real time so far: 33178ms, cpu time so far: 191ms
2019/03/08 16:00:15 [34] burn took 5ms, real time so far: 34184ms, cpu time so far: 197ms
2019/03/08 16:00:16 [35] burn took 5ms, real time so far: 35189ms, cpu time so far: 202ms
2019/03/08 16:00:17 [36] burn took 5ms, real time so far: 36194ms, cpu time so far: 208ms
2019/03/08 16:00:18 [37] burn took 5ms, real time so far: 37199ms, cpu time so far: 214ms
2019/03/08 16:00:19 [38] burn took 5ms, real time so far: 38205ms, cpu time so far: 219ms
2019/03/08 16:00:20 [39] burn took 5ms, real time so far: 39210ms, cpu time so far: 225ms
2019/03/08 16:00:21 [40] burn took 5ms, real time so far: 40215ms, cpu time so far: 230ms
2019/03/08 16:00:22 [41] burn took 5ms, real time so far: 41220ms, cpu time so far: 236ms
2019/03/08 16:00:23 [42] burn took 5ms, real time so far: 42226ms, cpu time so far: 241ms
2019/03/08 16:00:24 [43] burn took 5ms, real time so far: 43231ms, cpu time so far: 247ms
2019/03/08 16:00:25 [44] burn took 5ms, real time so far: 44236ms, cpu time so far: 253ms
2019/03/08 16:00:26 [45] burn took 14ms, real time so far: 45251ms, cpu time so far: 264ms
2019/03/08 16:00:27 [46] burn took 5ms, real time so far: 46256ms, cpu time so far: 275ms
2019/03/08 16:00:28 [47] burn took 5ms, real time so far: 47261ms, cpu time so far: 281ms
2019/03/08 16:00:29 [48] burn took 5ms, real time so far: 48266ms, cpu time so far: 286ms
2019/03/08 16:00:30 [49] burn took 5ms, real time so far: 49272ms, cpu time so far: 292ms
2019/03/08 16:00:31 [50] burn took 5ms, real time so far: 50277ms, cpu time so far: 298ms
2019/03/08 16:00:32 [51] burn took 5ms, real time so far: 51282ms, cpu time so far: 303ms
2019/03/08 16:00:33 [52] burn took 5ms, real time so far: 52287ms, cpu time so far: 309ms
2019/03/08 16:00:34 [53] burn took 5ms, real time so far: 53293ms, cpu time so far: 314ms
2019/03/08 16:00:35 [54] burn took 5ms, real time so far: 54298ms, cpu time so far: 320ms
2019/03/08 16:00:36 [55] burn took 5ms, real time so far: 55303ms, cpu time so far: 325ms
2019/03/08 16:00:37 [56] burn took 5ms, real time so far: 56308ms, cpu time so far: 331ms
2019/03/08 16:00:38 [57] burn took 5ms, real time so far: 57314ms, cpu time so far: 337ms
2019/03/08 16:00:39 [58] burn took 5ms, real time so far: 58319ms, cpu time so far: 342ms
2019/03/08 16:00:40 [59] burn took 5ms, real time so far: 59324ms, cpu time so far: 347ms
2019/03/08 16:00:41 [60] burn took 5ms, real time so far: 60330ms, cpu time so far: 352ms
2019/03/08 16:00:42 [61] burn took 11ms, real time so far: 61368ms, cpu time so far: 356ms
2019/03/08 16:00:43 [62] burn took 5ms, real time so far: 62373ms, cpu time so far: 362ms
2019/03/08 16:00:44 [63] burn took 5ms, real time so far: 63379ms, cpu time so far: 367ms
2019/03/08 16:00:45 [64] burn took 5ms, real time so far: 64384ms, cpu time so far: 373ms
2019/03/08 16:00:46 [65] burn took 5ms, real time so far: 65389ms, cpu time so far: 378ms
2019/03/08 16:00:47 [66] burn took 5ms, real time so far: 66394ms, cpu time so far: 384ms
2019/03/08 16:00:48 [67] burn took 5ms, real time so far: 67399ms, cpu time so far: 390ms
2019/03/08 16:00:49 [68] burn took 5ms, real time so far: 68405ms, cpu time so far: 395ms
2019/03/08 16:00:50 [69] burn took 5ms, real time so far: 69410ms, cpu time so far: 403ms
2019/03/08 16:00:51 [70] burn took 5ms, real time so far: 70415ms, cpu time so far: 409ms
2019/03/08 16:00:52 [71] burn took 5ms, real time so far: 71421ms, cpu time so far: 415ms
2019/03/08 16:00:53 [72] burn took 5ms, real time so far: 72426ms, cpu time so far: 420ms
2019/03/08 16:00:54 [73] burn took 5ms, real time so far: 73431ms, cpu time so far: 426ms
2019/03/08 16:00:55 [74] burn took 5ms, real time so far: 74436ms, cpu time so far: 431ms
2019/03/08 16:00:56 [75] burn took 5ms, real time so far: 75442ms, cpu time so far: 436ms
2019/03/08 16:00:57 [76] burn took 5ms, real time so far: 76447ms, cpu time so far: 442ms
2019/03/08 16:00:58 [77] burn took 5ms, real time so far: 77452ms, cpu time so far: 447ms
2019/03/08 16:00:59 [78] burn took 5ms, real time so far: 78457ms, cpu time so far: 453ms
2019/03/08 16:01:00 [79] burn took 5ms, real time so far: 79463ms, cpu time so far: 459ms
2019/03/08 16:01:01 [80] burn took 5ms, real time so far: 80468ms, cpu time so far: 464ms
2019/03/08 16:01:02 [81] burn took 5ms, real time so far: 81473ms, cpu time so far: 470ms
2019/03/08 16:01:03 [82] burn took 5ms, real time so far: 82478ms, cpu time so far: 475ms
2019/03/08 16:01:04 [83] burn took 5ms, real time so far: 83484ms, cpu time so far: 481ms
2019/03/08 16:01:05 [84] burn took 5ms, real time so far: 84489ms, cpu time so far: 487ms
2019/03/08 16:01:06 [85] burn took 5ms, real time so far: 85494ms, cpu time so far: 492ms
2019/03/08 16:01:07 [86] burn took 5ms, real time so far: 86499ms, cpu time so far: 498ms
2019/03/08 16:01:08 [87] burn took 5ms, real time so far: 87505ms, cpu time so far: 504ms
2019/03/08 16:01:09 [88] burn took 5ms, real time so far: 88510ms, cpu time so far: 509ms
2019/03/08 16:01:10 [89] burn took 5ms, real time so far: 89515ms, cpu time so far: 514ms
2019/03/08 16:01:11 [90] burn took 5ms, real time so far: 90520ms, cpu time so far: 519ms
2019/03/08 16:01:12 [91] burn took 5ms, real time so far: 91526ms, cpu time so far: 525ms
2019/03/08 16:01:13 [92] burn took 5ms, real time so far: 92531ms, cpu time so far: 531ms
2019/03/08 16:01:14 [93] burn took 5ms, real time so far: 93536ms, cpu time so far: 539ms
2019/03/08 16:01:15 [94] burn took 5ms, real time so far: 94542ms, cpu time so far: 544ms
2019/03/08 16:01:16 [95] burn took 5ms, real time so far: 95547ms, cpu time so far: 550ms
2019/03/08 16:01:17 [96] burn took 5ms, real time so far: 96552ms, cpu time so far: 556ms
2019/03/08 16:01:18 [97] burn took 5ms, real time so far: 97557ms, cpu time so far: 561ms
2019/03/08 16:01:19 [98] burn took 5ms, real time so far: 98563ms, cpu time so far: 567ms
2019/03/08 16:01:20 [99] burn took 5ms, real time so far: 99568ms, cpu time so far: 572ms

docker run --rm -it --cpu-quota 20000 --cpu-period 100000 -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 10 -sleep 5000ms
2019/03/08 16:01:36 [0] burn took 5ms, real time so far: 5ms, cpu time so far: 6ms
2019/03/08 16:01:41 [1] burn took 5ms, real time so far: 5010ms, cpu time so far: 12ms
2019/03/08 16:01:46 [2] burn took 5ms, real time so far: 10015ms, cpu time so far: 18ms
2019/03/08 16:01:51 [3] burn took 5ms, real time so far: 15020ms, cpu time so far: 23ms
2019/03/08 16:01:56 [4] burn took 5ms, real time so far: 20026ms, cpu time so far: 29ms
2019/03/08 16:02:01 [5] burn took 5ms, real time so far: 25031ms, cpu time so far: 35ms
2019/03/08 16:02:06 [6] burn took 5ms, real time so far: 30036ms, cpu time so far: 40ms
2019/03/08 16:02:11 [7] burn took 5ms, real time so far: 35041ms, cpu time so far: 46ms
2019/03/08 16:02:16 [8] burn took 5ms, real time so far: 40047ms, cpu time so far: 52ms
2019/03/08 16:02:21 [9] burn took 5ms, real time so far: 45052ms, cpu time so far: 57ms

@confiq
Copy link

confiq commented Feb 18, 2020

is there anything about this ^ in changelog of kernel ?

@Nuru
Copy link

Nuru commented Apr 22, 2020 •
edited
Loading

@confiq not exactly sure what "this" is. Seems like the root problem was the idea of expiring CFS quotas, and the problem was made more prominent by 512ac999 sched/fair: Fix bandwidth timer clock drift condition which was introduced to 4.14 in 4.14.95 and was in 4.18-rc4 and thus every 4.18 and later version until fixed. Dave Chiluk explains the history along with providing a fix in de53fd7ae sched/fair: Fix low cpu usage with high throttling by removing expiration of cpu-local slices which was released in Linux 5.4-rc1 and eventually got backported to kernels v4.14.154 && v4.19.84 but AFAIK never made it back into 4.9.

I still have not figured out why Debian stretch, based on the 4.9 kernel, shows the same bug, even though the kernel does not have either patch, and CoreOS 2023.4.0 running kernel 4.19.23 is NOT affected.

Note that Kubernetes throttling (which prompted the bulk of the interest in this bug) was observed by someone using CoreOS 4.19.25-coreos. This is to be expected, as the 4.19.25 kernel is known to have the bug, but together with this cfs.go test not showing any problems on CoreOS, suggests a limitation of this test. I see nothing in the change log to suggest a problem was introduced between 4.19.23 and 4.19.25 and indeed I expect all versions of 4.19, including release candidates, prior to 4.19.84 would have this problem.

You can read a lot more about this at kubernetes/kubernetes#67577

@alok87
Copy link

alok87 commented Apr 22, 2020

thanks for sharing this

@wuestkamp
Copy link

wuestkamp commented Jul 16, 2020 •
edited
Loading

Thanks for this test! But in our case it doesn't prove unwanted k8s throttling as described here.

On Ubuntu 4.4.0-159-generic|4.4.0-169-generic|4.4.0-185-generic|4.15.0-54-generic this test shows the throttling issues as well as Kubernetes pods.

On Ubuntu 4.15.0-109-generic this test shows no issues, but the unwanted throttling of Kubernetes pods still exist.

@tommynguyen-vungle
Copy link

Our testing result on EKS-AMI version 1.18 with kernel 4.14.219-164.354.amzn2.x86_64 on AWS show no throttling.
But the throttling in kubernetes pod is still observed.
See: https://gist.github.com/tommynguyen-vungle/d77f236681be7c004d38a81d8adc14df

@free-y
Copy link

free-y commented Dec 8, 2022 •
edited
Loading

@tommynguyen-vungle how did you observe the throttling in kubernetes pod?

Recently, I'm working on debuging the cpu throttling issue, with the following 5 tests, I've tested out the bug in kernel (Linux version 4.18.0-041800rc4-generic)

This test case is intended to hit 100% throttling for the test 5000ms / 100 ms periods = 50 periods. A kernel without this bug should be able to have a CPU usage stats about 500ms.

Maybe you can try these tests to check whether your kernel will be throttlled.

[Multi Thread Test 1]

./runfibtest 1; ./runfibtest

From <https://github.com/indeedeng/fibtest>

[Result]

Throttled

./runfibtest 1
Iterations Completed(M): 465 
Throttled for: 52 
CPU Usage (msecs) = 508

./runfibtest 8
Iterations Completed(M): 283 
Throttled for: 52 
CPU Usage (msecs) = 327

[Multi Thread Test 2]

docker run -it --rm --cpu-quota 10000 --cpu-period 100000 hipeteryang/fibtest:latest /bin/sh -c "runfibtest 8 && cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu"

[Result]

Throttled

Iterations Completed(M): 192 
Throttled for: 53 
CPU Usage (msecs) = 227
nr_periods 58
nr_throttled 56
throttled_time 10136239267
267434463
209397643 2871651 8044402 4226146 5891926 5532789 27939741 4104364

[Multi Thread Test 3]

docker run -it --rm --cpu-quota 10000 --cpu-period 100000 hipeteryang/stress-ng:cpu-delay /bin/sh -c "stress-ng --taskset 0 --cpu 1 --timeout 5s & stress-ng  --taskset 1-7 --cpu 7 --cpu-load-slice -1 --cpu-delay 10 --cpu-method fibonacci  --timeout 5s && cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu"

Result

Throttled

nr_periods 56
nr_throttled 53
throttled_time 7893876370
379589091
330166879 3073914 6581265 724144 706787 5605273 29455102 3849694

For the following kubernetes test, we can use "kubectl logs pod-name" to get the result once the job is done

[Multi Thread Test 4]

apiVersion: batch/v1
kind: Job
metadata:
  name: fibtest
  namespace: default
spec:
  template:
    spec:
      containers:
      - name: fibtest
        image: hipeteryang/fibtest
        command: ["/bin/bash", "-c", "runfibtest 8 && cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu"]
        resources:
          requests:
            cpu: "50m"
          limits:
            cpu: "100m"
      restartPolicy: Never

Result

Throttled

Iterations Completed(M): 195 
Throttled for: 52 
CPU Usage (msecs) = 230
nr_periods 56
nr_throttled 54
throttled_time 9667667360
255738571
213621103 4038989 2814638 15869649 4435168 5459186 4549675 5437010

[Multi Thread Test 5]

apiVersion: batch/v1
kind: Job
metadata:
  name: stress-ng-test
  namespace: default
spec:
  template:
    spec:
      containers:
      - name: stress-ng-test
        image: hipeteryang/stress-ng:cpu-delay
        command: ["/bin/bash", "-c", "stress-ng --taskset 0 --cpu 1 --timeout 5s & stress-ng  --taskset 1-7 --cpu 7 --cpu-load-slice -1 --cpu-delay 10 --cpu-method fibonacci  --timeout 5s && cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu
"]
        resources:
          requests:
            cpu: "50m"
          limits:
            cpu: "100m"
      restartPolicy: Never

Result

Throttled

nr_periods 53
nr_throttled 50
throttled_time 6827221694
417331622
381601924 1267814 8332150 3933171 13654620 184120 6376208 2623172

Feel free to leave any comment, I’ll reply as soon as possible.

@cstk502
Copy link

cstk502 commented Jan 8, 2024

@free-y hi free-y, is there any update about this issue ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment