bobrik

Overly aggressive CFS

We're looking at CPU bandwidth control via CFS:

• https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt

Reproduction program

Program does number of iterations, in each iteration we burn CPU in small chunks until we get 5ms of real time spent. On each iteration we also print how much

bobrik

Amcrest cleanup script

Handy if you want to keep the most recent Amcrest recordings around.

Example output:

$ ./cleanup.py -s $((400 * 1024)) /Volumes/cameras
Checked /Volumes/cameras/amcrest02/2023-11-11/001/dav
Checked /Volumes/cameras/amcrest02/2023-11-07/001/dav

bobrik

unix_gc overload

The issue

Linux kernel has a garbage collection mechanism for inflight unix sockets passed to other unix sockets. This mechanism can be used to cause excessive load onto well behaved processes that are using regular unix sockets without any fd passing, because garbage collection is called in the socket write path.

The repro

bobrik

Expensive memory.stat + cpu.stat

We're seeing issues where cadvisor spends a lot of time on CPU.

Repro

The following should run on any system with systemd (code is in this gist):

go run main.go

bobrik

Calls into skb:kfree_skb

As requested in netdev, this gist contains the stacks leading into skb:kfree_skb:

• https://lore.kernel.org/netdev/20230713201427.2c50fc7b@kernel.org/

The easiest way to look at the results is the flamegraphs, where reason is the top frame.

The results are from v6.1.38.

bobrik

{
  "cards": {
    "cardPadding": null,
    "cardRound": null
  },
  "color": {
    "cardColor": "#bf1b00",
    "colorScale": "sqrt",
    "colorScheme": "interpolateRdYlGn",
    "exponent": 0.25,

bobrik

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",

bobrik

{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {
          "__name__": "node_cpu",
          "colo_id": "28",
          "colo_name": "sea01",

bobrik

<?php

// download some country file (RU.zip for example), unzip it and run:
// php -d memory_limit=512M -f checker.php ~/RU.txt RU
// summary will appear in stderr, actual duplicates in stdout

if (count($argv) < 3) {
    fwrite(STDERR, "please specify path to file with cities and country code\n");
    exit(1);
}

bobrik

Redis KPTI benchmark

Other benchmarks make me sad. People run them in the cloud where VMs can migrate between different hosts on reboots and use old kernels. Here we use bare metal with the same recent kernel to remove extra variables and measure KPTI impact and KPTI impact only.

Setup

• Kernel: 4.14.11