Skip to content

Instantly share code, notes, and snippets.

@bobrik

bobrik/README.md

Last active August 11, 2023 21:52
Show Gist options
  • Save bobrik/5ba58fb75a48620a1965026ad30a0a13 to your computer and use it in GitHub Desktop.
Save bobrik/5ba58fb75a48620a1965026ad30a0a13 to your computer and use it in GitHub Desktop.
Download ZIP
Expensive `memory.stat` + `cpu.stat`
Raw
README.md

Expensive memory.stat + cpu.stat

We're seeing issues where cadvisor spends a lot of time on CPU.

Repro

The following should run on any system with systemd (code is in this gist):

go run main.go

The output for is is something like this:

completed:  4.32s [manual / mem-stat + cpu-stat]
completed:  4.25s [manual / cpu-stat + mem-stat]
completed:  0.47s [manual / mem-stat]
completed:  0.04s [manual / cpu-stat]
completed:  5.17s [manual / mem-stat + cpu-stat]
completed:  5.59s [manual / cpu-stat + mem-stat]
completed:  0.52s [manual / mem-stat]
completed:  0.04s [manual / cpu-stat]

The first three runs are a warm-up, and the second three runs is the meat and potatoes.

In cpu-stat + mem-stat we do effectively the following:

for _ in $(seq 1 1000); do cat /sys/fs/cgroup/system.slice/memory.stat /sys/fs/cgroup/system.slice/cpu.stat > /dev/null

In cpu-stat it's just cpu.stat file:

for _ in $(seq 1 1000); do cat /sys/fs/cgroup/system.slice/cpu.stat > /dev/null; done

In memory.stat it's just memory.stat file:

for _ in $(seq 1 1000); do cat /sys/fs/cgroup/system.slice/memory.stat > /dev/null; done

The weird thing is: combining the latter two is much faster than the former and there's easily 10x difference in the time spent in kernel between them.

Unfortunately, the former is how monitoring software normally works: all metrics are fetched for one service before proceeding to the next one. One prominent example that prompted this examination is cAdvisor:

Flamegraphs for a single combined loop (one-loop.svg) and for two loops back to back (two-loops.svg) are attached to this gist.

This doesn't happen on a mostly idle machine, but we do see this on loaded servers (128 busy logical CPUs, lots of memory churn).

Raw
main.go
package main
import (
"fmt"
"os"
"time"
)
var suffixes = map[string][]string{
"mem-all+cpu-stat": []string{"memory.stat", "memory.current", "memory.max", "memory.swap.current", "memory.swap.max", "cpu.stat"},
"mem-stat+cpu-stat": []string{"memory.stat", "cpu.stat"},
"cpu-stat+mem-stat": []string{"cpu.stat", "memory.stat"},
"mem-all": []string{"memory.stat", "memory.current", "memory.max", "memory.swap.current", "memory.swap.max"},
"mem-stat": []string{"memory.stat"},
"cpu-stat": []string{"cpu.stat"},
}
func main() {
onePathRepeated := make([]string, 1000)
for i := range onePathRepeated {
onePathRepeated[i] = "system.slice"
}
// Warm up round
runBenchmarks(onePathRepeated)
// Real run
runBenchmarks(onePathRepeated)
}
func runBenchmarks(paths []string) {
// benchmarkFunc("manual / mem-all + cpu-stat", func() {
// manualBenchmark(paths, suffixes["mem-all+cpu-stat"])
// })
benchmarkFunc("manual / mem-stat + cpu-stat", func() {
manualBenchmark(paths, suffixes["mem-stat+cpu-stat"])
})
benchmarkFunc("manual / cpu-stat + mem-stat", func() {
manualBenchmark(paths, suffixes["cpu-stat+mem-stat"])
})
// benchmarkFunc("manual / mem-all", func() {
// manualBenchmark(paths, suffixes["mem-all"])
// })
benchmarkFunc("manual / mem-stat", func() {
manualBenchmark(paths, suffixes["mem-stat"])
})
benchmarkFunc("manual / cpu-stat", func() {
manualBenchmark(paths, suffixes["cpu-stat"])
})
}
func benchmarkFunc(name string, bench func()) {
started := time.Now()
bench()
fmt.Printf("completed: %5.2fs [%s]\n", time.Since(started).Seconds(), name)
}
func manualBenchmark(paths []string, suffixes []string) {
for _, path := range paths {
_, err := getStatsManual("/sys/fs/cgroup/"+path, suffixes)
if err != nil {
fmt.Printf("Error getting stats: %v\n", err)
}
}
}
func getStatsManual(path string, suffixes []string) ([][]byte, error) {
results := [][]byte{}
for _, suffix := range suffixes {
data, err := os.ReadFile(path + "/" + suffix)
if err != nil {
return nil, err
}
results = append(results, data)
}
return results, nil
}
Display the source blob
Display the rendered blob
Raw
one-loop.svg
<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg version="1.1" width="1440" height="758" onload="init(evt)" viewBox="0 0 1440 758" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:fg="http://github.com/jonhoo/inferno"><!--Flame graph stack visualization. See https://github.com/brendangregg/FlameGraph for latest version, and http://www.brendangregg.com/flamegraphs.html for examples.--><!--NOTES: --><defs><linearGradient id="background" y1="0" y2="1" x1="0" x2="0"><stop stop-color="#eeeeee" offset="5%"/><stop stop-color="#eeeeb0" offset="95%"/></linearGradient></defs><style type="text/css">
text { font-family:"Verdana"; font-size:12px; fill:rgb(0,0,0); }
#title { text-anchor:middle; font-size:17px; }
#search { opacity:0.1; cursor:pointer; }
#search:hover, #search.show { opacity:1; }
#subtitle { text-anchor:middle; font-color:rgb(160,160,160); }
#unzoom { cursor:pointer; }
#frames > *:hover { stroke:black; stroke-width:0.5; cursor:pointer; }
.hide { display:none; }
.parent { opacity:0.5; }
</style><script type="text/ecmascript"><![CDATA[
var nametype = 'Function:';
var fontsize = 12;
var fontwidth = 0.59;
var xpad = 10;
var inverted = false;
var searchcolor = 'rgb(230,0,230)';
var fluiddrawing = false;
var truncate_text_right = true;
]]><![CDATA["use strict";
var details, searchbtn, unzoombtn, matchedtxt, svg, searching, frames;
function init(evt) {
details = document.getElementById("details").firstChild;
searchbtn = document.getElementById("search");
unzoombtn = document.getElementById("unzoom");
matchedtxt = document.getElementById("matched");
svg = document.getElementsByTagName("svg")[0];
frames = document.getElementById("frames");
total_samples = parseInt(frames.attributes.total_samples.value);
searching = 0;
// Use GET parameters to restore a flamegraph's state.
var restore_state = function() {
var params = get_params();
if (params.x && params.y)
zoom(find_group(document.querySelector('[*|x="' + params.x + '"][y="' + params.y + '"]')));
if (params.s)
search(params.s);
};
if (fluiddrawing) {
// Make width dynamic so the SVG fits its parent's width.
svg.removeAttribute("width");
// Edge requires us to have a viewBox that gets updated with size changes.
var isEdge = /Edge\/\d./i.test(navigator.userAgent);
if (!isEdge) {
svg.removeAttribute("viewBox");
}
var update_for_width_change = function() {
if (isEdge) {
svg.attributes.viewBox.value = "0 0 " + svg.width.baseVal.value + " " + svg.height.baseVal.value;
}
// Keep consistent padding on left and right of frames container.
frames.attributes.width.value = svg.width.baseVal.value - xpad * 2;
// Text truncation needs to be adjusted for the current width.
var el = frames.children;
for(var i = 0; i < el.length; i++) {
update_text(el[i]);
}
// Keep search elements at a fixed distance from right edge.
var svgWidth = svg.width.baseVal.value;
searchbtn.attributes.x.value = svgWidth - xpad - 100;
matchedtxt.attributes.x.value = svgWidth - xpad - 100;
};
window.addEventListener('resize', function() {
update_for_width_change();
});
// This needs to be done asynchronously for Safari to work.
setTimeout(function() {
unzoom();
update_for_width_change();
restore_state();
}, 0);
} else {
restore_state();
}
}
// event listeners
window.addEventListener("click", function(e) {
var target = find_group(e.target);
if (target) {
if (target.nodeName == "a") {
if (e.ctrlKey === false) return;
e.preventDefault();
}
if (target.classList.contains("parent")) unzoom();
zoom(target);
// set parameters for zoom state
var el = target.querySelector("rect");
if (el && el.attributes && el.attributes.y && el.attributes["fg:x"]) {
var params = get_params()
params.x = el.attributes["fg:x"].value;
params.y = el.attributes.y.value;
history.replaceState(null, null, parse_params(params));
}
}
else if (e.target.id == "unzoom") {
unzoom();
// remove zoom state
var params = get_params();
if (params.x) delete params.x;
if (params.y) delete params.y;
history.replaceState(null, null, parse_params(params));
}
else if (e.target.id == "search") search_prompt();
}, false)
// mouse-over for info
// show
window.addEventListener("mouseover", function(e) {
var target = find_group(e.target);
if (target) details.nodeValue = nametype + " " + g_to_text(target);
}, false)
// clear
window.addEventListener("mouseout", function(e) {
var target = find_group(e.target);
if (target) details.nodeValue = ' ';
}, false)
// ctrl-F for search
window.addEventListener("keydown",function (e) {
if (e.keyCode === 114 || (e.ctrlKey && e.keyCode === 70)) {
e.preventDefault();
search_prompt();
}
}, false)
// functions
function get_params() {
var params = {};
var paramsarr = window.location.search.substr(1).split('&');
for (var i = 0; i < paramsarr.length; ++i) {
var tmp = paramsarr[i].split("=");
if (!tmp[0] || !tmp[1]) continue;
params[tmp[0]] = decodeURIComponent(tmp[1]);
}
return params;
}
function parse_params(params) {
var uri = "?";
for (var key in params) {
uri += key + '=' + encodeURIComponent(params[key]) + '&';
}
if (uri.slice(-1) == "&")
uri = uri.substring(0, uri.length - 1);
if (uri == '?')
uri = window.location.href.split('?')[0];
return uri;
}
function find_child(node, selector) {
var children = node.querySelectorAll(selector);
if (children.length) return children[0];
return;
}
function find_group(node) {
var parent = node.parentElement;
if (!parent) return;
if (parent.id == "frames") return node;
return find_group(parent);
}
function orig_save(e, attr, val) {
if (e.attributes["fg:orig_" + attr] != undefined) return;
if (e.attributes[attr] == undefined) return;
if (val == undefined) val = e.attributes[attr].value;
e.setAttribute("fg:orig_" + attr, val);
}
function orig_load(e, attr) {
if (e.attributes["fg:orig_"+attr] == undefined) return;
e.attributes[attr].value = e.attributes["fg:orig_" + attr].value;
e.removeAttribute("fg:orig_" + attr);
}
function g_to_text(e) {
var text = find_child(e, "title").firstChild.nodeValue;
return (text)
}
function g_to_func(e) {
var func = g_to_text(e);
// if there's any manipulation we want to do to the function
// name before it's searched, do it here before returning.
return (func);
}
function update_text(e) {
var r = find_child(e, "rect");
var t = find_child(e, "text");
var w = parseFloat(r.attributes.width.value) * frames.attributes.width.value / 100 - 3;
var txt = find_child(e, "title").textContent.replace(/\([^(]*\)$/,"");
t.attributes.x.value = format_percent((parseFloat(r.attributes.x.value) + (100 * 3 / frames.attributes.width.value)));
// Smaller than this size won't fit anything
if (w < 2 * fontsize * fontwidth) {
t.textContent = "";
return;
}
t.textContent = txt;
// Fit in full text width
if (/^ *\$/.test(txt) || t.getComputedTextLength() < w)
return;
if (truncate_text_right) {
// Truncate the right side of the text.
for (var x = txt.length - 2; x > 0; x--) {
if (t.getSubStringLength(0, x + 2) <= w) {
t.textContent = txt.substring(0, x) + "..";
return;
}
}
} else {
// Truncate the left side of the text.
for (var x = 2; x < txt.length; x++) {
if (t.getSubStringLength(x - 2, txt.length) <= w) {
t.textContent = ".." + txt.substring(x, txt.length);
return;
}
}
}
t.textContent = "";
}
// zoom
function zoom_reset(e) {
if (e.tagName == "rect") {
e.attributes.x.value = format_percent(100 * parseInt(e.attributes["fg:x"].value) / total_samples);
e.attributes.width.value = format_percent(100 * parseInt(e.attributes["fg:w"].value) / total_samples);
}
if (e.childNodes == undefined) return;
for(var i = 0, c = e.childNodes; i < c.length; i++) {
zoom_reset(c[i]);
}
}
function zoom_child(e, x, zoomed_width_samples) {
if (e.tagName == "text") {
var parent_x = parseFloat(find_child(e.parentNode, "rect[x]").attributes.x.value);
e.attributes.x.value = format_percent(parent_x + (100 * 3 / frames.attributes.width.value));
} else if (e.tagName == "rect") {
e.attributes.x.value = format_percent(100 * (parseInt(e.attributes["fg:x"].value) - x) / zoomed_width_samples);
e.attributes.width.value = format_percent(100 * parseInt(e.attributes["fg:w"].value) / zoomed_width_samples);
}
if (e.childNodes == undefined) return;
for(var i = 0, c = e.childNodes; i < c.length; i++) {
zoom_child(c[i], x, zoomed_width_samples);
}
}
function zoom_parent(e) {
if (e.attributes) {
if (e.attributes.x != undefined) {
e.attributes.x.value = "0.0%";
}
if (e.attributes.width != undefined) {
e.attributes.width.value = "100.0%";
}
}
if (e.childNodes == undefined) return;
for(var i = 0, c = e.childNodes; i < c.length; i++) {
zoom_parent(c[i]);
}
}
function zoom(node) {
var attr = find_child(node, "rect").attributes;
var width = parseInt(attr["fg:w"].value);
var xmin = parseInt(attr["fg:x"].value);
var xmax = xmin + width;
var ymin = parseFloat(attr.y.value);
unzoombtn.classList.remove("hide");
var el = frames.children;
for (var i = 0; i < el.length; i++) {
var e = el[i];
var a = find_child(e, "rect").attributes;
var ex = parseInt(a["fg:x"].value);
var ew = parseInt(a["fg:w"].value);
// Is it an ancestor
if (!inverted) {
var upstack = parseFloat(a.y.value) > ymin;
} else {
var upstack = parseFloat(a.y.value) < ymin;
}
if (upstack) {
// Direct ancestor
if (ex <= xmin && (ex+ew) >= xmax) {
e.classList.add("parent");
zoom_parent(e);
update_text(e);
}
// not in current path
else
e.classList.add("hide");
}
// Children maybe
else {
// no common path
if (ex < xmin || ex >= xmax) {
e.classList.add("hide");
}
else {
zoom_child(e, xmin, width);
update_text(e);
}
}
}
}
function unzoom() {
unzoombtn.classList.add("hide");
var el = frames.children;
for(var i = 0; i < el.length; i++) {
el[i].classList.remove("parent");
el[i].classList.remove("hide");
zoom_reset(el[i]);
update_text(el[i]);
}
}
// search
function reset_search() {
var el = document.querySelectorAll("#frames rect");
for (var i = 0; i < el.length; i++) {
orig_load(el[i], "fill")
}
var params = get_params();
delete params.s;
history.replaceState(null, null, parse_params(params));
}
function search_prompt() {
if (!searching) {
var term = prompt("Enter a search term (regexp " +
"allowed, eg: ^ext4_)", "");
if (term != null) {
search(term)
}
} else {
reset_search();
searching = 0;
searchbtn.classList.remove("show");
searchbtn.firstChild.nodeValue = "Search"
matchedtxt.classList.add("hide");
matchedtxt.firstChild.nodeValue = ""
}
}
function search(term) {
var re = new RegExp(term);
var el = frames.children;
var matches = new Object();
var maxwidth = 0;
for (var i = 0; i < el.length; i++) {
var e = el[i];
// Skip over frames which are either not visible, or below the zoomed-to frame
if (e.classList.contains("hide") || e.classList.contains("parent")) {
continue;
}
var func = g_to_func(e);
var rect = find_child(e, "rect");
if (func == null || rect == null)
continue;
// Save max width. Only works as we have a root frame
var w = parseInt(rect.attributes["fg:w"].value);
if (w > maxwidth)
maxwidth = w;
if (func.match(re)) {
// highlight
var x = parseInt(rect.attributes["fg:x"].value);
orig_save(rect, "fill");
rect.attributes.fill.value = searchcolor;
// remember matches
if (matches[x] == undefined) {
matches[x] = w;
} else {
if (w > matches[x]) {
// overwrite with parent
matches[x] = w;
}
}
searching = 1;
}
}
if (!searching)
return;
var params = get_params();
params.s = term;
history.replaceState(null, null, parse_params(params));
searchbtn.classList.add("show");
searchbtn.firstChild.nodeValue = "Reset Search";
// calculate percent matched, excluding vertical overlap
var count = 0;
var lastx = -1;
var lastw = 0;
var keys = Array();
for (k in matches) {
if (matches.hasOwnProperty(k))
keys.push(k);
}
// sort the matched frames by their x location
// ascending, then width descending
keys.sort(function(a, b){
return a - b;
});
// Step through frames saving only the biggest bottom-up frames
// thanks to the sort order. This relies on the tree property
// where children are always smaller than their parents.
for (var k in keys) {
var x = parseInt(keys[k]);
var w = matches[keys[k]];
if (x >= lastx + lastw) {
count += w;
lastx = x;
lastw = w;
}
}
// display matched percent
matchedtxt.classList.remove("hide");
var pct = 100 * count / maxwidth;
if (pct != 100) pct = pct.toFixed(1);
matchedtxt.firstChild.nodeValue = "Matched: " + pct + "%";
}
function format_percent(n) {
return n.toFixed(4) + "%";
}
]]></script><rect x="0" y="0" width="100%" height="758" fill="url(#background)"/><text id="title" x="50.0000%" y="24.00">39m139</text><text id="details" x="10" y="741.00"> </text><text id="unzoom" class="hide" x="10" y="24.00">Reset Zoom</text><text id="search" x="1330" y="24.00">Search</text><text id="matched" x="1330" y="741.00"> </text><svg id="frames" x="10" width="1420" total_samples="1776"><g><title>runtime.casgstatus (1 samples, 0.06%)</title><rect x="0.0563%" y="469" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1" fg:w="1"/><text x="0.2646%" y="479.50"></text></g><g><title>asm_sysvec_apic_timer_interrupt (1 samples, 0.06%)</title><rect x="0.4505%" y="453" width="0.0563%" height="15" fill="rgb(231,131,0)" fg:x="8" fg:w="1"/><text x="0.6588%" y="463.50"></text></g><g><title>sysvec_apic_timer_interrupt (1 samples, 0.06%)</title><rect x="0.4505%" y="437" width="0.0563%" height="15" fill="rgb(237,137,0)" fg:x="8" fg:w="1"/><text x="0.6588%" y="447.50"></text></g><g><title>__sysvec_apic_timer_interrupt (1 samples, 0.06%)</title><rect x="0.4505%" y="421" width="0.0563%" height="15" fill="rgb(227,127,0)" fg:x="8" fg:w="1"/><text x="0.6588%" y="431.50"></text></g><g><title>hrtimer_interrupt (1 samples, 0.06%)</title><rect x="0.4505%" y="405" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="8" fg:w="1"/><text x="0.6588%" y="415.50"></text></g><g><title>__hrtimer_run_queues (1 samples, 0.06%)</title><rect x="0.4505%" y="389" width="0.0563%" height="15" fill="rgb(226,126,0)" fg:x="8" fg:w="1"/><text x="0.6588%" y="399.50"></text></g><g><title>tick_sched_timer (1 samples, 0.06%)</title><rect x="0.4505%" y="373" width="0.0563%" height="15" fill="rgb(230,130,0)" fg:x="8" fg:w="1"/><text x="0.6588%" y="383.50"></text></g><g><title>tick_sched_handle (1 samples, 0.06%)</title><rect x="0.4505%" y="357" width="0.0563%" height="15" fill="rgb(230,130,0)" fg:x="8" fg:w="1"/><text x="0.6588%" y="367.50"></text></g><g><title>update_process_times (1 samples, 0.06%)</title><rect x="0.4505%" y="341" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="8" fg:w="1"/><text x="0.6588%" y="351.50"></text></g><g><title>account_user_time (1 samples, 0.06%)</title><rect x="0.4505%" y="325" width="0.0563%" height="15" fill="rgb(236,136,0)" fg:x="8" fg:w="1"/><text x="0.6588%" y="335.50"></text></g><g><title>__cgroup_account_cputime_field (1 samples, 0.06%)</title><rect x="0.4505%" y="309" width="0.0563%" height="15" fill="rgb(230,130,0)" fg:x="8" fg:w="1"/><text x="0.6588%" y="319.50"></text></g><g><title>kernfs_fop_read_iter (2 samples, 0.11%)</title><rect x="0.5631%" y="389" width="0.1126%" height="15" fill="rgb(230,130,0)" fg:x="10" fg:w="2"/><text x="0.7714%" y="399.50"></text></g><g><title>__check_object_size (1 samples, 0.06%)</title><rect x="0.6757%" y="373" width="0.0563%" height="15" fill="rgb(230,130,0)" fg:x="12" fg:w="1"/><text x="0.8840%" y="383.50"></text></g><g><title>cgroup_seqfile_show (1 samples, 0.06%)</title><rect x="0.7320%" y="373" width="0.0563%" height="15" fill="rgb(223,123,0)" fg:x="13" fg:w="1"/><text x="0.9403%" y="383.50"></text></g><g><title>_raw_spin_unlock_irq (1 samples, 0.06%)</title><rect x="0.7883%" y="341" width="0.0563%" height="15" fill="rgb(237,137,0)" fg:x="14" fg:w="1"/><text x="0.9966%" y="351.50"></text></g><g><title>asm_sysvec_apic_timer_interrupt (1 samples, 0.06%)</title><rect x="0.7883%" y="325" width="0.0563%" height="15" fill="rgb(231,131,0)" fg:x="14" fg:w="1"/><text x="0.9966%" y="335.50"></text></g><g><title>sysvec_apic_timer_interrupt (1 samples, 0.06%)</title><rect x="0.7883%" y="309" width="0.0563%" height="15" fill="rgb(237,137,0)" fg:x="14" fg:w="1"/><text x="0.9966%" y="319.50"></text></g><g><title>__sysvec_apic_timer_interrupt (1 samples, 0.06%)</title><rect x="0.7883%" y="293" width="0.0563%" height="15" fill="rgb(227,127,0)" fg:x="14" fg:w="1"/><text x="0.9966%" y="303.50"></text></g><g><title>hrtimer_interrupt (1 samples, 0.06%)</title><rect x="0.7883%" y="277" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="14" fg:w="1"/><text x="0.9966%" y="287.50"></text></g><g><title>__hrtimer_run_queues (1 samples, 0.06%)</title><rect x="0.7883%" y="261" width="0.0563%" height="15" fill="rgb(226,126,0)" fg:x="14" fg:w="1"/><text x="0.9966%" y="271.50"></text></g><g><title>hrtimer_wakeup (1 samples, 0.06%)</title><rect x="0.7883%" y="245" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="14" fg:w="1"/><text x="0.9966%" y="255.50"></text></g><g><title>try_to_wake_up (1 samples, 0.06%)</title><rect x="0.7883%" y="229" width="0.0563%" height="15" fill="rgb(234,134,0)" fg:x="14" fg:w="1"/><text x="0.9966%" y="239.50"></text></g><g><title>ttwu_do_activate (1 samples, 0.06%)</title><rect x="0.7883%" y="213" width="0.0563%" height="15" fill="rgb(222,122,0)" fg:x="14" fg:w="1"/><text x="0.9966%" y="223.50"></text></g><g><title>enqueue_task_fair (1 samples, 0.06%)</title><rect x="0.7883%" y="197" width="0.0563%" height="15" fill="rgb(248,148,0)" fg:x="14" fg:w="1"/><text x="0.9966%" y="207.50"></text></g><g><title>enqueue_entity (1 samples, 0.06%)</title><rect x="0.7883%" y="181" width="0.0563%" height="15" fill="rgb(248,148,0)" fg:x="14" fg:w="1"/><text x="0.9966%" y="191.50"></text></g><g><title>__list_add_valid (1 samples, 0.06%)</title><rect x="0.7883%" y="165" width="0.0563%" height="15" fill="rgb(233,133,0)" fg:x="14" fg:w="1"/><text x="0.9966%" y="175.50"></text></g><g><title>_raw_spin_lock_irqsave (9 samples, 0.51%)</title><rect x="1.2387%" y="325" width="0.5068%" height="15" fill="rgb(237,137,0)" fg:x="22" fg:w="9"/><text x="1.4471%" y="335.50"></text></g><g><title>blkcg_rstat_flush (21 samples, 1.18%)</title><rect x="1.7455%" y="325" width="1.1824%" height="15" fill="rgb(214,114,0)" fg:x="31" fg:w="21"/><text x="1.9538%" y="335.50"></text></g><g><title>blkcg_iostat_update (4 samples, 0.23%)</title><rect x="2.7027%" y="309" width="0.2252%" height="15" fill="rgb(214,114,0)" fg:x="48" fg:w="4"/><text x="2.9110%" y="319.50"></text></g><g><title>mem_cgroup_css_rstat_flush (22 samples, 1.24%)</title><rect x="2.9279%" y="325" width="1.2387%" height="15" fill="rgb(230,130,0)" fg:x="52" fg:w="22"/><text x="3.1363%" y="335.50"></text></g><g><title>_find_next_bit (1 samples, 0.06%)</title><rect x="4.1104%" y="309" width="0.0563%" height="15" fill="rgb(231,131,0)" fg:x="73" fg:w="1"/><text x="4.3187%" y="319.50"></text></g><g><title>cgroup_rstat_flush_locked (71 samples, 4.00%)</title><rect x="0.8446%" y="341" width="3.9977%" height="15" fill="rgb(223,123,0)" fg:x="15" fg:w="71"/><text x="1.0529%" y="351.50">cgroup..</text></g><g><title>queued_spin_lock_slowpath (12 samples, 0.68%)</title><rect x="4.1667%" y="325" width="0.6757%" height="15" fill="rgb(233,133,0)" fg:x="74" fg:w="12"/><text x="4.3750%" y="335.50"></text></g><g><title>queued_spin_lock_slowpath (75 samples, 4.22%)</title><rect x="4.8423%" y="341" width="4.2230%" height="15" fill="rgb(233,133,0)" fg:x="86" fg:w="75"/><text x="5.0507%" y="351.50">queued..</text></g><g><title>cpu_stat_show (148 samples, 8.33%)</title><rect x="0.7883%" y="373" width="8.3333%" height="15" fill="rgb(223,123,0)" fg:x="14" fg:w="148"/><text x="0.9966%" y="383.50">cpu_stat_show</text></g><g><title>cgroup_base_stat_cputime_show (148 samples, 8.33%)</title><rect x="0.7883%" y="357" width="8.3333%" height="15" fill="rgb(223,123,0)" fg:x="14" fg:w="148"/><text x="0.9966%" y="367.50">cgroup_base_st..</text></g><g><title>seq_printf (1 samples, 0.06%)</title><rect x="9.0653%" y="341" width="0.0563%" height="15" fill="rgb(236,136,0)" fg:x="161" fg:w="1"/><text x="9.2736%" y="351.50"></text></g><g><title>vsnprintf (1 samples, 0.06%)</title><rect x="9.0653%" y="325" width="0.0563%" height="15" fill="rgb(227,127,0)" fg:x="161" fg:w="1"/><text x="9.2736%" y="335.50"></text></g><g><title>kfree (1 samples, 0.06%)</title><rect x="9.1216%" y="357" width="0.0563%" height="15" fill="rgb(229,129,0)" fg:x="162" fg:w="1"/><text x="9.3300%" y="367.50"></text></g><g><title>__rcu_read_unlock (4 samples, 0.23%)</title><rect x="16.7793%" y="293" width="0.2252%" height="15" fill="rgb(228,128,0)" fg:x="298" fg:w="4"/><text x="16.9876%" y="303.50"></text></g><g><title>_raw_spin_lock_irqsave (10 samples, 0.56%)</title><rect x="17.0045%" y="293" width="0.5631%" height="15" fill="rgb(237,137,0)" fg:x="302" fg:w="10"/><text x="17.2128%" y="303.50"></text></g><g><title>preempt_count_add (1 samples, 0.06%)</title><rect x="17.5113%" y="277" width="0.0563%" height="15" fill="rgb(240,140,0)" fg:x="311" fg:w="1"/><text x="17.7196%" y="287.50"></text></g><g><title>__rcu_read_lock (1 samples, 0.06%)</title><rect x="37.3311%" y="277" width="0.0563%" height="15" fill="rgb(228,128,0)" fg:x="663" fg:w="1"/><text x="37.5394%" y="287.50"></text></g><g><title>blkcg_rstat_flush (406 samples, 22.86%)</title><rect x="17.5676%" y="293" width="22.8604%" height="15" fill="rgb(214,114,0)" fg:x="312" fg:w="406"/><text x="17.7759%" y="303.50">blkcg_rstat_flush</text></g><g><title>blkcg_iostat_update (54 samples, 3.04%)</title><rect x="37.3874%" y="277" width="3.0405%" height="15" fill="rgb(214,114,0)" fg:x="664" fg:w="54"/><text x="37.5957%" y="287.50">blkc..</text></g><g><title>_find_first_bit (2 samples, 0.11%)</title><rect x="91.5541%" y="277" width="0.1126%" height="15" fill="rgb(231,131,0)" fg:x="1626" fg:w="2"/><text x="91.7624%" y="287.50"></text></g><g><title>__mem_cgroup_flush_stats (1,490 samples, 83.90%)</title><rect x="9.1779%" y="341" width="83.8964%" height="15" fill="rgb(232,132,0)" fg:x="163" fg:w="1490"/><text x="9.3863%" y="351.50">__mem_cgroup_flush_stats</text></g><g><title>cgroup_rstat_flush_irqsafe (1,490 samples, 83.90%)</title><rect x="9.1779%" y="325" width="83.8964%" height="15" fill="rgb(223,123,0)" fg:x="163" fg:w="1490"/><text x="9.3863%" y="335.50">cgroup_rstat_flush_irqsafe</text></g><g><title>cgroup_rstat_flush_locked (1,490 samples, 83.90%)</title><rect x="9.1779%" y="309" width="83.8964%" height="15" fill="rgb(223,123,0)" fg:x="163" fg:w="1490"/><text x="9.3863%" y="319.50">cgroup_rstat_flush_locked</text></g><g><title>mem_cgroup_css_rstat_flush (935 samples, 52.65%)</title><rect x="40.4279%" y="293" width="52.6464%" height="15" fill="rgb(230,130,0)" fg:x="718" fg:w="935"/><text x="40.6363%" y="303.50">mem_cgroup_css_rstat_flush</text></g><g><title>_find_next_bit (25 samples, 1.41%)</title><rect x="91.6667%" y="277" width="1.4077%" height="15" fill="rgb(231,131,0)" fg:x="1628" fg:w="25"/><text x="91.8750%" y="287.50"></text></g><g><title>__irq_exit_rcu (2 samples, 0.11%)</title><rect x="93.0743%" y="293" width="0.1126%" height="15" fill="rgb(225,125,0)" fg:x="1653" fg:w="2"/><text x="93.2827%" y="303.50"></text></g><g><title>tick_sched_do_timer (1 samples, 0.06%)</title><rect x="93.2432%" y="229" width="0.0563%" height="15" fill="rgb(230,130,0)" fg:x="1656" fg:w="1"/><text x="93.4516%" y="239.50"></text></g><g><title>perf_event_task_tick (1 samples, 0.06%)</title><rect x="93.2995%" y="181" width="0.0563%" height="15" fill="rgb(242,142,0)" fg:x="1657" fg:w="1"/><text x="93.5079%" y="191.50"></text></g><g><title>amd_pmu_check_overflow (1 samples, 0.06%)</title><rect x="93.2995%" y="165" width="0.0563%" height="15" fill="rgb(221,121,0)" fg:x="1657" fg:w="1"/><text x="93.5079%" y="175.50"></text></g><g><title>amd_pmu_test_overflow_topbit (1 samples, 0.06%)</title><rect x="93.2995%" y="149" width="0.0563%" height="15" fill="rgb(221,121,0)" fg:x="1657" fg:w="1"/><text x="93.5079%" y="159.50"></text></g><g><title>__hrtimer_run_queues (3 samples, 0.17%)</title><rect x="93.2432%" y="261" width="0.1689%" height="15" fill="rgb(226,126,0)" fg:x="1656" fg:w="3"/><text x="93.4516%" y="271.50"></text></g><g><title>tick_sched_timer (3 samples, 0.17%)</title><rect x="93.2432%" y="245" width="0.1689%" height="15" fill="rgb(230,130,0)" fg:x="1656" fg:w="3"/><text x="93.4516%" y="255.50"></text></g><g><title>tick_sched_handle (2 samples, 0.11%)</title><rect x="93.2995%" y="229" width="0.1126%" height="15" fill="rgb(230,130,0)" fg:x="1657" fg:w="2"/><text x="93.5079%" y="239.50"></text></g><g><title>update_process_times (2 samples, 0.11%)</title><rect x="93.2995%" y="213" width="0.1126%" height="15" fill="rgb(232,132,0)" fg:x="1657" fg:w="2"/><text x="93.5079%" y="223.50"></text></g><g><title>scheduler_tick (2 samples, 0.11%)</title><rect x="93.2995%" y="197" width="0.1126%" height="15" fill="rgb(236,136,0)" fg:x="1657" fg:w="2"/><text x="93.5079%" y="207.50"></text></g><g><title>task_tick_fair (1 samples, 0.06%)</title><rect x="93.3559%" y="181" width="0.0563%" height="15" fill="rgb(222,122,0)" fg:x="1658" fg:w="1"/><text x="93.5642%" y="191.50"></text></g><g><title>asm_sysvec_apic_timer_interrupt (7 samples, 0.39%)</title><rect x="93.0743%" y="325" width="0.3941%" height="15" fill="rgb(231,131,0)" fg:x="1653" fg:w="7"/><text x="93.2827%" y="335.50"></text></g><g><title>sysvec_apic_timer_interrupt (7 samples, 0.39%)</title><rect x="93.0743%" y="309" width="0.3941%" height="15" fill="rgb(237,137,0)" fg:x="1653" fg:w="7"/><text x="93.2827%" y="319.50"></text></g><g><title>__sysvec_apic_timer_interrupt (5 samples, 0.28%)</title><rect x="93.1869%" y="293" width="0.2815%" height="15" fill="rgb(227,127,0)" fg:x="1655" fg:w="5"/><text x="93.3953%" y="303.50"></text></g><g><title>hrtimer_interrupt (5 samples, 0.28%)</title><rect x="93.1869%" y="277" width="0.2815%" height="15" fill="rgb(232,132,0)" fg:x="1655" fg:w="5"/><text x="93.3953%" y="287.50"></text></g><g><title>clockevents_program_event (1 samples, 0.06%)</title><rect x="93.4122%" y="261" width="0.0563%" height="15" fill="rgb(219,119,0)" fg:x="1659" fg:w="1"/><text x="93.6205%" y="271.50"></text></g><g><title>lapic_next_event (1 samples, 0.06%)</title><rect x="93.4122%" y="245" width="0.0563%" height="15" fill="rgb(220,120,0)" fg:x="1659" fg:w="1"/><text x="93.6205%" y="255.50"></text></g><g><title>native_apic_mem_write (1 samples, 0.06%)</title><rect x="93.4122%" y="229" width="0.0563%" height="15" fill="rgb(234,134,0)" fg:x="1659" fg:w="1"/><text x="93.6205%" y="239.50"></text></g><g><title>_raw_spin_unlock_irqrestore (8 samples, 0.45%)</title><rect x="93.0743%" y="341" width="0.4505%" height="15" fill="rgb(237,137,0)" fg:x="1653" fg:w="8"/><text x="93.2827%" y="351.50"></text></g><g><title>asm_sysvec_call_function_single (1 samples, 0.06%)</title><rect x="93.4685%" y="325" width="0.0563%" height="15" fill="rgb(231,131,0)" fg:x="1660" fg:w="1"/><text x="93.6768%" y="335.50"></text></g><g><title>sysvec_call_function_single (1 samples, 0.06%)</title><rect x="93.4685%" y="309" width="0.0563%" height="15" fill="rgb(237,137,0)" fg:x="1660" fg:w="1"/><text x="93.6768%" y="319.50"></text></g><g><title>__sysvec_call_function_single (1 samples, 0.06%)</title><rect x="93.4685%" y="293" width="0.0563%" height="15" fill="rgb(227,127,0)" fg:x="1660" fg:w="1"/><text x="93.6768%" y="303.50"></text></g><g><title>sched_ttwu_pending (1 samples, 0.06%)</title><rect x="93.4685%" y="277" width="0.0563%" height="15" fill="rgb(236,136,0)" fg:x="1660" fg:w="1"/><text x="93.6768%" y="287.50"></text></g><g><title>ttwu_do_wakeup (1 samples, 0.06%)</title><rect x="93.4685%" y="261" width="0.0563%" height="15" fill="rgb(222,122,0)" fg:x="1660" fg:w="1"/><text x="93.6768%" y="271.50"></text></g><g><title>check_preempt_curr (1 samples, 0.06%)</title><rect x="93.4685%" y="245" width="0.0563%" height="15" fill="rgb(222,122,0)" fg:x="1660" fg:w="1"/><text x="93.6768%" y="255.50"></text></g><g><title>check_preempt_wakeup (1 samples, 0.06%)</title><rect x="93.4685%" y="229" width="0.0563%" height="15" fill="rgb(222,122,0)" fg:x="1660" fg:w="1"/><text x="93.6768%" y="239.50"></text></g><g><title>__memcpy (1 samples, 0.06%)</title><rect x="93.5811%" y="309" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="1662" fg:w="1"/><text x="93.7894%" y="319.50"></text></g><g><title>memory_stat_format.constprop.0 (1,502 samples, 84.57%)</title><rect x="9.1779%" y="357" width="84.5721%" height="15" fill="rgb(230,130,0)" fg:x="163" fg:w="1502"/><text x="9.3863%" y="367.50">memory_stat_format.constprop.0</text></g><g><title>seq_buf_printf (4 samples, 0.23%)</title><rect x="93.5248%" y="341" width="0.2252%" height="15" fill="rgb(236,136,0)" fg:x="1661" fg:w="4"/><text x="93.7331%" y="351.50"></text></g><g><title>vsnprintf (3 samples, 0.17%)</title><rect x="93.5811%" y="325" width="0.1689%" height="15" fill="rgb(227,127,0)" fg:x="1662" fg:w="3"/><text x="93.7894%" y="335.50"></text></g><g><title>number (2 samples, 0.11%)</title><rect x="93.6374%" y="309" width="0.1126%" height="15" fill="rgb(244,144,0)" fg:x="1663" fg:w="2"/><text x="93.8457%" y="319.50"></text></g><g><title>put_dec (1 samples, 0.06%)</title><rect x="93.6937%" y="293" width="0.0563%" height="15" fill="rgb(233,133,0)" fg:x="1664" fg:w="1"/><text x="93.9020%" y="303.50"></text></g><g><title>put_dec_full8 (1 samples, 0.06%)</title><rect x="93.6937%" y="277" width="0.0563%" height="15" fill="rgb(233,133,0)" fg:x="1664" fg:w="1"/><text x="93.9020%" y="287.50"></text></g><g><title>ksys_read (1,657 samples, 93.30%)</title><rect x="0.5068%" y="421" width="93.2995%" height="15" fill="rgb(231,131,0)" fg:x="9" fg:w="1657"/><text x="0.7151%" y="431.50">ksys_read</text></g><g><title>vfs_read (1,656 samples, 93.24%)</title><rect x="0.5631%" y="405" width="93.2432%" height="15" fill="rgb(226,126,0)" fg:x="10" fg:w="1656"/><text x="0.7714%" y="415.50">vfs_read</text></g><g><title>seq_read_iter (1,654 samples, 93.13%)</title><rect x="0.6757%" y="389" width="93.1306%" height="15" fill="rgb(236,136,0)" fg:x="12" fg:w="1654"/><text x="0.8840%" y="399.50">seq_read_iter</text></g><g><title>memory_stat_show (1,504 samples, 84.68%)</title><rect x="9.1216%" y="373" width="84.6847%" height="15" fill="rgb(230,130,0)" fg:x="162" fg:w="1504"/><text x="9.3300%" y="383.50">memory_stat_show</text></g><g><title>seq_puts (1 samples, 0.06%)</title><rect x="93.7500%" y="357" width="0.0563%" height="15" fill="rgb(236,136,0)" fg:x="1665" fg:w="1"/><text x="93.9583%" y="367.50"></text></g><g><title>strlen (1 samples, 0.06%)</title><rect x="93.7500%" y="341" width="0.0563%" height="15" fill="rgb(229,129,0)" fg:x="1665" fg:w="1"/><text x="93.9583%" y="351.50"></text></g><g><title>fpregs_restore_userregs (18 samples, 1.01%)</title><rect x="93.8063%" y="389" width="1.0135%" height="15" fill="rgb(242,142,0)" fg:x="1666" fg:w="18"/><text x="94.0146%" y="399.50"></text></g><g><title>restore_fpregs_from_fpstate (18 samples, 1.01%)</title><rect x="93.8063%" y="373" width="1.0135%" height="15" fill="rgb(236,136,0)" fg:x="1666" fg:w="18"/><text x="94.0146%" y="383.50"></text></g><g><title>asm_common_interrupt (6 samples, 0.34%)</title><rect x="94.8198%" y="341" width="0.3378%" height="15" fill="rgb(231,131,0)" fg:x="1684" fg:w="6"/><text x="95.0282%" y="351.50"></text></g><g><title>common_interrupt (6 samples, 0.34%)</title><rect x="94.8198%" y="325" width="0.3378%" height="15" fill="rgb(232,132,0)" fg:x="1684" fg:w="6"/><text x="95.0282%" y="335.50"></text></g><g><title>zen_untrain_ret (6 samples, 0.34%)</title><rect x="94.8198%" y="309" width="0.3378%" height="15" fill="rgb(245,145,0)" fg:x="1684" fg:w="6"/><text x="95.0282%" y="319.50"></text></g><g><title>hrtimer_update_next_event (1 samples, 0.06%)</title><rect x="95.4955%" y="277" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="1696" fg:w="1"/><text x="95.7038%" y="287.50"></text></g><g><title>__hrtimer_next_event_base (1 samples, 0.06%)</title><rect x="95.4955%" y="261" width="0.0563%" height="15" fill="rgb(226,126,0)" fg:x="1696" fg:w="1"/><text x="95.7038%" y="271.50"></text></g><g><title>os.(*File).Read (1,702 samples, 95.83%)</title><rect x="0.0563%" y="533" width="95.8333%" height="15" fill="rgb(92,239,92)" fg:x="1" fg:w="1702"/><text x="0.2646%" y="543.50">os.(*File).Read</text></g><g><title>internal/poll.(*FD).Read (1,702 samples, 95.83%)</title><rect x="0.0563%" y="517" width="95.8333%" height="15" fill="rgb(92,239,92)" fg:x="1" fg:w="1702"/><text x="0.2646%" y="527.50">internal/poll.(*FD).Read</text></g><g><title>syscall.read (1,702 samples, 95.83%)</title><rect x="0.0563%" y="501" width="95.8333%" height="15" fill="rgb(93,240,93)" fg:x="1" fg:w="1702"/><text x="0.2646%" y="511.50">syscall.read</text></g><g><title>syscall.Syscall (1,702 samples, 95.83%)</title><rect x="0.0563%" y="485" width="95.8333%" height="15" fill="rgb(93,240,93)" fg:x="1" fg:w="1702"/><text x="0.2646%" y="495.50">syscall.Syscall</text></g><g><title>runtime/internal/syscall.Syscall6 (1,701 samples, 95.78%)</title><rect x="0.1126%" y="469" width="95.7770%" height="15" fill="rgb(90,237,90)" fg:x="2" fg:w="1701"/><text x="0.3209%" y="479.50">runtime/internal/syscall.Syscall6</text></g><g><title>entry_SYSCALL_64_after_hwframe (1,694 samples, 95.38%)</title><rect x="0.5068%" y="453" width="95.3829%" height="15" fill="rgb(245,145,0)" fg:x="9" fg:w="1694"/><text x="0.7151%" y="463.50">entry_SYSCALL_64_after_hwframe</text></g><g><title>do_syscall_64 (1,694 samples, 95.38%)</title><rect x="0.5068%" y="437" width="95.3829%" height="15" fill="rgb(243,143,0)" fg:x="9" fg:w="1694"/><text x="0.7151%" y="447.50">do_syscall_64</text></g><g><title>syscall_exit_to_user_mode (37 samples, 2.08%)</title><rect x="93.8063%" y="421" width="2.0833%" height="15" fill="rgb(237,137,0)" fg:x="1666" fg:w="37"/><text x="94.0146%" y="431.50">sy..</text></g><g><title>exit_to_user_mode_prepare (37 samples, 2.08%)</title><rect x="93.8063%" y="405" width="2.0833%" height="15" fill="rgb(230,130,0)" fg:x="1666" fg:w="37"/><text x="94.0146%" y="415.50">ex..</text></g><g><title>schedule (19 samples, 1.07%)</title><rect x="94.8198%" y="389" width="1.0698%" height="15" fill="rgb(236,136,0)" fg:x="1684" fg:w="19"/><text x="95.0282%" y="399.50"></text></g><g><title>__schedule (19 samples, 1.07%)</title><rect x="94.8198%" y="373" width="1.0698%" height="15" fill="rgb(227,127,0)" fg:x="1684" fg:w="19"/><text x="95.0282%" y="383.50"></text></g><g><title>finish_task_switch.isra.0 (19 samples, 1.07%)</title><rect x="94.8198%" y="357" width="1.0698%" height="15" fill="rgb(240,140,0)" fg:x="1684" fg:w="19"/><text x="95.0282%" y="367.50"></text></g><g><title>asm_sysvec_apic_timer_interrupt (13 samples, 0.73%)</title><rect x="95.1577%" y="341" width="0.7320%" height="15" fill="rgb(231,131,0)" fg:x="1690" fg:w="13"/><text x="95.3660%" y="351.50"></text></g><g><title>sysvec_apic_timer_interrupt (13 samples, 0.73%)</title><rect x="95.1577%" y="325" width="0.7320%" height="15" fill="rgb(237,137,0)" fg:x="1690" fg:w="13"/><text x="95.3660%" y="335.50"></text></g><g><title>__sysvec_apic_timer_interrupt (13 samples, 0.73%)</title><rect x="95.1577%" y="309" width="0.7320%" height="15" fill="rgb(227,127,0)" fg:x="1690" fg:w="13"/><text x="95.3660%" y="319.50"></text></g><g><title>hrtimer_interrupt (7 samples, 0.39%)</title><rect x="95.4955%" y="293" width="0.3941%" height="15" fill="rgb(232,132,0)" fg:x="1696" fg:w="7"/><text x="95.7038%" y="303.50"></text></g><g><title>ktime_get_update_offsets_now (6 samples, 0.34%)</title><rect x="95.5518%" y="277" width="0.3378%" height="15" fill="rgb(222,122,0)" fg:x="1697" fg:w="6"/><text x="95.7601%" y="287.50"></text></g><g><title>runtime.exitsyscall (1 samples, 0.06%)</title><rect x="95.8896%" y="469" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1703" fg:w="1"/><text x="96.0980%" y="479.50"></text></g><g><title>runtime.exitsyscallfast (1 samples, 0.06%)</title><rect x="95.8896%" y="453" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1703" fg:w="1"/><text x="96.0980%" y="463.50"></text></g><g><title>perf_tp_event (3 samples, 0.17%)</title><rect x="95.9459%" y="325" width="0.1689%" height="15" fill="rgb(242,142,0)" fg:x="1704" fg:w="3"/><text x="96.1543%" y="335.50"></text></g><g><title>filter_match_preds (3 samples, 0.17%)</title><rect x="95.9459%" y="309" width="0.1689%" height="15" fill="rgb(241,141,0)" fg:x="1704" fg:w="3"/><text x="96.1543%" y="319.50"></text></g><g><title>regex_match_full (1 samples, 0.06%)</title><rect x="96.0586%" y="293" width="0.0563%" height="15" fill="rgb(236,136,0)" fg:x="1706" fg:w="1"/><text x="96.2669%" y="303.50"></text></g><g><title>asm_common_interrupt (1 samples, 0.06%)</title><rect x="96.2275%" y="309" width="0.0563%" height="15" fill="rgb(231,131,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="319.50"></text></g><g><title>common_interrupt (1 samples, 0.06%)</title><rect x="96.2275%" y="293" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="303.50"></text></g><g><title>__common_interrupt (1 samples, 0.06%)</title><rect x="96.2275%" y="277" width="0.0563%" height="15" fill="rgb(230,130,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="287.50"></text></g><g><title>handle_edge_irq (1 samples, 0.06%)</title><rect x="96.2275%" y="261" width="0.0563%" height="15" fill="rgb(231,131,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="271.50"></text></g><g><title>handle_irq_event (1 samples, 0.06%)</title><rect x="96.2275%" y="245" width="0.0563%" height="15" fill="rgb(231,131,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="255.50"></text></g><g><title>__handle_irq_event_percpu (1 samples, 0.06%)</title><rect x="96.2275%" y="229" width="0.0563%" height="15" fill="rgb(226,126,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="239.50"></text></g><g><title>nvme_irq (1 samples, 0.06%)</title><rect x="96.2275%" y="213" width="0.0563%" height="15" fill="rgb(242,142,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="223.50"></text></g><g><title>blk_mq_end_request_batch (1 samples, 0.06%)</title><rect x="96.2275%" y="197" width="0.0563%" height="15" fill="rgb(214,114,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="207.50"></text></g><g><title>clone_endio (1 samples, 0.06%)</title><rect x="96.2275%" y="181" width="0.0563%" height="15" fill="rgb(219,119,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="191.50"></text></g><g><title>clone_endio (1 samples, 0.06%)</title><rect x="96.2275%" y="165" width="0.0563%" height="15" fill="rgb(219,119,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="175.50"></text></g><g><title>iomap_finish_ioend (1 samples, 0.06%)</title><rect x="96.2275%" y="149" width="0.0563%" height="15" fill="rgb(241,141,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="159.50"></text></g><g><title>folio_end_writeback (1 samples, 0.06%)</title><rect x="96.2275%" y="133" width="0.0563%" height="15" fill="rgb(249,149,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="143.50"></text></g><g><title>folio_wake_bit (1 samples, 0.06%)</title><rect x="96.2275%" y="117" width="0.0563%" height="15" fill="rgb(249,149,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="127.50"></text></g><g><title>__wake_up_common (1 samples, 0.06%)</title><rect x="96.2275%" y="101" width="0.0563%" height="15" fill="rgb(223,123,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="111.50"></text></g><g><title>wake_page_function (1 samples, 0.06%)</title><rect x="96.2275%" y="85" width="0.0563%" height="15" fill="rgb(211,111,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="95.50"></text></g><g><title>try_to_wake_up (1 samples, 0.06%)</title><rect x="96.2275%" y="69" width="0.0563%" height="15" fill="rgb(234,134,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="79.50"></text></g><g><title>ttwu_queue_wakelist (1 samples, 0.06%)</title><rect x="96.2275%" y="53" width="0.0563%" height="15" fill="rgb(222,122,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="63.50"></text></g><g><title>send_call_function_single_ipi (1 samples, 0.06%)</title><rect x="96.2275%" y="37" width="0.0563%" height="15" fill="rgb(239,139,0)" fg:x="1709" fg:w="1"/><text x="96.4358%" y="47.50"></text></g><g><title>os.(*File).Stat (10 samples, 0.56%)</title><rect x="95.8896%" y="533" width="0.5631%" height="15" fill="rgb(92,239,92)" fg:x="1703" fg:w="10"/><text x="96.0980%" y="543.50"></text></g><g><title>internal/poll.(*FD).Fstat (10 samples, 0.56%)</title><rect x="95.8896%" y="517" width="0.5631%" height="15" fill="rgb(92,239,92)" fg:x="1703" fg:w="10"/><text x="96.0980%" y="527.50"></text></g><g><title>syscall.Fstat (10 samples, 0.56%)</title><rect x="95.8896%" y="501" width="0.5631%" height="15" fill="rgb(93,240,93)" fg:x="1703" fg:w="10"/><text x="96.0980%" y="511.50"></text></g><g><title>syscall.Syscall (10 samples, 0.56%)</title><rect x="95.8896%" y="485" width="0.5631%" height="15" fill="rgb(93,240,93)" fg:x="1703" fg:w="10"/><text x="96.0980%" y="495.50"></text></g><g><title>runtime/internal/syscall.Syscall6 (9 samples, 0.51%)</title><rect x="95.9459%" y="469" width="0.5068%" height="15" fill="rgb(90,237,90)" fg:x="1704" fg:w="9"/><text x="96.1543%" y="479.50"></text></g><g><title>entry_SYSCALL_64_after_hwframe (9 samples, 0.51%)</title><rect x="95.9459%" y="453" width="0.5068%" height="15" fill="rgb(245,145,0)" fg:x="1704" fg:w="9"/><text x="96.1543%" y="463.50"></text></g><g><title>do_syscall_64 (9 samples, 0.51%)</title><rect x="95.9459%" y="437" width="0.5068%" height="15" fill="rgb(243,143,0)" fg:x="1704" fg:w="9"/><text x="96.1543%" y="447.50"></text></g><g><title>__do_sys_newfstat (9 samples, 0.51%)</title><rect x="95.9459%" y="421" width="0.5068%" height="15" fill="rgb(230,130,0)" fg:x="1704" fg:w="9"/><text x="96.1543%" y="431.50"></text></g><g><title>vfs_fstat (9 samples, 0.51%)</title><rect x="95.9459%" y="405" width="0.5068%" height="15" fill="rgb(226,126,0)" fg:x="1704" fg:w="9"/><text x="96.1543%" y="415.50"></text></g><g><title>security_inode_getattr (9 samples, 0.51%)</title><rect x="95.9459%" y="389" width="0.5068%" height="15" fill="rgb(238,138,0)" fg:x="1704" fg:w="9"/><text x="96.1543%" y="399.50"></text></g><g><title>[unknown] (9 samples, 0.51%)</title><rect x="95.9459%" y="373" width="0.5068%" height="15" fill="rgb(243,112,112)" fg:x="1704" fg:w="9"/><text x="96.1543%" y="383.50"></text></g><g><title>kprobe_ftrace_handler (9 samples, 0.51%)</title><rect x="95.9459%" y="357" width="0.5068%" height="15" fill="rgb(230,130,0)" fg:x="1704" fg:w="9"/><text x="96.1543%" y="367.50"></text></g><g><title>kprobe_perf_func (9 samples, 0.51%)</title><rect x="95.9459%" y="341" width="0.5068%" height="15" fill="rgb(230,130,0)" fg:x="1704" fg:w="9"/><text x="96.1543%" y="351.50"></text></g><g><title>process_fetch_insn (6 samples, 0.34%)</title><rect x="96.1149%" y="325" width="0.3378%" height="15" fill="rgb(242,142,0)" fg:x="1707" fg:w="6"/><text x="96.3232%" y="335.50"></text></g><g><title>copy_from_kernel_nofault (3 samples, 0.17%)</title><rect x="96.2838%" y="309" width="0.1689%" height="15" fill="rgb(229,129,0)" fg:x="1710" fg:w="3"/><text x="96.4921%" y="319.50"></text></g><g><title>copy_from_kernel_nofault_allowed (1 samples, 0.06%)</title><rect x="96.3964%" y="293" width="0.0563%" height="15" fill="rgb(229,129,0)" fg:x="1712" fg:w="1"/><text x="96.6047%" y="303.50"></text></g><g><title>asm_sysvec_apic_timer_interrupt (1 samples, 0.06%)</title><rect x="96.3964%" y="277" width="0.0563%" height="15" fill="rgb(231,131,0)" fg:x="1712" fg:w="1"/><text x="96.6047%" y="287.50"></text></g><g><title>sysvec_apic_timer_interrupt (1 samples, 0.06%)</title><rect x="96.3964%" y="261" width="0.0563%" height="15" fill="rgb(237,137,0)" fg:x="1712" fg:w="1"/><text x="96.6047%" y="271.50"></text></g><g><title>__sysvec_apic_timer_interrupt (1 samples, 0.06%)</title><rect x="96.3964%" y="245" width="0.0563%" height="15" fill="rgb(227,127,0)" fg:x="1712" fg:w="1"/><text x="96.6047%" y="255.50"></text></g><g><title>hrtimer_interrupt (1 samples, 0.06%)</title><rect x="96.3964%" y="229" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="1712" fg:w="1"/><text x="96.6047%" y="239.50"></text></g><g><title>hrtimer_update_next_event (1 samples, 0.06%)</title><rect x="96.3964%" y="213" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="1712" fg:w="1"/><text x="96.6047%" y="223.50"></text></g><g><title>__hrtimer_next_event_base (1 samples, 0.06%)</title><rect x="96.3964%" y="197" width="0.0563%" height="15" fill="rgb(226,126,0)" fg:x="1712" fg:w="1"/><text x="96.6047%" y="207.50"></text></g><g><title>os.newFile (1 samples, 0.06%)</title><rect x="96.4527%" y="501" width="0.0563%" height="15" fill="rgb(92,239,92)" fg:x="1713" fg:w="1"/><text x="96.6610%" y="511.50"></text></g><g><title>syscall.SetNonblock (1 samples, 0.06%)</title><rect x="96.4527%" y="485" width="0.0563%" height="15" fill="rgb(93,240,93)" fg:x="1713" fg:w="1"/><text x="96.6610%" y="495.50"></text></g><g><title>syscall.fcntl (1 samples, 0.06%)</title><rect x="96.4527%" y="469" width="0.0563%" height="15" fill="rgb(93,240,93)" fg:x="1713" fg:w="1"/><text x="96.6610%" y="479.50"></text></g><g><title>syscall.Syscall (1 samples, 0.06%)</title><rect x="96.4527%" y="453" width="0.0563%" height="15" fill="rgb(93,240,93)" fg:x="1713" fg:w="1"/><text x="96.6610%" y="463.50"></text></g><g><title>runtime/internal/syscall.Syscall6 (1 samples, 0.06%)</title><rect x="96.4527%" y="437" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1713" fg:w="1"/><text x="96.6610%" y="447.50"></text></g><g><title>entry_SYSCALL_64 (1 samples, 0.06%)</title><rect x="96.4527%" y="421" width="0.0563%" height="15" fill="rgb(245,145,0)" fg:x="1713" fg:w="1"/><text x="96.6610%" y="431.50"></text></g><g><title>alloc_fd (1 samples, 0.06%)</title><rect x="96.5653%" y="389" width="0.0563%" height="15" fill="rgb(226,126,0)" fg:x="1715" fg:w="1"/><text x="96.7736%" y="399.50"></text></g><g><title>alloc_empty_file (1 samples, 0.06%)</title><rect x="96.6216%" y="357" width="0.0563%" height="15" fill="rgb(226,126,0)" fg:x="1716" fg:w="1"/><text x="96.8300%" y="367.50"></text></g><g><title>__alloc_file (1 samples, 0.06%)</title><rect x="96.6216%" y="341" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="1716" fg:w="1"/><text x="96.8300%" y="351.50"></text></g><g><title>__mutex_init (1 samples, 0.06%)</title><rect x="96.6216%" y="325" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="1716" fg:w="1"/><text x="96.8300%" y="335.50"></text></g><g><title>kmalloc_trace (1 samples, 0.06%)</title><rect x="96.6779%" y="325" width="0.0563%" height="15" fill="rgb(223,123,0)" fg:x="1717" fg:w="1"/><text x="96.8863%" y="335.50"></text></g><g><title>__kmem_cache_alloc_node (1 samples, 0.06%)</title><rect x="96.6779%" y="309" width="0.0563%" height="15" fill="rgb(223,123,0)" fg:x="1717" fg:w="1"/><text x="96.8863%" y="319.50"></text></g><g><title>__memset (1 samples, 0.06%)</title><rect x="96.6779%" y="293" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="1717" fg:w="1"/><text x="96.8863%" y="303.50"></text></g><g><title>do_dentry_open (2 samples, 0.11%)</title><rect x="96.6779%" y="357" width="0.1126%" height="15" fill="rgb(243,143,0)" fg:x="1717" fg:w="2"/><text x="96.8863%" y="367.50"></text></g><g><title>kernfs_fop_open (2 samples, 0.11%)</title><rect x="96.6779%" y="341" width="0.1126%" height="15" fill="rgb(230,130,0)" fg:x="1717" fg:w="2"/><text x="96.8863%" y="351.50"></text></g><g><title>seq_open (1 samples, 0.06%)</title><rect x="96.7342%" y="325" width="0.0563%" height="15" fill="rgb(236,136,0)" fg:x="1718" fg:w="1"/><text x="96.9426%" y="335.50"></text></g><g><title>kmem_cache_alloc (1 samples, 0.06%)</title><rect x="96.7342%" y="309" width="0.0563%" height="15" fill="rgb(220,120,0)" fg:x="1718" fg:w="1"/><text x="96.9426%" y="319.50"></text></g><g><title>get_obj_cgroup_from_current (1 samples, 0.06%)</title><rect x="96.7342%" y="293" width="0.0563%" height="15" fill="rgb(238,138,0)" fg:x="1718" fg:w="1"/><text x="96.9426%" y="303.50"></text></g><g><title>inode_permission (1 samples, 0.06%)</title><rect x="96.7905%" y="341" width="0.0563%" height="15" fill="rgb(240,140,0)" fg:x="1719" fg:w="1"/><text x="96.9989%" y="351.50"></text></g><g><title>kernfs_iop_permission (1 samples, 0.06%)</title><rect x="96.7905%" y="325" width="0.0563%" height="15" fill="rgb(230,130,0)" fg:x="1719" fg:w="1"/><text x="96.9989%" y="335.50"></text></g><g><title>security_inode_permission (1 samples, 0.06%)</title><rect x="96.8468%" y="341" width="0.0563%" height="15" fill="rgb(238,138,0)" fg:x="1720" fg:w="1"/><text x="97.0552%" y="351.50"></text></g><g><title>step_into (2 samples, 0.11%)</title><rect x="96.9032%" y="341" width="0.1126%" height="15" fill="rgb(230,130,0)" fg:x="1721" fg:w="2"/><text x="97.1115%" y="351.50"></text></g><g><title>mntput (1 samples, 0.06%)</title><rect x="96.9595%" y="325" width="0.0563%" height="15" fill="rgb(227,127,0)" fg:x="1722" fg:w="1"/><text x="97.1678%" y="335.50"></text></g><g><title>__d_lookup (1 samples, 0.06%)</title><rect x="97.0158%" y="309" width="0.0563%" height="15" fill="rgb(230,130,0)" fg:x="1723" fg:w="1"/><text x="97.2241%" y="319.50"></text></g><g><title>link_path_walk.part.0.constprop.0 (7 samples, 0.39%)</title><rect x="96.7905%" y="357" width="0.3941%" height="15" fill="rgb(226,126,0)" fg:x="1719" fg:w="7"/><text x="96.9989%" y="367.50"></text></g><g><title>walk_component (3 samples, 0.17%)</title><rect x="97.0158%" y="341" width="0.1689%" height="15" fill="rgb(221,121,0)" fg:x="1723" fg:w="3"/><text x="97.2241%" y="351.50"></text></g><g><title>lookup_fast (3 samples, 0.17%)</title><rect x="97.0158%" y="325" width="0.1689%" height="15" fill="rgb(232,132,0)" fg:x="1723" fg:w="3"/><text x="97.2241%" y="335.50"></text></g><g><title>kernfs_dop_revalidate (2 samples, 0.11%)</title><rect x="97.0721%" y="309" width="0.1126%" height="15" fill="rgb(230,130,0)" fg:x="1724" fg:w="2"/><text x="97.2804%" y="319.50"></text></g><g><title>strcmp (2 samples, 0.11%)</title><rect x="97.0721%" y="293" width="0.1126%" height="15" fill="rgb(229,129,0)" fg:x="1724" fg:w="2"/><text x="97.2804%" y="303.50"></text></g><g><title>lookup_fast (1 samples, 0.06%)</title><rect x="97.1847%" y="357" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="1726" fg:w="1"/><text x="97.3930%" y="367.50"></text></g><g><title>kernfs_dop_revalidate (1 samples, 0.06%)</title><rect x="97.1847%" y="341" width="0.0563%" height="15" fill="rgb(230,130,0)" fg:x="1726" fg:w="1"/><text x="97.3930%" y="351.50"></text></g><g><title>strcmp (1 samples, 0.06%)</title><rect x="97.1847%" y="325" width="0.0563%" height="15" fill="rgb(229,129,0)" fg:x="1726" fg:w="1"/><text x="97.3930%" y="335.50"></text></g><g><title>perf_tp_event (5 samples, 0.28%)</title><rect x="97.2973%" y="277" width="0.2815%" height="15" fill="rgb(242,142,0)" fg:x="1728" fg:w="5"/><text x="97.5056%" y="287.50"></text></g><g><title>filter_match_preds (5 samples, 0.28%)</title><rect x="97.2973%" y="261" width="0.2815%" height="15" fill="rgb(241,141,0)" fg:x="1728" fg:w="5"/><text x="97.5056%" y="271.50"></text></g><g><title>may_open (9 samples, 0.51%)</title><rect x="97.2410%" y="357" width="0.5068%" height="15" fill="rgb(220,120,0)" fg:x="1727" fg:w="9"/><text x="97.4493%" y="367.50"></text></g><g><title>[unknown] (8 samples, 0.45%)</title><rect x="97.2973%" y="341" width="0.4505%" height="15" fill="rgb(243,112,112)" fg:x="1728" fg:w="8"/><text x="97.5056%" y="351.50"></text></g><g><title>kprobe_ftrace_handler (8 samples, 0.45%)</title><rect x="97.2973%" y="325" width="0.4505%" height="15" fill="rgb(230,130,0)" fg:x="1728" fg:w="8"/><text x="97.5056%" y="335.50"></text></g><g><title>aggr_pre_handler (8 samples, 0.45%)</title><rect x="97.2973%" y="309" width="0.4505%" height="15" fill="rgb(227,127,0)" fg:x="1728" fg:w="8"/><text x="97.5056%" y="319.50"></text></g><g><title>kprobe_perf_func (8 samples, 0.45%)</title><rect x="97.2973%" y="293" width="0.4505%" height="15" fill="rgb(230,130,0)" fg:x="1728" fg:w="8"/><text x="97.5056%" y="303.50"></text></g><g><title>process_fetch_insn (3 samples, 0.17%)</title><rect x="97.5788%" y="277" width="0.1689%" height="15" fill="rgb(242,142,0)" fg:x="1733" fg:w="3"/><text x="97.7872%" y="287.50"></text></g><g><title>copy_from_kernel_nofault (1 samples, 0.06%)</title><rect x="97.6914%" y="261" width="0.0563%" height="15" fill="rgb(229,129,0)" fg:x="1735" fg:w="1"/><text x="97.8998%" y="271.50"></text></g><g><title>__rcu_read_unlock (1 samples, 0.06%)</title><rect x="97.7477%" y="341" width="0.0563%" height="15" fill="rgb(228,128,0)" fg:x="1736" fg:w="1"/><text x="97.9561%" y="351.50"></text></g><g><title>__x64_sys_openat (23 samples, 1.30%)</title><rect x="96.5653%" y="421" width="1.2950%" height="15" fill="rgb(233,133,0)" fg:x="1715" fg:w="23"/><text x="96.7736%" y="431.50"></text></g><g><title>do_sys_openat2 (23 samples, 1.30%)</title><rect x="96.5653%" y="405" width="1.2950%" height="15" fill="rgb(243,143,0)" fg:x="1715" fg:w="23"/><text x="96.7736%" y="415.50"></text></g><g><title>do_filp_open (22 samples, 1.24%)</title><rect x="96.6216%" y="389" width="1.2387%" height="15" fill="rgb(243,143,0)" fg:x="1716" fg:w="22"/><text x="96.8300%" y="399.50"></text></g><g><title>path_openat (22 samples, 1.24%)</title><rect x="96.6216%" y="373" width="1.2387%" height="15" fill="rgb(230,130,0)" fg:x="1716" fg:w="22"/><text x="96.8300%" y="383.50"></text></g><g><title>terminate_walk (2 samples, 0.11%)</title><rect x="97.7477%" y="357" width="0.1126%" height="15" fill="rgb(233,133,0)" fg:x="1736" fg:w="2"/><text x="97.9561%" y="367.50"></text></g><g><title>mntput (1 samples, 0.06%)</title><rect x="97.8041%" y="341" width="0.0563%" height="15" fill="rgb(227,127,0)" fg:x="1737" fg:w="1"/><text x="98.0124%" y="351.50"></text></g><g><title>exit_to_user_mode_prepare (6 samples, 0.34%)</title><rect x="97.8604%" y="405" width="0.3378%" height="15" fill="rgb(230,130,0)" fg:x="1738" fg:w="6"/><text x="98.0687%" y="415.50"></text></g><g><title>fpregs_restore_userregs (6 samples, 0.34%)</title><rect x="97.8604%" y="389" width="0.3378%" height="15" fill="rgb(242,142,0)" fg:x="1738" fg:w="6"/><text x="98.0687%" y="399.50"></text></g><g><title>restore_fpregs_from_fpstate (6 samples, 0.34%)</title><rect x="97.8604%" y="373" width="0.3378%" height="15" fill="rgb(236,136,0)" fg:x="1738" fg:w="6"/><text x="98.0687%" y="383.50"></text></g><g><title>__audit_syscall_exit (2 samples, 0.11%)</title><rect x="98.1982%" y="389" width="0.1126%" height="15" fill="rgb(232,132,0)" fg:x="1744" fg:w="2"/><text x="98.4065%" y="399.50"></text></g><g><title>audit_filter_syscall (2 samples, 0.11%)</title><rect x="98.1982%" y="373" width="0.1126%" height="15" fill="rgb(225,125,0)" fg:x="1744" fg:w="2"/><text x="98.4065%" y="383.50"></text></g><g><title>audit_filter_rules.constprop.0 (2 samples, 0.11%)</title><rect x="98.1982%" y="357" width="0.1126%" height="15" fill="rgb(225,125,0)" fg:x="1744" fg:w="2"/><text x="98.4065%" y="367.50"></text></g><g><title>syscall_exit_to_user_mode (14 samples, 0.79%)</title><rect x="97.8604%" y="421" width="0.7883%" height="15" fill="rgb(237,137,0)" fg:x="1738" fg:w="14"/><text x="98.0687%" y="431.50"></text></g><g><title>syscall_exit_work (8 samples, 0.45%)</title><rect x="98.1982%" y="405" width="0.4505%" height="15" fill="rgb(237,137,0)" fg:x="1744" fg:w="8"/><text x="98.4065%" y="415.50"></text></g><g><title>audit_reset_context.part.0.constprop.0 (6 samples, 0.34%)</title><rect x="98.3108%" y="389" width="0.3378%" height="15" fill="rgb(225,125,0)" fg:x="1746" fg:w="6"/><text x="98.5191%" y="399.50"></text></g><g><title>mntput (6 samples, 0.34%)</title><rect x="98.3108%" y="373" width="0.3378%" height="15" fill="rgb(227,127,0)" fg:x="1746" fg:w="6"/><text x="98.5191%" y="383.50"></text></g><g><title>os.OpenFile (40 samples, 2.25%)</title><rect x="96.4527%" y="533" width="2.2523%" height="15" fill="rgb(92,239,92)" fg:x="1713" fg:w="40"/><text x="96.6610%" y="543.50">os..</text></g><g><title>os.openFileNolog (40 samples, 2.25%)</title><rect x="96.4527%" y="517" width="2.2523%" height="15" fill="rgb(92,239,92)" fg:x="1713" fg:w="40"/><text x="96.6610%" y="527.50">os..</text></g><g><title>syscall.openat (39 samples, 2.20%)</title><rect x="96.5090%" y="501" width="2.1959%" height="15" fill="rgb(93,240,93)" fg:x="1714" fg:w="39"/><text x="96.7173%" y="511.50">sy..</text></g><g><title>syscall.Syscall6 (39 samples, 2.20%)</title><rect x="96.5090%" y="485" width="2.1959%" height="15" fill="rgb(93,240,93)" fg:x="1714" fg:w="39"/><text x="96.7173%" y="495.50">sy..</text></g><g><title>runtime/internal/syscall.Syscall6 (39 samples, 2.20%)</title><rect x="96.5090%" y="469" width="2.1959%" height="15" fill="rgb(90,237,90)" fg:x="1714" fg:w="39"/><text x="96.7173%" y="479.50">ru..</text></g><g><title>entry_SYSCALL_64_after_hwframe (39 samples, 2.20%)</title><rect x="96.5090%" y="453" width="2.1959%" height="15" fill="rgb(245,145,0)" fg:x="1714" fg:w="39"/><text x="96.7173%" y="463.50">en..</text></g><g><title>do_syscall_64 (38 samples, 2.14%)</title><rect x="96.5653%" y="437" width="2.1396%" height="15" fill="rgb(243,143,0)" fg:x="1715" fg:w="38"/><text x="96.7736%" y="447.50">do..</text></g><g><title>syscall_trace_enter.constprop.0 (1 samples, 0.06%)</title><rect x="98.6486%" y="421" width="0.0563%" height="15" fill="rgb(237,137,0)" fg:x="1752" fg:w="1"/><text x="98.8570%" y="431.50"></text></g><g><title>__audit_syscall_entry (1 samples, 0.06%)</title><rect x="98.6486%" y="405" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="1752" fg:w="1"/><text x="98.8570%" y="415.50"></text></g><g><title>internal/poll.runtime_pollClose (3 samples, 0.17%)</title><rect x="98.7050%" y="437" width="0.1689%" height="15" fill="rgb(92,239,92)" fg:x="1753" fg:w="3"/><text x="98.9133%" y="447.50"></text></g><g><title>runtime/internal/syscall.Syscall6 (3 samples, 0.17%)</title><rect x="98.7050%" y="421" width="0.1689%" height="15" fill="rgb(90,237,90)" fg:x="1753" fg:w="3"/><text x="98.9133%" y="431.50"></text></g><g><title>entry_SYSCALL_64_after_hwframe (3 samples, 0.17%)</title><rect x="98.7050%" y="405" width="0.1689%" height="15" fill="rgb(245,145,0)" fg:x="1753" fg:w="3"/><text x="98.9133%" y="415.50"></text></g><g><title>do_syscall_64 (3 samples, 0.17%)</title><rect x="98.7050%" y="389" width="0.1689%" height="15" fill="rgb(243,143,0)" fg:x="1753" fg:w="3"/><text x="98.9133%" y="399.50"></text></g><g><title>__x64_sys_epoll_ctl (3 samples, 0.17%)</title><rect x="98.7050%" y="373" width="0.1689%" height="15" fill="rgb(233,133,0)" fg:x="1753" fg:w="3"/><text x="98.9133%" y="383.50"></text></g><g><title>do_epoll_ctl (3 samples, 0.17%)</title><rect x="98.7050%" y="357" width="0.1689%" height="15" fill="rgb(243,143,0)" fg:x="1753" fg:w="3"/><text x="98.9133%" y="367.50"></text></g><g><title>ep_remove (1 samples, 0.06%)</title><rect x="98.8176%" y="341" width="0.0563%" height="15" fill="rgb(239,139,0)" fg:x="1755" fg:w="1"/><text x="99.0259%" y="351.50"></text></g><g><title>kmem_cache_free (1 samples, 0.06%)</title><rect x="98.8176%" y="325" width="0.0563%" height="15" fill="rgb(220,120,0)" fg:x="1755" fg:w="1"/><text x="99.0259%" y="335.50"></text></g><g><title>obj_cgroup_uncharge_pages (1 samples, 0.06%)</title><rect x="98.8176%" y="309" width="0.0563%" height="15" fill="rgb(229,129,0)" fg:x="1755" fg:w="1"/><text x="99.0259%" y="319.50"></text></g><g><title>__rcu_read_lock (1 samples, 0.06%)</title><rect x="98.8176%" y="293" width="0.0563%" height="15" fill="rgb(228,128,0)" fg:x="1755" fg:w="1"/><text x="99.0259%" y="303.50"></text></g><g><title>perf_tp_event (3 samples, 0.17%)</title><rect x="98.8739%" y="229" width="0.1689%" height="15" fill="rgb(242,142,0)" fg:x="1756" fg:w="3"/><text x="99.0822%" y="239.50"></text></g><g><title>filter_match_preds (1 samples, 0.06%)</title><rect x="98.9865%" y="213" width="0.0563%" height="15" fill="rgb(241,141,0)" fg:x="1758" fg:w="1"/><text x="99.1948%" y="223.50"></text></g><g><title>runtime/internal/syscall.Syscall6 (5 samples, 0.28%)</title><rect x="98.8739%" y="405" width="0.2815%" height="15" fill="rgb(90,237,90)" fg:x="1756" fg:w="5"/><text x="99.0822%" y="415.50"></text></g><g><title>entry_SYSCALL_64_after_hwframe (5 samples, 0.28%)</title><rect x="98.8739%" y="389" width="0.2815%" height="15" fill="rgb(245,145,0)" fg:x="1756" fg:w="5"/><text x="99.0822%" y="399.50"></text></g><g><title>do_syscall_64 (5 samples, 0.28%)</title><rect x="98.8739%" y="373" width="0.2815%" height="15" fill="rgb(243,143,0)" fg:x="1756" fg:w="5"/><text x="99.0822%" y="383.50"></text></g><g><title>syscall_exit_to_user_mode (5 samples, 0.28%)</title><rect x="98.8739%" y="357" width="0.2815%" height="15" fill="rgb(237,137,0)" fg:x="1756" fg:w="5"/><text x="99.0822%" y="367.50"></text></g><g><title>exit_to_user_mode_prepare (5 samples, 0.28%)</title><rect x="98.8739%" y="341" width="0.2815%" height="15" fill="rgb(230,130,0)" fg:x="1756" fg:w="5"/><text x="99.0822%" y="351.50"></text></g><g><title>task_work_run (5 samples, 0.28%)</title><rect x="98.8739%" y="325" width="0.2815%" height="15" fill="rgb(222,122,0)" fg:x="1756" fg:w="5"/><text x="99.0822%" y="335.50"></text></g><g><title>__fput (5 samples, 0.28%)</title><rect x="98.8739%" y="309" width="0.2815%" height="15" fill="rgb(228,128,0)" fg:x="1756" fg:w="5"/><text x="99.0822%" y="319.50"></text></g><g><title>ima_file_free (5 samples, 0.28%)</title><rect x="98.8739%" y="293" width="0.2815%" height="15" fill="rgb(228,128,0)" fg:x="1756" fg:w="5"/><text x="99.0822%" y="303.50"></text></g><g><title>[unknown] (5 samples, 0.28%)</title><rect x="98.8739%" y="277" width="0.2815%" height="15" fill="rgb(243,112,112)" fg:x="1756" fg:w="5"/><text x="99.0822%" y="287.50"></text></g><g><title>kprobe_ftrace_handler (5 samples, 0.28%)</title><rect x="98.8739%" y="261" width="0.2815%" height="15" fill="rgb(230,130,0)" fg:x="1756" fg:w="5"/><text x="99.0822%" y="271.50"></text></g><g><title>kprobe_perf_func (5 samples, 0.28%)</title><rect x="98.8739%" y="245" width="0.2815%" height="15" fill="rgb(230,130,0)" fg:x="1756" fg:w="5"/><text x="99.0822%" y="255.50"></text></g><g><title>process_fetch_insn (2 samples, 0.11%)</title><rect x="99.0428%" y="229" width="0.1126%" height="15" fill="rgb(242,142,0)" fg:x="1759" fg:w="2"/><text x="99.2511%" y="239.50"></text></g><g><title>copy_from_kernel_nofault (2 samples, 0.11%)</title><rect x="99.0428%" y="213" width="0.1126%" height="15" fill="rgb(229,129,0)" fg:x="1759" fg:w="2"/><text x="99.2511%" y="223.50"></text></g><g><title>copy_from_kernel_nofault_allowed (1 samples, 0.06%)</title><rect x="99.0991%" y="197" width="0.0563%" height="15" fill="rgb(229,129,0)" fg:x="1760" fg:w="1"/><text x="99.3074%" y="207.50"></text></g><g><title>internal/poll.(*FD).decref (9 samples, 0.51%)</title><rect x="98.7050%" y="469" width="0.5068%" height="15" fill="rgb(92,239,92)" fg:x="1753" fg:w="9"/><text x="98.9133%" y="479.50"></text></g><g><title>internal/poll.(*FD).destroy (9 samples, 0.51%)</title><rect x="98.7050%" y="453" width="0.5068%" height="15" fill="rgb(92,239,92)" fg:x="1753" fg:w="9"/><text x="98.9133%" y="463.50"></text></g><g><title>syscall.Close (6 samples, 0.34%)</title><rect x="98.8739%" y="437" width="0.3378%" height="15" fill="rgb(93,240,93)" fg:x="1756" fg:w="6"/><text x="99.0822%" y="447.50"></text></g><g><title>syscall.Syscall (6 samples, 0.34%)</title><rect x="98.8739%" y="421" width="0.3378%" height="15" fill="rgb(93,240,93)" fg:x="1756" fg:w="6"/><text x="99.0822%" y="431.50"></text></g><g><title>syscall.RawSyscall6 (1 samples, 0.06%)</title><rect x="99.1554%" y="405" width="0.0563%" height="15" fill="rgb(93,240,93)" fg:x="1761" fg:w="1"/><text x="99.3637%" y="415.50"></text></g><g><title>internal/poll.(*FD).Close (10 samples, 0.56%)</title><rect x="98.7050%" y="485" width="0.5631%" height="15" fill="rgb(92,239,92)" fg:x="1753" fg:w="10"/><text x="98.9133%" y="495.50"></text></g><g><title>internal/poll.(*fdMutex).increfAndClose (1 samples, 0.06%)</title><rect x="99.2117%" y="469" width="0.0563%" height="15" fill="rgb(92,239,92)" fg:x="1762" fg:w="1"/><text x="99.4200%" y="479.50"></text></g><g><title>os.(*file).close (11 samples, 0.62%)</title><rect x="98.7050%" y="501" width="0.6194%" height="15" fill="rgb(92,239,92)" fg:x="1753" fg:w="11"/><text x="98.9133%" y="511.50"></text></g><g><title>runtime.SetFinalizer (1 samples, 0.06%)</title><rect x="99.2680%" y="485" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1763" fg:w="1"/><text x="99.4764%" y="495.50"></text></g><g><title>os.ReadFile.func1 (12 samples, 0.68%)</title><rect x="98.7050%" y="533" width="0.6757%" height="15" fill="rgb(92,239,92)" fg:x="1753" fg:w="12"/><text x="98.9133%" y="543.50"></text></g><g><title>os.(*File).Close (12 samples, 0.68%)</title><rect x="98.7050%" y="517" width="0.6757%" height="15" fill="rgb(92,239,92)" fg:x="1753" fg:w="12"/><text x="98.9133%" y="527.50"></text></g><g><title>runtime.SetFinalizer (1 samples, 0.06%)</title><rect x="99.3243%" y="501" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1764" fg:w="1"/><text x="99.5327%" y="511.50"></text></g><g><title>runtime.growslice (4 samples, 0.23%)</title><rect x="99.3806%" y="533" width="0.2252%" height="15" fill="rgb(90,237,90)" fg:x="1765" fg:w="4"/><text x="99.5890%" y="543.50"></text></g><g><title>runtime.mallocgc (2 samples, 0.11%)</title><rect x="99.4932%" y="517" width="0.1126%" height="15" fill="rgb(90,237,90)" fg:x="1767" fg:w="2"/><text x="99.7016%" y="527.50"></text></g><g><title>runtime.(*mcache).nextFree (1 samples, 0.06%)</title><rect x="99.5495%" y="501" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1768" fg:w="1"/><text x="99.7579%" y="511.50"></text></g><g><title>runtime.(*mcache).refill (1 samples, 0.06%)</title><rect x="99.5495%" y="485" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1768" fg:w="1"/><text x="99.7579%" y="495.50"></text></g><g><title>runtime.(*mcentral).cacheSpan (1 samples, 0.06%)</title><rect x="99.5495%" y="469" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1768" fg:w="1"/><text x="99.7579%" y="479.50"></text></g><g><title>runtime.(*mcentral).grow (1 samples, 0.06%)</title><rect x="99.5495%" y="453" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1768" fg:w="1"/><text x="99.7579%" y="463.50"></text></g><g><title>runtime.systemstack.abi0 (1 samples, 0.06%)</title><rect x="99.5495%" y="437" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1768" fg:w="1"/><text x="99.7579%" y="447.50"></text></g><g><title>runtime.(*mheap).alloc.func1 (1 samples, 0.06%)</title><rect x="99.5495%" y="421" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1768" fg:w="1"/><text x="99.7579%" y="431.50"></text></g><g><title>runtime.(*mheap).allocSpan (1 samples, 0.06%)</title><rect x="99.5495%" y="405" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1768" fg:w="1"/><text x="99.7579%" y="415.50"></text></g><g><title>runtime.(*mheap).initSpan (1 samples, 0.06%)</title><rect x="99.5495%" y="389" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1768" fg:w="1"/><text x="99.7579%" y="399.50"></text></g><g><title>runtime.(*mheap).allocNeedsZero (1 samples, 0.06%)</title><rect x="99.5495%" y="373" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1768" fg:w="1"/><text x="99.7579%" y="383.50"></text></g><g><title>runtime.makeslice (1 samples, 0.06%)</title><rect x="99.6059%" y="533" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1769" fg:w="1"/><text x="99.8142%" y="543.50"></text></g><g><title>runtime.mallocgc (1 samples, 0.06%)</title><rect x="99.6059%" y="517" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1769" fg:w="1"/><text x="99.8142%" y="527.50"></text></g><g><title>runtime.(*mcache).nextFree (1 samples, 0.06%)</title><rect x="99.6059%" y="501" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1769" fg:w="1"/><text x="99.8142%" y="511.50"></text></g><g><title>runtime.(*mcache).refill (1 samples, 0.06%)</title><rect x="99.6059%" y="485" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1769" fg:w="1"/><text x="99.8142%" y="495.50"></text></g><g><title>runtime.(*mcentral).uncacheSpan (1 samples, 0.06%)</title><rect x="99.6059%" y="469" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1769" fg:w="1"/><text x="99.8142%" y="479.50"></text></g><g><title>runtime.(*spanSet).push (1 samples, 0.06%)</title><rect x="99.6059%" y="453" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1769" fg:w="1"/><text x="99.8142%" y="463.50"></text></g><g><title>find_vma (1 samples, 0.06%)</title><rect x="99.6622%" y="469" width="0.0563%" height="15" fill="rgb(240,140,0)" fg:x="1770" fg:w="1"/><text x="99.8705%" y="479.50"></text></g><g><title>mt_find (1 samples, 0.06%)</title><rect x="99.6622%" y="453" width="0.0563%" height="15" fill="rgb(216,116,0)" fg:x="1770" fg:w="1"/><text x="99.8705%" y="463.50"></text></g><g><title>__handle_mm_fault (1 samples, 0.06%)</title><rect x="99.7185%" y="453" width="0.0563%" height="15" fill="rgb(226,126,0)" fg:x="1771" fg:w="1"/><text x="99.9268%" y="463.50"></text></g><g><title>page_add_new_anon_rmap (1 samples, 0.06%)</title><rect x="99.7185%" y="437" width="0.0563%" height="15" fill="rgb(231,131,0)" fg:x="1771" fg:w="1"/><text x="99.9268%" y="447.50"></text></g><g><title>__mod_lruvec_page_state (1 samples, 0.06%)</title><rect x="99.7185%" y="421" width="0.0563%" height="15" fill="rgb(232,132,0)" fg:x="1771" fg:w="1"/><text x="99.9268%" y="431.50"></text></g><g><title>__rcu_read_lock (1 samples, 0.06%)</title><rect x="99.7185%" y="405" width="0.0563%" height="15" fill="rgb(228,128,0)" fg:x="1771" fg:w="1"/><text x="99.9268%" y="415.50"></text></g><g><title>runtime.memclrNoHeapPointers (3 samples, 0.17%)</title><rect x="99.6622%" y="533" width="0.1689%" height="15" fill="rgb(90,237,90)" fg:x="1770" fg:w="3"/><text x="99.8705%" y="543.50"></text></g><g><title>asm_exc_page_fault (3 samples, 0.17%)</title><rect x="99.6622%" y="517" width="0.1689%" height="15" fill="rgb(231,131,0)" fg:x="1770" fg:w="3"/><text x="99.8705%" y="527.50"></text></g><g><title>exc_page_fault (3 samples, 0.17%)</title><rect x="99.6622%" y="501" width="0.1689%" height="15" fill="rgb(235,135,0)" fg:x="1770" fg:w="3"/><text x="99.8705%" y="511.50"></text></g><g><title>do_user_addr_fault (3 samples, 0.17%)</title><rect x="99.6622%" y="485" width="0.1689%" height="15" fill="rgb(243,143,0)" fg:x="1770" fg:w="3"/><text x="99.8705%" y="495.50"></text></g><g><title>handle_mm_fault (2 samples, 0.11%)</title><rect x="99.7185%" y="469" width="0.1126%" height="15" fill="rgb(231,131,0)" fg:x="1771" fg:w="2"/><text x="99.9268%" y="479.50"></text></g><g><title>mem_cgroup_from_task (1 samples, 0.06%)</title><rect x="99.7748%" y="453" width="0.0563%" height="15" fill="rgb(230,130,0)" fg:x="1772" fg:w="1"/><text x="99.9831%" y="463.50"></text></g><g><title>os.ReadFile (1,773 samples, 99.83%)</title><rect x="0.0563%" y="549" width="99.8311%" height="15" fill="rgb(92,239,92)" fg:x="1" fg:w="1773"/><text x="0.2646%" y="559.50">os.ReadFile</text></g><g><title>runtime.memmove (1 samples, 0.06%)</title><rect x="99.8311%" y="533" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1773" fg:w="1"/><text x="100.0394%" y="543.50"></text></g><g><title>main.manualBenchmark (1,775 samples, 99.94%)</title><rect x="0.0000%" y="581" width="99.9437%" height="15" fill="rgb(71,219,71)" fg:x="0" fg:w="1775"/><text x="0.2083%" y="591.50">main.manualBenchmark</text></g><g><title>main.getStatsManual (1,774 samples, 99.89%)</title><rect x="0.0563%" y="565" width="99.8874%" height="15" fill="rgb(71,219,71)" fg:x="1" fg:w="1774"/><text x="0.2646%" y="575.50">main.getStatsManual</text></g><g><title>runtime.growslice (1 samples, 0.06%)</title><rect x="99.8874%" y="549" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1774" fg:w="1"/><text x="100.0957%" y="559.50"></text></g><g><title>runtime.mallocgc (1 samples, 0.06%)</title><rect x="99.8874%" y="533" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1774" fg:w="1"/><text x="100.0957%" y="543.50"></text></g><g><title>all (1,776 samples, 100%)</title><rect x="0.0000%" y="709" width="100.0000%" height="15" fill="rgb(255,130,130)" fg:x="0" fg:w="1776"/><text x="0.2083%" y="719.50"></text></g><g><title>derp (1,776 samples, 100.00%)</title><rect x="0.0000%" y="693" width="100.0000%" height="15" fill="rgb(247,119,119)" fg:x="0" fg:w="1776"/><text x="0.2083%" y="703.50">derp</text></g><g><title>runtime.goexit.abi0 (1,776 samples, 100.00%)</title><rect x="0.0000%" y="677" width="100.0000%" height="15" fill="rgb(90,237,90)" fg:x="0" fg:w="1776"/><text x="0.2083%" y="687.50">runtime.goexit.abi0</text></g><g><title>runtime.main (1,776 samples, 100.00%)</title><rect x="0.0000%" y="661" width="100.0000%" height="15" fill="rgb(90,237,90)" fg:x="0" fg:w="1776"/><text x="0.2083%" y="671.50">runtime.main</text></g><g><title>main.main (1,776 samples, 100.00%)</title><rect x="0.0000%" y="645" width="100.0000%" height="15" fill="rgb(71,219,71)" fg:x="0" fg:w="1776"/><text x="0.2083%" y="655.50">main.main</text></g><g><title>main.runBenchmarks (1,776 samples, 100.00%)</title><rect x="0.0000%" y="629" width="100.0000%" height="15" fill="rgb(71,219,71)" fg:x="0" fg:w="1776"/><text x="0.2083%" y="639.50">main.runBenchmarks</text></g><g><title>main.benchmarkFunc (1,776 samples, 100.00%)</title><rect x="0.0000%" y="613" width="100.0000%" height="15" fill="rgb(71,219,71)" fg:x="0" fg:w="1776"/><text x="0.2083%" y="623.50">main.benchmarkFunc</text></g><g><title>main.runBenchmarks.func1 (1,776 samples, 100.00%)</title><rect x="0.0000%" y="597" width="100.0000%" height="15" fill="rgb(71,219,71)" fg:x="0" fg:w="1776"/><text x="0.2083%" y="607.50">main.runBenchmarks.func1</text></g><g><title>runtime.morestack.abi0 (1 samples, 0.06%)</title><rect x="99.9437%" y="581" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1775" fg:w="1"/><text x="100.1520%" y="591.50"></text></g><g><title>runtime.newstack (1 samples, 0.06%)</title><rect x="99.9437%" y="565" width="0.0563%" height="15" fill="rgb(90,237,90)" fg:x="1775" fg:w="1"/><text x="100.1520%" y="575.50"></text></g></svg></svg>
Display the source blob
Display the rendered blob
Raw
two-loops.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@bobrik
Copy link
Author

bobrik commented Aug 11, 2023

Possible fix: bobrik/linux@50b6278

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment