perfetto.rst

Using AGI and Perfetto for Vulkan CPU profiling

In cases where the application is not GPU bound, it is useful to look into how much time were spent in each Vulkan API calls: this can reveal driver overhead as well as unexpected fence wait/stalls.

This document describes how you can use the CPU timing layer from AGI to profile Vulkan API calls.

Run

To capture a trace with perfetto you need to take the following steps:

1. Download and extract AGI binaries from GitHub.

2. Copy lib/CPUTiming.json to ~/.local/share/vulkan/implicit_layer.d/.

   Edit the copied file, replacing <library> with /absolute/path/to/AGI/lib/libVkLayer_CPUTiming.so.

   Add the following lines inside the "layer" object.

   "disable_environment": { "DISABLE_CPU_TIMING": "1" },
   "enable_environment": { "ENABLE_CPU_TIMING": "1" },

3. Build perfetto from sources available at subprojects/perfetto following this guide.

4. Copy the following trace config into a cpu_timing.cfg file:

   buffers {
       size_kb: 20480,
       fill_policy: DISCARD,
   }
   
   data_sources {
       config {
           name: "VulkanCPUTiming"
           target_buffer: 0
           legacy_config: "VkDevice:VkPhysicalDevice:VkCommandBuffer:VkQueue"
       }
   }
   
   duration_ms: 1000

   You can modify the trace duration (duration_ms) according to your need. The legacy_config line specifies a colon-separated filter: it's based on the list here and entries can be removed to reduce trace file size.

5. Change directory to subprojects/perfetto and run a convenience script to start the tracing service:

   cd subprojects/perfetto
   CONFIG=<path/to/cpu_timing.cfg> OUT=out/linux_clang_release ./tools/tmux -n

6. Start the target application with ENABLE_CPU_TIMING=1 in the environment. This can be done from another terminal or from a GUI launcher.
7. Start perfetto under the tmux session initiated in step 5. You can modify the command line to "sleep <DELAY> && ... perfetto ..." to insert a delay before tracing; which is convenient when you need to switch back to the application before tracing.
8. Once tracing has finished, you can detach from tmux with Ctrl+b, d, and the convenience script should automatically copy the trace files into $HOME/Downloads.
9. Go to ui.perfetto.dev and upload $HOME/Downloads/trace.protobuf by clicking on Open trace file.
10. Alternatively you can open the trace in AGI (which despite the name can be used to view non-android traces).