dvessel/F-Zero-Sand-Ocean-Intro.dtm

## readme.md

      
    Raw
  

              readme.md
            
          
    Benchmarking Dolphin on MacOS

This script will automate repeatable frame times for Metal, Vulkan or OpenGL backends.
A .dtm replay file is required. It is used to repeat the exact same sequence for every run.
usage: ./gfx-render-time.sh --help
  --dtm|-p replay.dtm
  Path to dtm replay file. This is required.

  --metal|--vulkan|--opengl
  The graphics backend to test. If none are set, only Metal will be tested.

  --repeat|-r 1
  How many runs for each graphics backend. Defaults to 1.

  --dolphin|-o /Application/Dolphin.app
  Path to Dolphin.app. Defaults to Application folder.

  --game|-g game.gcz
  The path will be detected automatically for rvz, iso and wia files.
  For other rom types, the path must be set manually.

  --hud
  Performance HUD for Metal and Vulkan graphics backend's. Ventura and later.

  --config=|-C System.Section.Key=Value
  Options accepted by Dolphin will be passed to it.
  https://wiki.dolphin-emu.org/index.php?title=GameINI

Example:

Using the included sample file: F-Zero-Sand-Ocean-Intro.dtm
enter in terminal: (Ctrl-C to cancel)
./gfx-render-time.sh --dtm F-Zero-Sand-Ocean-Intro.dtm --metal --vulkan --repeat 3
output: (This is also written to summary.txt.)
1/3 metal................5.085ms
Run completed. Sampled 11436 frames.
2/3 metal................5.088ms
Run completed. Sampled 11436 frames.
3/3 metal................5.104ms
Run completed. Sampled 11436 frames.
11436 frames 5.085ms metal pass 1
11436 frames 5.088ms metal pass 2
11436 frames 5.104ms metal pass 3
34308 frames 5.092ms complete

1/3 vulkan...................6.075ms
Run completed. Sampled 11436 frames.
2/3 vulkan...................6.085ms
Run completed. Sampled 11436 frames.
3/3 vulkan...................6.070ms
Run completed. Sampled 11436 frames.
11436 frames 6.075ms vulkan pass 1
11436 frames 6.085ms vulkan pass 2
11436 frames 6.070ms vulkan pass 3
34308 frames 6.077ms complete

Results: F-Zero\ Sand\ Ocean\ Intro\ 09.15,09.36.04

The output will be located next to the .dtm file in a dated folder. Each backend pass will list frame time in milliseconds. Console output will be hidden from the terminal and redirected to console.txt.
├── F-Zero-Sand-Ocean-Intro 09.15,09.36.04
│   ├── _summary.txt
│   ├── console.txt
│   ├── metal pass 1.txt
│   ├── metal pass 2.txt
│   ├── metal pass 3.txt
│   ├── vulkan pass 1.txt
│   ├── vulkan pass 2.txt
│   └── vulkan pass 3.txt
└── F-Zero-Sand-Ocean-Intro.dtm

More examples:

Run a copy of Dolphin outside the default /Applications folder:
./gfx-render-time.sh --dolphin ~/Downloads/Dolphin.app --dtm F-Zero-Sand-Ocean-Intro.dtm
The following will set the resolution to 4x native and disable custom texture loading. More can be found in the Dolphin Wiki:
./gfx-render-time.sh --dtm F-Zero-Sand-Ocean-Intro.dtm \
  -C Graphics.Settings.InternalResolution=4 -C Graphics.Settings.HiresTextures=False
Notes and cavets:


Replay.dtm files are portable between Apple ARM and Intel machines.
Errors emitted by Dolphin will not be visible in the terminal. This is due to Dolphin emitting all console output to stderr making it difficult to differentiate in the script. Check the console.txt file if it's not running properly.
Disable Config / General / Enable Cheats for recording and playback or risk desyncronization.
Disable Graphics / Hacks / Save Texture Cache to State when recording with save states.
Using save states is fragile. They tend to break between Dolphin revisions.
Save states are tied to the Game Cube bios. Using a real bios during recording will fail to load when playback depends on the default bios provided by Dolphin.
When sharing or moving replay files between machines, try to minimize any custom settings while recording unless they are part of what's being measured. Keep in mind any game specific configurations (from game properties) since the performance numbers can be affected by it.
The .dtm file will embed some settings when the recording is made. By consequence, they will be inherited during playback.  They cannot be changed through --config= options.
The graphics backend is written to the .dtm file. The script patches it so it's so it runs for all backends. This is generally not a problem. The only side effect is that games that depend on EFB CPU access can desync. Metroid Prime 2 visor scan lock is an example.


embeded settings
settings location


MAIN_CPU_THREAD
Config / General / Dual Core


MAIN_DSP_HLE
Config / Audio / DSP Emulation Engine


MAIN_CPU_CORE
Config / Advanced / CPU Emulation Engine


MAIN_SYNC_GPU
Properties Menu / Game Config / Syncronize GPU Thread


MAIN_FAST_DISC_SPEED
Properties Menu / Game Config / Speed up Disc Transfer Rate


SYSCONF_LANGUAGE or MAIN_GC_LANGUAGE
Config / Wii or GameCube / System Language


SYSCONF_PAL60
Config / Wii / Use PAL60 Mode


SYSCONF_PROGRESSIVE_SCAN
Graphics / Advanced / Enable Progressive Scan


GFX_HACK_EFB_ACCESS_ENABLE
Graphics / Hacks / Skip EFB Access from CPU


GFX_HACK_SKIP_EFB_COPY_TO_RAM
Graphics / Hacks / Store EFB Copies to Texture Only


GFX_HACK_EFB_EMULATE_FORMAT_CHANGES
Graphics / Hacks / Ignore Format Changes


GFX_HACK_SKIP_XFB_COPY_TO_RAM
Graphics / Hacks / Store XFB Copies to Texture Only


GFX_HACK_IMMEDIATE_XFB
Graphics / Hacks / Immediatly Present XFB


SESSION_USE_FMA


MAIN_JIT_FOLLOW_BRANCH


## F-Zero-Sand-Ocean-Intro.dtm

      
    Raw
  

              F-Zero-Sand-Ocean-Intro.dtm
            
          
            View raw
        
    
## gfx-render-time.sh
#!/usr/bin/env zsh

set -e -o pipefail

if (( $argv[(I)(--help|-h)] )); then
  printf "%s\n" "
  --dtm|-p replay.dtm
  Path to dtm replay file. This is required.

  --metal|--vulkan|--opengl|--null
  The graphics backend to test. If none are set, only Metal will be tested.

  --repeat|-r 1
  How many runs for each graphics backend. Defaults to 1.

  --dolphin|-o /Application/Dolphin.app
  Path to Dolphin.app. Defaults to Application folder.

  --game|-g game.gcz
  The path will be detected automatically for rvz, iso and wia files.
  For other rom types, the path must be set manually.

  --hud
  Performance HUD for Metal and Vulkan graphics backend's. Ventura and later.

  --fps
  Show fps/vps/% and frame counter. Incompatibel with --hud.

  --config=|-C System.Section.Key=Value
  Options accepted by Dolphin will be passed to it.
  https://wiki.dolphin-emu.org/index.php?title=GameINI
  "
  exit
fi

dtmpath=${argv[( $argv[(i)--dtm|-p] + 1 )]##-*}
if [[ -f $dtmpath ]]; then
  gameid=`xxd -seek 4 -len 6 $dtmpath | xxd -rp -seek -4`
else
  printf "%s\n" "[--dtm replay.dtm] not set or the file doesn't exist." >&2
  exit 1
fi

backends=( metal vulkan opengl null )
for backend in $backends; if (( $argv[(I)--$backend] )); then
  _backends+=( $backend )
fi
backends=( ${_backends:-metal} )

reploop=${${argv[( $argv[(i)--repeat|-r] + 1 )]##-*}:-1}

dolphin=${${argv[( $argv[(i)--dolphin|-o] + 1 )]##-*}:-/Applications/Dolphin.app}
if [[ ! -d $dolphin ]]; then
  printf "%s\n" "$dolphin not found. Set --dolphin with a path to the application." >&2
  exit 1
fi

# Follow --user= path.
usr_default=~/Library/Application\ Support/Dolphin
usr_dir=${${${argv[$argv[(i)--user=*]]##--user=}:-${argv[( $argv[(i)-u] + 1 )]##-*}}:-$usr_default}
if [[ ! -d $usr_dir ]]; then
  printf "%s\n" "$usr_dir not found. Set --user path to an existing support folder." >&2
  exit 1
fi

gamepath=${argv[( $argv[(i)--game|-g] + 1 )]##-*}
if [[ -z $gamepath ]]; then
  isopaths=( `sed -n "s|ISOPath[0-9][^/]*\(/.*\)|\1|p" $usr_dir/Config/Dolphin.ini` )
  while read -r g; do if [[ `file -b $g` =~ $gameid,.*\)$ ]]; then
    gamepath=$g; break
  fi; done < <( find -E $isopaths(N) -regex ".+\.(rvz|iso|wia)" )

  if [[ -z $gamepath ]]; then
    printf "%s\n" "Game not found for $gameid. Set it with --game path2/game.xyz" >&2; exit 1
  fi
elif [[ ! -f $gamepath ]]; then
  printf "%s\n" "File not found: --game $gamepath" >&2; exit 1
fi

if (( $argv[(I)--hud] )); then
  export MTL_HUD_ENABLED=1
elif (( $argv[(I)--fps] )); then
  argv+=--config=Graphics.Settings.ShowFPS=True
  argv+=--config=Graphics.Settings.ShowVPS=True
  argv+=--config=Graphics.Settings.ShowSpeed=True
  argv+=--config=Dolphin.General.ShowFrameCount=True
fi

cl=(
  --dtm -p $dtmpath
  --metal --vulkan --opengl --null
  --repeat -r $reloop
  --dolphin -o $dolphin
  --game -g $game
  --hud
)

outputdir="$dtmpath:r `date "+%m.%d,%H.%M.%S"`"
mkdir -p $outputdir

summary=$outputdir/_summary.txt
printf "%s\n\n" "Summary for $gamepath:t [$gameid] - $dtmpath:t." > $summary

# Patch the .dtm file since the backend is written to it.
# This will only be a problem if a game depends on EFB CPU access such
# as Metroid Prime 2 or Super Mario Galaxy. Does not affect most games.
# Extracted with: `xxd -s 81 -l 6 replay.dtm`
typeset -A hexsplice=(
  metal  "00000051: 4d65 7461 6c00                           Metal."
  vulkan "00000051: 5675 6c6b 616e                           Vulkan"
  opengl "00000051: 4f47 4c00 0000                           OGL..."
  null   "00000051: 4e75 6c6c 0000                           Null.."
)

# Clean-up after manual interrupt.
trap 'rm -f $outputdir/.$backend.dtm*; exit' INT

for backend in $backends; do

  head -c 81 $dtmpath > $outputdir/.$backend.dtm
  echo -n $hexsplice[$backend] | xxd -rp >> $outputdir/.$backend.dtm
  tail -c +88 $dtmpath >> $outputdir/.$backend.dtm
  if [[ -f $dtmpath.sav ]]; then
    ln -sf ../$dtmpath:t.sav $outputdir/.$backend.dtm.sav
  fi

  for i in {1..$reploop}; do
    find $usr_dir/Logs -name '*time*.txt' -delete

    # Launch asyncronously.
    $dolphin/Contents/MacOS/Dolphin ${@:|cl} \
      --batch \
      --exec=$gamepath \
      --movie=$outputdir/.$backend.dtm \
      --config=Graphics.Settings.WaitForShadersBeforeStarting=True \
      --config=Graphics.Settings.LogRenderTimeToFile=True \
      --config=Dolphin.Core.EmulationSpeed=0 \
      --config=Dolphin.DSP.Backend=No\ Audio\ Output \
      --config=Dolphin.Movie.PauseMovie=True \
      --config=Dolphin.Interface.ConfirmStop=False \
      --config=Dolphin.Interface.OnScreenDisplayMessages=True \
      &>> $outputdir/console.txt & pid=$!

    in=${(l:${#reploop}::0:)i}

    printf "$in/$reploop $backend" >&1 >> $summary
    printf "\n%s %s\n\n" "$_" "$*" >> $outputdir/console.txt

    # Ensure Dolphin is running and the render log is created before continuing.
    while [[ ! -f $renderlog && `ps -p $pid` =~ $pid ]]; do
      sleep .5; renderlog=`find $usr_dir/Logs -name 'render_time*'`
    done

    fcount=0 l=, c=.
    if [[ -f $renderlog ]]; then
      while test $l != $c && sleep 2; do
        printf . >&1 >> $summary
        l=$c c=`date -r $renderlog +%s`
      done

      if [[ -s $renderlog ]]; then
        result=$outputdir/$backend\ pass\ $in.txt
        tail -n +20 $renderlog > $result
        printf "%.3fms" `awk '!(T+= $NF) END { print T/NR }' $result` >&1 >> $summary
        fcount=`printf "%i" $(wc -l < $result)`

        vblanklog=`find $usr_dir/Logs -name 'v*blank_times*'`
        if [[ -n $vblanklog ]]; then
          tail -n +20 $vblanklog > $outputdir/$backend\ v-blanks\ $in.txt
        fi
      fi
    fi

    if [[ `ps -p $pid` =~ $pid ]]; then
      kill $pid
      printf "\n%s\n" "Run completed. Sampled $fcount frames." >&1 >> $summary
      # Wii games need a moment to shut down. Wait until it completely quits.
      while [[ `ps -p $pid` =~ $pid ]] sleep .5
    else
      printf "\n%s\n" "Warning! Early termination! Sampled $fcount frames." >&2 >> $summary
    fi
  done

  set=( $outputdir/$backend\ pass\ *.txt(N) )
  vbs=( $outputdir/$backend\ v-blanks\ *.txt(N) )

  if [[ ${#set[@]} > 0 ]]; then
    l=`printf "%i" $(wc -l < $set)`

    if [[ ${#set[@]} > 1 ]]; then
      for ((i = 1; i <= $#set; i++)); do
        [[ ${#vbs[@]} > 0 ]] &&
          vb=`printf ", v-blanks %.3fms - " $(awk '!(T+= $NF) END { print T/NR }' $vbs[i])` || vb=' - '
        printf "%${#l}i frames %.3fms%s%s\n" \
          `wc -l < $set[i]` `awk '!(T+= $NF) END { print T/NR }' $set[i]` $vb $set[i]:t:r >&1 >> $summary
      done
    fi

    [[ ${#vbs[@]} > 0 ]] &&
      vb=`printf ", v-blanks %.3fms" $(awk '!(T+= $NF) END { print T/NR }' $vbs)` || vb=''
    printf "%${#l}i frames %.3fms%s complete \n\n" \
      `wc -l < $set` `awk '!(T+= $NF) END { print T/NR }' $set` $vb >&1 >> $summary

  else
    printf "%s\n\n" "Nothing sampled for $backend." >&2 >> $summary
  fi

  rm -f $outputdir/.$backend.dtm*
done

printf "Results: %q\n" $outputdir
embeded settings	settings location
`MAIN_CPU_THREAD`	`Config` / `General` / `Dual Core`
`MAIN_DSP_HLE`	`Config` / `Audio` / `DSP Emulation Engine`
`MAIN_CPU_CORE`	`Config` / `Advanced` / `CPU Emulation Engine`
`MAIN_SYNC_GPU`	`Properties Menu` / `Game Config` / `Syncronize GPU Thread`
`MAIN_FAST_DISC_SPEED`	`Properties Menu` / `Game Config` / `Speed up Disc Transfer Rate`
`SYSCONF_LANGUAGE` or `MAIN_GC_LANGUAGE`	`Config` / `Wii` or `GameCube` / `System Language`
`SYSCONF_PAL60`	`Config` / `Wii` / `Use PAL60 Mode`
`SYSCONF_PROGRESSIVE_SCAN`	`Graphics` / `Advanced` / `Enable Progressive Scan`
`GFX_HACK_EFB_ACCESS_ENABLE`	`Graphics` / `Hacks` / `Skip EFB Access from CPU`
`GFX_HACK_SKIP_EFB_COPY_TO_RAM`	`Graphics` / `Hacks` / `Store EFB Copies to Texture Only`
`GFX_HACK_EFB_EMULATE_FORMAT_CHANGES`	`Graphics` / `Hacks` / `Ignore Format Changes`
`GFX_HACK_SKIP_XFB_COPY_TO_RAM`	`Graphics` / `Hacks` / `Store XFB Copies to Texture Only`
`GFX_HACK_IMMEDIATE_XFB`	`Graphics` / `Hacks` / `Immediatly Present XFB`
`SESSION_USE_FMA`
`MAIN_JIT_FOLLOW_BRANCH`
	#!/usr/bin/env zsh

	set -e -o pipefail

	if (( $argv[(I)(--help\|-h)] )); then
	printf "%s\n" "
	--dtm\|-p replay.dtm
	Path to dtm replay file. This is required.

	--metal\|--vulkan\|--opengl\|--null
	The graphics backend to test. If none are set, only Metal will be tested.

	--repeat\|-r 1
	How many runs for each graphics backend. Defaults to 1.

	--dolphin\|-o /Application/Dolphin.app
	Path to Dolphin.app. Defaults to Application folder.

	--game\|-g game.gcz
	The path will be detected automatically for rvz, iso and wia files.
	For other rom types, the path must be set manually.

	--hud
	Performance HUD for Metal and Vulkan graphics backend's. Ventura and later.

	--fps
	Show fps/vps/% and frame counter. Incompatibel with --hud.

	--config=\|-C System.Section.Key=Value
	Options accepted by Dolphin will be passed to it.
	https://wiki.dolphin-emu.org/index.php?title=GameINI
	"
	exit
	fi

	dtmpath=${argv[( $argv[(i)--dtm\|-p] + 1 )]##-*}
	if [[ -f $dtmpath ]]; then
	gameid=`xxd -seek 4 -len 6 $dtmpath \| xxd -rp -seek -4`
	else
	printf "%s\n" "[--dtm replay.dtm] not set or the file doesn't exist." >&2
	exit 1
	fi

	backends=( metal vulkan opengl null )
	for backend in $backends; if (( $argv[(I)--$backend] )); then
	_backends+=( $backend )
	fi
	backends=( ${_backends:-metal} )

	reploop=${${argv[( $argv[(i)--repeat\|-r] + 1 )]##-*}:-1}

	dolphin=${${argv[( $argv[(i)--dolphin\|-o] + 1 )]##-*}:-/Applications/Dolphin.app}
	if [[ ! -d $dolphin ]]; then
	printf "%s\n" "$dolphin not found. Set --dolphin with a path to the application." >&2
	exit 1
	fi

	# Follow --user= path.
	usr_default=~/Library/Application\ Support/Dolphin
	usr_dir=${${${argv[$argv[(i)--user=]]##--user=}:-${argv[( $argv[(i)-u] + 1 )]##-}}:-$usr_default}
	if [[ ! -d $usr_dir ]]; then
	printf "%s\n" "$usr_dir not found. Set --user path to an existing support folder." >&2
	exit 1
	fi

	gamepath=${argv[( $argv[(i)--game\|-g] + 1 )]##-*}
	if [[ -z $gamepath ]]; then
	isopaths=( `sed -n "s\|ISOPath[0-9][^/]\(/.\)\|\1\|p" $usr_dir/Config/Dolphin.ini` )
	while read -r g; do if [[ `file -b $g` =~ $gameid,.*\)$ ]]; then
	gamepath=$g; break
	fi; done < <( find -E $isopaths(N) -regex ".+\.(rvz\|iso\|wia)" )

	if [[ -z $gamepath ]]; then
	printf "%s\n" "Game not found for $gameid. Set it with --game path2/game.xyz" >&2; exit 1
	fi
	elif [[ ! -f $gamepath ]]; then
	printf "%s\n" "File not found: --game $gamepath" >&2; exit 1
	fi

	if (( $argv[(I)--hud] )); then
	export MTL_HUD_ENABLED=1
	elif (( $argv[(I)--fps] )); then
	argv+=--config=Graphics.Settings.ShowFPS=True
	argv+=--config=Graphics.Settings.ShowVPS=True
	argv+=--config=Graphics.Settings.ShowSpeed=True
	argv+=--config=Dolphin.General.ShowFrameCount=True
	fi

	cl=(
	--dtm -p $dtmpath
	--metal --vulkan --opengl --null
	--repeat -r $reloop
	--dolphin -o $dolphin
	--game -g $game
	--hud
	)

	outputdir="$dtmpath:r `date "+%m.%d,%H.%M.%S"`"
	mkdir -p $outputdir

	summary=$outputdir/_summary.txt
	printf "%s\n\n" "Summary for $gamepath:t [$gameid] - $dtmpath:t." > $summary

	# Patch the .dtm file since the backend is written to it.
	# This will only be a problem if a game depends on EFB CPU access such
	# as Metroid Prime 2 or Super Mario Galaxy. Does not affect most games.
	# Extracted with: `xxd -s 81 -l 6 replay.dtm`
	typeset -A hexsplice=(
	metal "00000051: 4d65 7461 6c00 Metal."
	vulkan "00000051: 5675 6c6b 616e Vulkan"
	opengl "00000051: 4f47 4c00 0000 OGL..."
	null "00000051: 4e75 6c6c 0000 Null.."
	)

	# Clean-up after manual interrupt.
	trap 'rm -f $outputdir/.$backend.dtm*; exit' INT

	for backend in $backends; do

	head -c 81 $dtmpath > $outputdir/.$backend.dtm
	echo -n $hexsplice[$backend] \| xxd -rp >> $outputdir/.$backend.dtm
	tail -c +88 $dtmpath >> $outputdir/.$backend.dtm
	if [[ -f $dtmpath.sav ]]; then
	ln -sf ../$dtmpath:t.sav $outputdir/.$backend.dtm.sav
	fi

	for i in {1..$reploop}; do
	find $usr_dir/Logs -name 'time.txt' -delete

	# Launch asyncronously.
	$dolphin/Contents/MacOS/Dolphin ${@:\|cl} \
	--batch \
	--exec=$gamepath \
	--movie=$outputdir/.$backend.dtm \
	--config=Graphics.Settings.WaitForShadersBeforeStarting=True \
	--config=Graphics.Settings.LogRenderTimeToFile=True \
	--config=Dolphin.Core.EmulationSpeed=0 \
	--config=Dolphin.DSP.Backend=No\ Audio\ Output \
	--config=Dolphin.Movie.PauseMovie=True \
	--config=Dolphin.Interface.ConfirmStop=False \
	--config=Dolphin.Interface.OnScreenDisplayMessages=True \
	&>> $outputdir/console.txt & pid=$!

	in=${(l:${#reploop}::0:)i}

	printf "$in/$reploop $backend" >&1 >> $summary
	printf "\n%s %s\n\n" "$_" "$*" >> $outputdir/console.txt

	# Ensure Dolphin is running and the render log is created before continuing.
	while [[ ! -f $renderlog && `ps -p $pid` =~ $pid ]]; do
	sleep .5; renderlog=`find $usr_dir/Logs -name 'render_time*'`
	done

	fcount=0 l=, c=.
	if [[ -f $renderlog ]]; then
	while test $l != $c && sleep 2; do
	printf . >&1 >> $summary
	l=$c c=`date -r $renderlog +%s`
	done

	if [[ -s $renderlog ]]; then
	result=$outputdir/$backend\ pass\ $in.txt
	tail -n +20 $renderlog > $result
	printf "%.3fms" `awk '!(T+= $NF) END { print T/NR }' $result` >&1 >> $summary
	fcount=`printf "%i" $(wc -l < $result)`

	vblanklog=`find $usr_dir/Logs -name 'vblank_times'`
	if [[ -n $vblanklog ]]; then
	tail -n +20 $vblanklog > $outputdir/$backend\ v-blanks\ $in.txt
	fi
	fi
	fi

	if [[ `ps -p $pid` =~ $pid ]]; then
	kill $pid
	printf "\n%s\n" "Run completed. Sampled $fcount frames." >&1 >> $summary
	# Wii games need a moment to shut down. Wait until it completely quits.
	while [[ `ps -p $pid` =~ $pid ]] sleep .5
	else
	printf "\n%s\n" "Warning! Early termination! Sampled $fcount frames." >&2 >> $summary
	fi
	done

	set=( $outputdir/$backend\ pass\ *.txt(N) )
	vbs=( $outputdir/$backend\ v-blanks\ *.txt(N) )

	if [[ ${#set[@]} > 0 ]]; then
	l=`printf "%i" $(wc -l < $set)`

	if [[ ${#set[@]} > 1 ]]; then
	for ((i = 1; i <= $#set; i++)); do
	[[ ${#vbs[@]} > 0 ]] &&
	vb=`printf ", v-blanks %.3fms - " $(awk '!(T+= $NF) END { print T/NR }' $vbs[i])` \|\| vb=' - '
	printf "%${#l}i frames %.3fms%s%s\n" \
	`wc -l < $set[i]` `awk '!(T+= $NF) END { print T/NR }' $set[i]` $vb $set[i]:t:r >&1 >> $summary
	done
	fi

	[[ ${#vbs[@]} > 0 ]] &&
	vb=`printf ", v-blanks %.3fms" $(awk '!(T+= $NF) END { print T/NR }' $vbs)` \|\| vb=''
	printf "%${#l}i frames %.3fms%s complete \n\n" \
	`wc -l < $set` `awk '!(T+= $NF) END { print T/NR }' $set` $vb >&1 >> $summary

	else
	printf "%s\n\n" "Nothing sampled for $backend." >&2 >> $summary
	fi

	rm -f $outputdir/.$backend.dtm*
	done

	printf "Results: %q\n" $outputdir