Benchmarking Remote CMD and Powershell Invocations with WinRM in Test Kitchen

README.md

Benchmarking Remote CMD and Powershell Invocations with WinRM in Test Kitchen

Notes

• The executor in the code is a CommandExecutor which sets up and reuses a remote shell session across calls and recyles it before the maxiumum commands-per-shell limit is reached.
• The vanilla object is a regular WinRM::WinRMWebService instance which sets up and tears down a shell for every CMD and Powershell script invocation.
• The Powershell script variant is really running a powershell -encodedCommand <ENCODED_SCRIPT> over CMD.
• The file appending operation isn't very taxing on the Windows host but simulates a chunked file upload and so tries to measure the relative cost of command invocations vs. measuring the time for the commands to execute.

Observations

• A 2x speed up is great for CMD commands!
• Any speed gains in invoking Powershell scripts is wiped out by the setup/teardown cost
• A 100x cost penalty for running Powershell scripts is enough to give the author pause

generate.rb

#/usr/bin/env
# **Note** this program require the benchmark-ips gem, installed with:
#     gem install benchmark-ips
#
# To run, from the root of the Test Kitchen codebase:
#     ruby -Ilib benchmark.rb
#
require "benchmark"
require "benchmark/ips"
require "kitchen/transport/winrm"

t = Kitchen::Transport::Winrm.new(kitchen_root: "./tmp").finalize_config!(OpenStruct.new(name: "coolbeans"))
c = t.connection(hostname: "127.0.0.1", username: "vagrant", password: "vagrant")

# yank out the raw CommandExecutor
executor = c.send(:session)
# yank out the WinRMWebService from the executor
vanilla = executor.instance_variable_get(:@service)

# with CMD we can get closer to 8K (max command line size), but this
# will be Base64-encoded for the Powershell script, so we'll go with 2K
output = ("." * 2000).freeze

# start fresh new files
%w[executor-cmd vanilla-cmd executor-ps1 vanilla-ps1].each do |file|
  executor.run_cmd("echo START > %TEMP%\\#{file}.txt")
end

puts "## Profiling CMD and Powershell script commands (uploading a file)\n\n"

Benchmark.ips do |x|
  x.time = 20
  x.warmup = 5
  
  x.report("cmd_executor") do
    executor.run_cmd("echo #{output} >> %TEMP%\\executor-cmd.txt")
  end
  
  x.report("cmd_vanilla") do
    vanilla.run_cmd("echo #{output} >> %TEMP%\\vanilla-cmd.txt") 
  end
  
  x.report("ps1_executor") do
    executor.run_powershell_script(%{Add-Content $env:Temp\\executor-ps1.txt "#{output}"})
  end
  
  x.report("ps1_vanilla") do
    vanilla.run_powershell_script(%{Add-Content $env:Temp\\vanilla-ps1.txt "#{output}"})
  end
  
  x.compare!
end

Report.txt

## Profiling CMD and Powershell script commands (uploading a file)

Calculating -------------------------------------
        cmd_executor     3.000  i/100ms
         cmd_vanilla     1.000  i/100ms
        ps1_executor     1.000  i/100ms
         ps1_vanilla     1.000  i/100ms
-------------------------------------------------
        cmd_executor     38.487  (± 5.2%) i/s -    771.000
         cmd_vanilla     16.925  (± 5.9%) i/s -    338.000
        ps1_executor      0.355  (± 0.0%) i/s -      8.000  in  22.535856s
         ps1_vanilla      0.351  (± 0.0%) i/s -      8.000  in  22.789677s

Comparison:
        cmd_executor:       38.5 i/s
         cmd_vanilla:       16.9 i/s - 2.27x slower
        ps1_executor:        0.4 i/s - 108.40x slower
         ps1_vanilla:        0.4 i/s - 109.61x slower