Lua vararg iteration benchmark

0. Abstract

This small benchmark test compares the performance of using `ipairs{...}` vs.
`select('#',<n>)` to iterate over the arguments inside a variadic function.

1. The code.lua

local unpack = table.unpack or
function(tab)
	return table.remove(tab,1), unpack(tab)
end

function try(fnc,n,...)
	name = name or ""
	ts = os.clock()
	for i=1,n do
		fnc(...)
	end
	te = os.clock()
	print(te-ts)
end

for i=1,8 do
	values = {}
	n = 2^i
	print(("Number of elements: %i"):format(n))
	for i=1,n*2 do
		values[#values+1]=i
	end
	n = 2^22 / n
	print(("Number of iterations: %i"):format(n))

	print("---- STARTING TEST ----")

	io.write "numeric for: "
	try(function(...)
		args = {...}
		for i=1,#args do
			a = args[i]
		end
	end, n, unpack(values))

	io.write "ipairs{...}: "
	try(function(...)
		args = {...}
		for key, value in ipairs(args) do
			a = value
		end
	end, n, unpack(values))

	io.write "select     : "
	try(function(...)
		for i=1,select("#",...) do
			a = select(i,...)
		end
	end, n, unpack(values))

	---[[
	io.write "combination: "
	try(function(...)
		n_args = select("#",...)
		if n_args<=16 then
			for i=1,n_args do
				a = select(i,...)
			end
		else
			args = {...}
			for key, value in ipairs(args) do
				a = value
			end
		end
	end, n, unpack(values))
	--]]
	print()
end

2.1 The results

Number of elements: 1
Number of iterations: 4194304
---- STARTING TEST ----
numeric for: 1.314748
ipairs{...}: 1.786948
select     : 0.825248
combination: 1.115971

Number of elements: 2
Number of iterations: 2097152
---- STARTING TEST ----
numeric for: 0.838786
ipairs{...}: 1.09825
select     : 0.578854
combination: 0.722172

Number of elements: 4
Number of iterations: 1048576
---- STARTING TEST ----
numeric for: 0.521183
ipairs{...}: 0.66698
select     : 0.459284
combination: 0.535527

Number of elements: 8
Number of iterations: 524288
---- STARTING TEST ----
numeric for: 0.370473
ipairs{...}: 0.469861
select     : 0.384568
combination: 0.443036

Number of elements: 16
Number of iterations: 262144
---- STARTING TEST ----
numeric for: 0.307234
ipairs{...}: 0.398395
select     : 0.43302
combination: 0.447391

Number of elements: 32
Number of iterations: 131072
---- STARTING TEST ----
numeric for: 0.26259
ipairs{...}: 0.352319
select     : 0.476176
combination: 0.342218

Number of elements: 64
Number of iterations: 65536
---- STARTING TEST ----
numeric for: 0.222863
ipairs{...}: 0.306345
select     : 0.621975
combination: 0.327539

Number of elements: 128
Number of iterations: 32768
---- STARTING TEST ----
numeric for: 0.208248
ipairs{...}: 0.286315
select     : 0.920346
combination: 0.309997

Number of elements: 256
Number of iterations: 16384
---- STARTING TEST ----
numeric for: 0.230745
ipairs{...}: 0.295762
select     : 1.526707
combination: 0.30213

2.2 Aditional tests

Number of elements: 16
Number of iterations: 262144
---- STARTING TEST ----
numeric for: 0.53651
ipairs{...}: 0.658203
select     : 1.006718
combination: 0.738774

Number of elements: 32
Number of iterations: 131072
---- STARTING TEST ----
numeric for: 0.532007
ipairs{...}: 0.649775
select     : 1.298958
combination: 0.691573

Number of elements: 64
Number of iterations: 65536
---- STARTING TEST ----
numeric for: 0.506338
ipairs{...}: 0.607839
select     : 1.91865
combination: 0.656192

Number of elements: 128
Number of iterations: 32768
---- STARTING TEST ----
numeric for: 0.507063
ipairs{...}: 0.60743
select     : 3.321204
combination: 0.64475

Number of elements: 256
Number of iterations: 16384
---- STARTING TEST ----
numeric for: 0.504592
ipairs{...}: 0.625383
select     : 7.188905
combination: 0.643067

Number of elements: 512
Number of iterations: 8192
---- STARTING TEST ----
numeric for: 0.488121
ipairs{...}: 0.596162
select     : 13.666135
combination: 0.609433

Number of elements: 1024
Number of iterations: 4096
---- STARTING TEST ----
numeric for: 0.487915
ipairs{...}: 0.57412900000001
select     : 25.984348
combination: 0.589311

2.3 LuaJIT results

Number of elements: 32
Number of iterations: 131072
---- STARTING TEST ----
numeric for: 0.067259
ipairs{...}: 0.061422
select     : 0.022591
combination: 0.071471

Number of elements: 64
Number of iterations: 65536
---- STARTING TEST ----
numeric for: 0.046058
ipairs{...}: 0.046445
select     : 0.014507
combination: 0.054622

Number of elements: 128
Number of iterations: 32768
---- STARTING TEST ----
numeric for: 0.041115
ipairs{...}: 0.03876
select     : 0.011633
combination: 0.0432

Number of elements: 256
Number of iterations: 16384
---- STARTING TEST ----
numeric for: 0.029038
ipairs{...}: 0.031937
select     : 0.010491
combination: 0.034294

Number of elements: 512
Number of iterations: 8192
---- STARTING TEST ----
numeric for: 0.030625
ipairs{...}: 0.030857
select     : 0.01198
combination: 0.038555

Number of elements: 1024
Number of iterations: 4096
---- STARTING TEST ----
numeric for: 0.0268
ipairs{...}: 0.030364
select     : 0.018461
combination: 0.033491

Number of elements: 2048
Number of iterations: 2048
---- STARTING TEST ----
numeric for: 0.023028
ipairs{...}: 0.028488
select     : 0.013719
combination: 0.029133

Number of elements: 4096
Number of iterations: 1024
---- STARTING TEST ----
numeric for: 0.021597
ipairs{...}: 0.024167
select     : 0.011663
combination: 0.029756

3. Conclusion

# Conclusion

The tests clearly show that with up to around 20 arguments, using `select()`
seems to be faster, while above that it grows increasingly slower.

In practice this means that most variadic functions can perform better if
implemented that way. With functions that expect more arguments than around 16
to 20 though it is more effective to use the traditional method of building a
table and iterating it with `ipairs`.

## Update

After adding a version with a numeric for loop, the results didn't change all
that much. Select is still a bit faster for small numbers of arguments, but the
tipping point has moved to around 8 arguments. The difference between the mixed
implementation and the pire numeric for implementation is also way bigger than
with the ipairs implementation. What this means is that one should probably go
for that when expecting lots of arguments, though the mixed one is still faster
for very few arguments and still safe for large numbers of arguments, when the
pure select loop would start taking a lot longer for each doubling of the
argument count. (see 2.5)

## LuaJIT

The results when using LuaJIT are somewhat surprising, though not completely
unexpected. The first iteration is not all that representative, as is usually
the case with JIT compilation. Once the interpreter knows what's going on though
the optimization kicks in and there's a massive speed gain. What's clear
throughout all the iterations though,  is that select is a lot faster, almost
x2 the speed ot the numeric for loop and more than x2 as fast as ipairs{...}.