GMod file.Exists loadtime test setup (500 addons)

README_500Addons.md

Setup

This is a testing setup I used to write up the report: Facepunch/garrysmod-issues#5674

Other setups

• GMod file.Exists loadtime test setup (1 addon with 50k sounds)
• _test_addon_many_luas.zip (5k Lua files)

Benchmark

• GMod file.Exists loadtime benchmark setup

The file zzz_loadtime_benchmark_(outdated).lua is outdated. I left it there to keep old results reproducible. For newer results please go to above link.

Test case: 500 addons (Lua and Non-Lua)

Addon generator (create_dummy_addons500.bat)

To reproduce the issue I use a custom written batch script to generate and regenerate hundreds of addons with each containing a single unique Lua script and a sound file for testing the "non-lua cases".

Put the script create_dummy_addons.bat in the addons folder you want to test in and run it. It will tell you what it does. You may want to adjust script internal configs to test different cases. It should be safe to use, as I user-error-proved it quite a bit, but I still recommend to be careful on where you run it. I recommend to review and analyze the script before running it.

Test addon content

Each test addon generated by create_dummy_addons.bat has content to ensure that the testing environment is as realistic as possible. The game can never see the addons as empty. They contain:

• A very short but valid sound file (content defined in dummySoundFileData)
• A short but valid Lua script that can be shared, clientside only or serverside only (default) depending on the config
• Each file is uniquely named to avoid internal conflicts
• Each Lua file has unique content for distinct operation during tests

Each lua looks similar to this. It varies a bit depending on config. Shared scripts are AddCSLuaFile'ed.

-- Dummyfile _test_addon_dummy_00496/lua/autorun/server/_test_addon_dummy_00496.lua

print("_test_addon_dummy_00496/lua/autorun/server/_test_addon_dummy_00496.lua")

return true

The sound file represent any non lua content such as models, textures or maps.

Performance

This test case does have a VERY significant influence to the reported issue.

create_dummy_addons500.bat

@echo off

rem Put in "addons" folder

set basedir=%~dp0

rem Make sure you are in the addons directory
cd %basedir%

setlocal ENABLEDELAYEDEXPANSION


rem How many addons do you need?
set /A addonCount=500

echo This will create %addonCount% dummy addons.
set /P continueScript=Do you want to continue (Y/[N])?

IF /I "%continueScript%" NEQ "Y" GOTO FINISH


rem Insert the dummy script as serverside only script (1 = On, 0 = Off)
set /A serversideOnlyLua=1

rem Insert the dummy script as clientside only script (1 = On, 0 = Off)
set /A clientsideOnlyLua=0

rem Path of the dummy files to create, unwanted files can be removed by commenting them out
set luaPath=lua\autorun
set luaPathServerOnly=lua\autorun\server
set luaPathClientOnly=lua\autorun\client
set soundPath=sound\_test_addon_dummies


rem Prefix for addon folders
set addonPrefix=_test_addon_dummy_

rem Temporary file name for the sound file
set dummySoundFile=_dummy_sound.wav

rem A copy of sound/synth/sine_1760.wav from Wiremod as hex code
set dummySoundFileData= ^
52 49 46 46 B6 00 00 00 57 41 56 45 66 6D 74 20 ^
10 00 00 00 01 00 01 00 44 AC 00 00 88 58 01 00 ^
02 00 10 00 64 61 74 61 32 00 00 00 8E F5 10 E0 ^
91 CC 49 BC 3F B0 31 A9 92 A7 79 AB AC B4 94 C2 ^
54 D4 CF E8 BD FE BF 14 76 29 94 3B F9 49 BD 53 ^
44 58 46 57 D2 50 50 45 78 35 48 22 F2 0C 63 75 ^
65 20 1C 00 00 00 01 00 00 00 01 00 00 00 00 00 ^
00 00 64 61 74 61 00 00 00 00 00 00 00 00 00 00 ^
00 00 4C 49 53 54 34 00 00 00 61 64 74 6C 6C 74 ^
78 74 14 00 00 00 01 00 00 00 19 00 00 00 72 67 ^
6E 20 00 00 00 00 00 00 00 00 6C 61 62 6C 0C 00 ^
00 00 01 00 00 00 4C 6F 6F 70 20 30 31 00

rem Clean up old addons starting with _test_addon_dummy_ (%addonPrefix%)
echo.
echo Deleteing old dummy addons starting with %addonPrefix%* ...
for /d %%G in (%addonPrefix%*) do (
   rd /S /Q "%%~G" 2> nul
   echo Deleted addon %%~G
)

rem Config validation
set /A addLuaFile=1
set /A addSoundFile=1
set /A addCSLuaFile=1

if %serversideOnlyLua% EQU 1 (
   set luaPath=%luaPathServerOnly%
)

if %clientsideOnlyLua% EQU 1 (
   set luaPath=%luaPathClientOnly%
)

if "%luaPath%"=="" (
   set addLuaFile=0
)

if %addonCount% LEQ 0 (
   set addLuaFile=0
)

if "%soundPath%"=="" (
   set addSoundFile=0
)

if "%dummySoundFile%"=="" (
   set addSoundFile=0
)

if "%dummySoundFileData%"=="" (
   set addSoundFile=0
)

if %addonCount% LEQ 0 (
   set addSoundFile=0
)

if %serversideOnlyLua% EQU 1 (
   set addCSLuaFile=0
)

if %clientsideOnlyLua% EQU 1 (
   set addCSLuaFile=0
)

rem create dummy sound dummy file from hex
if %addSoundFile% EQU 1 (

   echo.
   echo Creating dummy sound file: !dummySoundFile!

   del /F /Q %dummySoundFile% 2> nul

   echo %dummySoundFileData%> %dummySoundFile%.tmp
   certutil -f -decodehex %dummySoundFile%.tmp %dummySoundFile% >nul
   del /F /Q %dummySoundFile%.tmp 2> nul
)

if %addLuaFile% NEQ 1 (
   if %addSoundFile% NEQ 1 (
      goto FINISH
   )
)

echo.
set /A addonsCreated=0

for /l %%x in (1, 1, %addonCount%) do (
   set fx=000000%%x
   set fx=!fx:~-5!

   set addonName=%addonPrefix%!fx!
   mkdir !addonName! 2> nul

   set /A addonsCreated=addonsCreated+1

   rem create the lua dummy file

   if %addLuaFile% EQU 1 (
      set folderAddonLua=!addonName!\%luaPath%

      mkdir !folderAddonLua! 2> nul

      set filename=!addonName!.lua
      set file=!folderAddonLua!\!filename!
      set fileOutput=!file:\=/!

      echo Creating file: !file! ^(%%x / %addonCount%^)

      echo -- Dummyfile !fileOutput!>> !file!
      echo.>> !file!

      if %addCSLuaFile% EQU 1 (
         echo if SERVER then AddCSLuaFile^(^) end>> !file!
         echo.>> !file!
      )

      echo print^("!fileOutput!"^)>> !file!
      echo.>> !file!

      echo return true>> !file!

      echo.>> !file!
      echo.>> !file!
   )

   rem copy the sound dummy file

   if %addSoundFile% EQU 1 (
      set folderAddonSound=!addonName!\%soundPath%

      mkdir !folderAddonSound! 2> nul

      set filename=!addonName!.wav
      set file=!folderAddonSound!\!filename!

      echo Creating file: !file! ^(%%x / %addonCount%^)

      copy /B /Y %dummySoundFile% !file! /B 1> nul
   )
)

if %addSoundFile% EQU 1 (
   del /F /Q %dummySoundFile% 2> nul

   echo.
   echo Deleted temp dummy sound file %dummySoundFile%
)

:FINISH

set /A addonsCreated=addonsCreated

cd %basedir%

echo.
echo %addonsCreated% addons created.

endlocal

echo Press any key to close.
pause > nul

@echo on
@exit /B 0

zzz_loadtime_benchmark_(outdated).lua

-- Put in "lua\autorun" folder

-- this is outdated, please look at this instead: 
-- https://gist.github.com/Grocel/f8eff1ba32f2ef0e46f91e0ae1a79c4f

AddCSLuaFile()

local function CustomFileExists( path, dir )
    -- Note: It will NOT find AddCSLuaFile() files on the client if file is not physically present as an actual file on said client.
    -- This is the case for server exclusive content or not installed addons that are used on the server.

    local f = file.Open( path, "r", dir )

    if f then
        f:Close()
 
        return true
    end

    return false
end

local function Benchmark()
    print("\n----------------------------\n")

    print( "REALM:", SERVER and "SERVER" or "CLIENT")

    print( "file.Exists LUA" )
    print( file.Exists( "entities/sent_ball.lua", "LUA" ) )
    local start = SysTime()
    for i = 1, 100 do
        file.Exists( "entities/sent_ball.lua", "LUA" )
    end
    print( SysTime() - start )

    print( "file.Read GAME" )
    print( file.Exists( "lua/entities/sent_ball.lua", "GAME" ) )
    local start = SysTime()
    for i = 1, 100 do
        file.Exists( "lua/entities/sent_ball.lua", "GAME" )
    end
    print( SysTime() - start )

    print( "file.Read LUA" )
    print( tobool( file.Read( "entities/sent_ball.lua", "LUA" ) ) )
    local start = SysTime()
    for i = 1, 100 do
        tobool( file.Read( "entities/sent_ball.lua", "LUA" ) )
    end
    print( SysTime() - start )

    print( "file.Read GAME" )
    print( tobool( file.Read( "lua/entities/sent_ball.lua", "GAME" ) ) )
    local start = SysTime()
    for i = 1, 100 do
        tobool( file.Read( "lua/entities/sent_ball.lua", "GAME" ) )
    end
    print( SysTime() - start )

    print( "Custom file exist LUA" )
    print( CustomFileExists( "entities/sent_ball.lua", "LUA" ) )
    local start = SysTime()
    for i = 1, 100 do
        CustomFileExists( "entities/sent_ball.lua", "LUA" )
    end
    print( SysTime() - start )

    print( "Custom file exist GAME" )
    print( CustomFileExists( "lua/entities/sent_ball.lua", "GAME" ) )
    local start = SysTime()
    for i = 1, 100 do
        CustomFileExists( "lua/entities/sent_ball.lua", "GAME" )
    end
    print( SysTime() - start )
    
    print("\n----------------------------\n")

    print("PrintTable( debug.getinfo( file.Exists ) ):\n")
    PrintTable( debug.getinfo( file.Exists ) )

    print("\n")

    print("PrintTable( debug.getinfo( file.Read ) ):\n")
    PrintTable( debug.getinfo( file.Read ) )

    print("\n----------------------------")
end

timer.Simple(1, Benchmark)