Created
July 9, 2017 00:54
-
-
Save se4u/dcdd95aa2c1c6743473a1dd5dd3fde39 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
-- Usecase for a script runner. | |
-- Author: Pushpendre Rastogi | |
-- Date: 8 July 2017 | |
-- Multiple small changes in code are a norm when developing machine learning software | |
-- in dynamic languages, such as lua, python, because of a large number of hyper-parameters | |
-- in machine learning code, and large number of typos because of lack of compile time | |
-- type checking. For example, lua does not even enforce the number of arguments to a | |
-- function to be the same as its signature. Because of this ML code typically needs to be | |
-- run multiple times during development with small changes/fixes. | |
-- ML code also commonly has costly processing that must be done on a dataset. | |
-- For reasons of performance / simplicity often such preprocessing is done in a | |
-- batch manner instead of an online manner through iterators. | |
-- Often time the datasets are so large that even serializing/deserializing | |
-- the processed datasets can take a large amount of time. | |
-- This combination of large number of restarts, with each restart having a large fixed | |
-- startup cost causes a lot of difficulty for developers. | |
-- | |
-- A script runner is basically eval-loop that keeps time-consuming data structures | |
-- in memory, but it reloads the code/script that must be run each time. Finally, | |
-- it also allows incremental changes to hyper-parameters between the run of a script | |
-- by reading the user's input. | |
-- | |
-- The following is a script runner written in `lua` | |
-- The `torch.CmdLine` library is used for command line processing. | |
-- USAGE: lua script_runner.lua -arg 2 | |
--[[ script.lua | |
function p() print(opt.arg) end p() | |
--]] | |
cmd = torch.CmdLine() | |
cmd:text() | |
cmd:option('-arg', 1,'') | |
cmd:text() | |
cmd:text('Options') | |
opt = cmd:parse(arg) | |
while true do | |
local r, errmsg = pcall(dofile, 'tmp3.lua') | |
if(not r) then print('ERROR: ' .. errmsg) end | |
io.write('Continue? (C-q C-c to exit, edit cmdline by typing here) ') | |
v = io.read("*l") | |
if v ~= "" then | |
local input = {} | |
local i = 0 | |
for e in string.gmatch(v, "([^ ]+) *") do | |
i = i + 1 | |
input[i] = e | |
end | |
opt2 = cmd:parse(input) | |
for a,b in string.gmatch(v, "-+([^ ]+) *([^ ]+) *") do | |
print('Updated GLOBAL_OPT[' .. a .. '] = ' .. opt2[a]) | |
opt[a] = opt2[a] | |
end | |
end | |
end |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
good