Skip to content

Instantly share code, notes, and snippets.

@uDude
Last active March 17, 2018 19:10
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save uDude/da262b811ba28c8b8204daef5ff5094e to your computer and use it in GitHub Desktop.
Iteratively copy files in Elixir
# We use streams (open the file convert the device to a stream, using 4MB chunks)
# as a demo, using this method we copied a 2.5GB file with no significant memory usage on
# an Odroid with only 2GB of RAM. Not super fast... Do line oriented by replacing the byte
# count in IO.binstream(:line), very very slow. Process line by line or chunk by chunk
# by passing a function to Stream.into(out, fn(line) -> do_someting(line) end).
fin = File.open!("file.in", [:read, :read_ahead]) |> IO.binstream(4194304)
fout = File.open!("file.out", [:write] |> IO.binstream(4194304)
fin |> Stream.into(fout) |> Stream.run
# to close the files
fin.device |> File.close()
fout.device |> File.close()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment