Last active
March 17, 2018 19:10
-
-
Save uDude/da262b811ba28c8b8204daef5ff5094e to your computer and use it in GitHub Desktop.
Iteratively copy files in Elixir
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# We use streams (open the file convert the device to a stream, using 4MB chunks) | |
# as a demo, using this method we copied a 2.5GB file with no significant memory usage on | |
# an Odroid with only 2GB of RAM. Not super fast... Do line oriented by replacing the byte | |
# count in IO.binstream(:line), very very slow. Process line by line or chunk by chunk | |
# by passing a function to Stream.into(out, fn(line) -> do_someting(line) end). | |
fin = File.open!("file.in", [:read, :read_ahead]) |> IO.binstream(4194304) | |
fout = File.open!("file.out", [:write] |> IO.binstream(4194304) | |
fin |> Stream.into(fout) |> Stream.run | |
# to close the files | |
fin.device |> File.close() | |
fout.device |> File.close() |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment