Skip to content

Instantly share code, notes, and snippets.

@hpoit
Last active May 9, 2017 18:25
Show Gist options
  • Save hpoit/a64edf6f3b9169b853a0fe7f6cf79239 to your computer and use it in GitHub Desktop.
Save hpoit/a64edf6f3b9169b853a0fe7f6cf79239 to your computer and use it in GitHub Desktop.
Demo of sequence, not tested:
julia> using JuliaDB
julia> # first use `glob` to list all the compressed streams
compressed_streams = glob("*.csv.gz", Pkg.dir("JuliaDB", "test", "tickdata"))
10-element Array{String,1}:
"/home/shashi/.julia/v0.5/JuliaDB/test/tick-2010-03.csv.gz"
"/home/shashi/.julia/v0.5/JuliaDB/test/tick-2009-09.csv.gz"
"/home/shashi/.julia/v0.5/JuliaDB/test/tick-2010-08.csv.gz"
"/home/shashi/.julia/v0.5/JuliaDB/test/tick-2009-11.csv.gz"
"/home/shashi/.julia/v0.5/JuliaDB/test/tick-2011-07.csv.gz"
"/home/shashi/.julia/v0.5/JuliaDB/test/tick-2016-03.csv.gz"
"/home/shashi/.julia/v0.5/JuliaDB/test/tick-2011-07.csv.gz"
"/home/shashi/.julia/v0.5/JuliaDB/test/tick-2016-03.csv.gz"
"/home/shashi/.julia/v0.5/JuliaDB/test/tick-2011-03.csv.gz"
"/home/shashi/.julia/v0.5/JuliaDB/test/tick-2012-08.csv.gz"
Metadata for 10 / 10 compressed streams can be loaded from cache.
julia> # load_compressed_streams can load the compressed streams in parallel
tickdata = load_compressed_streams(compressed_streams, header_exists=false,
colnames=["Type", "timestamp", "price", "volume"],
indexcols=[1, 2])
DTable with 150 rows in 10 chunks:
Type timestamp │ price volume
───────────────────────────────────┼─────────────────
"Trade" 2010-03-03T20:27:10.715 │ 19.48 100
"Quote" 2010-03-03T20:27:10.718 │ NA NA
"Trade" 2010-03-03T20:27:10.966 │ 19.48 300
"Trade" 2010-03-03T20:27:10.975 │ 19.48 200
"Quote" 2010-03-03T20:27:11.217 │ NA NA
...
julia> select(tickdata, 1 .== "Trade")
DTable with 10 chunks:
Type timestamp │ price volume
───────────────────────────────────┼─────────────────
"Trade" 2011-07-15T06:16:20.534 │ 19.56 100
"Trade" 2011-07-15T06:16:20.707 │ 19.49 400
"Trade" 2011-07-15T06:16:20.806 │ 19.32 800
"Trade" 2011-07-15T06:16:20.807 │ 19.43 900
"Trade" 2011-07-15T06:16:21.006 │ 19.32 300
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment