with JuliaDB and https://github.com/JuliaLang/julia/tree/master/base

Loading compressed streams and selecting subset

Demo of sequence, not tested:

julia> using JuliaDB

julia> # first use `glob` to list all the compressed streams
       compressed_streams = glob("*.csv.gz", Pkg.dir("JuliaDB", "test", "tickdata"))
10-element Array{String,1}:
 "/home/shashi/.julia/v0.5/JuliaDB/test/tick-2010-03.csv.gz"
 "/home/shashi/.julia/v0.5/JuliaDB/test/tick-2009-09.csv.gz"
 "/home/shashi/.julia/v0.5/JuliaDB/test/tick-2010-08.csv.gz"
 "/home/shashi/.julia/v0.5/JuliaDB/test/tick-2009-11.csv.gz"
 "/home/shashi/.julia/v0.5/JuliaDB/test/tick-2011-07.csv.gz"
 "/home/shashi/.julia/v0.5/JuliaDB/test/tick-2016-03.csv.gz"
 "/home/shashi/.julia/v0.5/JuliaDB/test/tick-2011-07.csv.gz"
 "/home/shashi/.julia/v0.5/JuliaDB/test/tick-2016-03.csv.gz"
 "/home/shashi/.julia/v0.5/JuliaDB/test/tick-2011-03.csv.gz"
 "/home/shashi/.julia/v0.5/JuliaDB/test/tick-2012-08.csv.gz"

Metadata for 10 / 10 compressed streams can be loaded from cache.
julia> # load_compressed_streams can load the compressed streams in parallel
       tickdata = load_compressed_streams(compressed_streams, header_exists=false,
                          colnames=["Type", "timestamp", "price", "volume"],
                          indexcols=[1, 2])
DTable with 150 rows in 10 chunks:

Type       timestamp               │ price      volume
───────────────────────────────────┼─────────────────
"Trade"  2010-03-03T20:27:10.715 │ 19.48  100
"Quote"  2010-03-03T20:27:10.718 │ NA  NA
"Trade"  2010-03-03T20:27:10.966 │ 19.48  300
"Trade"  2010-03-03T20:27:10.975 │ 19.48  200
"Quote"  2010-03-03T20:27:11.217 │ NA  NA
...

julia> select(tickdata, 1 .== "Trade")
DTable with 10 chunks:

Type       timestamp               │ price      volume
───────────────────────────────────┼─────────────────
"Trade"  2011-07-15T06:16:20.534 │ 19.56  100
"Trade"  2011-07-15T06:16:20.707 │ 19.49  400
"Trade"  2011-07-15T06:16:20.806 │ 19.32  800
"Trade"  2011-07-15T06:16:20.807 │ 19.43  900
"Trade"  2011-07-15T06:16:21.006 │ 19.32  300
...