Skip to content

Instantly share code, notes, and snippets.

@hpoit

hpoit/pv Secret

Created May 4, 2017 00:53
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hpoit/fea6568a6fb2e21d7f364828d928c6f6 to your computer and use it in GitHub Desktop.
Save hpoit/fea6568a6fb2e21d7f364828d928c6f6 to your computer and use it in GitHub Desktop.
Download ZIP
PV
Raw
pv
Last login: Wed May 3 21:44:12 on ttys001
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.5.0 (2016-09-19 18:14 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org/ release
|__/ | x86_64-apple-darwin13.4.0
julia> using DataFrames
julia> using Query
WARNING: Method definition require(Symbol) in module Base at loading.jl:345 overwritten in module Query at /Users/Corvus/.julia/v0.5/Requires/src/require.jl:12.
WARNING: Method definition ==(Base.Nullable{#T<:Any}, Base.Nullable{Union{}}) in module NullableArrays at /Users/Corvus/.julia/v0.5/NullableArrays/src/operators.jl:138 overwritten in module Query at /Users/Corvus/.julia/v0.5/Query/src/operators.jl:8.
WARNING: Method definition ==(Base.Nullable{Union{}}, Base.Nullable{#T<:Any}) in module NullableArrays at /Users/Corvus/.julia/v0.5/NullableArrays/src/operators.jl:137 overwritten in module Query at /Users/Corvus/.julia/v0.5/Query/src/operators.jl:9.
WARNING: Method definition ==(Base.Nullable{#T1<:Any}, Base.Nullable{#T2<:Any}) in module NullableArrays at /Users/Corvus/.julia/v0.5/NullableArrays/src/operators.jl:128 overwritten in module Query at /Users/Corvus/.julia/v0.5/Query/src/operators.jl:77.
julia> using CSV
julia> q = @from i in CSV.Source("/file.csv") begin
@where i.Type == "Trade"
@select {i.Price, i.Volume}
end
Query.EnumerableSelect{NamedTuples._NT_PriceVolume{Nullable{Float64},Nullable{Int64}},Query.EnumerableWhere{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}},Query.EnumerableDataStream{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}},CSV.Source,Tuple{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}}},FunctionWrappers.FunctionWrapper{Bool,Tuple{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}}}}},FunctionWrappers.FunctionWrapper{NamedTuples._NT_PriceVolume{Nullable{Float64},Nullable{Int64}},Tuple{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}}}}}(Query.EnumerableWhere{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}},Query.EnumerableDataStream{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}},CSV.Source,Tuple{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}}},FunctionWrappers.FunctionWrapper{Bool,Tuple{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}}}}}(Query.EnumerableDataStream{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}},CSV.Source,Tuple{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}}}(CSV.Source: /Users/Corvus/Desktop/reuters1A.csv
CSV.Options:
delim: ','
quotechar: '"'
escapechar: '\\'
null: ""
dateformat: Base.Dates.DateFormat(Base.Dates.Slot[Base.Dates.DelimitedSlot{Base.Dates.Year}(Base.Dates.Year,'y',4,"-"),Base.Dates.DelimitedSlot{Base.Dates.Month}(Base.Dates.Month,'m',2,"-"),Base.Dates.DelimitedSlot{Base.Dates.Day}(Base.Dates.Day,'d',2,r"(?=\s|$)")],"","english")
Data.Schema{true}:
rows: 80000534 cols: 12
Columns:
"#RIC" Nullable{WeakRefString{UInt8}}
"Date[G]" Nullable{WeakRefString{UInt8}}
"Time[G]" Nullable{WeakRefString{UInt8}}
"GMT Offset" Nullable{Int64}
"Type" Nullable{WeakRefString{UInt8}}
"Price" Nullable{Float64}
"Volume" Nullable{Int64}
"Bid Price" Nullable{Float64}
"Bid Size" Nullable{Int64}
"Ask Price" Nullable{Float64}
"Ask Size" Nullable{Int64}
"Qualifiers" Nullable{WeakRefString{UInt8}},Data.Schema{true}:
rows: 80000534 cols: 12
Columns:
"#RIC" Nullable{WeakRefString{UInt8}}
"Date[G]" Nullable{WeakRefString{UInt8}}
"Time[G]" Nullable{WeakRefString{UInt8}}
"GMT Offset" Nullable{Int64}
"Type" Nullable{WeakRefString{UInt8}}
"Price" Nullable{Float64}
"Volume" Nullable{Int64}
"Bid Price" Nullable{Float64}
"Bid Size" Nullable{Int64}
"Ask Price" Nullable{Float64}
"Ask Size" Nullable{Int64}
"Qualifiers" Nullable{WeakRefString{UInt8}}),FunctionWrappers.FunctionWrapper{Bool,Tuple{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}}}}(Ptr{Void} @0x000000031ce66180,Ptr{Void} @0x000000011243c080,Base.RefValue{##1#3}(#1),##1#3)),FunctionWrappers.FunctionWrapper{NamedTuples._NT_PriceVolume{Nullable{Float64},Nullable{Int64}},Tuple{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}}}}(Ptr{Void} @0x000000031ce67390,Ptr{Void} @0x000000011243c090,Base.RefValue{##2#4}(#2),##2#4))
julia> df = DataFrame(take(q, 3_000_000))
ERROR: MethodError: Cannot `convert` an object of type Base.Take{Query.EnumerableSelect{NamedTuples._NT_PriceVolume{Nullable{Float64},Nullable{Int64}},Query.EnumerableWhere{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}},Query.EnumerableDataStream{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}},CSV.Source,Tuple{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}}},FunctionWrappers.FunctionWrapper{Bool,Tuple{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}}}}},FunctionWrappers.FunctionWrapper{NamedTuples._NT_PriceVolume{Nullable{Float64},Nullable{Int64}},Tuple{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}}}}}} to an object of type DataFrames.DataFrame
This may have arisen from a call to the constructor DataFrames.DataFrame(...),
since type constructors fall back to convert methods.
in DataFrames.DataFrame(::Base.Take{Query.EnumerableSelect{NamedTuples._NT_PriceVolume{Nullable{Float64},Nullable{Int64}},Query.EnumerableWhere{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}},Query.EnumerableDataStream{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}},CSV.Source,Tuple{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}}},FunctionWrappers.FunctionWrapper{Bool,Tuple{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}}}}},FunctionWrappers.FunctionWrapper{NamedTuples._NT_PriceVolume{Nullable{Float64},Nullable{Int64}},Tuple{NamedTuples._NT_#RICDate[G]Time[G]GMT OffsetTypePriceVolumeBid PriceBid SizeAsk PriceAsk SizeQualifiers{Nullable{String},Nullable{String},Nullable{String},Nullable{Int64},Nullable{String},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{Float64},Nullable{Int64},Nullable{String}}}}}}) at ./sysimg.jl:53
julia>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment