Skip to content

Instantly share code, notes, and snippets.

@ssanderson
Created June 19, 2014 17:43
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ssanderson/598f14937be2f044fd06 to your computer and use it in GitHub Desktop.
Save ssanderson/598f14937be2f044fd06 to your computer and use it in GitHub Desktop.
### Loading CSV for EIDO Data since 2014
## Branch
2428414 function calls (2416047 primitive calls) in 9.570 seconds
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 8.094 8.094 qexec/qexec/sources/requests_csv.py:384(__init__)
1 0.254 0.254 8.094 8.094 qexec/qexec/sources/requests_csv.py:269(load_df)
## Master
9529341 function calls (9517154 primitive calls) in 30.053 seconds
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 28.148 28.148 qexec/qexec/sources/requests_csv.py:350(__init__)
1 1.049 1.049 28.148 28.148 qexec/qexec/sources/requests_csv.py:246(load_df)
### Loading Full EIDO Data (2006 start - 2013 end)
## Branch
31907547 function calls (31895180 primitive calls) in 140.782 seconds
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 139.116 139.116 qexec/qexec/sources/requests_csv.py:384(__init__)
1 4.853 4.853 139.115 139.115 qexec/qexec/sources/requests_csv.py:269(load_df)
## Master (NOTE: Needed to raise fetcher timeout from 360 seconds to complete at all).
160150257 function calls (160138070 primitive calls) in 493.022 seconds
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 491.342 491.342 qexec/qexec/sources/requests_csv.py:350(__init__)
1 16.794 16.794 491.342 491.342 qexec/qexec/sources/requests_csv.py:246(load_df)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment