Skip to content

Instantly share code, notes, and snippets.

@hnykda
Created January 3, 2018 19:15
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save hnykda/921eb78dfd9a33bf9c4565e333fdafad to your computer and use it in GitHub Desktop.
Save hnykda/921eb78dfd9a33bf9c4565e333fdafad to your computer and use it in GitHub Desktop.
Different compression between versions in hdf (blosc:snappy)
dan at think460s in ~/load/FinalWave422122017/tst
$ ll
total 68M
-rw-r--r-- 1 dan dan 68M Jan 3 19:54 a.hdf
(pygwi)
dan at think460s in ~/load/FinalWave422122017/tst
$ python -c "import pandas; df = pandas.read_hdf('a.hdf'); df.to_hdf('b.hdf', 'df', complib='blosc:snappy'); pandas.show_versions()"
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.14.10-1-ARCH
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
pandas: 0.22.0
pytest: None
pip: 9.0.1
setuptools: 36.5.0.post20170921
Cython: None
numpy: 1.13.3
scipy: None
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
(pygwi)
dan at think460s in ~/load/FinalWave422122017/tst
$ conda install pandas==0.20.3
Fetching package metadata .............
Solving package specifications: .
Package plan for installation in environment /home/dan/.local/opt/miniconda/envs/pygwi:
The following packages will be DOWNGRADED:
pandas: 0.22.0-py36hf484d3e_0 --> 0.20.3-py36h6022372_2
Proceed ([y]/n)? y
(pygwi)
dan at think460s in ~/load/FinalWave422122017/tst
$ python -c "import pandas; df = pandas.read_hdf('a.hdf'); df.to_hdf('c.hdf', 'df', complib='blosc:snappy'); pandas.show_versions()"
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.14.10-1-ARCH
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
pandas: 0.20.3
pytest: None
pip: 9.0.1
setuptools: 36.5.0.post20170921
Cython: None
numpy: 1.13.3
scipy: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
pandas_gbq: None
pandas_datareader: None
(pygwi)
dan at think460s in ~/load/FinalWave422122017/tst
$ ll
total 149M
-rw-r--r-- 1 dan dan 68M Jan 3 19:54 a.hdf
-rw-r--r-- 1 dan dan 68M Jan 3 20:11 b.hdf
-rw-r--r-- 1 dan dan 14M Jan 3 20:11 c.hdf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment