AVRO and Snappy within Python 3

try_snappy_py3.py

"""Checking to see if snappy works with Python 3 recent updates to libraries

Requires:
    python-snappy==0.5
    fastavro==0.9.9

"""
import random
import string

from fastavro import writer


def main():

    words_dict = open('/usr/share/dict/words').read().splitlines()

    schema = {
        'doc': 'Boring Example',
        'name': 'YT_LIKE',
        'namespace': 'test',
        'type': 'record',
        'fields': [
            {'name': 'index', 'type': 'int'},
            {'name': 'yt_video', 'type': 'string'},
            {'name': 'views', 'type': 'long'},
            {'name': 'description', 'type': 'string'},
        ],
    }

    def get_video():
        # Get a random string of valid characters (max 2 each)
        yt_set = (string.ascii_letters + string.digits + "-_") * 2
        return ''.join(random.sample(yt_set, 11))

    def get_desc():
        return ' '.join(random.sample(words_dict, random.randint(10, 100)))

    def gen_data():
        random.seed(100)
        for index in range(0, 10000):
            yield {
                'index': index,
                'yt_video': get_video(),
                'views': random.randint(1000, 1000000000),
                'description': get_desc(),
            }

    with open('try_snappy_snappy.avro', 'wb') as target:
        writer(target, schema, gen_data(), codec='snappy')

    with open('try_snappy_deflate.avro', 'wb') as target:
        writer(target, schema, gen_data(), codec='deflate')

if __name__ == '__main__':
    main()

z_output.md

$ pyenv virtualenv 3.5.1 trying

Ignoring indexes: https://pypi.python.org/simple Requirement already satisfied (use --upgrade to upgrade): setuptools in /Users/kyle.walker/.pyenv/versions/3.5.1/envs/trying/lib/python3.5/site-packages Requirement already satisfied (use --upgrade to upgrade): pip in /Users/kyle.walker/.pyenv/versions/3.5.1/envs/trying/lib/python3.5/site-packages

$ pyenv local trying
$ pip install -U setuptools pip python-snappy==0.5 fastavro==0.9.9
> ```
Collecting setuptools
  Downloading http://nexus.zefr.com/repository/pypi/packages/69/19/b1dff551058ce79d88b1e3688f1c735590d7ddf44d10681512133b35019f/setuptools-32.3.1-py2.py3-none-any.whl (479kB)
    100% |████████████████████████████████| 483kB 582kB/s
Collecting pip
  Downloading http://nexus.zefr.com/repository/pypi/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 10.8MB/s
Collecting python-snappy==0.5
  Downloading http://nexus.zefr.com/repository/pypi/packages/70/fa/621594e313b2b9074585af63668f7a31839c4e4d69df536ab592e6e1e353/python-snappy-0.5.tar.gz
Collecting fastavro==0.9.9
  Downloading http://nexus.zefr.com/repository/pypi/packages/fe/fe/df11c95c997f8cdc9f530d8e5ce741c55c8ec3c0e06651bd320bd8179a79/fastavro-0.9.9.tar.gz (286kB)
    100% |████████████████████████████████| 286kB 767kB/s
Installing collected packages: setuptools, pip, python-snappy, fastavro
  Found existing installation: setuptools 18.2
    Uninstalling setuptools-18.2:
      Successfully uninstalled setuptools-18.2
  Found existing installation: pip 7.1.2
    Uninstalling pip-7.1.2:
      Successfully uninstalled pip-7.1.2
  Running setup.py install for python-snappy
  Running setup.py install for fastavro
Successfully installed fastavro-0.9.9 pip-9.0.1 python-snappy-0.5 setuptools-32.3.1

$ python try_snappy.py

$ ls -al try* -rw-r--r-- 1 kyle.walker 10000 1501 Jan 5 11:58 try_snappy.py -rw-r--r-- 1 kyle.walker 10000 3347082 Jan 5 11:59 try_snappy_deflate.avro -rw-r--r-- 1 kyle.walker 10000 5085334 Jan 5 11:59 try_snappy_snappy.avro