Skip to content

Instantly share code, notes, and snippets.

@thearn
Created April 20, 2013 00:57
Show Gist options
  • Star 3 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save thearn/5424244 to your computer and use it in GitHub Desktop.
Save thearn/5424244 to your computer and use it in GitHub Desktop.
Functions for saving and loading python objects using pickle with gzip compression.
import pickle
import gzip
def save(object, filename, protocol = 0):
"""Saves a compressed object to disk
"""
file = gzip.GzipFile(filename, 'wb')
file.write(pickle.dumps(object, protocol))
file.close()
def load(filename):
"""Loads a compressed object from disk
"""
file = gzip.GzipFile(filename, 'rb')
buffer = ""
while True:
data = file.read()
if data == "":
break
buffer += data
object = pickle.loads(buffer)
file.close()
return object
@Teketel
Copy link

Teketel commented Jun 21, 2018

Failing on Python:3.6

TypeError                                 Traceback (most recent call last)
/test.py in <module>()
     37 filename = 'machine.pickle'
     38 save(machine, filename)
---> 39 m = load(filename)
     40 print(m.y)

/test.py in load(filename)
     30                 if data == "":
     31                         break
---> 32                 buffer += data
     33         object = pickle.loads(buffer)
     34         file.close()
TypeError: must be str, not bytes 

Tested with the following code

filename = 'machine.pickle'
machine = dict(name="testing dict")
save(machine, filename)
m = load(filename) 

@sonictl
Copy link

sonictl commented Dec 18, 2019

def load(filename):
        """Loads a compressed object from disk
        """
        file = gzip.GzipFile(filename, 'rb')
        # buffer = ""
        # while True:
        #         data = file.read()
        #         if data == "":
        #                 break
        #         buffer += data
        # object = pickle.loads(buffer)
        data = file.read()
        object = pickle.loads(data)
        file.close()
        return object

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment