Skip to content

Instantly share code, notes, and snippets.

@astanway
Created December 11, 2012 15:59
Show Gist options
  • Save astanway/4259639 to your computer and use it in GitHub Desktop.
Save astanway/4259639 to your computer and use it in GitHub Desktop.
import socket
import sys
import pickle
import redis
"""
Basically, this bad boy needs to receive 500k pickled metrics of the type [name, (timestamp,
value)] every 10 seconds. It needs to then unpickle them and pop them in Redis. When you fire
it up, it works for about five seconds (just listening - not even Redising yet), and then it
stops, with a weird exception in gen_unpickle:
'\x00'
'\x00'
'\x93'
'\x0b'
So I have no idea what's going on.
"""
def gen_unpickle(infile):
while True:
try:
item = pickle.load(infile)
yield item
except Exception as e:
print e
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
s.bind((socket.gethostname(), 2014))
s.listen(5)
r = redis.StrictRedis(host='localhost', port=6379, db=0)
r.flushall()
while True:
(conn, address) = s.accept()
try:
print >>sys.stderr, 'connection from', address
while True:
for item in gen_unpickle(conn.makefile()):
for metric in item:
name = "metric." + metric[0]
print name
#r.rpush(name, metric[1])
except Exception as e:
print e
finally:
conn.close()
@astanway
Copy link
Author

we're passing it a makefile, so it can continually grow while in the gen_unpickle function. these are streams, remember

@marthakelly
Copy link

right I get that, I just haven't seen it used outside of an iterable before. I figured the buffer would take care of that rather than using yield. anyways, way over my head. good luck!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment