Skip to content

Instantly share code, notes, and snippets.

@astanway
Created December 11, 2012 15:59
Show Gist options
  • Save astanway/4259639 to your computer and use it in GitHub Desktop.
Save astanway/4259639 to your computer and use it in GitHub Desktop.
import socket
import sys
import pickle
import redis
"""
Basically, this bad boy needs to receive 500k pickled metrics of the type [name, (timestamp,
value)] every 10 seconds. It needs to then unpickle them and pop them in Redis. When you fire
it up, it works for about five seconds (just listening - not even Redising yet), and then it
stops, with a weird exception in gen_unpickle:
'\x00'
'\x00'
'\x93'
'\x0b'
So I have no idea what's going on.
"""
def gen_unpickle(infile):
while True:
try:
item = pickle.load(infile)
yield item
except Exception as e:
print e
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
s.bind((socket.gethostname(), 2014))
s.listen(5)
r = redis.StrictRedis(host='localhost', port=6379, db=0)
r.flushall()
while True:
(conn, address) = s.accept()
try:
print >>sys.stderr, 'connection from', address
while True:
for item in gen_unpickle(conn.makefile()):
for metric in item:
name = "metric." + metric[0]
print name
#r.rpush(name, metric[1])
except Exception as e:
print e
finally:
conn.close()
@marthakelly
Copy link

I'm not certain, but I think

 for item in gen_unpickle(conn.makefile()): # (ln 38)

makefile needs to be on the socket itself, not the connection?

 for item in gen_unpickle(s.makefile()): # (ln 38)

@marthakelly
Copy link

ha! totally ln 41 not 38, need moar coffee

@astanway
Copy link
Author

yeah the unpickling works for a hot second, and it prints out all the metrics. but then it fucks up

@marthakelly
Copy link

I'm suspicious of "yield" here (ln 22), does return work? or why are you using yield (so I can understand)

@astanway
Copy link
Author

we're passing it a makefile, so it can continually grow while in the gen_unpickle function. these are streams, remember

@marthakelly
Copy link

right I get that, I just haven't seen it used outside of an iterable before. I figured the buffer would take care of that rather than using yield. anyways, way over my head. good luck!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment