brendano (owner)

Revisions

gist: 28439 Download_button fork
public
Description:
pipe fiddling: (1) kill buffering (2) output redir kills stdout encoding, so force it
Public Clone URL: git://gist.github.com/28439.git
Embed All Files: show embed
Python #
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Pipe-oriented I/O in Python. This is harder than it should be.
# (1) Kill stdout buffering. makes redirects and tee easier to use.
if "<fdopen>" not in str(sys.stdout): sys.stdout = os.fdopen(1,'w',0)
# (2) Encoding madness. Note codecs.open() isn't available to us since we're using pipes.
import codecs
sys.stdout = codecs.EncodedFile(sys.stdout,'utf-8','utf-8','ignore')
# or this too .. sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
 
# I'm interested in safely handling potentially garbled input data, so want to protect stdin.
# You'd think this would work:
sys.stdin = codecs.EncodedFile(sys.stdin,'utf-8','utf-8','ignore')
 
# But it fails on the line: "Smokey Joe's Caf\xe9\n" (which appears to not be valid utf-8)
# It thought the newline wasn't a newline and blended to the next line. wtf.
# So, if line-oriented input is what you want (usually is for me):
for line in codecs.iterdecode(sys.stdin,'utf-8','ignore'):
  print line,