Skip to content

Instantly share code, notes, and snippets.

@thewoolleyman
Last active January 16, 2017 06:42
Show Gist options
  • Save thewoolleyman/09032fb1b5f6e8e5d798bf91ba4daca5 to your computer and use it in GitHub Desktop.
Save thewoolleyman/09032fb1b5f6e8e5d798bf91ba4daca5 to your computer and use it in GitHub Desktop.
ruby PTY turns LF into CF+LF
irb
2.3.3 :001 > puts `uname -a`
Darwin pima 15.6.0 Darwin Kernel Version 15.6.0: Wed Nov 2 20:30:56 PDT 2016; root:xnu-3248.60.11.1.2~2/RELEASE_X86_64 x86_64
=> nil
2.3.3 :002 > `printf "a\n"`.chars
=> ["a", "\n"]
2.3.3 :003 > require 'pty'; PTY.spawn('printf "a\n"')[0].read(3).chars
=> ["a", "\r", "\n"]
[10:45pm] ChanServ: [#bash] Welcome to #bash, we discuss the GNU bash shell. Be explicit if your question is about another shell. Don't trust the Internet on bash, learn from the resources in the topic instead.
[10:49pm] thewoolleyman: When I write to a pseudo-terminal on POSIX (OSX), I'm seeing LF (\n) be turned into CR+LF (\r\n). Is this expected behavior for all pseudo-terminals, or something wrong with my implementation? I'm trying it in Ruby via the built-in PTY support: https://gist.github.com/thewoolleyman/09032fb1b5f6e8e5d798bf91ba4daca5 Thanks!
[10:49pm] gautamsomani left the chat room. (Quit: Leaving.)
[10:51pm] gentunian left the chat room. (Ping timeout: 248 seconds)
[10:52pm] izabera: thewoolleyman: can you control some options for that pty?
[10:53pm] nnpnv6: thewoolleyman: not a bash question, but File.open in ruby respects the letter b second argument for "binary", so that LF to CF-LF conversion shall not happen
[10:54pm] izabera: i don't know much ruby but that doesn't look like File.open
[10:54pm] tdy joined the chat room.
[10:54pm] thewoolleyman: No, it's a PTY, and no, you can't control options. It inherits from the parent master terminal
[10:55pm] thewoolleyman: Right, I know not bash, but I was just trying to understand if this is expected general behavior for pseudo terminals. I can't find the answer on google, so I assume no...
[10:55pm] izabera: uh then PTY.spawn is a weird name
[10:55pm] thewoolleyman: https://ruby-doc.org/stdlib-2.2.3/libdoc/pty/rdoc/PTY.html
[10:56pm] izabera: are you sure it's the parent's?
[10:56pm] austin987 left the chat room. (Ping timeout: 245 seconds)
[10:57pm] thewoolleyman: I could try in another implementation of pseudo terminal to compare - is there a standard bash/posix utility that will let me easily invoke a command in a pseudo terminal?
[10:57pm] w0ng left the chat room. (Quit: leaving)
[10:57pm] thewoolleyman: Well, the slave pty inherits the parents TERM type, and AFAIK that's all that really controls its behavior? But maybe I'm wrong?
[10:58pm] w0ng joined the chat room.
[10:59pm] linuxaddicts_ joined the chat room.
[10:59pm] izabera: faketty () { script -eqfc "$(printf "%q " "$@")" /dev/null; }
[10:59pm] nnpnv6: it's not a problem of a terminal and probably not a problem of bash or shell
[10:59pm] izabera: $ faketty echo hi | od -An -c
[10:59pm] izabera: h i \r \n
[10:59pm] nnpnv6: you can use ... | hexdump -C or ...| od -A x -t x1z -v of ...| xxd '(if you have from vim)' to see what bytes your shell commands produce
[10:59pm] nnpnv6: *or ...| xxd
[10:59pm] izabera: nnpnv6: the pty is converting \n to \r\n
[11:00pm] nnpnv6: the pty in Ruby, so he needs to set up his Ruby pty to stop it, by Ruby ways
[11:01pm] Laif joined the chat room.
[11:01pm] izabera: thewoolleyman: try with the ruby equivalent of stty -onlcr
[11:01pm] nnpnv6: as a hack you can base64-encode output of your shell commands, grok encoded data in Ruby and decode it there for further processing
[11:01pm] izabera: ..what
[11:01pm] cdown joined the chat room.
[11:02pm] gr00t left the chat room. (Read error: Connection reset by peer)
[11:02pm] andries left the chat room. (Read error: Connection reset by peer)
[11:02pm] kimitux joined the chat room.
[11:03pm] thewoolleyman: izabera: Trying to get that faketty() function to work on OSX/BSD... -eqfc isn't valid, but I'm trying to fix. Thanks for helping....
[11:03pm] izabera: try with only -c
[11:04pm] NotThizTime joined the chat room.
[11:04pm] nnpnv6: izabera: about the base64 thing: he has some printf '\n'-like command in his PTY, producer of text on stdout, and he tries to read and process output of this producer; it is possible to encode output into safe intermediate encoding, swallow it in Ruby, ignore all \n and \r, decode it and process the text further
[11:05pm] cdown_ joined the chat room.
[11:07pm] thewoolleyman: nnpnv6: That's possible, but I'm actually working on a library that makes it easier to handle subprocess output in ruby, so I wouldn't want to do anything that complicated. It actually works fine with direct stream processing, but some commands that want a TTY fail, which is why I'm trying to convert it to use a pseudo terminal, and ran into this issue in my tests (which are now failing because they only expect \n, not \r\n).
[11:08pm] rigsby left the chat room. (Remote host closed the connection)
[11:08pm] thewoolleyman: izabera: So, that fails saying it can't write to /dev/null (?!?) but I do see a \r\n in it...
[11:08pm] nnpnv6: thewoolleyman: what about require 'pty'; PTY.spawn('stty -onlcr; printf "a\n"')[0].read(3).chars ?
[11:08pm] izabera: i don't know how your script(1) works...
[11:08pm] dodobrain joined the chat room.
[11:09pm] cdown left the chat room. (Ping timeout: 240 seconds)
[11:09pm] cdown_ is now known as cdown.
[11:09pm] Grapes left the chat room. (Quit: My MacBook has gone to sleep. ZZZzzz…)
[11:09pm] thewoolleyman: nnpnv6: Oh, didn't know about stty, that is useful.
[11:09pm] izabera: ;-;
[11:09pm] thewoolleyman: is stty standard across BSD/linux?
[11:09pm] nnpnv6: it's also possible to make a pty for subordinate process in GNU awk. thewoolleyman: do you have GNU awk installed, gawk?
[11:09pm] nnpnv6: thewoolleyman: who knows
[11:10pm] izabera: yes it's posix
[11:10pm] ballangddang: how can i prompt the user for a key press ?
[11:10pm] thewoolleyman: I don't want to assume anything is install, should run cross-platform with no dependencies.
[11:10pm] nnpnv6: ballangddang: read -N1 -s -p "press any key"
[11:10pm] gautamsomani joined the chat room.
[11:10pm] gautamsomani left the chat room. (Max SendQ exceeded)
[11:11pm] gautamsomani joined the chat room.
[11:11pm] ballangddang: nnpnv6: sorry i mean i need to get the value like in a '[y/N]' scenari
[11:11pm] izabera: thewoolleyman: ship a qemu binary that boots a linux system
[11:12pm] GesrobDR left the chat room. (Quit: Connection closed for inactivity)
[11:14pm] nnpnv6: ballangddang: while read -p "do you agree? [y/N]: " && [[ $REPLY != [yYnN] ]]; do printf %s\\n "Please enter y or n."; done; case $REPLY in y|Y) echo reply was yes ;; n|N) echo reply was no ;; esac
[11:14pm] thewoolleyman: izabera: yes, that's definitely the optimal solution
[11:15pm] nnpnv6: ballangddang: fixed N by default situation: while read -p "do you agree? [y/N]: " && [[ $REPLY != [yYnN] ]]; do printf %s\\n "Please enter y or n."; done; case $REPLY in y|Y) echo reply was yes ;; ''|n|N) echo reply was no ;; esac
[11:15pm] nnpnv6: ... ''|n|N) ...
[11:15pm] lambdaq joined the chat room.
[11:16pm] izabera: thewoolleyman: it's the docker way! ship the whole machine
[11:17pm] ballangddang: nnpnv6: oh $REPLY will automatically host the value from read ?
[11:17pm] nnpnv6: ballangddang: in bash yes
[11:18pm] ballangddang: ok thanks sir
[11:18pm] zer0dev joined the chat room.
[11:19pm] scoofy left the chat room. (Ping timeout: 255 seconds)
[11:19pm] thewoolleyman: nnpnv6: stty was the trick. No idea why onlcr is on by default, but that's what's happening. Thanks so much, nnpnv6 and izabera . This channel always kicks ass.
[11:21pm] node9_ left the chat room. (Ping timeout: 248 seconds)
[11:21pm] _ng joined the chat room.
[11:22pm] joebobjoe joined the chat room.
[11:22pm] Cloudish left the chat room. (Ping timeout: 258 seconds)
[11:22pm] nnpnv6: thewoolleyman: izabera suggested you the stty -onlcr in the first place, and she said that stty is POSIX, but -onlcr option may be non-POSIX extension
[11:23pm] nnpnv6: thewoolleyman: please consider the following: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/stty.html
[11:24pm] koldbrutality: what is a good json editor that supports xpath? something like xmlstarlet?
[11:24pm] izabera: jq but you have to be a wizard
[11:24pm] nnpnv6: thewoolleyman: just checked: onlcr (-onlcr) is standard option.
[11:24pm] thewoolleyman: right.
[11:24pm] nnpnv6: thewoolleyman: but on XSI-compliant systems
[11:25pm] nnpnv6: not all POSIX systems are XSI
[11:25pm] izabera: so it won't work on that solaris box from 1993 you have in your garage
[11:25pm] nnpnv6: *XSI-conformant
[11:25pm] josePHPagoda left the chat room. (Remote host closed the connection)
[11:26pm] linaak joined the chat room.
[11:26pm] DarkCat left the chat room. (Ping timeout: 240 seconds)
[11:26pm] joebobjoe left the chat room. (Ping timeout: 240 seconds)
[11:27pm] SuperficialyLost left the chat room. (Quit: WeeChat 1.6)
[11:28pm] thewoolleyman: so, here's another question - does POSIX by standard always correctly handle (ignore) a \r in a terminal? E.g. is it OK to just let the \r\n happen, and it should always do an EOL correctly?
[11:28pm] josePHPagoda joined the chat room.
[11:28pm] nnpnv6: some terminals need it, some don't
[11:28pm] thewoolleyman: what terminal property determines that?
[11:28pm] thewoolleyman: I know windows, but I don't care about windows
[11:29pm] thewoolleyman: nor editors or files
[11:29pm] thewoolleyman: just terminal IO
[11:29pm] blind: I don't know of \r getting "ignored" -- it's a carriage return
[11:31pm] nnpnv6: you know what, i take my words back. i do not know a single terminal that moves caret in the beginning of line without of explicit \r
[11:32pm] fatdragon joined the chat room.
[11:32pm] blind: Do the terminals automatically include a \r when you only do \n then? Like in: printf '%s\n' foo
[11:33pm] nnpnv6: it looks like so
[11:33pm] nnpnv6: if tty driver does not automatically add \r, \n moves caret to the next line, but its horizontal position does not change
[11:33pm] nnpnv6: so you end up with cursor in the middle of next line
[11:33pm] thewoolleyman: Well, it seems like `printf "a\r\nz"` behaves identically to printf "a\nz" on both linux and OSX with TERM of dumb and xterm-256color, so it seems like it doesn't care
[11:34pm] grumble left the chat room. (Remote host closed the connection)
[11:34pm] thewoolleyman: OH, if the line is longer than 1 char...
[11:34pm] zer0dev1 joined the chat room.
[11:34pm] nnpnv6: no, it does not happen on printf 'a\r\nz' level
[11:35pm] nnpnv6: tty driver (code inside your terminal emulator) turns \n from process's stdout and stderr into \r\n fed to terminal
[11:35pm] zer0dev left the chat room. (Ping timeout: 240 seconds)
[11:35pm] nnpnv6: run script, run some commands under its supervision, leave script, cat -A typescript, see that all \n are replaced by \r\n
[11:35pm] kyan left the chat room. (Quit: Leaving)
[11:36pm] nnpnv6: code inside script turns \n into \r\n
[11:36pm] fatdragon left the chat room. (Ping timeout: 256 seconds)
[11:37pm] nnpnv6: stty allows to change this behavior
[11:37pm] last_staff joined the chat room.
[11:37pm] grumble joined the chat room.
[11:37pm] genericuser123 left the chat room. (Quit: Leaving)
[11:37pm] scoofy joined the chat room.
[11:38pm] linuxaddicts_ is now known as linuxaddicts.
[11:38pm] linuxaddicts left the chat room. (Changing host)
[11:38pm] linuxaddicts joined the chat room.
[11:38pm] linuxaddicts left the chat room. (Changing host)
[11:38pm] linuxaddicts joined the chat room.
[11:38pm] thewoolleyman: Right, back to onlcr. Which seems to be default. So, I think I'm good just ignoring it, and if use cares, they can set -onlcr in the spawning master terminal.
[11:38pm] thewoolleyman: *user
[11:38pm] zer0dev1 left the chat room. (Ping timeout: 245 seconds)
[11:39pm] thewoolleyman: nnpnv6 and izabera - thanks so much for your help. Figuring this stuff out on my own by googling and reading man pages can take days...
[11:39pm] blind: !google
[11:39pm] greybot: Google is NOT a preferred source for learning bash, because almost all the "tutorials" and scripts out there are JUNK. Instead, ask a good question here or refer to the Guide and FAQ (see topic)
[11:39pm] thewoolleyman: haha don't I know greybot...
[11:39pm] iliv joined the chat room.
[11:40pm] zer0dev joined the chat room.
[11:40pm] KindOne_ joined the chat room.
[11:40pm] bel_ki left the chat room. (Quit: Leaving)
[11:40pm] zer0dev left the chat room.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment