Last active
January 16, 2017 06:42
-
-
Save thewoolleyman/09032fb1b5f6e8e5d798bf91ba4daca5 to your computer and use it in GitHub Desktop.
ruby PTY turns LF into CF+LF
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
irb | |
2.3.3 :001 > puts `uname -a` | |
Darwin pima 15.6.0 Darwin Kernel Version 15.6.0: Wed Nov 2 20:30:56 PDT 2016; root:xnu-3248.60.11.1.2~2/RELEASE_X86_64 x86_64 | |
=> nil | |
2.3.3 :002 > `printf "a\n"`.chars | |
=> ["a", "\n"] | |
2.3.3 :003 > require 'pty'; PTY.spawn('printf "a\n"')[0].read(3).chars | |
=> ["a", "\r", "\n"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[10:45pm] ChanServ: [#bash] Welcome to #bash, we discuss the GNU bash shell. Be explicit if your question is about another shell. Don't trust the Internet on bash, learn from the resources in the topic instead. | |
[10:49pm] thewoolleyman: When I write to a pseudo-terminal on POSIX (OSX), I'm seeing LF (\n) be turned into CR+LF (\r\n). Is this expected behavior for all pseudo-terminals, or something wrong with my implementation? I'm trying it in Ruby via the built-in PTY support: https://gist.github.com/thewoolleyman/09032fb1b5f6e8e5d798bf91ba4daca5 Thanks! | |
[10:49pm] gautamsomani left the chat room. (Quit: Leaving.) | |
[10:51pm] gentunian left the chat room. (Ping timeout: 248 seconds) | |
[10:52pm] izabera: thewoolleyman: can you control some options for that pty? | |
[10:53pm] nnpnv6: thewoolleyman: not a bash question, but File.open in ruby respects the letter b second argument for "binary", so that LF to CF-LF conversion shall not happen | |
[10:54pm] izabera: i don't know much ruby but that doesn't look like File.open | |
[10:54pm] tdy joined the chat room. | |
[10:54pm] thewoolleyman: No, it's a PTY, and no, you can't control options. It inherits from the parent master terminal | |
[10:55pm] thewoolleyman: Right, I know not bash, but I was just trying to understand if this is expected general behavior for pseudo terminals. I can't find the answer on google, so I assume no... | |
[10:55pm] izabera: uh then PTY.spawn is a weird name | |
[10:55pm] thewoolleyman: https://ruby-doc.org/stdlib-2.2.3/libdoc/pty/rdoc/PTY.html | |
[10:56pm] izabera: are you sure it's the parent's? | |
[10:56pm] austin987 left the chat room. (Ping timeout: 245 seconds) | |
[10:57pm] thewoolleyman: I could try in another implementation of pseudo terminal to compare - is there a standard bash/posix utility that will let me easily invoke a command in a pseudo terminal? | |
[10:57pm] w0ng left the chat room. (Quit: leaving) | |
[10:57pm] thewoolleyman: Well, the slave pty inherits the parents TERM type, and AFAIK that's all that really controls its behavior? But maybe I'm wrong? | |
[10:58pm] w0ng joined the chat room. | |
[10:59pm] linuxaddicts_ joined the chat room. | |
[10:59pm] izabera: faketty () { script -eqfc "$(printf "%q " "$@")" /dev/null; } | |
[10:59pm] nnpnv6: it's not a problem of a terminal and probably not a problem of bash or shell | |
[10:59pm] izabera: $ faketty echo hi | od -An -c | |
[10:59pm] izabera: h i \r \n | |
[10:59pm] nnpnv6: you can use ... | hexdump -C or ...| od -A x -t x1z -v of ...| xxd '(if you have from vim)' to see what bytes your shell commands produce | |
[10:59pm] nnpnv6: *or ...| xxd | |
[10:59pm] izabera: nnpnv6: the pty is converting \n to \r\n | |
[11:00pm] nnpnv6: the pty in Ruby, so he needs to set up his Ruby pty to stop it, by Ruby ways | |
[11:01pm] Laif joined the chat room. | |
[11:01pm] izabera: thewoolleyman: try with the ruby equivalent of stty -onlcr | |
[11:01pm] nnpnv6: as a hack you can base64-encode output of your shell commands, grok encoded data in Ruby and decode it there for further processing | |
[11:01pm] izabera: ..what | |
[11:01pm] cdown joined the chat room. | |
[11:02pm] gr00t left the chat room. (Read error: Connection reset by peer) | |
[11:02pm] andries left the chat room. (Read error: Connection reset by peer) | |
[11:02pm] kimitux joined the chat room. | |
[11:03pm] thewoolleyman: izabera: Trying to get that faketty() function to work on OSX/BSD... -eqfc isn't valid, but I'm trying to fix. Thanks for helping.... | |
[11:03pm] izabera: try with only -c | |
[11:04pm] NotThizTime joined the chat room. | |
[11:04pm] nnpnv6: izabera: about the base64 thing: he has some printf '\n'-like command in his PTY, producer of text on stdout, and he tries to read and process output of this producer; it is possible to encode output into safe intermediate encoding, swallow it in Ruby, ignore all \n and \r, decode it and process the text further | |
[11:05pm] cdown_ joined the chat room. | |
[11:07pm] thewoolleyman: nnpnv6: That's possible, but I'm actually working on a library that makes it easier to handle subprocess output in ruby, so I wouldn't want to do anything that complicated. It actually works fine with direct stream processing, but some commands that want a TTY fail, which is why I'm trying to convert it to use a pseudo terminal, and ran into this issue in my tests (which are now failing because they only expect \n, not \r\n). | |
[11:08pm] rigsby left the chat room. (Remote host closed the connection) | |
[11:08pm] thewoolleyman: izabera: So, that fails saying it can't write to /dev/null (?!?) but I do see a \r\n in it... | |
[11:08pm] nnpnv6: thewoolleyman: what about require 'pty'; PTY.spawn('stty -onlcr; printf "a\n"')[0].read(3).chars ? | |
[11:08pm] izabera: i don't know how your script(1) works... | |
[11:08pm] dodobrain joined the chat room. | |
[11:09pm] cdown left the chat room. (Ping timeout: 240 seconds) | |
[11:09pm] cdown_ is now known as cdown. | |
[11:09pm] Grapes left the chat room. (Quit: My MacBook has gone to sleep. ZZZzzz…) | |
[11:09pm] thewoolleyman: nnpnv6: Oh, didn't know about stty, that is useful. | |
[11:09pm] izabera: ;-; | |
[11:09pm] thewoolleyman: is stty standard across BSD/linux? | |
[11:09pm] nnpnv6: it's also possible to make a pty for subordinate process in GNU awk. thewoolleyman: do you have GNU awk installed, gawk? | |
[11:09pm] nnpnv6: thewoolleyman: who knows | |
[11:10pm] izabera: yes it's posix | |
[11:10pm] ballangddang: how can i prompt the user for a key press ? | |
[11:10pm] thewoolleyman: I don't want to assume anything is install, should run cross-platform with no dependencies. | |
[11:10pm] nnpnv6: ballangddang: read -N1 -s -p "press any key" | |
[11:10pm] gautamsomani joined the chat room. | |
[11:10pm] gautamsomani left the chat room. (Max SendQ exceeded) | |
[11:11pm] gautamsomani joined the chat room. | |
[11:11pm] ballangddang: nnpnv6: sorry i mean i need to get the value like in a '[y/N]' scenari | |
[11:11pm] izabera: thewoolleyman: ship a qemu binary that boots a linux system | |
[11:12pm] GesrobDR left the chat room. (Quit: Connection closed for inactivity) | |
[11:14pm] nnpnv6: ballangddang: while read -p "do you agree? [y/N]: " && [[ $REPLY != [yYnN] ]]; do printf %s\\n "Please enter y or n."; done; case $REPLY in y|Y) echo reply was yes ;; n|N) echo reply was no ;; esac | |
[11:14pm] thewoolleyman: izabera: yes, that's definitely the optimal solution | |
[11:15pm] nnpnv6: ballangddang: fixed N by default situation: while read -p "do you agree? [y/N]: " && [[ $REPLY != [yYnN] ]]; do printf %s\\n "Please enter y or n."; done; case $REPLY in y|Y) echo reply was yes ;; ''|n|N) echo reply was no ;; esac | |
[11:15pm] nnpnv6: ... ''|n|N) ... | |
[11:15pm] lambdaq joined the chat room. | |
[11:16pm] izabera: thewoolleyman: it's the docker way! ship the whole machine | |
[11:17pm] ballangddang: nnpnv6: oh $REPLY will automatically host the value from read ? | |
[11:17pm] nnpnv6: ballangddang: in bash yes | |
[11:18pm] ballangddang: ok thanks sir | |
[11:18pm] zer0dev joined the chat room. | |
[11:19pm] scoofy left the chat room. (Ping timeout: 255 seconds) | |
[11:19pm] thewoolleyman: nnpnv6: stty was the trick. No idea why onlcr is on by default, but that's what's happening. Thanks so much, nnpnv6 and izabera . This channel always kicks ass. | |
[11:21pm] node9_ left the chat room. (Ping timeout: 248 seconds) | |
[11:21pm] _ng joined the chat room. | |
[11:22pm] joebobjoe joined the chat room. | |
[11:22pm] Cloudish left the chat room. (Ping timeout: 258 seconds) | |
[11:22pm] nnpnv6: thewoolleyman: izabera suggested you the stty -onlcr in the first place, and she said that stty is POSIX, but -onlcr option may be non-POSIX extension | |
[11:23pm] nnpnv6: thewoolleyman: please consider the following: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/stty.html | |
[11:24pm] koldbrutality: what is a good json editor that supports xpath? something like xmlstarlet? | |
[11:24pm] izabera: jq but you have to be a wizard | |
[11:24pm] nnpnv6: thewoolleyman: just checked: onlcr (-onlcr) is standard option. | |
[11:24pm] thewoolleyman: right. | |
[11:24pm] nnpnv6: thewoolleyman: but on XSI-compliant systems | |
[11:25pm] nnpnv6: not all POSIX systems are XSI | |
[11:25pm] izabera: so it won't work on that solaris box from 1993 you have in your garage | |
[11:25pm] nnpnv6: *XSI-conformant | |
[11:25pm] josePHPagoda left the chat room. (Remote host closed the connection) | |
[11:26pm] linaak joined the chat room. | |
[11:26pm] DarkCat left the chat room. (Ping timeout: 240 seconds) | |
[11:26pm] joebobjoe left the chat room. (Ping timeout: 240 seconds) | |
[11:27pm] SuperficialyLost left the chat room. (Quit: WeeChat 1.6) | |
[11:28pm] thewoolleyman: so, here's another question - does POSIX by standard always correctly handle (ignore) a \r in a terminal? E.g. is it OK to just let the \r\n happen, and it should always do an EOL correctly? | |
[11:28pm] josePHPagoda joined the chat room. | |
[11:28pm] nnpnv6: some terminals need it, some don't | |
[11:28pm] thewoolleyman: what terminal property determines that? | |
[11:28pm] thewoolleyman: I know windows, but I don't care about windows | |
[11:29pm] thewoolleyman: nor editors or files | |
[11:29pm] thewoolleyman: just terminal IO | |
[11:29pm] blind: I don't know of \r getting "ignored" -- it's a carriage return | |
[11:31pm] nnpnv6: you know what, i take my words back. i do not know a single terminal that moves caret in the beginning of line without of explicit \r | |
[11:32pm] fatdragon joined the chat room. | |
[11:32pm] blind: Do the terminals automatically include a \r when you only do \n then? Like in: printf '%s\n' foo | |
[11:33pm] nnpnv6: it looks like so | |
[11:33pm] nnpnv6: if tty driver does not automatically add \r, \n moves caret to the next line, but its horizontal position does not change | |
[11:33pm] nnpnv6: so you end up with cursor in the middle of next line | |
[11:33pm] thewoolleyman: Well, it seems like `printf "a\r\nz"` behaves identically to printf "a\nz" on both linux and OSX with TERM of dumb and xterm-256color, so it seems like it doesn't care | |
[11:34pm] grumble left the chat room. (Remote host closed the connection) | |
[11:34pm] thewoolleyman: OH, if the line is longer than 1 char... | |
[11:34pm] zer0dev1 joined the chat room. | |
[11:34pm] nnpnv6: no, it does not happen on printf 'a\r\nz' level | |
[11:35pm] nnpnv6: tty driver (code inside your terminal emulator) turns \n from process's stdout and stderr into \r\n fed to terminal | |
[11:35pm] zer0dev left the chat room. (Ping timeout: 240 seconds) | |
[11:35pm] nnpnv6: run script, run some commands under its supervision, leave script, cat -A typescript, see that all \n are replaced by \r\n | |
[11:35pm] kyan left the chat room. (Quit: Leaving) | |
[11:36pm] nnpnv6: code inside script turns \n into \r\n | |
[11:36pm] fatdragon left the chat room. (Ping timeout: 256 seconds) | |
[11:37pm] nnpnv6: stty allows to change this behavior | |
[11:37pm] last_staff joined the chat room. | |
[11:37pm] grumble joined the chat room. | |
[11:37pm] genericuser123 left the chat room. (Quit: Leaving) | |
[11:37pm] scoofy joined the chat room. | |
[11:38pm] linuxaddicts_ is now known as linuxaddicts. | |
[11:38pm] linuxaddicts left the chat room. (Changing host) | |
[11:38pm] linuxaddicts joined the chat room. | |
[11:38pm] linuxaddicts left the chat room. (Changing host) | |
[11:38pm] linuxaddicts joined the chat room. | |
[11:38pm] thewoolleyman: Right, back to onlcr. Which seems to be default. So, I think I'm good just ignoring it, and if use cares, they can set -onlcr in the spawning master terminal. | |
[11:38pm] thewoolleyman: *user | |
[11:38pm] zer0dev1 left the chat room. (Ping timeout: 245 seconds) | |
[11:39pm] thewoolleyman: nnpnv6 and izabera - thanks so much for your help. Figuring this stuff out on my own by googling and reading man pages can take days... | |
[11:39pm] blind: !google | |
[11:39pm] greybot: Google is NOT a preferred source for learning bash, because almost all the "tutorials" and scripts out there are JUNK. Instead, ask a good question here or refer to the Guide and FAQ (see topic) | |
[11:39pm] thewoolleyman: haha don't I know greybot... | |
[11:39pm] iliv joined the chat room. | |
[11:40pm] zer0dev joined the chat room. | |
[11:40pm] KindOne_ joined the chat room. | |
[11:40pm] bel_ki left the chat room. (Quit: Leaving) | |
[11:40pm] zer0dev left the chat room. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment