delameter/shell-pipes-and-redirects.rst

## shell-pipes-and-redirects.rst

      
    Raw
  

              shell-pipes-and-redirects.rst
            
          
    Shell pipes & redirects

Summary, syntax, examples.

Contents

1.   stdout and stderr
2.   Redirection precedence
3.   Summary
4.   Temporary file
5.   Sequential output without line shuffling
6.   References


1.   stdout and stderr

( cmd ) > /dev/null # discard stdout; same as 1> /dev/null
( cmd ) 2> /dev/null # discard stderr
( cmd ) >&2 # redirect stdout to stderr
( cmd ) 2>&1 # redirect stderr to stdout
This command outputs to stdout as well as to stderr (normally) [1]:
$ ls /tmp /tnt -ld
ls: cannot access '/tnt': No such file or directory
drwxrwxrwt 41 root root 16384 Jan  3 04:39 /tmp

$ ls /tmp /tnt -ld > /dev/null
ls: cannot access '/tnt': No such file or directory

$ ls /tmp /tnt -ld 2> /dev/null
drwxrwxrwt 41 root root 16384 Jan  3 04:39 /tmp
Operator order importance (see precedence):
$ ls /tmp /tnt -ld >/dev/null 2>&1

$ ls /tmp /tnt -ld 2>&1 >/dev/null
ls: cannot access '/tnt': No such file or directory
Invoking pipes:
$ ls /tmp /tnt -ld | sed "s/^.*$/<<< & >>>/"
ls: cannot access '/tnt': No such file or directory
<<< drwxrwxrwt 41 root root 16384 Jan  3 04:49 /tmp >>>

$ ls /tmp /tnt -ld 2>&1 | sed "s/^.*$/<<< & >>>/" # same as ls |& sed
<<< ls: cannot access '/tnt': No such file or directory >>>
<<< drwxrwxrwt 41 root root 16384 Jan  3 04:49 /tmp >>>

$ ls /tmp /tnt -ld >/dev/null 2>&1 | sed "s/^.*$/<<< & >>>/"

$ ls /tmp /tnt -ld 2>&1 >/dev/null | sed "s/^.*$/<<< & >>>/"
<<< ls: cannot access '/tnt': No such file or directory >>>
Doing different operation on both outputs:
$ ( ls /tmp /tnt -ld | sed "s/^/OUT: /" >&9 ) 9>&2 2>&1 | sed "s/^/ERR: /"
ERR: ls: cannot access '/tnt': No such file or directory
OUT: drwxrwxrwt 41 root root 16384 Jan  3 05:58 /tmp
$ # (no idea wtf is &9)
Updated syntax (bash >4.0) for the same thing:
$ ls /tmp /tnt -ld 2> >(sed "s/^/ERR: /") > >(sed "s/^/OUT: /")
ERR: ls: cannot access '/tnt': No such file or directory
OUT: drwxrwxrwt 41 root root 16384 Jan  3 05:58 /tmp

$ cat <(ls /tmp /tnt -ld 2> >(sed "s/^/ERR: /") > >(sed "s/^/OUT: /"))
OUT: drwxrwxrwt 41 root root 16384 Jan  3 06:09 /tmp
ERR: ls: cannot access '/tnt': No such file or directory

$ rev <(ls /tmp /tnt -ld 2> >(sed "s/^/ERR: /") > >(sed "s/^/OUT: /"))
pmt/ 90:60 3  naJ 48361 toor toor 14 twrxwrxwrd :TUO
yrotcerid ro elif hcus oN :'tnt/' ssecca tonnac :sl :RRE

2.   Redirection precedence

$ ( echo stdout ; >&2 echo stderr  ) > >( rev ) 2> >( cat )
tuodts
rredts
The stderr does get passed to cat but cat sends its output
to stdout and stdout has already been redirected to rev. Hence,
both stdout and stderr eventually go to rev.
If we reverse the order, cat sends it output to the terminal because
stdout has not yet been redirected [2]:
$ ( echo stdout ; >&2 echo stderr  ) 2> >( cat ) > >( rev )
stderr
tuodts
Another approach:
$ ( echo stdout ; >&2 echo stderr  ) > >( rev ) 2> >( cat >&2 )
tuodts
stderr

3.   Summary


Syntax
visible in terminal
visible in file
existing
file

StdOut
StdErr
StdOut
StdErr


>
no
yes
yes
no
overwrite

>>
append

2>
yes
no
no
yes
overwrite

2>>
append

&>
no
no
yes
yes
overwrite

&>>
append

| tee
yes
yes
yes
no
overwrite

| tee -a
append

n.e.
[a]
yes
yes
no
yes
overwrite

append

|& tee
yes
yes
yes
yes
overwrite

|& tee -a
append


command > output.txt
The standard output stream will be redirected to the file only, it
will not be visible in the terminal. If the file already exists, it
gets overwritten.  shell
command >> output.txt
The standard output stream will be redirected to the file only, it
will not be visible in the terminal. If the file already exists, the
new data will get appended to the end of the file.
command 2> output.txt
The standard error stream will be redirected to the file only, it
will not be visible in the terminal. If the file already exists, it
gets overwritten.
command 2>> output.txt
The standard error stream will be redirected to the file only, it
will not be visible in the terminal. If the file already exists, the
new data will get appended to the end of the file.
command &> output.txt
Both the standard output and standard error stream will be redirected
to the file only, nothing will be visible in the terminal. If the
file already exists, it gets overwritten.
command &>> output.txt
Both the standard output and standard error stream will be redirected
to the file only, nothing will be visible in the terminal. If the
file already exists, the new data will get appended to the end of the
file..
command | tee output.txt
The standard output stream will be copied to the file, it will still
be visible in the terminal. If the file already exists, it gets
overwritten.
command | tee -a output.txt
The standard output stream will be copied to the file, it will still
be visible in the terminal. If the file already exists, the new data
will get appended to the end of the file.


[a] Bash has no shorthand syntax that allows piping only StdErr to a
second command, which would be needed here in combination with tee
again to complete the table. If you really need something like that,
please look at "How to pipe stderr, and not stdout?" on Stack
Overflow for some ways how this can be done e.g. by swapping streams
or using process substitution.


command |& tee output.txt
Both the standard output and standard error streams will be copied to
the file while still being visible in the terminal. If the file
already exists, it gets overwritten.
command |& tee -a output.txt
Both the standard output and standard error streams will be copied to
the file while still being visible in the terminal. If the file
already exists, the new data will get appended to the end of the
file.

Source: [3]

4.   Temporary file

tmpfile=$(mktemp)
exec {FDW}>"$tmpfile"
exec {FDR}<"$tmpfile"
rm "$tmpfile"
# ... script continues
echo A >&$FDW  # write some data
wc -l <&$FDR  # read some data
Source: [4]

5.   Sequential output without line shuffling

The key is to use flock in exclusive mode [5]:
#!/bin/bash

PIPE=/tmp/temppipe
[[ ! -p $PIPE ]] && mkfifo $PIPE

for i in {1..3} ; do
    echo "Spawned writer $i"
    flock -x $PIPE seq -f "[$i] %.0f" 4 > $PIPE &
done

echo "Reading starts"
cat $PIPE
echo "Reading done"

rm $PIPE
Spawned writer 1
Spawned writer 2
Spawned writer 3
Reading starts
[2] 1
[2] 2
[2] 3
[2] 4
[1] 1
[1] 2
[1] 3
[1] 4
[3] 1
[3] 2
[3] 3
[3] 4
Reading done

6.   References


[1] https://stackoverflow.com/a/16283739/5834973


[2] https://stackoverflow.com/a/V16283739/5834973


[3] https://askubuntu.com/a/731237/1498488


[4] https://stackoverflow.com/a/16283739/5834973


[5] https://stackoverflow.com/a/4113995/5834973


ReST layout: Alexandr Shavykin

Contact: 0.delameter@gmail.com

Date: 24-Jul-24 08:12:16 PDT
Syntax	visible in terminal		visible in file		existing file
Syntax	StdOut	StdErr	StdOut	StdErr	existing file
>	no	yes	yes	no	overwrite
>>	no	yes	yes	no	append
2>	yes	no	no	yes	overwrite
2>>	yes	no	no	yes	append
&>	no	no	yes	yes	overwrite
&>>	no	no	yes	yes	append
\| tee	yes	yes	yes	no	overwrite
\| tee -a	yes	yes	yes	no	append
n.e. [a]	yes	yes	no	yes	overwrite
n.e. [a]	yes	yes	no	yes	append
\|& tee	yes	yes	yes	yes	overwrite
\|& tee -a	yes	yes	yes	yes	append
ReST layout:	Alexandr Shavykin
Contact:	0.delameter@gmail.com
Date:	24-Jul-24 08:12:16 PDT