shell utilities

awk

AWK

URLs:
https://blog.jpalardy.com/posts/why-learn-awk/
https://github.com/learnbyexample/Command-line-text-processing/blob/master/gnu_awk.md <- great intro to features

Sometimes called the Excel of the command-line
Follows the UNIX philosophy of "write programs as text filters"
Generally, a hint that you want AWK is when you write a shell script where you are piping sed to cut to grep to grep to tail etc.
It's everywhere due to being a requirement of the POSIX standard

Size of programs
Kernighan said that most AWK programs (scripts) don't go past 5 LOC
However, I've read at least one interview where, at Bell Labs, they were surprised to find out that one of the users of the UNIX project went ahead and built an AWK programs >100 LOC and it worked just fine.

When is it appropriate?
- A program needs to run through the input only once
- The input might be big enough that you almost certainly want to stream your way through it (Not that big a requirement for us since we don't have "big data").
- Your input is in a streamable form.  That leaves out fancy things like XML and JSON.  If you can get the data into something that can be imported into a spreadsheet, you are good to go.
- CSV, ADV, TSV, etc.
- Read https://ronaldduncan.wordpress.com/2009/10/31/text-file-formats-ascii-delimited-text-not-csv-or-tab-delimited-text/ and then rail against the injustices in this world.  This problem was solved in the f*$%#@g 1960s.
- https://en.wikipedia.org/wiki/ASCII

busybox

Busybox

URLs
https://en.wikipedia.org/wiki/BusyBox
https://www.busybox.net/

It has a bunch of stripped down common UNIX shell programs all rolled into a single package.

- nc
- httpd
- see https://www.busybox.net/downloads/BusyBox.html for a full list of commands

httpd
Its a tiny little webserver.  Command follows.
httpd -fvp 3567 .
Put files you want to share into a dir.  "cd" into the dir.  Run the above command, and hand your IP addr and port to whoever you want to share files with, with the name of the file.  They run
http://YOURIP:PORT/filename.foo
and they get to download the file.  Then you shut your webserver down.
It's similar to the "python -m http " command, but with a much smaller footprint.

nc
I find that I will sometimes just use the netcat from this project because I know it's under regular development, and it's feature-set is small enough that there are no surprises.  A busybox docker container weighs in at almost nothing, and I know it will work the same everywhere.
https://hub.docker.com/_/busybox

netcat

Netcat

https://en.wikipedia.org/wiki/Netcat
https://www.digitalocean.com/community/tutorials/how-to-use-netcat-to-establish-and-test-tcp-and-udp-connections-on-a-vps

Homepage?
- GNU - http://netcat.sourceforge.net/
- https://nmap.org/ncat/ <- This is new and fancy and has SSL support
- OpenBSD - https://man.openbsd.org/nc

This program is so old, I couldn't figure out how it was versioned.

Protocols:
- I've only used it for direct TCP connections
- Never used it for UDP packets (fire and forget)

Ways I've used it:
- Great as a 1-on-1 chat client, as long as you know each other's IP addrs or DNS hostnames, and the Port on the receiving end.
- Great for file-transfer, and can be used almost similarly to how we use AirDrop to transfer files.
- Use Openbsd's netcat (maybe?) if you want SSL, or use the Nmap version.
- On my Ubuntu box at home, nc is from GNU, but the manpage is from BSD. #shrug #chaos
- If you happen to not have a version of nc with ssl integrated, that's ok.  Just use the openssl tool directly to encrypt the file using a symmetric key (password) before sending it over the wire.

openssl

OpenSSL

So good.  Such a base part of the interwebz.
SSH depends on this thing working.

I've nearly always never used it directly, till a few months ago when I read through it's manpage.

Use case:
- Create a CA (private and public)
- Create SSL certs (private and public)
- Sign certs using your CA's private key
- Give people your CA's public key so they can add it as a trusted key
- Profit

Use case:
- http://stackoverflow.com/questions/16056135/ddg#16056298
- Encrypt a file (large or small) using a password (symmetric key)
- openssl aes-256-cbc -salt < foo > foo.enc
- openssl aes-256-cbc -d < foo.enc > foo2

Use case:
- Create random password
- openssl rand 30
- openssl rand -hex 30
- openssl rand -base64 30