-
-
Save ravster/fb92697cea6d1fb14e760c34361cc779 to your computer and use it in GitHub Desktop.
shell utilities
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AWK | |
URLs: | |
https://blog.jpalardy.com/posts/why-learn-awk/ | |
https://github.com/learnbyexample/Command-line-text-processing/blob/master/gnu_awk.md <- great intro to features | |
Sometimes called the Excel of the command-line | |
Follows the UNIX philosophy of "write programs as text filters" | |
Generally, a hint that you want AWK is when you write a shell script where you are piping sed to cut to grep to grep to tail etc. | |
It's everywhere due to being a requirement of the POSIX standard | |
Size of programs | |
Kernighan said that most AWK programs (scripts) don't go past 5 LOC | |
However, I've read at least one interview where, at Bell Labs, they were surprised to find out that one of the users of the UNIX project went ahead and built an AWK programs >100 LOC and it worked just fine. | |
When is it appropriate? | |
- A program needs to run through the input only once | |
- The input might be big enough that you almost certainly want to stream your way through it (Not that big a requirement for us since we don't have "big data"). | |
- Your input is in a streamable form. That leaves out fancy things like XML and JSON. If you can get the data into something that can be imported into a spreadsheet, you are good to go. | |
- CSV, ADV, TSV, etc. | |
- Read https://ronaldduncan.wordpress.com/2009/10/31/text-file-formats-ascii-delimited-text-not-csv-or-tab-delimited-text/ and then rail against the injustices in this world. This problem was solved in the f*$%#@g 1960s. | |
- https://en.wikipedia.org/wiki/ASCII |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Busybox | |
URLs | |
https://en.wikipedia.org/wiki/BusyBox | |
https://www.busybox.net/ | |
It has a bunch of stripped down common UNIX shell programs all rolled into a single package. | |
- nc | |
- httpd | |
- see https://www.busybox.net/downloads/BusyBox.html for a full list of commands | |
httpd | |
Its a tiny little webserver. Command follows. | |
httpd -fvp 3567 . | |
Put files you want to share into a dir. "cd" into the dir. Run the above command, and hand your IP addr and port to whoever you want to share files with, with the name of the file. They run | |
http://YOURIP:PORT/filename.foo | |
and they get to download the file. Then you shut your webserver down. | |
It's similar to the "python -m http " command, but with a much smaller footprint. | |
nc | |
I find that I will sometimes just use the netcat from this project because I know it's under regular development, and it's feature-set is small enough that there are no surprises. A busybox docker container weighs in at almost nothing, and I know it will work the same everywhere. | |
https://hub.docker.com/_/busybox |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Netcat | |
https://en.wikipedia.org/wiki/Netcat | |
https://www.digitalocean.com/community/tutorials/how-to-use-netcat-to-establish-and-test-tcp-and-udp-connections-on-a-vps | |
Homepage? | |
- GNU - http://netcat.sourceforge.net/ | |
- https://nmap.org/ncat/ <- This is new and fancy and has SSL support | |
- OpenBSD - https://man.openbsd.org/nc | |
This program is so old, I couldn't figure out how it was versioned. | |
Protocols: | |
- I've only used it for direct TCP connections | |
- Never used it for UDP packets (fire and forget) | |
Ways I've used it: | |
- Great as a 1-on-1 chat client, as long as you know each other's IP addrs or DNS hostnames, and the Port on the receiving end. | |
- Great for file-transfer, and can be used almost similarly to how we use AirDrop to transfer files. | |
- Use Openbsd's netcat (maybe?) if you want SSL, or use the Nmap version. | |
- On my Ubuntu box at home, nc is from GNU, but the manpage is from BSD. #shrug #chaos | |
- If you happen to not have a version of nc with ssl integrated, that's ok. Just use the openssl tool directly to encrypt the file using a symmetric key (password) before sending it over the wire. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
OpenSSL | |
So good. Such a base part of the interwebz. | |
SSH depends on this thing working. | |
I've nearly always never used it directly, till a few months ago when I read through it's manpage. | |
Use case: | |
- Create a CA (private and public) | |
- Create SSL certs (private and public) | |
- Sign certs using your CA's private key | |
- Give people your CA's public key so they can add it as a trusted key | |
- Profit | |
Use case: | |
- http://stackoverflow.com/questions/16056135/ddg#16056298 | |
- Encrypt a file (large or small) using a password (symmetric key) | |
- openssl aes-256-cbc -salt < foo > foo.enc | |
- openssl aes-256-cbc -d < foo.enc > foo2 | |
Use case: | |
- Create random password | |
- openssl rand 30 | |
- openssl rand -hex 30 | |
- openssl rand -base64 30 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment