Last active
April 8, 2022 11:38
-
-
Save erikdw/670f4138914a524725921ae48cff265b to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Raw version of https://github.com/erikdw/tips-and-tricks/wiki/Tips-and-Tricks without ugly formatting. | |
| 1. SSH break-out: | |
| ``` | |
| <enter>~. | |
| <enter>~? | |
| ``` | |
| 2. Shadowed window screenshot in mac: | |
| ``` | |
| command-shift-4 then hit <spacebar>, then click in the window | |
| ``` | |
| 3. selecting a path when double clicking in Terminal.app in mac: | |
| ``` | |
| command-shift-doubleclick | |
| ``` | |
| 4. Copy | |
| ``` | |
| rsync -ave ssh ~/dir/ 192.168.16.131:dir/ | |
| rsync -ave ssh HOSTNAME:/dir /dir/ | |
| ``` | |
| 5. Show hidden files in Finder: | |
| ``` | |
| defaults write com.apple.finder AppleShowAllFiles TRUE | |
| killall Finder | |
| (to disable set to FALSE) | |
| ``` | |
| 6. Show space-available at bottom of Finder windows: | |
| ``` | |
| View > Show Status Bar | |
| ``` | |
| 7. Bash loops | |
| ``` | |
| % for h in foobar-{app1,app2,lb1,lb2,utility1}.baz; do ssh $h 'echo `hostname -f` `readlink /usr/local | cut -d/ -f4`'; done | |
| % i=0 ; while [ $i -ne 50 ]; do i=$(($i+1)); echo -n "=== $i ===" ; time curl http://assets1.grouponcdn.com/static/stylesheets/app/home/index-iFTURNG0.css -s -o /dev/null; done | |
| % for h in splunk{1,2,3,4,5,6,7}; do ssh $h 'hostname -f && df -h'; done | |
| % for h in 192.168.40.{70,71,84,85}; do echo -n "$h:"; ipmitool -I lanplus -H $h -U ADMIN -P ADMIN mc info; done | |
| % while : ; do echo foo ; sleep 5; done | |
| % sh -c 'while : ; do sh -c "date ; time curl -s http://config/host/HOSTNAME -o /dev/null ; date" >> curl.1.log 2>&1 ; sleep 5 ; done' | |
| ``` | |
| 8. cut use | |
| ``` | |
| bin/add_package lockrun-2011.09.02_15.52 `grep lockrun hostclasses/*.yml | cut -d: -f1 | xargs echo` | |
| vim `git stat | grep modified | cut -d: -f 2` | |
| ``` | |
| 9. Package/hostclass manipulation: | |
| ``` | |
| date +%Y.%m.%d_%H.%M | |
| git stat hostclasses | grep modified | cut -d: -f2 | cut -d/ -f2 | cut -d. -f1 | awk '{print echo "git tag " $1 "-2011.09.02_16.48"}' | |
| ``` | |
| 10. Finding non-ascii characters: | |
| ``` | |
| find app | xargs grep -P "[\x80-\xFF]" | awk -F':' '{print $1}' | sort | uniq -c | |
| ``` | |
| 11. mv with shell expansion with {} | |
| ``` | |
| 0 eweathers@G12885(~) % mv .rvm{,.bak} | |
| 0 eweathers@G12885(~) % echo .rvm{,.bak} | |
| .rvm.bak | |
| 0 eweathers@G12885(~) % echo .rvm{.bak,123,1,1} | |
| .rvm.bak .rvm123 .rvm1 .rvm1 | |
| for i in `ls -l /usr/bin/* | grep Ruby.framework | awk '{print $9}'`; do echo sudo mv `dirname $i`/`basename $i @`{,.bak}; done | |
| ``` | |
| 12. Using diff+ssh in a single line to compare across hosts: | |
| ``` | |
| diff <(ssh HOSTNAME cat /etc/ssh/sshd_config) <(ssh HOSTNAME2 cat /etc/ssh/sshd_config) | |
| ``` | |
| That trick is called "process substitution". | |
| Here are some related tips: | |
| http://www.linuxtutorialblog.com/post/tutorial-the-best-tips-tricks-for-bash | |
| http://www.commandlinefu.com/commands/using/diff | |
| https://gist.github.com/erikdw/0670fe94fa6027fbf1ebb6a44efab99c | |
| 13. tcpdumps: (no -s 65535 needed on recent tcpdumps) | |
| http://danielmiessler.com/study/tcpdump/ | |
| (a) find dns requests (adjusted for being on hosts in terremark): | |
| sudo tcpdump -i eth1 -c3000 -l -n dst port 53 | grep api | |
| (thepoint@tm22-s00311) | |
| sudo tcpdump -i eth0 -c3000 -l -n port 3133 | |
| (b) capture all data: | |
| sudo tcpdump -i eth0 -w /tmp/1.pcap | |
| (c) capture all data except port 22 (SSH): | |
| sudo tcpdump -i eth0 -w /tmp/1.pcap port not 22 | |
| (d) capture just dns: | |
| sudo tcpdump -i eth0 -w /tmp/1.pcap port 53 | |
| (e) find nfs requests | |
| sudo tcpdump -n -i any -c10000 dst port nfs | grep --line-buffered '"' | cut '-d"' -f2 | |
| (f) print HTTP GET requests | |
| sudo tcpdump -A -i eth0 -vvv -s 500 'tcp port 80 and ip[2:2] > 40 and tcp[tcpflags] & tcp-push != 0 and dst port 80' -f | |
| (g) debugging ganglia's gmetad | |
| sudo tcpdump -w /tmp/gmetad.pcap -s 0 -vxli lo src host localhost and dst host localhost and dst portrange 8660-8777 | |
| (h) capturing traffic for a particular source subnet | |
| sudo tcpdump -i eth0 -w /tmp/1.pcap src net 50.115.209.0/24 | |
| (i) capturing traffic for a particular host with a particular port | |
| sudo tcpdump -i eth0 -w /tmp/1.pcap host 10.20.43.21 and port 22 | |
| (j) capture multicast or broadcast | |
| sudo tcpdump -i eth0 -w /tmp/1.pcap -n "multicast or broadcast" | |
| (k) show outbound TCP traffic that is not SSH | |
| sudo tcpdump -i eth0 tcp and src host `ip address show dev eth0 primary | sed -n 's,^ inet ,,p' | sed 's,/.*$,,'` and port not 22 | |
| (l) rotating output file every 600 secs, retain only 18 files | |
| sudo tcpdump -w avatax-slow-pings-%Y%m%d%H%M%S.dump -G 600 -W 18 -vv '(host avatax.avalara.net and port 443)' | |
| (j) tcpdump -i eth0 -w /tmp/1.pcap 'tcp port 8080 or tcp portrange 31000-31100' | |
| 14. SSHing thru a bastion to another host using 2 different keys: | |
| ssh -i ~/.ssh/us_east.pem -o ProxyCommand='ssh bastion.east-aws.domainname nc %h %p' root@SOMETARGETHOST | |
| - The pem key here is for sshing into the SOMETARGETHOST host as root. | |
| - My default ssh key is being used for executing the proxy command of sshing into the bastion.east-aws.domainname | |
| 15. Find out which processes are listening on TCP ports on Mac: | |
| sudo lsof -P -i tcp | grep -i listen | |
| 16. Create SSH tunnel to a particular port on a server (TARGETHOST) thru a bastion (ssh jump) host (BASTIONHOST): | |
| To use localhost:8001 to connect to port 80 of TARGETHOST | |
| ssh -L8001:BASTIONHOST:80 -qnN BASTIONHOST | |
| To use localhost:2200 to connect to port 22 of TARGETHOST | |
| ssh -L2200:BASTIONHOST:22 -qnN BASTIONHOST | |
| Another version: | |
| ssh -f -N -L 50000:TARGETHOSTSHORTNAME:50000 BASTIONHOST | |
| 17. Create a general HTTP proxy over an SSH tunnel thru a bastion: | |
| ssh -L 21785:127.0.0.1:21785 -R 21785:127.0.0.1:21786 -L3133:TARGETHOST:3128 BASTIONHOST -qnN | |
| Then you set your HTTP proxy (in your app's config) to localhost:3133, and send your HTTP requests thru it, and they will be proxied out thru the final host specified (TARGETHOST in the case above). | |
| TODO: study the options here and grok them. | |
| NOTE: this cmd is created automatically using the autossh package and this cmd: | |
| autossh -M "20001" -f '-L3131:TARGETHOST:3128 BASTIONHOST' -qnN | |
| 18. Translate Excel CSV to have unix-parseable newlines instead of ^M chars: | |
| tr -s '^M' '\n' < foo-types.csv > foo-types-newlines.csv | |
| (where ^M is entered as "ctrl-V-M") | |
| Alternatively: | |
| tr -s '\015' '\012' | |
| 19. History expansion in zsh | |
| http://www.acm.uiuc.edu/workshops/zsh/history/hist_expn.html | |
| !^ - first arg | |
| !$ - last arg | |
| 20. Printing last N commands in zsh, and getting the time spent in each: | |
| fc -D -l -1000 | |
| 21. Printing all commands in zsh history: | |
| fc -l 1 | |
| 22. Suppress command numbers in zsh history: | |
| fc -ln 1 | |
| 23. add prefix to input lines coming in, and combine into a single line with the strings separated by spaces: | |
| some-cmd | perl -pe '$_ = "hostclasses/" . $_' | tr -s '\012' ' ' | |
| 24. Fix broken ssh-agent forwarding that is blocking a deploy. Try in a fresh shell: | |
| eval `ssh-agent` | |
| ssh-add | |
| (re-run your deploy command) | |
| 25. Ruby 1-liner to print a range of numbers in a comma sequence | |
| ruby -e '(51..75).each { |n| print "#{n}," }' | |
| Better (no trailing comma): | |
| ruby -e 'puts (51..75).inject{|s,n| "#{s},#{n}"}' | |
| works: | |
| for h in 192.168.64.{51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75}; do echo -n "$h:"; ipmitool -I lanplus -H $h -U ADMIN -P ADMIN power status; done | |
| does not work: | |
| for h in 192.168.64.{`ruby -e 'puts (51..75).inject{|s,n| "#{s},#{n}"}'`}; do echo -n "$h:"; ipmitool -I lanplus -H $h -U ADMIN -P ADMIN power status; done | |
| 26. Slow-running tail of large file across NFS to keep filehandle open: | |
| thepoint@tm50-s00123 ~ $ perl -e 'open my $fh, "/shared39/thepoint/sitemap.xml" or die; while(<$fh>) { print $_; select undef, undef, undef, 0.5 } close $fh' | |
| For the record, here's the change he applied on both tm50-s00039 (NFS primary) and tm27-s00202 (warm backup): | |
| 27. See entire history of changes for a file: | |
| git log --follow -p <filename> | |
| 28. Debugging "rake test" tasks: | |
| a) use rdebug: | |
| http://www.offbyzero.com/resources/debugging_rails_rdebug | |
| b) print messages: | |
| $stderr.puts | |
| c) can run the test directly; e.g., | |
| ruby test/colo_test.rb | |
| d) enable verbose output: | |
| rake test TESTOPTS="-v" | |
| 29. git: Show entirety of a file at a particular change (hash): | |
| git show <hash>:<path> | |
| 30. Generate SSH thumbprint from private key: | |
| ssh-keygen -lf <keyfile> | |
| 31. If you notice inconsistencies with order of output of "ls" cmds for files on different hosts, it is likely due to inconsistent locale settings. | |
| 32. If a username is longer than 8 characters, then "ps aux" will substitute the uid for username. | |
| Explanation for why "ps aux" shows a UID instead of the name touch_deploy | |
| 33. If linux is hanging after the "SELinux: Disabled at runtime" prompt, you probably have bad entries in /etc/fstab: | |
| http://www.spamstopshere.com/blog/2008/07/21/linux-hangs-at-selinux-disabled-at-runtime/ | |
| 34. Connecting to someone's screen session in readonly mode: | |
| 35. Executing bash aliases in remote SSH cmd: | |
| bash -ic <cmd> | |
| http://stackoverflow.com/questions/1198378/ssh-command-execution-doesnt-consider-bashrc-bash-login-ssh-rc | |
| for h in {foo,bar,baz}; do echo -n "$h: " ; ssh -T $h bash -ic hostclass; done | |
| 36. Screwed up terminal (e.g., missing cursor, cat'ed a binary file and screen is full of gibberish): | |
| reset | |
| 38. Delete deferred mail from postfix (e.g., on our nagios hosts) | |
| [root@nagios1 log]# postsuper -d ALL | |
| postsuper: Deleted: 22718 messages | |
| 39. Restart network interface in gentoo: | |
| sudo /etc/init.d/net.eth1 restart | |
| 40. Checking your current "run level" in linux: | |
| /sbin/runlevel | |
| who -r | |
| 41. Reverse the lines in a file | |
| tail -r | |
| 42. Running sar | |
| sar -q -f /var/log/sa/sa25 | |
| (figure out which file from the file timestamp) | |
| 43. Escape shell meta characters (from my-vip-flip on bzr) | |
| # Escape shell meta-characters | |
| for(my $i = 0; $i <= $#args; $i++) { | |
| $args[$i] =~ s|([^\w/:\-])|\\$1|g; | |
| } | |
| 44. top w/ resident memory in bytes: | |
| top -o res -b | |
| 45. disable/enable crontabs for current user: | |
| VISUAL="/bin/sed -i 's/^/# /' \$1" crontab -e | |
| re-enable crontabs: | |
| VISUAL="/bin/sed -i 's/^# //' \$1" crontab -e | |
| 46. Checking whether a URL is marked for CDN caching (by going to the origin server): | |
| not cachable: | |
| curl -I "https://origin.foobar.com" | |
| returns header: Cache-Control: private, max-age=0, must-revalidate | |
| cachable by CDN: | |
| curl -I "https://origin.foobar.com" | |
| returns header: Cache-Control: max-age=3354, public | |
| 47. fuser is akin to lsof | |
| 48. dmidecode | |
| e.g., sudo dmidecode -s system-serial-number | |
| 49. Use SOCKS proxy to upload over SSH: | |
| curl --upload-file $1 --socks5-hostname CONDUITHOST:1080 http://config/package/ | |
| 50. single user mode: | |
| select kernel in grub list | |
| e | |
| edit kernel line, put "single" at end | |
| b | |
| 51. cat from stdin into a new file: | |
| % cat - > /tmp/foo.txt | |
| <paste> | |
| control-D (EOF) | |
| 52. git: see file at specific revision: | |
| git show e4f434489529854064b5818d4a728806f577326d:./haproxy.cfg.erb | diff - haproxy_rewards.cfg.erb | |
| 53. kill -QUIT <pid> | |
| Get Java thread stack trace from running process | |
| 54. figlet - creating big ASCII text words on the console - ASCII art | |
| figlet -I2 gives directory where fonts are available | |
| e.g., /usr/local/Cellar/figlet/2.2.4/share/figlet/fonts | |
| 55. strace on Mac: dtruss -f | |
| NOTE: didn't work well for capistrano commands | |
| 56. expanding shell syntax for HOSTS | |
| cap roll HOSTS=`echo foo-app{2..5}.colo | tr ' ' ','` MAX_HOSTS=6 | |
| ==> use commify: | |
| 57. ERB syntax checking | |
| cat haproxy_config/config/templates/haproxy.cfg.erb | erb -x -T - | ruby -c | |
| 58. Shell globbing to exclude files: | |
| enable shell opt "extglob" | |
| ls hosts/!(pci*) | |
| 59. Teeing output to a file: | |
| command 2>&1 | tee logfile | |
| 60. Collaborating remotely: | |
| screen | |
| skype | |
| gotomeeting | |
| http://collabedit.com/ | |
| https://github.com/Pita/etherpad-lite | |
| 61. Viewing only DNS A requests and responses via a Wireshark view filter: | |
| dns.qry.type == A or dns.resp.type | |
| dns.qry.type == A or (dns.resp.type and udp.length == 143) | |
| (check udp.length against the expected length of DNS responses (avoids inclusion of other responses like AAAA)) | |
| 62. Network just totally broken in Mac, try these things: | |
| * Turned the wireless adapter off and on repeatedly | |
| * Remove /Library/Preferences/SystemConfiguration/com.apple.airport.preferences.plist. | |
| * Create a new Location in Network Preferences. | |
| * If specific networks cannot be joined, find them in Keychain Access, and remove them. | |
| 63. mpstat -P ALL 2 | |
| 64. netstat -su | |
| 65. cat /proc/interrupts | |
| 66. cat /proc/net/snmp | |
| 67. sar: options to enable: | |
| -S INT | |
| -S DISK | |
| -S XDISK | |
| -S SNMP (Note that IPv4 statistics depend on sadc option "-S SNMP" to be collected. The following values are displayed (formal SNMP names between square brackets):) | |
| 68. Patches w/ git: | |
| git diff --no-prefix > patchfile | |
| patch -p0 < patchfile | |
| git diff > patchfile | |
| patch -p1 < patchfile | |
| git diff > patchfile | |
| git apply patchfile | |
| git format-patch master --stdout > fix_empty_poster.patch | |
| 69. git delete tag | |
| git tag -d ops-ns_public_config-2012.02.27_15.46 | |
| git push origin :refs/tags/ops-ns_public_config-2012.02.27_15.46 | |
| 70. invoking cmds thru cap | |
| HOSTS=baz COMMAND=hostname cap rolled tat invoke | |
| HOSTS=foo,bar COMMAND=whoami cap invoke | |
| 71. Loopback mounting an ISO: | |
| mount -t iso9660 source.iso -o loop /mnt/iso_files | |
| 72. Equivalent curl & wget cmds for reading a file quietly: | |
| wget -qO- URL | |
| curl -s URL | |
| 73. Getting "alias" cmds to work over remote SSH | |
| ???? | |
| 74. Mac "Open" dialog shortcuts: | |
| Cmd-/ | |
| Allows typing path (e.g., ~/.ssh/config) | |
| Cmd-Shift-. | |
| Toggles hidden directories | |
| 75. Finding and chowning files owned by a specific user | |
| # you could use something like "find / -ls | grep someuser" to identify all of that user's files | |
| chown -R someuser: /some/dir | |
| # This can be a way to churn through the files if there are a lot or you don't know exactly which files are supposed to be owned by that user: | |
| find <path> -user someuser -exec chown someuser: {} \; | |
| 76. Convert unix timestamp into human readable time: | |
| function unixtime() { | |
| perl -le "print scalar localtime $1" | |
| } | |
| 77. Check ruby process size (e.g., w/in test) | |
| pid, size = `ps ax -o pid,rss | grep -E "^[[:space:]]*#{Process::pid}"`.chomp.split(/\s+/).map {|s| s.strip.to_i} | |
| assert(false, "size: #{size}") | |
| 78. Install ack (ack-grep) on mac: | |
| http://mdskinner.com/resources/how-install-ack-mac | |
| 79. Tool for obtaining info about a server's underlying hardware from SMBIOS: | |
| dmidecode | |
| http://en.wikipedia.org/wiki/Desktop_Management_Interface | |
| http://en.wikipedia.org/wiki/System_Management_BIOS | |
| 80. Shell locking: | |
| #!/bin/sh | |
| LOCKFILE=/var/tmp/s3sync.lock | |
| exec 200>$LOCKFILE | |
| if flock --exclusive --nonblock 200; then | |
| echo "Got lock. Do something." | |
| else | |
| echo "Another instance is already running. Exiting." | |
| fi | |
| 81. getting mtime of a file from unix cmd line: | |
| stat -c %y filename | |
| OR | |
| # Linux version of mtime (based on GNU stat) | |
| function mtime() { | |
| perl -e "print scalar(localtime(`stat -c %Y $1`)) . \"\n\"" | |
| } | |
| OR | |
| # Mac version of mtime (based on BSD stat) | |
| function mtime() { | |
| perl -e "print scalar(localtime(`stat -f %m $1`)) . \"\n\"" | |
| } | |
| 82. Secure erasing data: | |
| ATA has a SECURE ERASE option | |
| http://code.google.com/p/diskscrub/ | |
| http://www.dban.org/ | |
| 83. Synchronize 2 local directories using rsync (from 1 to another): | |
| rsync -avz --stats --exclude .git ~/dev/ops-tools-dns-*/ ~/dev/ops-tools-2 | |
| 84. A10 Networks: disable pager ('more' behavior): | |
| terminal length 0 | |
| 85. | |
| cut -f1-6,8- filename | |
| 86. VIM: change inside containment characters (e.g., #{afdafdasfd}) | |
| inside the middle of the containment chars, do: | |
| ci{ | |
| ({ being the containment char) | |
| Similarly, to delete everything inside containment characters, you can switch 'c' with 'd'; e.g., | |
| di{ | |
| 87. Delete photos from iPhone using Mac Application: | |
| Hook up phone to computer using USB cable | |
| Use ImageCapture application to select photos to delete | |
| Click Red X Icon at bottom left to delete | |
| 88. Deploy specific git revision via capistrano: | |
| cap $target deploy -s revision=$sha1 | |
| 89. | |
| curl ifconfig.me | |
| curl ifconfig.me/all/json | |
| {"connection":"","ip_addr":"173.241.26.84","lang":"","remote_host":"","user_agent":"curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5","charset":"","port":"59034","via":"","forwarded":"","mime":"*/*","keep_alive":"","encoding":""} | |
| 90. Delete deferred postfix emails: | |
| sudo postsuper -d ALL deferred | |
| (Useful when emails continue to trickle out of a host, e.g., from cronjob problems) | |
| 91. Check expiration date of cert | |
| openssl x509 -in cert -noout -enddate | |
| 92. ss -l | |
| listening sockets (like "netstat -plan | grep -i listen") | |
| 93. | |
| sed -i "s/usernameN.local usernameN/$etc_hosts_data/g" /etc/hosts | |
| 94. Hand edit /etc/passwd & /etc/group | |
| vipw | |
| vigr | |
| 96. TODO: string intersection between files via grep (i.e., multiple-grep) | |
| 97. Copy all files containing a specific string: | |
| grep -l 'elivered.*foobar' * | xargs -I {} cp {} /tmp/outputdir/. | |
| 98. git visual tool: gitk | |
| 99. Query DKIM record from DNS: | |
| dig +noall +answer TXT DKIMDNSHOSTNAME. | |
| 100. Find WiFi password stored on a Mac: | |
| Go to Applications --> Utilities --> Keychain Access. | |
| Search for your wifi network name. Double click the entry that pops up. Click on the show password box. | |
| 101. Set operations for string files on unix CLI | |
| http://www.scribd.com/doc/8643762/Set-Operations-in-Unix-Shell | |
| e.g., | |
| set union: sort set1 set2 | uniq | |
| set difference: sort set1 set2 | uniq -u | |
| set intersection: sort set1 set2 | uniq -d | |
| 102. Get total space used for a set of files: | |
| du -shc file1 file2 file3 | tail -n 1 | |
| 103. VIM: delete/change from current prompt up to another character | |
| df<somecharacter> | |
| dt<somecharacter> | |
| (dt —> delete til) | |
| (df —> delete thru ... or "fru" | |
| 104. htop | |
| (a) configuration file (~/.htoprc) | |
| (b) tips: | |
| http://www.thegeekstuff.com/2011/09/linux-htop-examples/ | |
| http://www.freesoftwaremagazine.com/articles/htop_tip_top_ncurses_interactive_tool_system_monitoring_your_desktop | |
| 105. SQL join explanations: | |
| http://www.codinghorror.com/blog/2007/10/a-visual-explanation-of-sql-joins.html | |
| 106. Safari tab selection (making it suck less): | |
| https://github.com/rs/SafariTabSwitching | |
| 107. Better Touch Tool for Mac to allow snapping and auto-resizing of windows: | |
| http://blog.boastr.net/?page_id=1722 | |
| 108. UTC Menu Clock: | |
| https://github.com/netik/UTCMenuClock | |
| NOTE: I have a modified version from Andrew Ho that has no ticking-seconds: it's in my dropbox account, in the "Mac Programs" folder | |
| 109. Enable Safari Debug menu: | |
| (Terminal)# defaults write com.apple.Safari IncludeInternalDebugMenu 1 | |
| Then relaunch safari | |
| 110. Function in Google Docs Spreadsheet (using IF with AND): | |
| =IF(AND(B2 = "snake", C2 <> "yes"), "YES", "no") | |
| 111. Search in Mac Finder: default to "this folder", instead of "this mac": | |
| http://superuser.com/questions/365731/finder-how-to-make-default-search-in-folder-the-current-folder | |
| 112. Do not put % into cron lines; e.g., | |
| $(($RANDOM\%30)) | |
| 113. Handle gigantic dirs (tons of contained files) | |
| http://www.olark.com/spw/2011/08/you-can-list-a-directory-with-8-million-files-but-not-with-ls/ | |
| 114. Typing various chars on Mac: | |
| http://tlt.its.psu.edu/suggestions/international/accents/codemac.html | |
| ¿: shift-option-? | |
| ñ: alt-n,n | |
| 115. yawk trick: avoid lots of if logic with rescue | |
| Perhaps this is super obvious to everybody but me, but this empty rescue statement saved me a ton of "if" logic, which is ugly when concatenated together in a one-liner: | |
| bin/yawk 'begin puts "#{d.hostname}.#{d.colo}: " + d.monitors.iostat.shell_command; rescue; end' hosts/* | |
| Or: "Show me the hostname.colo and shell command of all hosts which have overridden the iostat monitor from their hostclass" | |
| This can be shortened even further by using the "inline rescue"... | |
| bin/yawk 'puts "#{d.hostname}.#{d.colo}: " + d.monitors.iostat.shell_command rescue nil' hosts/* | |
| 116. To edit a cmdline string directly in vim | |
| ctrl-x ctrl-e | |
| EDITOR=vim | |
| 117. Apache: disable keepalives (to avoid requiring clients to send "Connection: close" HTTP header) | |
| http://httpd.apache.org/docs/2.2/mod/core.html#keepalive | |
| 118. Looking for logs in Mac? Console.app is your friend. | |
| 119. Ruby Gem File inclusion failure: | |
| require 'net/ssh' not working for ya? How about require 'whatever'? | |
| Could be that you need to require 'rubygems' first! | |
| http://docs.rubygems.org/read/chapter/3 | |
| http://guides.rubygems.org/what-is-a-gem/ | |
| 120. generating email and sending over SMTP | |
| swaks: swiss army knife smtp | |
| e.g., ~/bin/swaks --to someone@example.com --server SMTPHOSTNAME --port 2400 | |
| smtp-source: send crafted smtp messages (part of postfix) | |
| xtress: load testing suite for smtp | |
| 121. Get vi-mode in bash to work better: | |
| http://www.jukie.net/bart/blog/20040326082602 | |
| # ^l clear screen | |
| bind -m vi-insert "\C-l":clear-screen | |
| # allow all readline clients to use vi-mode too | |
| set editing-mode vi | |
| set keymap vi | |
| set convert-meta on | |
| 122. Gmail slowness? | |
| * disable/remove boomerang browser add-on | |
| * disable the "3rd party access" authorization within your account for b4g.baydin.com (that's boomerang for gmail's domain): | |
| http://support.google.com/accounts/bin/answer.py?hl=en&answer=41236 | |
| * disable ALL mail applications that were using IMAP to access gmail. For me that meant Mail.app on MacBook, iPad, iPhone. | |
| For iPad, iPhone, just did this: | |
| Settings > Mail, Contacts, Calendars > myaccountname > Mail > Off | |
| 123. pdsh : distributed ssh, executes cmd on multiple hosts | |
| pdsh -w foohost[1-6].colo df -h | |
| 124. Finding memory hog proceses: | |
| watch -n 30 top -c -o res -b >> mem.log | |
| 125. Clear / delete / remove IP address completely from interface | |
| So, if you have a pre-existing IP config on an interface, say eth1, and then you just remove that config, the | |
| IP config will linger on (visible in output of both ifconfig && ip addr), even though there is no ifcfg-eth1 script in | |
| /etc/sysconfig/network-scripts | |
| To fully remove the IP address from the interface, it is insufficient to just restart networking, | |
| nor can you just down and up the interface. Nor can "ifconfig del" help you. | |
| You need to use "ip addr del", or you need to set the IP to 0.0.0.0: | |
| http://superuser.com/questions/153559/clear-ip-address-of-ethernet-interface | |
| i.e., | |
| ip addr del 1.2.3.4 dev eth1 | |
| ifconfig eth1 0.0.0.0 | |
| 126. du and df being out of sync: | |
| Explanation: | |
| du will not show space used by files that are deleted from the filesystem (e.g., with rm) | |
| but are still being held by a process, since they are not reachable thru the directory | |
| tree anymore. But the space is not reclaimed by the filesystem until all references to a | |
| file are gone. | |
| df will still show the space as being used until the references are gone. | |
| Find the dangling references: | |
| sudo lsof | grep '(deleted)' | |
| 127. Getting cmd line text into Mac's paste buffer: | |
| echo "foo" | pbcopy | |
| 128. Make Safari less sucky: | |
| http://www.machangout.com | |
| 129. getting info on what the CPU/IO/memory is doing: | |
| mpstat -P ALL 1 | |
| iostat -mtx 1 | |
| vmstat 1 | |
| htop | |
| 130. split CSV list into separate lines with line-number prefixes: | |
| % echo 'foo,bar,baz' | tr ',' '\n' | nl -ba -s') ' | |
| 1) foo | |
| 2) bar | |
| 3) baz | |
| 131. working with git branches: | |
| http://stackoverflow.com/questions/1519006/git-how-to-create-remote-branch | |
| http://sandofsky.com/blog/git-workflow.html | |
| 132. paste cmd to combine multiple lines | |
| echo "foo\nbar" | paste -s -d' ' - | |
| 133. use awk as the opposite of paste cmd; i.e., to rip a each line from a file apart into multiple lines | |
| echo 'email-msys1 10.0.0.1' | awk '{print "- address: " $2":2525", "\n name: " $1}' | |
| 134. storing text records in DNS that are greater than 512 bytes | |
| 20120113._domainkey.google.com. 86400 IN TXT "k=rsa\; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAp5kQ31/aZDreQqR9/ikNe00ywRvZBFHod6dja+Xdui4C1y8SVrkUMQQLOO49UA+ROm4evxAru5nGPbSl7WJzyGLl0z8Lt+qjGSa3+qxf4ZhDQ2chLS+2g0Nnzi6coUpF8r" "juvuWHWXnzpvLxE5TQdfgp8yziNWUqCXG/LBbgeGqCIpaQjlaA6GtPbJbh0jl1NcQLqrOmc2Kj2urNJAW+UPehVGzHal3bCtnNz55sajugRps1rO8lYdPamQjLEJhwaEg6/E50m58BVVdK3KHvQzrQBwfvm99mHLALJqkFHnhyKARLQf8tQMy8wVtIwY2vOUwwJxt3e0KcIX6NtnjSSwIDAQAB" | |
| 135. Find which filesystem a file or directory resides within: | |
| [root@hostname log]# df -PTh /some/dir/log | |
| Filesystem Type Size Used Avail Use% Mounted on | |
| /dev/sda3 ext3 48G 42G 3.0G 94% / | |
| [root@hostname /]# df /some/dir/log | |
| Filesystem 1K-blocks Used Available Use% Mounted on | |
| /dev/sdb1 1133279828 536728136 538055816 50% /data | |
| 136. To get all git tags | |
| git fetch --tags | |
| git pull does not get all tags but only those reachable from the current branch heads. However, git pull --tags fetches all tags and is apparently equivalent to git fetch --tags. | |
| 137. Calculate time difference with respect to timezones | |
| http://www.wolframalpha.com/input/?i=13+hours+54+minutes+before+4%3A22AM | |
| 138. Run repeated DNS queries 10 times a second, saving the timing and result to a file & stdout using tee: | |
| while true; do date && (time dig @10.20.253.33 A DNSHOSTNAME +short +timeout=30) 2>&1 | tee -a dns.log; sleep 0.1; done | |
| 139. Use dig to perform reverse DNS lookup of 50.115.214.1: | |
| % dig +short PTR 1.214.115.50.in-addr.arpa | |
| mta1l1.r.grouponmail.co.uk. | |
| 140. Strace | |
| sudo strace -tt -s 1024 -o strace.log -ffv -x -p 16802 | |
| -x ==> print all non-ASCII in hex format | |
| 141. File where network preferences are stored on Mac (as of Lion): | |
| /Library/Preferences/SystemConfiguration/com.apple.airport.preferences.plist | |
| 140. Join all lines in a file | |
| cat file | tr -d '\n' | |
| 141. DKIM validator tool | |
| https://9vx.org/~dho/dkim_validate.php | |
| 142. Convert unix timestamp to human-readable time: | |
| date -d @1360370800 | |
| 143. | |
| curl --socks5-hostname conduit:1084 http://grapher/dashboard/144 | |
| 144. Does not exist: DNE: ¬∃ | |
| 145. Named / bind debugging: | |
| sudo rndc dumpdb -all | |
| 146. putting process in background: | |
| <some-cmd> | |
| control-z | |
| bg | |
| disown -h | |
| 147. find files of at least X bytes or at most X bytes (example use 4KB): | |
| at least: | |
| find . -type f -size +4096c | |
| at most: | |
| find . -type f -size -4096c | |
| 148. list all files with contents from a dir | |
| find . -type f -print | while read filename; do echo -n "$filename: " ; cat "$filename" ; done | |
| 149. how to daemonize a process: | |
| http://www.itp.uzh.ch/~dpotter/howto/daemonize | |
| To properly daemonize, the following steps must be followed. | |
| The fork() call is used to create a separate process. | |
| The setsid() call is used to detach the process from the parent (normally a shell). | |
| The file mask should be reset. | |
| The current directory should be changed to something benign. | |
| The standard files (stdin,stdout and stderr) need to be reopened. | |
| Failure to do any of these steps will lead to a daemon process that can misbehave. The typical symptoms are as follows. | |
| Starting the daemon and then logging out will cause the terminal to hang. This is particularly nasty with ssh. | |
| The directory from which the daemon was launched remains locked. | |
| Spurious output appears in the shell from which the daemon was started. | |
| 150. Frank's compulsory guide to postal addresses: | |
| http://www.columbia.edu/~fdc/postal/ | |
| Lists address formats for most (all?) countries | |
| 151. calling shell command (e.g., foo) directly without aliases or functions: | |
| command foo | |
| 152. problems with safari & fuzebox | |
| Finder > Go > Hold Option & Click Library | |
| Search for "safari" remove all of the plist files | |
| 153. pkill without pkill: # From: http://en.wikipedia.org/wiki/Kill_(command) | |
| kill `ps --no-headers -C firefox -o pid` | |
| 154. stop using "bundle exec" | |
| https://github.com/mpapis/rubygems-bundler | |
| 155. too many sockets in CLOSE_WAIT state (e.g., in unicorn) | |
| $ sudo lsof | grep CLOSE_WAIT | wc -l | |
| 27405 | |
| What is the other end of the CLOSE_WAIT sockets? | |
| Could you paste a sampling of "sudo netstat -alnp | grep CLOSE_WAIT | grep unicorn"? | |
| Does the situation change if you disable keepalive at the nginx or app level? (send "Connection: close" header) | |
| 156. What process string do pkill & pgrep & killall search through? | |
| http://askubuntu.com/questions/27501/whats-the-difference-between-killall-and-pkill | |
| By default: | |
| /proc/<PID>/stat | |
| If you use "-f" with pkill & pgrep, then it looks through: | |
| If you use " | |
| /proc/cmdline | |
| 157. Comparing version numbers in ruby | |
| http://stackoverflow.com/questions/2051229/how-to-compare-versions-in-ruby | |
| Gem::Version.new('0.4.1') > Gem::Version.new('0.10.1') | |
| richer: https://github.com/dazuma/versionomy | |
| 158. Use tcp device to make HTTP requests in the shell, just using exec, echo, & cat | |
| exec 3<>/dev/tcp/www.groupon.com/80 && echo -e "GET / HTTP/1.1\r\nhost: http://www.groupon.com\r\nConnection: close\r\n\r\n" >&3 && cat <&3 | |
| 159. Use iftop with tcpdump filters to find if traffic has stopped to a service | |
| sudo iftop -P -N -f 'dst portrange 5000-5049' | |
| 160. mou --> good for markdown wysiwyg editing, and then you can use pandoc to convert to mediawiki | |
| 161. Mac Top Menu Bar frozen? | |
| killall SystemUIServer | |
| http://apple.stackexchange.com/questions/39219/what-should-i-do-when-the-menu-bar-seems-frozen-unresponsive | |
| 162. Tail a file that hasn't yet been created: | |
| tail --follow=name <filename> | |
| 163. Determining which processes have various TCP connections on Mac (since "netstat -pan" fails) | |
| lsof -n -i4TCP -P | |
| 164. memcache top | |
| mctop | |
| 165. prevent logins | |
| sudo touch /etc/nologin | |
| (Actually, need to double check that, I think it varies greatly between BSD and Linux) | |
| 166. Shell enhancements for Git (zsh, bash): | |
| http://nuclearsquid.com/writings/git-tricks-tips-workflows/ | |
| http://stackoverflow.com/questions/1128496/to-get-a-prompt-which-indicates-git-branch-in-zsh | |
| http://dmytro.github.io/2012/11/27/unobtrusive-git-prompt.html | |
| http://stackoverflow.com/questions/6623649/disable-auto-completion-of-remote-branches-in-git-bash/ | |
| http://stackoverflow.com/questions/12175277/disable-auto-completion-of-remote-branches-in-zsh | |
| 167. Make yourself admin on a Mac (Mountain Lion) | |
| I just got a new MBAir from IT and they didn't set my account up as a computer admin. This made it so I couldn't change security settings (like FileVault), set up the VPN, or install certain software. | |
| The best solution is to have IT set you up as an admin, but if that's not an option (I'm working from home today), here's how to do it: | |
| 1) shut down the computer | |
| 2) boot into single-user mode by holding down CMD-s as you power on the computer | |
| 3) when it's done booting, run these commands: | |
| # fsck -fy | |
| # mount -uw / | |
| # launchctl load /System/Library/LaunchDaemons/com.apple.opendirectoryd.plist | |
| # sudo dseditgroup -o edit -a yourlogin -t user admin | |
| # sudo dseditgroup -o edit -a yourlogin -t user wheel | |
| # reboot | |
| 4) When the machine is done booting, log in and now you're an admin! | |
| http://steinkamp.us/934/single-user-admin-on-os-x-mountain-lion | |
| 168. Get your host's public-visible NAT IP | |
| # curl icanhazip.com | |
| # wget -q -O- icanhazip.com | |
| 169. Avoid clearing/polluting disk cache with a periodic process that scans a filesystem | |
| nocache ionice -c3 nice long_running.sh | |
| http://packages.debian.org/sid/nocache | |
| 170. Avoid data changes being synced to disk | |
| eatmydata | |
| http://packages.debian.org/sid/eatmydata | |
| 171. Set default shell in mac: | |
| chsh -s /bin/zsh | |
| chsh -s /usr/local/bin/fish # http://fishshell.com/ | |
| 172. Combine output from 2+ cmds into 1 stream by using a sub-shell which concatenates output into 1 stream: | |
| (foo; bar;) | baz | |
| Tests of it not being interleaved: | |
| (yes foo | head -n 50000; yes bar | head -n 50000;) > /tmp/5 | |
| (seq 0 99999; seq 1000000 1099999) > /tmp/5 | |
| 173. The "suck" effect for minimizing a window on OS X | |
| You can choose between the "scale" or "genie" effect when you minimize a window on your Mac. But there's a secret third choice called "suck." | |
| Start up Terminal and type "defaults write com.apple.dock mineffect -string suck" and restart your computer. The new effect is activated. | |
| Read more: http://www.businessinsider.com/mac-os-x-easter-eggs-2013-4?op=1#ixzz2n14Nkjf6 | |
| 174. copy and paste issues on Mac | |
| By default: command-shift-v : Paste and Match Style | |
| Can supposedly change to command-v doing "Paste and Match Style" by doing this: | |
| defaults write .GlobalPreferences -dict-add NSUserKeyEquivalents "Paste and Match Style" -string "@v" | |
| http://www.macstories.net/tutorials/customize-your-macs-paste-and-match-style/ | |
| 175. Emoji unicode characters that render as crossed out mushroom: | |
| http://graphemica.com/search?q=%F0%9F%8D%84%E2%83%A0 | |
| 176. Get first and last lines from the output of a command | |
| (curl -s -i http://localhost:8999/heartbeat.txt | tee /dev/fd/3 | head -n 1) 3> >(tail -n 1) | |
| http://stackoverflow.com/questions/8624669/unix-head-and-tail-of-file | |
| 177. Sound an alert from a terminal (causes beep sound and puts a bell icon in Terminal.app tab title) | |
| printf '\7' | |
| 178. Smart Questions: follow-up with solution: | |
| http://www.catb.org/~esr/faqs/smart-questions.html#followup | |
| 179. Install specific version of something in brew (e.g., to downgrade git): | |
| http://stackoverflow.com/questions/3987683/homebrew-install-specific-version-of-formula | |
| 180. cmds that go through pager have escape codes instead of colors | |
| http://unix.stackexchange.com/questions/64927/git-diff-displays-colors-incorrectly | |
| git config --global core.pager 'less -R' | |
| 181. slabtop | |
| ``` | |
| slabtop | |
| ``` | |
| Figuring out what "kernel slab cache" memory is being used for. | |
| http://www.toofishes.net/blog/linux-command-day-slabtop/ | |
| 182. Get a file's rename history in git | |
| ``` | |
| git log --format='%H%%' --name-only --follow -- somefile/in/your/git/repo | gsed ':a;N;$!ba;s/%\n\n/ /g' | awk '{print $2}' | uniq | |
| ``` | |
| 183. Work with JSON on the command line. | |
| ``` | |
| jq | |
| # e.g., sort the keys | |
| jq -S '.' | |
| ``` | |
| http://stedolan.github.io/jq/tutorial/ | |
| 184. Run previous command with sudo | |
| ``` | |
| sudo !! | |
| ``` | |
| 185. Substitute in previous command's final argument | |
| ``` | |
| !$ | |
| ``` | |
| 186. Get the number of connections to a port; e.g., 2181 | |
| ``` | |
| % ss -a -t -n state connected src :2181 | wc -l | |
| 37 | |
| ``` | |
| 187. Bash shell expansion | |
| http://wiki.bash-hackers.org/syntax/expansion/brace | |
| ``` | |
| foo{1..15} | |
| foo[1-9] # cannot do foo[1-15], since that expands to foo1,foo5; i.e., the square bracket syntax is useful only with single characters | |
| ``` | |
| 188. echo -n is bad emkay, use printf "%s" instead | |
| https://github.com/git/git/commit/19c3c5fdcb35b66b792534c5dc4e8d87a3952d2a | |
| POSIX-compatible, portable method for printing a string without a newline is: | |
| ``` | |
| printf "%s" "text" | |
| ``` | |
| instead of | |
| ``` | |
| echo -n "text" | |
| ``` | |
| 189. github / git tips and tricks | |
| http://owenou.com/2012/01/13/ten-things-you-didnt-know-git-and-github-could-do.html | |
| 190. Show definition of a shell alias or function: | |
| zsh: | |
| ``` | |
| type some-alias | |
| whence -f some-alias | |
| ``` | |
| bash: | |
| ``` | |
| type some-alias | |
| ``` | |
| 191. Curl cmd that provides detailed timing: | |
| when you test it again, perhaps try this curl command which provide | |
| more timing details: | |
| ``` | |
| curl -o /dev/null -s -w | |
| "time_namelookup: %{time_namelookup}\ntime_connect: | |
| %{time_connect}\ntime_appconnect: | |
| %{time_appconnect}\ntime_pretransfer: | |
| %{time_pretransfer}\ntime_redirect: | |
| %{time_redirect}\ntime_starttransfer: | |
| %{time_starttransfer}\ntime_total: %{time_total}\n" | |
| http://somehost/ | |
| ``` | |
| 192. Git: reverting a commit | |
| http://stackoverflow.com/questions/4114095/revert-to-a-previous-git-commit | |
| 193. Get age of a process | |
| ``` | |
| stat /proc/$pid | |
| ``` | |
| 194. Weird behavior of exempting "grep" from process searching: | |
| ``` | |
| % ps aux | grep [/]bin/bash | grep grep | |
| % ps aux | grep /bin/bas[h] | grep grep | |
| 1226 22256 0.0 0.0 61192 828 pts/20 S+ 01:59 0:00 grep /bin/bash | |
| ``` | |
| The [] is treated as a shell glob initially, with the shell trying to | |
| expand the argument to the full matching file name (which /bin/bash *is*). | |
| http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_04.html#sect_03_04_08 | |
| 195. Make colluquy shut up about joined/left messages | |
| http://colloquy.info/project/wiki/FAQs#Doescolloquyhaveawaytoignorehidepartsnicksetcinabusyroom | |
| http://wiki.xkcd.com/irc/Hide_join_part_messages#Colloquy | |
| 196. Controlling tabs in Mac Terminal | |
| http://superuser.com/a/176181/261282 | |
| http://www.culater.net/software/SIMBL/SIMBL.php | |
| 197. Figure out which java thread (in jstack) a Linux Thread corresponds to | |
| http://www.semicomplete.com/blog/geekery/debugging-java-performance.html | |
| jstack PID > stacktrace | |
| # have PID for Linux thread you care about (e.g., from running strace), convert that to hex: | |
| % printf "0x%x\n" 28993 | |
| 0x7141 | |
| % grep nid=0x7141 stacktrace | |
| "VM Thread" prio=10 tid=0x000000005e6d9000 nid=0x7141 runnable | |
| 198. Find which package a file in centos/redhat came from: | |
| rpm -qf <filename> | |
| 199. Create file with random bits of size X | |
| dd if=/dev/urandom of=2kB.rand bs=1K count=2 | |
| 200. Latency numbers of various I/O operations | |
| https://gist.github.com/jboner/2841832 | |
| Latency Comparison Numbers | |
| -------------------------- | |
| L1 cache reference 0.5 ns | |
| Branch mispredict 5 ns | |
| L2 cache reference 7 ns 14x L1 cache | |
| Mutex lock/unlock 25 ns | |
| Main memory reference 100 ns 20x L2 cache, 200x L1 cache | |
| Compress 1K bytes with Zippy 3,000 ns | |
| Send 1K bytes over 1 Gbps network 10,000 ns 0.01 ms | |
| Read 4K randomly from SSD* 150,000 ns 0.15 ms | |
| Read 1 MB sequentially from memory 250,000 ns 0.25 ms | |
| Round trip within same datacenter 500,000 ns 0.5 ms | |
| Read 1 MB sequentially from SSD* 1,000,000 ns 1 ms 4X memory | |
| Disk seek 10,000,000 ns 10 ms 20x datacenter roundtrip | |
| Read 1 MB sequentially from disk 20,000,000 ns 20 ms 80x memory, 20X SSD | |
| Send packet CA->Netherlands->CA 150,000,000 ns 150 ms | |
| Notes | |
| ----- | |
| 1 ns = 10-9 seconds | |
| 1 ms = 10-3 seconds | |
| * Assuming ~1GB/sec SSD | |
| Credit | |
| ------ | |
| By Jeff Dean: http://research.google.com/people/jeff/ | |
| Originally by Peter Norvig: http://norvig.com/21-days.html#answers | |
| Contributions | |
| ------------- | |
| Some updates from: https://gist.github.com/2843375 | |
| Great 'humanized' comparison version: https://gist.github.com/2843375 | |
| Visual comparison chart: http://i.imgur.com/k0t1e.png | |
| Nice animated presentation of the data: http://prezi.com/pdkvgys-r0y6/latency-numbers-for-programmers-web-development/ | |
| 201. Test for whether connections are open | |
| % for conn in {"foo 22","foo 80","foo 443","bar 9418"}; do nc -z -w1 $conn; if [ "$?" -ne 0 ]; then echo "$conn: FAILED, port not open"; fi; done | |
| Connection to foo 22 port [tcp/ssh] succeeded! | |
| foo 80: FAILED, port not open | |
| Connection to foo 443 port [tcp/https] succeeded! | |
| Connection to bar 9418 port [tcp/git] succeeded! | |
| 202. Running diff of lsof each time you run it | |
| 0 - chi@kafka-hadoop-consumer2:~$ for i in {1..10}; do sudo lsof -p 3800 > /tmp/lsof$((i % 2)) && diff /tmp/lsof0 /tmp/lsof1 >> /tmp/lsof-delta.txt; done | |
| diff: /tmp/lsof0: No such file or directory | |
| 1 - chi@kafka-hadoop-consumer2:~$ tail /tmp/lsof-delta.txt | |
| > java 3800 root *531r FIFO 0,6 0t0 97169587 pipe | |
| > java 3800 root *532w FIFO 0,6 0t0 97169587 pipe | |
| > java 3800 root *533r 0000 0,11 0 97169588 eventpoll | |
| 41542a41552,41553 | |
| > java 3800 root *537w FIFO 0,6 0t0 97172727 pipe | |
| > java 3800 root *538r 0000 0,11 0 97172728 eventpoll | |
| 41544a41556,41558 | |
| > java 3800 root *546r FIFO 0,6 0t0 97170919 pipe | |
| > java 3800 root *547r FIFO 0,6 0t0 97170919 pipe | |
| > java 3800 root *548w 0000 0,11 0 97170920 eventpoll | |
| 203. Stand up a simple HTTP server for serving files: | |
| ``` | |
| python -m SimpleHTTPServer 4004 | |
| # OR | |
| ruby -run -e httpd . -p 5000 | |
| ``` | |
| http://www.linuxjournal.com/content/tech-tip-really-simple-http-server-python | |
| http://www.benjaminoakes.com/2013/09/13/ruby-simple-http-server-minimalist-rake/ | |
| 204. Trim in shell using sed | |
| sed 's/^ *//;s/ *$//' | |
| 205. OS X tweaks | |
| https://www.learningosx.com/101-ways-to-tweak-os-x-using-terminal/ | |
| 206. Cannot put multiple ~ chars inside a single string in a shell and except them to be expanded; e.g., in a Java classpath with multiple jars. | |
| http://stackoverflow.com/questions/9122989/use-javac-with-multiple-specific-jars-in-linux | |
| https://bugs.openjdk.java.net/browse/JDK-4668475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel | |
| 207. Debugging class loading issues | |
| java -verbose:class | |
| http://www.ibm.com/developerworks/library/j-dclp1/ | |
| 208. Find traffic between 1 host and not some others: | |
| sudo iftop -i eth2 -f 'port 2181 and not host foobar and not host baz' | |
| 209. Reading through files with gigantic lines (with many many more characters than fit in your terminal's width) | |
| | less -S | |
| | cut -c1-200 | |
| # Or whatever your terminal's width in characters is | |
| 210. Pastebin CLI cmds for Mac | |
| pbcopy | |
| pbpaste | |
| 211. Take CSV and make it readable | |
| cat foo | column -s, -t | |
| column -s, -t < foo | |
| Alternatives: | |
| 1. $ csvgrep -e iso-8859-1 -c 1 -m "de" worldcitiespop | csvgrep -c 5 -r "\d+" | csvsort -r -c 5 -l | csvcut -c 1,2,4,6 | head -n 11 | csvlook | |
| i.e., from csvkit: https://csvkit.readthedocs.org/en/latest/ | |
| 2. csvtool readable filename | view - | |
| 3. csv plugin for VIM: http://vim.sourceforge.net/scripts/script.php?script_id=2830 | |
| 4. Tabview : https://github.com/firecat53/tabview | |
| 212. bash with various file descriptors using exec | |
| http://www.tldp.org/LDP/abs/html/x17974.html | |
| See LOCKFILE stuff above. | |
| 213. Switch java version | |
| ``` | |
| export JAVA_HOME=$(/usr/libexec/java_home -v1.8) | |
| ``` | |
| 215. Fancy use of /proc | |
| https://blogs.oracle.com/ksplice/entry/solving_problems_with_proc | |
| 216. Lots of good bash stuff: | |
| http://stackoverflow.com/questions/2188199/how-to-use-double-or-single-bracket-parentheses-curly-braces | |
| 217. Building a JSON-based API: | |
| http://jsonapi.org/ | |
| including great stuff for pagination: http://jsonapi.org/format/#fetching-pagination | |
| 218. All about redirects, including `exec` with redirects | |
| http://wiki.bash-hackers.org/howto/redirection_tutorial | |
| https://stackoverflow.com/questions/18351198/what-are-the-uses-of-the-exec-command-in-shell-scripts | |
| 219. Prefer long-form bash redirects | |
| i.e., | |
| Never use the Csh &>foo shorthand redirect. | |
| Use the long form >foo 2>&1. | |
| == | |
| Use the long form 1>foo 2>&1. | |
| 221. Get a process's env: | |
| % sudo sh -c 'xargs --null --max-args=1 < /proc/26672/environ' | |
| LD_LIBRARY_PATH=/usr/local/lib:/usr/local/jre/lib/amd64/server | |
| MESOS_LOG_DIR=/var/logs/mesos | |
| MESOS_CLUSTER=some_cluster_name | |
| MESOS_IP=192.168.0.1 | |
| MESOS_QUORUM=2 | |
| MESOS_WORK_DIR=/var/blah/mesos/work_dir | |
| MESOS_ZK=zk://zookeeperhostname:2181/mesos_dev | |
| 222. Undo last git commit: | |
| git reset --soft HEAD^ | |
| 223. Ansible use of paramiko vs. native OpenSSH | |
| When switching between source hosts for running ansible you might notice differing | |
| behaviors w.r.t. ansible respecting your ~/.ssh/config options. This happens because | |
| ansible (after 1.2) tries to use OpenSSH if it's version 5.6+, and if the version of | |
| OpenSSH is too old then ansible falls back to using paramiko, which doesn't respect | |
| the ~/.ssh/config parameters like OpenSSH does. | |
| Notably, Centos6 doesn't have OpenSSH 5.6+. | |
| To overcome this, add the following section to ~/.ansible.cfg: | |
| [defaults] | |
| host_key_checking = False | |
| http://docs.ansible.com/ansible/intro_getting_started.html#remote-connection-information | |
| http://docs.ansible.com/ansible/intro_getting_started.html#host-key-checking | |
| 224. ZooKeeper: delete nodes recursively in zookeeper-3.3* | |
| https://github.com/davidledwards/zookeeper/tree/master/zookeeper-cli | |
| 225. Capture packets remotely using tcpdump & wireshark: | |
| wireshark -k -i <(ssh REMOTE_HOST "sudo tcpdump -w - port 6557") | |
| 226. Use sed to clean up known_hosts | |
| gsed -i '/staas-worker.*staging/d' ~/.ssh/known_hosts | |
| 227. get CPU usage info for a PID: | |
| ps -p PID -o cputime ; ps -p PID -o etime ; ps -p PID -o %cpu | |
| NOTE: cputime is cumulative across all cores, so can be more than elapsed time (clock time) | |
| 228. List threads owned by monitord | |
| ps u -L -U monitord | |
| 229. Get actual cpu usage; ps's %cpu is cummulative for time process has run: | |
| ps -o%cpu,cputime,etime -p 28038 | |
| i.e., %cpu by default is just etime / cputime | |
| so really, we need to take the difference between 2 snapshots of cputime over a | |
| period of time to get the usage during that period | |
| TODO: write a wrapper for doing that | |
| 230. Vim: substitute newline | |
| use \r instead of \n | |
| https://stackoverflow.com/questions/71323/how-to-replace-a-character-for-a-newline-in-vim | |
| https://stackoverflow.com/questions/71417/why-is-r-a-newline-for-vim | |
| 231. tar pipe: copy a bunch of files quickly over SSH | |
| # PUSH: from this host to another one | |
| tar czf - kernel | ssh REMOTEHOST "cd tmp/ ; tar xzf -" | |
| # PULL: from remote host to this host: | |
| ssh REMOTEHOST "tar czf - kernel" | tar xzf - | |
| # PULL: with root-owned files | |
| ssh REMOTEHOST "cd /etc ; sudo tar czpf - ssh" | sudo tar xzpf - | |
| # where kernel & ssh are dirs with a bunch of files | |
| 232. Mirror Displays on Mac via keyboard shortcut: Command + F1 | |
| 233. Google+ comment cheatsheet: | |
| https://docs.google.com/document/d/1tkKOd0wKsQLKn5Dtkqy8p_DxQ0ikZTXJohzIfw-_gNQ/edit | |
| 234. Bash loops to search for processes with certain environment variables | |
| while true; do sudo sh -c 'strings /proc/*/environ | grep MESOS | grep -v SUDO_COMMAND'; done | |
| while true ; do for p in /proc/[0-9]*; do sudo sh -c "strings $p/environ | grep MESOS && cat $p/cmdline ; strings $p/environ | grep MESOS" ; done; done; | |
| 235. Bash check exit values | |
| http://blog.sanctum.geek.nz/testing-exit-values-bash/ | |
| # Bad practice | |
| grep -q regex options | |
| if (($? > 0)); then | |
| printf '%s\n' 'myscript: Pattern not found!' >&2 | |
| exit 1 | |
| fi | |
| # Better | |
| if grep -q regex options; then | |
| # Do nothing | |
| : | |
| else | |
| printf '%s\n' 'myscript: Pattern not found!\n' >&2 | |
| exit 1 | |
| fi | |
| # Best | |
| if ! grep -q regex options; then | |
| printf '%s\n' 'myscript: Pattern not found!' >&2 | |
| exit 1 | |
| fi | |
| 236. Get Instruction Pointer (32-bit: EIP, 64-bit: RIP) for Linux threads: | |
| ps -eo tid,pid,rip,state -L | |
| 237. Container visibility: sysdig / csysdig | |
| https://github.com/draios/sysdig/wiki/Csysdig-Overview | |
| http://www.sysdig.org/install/ | |
| sudo yum -y update kernel | |
| curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash | |
| 238. Enter container with same PID & mount namespaces as specified PID: | |
| sudo nsenter -p -m -t <PID> | |
| 239. Reset local git branch to be same as remote branch (useful after someone force-pushes): | |
| git reset --hard <remote-repo>/<branch> | |
| e.g., | |
| git reset --hard DarinJ/0.10.0 | |
| https://stackoverflow.com/questions/1628088/reset-local-repository-branch-to-be-just-like-remote-repository-head | |
| 240. Find processes running at a given instant | |
| grep -E '^R.*' /proc/sched_debug | |
| 241. Git tips / scripts | |
| http://nuclearsquid.com/writings/git-tricks-tips-workflows/ | |
| https://about.gitlab.com/2015/02/19/8-tips-to-help-you-work-better-with-git/ | |
| https://wiki.typo3.org/Tips_and_Tricks_%28Git%29 | |
| 242. HEREDOC for pasting file to be processed | |
| % column -t << EOL | |
| heredoc> iface bytes packets | |
| heredoc> lo 86654545 615357 | |
| heredoc> eth0 6096258286 40319326 | |
| heredoc> eth1 6119552036 40693237 | |
| heredoc> bond0 12215810436 81012564 | |
| heredoc> br0 2222175609 11065906 | |
| heredoc> vnet0 15989996953 73543679 | |
| heredoc> EOL | |
| iface bytes packets | |
| lo 86654545 615357 | |
| eth0 6096258286 40319326 | |
| eth1 6119552036 40693237 | |
| bond0 12215810436 81012564 | |
| br0 2222175609 11065906 | |
| vnet0 15989996953 73543679 | |
| https://stackoverflow.com/questions/2500436/how-does-cat-eof-work-in-bash | |
| 243. Use ansible to perform scp with root-privileges | |
| Copy to remote host using sudo | |
| ansible -i HOST, -s -m copy -a "src=SRC dest=DEST" HOST | |
| Copy from remote host using sudo | |
| ansible -i HOST, -s -m fetch -a "src=SRC dest=DEST flat=yes" HOST | |
| 243. Using ansible on localhost | |
| http://ansible.pickle.io/post/86598332429/running-ansible-playbook-in-localhost | |
| 244. Use maven to run *just* the tests, and bail after the 1st error: | |
| mvn surefire:test -DskipAfterFailureCount=1 | |
| # need surefire 2.19+ | |
| 245. When using maven clojure plugin, the test failures are squelched unless you run: | |
| mvn clojure:test | |
| i.e., `mvn test` is insufficient. | |
| http://alexott.net/en/clojure/ClojureMaven.html | |
| http://www.chaomancy.com/blog/clojure-in-eclipse-part-1-maven.html | |
| 246. Rename branch in git: | |
| https://gist.github.com/lttlrck/9628955 | |
| git checkout old_branch # Ensure you are on the to-be-renamed branch | |
| git branch -m new_branch # Rename branch locally | |
| git push origin :old_branch # Delete the old branch | |
| git push --set-upstream origin new_branch # Push the new branch, set local branch to track the new remote | |
| 245. Overriding SSH config per-host for a different SSH private key: | |
| Host 54.177.87.70 50.18.145.148 184.72.37.34 geespot-dev geespot-prod geespot-dev.willinglydumb.com geespot-prod.willinglydumb.com | |
| IdentityFile ~/.ssh/geespot/id_rsa | |
| 246. Clean up logs older than N minutes: | |
| find /some/path/ -type f -mmin +240 -name '*.log' -delete | |
| 247. Clean up all dirs with specific path prefix: | |
| find /tmp -path '/tmp/render-sandbox.*' -delete" | |
| 248. tig: git curses thing kinda like gitk | |
| 249. git : clean up repo | |
| git gc --prune=now | |
| git reflog expire --all --stale-fix | |
| git prune | |
| 250. tcpflow: create separate connections out of pcap | |
| 251. pretty-print json | |
| jq . ~/.vagrant.d/data/machine-index/index | |
| 252. GitHub permalinks to files: | |
| https://help.github.com/articles/getting-permanent-links-to-files/ | |
| type "y" | |
| 253. Understanding signals, jobs, process groups; why Ctrl-C might be different than kill -2 | |
| https://unix.stackexchange.com/questions/45426/why-would-ctrl-c-behave-differently-than-kill-2 | |
| 254. Track all processes | |
| psacct | |
| acct | |
| https://www.gnu.org/software/acct/ | |
| http://www.tecmint.com/how-to-monitor-user-activity-with-psacct-or-acct-tools/ | |
| http://www.tldp.org/HOWTO/text/Process-Accounting | |
| https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#safe=off&q=linux+acct | |
| http://www.dedoimedo.com/computers/linux-cool-hacks.html | |
| 255. Filter logs based on timestamp | |
| https://stackoverflow.com/questions/33000824/filter-linux-logs-based-on-unix-timestamp | |
| 256. Human readable timestamp from logs with unix timestamps | |
| https://www.commandlinefu.com/commands/view/8784/read-squid-logs-with-human-readable-timestamp | |
| 257. Colorize terminal environment | |
| https://unix.stackexchange.com/questions/148/colorizing-your-terminal-and-shell-environment | |
| 258. | |
| IntelliJ IDEA > Preferences > Editor > General > "Change font size (Zoom) with Ctrl+MouseWheel" | |
| 259. ls with full timestamp including milliseconds | |
| ``` | |
| % ls -altr --time-style=full-iso | |
| ``` | |
| 260. Edit paste-buffer contents before pasting into iTerm | |
| Command-Option-v | |
| 261. jq usage with mesos: | |
| curl -o- 'MESOSWORKERHOST:5051/slave(1)/state.json' | jq -C ".frameworks[]" | less -R | |
| 262. 4-finger swipe / gesture for trackpad stops working. | |
| killall Dock | |
| 263. Splunk - less-ish behaviors: | |
| 1. Get nearby events: click on event time - "Nearby Events" + "Events Before or After" | |
| 2. Get events in reverse (original) order: "| reverse" | |
| 264. Bash: [] vs. [[ ]] | |
| https://serverfault.com/questions/52034/what-is-the-difference-between-double-and-single-square-brackets-in-bash | |
| 265. Buffer output per-line | |
| stdbuf -oL -eL | |
| 266. Firefox telemetry info, including histograms | |
| about:telemetry | |
| 267. Same source port held by multiple processes? Probably inherited. | |
| 268. Use pmap when trying to look into JVM memory usage | |
| 269. Get process start time | |
| ps -eo pid,lstart,cmd | grep <PID> | |
| 270. watch of a pipe of commands: 'foo | bar'. Also "ps" output gets cut off, pass "www" | |
| sudo watch -d 'ps auxwww | grep logstas[h]' | |
| 271. Find and delete files that are not held open: | |
| find * ! -exec sudo fuser -s "{}" 2>/dev/null \; -exec echo {} \; | xargs -I{} sudo rm {} | |
| 272. Find and delete files that have not been modified within last hour: | |
| find * -name "strace.log.*" -type f -mmin +60 -delete | |
| 273. Mac video camera stuck on laptop cannot use external display | |
| sudo killall VDCAssistant | |
| 274. Find size of files excluding some dir | |
| sudo find /some/dir -type f -not -path "/some/dir/baz/*" -exec echo {} \; | sort | xargs -I{} sudo sh -c 'du -b {}' > ~/some-dir-except-baz.du | |
| 275. Extract file to stdout from jar | |
| unzip -q -c stormjar.jar META-INF/maven/org.apache.storm/storm-kafka/pom.xml | grep \<version | head -n1 | |
| 276. paste and bc for adding numbers that are in separate lines in a file / stream | |
| cat numberfile | paste -sd+ - | bc | |
| 277. | |
| URL="http://www.google.com" ; while true ; do sleep 5 ; echo -n "$(date) " ; if curl -sSf ${URL} >& /dev/null; then echo "${URL} OK" ; else echo "${URL} FAILED" ; fi ; done | |
| 278. to ensure proper script behavior follow these tips: | |
| For script authors: Every bash script that uses the cd command with a relative path needs | |
| to call unset CDPATH, or else it may not work correctly. Scripts that don’t use cd should | |
| probably do it anyway, in case someone puts a cd in later. | |
| For users: Never export CDPATH from your shell to the environment. If you use CDPATH then | |
| set it in your .bashrc file and don’t export it, so that it’s only set in interactive | |
| shells. | |
| https://issues.apache.org/jira/browse/STORM-2486 | |
| https://bosker.wordpress.com/2012/02/12/bash-scripters-beware-of-the-cdpath/ | |
| # STORM-2486: Prevent `cd` from printing the target directory. | |
| unset CDPATH | |
| 279. Create custom homebrew install | |
| https://github.com/Homebrew/legacy-homebrew/issues/40987 | |
| 280. git diff | |
| % git diff -W 0.9.0.1:storm-core/src/clj/backtype/storm/daemon/common.clj v1.0.6:storm-core/src/clj/org/apache/storm/daemon/common.clj | |
| 281. Enable Picture-in-Picture mode in Mac Sierra for YouTube videos: | |
| Step 1: Right click on the YouTube video that you wish to detach | |
| Step 2: Once the menu appears, right-click once more to reveal a second menu and then click Enter Picture-in-Picture | |
| For normal stuff: | |
| https://support.apple.com/en-us/HT206997 | |
| 282. Show hidden files toggle for Finder | |
| COMMAND + SHIFT + . | |
| 283. Stupid xfinity autologin crap causing my devices to think my xfinity service is the current house's xfinity service instead of my own. | |
| Disable Auto Authentication | |
| https://idm.xfinity.com/myaccount/userprofile | |
| Auto Authentication | |
| Automatically access TV content while connected to Xfinity Internet at home without having to sign in. Visit the FAQ section to learn more. | |
| 284. diff-so-fancy | |
| https://github.com/so-fancy/diff-so-fancy#screenshot | |
| 285. IntelliJ help with shortcuts & commands | |
| Command-Shift-A | |
| 286. Use Preview.app on Mac to join PDF files together | |
| https://support.apple.com/en-us/HT202945 | |
| 287. GitHub Search options for not searching only in the base fork & default (usually master) branch | |
| fork:true | |
| fork:only | |
| 288. Generate a text with an email message (for AT&T) | |
| 7775551212@mms.att.net | |
| 289. Mirror a web site | |
| https://superuser.com/questions/14403/how-can-i-download-an-entire-website | |
| wget -m -p -E -k --mirror "https://example.com/" | |
| For a mediawiki site: | |
| wget --limit-rate=500k -m -p -E -k --mirror --reject "*action=print*","*action=diff*","*action=upload*","*action=edit*" --load-cookies ~/tmp/cookies.txt "https://wiki.somecompany.com/" | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment