Useful bash tricks

useful_bash_tricks.md

1. Filter CSV based on number of words in a field using awk:

awk -F ',' 'split($2,a," ") == 2 {print $2}' fname

2. Convert Kubernetes events to local timezone (for comparison with other logs):

kubectl get events -o json | jq '.items[] | .lastTimestamp + "," + .message' | tr -d '"' | awk -F',' '{cmd="date +\"%d/%b/%Y:%T\" --date=" $1; cmd | getline conv_date; print "[" conv_date "] (kubectl events) " $2}'

3. Remove EOS/BOS from a text data file using sed

sed -r 's!([ ]+)?<(/)?s>([ ]+)?!!g' fname > plain_fname

4. Use curl to test a TF-Serving model (needs inputs key if model has been saved using tf.saved_model.simple_save)

curl -XPOST localhost:15213/v1/models/joint_training:predict -d '{"inputs":{"sent_token_ids": [[2,64,84,1]], "next_token_ids":[64,84]}}'

5. Running ERRANT on parallel M2 files, and extracting just FP:

python -m errant.commands.compare_m2 -hyp cor_inc_hypo_1.m2 -ref cor_inc_query.m2 -dt -cat 2 | tail -18 | head -12 | sed -r 's/[[:space:]]+/,/g' | awk -F',' '{print $1 "," $3}' | sort -rn -k2,2 -t','

6. Copy k8s secret from one namespace to another:

kubectl get secret registry-readonly-secret --namespace=ns-a --export -o yaml |\ kubectl apply --namespace=ns-b -f -

7. Using pipe volume to track ETA, elapsed time, rate of xfer with sweet bash progress bar

pv -pert some_file.txt | cut -d' ' -f1 --complement | sed -r 's!^(.*)$!<s> \1 </s>!g' > some_other_file.txt