Skip to content

Instantly share code, notes, and snippets.

@gwen001
Last active November 26, 2022 22:54
Show Gist options
  • Save gwen001/6843c83d878ff22783bd573ac48a2c93 to your computer and use it in GitHub Desktop.
Save gwen001/6843c83d878ff22783bd573ac48a2c93 to your computer and use it in GitHub Desktop.
onliner to quickly test some urls using curl
cat plainurls.txt | while read u;do echo "$(printf '%-100s' "$u")-> $((curl -I -s -m 5 -k "$u"||echo KO)|head -n 1 -)"; done
From a json file:
cat waybackurls.json|jq -r '.[]'|grep 'http'|cut -d '"' -f 2 | while read u;do echo "$(printf '%-100s' "$u")-> $((curl -I -s -m 5 -k "$u"||echo KO)|head -n 1 -)"; done
Using parallel to speed up the process:
cat plainurls.txt | parallel -j 20 -I# 'echo "$(printf "%-100s" "#") -> $((curl -I -s -m 5 -k "#"||echo KO)|head -n 1 -)"'
function otestu {
cat $1 | parallel -j 20 -I# 'echo "$(printf "%-100s" "#") -> $((curl -I -s -m 5 -k "#"||echo KO)|head -n 1 -)"'
#while read u;do echo "$(printf '%-100s' "$u")-> $((curl -I -s -m 5 -k "$u"||echo KO)|head -n 1 -)";done < $1
}
function otestuj {
cat $1 | jq -r '.[]'|grep 'http'|cut -d '"' -f 2 | parallel -j 20 -I# 'echo "$(printf "%-100s" "#") -> $((curl -I -s -m 5 -k "#"||echo KO)|head -n 1 -)"'
#while read u;do echo "$(printf '%-100s' "$u")-> $((curl -I -s -m 5 -k "$u"||echo KO)|head -n 1 -)";done < $1
}
ALTERNATIVES:
@4_N_K_1_T
xargs -a u -I{} curl {} -s -o /dev/null -w "%{http_code}""\t -> \t""{}""\n";
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment