Skip to content

Instantly share code, notes, and snippets.

@mr-rizwan-syed
Created December 2, 2023 08:42
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save mr-rizwan-syed/1e1dfe3aad931fc9e8b49c3679cab965 to your computer and use it in GitHub Desktop.
To extract URLs of technologies from httpx & webanalyze results combined
#!/bin/bash
if [ -z "$1" ]; then
echo "Usage: techdetect <technology>"
exit 1
fi
urls=()
# Check existence of httpxout*.json files and process each one
for file in httpxout*.json; do
[ -e "$file" ] && urls+=($(cat "$file" | jq -r 'select(.tech // [] | length > 0) | [.url, .tech[]] | @csv' | grep -i "$1" | cut -d , -f 1 | tr -d '"'))
done
# Check existence of webtech.json file
[ -e webtech.json ] && urls+=($(cat webtech.json | jq -r '. | [.hostname, .matches[].app_name] | @csv' | grep -i "$1" | cut -d , -f 1 | tr -d '"'))
result=$(printf "%s\n" "${urls[@]}")
for url in $result; do
echo "$url" | grep -oE "^https?://[^/]*(:[0-9]+)?"
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment