Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
crawl a web page, extract all domains and resolve them to IP addresses with bash and common GNU/Linux tools
#!/bin/bash
url='rawsec.ml'
domains=$(curl $url -s | grep -E 'https?://[^"]*' | cut -d '/' -f 3 | cut -d '"' -f 1 | uniq)
filename='/tmp/temporary_ips.txt'
for domain in $domains
do
dig +noall +answer $domain | awk '/\sA\s/ {print $5}' >> $filename
done
cat $filename | sort -u
@noraj

This comment has been minimized.

Copy link
Owner Author

noraj commented Apr 26, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.