Get all URLS from a website:
-
Does it have a sitemap.xml? Many CMSes generate these automatically.
-
Use
wget
set DOMAIN example.com
wget --mirror --delete-after --no-directories https://$DOMAIN 2>&1 \
| grep '^--' | awk '{print $3}' | sort > urls.txt- Use python with BeautifulSoup or scrapy
Mostly iterate over site, look at <a> elements with href, maintain a map of visited pages, only visit
particular list of domain(s) so it doesn't follow external links, add a delay in between requests to be
polite.