Skip to content

Instantly share code, notes, and snippets.

@ecoopnet
Last active April 4, 2024 02:34
Show Gist options
  • Save ecoopnet/97c89ff6329ef3a5ceb4feccd2402a26 to your computer and use it in GitHub Desktop.
Save ecoopnet/97c89ff6329ef3a5ceb4feccd2402a26 to your computer and use it in GitHub Desktop.
指定したURLから再帰的にURLを取ってくるスクリプト
#!/bin/bash
# Usage: fetch_urls.sh "URL"
# Example: fetch_urls.sh "https://example.com/" | tee urls.txt
# https://example.com/ に再帰的にアクセスして、 urls.txt にURL一覧を書き出す
# Requirements: wget perl
# wget が入ってない場合、 brew install wget などでインストールして下さい。
_url="$1"
(
wget -nd -nH --spider --recursive --level=inf --no-verbose --execute robots=off "$_url" 2>&1
) | perl -ne '$| = 1; if (/URL: *([^ ]+) /) { print "$1\n"; }'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment