Skip to content

Instantly share code, notes, and snippets.

View iwiwi's full-sized avatar
🏠
Working from home

Takuya Akiba iwiwi

🏠
Working from home
View GitHub Profile
@iwiwi
iwiwi / crawler.rb
Last active September 20, 2023 10:58
Lightweight parallel web graph crawler
#!/usr/bin/env ruby
#
# crawler.rb --- Lightweight parallel web graph crawler
#
# Usage:
# ./crawler.rb START_URL TARGET_REGEXP
#
# Output:
# stdout --- edge list (tab separated URLs)
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.