Skip to content

Instantly share code, notes, and snippets.

@h5y1m141
Created October 3, 2012 22:24
Show Gist options
  • Save h5y1m141/3830272 to your computer and use it in GitHub Desktop.
Save h5y1m141/3830272 to your computer and use it in GitHub Desktop.
CoffeeScriptで書きなおしたnode.jsの標準モジュールを使ったサンプルクローラー
class Crawler
fs = require 'fs'
sys = require 'sys'
http = require 'http'
url = require 'url'
constructor: ->
crawl:(list) ->
self = this
self.fetch u for u in list
fetch:(targetUrl) ->
parsedUrl = url.parse(targetUrl)
client = http.createClient(80,parsedUrl.hostname)
request = client.request('GET',parsedUrl.pathname,'host':parsedUrl.hostname)
request.end()
request.on('response',(response)->
body = ''
response.on('end',->
console.log(targetUrl + " - " + response.statusCode)
)
)
readFile:(path) ->
self = this
fs.readFile(path, (err,data) ->
console.log 'start'
if err
console.error "List file is not exists"
process.exit(1)
list = eval('(' + data + ')')
self.crawl(list)
)
module.exports = Crawler
Crawler = require 'crawler'
c = new Crawler()
listPath = process.argv[2]
c.readFile(listPath)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment