Skip to content

Instantly share code, notes, and snippets.

@darshanime
Created October 5, 2015 11:44
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save darshanime/16ab4bb86c11151a494e to your computer and use it in GitHub Desktop.
Save darshanime/16ab4bb86c11151a494e to your computer and use it in GitHub Desktop.
Stats for a scrapy run
{'downloader/exception_count': 217,
'downloader/exception_type_count/twisted.internet.error.DNSLookupError': 27,
'downloader/exception_type_count/twisted.internet.error.TimeoutError': 6,
'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived': 184,
'downloader/request_bytes': 93555,
'downloader/request_count': 304,
'downloader/request_method_count/GET': 304,
'downloader/response_bytes': 1127213,
'downloader/response_count': 87,
'downloader/response_status_count/200': 8,
'downloader/response_status_count/301': 12,
'downloader/response_status_count/302': 15,
'downloader/response_status_count/400': 6,
'downloader/response_status_count/401': 2,
'downloader/response_status_count/402': 14,
'downloader/response_status_count/403': 3,
'downloader/response_status_count/404': 15,
'downloader/response_status_count/503': 12,
'dupefilter/filtered': 12,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2015, 10, 5, 11, 40, 28, 914811),
'item_scraped_count': 7,
'log_count/DEBUG': 360,
'log_count/ERROR': 75,
'log_count/INFO': 24,
'response_received_count': 47,
'scheduler/dequeued': 304,
'scheduler/dequeued/memory': 304,
'scheduler/enqueued': 304,
'scheduler/enqueued/memory': 304,
'start_time': datetime.datetime(2015, 10, 5, 11, 24, 27, 524214)}
2015-10-05 17:10:28 [scrapy] INFO: Spider closed (finished)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment