Skip to content

Instantly share code, notes, and snippets.

Last active August 29, 2015 14:14
Show Gist options
  • Save otobrglez/f9eee98ef08a9fb53b7f to your computer and use it in GitHub Desktop.
Save otobrglez/f9eee98ef08a9fb53b7f to your computer and use it in GitHub Desktop.
Rollbar log observer for Scrapy / Python
import rollbar
import logging
# Needed if you use rollbar.report_exc_info
import sys
class RollbarLogObserver(object):
def __init__(self, rollbar_access_token, env="development", level=logging.INFO):
self.level = level
rollbar.init(rollbar_access_token, env)
def emit(self, event={}):
if event['isError']:
if 'failure' in event:
text = event['failure'].getTraceback()
text = " ".join([str(m) for m in event["message"]]) + "\n"
# You could also add more infor and exec info like this.
# rollbar.report_exc_info(sys.exc_info(), {})
extra_data = {}
if event['spider']:
spider = event['spider']
# Some usefull data like class, name and 'why', useful for debugging.
extra_data = {
'class': str(spider.__class__.__name__),
'why': event['why']
# To add Rollbar to your Scrapy project add this to
# ... settings ...
from scrapy import log
from collector.log import RollbarLogObserver # You could also save it somewhere else
log.log.addObserver(RollbarLogObserver("token goes here", env="production", level=LOG_LEVEL).emit)
Copy link

brianr commented Jan 24, 2015

Hey! Brian from Rollbar here -- I found this via a Google alert.

A quick suggestion: for the case where you do have a traceback, use the rollbar.report_exc_info() form. This way you'll send a nicely structured traceback to Rollbar, and that will make a lot of things work better (grouping, UI, etc.).

Copy link

yrunts commented May 22, 2015

Thanks for such good receipt, also I've tried rollbar.report_exc_info() approach but it crashes:

File "/tmp/eggs-19WeYZ/rollbar.egg/rollbar/", line 716, in _add_locals_data
TypeError: unhashable type: 'list'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment