skyline analyzer errors

gistfile1.txt

Sample logs at fresh startup/empty redis (horizon is started first a few seconds before analyzer in this case):
---------------------------------------------------------------------
started with pid 18445
2013-10-16 08:58:16 :: 18445 :: starting skyline analyzer
2013-10-16 08:58:16 :: 18445 :: seconds to run    :: 0.35
2013-10-16 08:58:16 :: 18445 :: total metrics     :: 40249
2013-10-16 08:58:16 :: 18445 :: total analyzed    :: -1
2013-10-16 08:58:16 :: 18445 :: total anomalies   :: 0
2013-10-16 08:58:16 :: 18445 :: exception stats   :: {'Incomplete': 40250}
2013-10-16 08:58:16 :: 18445 :: anomaly breakdown :: {}
2013-10-16 08:58:16 :: 18445 :: canary duration   :: 0.07
2013-10-16 08:58:16 :: 18445 :: sleeping due to low run time...
2013-10-16 08:58:27 :: 18445 :: seconds to run    :: 0.72
2013-10-16 08:58:27 :: 18445 :: total metrics     :: 40567
2013-10-16 08:58:27 :: 18445 :: total analyzed    :: -3
2013-10-16 08:58:27 :: 18445 :: total anomalies   :: 0
2013-10-16 08:58:27 :: 18445 :: exception stats   :: {'Incomplete': 40570}
2013-10-16 08:58:27 :: 18445 :: anomaly breakdown :: {}
2013-10-16 08:58:27 :: 18445 :: canary duration   :: 0.08
2013-10-16 08:58:27 :: 18445 :: sleeping due to low run time...
2013-10-16 08:58:37 :: 18445 :: seconds to run    :: 0.41
2013-10-16 08:58:37 :: 18445 :: total metrics     :: 40931
2013-10-16 08:58:37 :: 18445 :: total analyzed    :: -4
2013-10-16 08:58:37 :: 18445 :: total anomalies   :: 0
2013-10-16 08:58:37 :: 18445 :: exception stats   :: {'Incomplete': 40935}
2013-10-16 08:58:37 :: 18445 :: anomaly breakdown :: {}
2013-10-16 08:58:37 :: 18445 :: canary duration   :: 0.08
2013-10-16 08:58:38 :: 18445 :: sleeping due to low run time...
2013-10-16 08:58:48 :: 18445 :: seconds to run    :: 0.66
2013-10-16 08:58:48 :: 18445 :: total metrics     :: 41285
2013-10-16 08:58:48 :: 18445 :: total analyzed    :: 0
2013-10-16 08:58:48 :: 18445 :: total anomalies   :: 0
2013-10-16 08:58:48 :: 18445 :: exception stats   :: {'Incomplete': 41285}
2013-10-16 08:58:48 :: 18445 :: anomaly breakdown :: {}
2013-10-16 08:58:48 :: 18445 :: canary duration   :: 0.08
2013-10-16 08:58:48 :: 18445 :: sleeping due to low run time...
2013-10-16 08:58:59 :: 18445 :: seconds to run    :: 0.35
2013-10-16 08:58:59 :: 18445 :: total metrics     :: 41542
2013-10-16 08:58:59 :: 18445 :: total analyzed    :: -3
2013-10-16 08:58:59 :: 18445 :: total anomalies   :: 0
2013-10-16 08:58:59 :: 18445 :: exception stats   :: {'Incomplete': 41545}
2013-10-16 08:58:59 :: 18445 :: anomaly breakdown :: <DictProxy object, typeid 'dict' at 0x7f61846f5090; '__str__()' failed>
2013-10-16 08:58:59 :: 18445 :: canary duration   :: 0.08
2013-10-16 08:58:59 :: 18445 :: sleeping due to low run time...
multiprocessing/managers.py", line 216, in serve_client
    obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '40ecca0'
---------------------------------------------------------------------------
Process Process-43:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/usr/lib64/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/wayfair/pkg/skyline/src/analyzer/analyzer.py", line 120, in spin_process
    if key not in self.exceptions:
  File "<string>", line 2, in __contains__
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 216, in serve_client
    obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '40ecca0'
---------------------------------------------------------------------------
Process Process-41:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/usr/lib64/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "/wayfair/pkg/skyline/src/analyzer/analyzer.py", line 120, in spin_process
    if key not in self.exceptions:
  File "<string>", line 2, in __contains__
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 216, in serve_client
    obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '40ecca0'
---------------------------------------------------------------------------
2013-10-16 08:59:10 :: 18445 :: seconds to run    :: 0.74
2013-10-16 08:59:10 :: 18445 :: total metrics     :: 41918
, in __bootstrap_inner
    self.run()
  File "/wayfair/pkg/skyline/src/analyzer/analyzer.py", line 192, in run
    logger.info('total analyzed    :: %d' % (len(unique_metrics) - sum(self.exceptions.values())))
  File "<string>", line 2, in values
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod
    raise convert_to_error(kind, result)
RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 216, in serve_client
    obj, exposed, gettypeid = id_to_obj[ident]
KeyError: '40ecca0'
---------------------------------------------------------------------------