I'm having issues inserting new indicators into CIF.
When I use the python SDK as described in https://github.com/csirtgadgets/bearded-avenger-sdk-py/wiki (changing cli.submit() to cli.indicators_create()), using the literal string '{"indicator":"example-site-malware.com","tlp":"amber","confidence":"8","tags":"test","provider":"example.com","group":"everyone"}' as my data argument, I get a 404 response after quite a long time and CIF stops responding to queries until I restart the cif-router service. The log message that appears to be associated with this is as follows:
Aug 20 18:29:36 cif cif-router[1205]: Process Process-6:
Aug 20 18:29:36 cif cif-router[1205]: Traceback (most recent call last):
Aug 20 18:29:36 cif cif-router[1205]: File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Aug 20 18:29:36 cif cif-router[1205]: self.run()
Aug 20 18:29:36 cif cif-router[1205]: File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
Aug 20 18:29:36 cif cif-router[1205]: self._target(*self._args, **self._kwargs)
Aug 20 18:29:36 cif cif-router[1205]: File "/usr/local/lib/python2.7/dist-packages/cif-3.0.0rc5-py2.7.egg/cif/gatherer/__init__.py", line 80, in start
Aug 20 18:29:36 cif cif-router[1205]: data = json.loads(data)
Aug 20 18:29:36 cif cif-router[1205]: ValueError: Expected object or value
If I try cli.indicators_create(json.dumps(data)), cif-router does not need to be restarted but I still get a failure as follows:
Aug 20 18:49:10 cif cif-router[2449]: Process Process-6:
Aug 20 18:49:10 cif cif-router[2449]: Traceback (most recent call last):
Aug 20 18:49:10 cif cif-router[2449]: File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Aug 20 18:49:10 cif cif-router[2449]: self.run()
Aug 20 18:49:10 cif cif-router[2449]: File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
Aug 20 18:49:10 cif cif-router[2449]: self._target(*self._args, **self._kwargs)
Aug 20 18:49:10 cif cif-router[2449]: File "/usr/local/lib/python2.7/dist-packages/cif-3.0.0rc5-py2.7.egg/cif/gatherer/__init__.py", line 87, in start
Aug 20 18:49:10 cif cif-router[2449]: i = Indicator(**d)
Aug 20 18:49:10 cif cif-router[2449]: TypeError: type object argument after ** must be a mapping, not unicode
If I attempt to use the cURL method:
curl -v -i -k -H "Content-Type: application/json" \
-XPOST -H "Authorization: Token token=50m3t0k3n" \
'https://cif-host-url.web/indicators' \
-d '{"indicator":"example-site-malware.com","tlp":"amber","confidence":"8","tags":"test","provider":"example.com","group":"everyone"}'
I get a 404 and log messages similar to what I get from indicators_create().
I suspect this has something to do with the way my JSON is structured.
Further, I see a lot of these when cif-router fails and needs restarting:
Aug 20 19:16:35 cif gunicorn[1284]: [2018-08-20 19:16:35,141] ERROR in app: Exception on /ping [GET]
Aug 20 19:16:35 cif gunicorn[1284]: Traceback (most recent call last):
Aug 20 19:16:35 cif gunicorn[1284]: File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
Aug 20 19:16:35 cif gunicorn[1284]: response = self.full_dispatch_request()
Aug 20 19:16:35 cif gunicorn[1284]: File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in full_dispatch_request
Aug 20 19:16:35 cif gunicorn[1284]: rv = self.handle_user_exception(e)
Aug 20 19:16:35 cif gunicorn[1284]: File "/usr/local/lib/python2.7/dist-packages/flask_cors/extension.py", line 161, in wrapped_function
Aug 20 19:16:35 cif gunicorn[1284]: return cors_after_request(app.make_response(f(*args, **kwargs)))
Aug 20 19:16:35 cif gunicorn[1284]: File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in handle_user_exception
Aug 20 19:16:35 cif gunicorn[1284]: reraise(exc_type, exc_value, tb)
Aug 20 19:16:35 cif gunicorn[1284]: File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
Aug 20 19:16:35 cif gunicorn[1284]: rv = self.dispatch_request()
Aug 20 19:16:35 cif gunicorn[1284]: File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
Aug 20 19:16:35 cif gunicorn[1284]: return self.view_functions[rule.endpoint](**req.view_args)
Aug 20 19:16:35 cif gunicorn[1284]: File "/usr/local/lib/python2.7/dist-packages/flask/views.py", line 84, in view
Aug 20 19:16:35 cif gunicorn[1284]: return self.dispatch_request(*args, **kwargs)
Aug 20 19:16:35 cif gunicorn[1284]: File "/usr/local/lib/python2.7/dist-packages/flask/views.py", line 149, in dispatch_request
Aug 20 19:16:35 cif gunicorn[1284]: return meth(*args, **kwargs)
Aug 20 19:16:35 cif gunicorn[1284]: File "/usr/local/lib/python2.7/dist-packages/cif-3.0.0rc5-py2.7.egg/cif/httpd/views/ping.py", line 24, in get
Aug 20 19:16:35 cif gunicorn[1284]: r = Client(remote, pull_token()).ping(write=write)
Aug 20 19:16:35 cif gunicorn[1284]: File "/usr/local/lib/python2.7/dist-packages/cifsdk/client/zeromq.py", line 110, in ping
Aug 20 19:16:35 cif gunicorn[1284]: return self._send(Msg.PING_WRITE)
Aug 20 19:16:35 cif gunicorn[1284]: File "/usr/local/lib/python2.7/dist-packages/cifsdk/client/zeromq.py", line 106, in _send
Aug 20 19:16:35 cif gunicorn[1284]: return self._recv(decode=decode)
Aug 20 19:16:35 cif gunicorn[1284]: File "/usr/local/lib/python2.7/dist-packages/cifsdk/client/zeromq.py", line 53, in _recv
Aug 20 19:16:35 cif gunicorn[1284]: mtype, data = Msg().recv(self.socket)
Aug 20 19:16:35 cif gunicorn[1284]: File "/usr/local/lib/python2.7/dist-packages/cifsdk/msg.py", line 72, in recv
Aug 20 19:16:35 cif gunicorn[1284]: m = s.recv_multipart()
Aug 20 19:16:35 cif gunicorn[1284]: File "/usr/local/lib/python2.7/dist-packages/zmq/sugar/socket.py", line 395, in recv_multipart
Aug 20 19:16:35 cif gunicorn[1284]: parts = [self.recv(flags, copy=copy, track=track)]
Aug 20 19:16:35 cif gunicorn[1284]: File "zmq/backend/cython/socket.pyx", line 693, in zmq.backend.cython.socket.Socket.recv
Aug 20 19:16:35 cif gunicorn[1284]: File "zmq/backend/cython/socket.pyx", line 727, in zmq.backend.cython.socket.Socket.recv
Aug 20 19:16:35 cif gunicorn[1284]: File "zmq/backend/cython/socket.pyx", line 150, in zmq.backend.cython.socket._recv_copy
Aug 20 19:16:35 cif gunicorn[1284]: File "zmq/backend/cython/socket.pyx", line 145, in zmq.backend.cython.socket._recv_copy
Aug 20 19:16:35 cif gunicorn[1284]: File "zmq/backend/cython/checkrc.pxd", line 19, in zmq.backend.cython.checkrc._check_rc
Aug 20 19:16:35 cif gunicorn[1284]: raise Again(errno)
Aug 20 19:16:35 cif gunicorn[1284]: Again: Resource temporarily unavailable
I am also seeing the following, but am unsure what action is associated with these log messages (I see quite a few at a time when they do appear):
Aug 21 16:13:44 cif cif-router[2654]: ': {u'caused_by': {u'reason': u"Can't get text on a START_OBJECT at 1:516", u'type': u'illegal_state_exception'}, u'reason': u'failed to parse [additional_data]', u'type': u'mapper_parsing_exception'}, u'_ind
ex': u'indicators-2018.08'}}, {u'index': {u'status': 400, u'_type': u'indicator', u'_id': u'AWVdQ2xf5G-aQFjlcQvS', u'error': {u'caused_by': {u'reason': u"Can't get text on a START_OBJECT at 1:603", u'type': u'illegal_state_exception'}, u'reason':
u'failed to parse [additional_data]', u'type': u'mapper_parsing_exception'}, u'_index': u'indicators-2018.08'}}])