Skip to content

Instantly share code, notes, and snippets.

View sacreman's full-sized avatar
💭
Adding tremendous value

Steven Acreman sacreman

💭
Adding tremendous value
  • ThunderOps
  • United Kingdom
View GitHub Profile
@sacreman
sacreman / requirements.txt
Created March 15, 2016 15:41
Dataloop Agent requirements.txt
###
# Core Agent dependencies. These are require for the Agent to run
###
APScheduler==3.0.3
netifaces==0.10.4
nose==1.3.7
psutil==3.2.2
pytz==2015.2
PyYAML==3.11
requests==2.7.0
@sacreman
sacreman / plugin_requirements.txt
Created March 15, 2016 15:42
Dataloop Agent plugin_requirements.txt
##
# Plugin dependencies. These are required to run the out of the box plugins
##
backports.ssl-match-hostname==3.4.0.2
BeautifulSoup==3.2.1
boto==2.23.0
cffi==0.8.6
cryptography==0.6.1
docker-py==1.4.0
futures==2.2.0
Steven-MacPro:dalmatiner-docker sacreman$ docker logs b5cd325303fd
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/rc.local...
*** Booting runit daemon...
*** Runit started as PID 7
2016/03/29 12:06:33 [I] Starting Grafana
2016/03/29 12:06:33 [I] Version: 2.6.0, Commit: v2.6.0, Build date: 2015-12-14 14:18:01 +0000 UTC
2016/03/29 12:06:33 [I] Configuration Info
Config files:
[0]: /usr/share/grafana/conf/defaults.ini
#!/usr/bin/env python
from gevent import monkey, socket
monkey.patch_all()
from gevent.pool import Pool
import requests
import uuid
settings = {
'url': 'https://app.dataloop.io/api/v1',
'org': '',
#!/usr/bin/env python
from gevent import monkey
monkey.patch_all()
from gevent.pool import Pool
import requests
import uuid
settings = {
'url': 'https://app.dataloop.io/api/v1',
'org': 'yolo',

Problem

tldr; We don't want to be a Datadog level solution forever that keeps the price low(ish) by only supporting light weight events. We also don't want to provide standard SaaS log pricing with one of those slide bars that increases the $$$ as the GB amount goes up until you give up and setup open source.

We want a middle ground that's cheap enough not to worry constantly about log volume, that's incredibly quick so it's nice to use in the app and covers most of the use cases for DevOps troubleshooting. So people can get alerted by time series data and drill down into the context of what happened with log data.

Long Version

Now is the opportune moment to look at event data while the time series analytics front-end is being finished. Event data is similar to time series data but differs just enough that it requires a different data store and a bunch of additional services.

#!/usr/bin/env python
import sys
from dlcli import api
# need to set the org, account and key
settings = {
'url': 'https://app.dataloop.io/api/v1',
'org': '',
'account': '',
'key': ''
#!/usr/bin/env python
import sys
from dlcli import api
'''
Returns a sum of the number of agents that have returned the base.count metric in the last minute.
You will need to update the TAG, org, account and key variables in the settings below.
'''

Introduction

Welcome to the Dataloop API documentation!

To use the API you'll need an api key which can be created in Dataloop under your user account settings. When integrating services you may want to consider creating an application specific user in Dataloop with access to accounts at the correct role level.

You will also need to know the organisation name and account name that you want to work with. These match the organisation and account names in Dataloop. Use these details where you see <org name> and <account name> in the examples.

  • aws
  • awsapi
  • metricsBrowser
  • aws-i