| L1 cache reference 0.5 ns | |
| Branch mispredict 5 ns | |
| L2 cache reference 7 ns | |
| Mutex lock/unlock 25 ns | |
| Main memory reference 100 ns | |
| Compress 1K bytes with Zippy 3,000 ns | |
| Send 2K bytes over 1 Gbps network 20,000 ns | |
| Read 1 MB sequentially from memory 250,000 ns | |
| Round trip within same datacenter 500,000 ns | |
| Disk seek 10,000,000 ns |
| # encoding: utf-8 | |
| require "cgi" | |
| require "base64" | |
| require "openssl" | |
| require "net/http" | |
| ACCESS_KEY = ENV["AWS_ACCESS_KEY_ID"] | |
| SECRET_KEY = ENV["AWS_SECRET_ACCESS_KEY"] |
First, you need to determine the level of precision you need to query by. In the following example, it ends up being timeboxed into 10 second bins, but in groups of 100 different entities (or meters in this example):
Then, answering a query like:
Give me <45 minutes> worth of data from <18 hours ago> for entities <1, 2, 226, 301>
| ### Keybase proof | |
| I hereby claim: | |
| * I am hectcastro on github. | |
| * I am hector (https://keybase.io/hector) on keybase. | |
| * I have a public key whose fingerprint is 14F5 3D5C 943D 8DD8 DD6C E79C 4E76 EF6B A3A6 8557 | |
| To claim this, I am signing this object: |
| import logging | |
| import socket | |
| class PapertrailHandler(logging.Handler): | |
| """An RFC 5425 log handler that has been tested to interact | |
| with Papertrail. | |
| """ | |
| # RFC 5424 Severities | |
| LOG_EMERG = 0 |
| CREATE EXTERNAL TABLE IF NOT EXISTS azavea_datahub_logs.azavea_datahub ( | |
| bucket_owner string, | |
| bucket string, | |
| requested_at string, | |
| remote_ip string, | |
| requester string, | |
| request_id string, | |
| operation string, | |
| key string, | |
| request_uri_operation string, |
| CREATE EXTERNAL TABLE IF NOT EXISTS cicero.papertrail ( | |
| `id` bigint, | |
| `generated_at` string, | |
| `received_at` string, | |
| `source_id` bigint, | |
| `source_name` string, | |
| `source_ip` string, | |
| `facility_name` string, | |
| `severity_name` string, | |
| `program` string, |