Skip to content

Instantly share code, notes, and snippets.

View derickson's full-sized avatar

David Erickson derickson

16 followers · 5 following
View GitHub Profile
@derickson
derickson / logstash-bikehare-status.conf
Last active March 10, 2021 14:26
Logstash config for http_poller pulling of DC Capital bikeshare data and doing an XML split of the contents.
## Example of pulling data from DC Capital bikeshare to Elasticsearch in real time
## HTTP Poller -> XML Splitting -> Elasticsearch
input {
## pull data from Capital Bikeshare every 60 seconds
http_poller {
urls => {
bikeshare_dc => "https://www.capitalbikeshare.com/data/stations/bikeStations.xml"
}
request_timeout => 30
@derickson
derickson / mongoToES.js
Created July 16, 2015 15:18
Example of NodeJS Loop of Mongo to Elasticsearch
// npm install elasticsearch
// setup nodejs client for elasticsearch
// documentation: https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/index.html
var elasticsearch = require('elasticsearch');
var EsClient = new elasticsearch.Client({
host: 'localhost:9200',
log: 'info'
});
@derickson
derickson / elasticsearch_sql.json
Created August 7, 2018 18:46
## 1) Pull all records from index named fedeastsql
POST /_xpack/sql?format=txt
{
"query": "SELECT * FROM fedeastsql"
}
## 2) List all columns and datatype of this index
@derickson
derickson / ingestBeer.py
Last active July 18, 2018 13:58
#!/bin/python3
import json
import moment
from elasticsearch import helpers
from elasticsearch import Elasticsearch
esConnString = 'http://elastic:changeme@localhost:9200'
esTo = Elasticsearch([esConnString],request_timeout=100)
@derickson
derickson / abac with covery query proposal
Created October 11, 2017 16:55
PUT my_index
{
"mappings": {
"properties": {
"securityTags": {
"type": "keyword",
"fields": {
"length": {
"type": "token_count",
"analyzer": "standard"
@derickson
derickson / combinatorialSet.py
Created August 2, 2017 14:36
#!/usr/bin/python
# -*- coding: utf-8 -*-
import json
def prettyPrint(doc):
print(json.dumps(doc, indent=4, sort_keys=True))
@derickson
derickson / bigBool.json
Created August 2, 2017 14:28
{
"bool": {
"should": [
[
{
"bool": {
"must": [
[
{
"term": {
@derickson
derickson / watcher examples .txt
Last active September 1, 2016 15:03
### EXAMPLE SEARCH: Has Spotify run in the last 30 seconds?
GET /topbeat-*/_search
{
"query": {
"bool": {
"must": [
{"term": {
"proc.name": {
"value": "Spotify"
@derickson
derickson / spectator.html
Last active August 26, 2016 01:40
A-Frame spectator camera example code
<html>
<head>
<script src="../dist/aframe.min.js"></script>
<script>
AFRAME.registerComponent('spectator',{
'schema': {
canvas: {
type: 'string',
default: ''
@derickson
derickson / splitHelper.js
Last active December 24, 2015 16:29
YCSB splittin code for mongodb
var ycsbUtil = {};
ycsbUtil.dropAndSplitYCSB = function (shardCount, chunksPerShard) {
db.getMongo().getDB( 'ycsb' ).dropDatabase();
db.adminCommand( { "enablesharding" : "ycsb" } ) ;
db.adminCommand( { "shardcollection" : "ycsb.usertable", "key" : { "_id" : 1 } } ) ;
var splitCount = shardCount * chunksPerShard;
var splitSize = Math.ceil( 10000 / splitCount; )
for(var i=1; i < splitCount; ++i){