Skip to content

Instantly share code, notes, and snippets.

View thinkerbot's full-sized avatar

Simon Chiang thinkerbot

View GitHub Profile
@thinkerbot
thinkerbot / fix_timestamps.sh
Created November 24, 2016 20:24
Google Uploader
#!/bin/bash
# Google Uploader looks at a file timestamp not the video metadata. Run
# this to get the right timestamp to appear on the video. Requires
# exiftool. https://productforums.google.com/forum/#!topic/photos/oj96JZK14Fs
#
# brew install exiftool
# ./fix_timestamps.sh
#
cd "DIR_WITH_THE_m4v_VIDEO_FILES"
ruby -rtime -ryaml -rshellwords -e '
@thinkerbot
thinkerbot / example.sh
Last active October 29, 2016 03:47
Stream compress, encrypt, decrypt, uncompress
#!/bin/bash
# http://unix.stackexchange.com/questions/79525/openssl-buffering-problem
# http://stackoverflow.com/questions/570984/how-can-i-gzip-standard-in-to-a-file-and-also-print-standard-in-to-standard-out
time (
ruby -e "STDOUT.sync=true; 100000.times {|i| puts(i.to_s + ' knowledge is power') }" |
gzip --stdout |
openssl enc -aes-128-cfb -pass pass:test -bufsize 256 |
openssl enc -base64 -bufsize 256 |
openssl enc -base64 -d -bufsize 256 |
@thinkerbot
thinkerbot / sign_requests_with_instance_profile_for_aws_es.rb
Created August 4, 2016 20:03
Sign requests to get access to a role-controlled AWS elasticsearch
require 'aws-sdk'
require 'faraday_middleware'
require 'faraday_middleware/aws_signers_v4'
require 'pp'
conn = Faraday.new(url: 'http://ENDPOINT') do |faraday|
faraday.request :aws_signers_v4,
credentials: Aws::InstanceProfileCredentials.new.credentials,
service_name: 'es',
region: 'REGION'
x, y = 0, 0
xmin, xmax, ymin, ymax = 0, 0, 0, 0
e = Enumerator.new do |yy|
loop do
while x > xmin
yy << [x, y]
x -= 1
end
xmin -= 1
while y < ymax

Summary

Many types of functions produce data amenable for charting in the xy dimensions. These functions are defined as f(x[n]) = y. To transform this function into an f(x) = y function, provide a projection that takes a set of (x[n], y) values and picks a single (x,y) pair. In practice the set (x[n], y) values must be limited to fit within memory/runtime bounds... ergo for a given x there must be a limited number of x[n-1].

In summary a projection can produce xy data from x[n]y data provided:

  • there is one value y for each (x[n])
  • the number of x[n-1] is limited for a given x

Many types of data end up satisfying these conditions, in particular measurement data. Measurement data can be modeled as ty. If measurement data can be overridden then add a dimension to track when the data was recieved, ie tty.

@thinkerbot
thinkerbot / README.md
Last active August 29, 2015 14:14
Clean merge

One way to merge nested configs cleanly:

  • flatten key paths
  • merge

This approach overcomes the issue of performing a deep merge when an overlay has the ability to specify a value or a hash. This does not deal with merging array values.

Here is the issue:

c1 = {"a" => 1}

RSpec issues with include

We're extracting modules from intellisource and that's a good engineering practice. We can localize and document the footprint of modules by requiring and including them where they're used.

[helper-module.gem]
module HelperModule

class HelperClass

@thinkerbot
thinkerbot / pipe-exec
Last active August 29, 2015 14:06
Hold open a fifo from both ends, allowing input from external processes
#!/usr/bin/env ruby
begin
require 'optparse'
options = {
:direction => "in"
}
OptionParser.new do |opts|
opts.banner = %{
usage: pipe-exec [options] FIFO_FILE COMMAND....
@thinkerbot
thinkerbot / xyz.cql
Created August 26, 2014 16:48
Examples of cql queries
DROP KEYSPACE IF EXISTS example;
CREATE KEYSPACE example WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 };
USE example;
DROP TABLE IF EXISTS xyz;
create table xyz (
xpart int,
x int,
z int,
y int,

Notes

Starting/stopping processes occurs in serial, ergo make sure the start/stop commands go quickly or the whole thing hangs. I It's a good idea to use env.sh scripts to setup the execution environment so you have something fully reproducible. Here is the command to get the ruby environment via rvm.

rvm info environment | sed -n -e 's/: \{1,\}/=/' -e 's|.rvm/bin:.*|.rvm/bin:$PATH"|' -e 's/    \([A-Z]\)/export \1/p'

An example usage:

check process process_name with pidfile /path/to/pidfile.pid

start program = "/usr/bin/env -i KEY=VALUE /path/to/env.sh /path/to/monit_exec start /path/to/pidfile.pid /path/to/stdin_file /path/to/stdout_file /path/to/logfile.log commmand args..."