Skip to content

Instantly share code, notes, and snippets.

View lbjay's full-sized avatar

Jay Luker lbjay

View GitHub Profile
@lbjay
lbjay / gitwishes.json
Last active June 9, 2022 13:27
gitwishes exclude repos & patterns
{
"exclude_repos": ["oh_my_fortune","gotgit","GithubBotRepo","dlpkuhole2"],
"exclude_terms": ["Garbage Man", "roomservice", "fluent in silence"],
"queries": [
"i wish",
"i just wish",
"i really wish",
"i seriously wish",
"i seriously hope",
"i really hope"

Keybase proof

I hereby claim:

  • I am lbjay on github.
  • I am lbjay (https://keybase.io/lbjay) on keybase.
  • I have a public key ASAthX1IDfS1Dlsz8-JBZaXF6ScVlVXY7OqGpmBOTO1I9wo

To claim this, I am signing this object:

Hi all,
At Harvard DCE we've been developing a few strategies for automated horizontal scaling of workers. Some involve analysis of AWS CloudWatch metrics and others look at data from Matterhorn itself, e.g. queued jobs. All of them, to a greater or lesser degree, are hindered by the behavior of the job dispatching mechanisms. I'll give a quick example of a typical scenario and why it causes problems for both types of strategies.
Say we have a cluster that has 10 workers at it's disposal but only 1 currently online. A burst of producer activity creates a bunch of new workflows and quickly there are a dozen media inspection and compose jobs generated. What we would expect is that the `max_jobs: n` setting on the worker host would prevent more than n of those jobs from being assigned to the single worker, the rest being put into some sort of queued state. What actually happens is that the 1 worker happily accepts all of them because the dispatching logic in `matterhorn-serviceregistry/.../ServiceRegistryJpaIm
@lbjay
lbjay / bench.py
Last active April 5, 2016 18:18
mh_user_action inserter
import sys
import time
import boto3
import logging
import argparse
from faker import Factory
from random import choice, shuffle
from multiprocessing import Process, JoinableQueue
from mysql.connector import connect
Starting storage, including a deploy
Starting db, including a deploy
Waiting on "bash /usr/bin/vagrant up storage" to finish . . .
There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.
Command: ["setextradata", "07be0611-9249-468c-b3ee-96b6b6894db8", "VBoxInternal2/SharedFoldersEnableSymlinksCreate/tmp_vagrant-opsworks", "1"]
Stderr: VBoxManage: error: The machine 'mh-opsworks_storage_1459356738934_15160' already has a lock request pending
VBoxManage: error: Details: code VBOX_E_INVALID_OBJECT_STATE (0x80bb0007), component MachineWrap, interface IMachine, callee nsISupports
// put this in yr custom json
"elk": {
"kibana_auth": {
"user": "foo",
"pass": "bar"
},
"harvester_release": "v0.1.0"
}
@lbjay
lbjay / get_stack_name.py
Created February 1, 2016 16:28
Get opsworks stack name from agent config
import os
import boto3
from yaml import load
os.environ.setdefault('AWS_DEFAULT_REGION', 'us-east-1')
if __name__ == '__main__':
with open('/etc/aws/opsworks/instance-agent.yml') as f:
agent_config = load(f)
@lbjay
lbjay / 0_reuse_code.js
Created December 15, 2015 16:16
Here are some things you can do with Gists in GistBox.
// Use Gists to store code you would like to remember later on
console.log(window); // log the "window" object to the console
@lbjay
lbjay / harvard.css
Last active August 29, 2015 14:22
canvas styleguide css foolery
* {
font-family: "monospace";
}
a.name {
color: red;
}