Skip to content

Instantly share code, notes, and snippets.

View soldair's full-sized avatar

Ryan Day soldair

  • Google
  • San Jose CA
View GitHub Profile
@soldair
soldair / vertica_copy.js
Last active December 14, 2015 23:49
how to run a copy command with data from any node stream into to a vertica database
var through = require('through');
// pure js vertica driver. im not into the coffee script but this is so useful even i domnt hold it against them
var vertica = require('vertica');
var stream = through();
//
// pipe some bar delimited rows of data to your stream.
// until you call stream.end() the data will not commit.
// because you can use logical streams rather than "real" streams..
// .. you can break a job up simply by ending the stream and starting another copy command ..
@soldair
soldair / leak_levelup.js
Created March 22, 2013 23:42
levelup writeStream doesn't pause when behind. this leads to huge memory leaks importing lots of data especially from stdin.
var level = require('levelup')
, db = level('./streamtest.db')
var dbws = db.createWriteStream()
, c = 0
, start = Date.now()
, intervalCount = 0
, bytes = 0
, paused = false
@soldair
soldair / gist:5911432
Created July 2, 2013 17:42
what i do instead of using the "async" module
// do series or exactly N parallel its the same
var c = jobs.length
, job = function(){
doThing(function(){
next()
})
}, next = function(){
c--;
if(!c) done();
else job()
@soldair
soldair / gist:5943759
Created July 7, 2013 15:10
me learns pythons
// write a log file!
file = open(logname,'a+');
file.write("log line!\n");
// stringify json
import json
json.dumps("hi");
@soldair
soldair / gist:5951789
Created July 8, 2013 19:34
process large file with node
process.stdin.pipe(require('line-stream')()).pipe(require('through')(function(line){
// do analysis on a line
})).pipe(process.stdout)
@soldair
soldair / db.js
Last active February 7, 2022 01:01
backup a running node levelup process
var levelup = require('level')
var zlib = require('zlib')
var fs = require('fs')
var through = require('through')
//hot backup
db = levelup(...)
var backingup = false
@soldair
soldair / package.json
Last active December 19, 2015 23:19
multilevel ending does not error active streams
{
"name": "multilevel-end-test",
"description": "analyze end behavior",
"version": "0.0.0",
"repository": {
"url": "git@gist.github.com:6034011"
},
"main": "index.js",
"scripts": {
"test": "tape test/*.js"
@soldair
soldair / gist:6703702
Created September 25, 2013 18:15
npm global broken?
sudo npm install -g npm
npm http GET https://registry.npmjs.org/npm
npm ERR! cb() never called!
npm ERR! not ok code 0
@soldair
soldair / hmm.js
Last active December 25, 2015 17:19
wrap a pipe chain in an exposed through stream
var through = require('through')
module.exports = function(chain){
chain = chain();
var s = through(function(data){
if(!chain.write(data)) this.pause();
});
chain.on('drain',function(){
s.resume();
})
@soldair
soldair / procfs-pipes.md
Last active January 3, 2016 15:09
procfs stats pipeline!!!!

i've exposed quite a few stats in my procfs-stats module. now i need to be able to make use of the data!

###poll-stream

###subtract-values-stream