travisbhartwell

NOTE: a more up-to-date version of this can be found on my blog

nix-shell and Shebang Lines

A few days ago, version 1.9 of the Nix package manager was released. From the release notes:

nix-shell can now be used as a #!-interpreter. This allows you to write scripts that dynamically fetch their own dependencies.

olgabot

max-mapper

• dat data model
  • commit graph (mutable)
    • Merkle DAG
    • integrity check a root hash, and know all children are ok too (efficient!)
    • deduplication of common children
    • can content address all the nodes in the datastructure
    • can reveal single path from node to a root (with adjacent hashes) and prove object must be in dag
  • operation database (mutable)
    • protobufs in leveldb
• prototbufs support efficient encodings and backwards compat

rwaldron

It's ok to say or write ES6 when you mean ES2015

Allen Wirfs-Brock said "ES6 is the twitter friendly abbreviation for ECMAScript 2015" and Alex Sexton provided a rationalization.

lindenb

define run_fastqc

$$(addsuffix .ok,$(1)) : $(1)
	mkdir -p $$(dir $$@) && \
	unzip -p $$< $$(addsuffix .tmp_fastqc/fastqc_data.txt,$$(notdir $$<)) |\
		grep ">>" | grep -v ">>END_MODULE" | cut -c 3- |\
		awk -F '	' 'BEGIN{N=0;} {if(($$$$1=="Basic Statistics" || $$$$1=="Per base sequence quality" || $$$$1=="Per base N content") && $$$$2=="pass" ) N++;} END { if(N>2) {print "__PASS__"}}' |\
		grep __PASS__ && echo "FASTQC ok for $$<" > $$@
		

vsbuffalo

library(magrittr)

write.table(iris, file="iris.txt", sep="\t", row.names=FALSE, quote=FALSE)

plines <- function(cmd) readLines(pipe(cmd))
ptabs <- function(cmd) read.delim(pipe(cmd))
 
# just as a trivial example, process some shuffled lines using GNU shuf, sort,
# etc. (we could do this in R, but this is to simulate command line operations)

hadley

If you were to give recommendations to your "little brother/sister" on things that they need to do to become a data scientist, what would those things be?

I think the "Data Science Venn Diagram" (http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram) is a great place to start. You need three things to be a good data scientist:

• Statistical knowledge
• Programming/hacking skills
• Domain expertise

Statistical knowledge

mafintosh

var atomicQueue = require(...)

var queue = atomicQueue(db, function (data, cb) {
  console.log('got some work', data)
  // do stuff ...
  cb(null, result) // if no error this work will be removed from the leveldb
})

// write will add this 'work' to the leveldb
queue.write({some: 'world'})

audy

#!/usr/bin/env node

var fs = require('fs');
var fasta = require('bionode-fasta');
var crypto = require('crypto');

fasta.obj('proteins.fasta').on('data', function(x) {
  var seq = x.seq
        .replace(/\*/g, '') // remove asterisks (stop codons)
        .toUpperCase();     // enforce case

ctb

https://www.youtube.com/watch?x-yt-cl=84359240&x-yt-ts=1421782837&feature=player_embedded&v=ZACVcJt0oJA#t=7303

Kai Blin: If we containerize all these things won’t it just encourage
worse software development practices; right now developers still need
to consider someone other than themselves installing the software.

Michael Barton:

“It’s a good point. Ultimately, though, if I can get a container, and
it works, and I know it will work, do you care how well it was