brendano

# by brendan o'connor (http://brenocon.com) written in early 2012
# parallelized collapsed gibbs sampling for LDA with threads in cython
# need to delete these lines to get the cython instructions to work...

#cython: boundscheck=False, cdivision=True
# vim:sts=4:sw=4
import numpy as np
cimport numpy as np
cimport cython
cimport openmp

brendano

TEXT	Gradpics Today <3 !!
OLD	Gradpics Today < 3 !!
NEW	Gradpics Today <3 !!

TEXT	Great Review on #JBSpinshttp://jbspins.blogspot.com/2010/10/fandom-love-saturday-nightmares.html
OLD	Great Review on #JBSpinshttp :/ / jbspins.blogspot.com/2010/10/fandom-love-saturday-nightmares.html
NEW	Great Review on #JBSpinshttp :// jbspins.blogspot.com/2010/10/fandom-love-saturday-nightmares.html

TEXT	RT @kamilaluvsdemi: @IselgomezI dont really know where i would be without you and Demi<3 i luv youuuu
OLD	RT @kamilaluvsdemi : @IselgomezI dont really know where i would be without you and Demi < 3 i luv youuuu

brendano

#!/usr/bin/env python
""" set operations on files as lists.  symlink this as:
* setdiff [-c] <set1> <set2> - set difference
* setand  [-c] <set1> <set2> - set intersection
* setor   [-c] <set1> <set2> - set union
-c means: give count of the result
Output order is randomish
We don't newline chomp, so a bug if your file doesnt end with a newline 
Dash - for stdin (e.g. cut/awk/sed/grep)
Though in zsh, =(bla bla) syntax is superior: can do 2 pipeline inputs

brendano

# TODO this file should eventually go away and be folded into the
# infrastructure for all the other samplers.
#
# This samples the funnel distribution, then tests that the inferences are correct.
# Can trigger the warnings by modifying funnel_Lpstar; e.g. make the mean 0.8
# and it will detect the inferred mean is so far from 0 that there's a problem.
#
#Example output:
#julia> include("test_ss.jl")
#ls2 ess 2324.94, out of raw 99990

brendano

Colons can be used to align columns.

Tables	Are	Cool
`col 3 is
good `	right-aligned	$1600
col 2 is	centered	$12
zebra stripes	are neat	$1

The outer pipes (|) are optional, and you don't need to make the raw Markdown line up prettily. You can also use inline Markdown.

brendano

13150 samples ~ 0.4% se, though more in reality
       File                                            Function            Line
50.3 % treetm.jl cgsIterPath          383             qnewLL,pnewLL = proposePath!(newpath, mm, V,di,word, first ? nothing : oldpath, :simulate)
44.1 % treetm.jl cgsIterPath          384             qoldLL,poldLL = proposePath!(oldpath, mm, V,di,word, first ? nothing : oldpath, :evaluate)
 4.1 % treetm.jl getindex              79 getindex(c::ClassCountTable, k) = c.counts[k]
 2.1 % treetm.jl incrementFullpath!   122         x = b ? x.right : x.left
22.0 % treetm.jl proposePath!         337         w0 = (n0.counts[wordID] + betaHere/V - on0) / (n0.counts.total + betaHere - int(on_cur))
18.9 % treetm.jl proposePath!         338         w1 = (n1.counts[wordID] + betaHere/V - on1) / (n1.counts.total + betaHere - int(on_cur))
 5.2 % treetm.jl proposePath!         339         p0 = (cur_docnode.left.count + mm.gammaConc/2 - on0)
 6.5 % treetm.jl proposePath!         340         p1 = (cur_do

brendano

       File                                            Function            Line
48.0 % /Users/brendano/Desktop/hier_lda/code/treetm.jl cgsIterPath          389             qnewLL,pnewLL = proposePath!(newpath, mm, V,di,word, first ? nothing : oldpath, :simulate)
41.6 % /Users/brendano/Desktop/hier_lda/code/treetm.jl cgsIterPath          390             qoldLL,poldLL = proposePath!(oldpath, mm, V,di,word, first ? nothing : oldpath, :evaluate)
 1.1 % /Users/brendano/Desktop/hier_lda/code/treetm.jl cgsIterPath          391             logA = pnewLL-poldLL + qoldLL-qnewLL  #  (pnew-qnew) - (pold-qnew)
 3.0 % /Users/brendano/Desktop/hier_lda/code/treetm.jl cgsIterPath          402                 incrementFullpath!(mm.cTopicWord, newpath, word, +1)
 3.1 % /Users/brendano/Desktop/hier_lda/code/treetm.jl cgsIterPath          408                     incrementFullpath!(mm.cTopicWord, oldpath, word, -1)
 2.7 % /Users/brendano/Desktop/hier_lda/code/treetm.jl getindex              82 getindex(c::ClassCountTable, k) = c.c

brendano

julinclude("gotree.jl")
Array{CountTrie,1}
accept rate = 850654/850654 = 1.000
elapsed time: 33.986743839 seconds (5921922372 bytes allocated)
.ITER 1
accept rate = 809118/850654 = 0.951
elapsed time: 36.449559574 seconds (6175698392 bytes allocated)
.ITER 2
accept rate = 796254/850654 = 0.936
elapsed time: 30.21721326 seconds (6166426280 bytes allocated)

brendano

# http://mikelove.wordpress.com/2013/11/07/empirical-bayes/

# Stein's estimation rule and its competitors - an empirical Bayes approach
# B Efron, C Morris, Journal of the American Statistical, 1973
n <- 1000
sigma.means <- 5
means <- rnorm(n, 0, sigma.means)
# sigma.y <- 5
library(manipulate)
manipulate({

brendano

package nlp;

import java.io.IOException;
import java.io.StringReader;

import edu.stanford.nlp.io.IOUtils;
import edu.stanford.nlp.ling.CoreAnnotations;
import edu.stanford.nlp.ling.CoreLabel;
import edu.stanford.nlp.trees.LabeledScoredTreeFactory;
import edu.stanford.nlp.trees.PennTreeReader;