Skip to content

Instantly share code, notes, and snippets.

@kaja47
kaja47 / csfdsim.scala
Created December 30, 2014 04:58
How to compute similar movies from CSFD data in 10 minutes and find love of your life
import breeze.linalg._
import breeze.stats
import breeze.numerics._
val dataFile = new File(???)
val userItems: Array[SparseVector[Double]] = loaderUserItemsWithRatings(dataFile, """[ ,:]""".r)
val itemUsers: Array[SparseVector[Double]] = transpose(userItems) map { vec => normalize(vec, 2) }
// weights
val N = DenseVector.fill[Double](itemIndex.size)(userIndex.size) // vector where total numbers of users is repeated
@kaja47
kaja47 / svd-img.scala
Created May 13, 2014 21:57
Visualization of truncated SVD
import breeze._
import breeze.linalg._
import breeze.numerics._
import java.awt.image.BufferedImage
import javax.imageio.ImageIO
val f = ???
val img = javax.imageio.ImageIO.read(new File(f))
val gray = new BufferedImage(img.getWidth, img.getHeight, BufferedImage.TYPE_BYTE_GRAY)
val g = gray.createGraphics()
@kaja47
kaja47 / gist:554f62c61f21b0420720
Created May 9, 2014 19:47
minhash vs. HyperLogLog
// min-hash
val fs: Vector[Int => Int] // hash funkce
items map { it => fs map { f => f(it) } } fold (vectorPairwise(min), initialValue = Vector.fill(infinity))
// HyperLogLog
@bnyeggen
bnyeggen / raid_mtbf.py
Created July 11, 2011 22:54
A RAID MTBF calculator
#redundancy is the max number of survivable failures, so eg 1 for RAID5
#mtbf_array is an array of either actual mean-time-between-failures, or a nested RAID array
# RAID([100]*7,2) #7 disk RAID 6
# RAID([RAID([100]*3,1),RAID([1000]*3,1)],0) # RAID 50, 2 arrays of 3
# RAID([100,100,50,50],1) #RAID 5 with varying reliabilities
from random import random
class RAID(object):
@steipete
steipete / iOSDocumentMigrator.m
Created December 6, 2011 15:21
Helps migrating documents between iOS <= 5.0 and >= 5.0.1 to comply with Apple's iCloud guidelines. Follow @steipete on Twitter for updates.
#include <sys/xattr.h>
/// Set a flag that the files shouldn't be backuped to iCloud.
+ (void)addSkipBackupAttributeToFile:(NSString *)filePath {
u_int8_t b = 1;
setxattr([filePath fileSystemRepresentation], "com.apple.MobileBackup", &b, 1, 0, 0);
}
/// Returns the legacy storage path, used when the com.apple.MobileBackup file attribute is not available.
+ (NSString *)legacyStoragePath {
@michaelaguiar
michaelaguiar / d3donut.js
Created December 17, 2011 00:37
d3.js donut chart test
var donutVal = 85;
var donutFull = 100 - donutVal;
var d3_category_socialmedia = ["#0054a6", "#dbdbdb"];
if(donutVal < 50) {
donutVal = -donutVal;
donutFull = -donutFull;
@darkseed
darkseed / apache-logs-hive.sql
Created March 12, 2012 00:04 — forked from emk/apache-logs-hive.sql
Apache log analysis with Hadoop, Hive and HBase
-- This is a Hive program. Hive is an SQL-like language that compiles
-- into Hadoop Map/Reduce jobs. It's very popular among analysts at
-- Facebook, because it allows them to query enormous Hadoop data
-- stores using a language much like SQL.
-- Our logs are stored on the Hadoop Distributed File System, in the
-- directory /logs/randomhacks.net/access. They're ordinary Apache
-- logs in *.gz format.
--
-- We want to pretend that these gzipped log files are a database table,
@darkseed
darkseed / apache-logs-hive.sql
Created June 1, 2012 21:00 — forked from emk/apache-logs-hive.sql
Apache log analysis with Hadoop, Hive and HBase
-- This is a Hive program. Hive is an SQL-like language that compiles
-- into Hadoop Map/Reduce jobs. It's very popular among analysts at
-- Facebook, because it allows them to query enormous Hadoop data
-- stores using a language much like SQL.
-- Our logs are stored on the Hadoop Distributed File System, in the
-- directory /logs/randomhacks.net/access. They're ordinary Apache
-- logs in *.gz format.
--
-- We want to pretend that these gzipped log files are a database table,
@NorthIsUp
NorthIsUp / spawn.py
Created July 5, 2012 12:46
gevent spawn helpers
"""
realertime.lib.spawn
~~~~~~~~~~~~~~~~~~~~
:author: Adam Hitchcock
:copyright: (c) 2012 DISQUS.
:license: Apache License 2.0, see LICENSE for more details.
"""
from __future__ import absolute_import
@stucchio
stucchio / monte_carlo_compare_theory_to_practice.py
Created June 17, 2014 13:34
Code to make other graph in equal weights post
from pylab import *
from numpy.random import dirichlet, rand, binomial, uniform, normal
def _unit_weight(dim):
return ones(dim) / float(dim)
ONE_FRAC = 0.5
SQRT_TWO_INV = 1.0 / sqrt(2.0)
def _feature_vec(dim, method="bernoulli"):
if method == "bernoulli":