This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from redis import Redis | |
import simplejson | |
class Resque(object): | |
"""Dirt simple Resque client in Python. Can be used to create jobs.""" | |
redis_server = 'localhost:6379' | |
def __init__(self): | |
host, port = self.redis_server.split(':') | |
self.redis = Redis(host=host, port=int(port)) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SELECT 1.0 + SUM((1.0 + #{p} * tunkrank_score) / (1.0 + num_friends)) AS tunkrank_score | |
FROM twitter_users | |
INNER JOIN twitter_id_follows ON (twitter_users.twitter_id = twitter_id_follows.follower_twitter_id) | |
WHERE twitter_id_follows.user_twitter_id = #{twitter_id}; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
class TwitterUser | |
def calculate_tunkrank(p=0.05) | |
self.followers.inject(0.0) do |sum, follower| | |
sum + ((1.0 + (p * follower.tunkrank_score)) / (1.0 + follower.num_friends)) | |
end | |
end | |
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
diff -Naur whirr-0.3.0-incubating/contrib/python/src/py/hadoop/cloud/cli.py whirr-0.3.0-incubating-backtype/contrib/python/src/py/hadoop/cloud/cli.py | |
--- whirr-0.3.0-incubating/contrib/python/src/py/hadoop/cloud/cli.py 2011-01-15 23:03:44.000000000 -0800 | |
+++ whirr-0.3.0-incubating-backtype/contrib/python/src/py/hadoop/cloud/cli.py 2011-02-15 11:51:49.000000000 -0800 | |
@@ -296,7 +296,7 @@ | |
opt.get('user_data_file'), | |
opt.get('availability_zone'), opt.get('user_packages'), | |
opt.get('auto_shutdown'), opt.get('env'), | |
- opt.get('security_group')) | |
+ opt.get('security_group'), opt.get('spot_price')) | |
service.launch_master(template, config_dir, opt.get('client_cidr')) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(ns gist.globhfs | |
(:import [cascading.tap GlobHfs])) | |
;; ### Bucket to Cluster | |
;; | |
;;; To get tuples back out of our directory structure on S3, we employ | |
;; Cascading's [GlobHFS] (http://goo.gl/1Vwdo) tap, along with an | |
;; interface tailored for datasets stored in the MODIS sinusoidal | |
;; projection. For details on the globbing syntax, see | |
;; [here](http://goo.gl/uIEzu). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# mac port installs bash_completion in /opt/local | |
if [ -f /opt/local/etc/bash_completion ]; then | |
. /opt/local/etc/bash_completion | |
# * | |
export GIT_PS1_SHOWDIRTYSTATE=1 | |
# $ | |
export GIT_PS1_SHOWSTASHSTATE=1 | |
# % | |
# export GIT_PS1_SHOWUNTRACKEDFILES=1 | |
export PS1='\[\e[32m\]λ \w\[\e[36m\]$(__git_ps1 " (%s)") [$(~/.rvm/bin/rvm-prompt i v)]\[\e[0m\]\n\[\e[32m\]→\[\e[0m\] ' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(defn hfs-report | |
[path] | |
"Loads the log data from an HDFS path into Hbase." | |
(?<- (hbase-tap "urls" "?url-hash" "urls" "?url" "?crawl-date" | |
"?crawl-time" "?response-code" "?status" "?host") | |
[?url-hash ?url ?crawl-date ?crawl-time ?response-code ?status ?host] | |
((hfs-textline path) ?text) | |
(fetch-value-hash ?text :url :> ?url-hash) | |
(fetch-value ?text :url :> ?url) | |
(fetch-value ?text :crawl-date :> ?crawl-date) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package forma.tap; | |
import backtype.hadoop.pail.PailStructure; | |
import java.util.Collections; | |
import java.util.List; | |
import org.apache.thrift.TBase; | |
import org.apache.thrift.TDeserializer; | |
import org.apache.thrift.TException; | |
import org.apache.thrift.TSerializer; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(defn counts-per-day [intent-tap] | |
(<- [!intent !bucket !count] | |
(intent-tap _ !intent !ped) | |
(g/extract-true-as-of !ped :> !ms) | |
(o/time-buckets ["d"] !ms :> !bucket) | |
(c/count !count))) | |
(defn avg-sd [counts-per-day-sq min-avg] | |
(<- [!intent !a !sd] | |
(counts-per-day-sq !intent !bucket !count) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(require 'font-lock) | |
(defvar thrift-mode-hook nil) | |
(add-to-list 'auto-mode-alist '("\\.thrift\\'" . thrift-mode)) | |
(defvar thrift-indent-level 2 | |
"Defines 2 spaces for thrift indentation.") | |
;; syntax coloring |
OlderNewer