davidsnyder

require 'yaml'
require 'json'
require 'icss'
require 'icss/brevity'

yaml_file = File.read("config.yaml")

#load a YAML ICSS
yaml_icss = Icss::Protocol.receive YAML.load(yaml_file)

davidsnyder

=> #<Icss::RecordField:0x7f7189e34458 @type=#<Icss::PrimitiveType:0x7f7189fbca28 @name=:string, @mysql_name="VARCHAR", @ruby_klass=String, @pig_name="chararray">, @doc="Word type headword - see pp.4-5", @name="head_word", @is_reference=true>
irb(main):007:0> f.length
=> 3
irb(main):008:0> f.class
=> Icss::RecordField
irb(main):009:0> f.keys
=> [:name, :doc, :type]

davidsnyder

OpenMenu API Wrapper

Check the documentation here http://openmenu.org/api/

require 'open_menu'

apikey = '1219-ab23-139b-de3-c23'
client = OpenMenu::Client.new(apikey)

menu_uuid = '12db-12b3-23bde-23bef'

davidsnyder

Factual API Wrapper

The ruby-factual gem is clunky and out of date, so I tossed together a script to access the new v3 API.

The v3 Factual API encourages 2-legged OAuth for requests. Request a key here:

http://developer.factual.com/display/docs/Factual+Developer+APIs+Version+3

require 'factual'

davidsnyder

‘foo.pig.erb’:

fips = LOAD '<%= in_path %>' AS (fips_id:int,state_name:chararray);
DUMP fips;

Ruby irb interpreter:

require 'swineherd'
script = Swineherd::Script.new('foo.pig.erb',{"in_path" => "fips_to_state.tsv"}) 
script.run(:run_mode => :local,:combine_splits => false,:map_tasks => 10,:param => {:foo => "bar"}) 
--> ENV['PIG_OPTS'] = '-Dmapred.map.tasks=10 -Dpig.splitCombination=false'

davidsnyder

-- 1,2000,Dinosaur Planet,3.74
-- 2,2000,Isle of Man TT 2004 Review,3.53
-- 3,1990,Character,3.64
-- 4,1990,Paula Abdul's Get Up & Dance,2.72
movie_decades   = LOAD '$DECADE_TITLES' USING PigStorage(',') AS (movie_id:chararray,decade:int,title:chararray,avg_rating:float);
snipped_decades = FOREACH movie_decades GENERATE movie_id,decade; 

-- 9983,462930,3
-- 9983,1149472,4
-- 9983,238407,4

davidsnyder

edges     = LOAD '$GRAPH' AS (user_id:chararray,movie_id:chararray);
edges_dup = LOAD '$GRAPH' AS (user_id:chararray,movie_id:chararray); -- hack for self-join

-- (user_id,|A|)
grouped_edges  = GROUP edges by user_id; --reduce 
outgoing_links = FOREACH grouped_edges GENERATE --map
                 group        AS user_id,
                 COUNT(edges) AS user_out;

-- (movie_id,user_a_id,movie_id,user_b_id)

davidsnyder

You can use the consumer script to verify that data is being received by the http listener (http_producer) on port 80 of the Kafka broker node.

sudo /usr/local/share/kafka/bin/kafka-console-consumer.sh --zookeeper [zk-private-ip]:2181 --topic [topic] --from-beginning

If you curl records at the kafka http listener, you should see them come out at the kafka broker node:

(On the Kafka listener node)

curl -XGET localhost:80 -d '{"name":"Bill","position":"Sales"}'

davidsnyder

java.lang.AbstractMethodError: com.infochimps.storm.trident.ElasticsearchStateFactory.makeState(Ljava/util/Map;Lbacktype/
Tue, 26          at storm.trident.planner.SubtopologyBolt.prepare(SubtopologyBolt.java:52)                                        
Nov 2013         at storm.trident.topology.TridentBoltExecutor.prepare(TridentBoltExecutor.java:214)                              
23:34:46         at backtype.storm.daemon.executor$fn__3557$fn__3569.invoke(executor.clj:659)                                     
+0000            at backtype.storm.util$async_loop$fn__436.invoke(util.clj:375)                                                   
                 at clojure.lang.AFn.run(AFn.java:24)                                                                             
                 at java.lang.Thread.run(Thread.java:662)                                                  

davidsnyder

 +---------------------------+-------+-------------+-----------+------------+-------------+-----------+
  | Name                      | Chef? | State       | Flavor    | AZ         | Env         | relevant? |
  +---------------------------+-------+-------------+-----------+------------+-------------+-----------+
  | p1_davidcontrol-dashpot-0 | no    | not running | m1.large  | us-east-1d | development | -         |
  | p1_davidcontrol-nfs-0     | no    | not running | c1.medium | us-east-1d | development | -         |
  | p1_davidcontrol-mongo-0   | no    | not running | m1.large  | us-east-1d | development | -         |
  | p1_davidcontrol-zabbix-0  | no    | not running | m1.large  | us-east-1d | development | -         |
  | p1_davidcontrol-vcd-0     | no    | not running | c1.xlarge | us-east-1d | development | -         |
  | p1_davidcontrol-backup-0  | no    | not running | m1.small  | us-east-1d | development | -         |
  +---------------------------+-------+-------------+-----------+------------+--