mrflip

package com.infochimps.storm.trident;

import kafka.javaapi.producer.Producer;
import kafka.javaapi.producer.ProducerData;
import kafka.message.Message;
import kafka.serializer.Encoder;
import kafka.producer.ProducerConfig;
import backtype.storm.task.IMetricsContext;
import storm.trident.operation.TridentCollector;

mrflip

Next Steps

• Measure time spend on index, flush, refresh, merge, query, etc. (TD - done)
• Take hot threads snapshots under read+write, read-only, write-only (TD - done)
• Adjust refresh time to 10s (from 1s) and see how load changes (TD)
• Measure time of a rolling restart doing disable_flush and disable_recovery (TD)
• Specify routing on query -- make it choose same node for each shard each time (MD)
• GC new generation size (TD)
• Warmers

• measure before/after of client query time with and without warmers (MD)

mrflip

Performance Qualification

Identify all reasons why (eg) Elasticsearch cannot provide acceptable performance for standard requests and Qualifying load. The "Qualifying load" for each performance bound is double (or so) the worst-case scenario for that bound across all our current clients.

• Performance
  • bandwidth (rec/s, MB/s) and latency (s) for 100B, 1kB, 100kB records
  • under read, write, read/write
  • in degraded state: a) loss of one/two servers and recovering; b) elevated packet latency + drop rate between "regions"
  • High concurrency
• keepalive

mrflip

Things

"Big Five" == Elasticsearch, Storm, Kafka, HBase, wukong decorators

• Faster chef convergence (custom packages; local physical cluster)
• Centralized log archiving
• Performance qualification of
• Visibility and request manipulation
• Metarepo (deb/rpm, gem, egg, maven)

mrflip

-- correct a couple errors in the 2012 Baseball Databank (the 'January 9, 3:00 pm' release)

UPDATE `master`        SET `playerID` = 'baezjo01'  WHERE `lahmanID` = 460   AND `playerID` = 'baezda01';
UPDATE `master`        SET `bbrefID`  = 'snydech03' WHERE `lahmanID` = 19419 AND `playerID` = 'snydech03';
UPDATE `master`        SET `bbrefID`  = 'gilgahu01' WHERE `lahmanID` = 19417 AND `playerID` = 'gilgahu01';
UPDATE `AwardsPlayers` SET `playerID` = 'braunry02' WHERE `playerID` = 'braunry01' AND `awardID` = 'Silver Slugger' AND yearID = 2012 AND `lgID` = 'NL';
UPDATE `AwardsPlayers` SET `playerID` = 'brechha01' WHERE `playerID` = 'Brecheen'  AND `awardID` = 'Baseball Magazine All-Star';

mrflip

package com.infochimps.kafka.consumers;

import java.io.IOException;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import java.nio.ByteBuffer;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

mrflip

package com.infochimps.kafka.consumers;

import java.io.IOException;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import java.nio.ByteBuffer;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

mrflip

Outline

• Cube is awesome
  • streaming dashboards, yay
  • etsy says so
  • flexible, pretty, and streaming: meets our needs
  • what's so great?
    • metrics and events
      • event data
• metrics are calculation caches

mrflip

Analysis:

Assuming these goals, from most to least important:

1. that you, shaving finally learned (mostly) how to switch inputs, continue to have a system whose behavior you can predict
2. reasonable-quality sound in den and living room ...
3. ... while playing CDs
4. ... while playing radio
5. ---- line of essential ^^^ vs. important vvv ---
6. ... while playing songs from computer

mrflip

From this Language Log post about the mysterious language on two postcards from the 1910's --

• postcard 1
• postcard 2