mrflip

TED 2014 Friday

Friday mid-Morning: Onward (final session)

Andrew Solomon, author

• Reports on experience of people in extreme circumstances
• Avoidance and Endurance
• Take traumas and make them part of who you'll be
• Mother of a child due to rape: I think of him (rapist) with pity -- he has a beautiful daughter he doesn't know, and I do, and so I’m the lucky one

mrflip

votes CF 

"back in black" => { 201005211200 => '1', 201005201159 => '1', 201005201157 => '1',  201005011900 => '1', 201004190600 => '1' },
"black album"   => { 201005021800 => '1', 201005010600 => '1' },
"black star"    => { 201005011000 => '1' }

cached_counts CF

"back in black" => { 'cached_count' => 2, 'counted_until' => 201005011931 },
"black album"   => { 'cached_count' => 1, 'counted_until' => 201005010600 }

mrflip

package com.infochimps.storm.trident;

import kafka.javaapi.producer.Producer;
import kafka.javaapi.producer.ProducerData;
import kafka.message.Message;
import kafka.serializer.Encoder;
import kafka.producer.ProducerConfig;
import backtype.storm.task.IMetricsContext;
import storm.trident.operation.TridentCollector;

mrflip

Performance Qualification

Identify all reasons why (eg) Elasticsearch cannot provide acceptable performance for standard requests and Qualifying load. The "Qualifying load" for each performance bound is double (or so) the worst-case scenario for that bound across all our current clients.

• Performance
  • bandwidth (rec/s, MB/s) and latency (s) for 100B, 1kB, 100kB records
  • under read, write, read/write
  • in degraded state: a) loss of one/two servers and recovering; b) elevated packet latency + drop rate between "regions"
  • High concurrency
• keepalive

mrflip

Things

"Big Five" == Elasticsearch, Storm, Kafka, HBase, wukong decorators

• Faster chef convergence (custom packages; local physical cluster)
• Centralized log archiving
• Performance qualification of
• Visibility and request manipulation
• Metarepo (deb/rpm, gem, egg, maven)

mrflip

-- correct a couple errors in the 2012 Baseball Databank (the 'January 9, 3:00 pm' release)

UPDATE `master`        SET `playerID` = 'baezjo01'  WHERE `lahmanID` = 460   AND `playerID` = 'baezda01';
UPDATE `master`        SET `bbrefID`  = 'snydech03' WHERE `lahmanID` = 19419 AND `playerID` = 'snydech03';
UPDATE `master`        SET `bbrefID`  = 'gilgahu01' WHERE `lahmanID` = 19417 AND `playerID` = 'gilgahu01';
UPDATE `AwardsPlayers` SET `playerID` = 'braunry02' WHERE `playerID` = 'braunry01' AND `awardID` = 'Silver Slugger' AND yearID = 2012 AND `lgID` = 'NL';
UPDATE `AwardsPlayers` SET `playerID` = 'brechha01' WHERE `playerID` = 'Brecheen'  AND `awardID` = 'Baseball Magazine All-Star';

mrflip

package com.infochimps.kafka.consumers;

import java.io.IOException;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import java.nio.ByteBuffer;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

mrflip

package com.infochimps.kafka.consumers;

import java.io.IOException;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import java.nio.ByteBuffer;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

mrflip

Outline

• Cube is awesome
  • streaming dashboards, yay
  • etsy says so
  • flexible, pretty, and streaming: meets our needs
  • what's so great?
    • metrics and events
      • event data
• metrics are calculation caches

mrflip

Analysis:

Assuming these goals, from most to least important:

1. that you, shaving finally learned (mostly) how to switch inputs, continue to have a system whose behavior you can predict
2. reasonable-quality sound in den and living room ...
3. ... while playing CDs
4. ... while playing radio
5. ---- line of essential ^^^ vs. important vvv ---
6. ... while playing songs from computer