pnc

Kinesis Firehose Survival Guide

Avoid regular Kinesis if you can

It's every bit as complicated to build software against as Apache Kafka and reasoning about repartioning, retry, and failover is a huge pain and only matters if you have TB/s of data. Kinesis Firehose will happily push many, many MB/s or maybe even GB/s without the complexity.

Kinesis Firehose is pay-by-the-drink (price per byte) and pretty darn cheap. Real Kinesis (price per hour) gets expensive fast, since it involves provisioned infrastructure. Ditto Kinesis Data Analytics. One more reason not to use it.

If you find yourself writing a Kinesis Firehose consumer, run screaming into the night. Here is the first half of the steps to connect to Kinesis and consume data:

The KCL acts as an intermediary between your record processing logic and Kinesis Data Streams. The KCL performs the following tasks:

pnc

# Simulate a fairly realistic "bad network" (e.g. bufferbloat,
# flaky Wi-Fi, LTE from a brick building) to help find bugs
# in code that never expects network responses to arrive out of order.

# Run with `sudo ruby network-chaos.rb` and stop with control-C.
# Test with:
#
#       $ ping 1.1.1.1
#       PING 1.1.1.1 (1.1.1.1): 56 data bytes
#       Request timeout for icmp_seq 0

pnc

This one is a real challenge! The wide design of the Grand Army Plaza circle, combined with fairly long light cycles, permits drivers to feel safe going 40-50mph, and brings them off the circle onto Vanderbilt (and into this intersection) at speed. This intersection marks the start of significant bicycle (rare and unexpected to motorists two-way bike traffic on Plaza St E) and pedestrian (four crosswalks) activity not present on the stretch of Grand Army Plaza circle from the last light until this light. This shift from parkway/highway-like design in the traffic circle to residential street design surprises drivers, and they often have no time to adjust their speed appropriately when the light is green.

Additional issues at this intersection:

1. Drivers at speed going northbound cannot see pedestrians or cyclists entering the intersection from the east side due to the advertising portion of the MTA bus stop and the allocation of the corner parking spot. Physically blocking this with a low planter or parking

pnc

Prepare to spend a little cash on a wild ride. You are about to go from an expensive, Jet-A-enriched glass-and-steel shopping mall to an acoustic-tile-ceilinged underground train pit with a bowling alley on top.

Load up Google Maps and pick the result that has you taking AirTrain to Jamaica, and then a cluster of colorful names including Babylon, Hempstead, Long Beach, Port Jefferson, and Ronkonkoma to Penn Station.

🍹 True fact: Ronkonkoma isn't a real place!

Once in the baggage area of any of the terminals follow the signs for Ground Transportation and then AirTrain. Board (ticketless) a Jamaica Station-bound AirTrain. Do not board a Howard Beach-bound AirTrain. Once at Jamaica Station, swarm with the others to purchase a MetroCard of at least $5 value. I recommend just getting a $20 (plus $1.4536̅6̅ bonus!!!) card for your time here. The UI will try to create a false distinction between an "AirTrain MetroCard" and a normal MetroCard. Do not be fooled. You simply need a card with at least $5 v

pnc

npm install --save --build-from-source canvas                                                                                       on master 

> canvas@2.3.1 install /Users/phil/Development/creator/node_modules/canvas
> node-pre-gyp install --fallback-to-build

node-pre-gyp WARN Using needle for node-pre-gyp https download 
gyp ERR! configure error 
gyp ERR! stack Error: Command failed: /usr/local/opt/python/libexec/bin/python -c import sys; print "%s.%s.%s" % sys.version_info[:3];
gyp ERR! stack   File "<string>", line 1
gyp ERR! stack     import sys; print "%s.%s.%s" % sys.version_info[:3];

pnc

#!/bin/bash

set -e

CONTENTS=$(tesseract -c language_model_penalty_non_dict_word=0.8 --tessdata-dir /usr/local/share/tessdata/ "$1" stdout -l eng | xml esc)

hex=$((cat <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">

pnc

phil@Supernova > java -jar lib/swagger-codegen-cli.jar version                             
3.0.0-SNAPSHOT
phil@Supernova > java -jar lib/swagger-codegen-cli.jar generate -v -i specs/client/petstore.yaml -l swagger -o output/client/petstore
[main] INFO io.swagger.codegen.config.CodegenConfigurator - 
VERBOSE MODE: ON. Additional debug options are injected
 - [debugSwagger] prints the swagger specification as interpreted by the codegen
 - [debugModels] prints models passed to the template engine
 - [debugOperations] prints operations passed to the template engine
 - [debugSupportingFiles] prints additional data passed to the template engine
[main] INFO io.swagger.parser.Swagger20Parser - reading from specs/client/petstore.yaml

pnc

### Keybase proof

I hereby claim:

  * I am pnc on github.
  * I am pnc (https://keybase.io/pnc) on keybase.
  * I have a public key whose fingerprint is CAE3 A08A 3CFB 5042 82BD  80B8 37B7 21F0 DAE6 961F

To claim this, I am signing this object:

pnc

Just about the only thing that'll crash the Erlang VM is running out of memory. Although processes themselves are cheap, a good way to run out of memory is to start an unbounded number of processes from an external signal (such as one per web request), and then make each of those processes hold onto some more expensive resource. If you're not careful about cleanup, you can also leak processes, which compounds the problem. Anyway, nobody's perfect, so to avoid these cases, we use the poolboy library in a bunch of places to handle process pools that handle cleanup and limit the total amount of concurrency to some reasonable amount.

When:

1. all a pool's workers are checked out (by actual organic load/concurrency)
2. some process is leaking pool workers by not checking them back in (rude)
3. the pool supervisor isn't responding (it's deadlocked, bugs, etc.)

you'll see "checkout" timeouts (waited longer than the timeout to check out a worker from the pool):

pnc

AWS Lambda allows you to run custom binaries as child processes.

However, if the packaged binary you're running on AWS Lambda uses shared libraries, they may not be available in the Lambda environment. If this is the case, your binary will terminate without any output. In my case, the exit status code was 127, which wasn't very helpful (typically this is "command not found.")

2015-11-18T00:50:10.731Z	521db901-8d8e-11e5-b9df-cd31cc90ece2	Calling phantom:  /var/task/phantomjs [ '/var/task/phantomjs-script.js' ]
2015-11-18T00:50:10.809Z	521db901-8d8e-11e5-b9df-cd31cc90ece2	child process exited with code 127

Linux's loader, ld.so, allows you (see manpage) to set an environment variable called LD_DEBUG that will output verbose information while the shared libraries are loaded.

Since Lambda doesn't let you set arbitrary environment variables, you need to set the environment