A stream consumption tool.
Status: Vaporware
In 2015, stream-processing is the it-girl of software architecture. This sub-field is driven forward by a growing number of large-scale, analysis-oriented storage systems developed by open source communities and SaaS companies alike. Tooling, on the other hand, feels as though it's fallen by the wayside.
A "stream-processing engine" seems to comprise a clustered set of servers running a complicated application in a datacenter somewhere. Treating the engine as a black box, programs write data to it and are able to read values back, presumably via a vendor-provided client library.
But what if we want to inspect values passing through the stream? An SDK provides programmatic access, but is a long way off from out-of-the-box observability into the operation of a system. Rarely are legitimately-useful command-line programs provided for debugging and consuming streams.
This developer feels as if we're constructing a pristine edifice with limited regard for how occupants are supposed to interact with the new environment day-to-day.
Enter scat, a proposed command-line tool for interacting with
stream-processing systems. Along similar lines as existing tools such
as kafkacat and
aws kinesis get-records,
a developer uses scat to read stream contents directly from a
terminal. In this way, it can be thought of as cat for
streams-processing systems.
scat will initially target Amazon Kinesis.
A lofty goal of this project is for the tool to be adaptable to multiple stream-processing systems (e.g. Kafka, Flume, WhateverMQ).
$ scat -h
usage: scat -s <stream-name> [-r <aws-region>] [-q <seqno>] [-n <interval>]
Print messages from a user-events Kinesis stream in the us-west-2
region to stdout in real time:
$ scat -s user-events -r us-west-2
Print messages from a Kinesis stream starting at a given sequence number to stdout:
$ scat -s user-events -r us-west-2 -n 21267647932558653966460912964485513216
Filter GZIP'd messages from a Kinesis stream and write them to a file:
$ scat -s user-events -r us-west-2 | gunzip | grep 'user_id:1583331' > /var/log/my-user-events.log