Skip to content

Instantly share code, notes, and snippets.

@ayakout
Forked from jkvor/gist:929428
Created April 21, 2011 05:33
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ayakout/933773 to your computer and use it in GitHub Desktop.
Save ayakout/933773 to your computer and use it in GitHub Desktop.

Overview

Logplex is the Heroku logging multiplexor. Components throughout the Heroku platform publish syslog packets that are received and processed by a grid of logplex nodes. Users may retrieve their logs through requests to the logplex API.

Logplex relies on channel, token, drain and session objects. This data is stored in redis, but also cached for fast lookups in a normalized format in ETS tables. When logplex boots, all of the keys are loaded from redis and cached in ETS. When an object is created or deleted through the logplex API, other logplex nodes are notified of the change (via Erlang rpc calls) so that they may update their local ETS cache. We would like to explore the possibility of using nsync to manage replicating the dataset into ETS. This would speed up logplex boot times and save us from making Erlang RPC calls to update local caches on remote nodes.

To get started, fork logplex on github and follow the directions below to get logplex running locally.

Once you are up and running, add nsync as an application dependency and ensure nsync is successfully replicating from the master redis when logplex boots.

Next, take a shot at refactoring logplex_worker to perform the token -> channel lookup from the nsync ETS cache. After that, find the other places in the code that ETS reads are performed and switch those to read from the nsync ETS cache. Then remove the code that loads the ETS tables from redis on boot and remove the code that publishes cache updates via RPC.

Repos

Logplex: https://github.com/heroku/logplex
Nsync https://github.com/JacobVorreuter/nsync

Running logplex locally

TERMINAL 1

Boot redis server

$ echo "timeout 0" | ./redis-server -
[83656] 19 Apr 12:49:08 * Server started, Redis version 2.3.0
[83656] 19 Apr 12:49:08 * DB loaded from disk: 0 seconds
[83656] 19 Apr 12:49:08 * The server is now ready to accept connections on port 6379

TERMINAL 2

Compile and boot logplex

$ cd logplex
$ git submodule update --init
$ make
$ bin/console
=PROGRESS REPORT==== 19-Apr-2011::12:47:49 ===
         application: logplex
Eshell V5.8.3  (abort with ^G)
1>

TERMINAL 3

Create a channel (channel_id == 1)

$ curl -H "Authorization: secret" -d "{\"name\": \"foo\", \"app_id\": 1000, \"addon\": \"basic\"}" http://localhost:8002/channels
1

Create two tokens associated with this channel

$ curl -H "Authorization: secret" -d "{\"name\": \"heroku\"}" http://localhost:8002/channels/1/token
t.AFBE76CE-0726-4B26-9309-5318B5906B7E
$ curl -H "Authorization: secret" -d "{\"name\": \"app\"}" http://localhost:8002/channels/1/token
t.35268A1A-B252-4BCE-9CBD-1E6F80E06159

TERMINAL 2

Verify that our channel and tokens were created and cached in logplex

2> ets:tab2list(logplex_channel).
[{channel,1,<<"foo">>,1000,<<"basic">>}]
3> ets:tab2list(logplex_token).
[{token,<<"t.35268A1A-B252-4BCE-9CBD-1E6F80E06159">>,1,
        <<"app">>,1000,<<"basic">>},
 {token,<<"t.AFBE76CE-0726-4B26-9309-5318B5906B7E">>,1,
        <<"heroku">>,1000,<<"basic">>}]

TERMINAL 3

Publish some log packets containing our newly created tokens to logplex via UDP

$ ncat -u 127.0.0.1 9999
<40>1 2010-11-10T17:16:33-08:00 localhost t.35268A1A-B252-4BCE-9CBD-1E6F80E06159 web.1 - - State changed from created to starting
<40>1 2010-11-10T17:17:33-08:00 localhost t.AFBE76CE-0726-4B26-9309-5318B5906B7E router - - unidling web.1
^C

Create a session and retrieve our logs

$ curl -H "Authorization: secret" -d "{\"channel_id\": \"1\"}" http://localhost:8002/sessions
/sessions/9610A4BB-FF1A-45CE-B372-8D6354BBFA4A
$ curl -H "Authorization: secret" http://localhost:8002/sessions/9610A4BB-FF1A-45CE-B372-8D6354BBFA4A
2010-11-10T17:16:33-08:00 app[web.1]: State changed from created to starting
2010-11-10T17:17:33-08:00 heroku[router]: unidling web.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment