blep

README.md

Blep

A high availability log sourcing tool for modern DevOps

Our implementation

---

• Decentralizing calls to a distributed log-store, rqlite
• Fast logs sourcing using NATS and gRPC
• Low bottlenecking
• Abstraction between persistent (subscriber) and application (publisher) layer
• Easily deployable and testable image build
• Easy integration with an existing ELK stack
• Seamless integration across different codebases
• Self container binaries
• Inbuilt analytics and benchmarks
• Mormon CLI interface

Why NATS?

---

NATS is an event sourcing tool which we will be using to publish logs and distribute related data between different services. The reason for NATS is:

• RabbitMQ is limited to HTTP and HTTPS natively

• NATS supports gRPC and works fast due to being type safe as well as removing extra overhead of marshalling and unmarshalling

• Publishing events on a different thread and subscribing from a different thread allows non-blocking log sourcing.

• NATS is very high throughput when it comes to requests per second

The need for a distributed DB

---

Different containers consuming from a single DB lead to network bottlenecking. distributed databases give us the advantage in terms of reducing request overhead. In this implementation we have used rqlite, because:

• It is very lightweight, and written in go

• Forming a cluster is very easy.

• It is fast and fault tolerant

• It has an inbuilt backup and recovery mechanism

Introducing the mormon CLI

usage: mormon [subcommand] [args]
mormon build <----------------------------------------------------------> build from source, go needed
mormon run-env <--------------------------------------------------------> run the nats and rqlite env
mormon run-safe-env <---------------------------------------------------> run-env for the case docker-compose is not present
mormon run-env stop <---------------------------------------------------> stop the nats and rqlite env
mormon start -p [port] <------------------------------------------------> stop the nats and rqlite env
mormon publisher [logId] [timestamp] [log] [host] <---------------------> publish an event
mormon subscriber logs.[host] [callback executible] <-------------------> subscribe for an event
mormon mock-logs <------------------------------------------------------> generate logs for testing

Project instructions

Build environment specification

Run NATS and rqlite

git clone https://angadsharma1016/mormon
cd mormon
chmod +x bin/*
./bin/build
./bin/run-env

Stop both

./bin/run-env stop

Run server

./bin/start -p 3000

Publisher side

./bin/publisher "[log ID]" "[Timestamp]" "[Log]" "[Host]"

Subscriber side

On the occurence of the event, subscriber will run ./bin/callback

./bin/subscriber "[event name]" "[callback executible]"