fake log generator

README.md

This simulates cloud firewall log entries that are arriving as fast as a CPU can process them. The goal of this exercise was to understand how well the log compresses under such a circumstance. Oh, and I felt the need to write a rust program that didn't bear a striking resemblance to those found in "the book".

$ time ./target/release/logen | gzip --verbose >/dev/null
 95.2%

real	0m4.909s
user	0m7.587s
sys	0m1.917s
$ ./target/release/logen | gzip -9 --verbose >/dev/null
 95.3%
$ time ./target/release/logen | bzip2 --verbose >/dev/null
  (stdin): 33.624:1,  0.238 bits/byte, 97.03% saved, 297144041 in, 8837326 out.

real	0m45.347s
user	0m46.122s
sys	0m2.779s
$ time ./target/release/logen | xz --verbose >/dev/null
  100 %        11.4 MiB / 285.3 MiB = 0.040   3.9 MiB/s       1:12

real	1m12.719s
user	1m16.609s
sys	0m1.132s

gzip gives about 20x compression and gzip -9 does not improve it enough to matter. While bzip2 gives almost 33.6x compression, it also took 9 times longer. This translates in going from 200,000 events per second down to about 22,000 events per second. And don't even get me started on xz (lzma).

Cargo.toml

[package]
name = "logen"
version = "0.1.0"
authors = ["Mike Gerdts <mike.gerdts@joyent.com>"]

[dependencies]
chrono = "0.4"
rand = "0.6"
serde= "1.0"
serde_derive = "1.0"
serde_json = "1.0"

main.rs

extern crate chrono;
extern crate rand;
extern crate serde;
extern crate serde_json;
use chrono::prelude::*;
use rand::prelude::*;

#[macro_use]
extern crate serde_derive;

#[derive(Serialize, Deserialize)]
struct Entry {
    event: String,
    time: String,
    source_ip: String,
    source_port: u16,
    destination_ip: String,
    destination_port: u16,
    protocol: String,
    rule: String,
    vm: String,
    alias: String,
}

fn main() {
    let mut i = 1_000_000;
    let mut entry = Entry {
        event: String::from("allow"),
        time: String::from(""),
        source_ip: String::from(""),
        source_port: 0,
        destination_ip: rand_ipv4(),
        destination_port: rand_port(),
        protocol: String::from("tcp"),
        rule: String::from("7f789617-84c9-c7f6-8914-c0568cdbbbd5"),
        vm: String::from("260cd903-8692-47d3-93fa-eca8ec93c3b1"),
        alias: String::from("some-friendly-name-here")
    };

    while i != 0 {
        let utc: DateTime<Utc> = Utc::now();

        entry.time = utc.format("%Y-%m-%dT%H:%M:%S%.3fZ").to_string();
        entry.source_ip = rand_ipv4();
        entry.source_port = rand_port();
        println!("{}", serde_json::to_string(&entry).unwrap());
        i -= 1;
    }
}

fn rand_port() -> u16 {
    let mut rng = rand::thread_rng();

    rng.gen_range(1025, 65535)
}

fn rand_ipv4() -> String {
    let mut rng = rand::thread_rng();
    let ip: [u8; 4] = rng.gen();

    format!("{}.{}.{}.{}", ip[0], ip[1], ip[2], ip[3])
}