cameronkerrnz

Logstash 7.14 introduces the concept of the databasemanager into logstash-filter-geoip, which downloads Maxmind GeoIP databases in accordance with Maxmind licencing terms. This does not work behind a proxy. Thus, I updated my deployment with the ability to download install the newer (more appropriate) version of geoipupdate from MaxMind, and configure that to download the databases.

A better (more scalable, easier on upstream) would be to have this on one machine, and then share out the resulting databases internally, on a recurring schedule.

This is an extract of a deployment Ansible (2.10ish) targetting RHEL7 / CentOS7, and while its use-case is to support Logstash, there is nothing in this gist that is specifically about Logstash.

cameronkerrnz

I've been working on improving operational visibility of a system that uses Kafka as its data backbone that have a combination of size and age-based retention policies [with no compaction]. One of the key metrics I want to track is how much time is actually held in some of these topics, before I start making changes, as a longer amount of time means I have a longer range of recovery if I need to reprocess that data. Although kafka_exporter does give me useful visibility of topic lag, Kafka doesn't provide any tools specifically for reporting on the timestamp of the earliest message, so with the aid of kafka-console-consumer, some basic scripting, Prometheus and Grafana, now I have the visibility I need, and could set up some useful thresholds for alerting.

To use with Grafana

To use this in a Grafana dashboard, here's a sample Prometheus query you can use as part of a Singlestat (format the Singlestat with a unit of 'seconds')

time() - max(kafka_topic_partition_earliest_time_seconds{topic=~

cameronkerrnz

Get-Content -Head 5 c:\windows\system32\LogFiles\Firewall\pfirewall.log

Get-Content -Wait -Tail 5 C:\Windows\System32\LogFiles\Firewall\pfirewall.log | % {
    do { 
        $a=$_.split(' ')

        # DROP or ACCEPT (the only values AFAIK)
        #
        if ($a[2] -ne 'DROP') {continue}