Skip to content

Instantly share code, notes, and snippets.

@JoeKarlsson
Last active April 15, 2024 21:41
Show Gist options
  • Save JoeKarlsson/19a7d5386e21e81179dc7b6be4d16c8e to your computer and use it in GitHub Desktop.
Save JoeKarlsson/19a7d5386e21e81179dc7b6be4d16c8e to your computer and use it in GitHub Desktop.
Script for infra-monitoring-demo for Tinybird

Slack Status

Real-Time Infrastructure Monitoring with Tinybird

This demo shows you how to build an infrastructure monitoring tool, similar to a mini-datadog, that keeps track of some stats from all of the computers running this included script. Using this data, we can plot time series and build something based on "real" data being captured in the moment.

The script

This script essentially is a Python script that gathers some information (what's your OS - e.g: Windows, macOS or Ubuntu), how many CPUs you have, available memory and available disk and sends an event every second with it.

This is how the messages look:

{
    'timestamp': '2023-02-21 03:11:31.709',
    'nickname': 'Sancha',
    'team': 'Tinybird',
    'text': 'Good luck!',
    'os': 'Darwin',
    'cpus': 8,
    'mem_size': 34359738368,
    'mem_free': 15164309504,
    'disk_size': 494384795648,
    'disk_free': 112696381440
}

I'll build a time series in Tinybird that plots free memory over time per OS and/or number of CPUS, and a couple of endpoints that expose the data.

  1. Save in a local folder

  2. Go that folder in the terminal and run the following:

python3 -m venv .venv
source .venv/bin/activate
pip install pytz kafka-python psutil confluent-kafka
  1. Configure the script

Open the index.py and customize lines 22 and 24:

your_nickname = 'Joe' #enter a nickname
message_to_tinybird = 'Good luck!' #Or say something funny, but keep it tidy... 8)
  1. Run the script

In a terminal, run the following:

python3 index.py

Note: If you are running this on macOS, be sure to run the following to prevent the computer from sleeping:

caffeinate -s python3 index.py

If you are in Ubuntu, you probably know better than me how to prevent the computer from sleeping. If you are in Windows, your guess is as good as mine.

The script will sleep until an hour before the start of the demo. Just leave it running. Once the time of the event has past, the script will stop automatically.

Authors


Need help?: Community SlackTinybird DocsEmail

import time
from datetime import datetime
import pytz
import psutil
import platform
import json
from confluent_kafka import Producer
from kafka import KafkaProducer
def send_data_to_kafka():
producer = KafkaProducer(
bootstrap_servers=['pkc-l6wr6.europe-west2.gcp.confluent.cloud:9092'],
sasl_mechanism='PLAIN',
security_protocol='SASL_SSL',
sasl_plain_username='UQR4M6LACV5WWD7E',
sasl_plain_password='9w/zYWlOzMJePfyNleAfAYjgJ19UgqWDZe51/I9zs1KKsUGQ0skk+Su+CwQuM1rT',
value_serializer=lambda x: json.dumps(x).encode('utf-8')
)
print('Sending events...')
your_nickname = 'Joe Karlsson' # enter a nickname
# Say something funny, but keep it clean...
message_to_tinybird = 'I love Kafka and Tinybird'
TOPIC = 'infra-monitoring-demo'
# Convert the start and end times from local time to ET
ny_tz = pytz.timezone('America/New_York')
# Send a message every second
while True:
# Get the current time in ET
timestamp = ny_tz.localize(datetime.now()).strftime(
'%Y-%m-%d %H:%M:%S.%f')[:-3]
message = {}
message['timestamp'] = timestamp
message['nickname'] = your_nickname
message['team'] = 'Tinybird'
message['text'] = message_to_tinybird
message['os'] = platform.system()
message['cpus'] = psutil.cpu_count()
mem = psutil.virtual_memory()
message['mem_size'] = mem.total
message['mem_free'] = mem.available
disk = psutil.disk_usage('/')
message['disk_size'] = disk.total
message['disk_free'] = disk.free
# Send the message to Kafka
producer.send(TOPIC, value=message)
# Wait for a few seconds before sending the next message
print(message)
time.sleep(1)
send_data_to_kafka()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment