-
-
Save gnzjgo/f1a80186a301cd8770a946d02343bafd to your computer and use it in GitHub Desktop.
Sample of how to send dummy data to a Kinesis data stream
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
""" | |
Kinesis to Tinybird demo | |
Complement to this guide(tbd). | |
Create a virtual environment and install the dependencies. | |
``` | |
python3 -mvenv .e | |
. .e/bin/activate | |
pip install boto3 | |
``` | |
Be sure your ~/.aws/credentials and ~/.aws/config files are OK according to [boto3 docs](https://github.com/boto/boto3#getting-started), that your account has the right privileges. | |
Start sending dummy data: | |
``` | |
python3 send_to_kinesis.py | |
``` | |
""" | |
import json | |
import boto3 | |
from datetime import datetime | |
import time | |
import random | |
STREAM_NAME = "demo-tb-kinesis-stream" | |
products = ["sZzx0cUDX98","5d0cgAl5BTk","YY4YaHKh2jQ","p8Drpg_duLw","xFmXLq_KJxg","fSdBxY0NxVI","6cHumpSxTvs","Zu7A1GCSjZE","Fg15LdqpWrs"] | |
events = ["view","cart","sale"] | |
events_weights = [60, 33, 24] | |
while True: | |
event = { | |
'datetime': datetime.utcnow().isoformat(), | |
'event': random.choices(events, weights=events_weights,k=1)[0], | |
'product': random.choice(products) | |
} | |
print(event) | |
boto3.client('kinesis').put_record( | |
StreamName=STREAM_NAME, | |
Data=json.dumps(event), | |
PartitionKey="partitionkey") | |
time.sleep(0.1) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment