Anand Prakash anand086
anand086
/ parallel_session_pid
Created
February 17, 2022 08:38
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [ec2-user@ip-10-0-1-43 ~]$ nohup python3 cc-data-gen.py > session-1-cc-data-gen.log 2>&1 & | |
| [1] 10649 | |
| [ec2-user@ip-10-0-1-43 ~]$ ps -ef | grep cc-data-gen | |
| ec2-user 10649 1794 1 03:17 pts/0 00:00:02 python3 cc-data-gen.py | |
| ec2-user 10657 9736 1 03:17 pts/1 00:00:02 python3 cc-data-gen.py | |
| ec2-user 10665 9800 1 03:17 pts/2 00:00:02 python3 cc-data-gen.py | |
| ec2-user 10667 9858 1 03:17 pts/3 00:00:02 python3 cc-data-gen.py | |
| ec2-user 10675 9922 1 03:17 pts/4 00:00:02 python3 cc-data-gen.py |
anand086
/ record_count_per_partition
Created
February 17, 2022 08:37
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [ec2-user@ip-10-0-1-43 ~]$ kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list $BOOTSTRAP --topic transaction_events | awk -F ":" '{print $3}' | |
| 1 | |
| 2 | |
| 2 | |
| 2 | |
| 3 | |
| [ec2-user@ip-10-0-1-43 ~]$ |
anand086
/ create_kafka_topic
Created
February 17, 2022 08:36
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| kafka-topics.sh --create --zookeeper $ZOOKEEPER --replication-factor 2 --partitions 5 --topic transaction_events |
anand086
/ MSK-Connector-Transaction-Events-S3-Sink-Role-Trust-Relationship.json
Created
February 17, 2022 08:35
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "Version": "2012-10-17", | |
| "Statement": [ | |
| { | |
| "Effect": "Allow", | |
| "Principal": { | |
| "Service": "kafkaconnect.amazonaws.com" | |
| }, | |
| "Action": "sts:AssumeRole" | |
| } |
anand086
/ amazon_msk_connect_topics
Created
February 17, 2022 08:34
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| __amazon_msk_connect_status_lensesio-msk-transacton-events-src-s3-sink-iam_165f318b-28c2-46ac-a41c-bdc3e2dc3f87-3 | |
| __amazon_msk_connect_offsets_lensesio-msk-transacton-events-src-s3-sink-iam_165f318b-28c2-46ac-a41c-bdc3e2dc3f87-3 | |
| __amazon_msk_connect_configs_lensesio-msk-transacton-events-src-s3-sink-iam_165f318b-28c2-46ac-a41c-bdc3e2dc3f87-3 |
anand086
/ MSK-Connector-Transaction-Events-S3-Sink-Role-Trust-Relationship.json
Created
February 17, 2022 08:33
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "Version": "2012-10-17", | |
| "Statement": [ | |
| { | |
| "Effect": "Allow", | |
| "Principal": { | |
| "Service": "kafkaconnect.amazonaws.com" | |
| }, | |
| "Action": "sts:AssumeRole" | |
| } |
anand086
/ IAM-Auth-MSK-Transaction-Events-Inline-Policy.json
Last active
February 17, 2022 08:31
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "Version": "2012-10-17", | |
| "Statement": [ | |
| { | |
| "Effect": "Allow", | |
| "Action": [ | |
| "kafka-cluster:Connect", | |
| "kafka-cluster:DescribeCluster", | |
| "kafka-cluster:ReadData", | |
| "kafka-cluster:DescribeTopic" |
anand086
/ S3-Read-Write-Inline-Policy.json
Created
February 17, 2022 08:30
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "Version": "2012-10-17", | |
| "Statement": [ | |
| { | |
| "Effect": "Allow", | |
| "Action": [ | |
| "s3:Get*", | |
| "s3:List*", | |
| "s3:Put*" | |
| ], |
anand086
/ msk_lenses_connector_config
Created
February 17, 2022 08:29
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| connector.class=io.lenses.streamreactor.connect.aws.s3.sink.S3SinkConnector | |
| tasks.max=2 | |
| topics=transaction_events | |
| connect.s3.vhost.bucket=true | |
| schema.enable=false | |
| connect.s3.kcql=INSERT INTO learn-share-repeat-us-west-2:MSKBuildLabCluster SELECT * FROM transaction_events STOREAS `JSON` WITH_FLUSH_INTERVAL = 300 | |
| aws.region=us-west-2 | |
| aws.custom.endpoint=https://s3.us-west-2.amazonaws.com/ | |
| connect.s3.aws.region=us-west-2 | |
| key.converter.schemas.enable=false |
anand086
/ fake_transaction_record.json
Created
February 17, 2022 08:28
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "additionalData":{ | |
| "cardHolderName":"Cathy Martinez", | |
| "shopperCountry":"Cote d'Ivoire", | |
| "shopperIP":"177.244.104.76" | |
| }, | |
| "card_detail":{ | |
| "cardNumber":"4869141314562087", | |
| "provider":"Diners Club / Carte Blanche", | |
| "expiration":"10/22", |