vikasbajaj

0. Make sure you have your Cloud9 environment up and running from the previous lab

1. Create ES domain
    - Development type = Development and Testing
    - Elasticsearch Version = 7.10
    - ES Domain name = <Name>
    - Auto Tune = Disable
    - Data Nodes 
        Instance Type = t3.small.elasticsearch
        Number of nodes = 1

vikasbajaj

0. Create an IAM Role and attach the following managed policies with it.
- IAMFullAccess
- AmazonDynamoDBFullAccess
- AmazonSNSFullAccess
- AWSLambda_FullAccess
1. Create cloud 9 environment and attach IAM role that was created in the previous step with the Cloud9 EC2 instance. Open Cloud 9 environment

2. Switch of AWS managed temporary credentials in cloud 9 and assign role created by the Cloudformation stack
    - Set a default region to ap-southeast-2

vikasbajaj

Lab-1: Cluster Creation
---------------------------
1. MSK Workshop link = https://amazonmsk-labs.workshop.aws/en/clustercreation.html

2. Use the following cloudformation template, download it on your laptop 

https://github.com/vikasbajaj/msk-kafka-workshop/blob/master/msk-infra-and-kafka-clients/MSK-VPC-Clients.yaml

3. Make sure you are in running this lab in ap-southeast-2 (sydney) region

vikasbajaj

Lab-3: MSK with Glue Schema Registery
-------------------------------------

1. Go to Cloud9 console and open your environment IDE
2. In a Cloud 9 terminal use the following command to ssh into Kafka EC2 instance

    Note: change the IP address with Kafka EC2 instance private IP address running in your AWS account
    
    ssh -i msk-workshop-pem.pem ec2-user@10.0.1.124

vikasbajaj

Lab-2: Cluster Expansion
------------------------

1. Go to Cloud9 console and open your environment IDE
2. In a Cloud 9 terminal use the following command to ssh into Kafka EC2 instance

    Note: change the IP address with Kafka EC2 instance private IP address running in your AWS account
    
    ssh -i msk-workshop-pem.pem ec2-user@10.0.1.124

vikasbajaj

zookeeper-server-start.sh config/zookeeper.properties
kafka-server-start.sh config/server.properties
kafka-topics.sh --bootstrap-server localhost:9092 --list
kafka-topics.sh --bootstrap-server localhost:9092 --topic test-topic --create --partitions 3 --replication-factor 1
kafka-topics.sh --bootstrap-server localhost:9092 --topic test-topic --describe
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning --group group-1
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-topic

vikasbajaj

AWSTemplateFormatVersion: '2010-09-09'
Description: This template deploys IAM service control policies required for managing AWS organizations.
Resources:
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      Description: IAM Role for lambda to access Redshift
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
        - arn:aws:iam::aws:policy/AWSOrganizationsFullAccess

vikasbajaj

Description: "AWS Streaming Data APIGWv2 -> KDS -> kdf -> s3"
Parameters:
  ShardCount:
    Type: Number
    Default: 2
    MaxValue: 20
    MinValue: 1
  RetentionHours:
    Type: Number
    Default: 24

vikasbajaj

#!/usr/bin/python3
import requests
import datetime
import time
import threading
import random

# replace URL string with Invoke URL (APIGW V2 --> Invoke URL)
url = "https://xxxxxxxxxxxxxxxxxxxxxxxxx.amazonaws.com/beta"
for i in range(10000):