vikasbajaj

Lab-1: Cluster Creation
---------------------------
1. MSK Workshop link = https://amazonmsk-labs.workshop.aws/en/clustercreation.html

2. Use the following cloudformation template, download it on your laptop 

https://github.com/vikasbajaj/msk-kafka-workshop/blob/master/msk-infra-and-kafka-clients/MSK-VPC-Clients.yaml

3. Make sure you are in running this lab in ap-southeast-2 (sydney) region

vikasbajaj

Lab-2: Cluster Expansion
------------------------

1. Go to Cloud9 console and open your environment IDE
2. In a Cloud 9 terminal use the following command to ssh into Kafka EC2 instance

    Note: change the IP address with Kafka EC2 instance private IP address running in your AWS account
    
    ssh -i msk-workshop-pem.pem ec2-user@10.0.1.124

eladroz

I'm now working on big data processing with Pandas at scale, as a lightweight alternative to Spark. Fortunately, the Apache Arrow project brings with it an excellent and very fast Parquet reader and writer.

With the current push to ARM in both personal computers and the data center, I was curious to check the performance of my code on ARM - running on AWS' homegrown Graviton2 processor. Their c6g instance types are 20% cheaper than the equivalent Intel-based c5's, while promising faster performance. If that's the future, why not start getting ready now?

While there are already Python wheels for NumPy and Pandas, there is no official build yet for PyArrow. There's a pull request in the works,

syntaqx

#cloud-config

# Option 1 - Full installation using cURL
package_update: true
package_upgrade: true

groups:
  - docker

system_info:

kuczmama

# Inspired by http://www.madebyloren.com/posts/migrating-to-uuids-as-primary-keys

task id_to_uuid: :environment do
  puts "[START] Convert id to uuid"
  ActiveRecord::Base.connection.enable_extension 'uuid-ossp' unless ActiveRecord::Base.connection.extensions.include? 'uuid-ossp'
  ActiveRecord::Base.connection.enable_extension 'pgcrypto' unless ActiveRecord::Base.connection.extensions.include? 'pgcrypto'

  table_names = ActiveRecord::Base.connection.tables - ["schema_migrations", "ar_internal_metadata", "migration_validators"]
  table_names.each do |table_name|
    puts "[CREATE] uuid column for #{table_name}"

daboross

# daboross's kakrc
#
# ###
#
# Copyright (c) 2019 David Ross
#
# This software is provided 'as-is', without any express or implied
# warranty. In no event will the authors be held liable for any damages
# arising from the use of this software.
#

privatwolke

async def gather_dict(tasks: dict):
    async def mark(key, coro):
        return key, await coro

    return {
        key: result
        for key, result in await gather(
            *(mark(key, coro) for key, coro in tasks.items())
        )
    }

itzg

Here is how to create a cloud-init disk image and OS disk image suitable for configuring into a libvirt domain file.

In my case I am naming my domain (a.k.a. virtual machine or VM) xenial with a static IP address of 192.168.0.101. The filenames "network-config" and "user-data" files are arbitrary, so they can be named with a prefix for the domain, etc.

First, get the cloud image and convert into QCOW2 format:

qemu-img convert -O qcow2 xenial-server-cloudimg-amd64-disk1.img xenial-server-cloudimg-amd64-disk1.qcow2

jag3773

[root@ip-172-26-177-115 ~]# gluster volume info
 
Volume Name: supportgfs
Type: Distributed-Replicate
Volume ID: 695f6857-de4a-441f-bbf1-a57ec047eea6
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 172.26.178.4:/media/ephemeral0/supportgfs-readonly

kachayev

# Simple:
# a --> b
#   --> c --> d
#   --> d 
graph1 = {
    "a": ["b", "c", "d"],
    "b": [],
    "c": ["d"],
    "d": []
}