vikaslalwani

Heap dumping a Java process running in AWS ECS

Note: this guide is designed for AWS ECS services, but starting from Step 4 is functionally equivalent to any Docker container on a Linux host.

Step 1. Look up ECS service

1. Log into the AWS Console using the appropriate AWS account
2. Navigate to AWS ECS service clusters (https://console.aws.amazon.com/ecs/home)
3. Make sure you are in the correct region, if not, switch to the correct region (second drop-down menu in top right corner)
4. Select the correct cluster (ex: https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters//services)
5. In the Services tab, In the 'Filter in this page' text box, type the name of the service

vikaslalwani

openssl genrsa -out CAroot.key 2048
openssl req -new -key CAroot.key -out CAroot.csr # CN should be different from the certificates below
openssl req -x509 -days 1825 -key CAroot.key -in CAroot.csr -out CAroot.crt
cat CAroot.crt CAroot.key > CAroot.pem

openssl genrsa -out mongod.key 2048
openssl req -new -key mongod.key -out mongod.csr
openssl x509 -req -days 1825 -in mongod.csr -CA CAroot.pem -CAkey CAroot.key -CAcreateserial -out mongod.crt
cat mongod.crt mongod.key > mongod.pem

vikaslalwani

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *

vikaslalwani

import numpy as np
import tensorflow as tf
import tensorflow_io.kafka as kafka_io

# 1. Consume streaming data with Kafka and TensorFlow I/O
def func_x(x):
  # Decode image to (28, 28)
  x = tf.io.decode_raw(x, out_type=tf.uint8)
  x = tf.reshape(x, [28, 28])
  # Convert to float32 for tf.keras

vikaslalwani

// Continuously aggregating a KStream into a KTable.
KStream<String, String> locationUpdatesStream = ...;

KTable<String, Long> locationsPerUser
   = locationUpdatesStream
       .groupBy((k, v) -> v.username)
       .count();	

vikaslalwani

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *

vikaslalwani

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *

vikaslalwani

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *

vikaslalwani

Producer

Setup
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test-rep-one --partitions 6 --replication-factor 1
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test --partitions 6 --replication-factor 3

Single thread, no replication

bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 100 -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196

vikaslalwani

import multiprocessing

# split a list into evenly sized chunks
def chunks(l, n):
    return [l[i:i+n] for i in range(0, len(l), n)]


def do_job(job_id, data_slice):
    for item in data_slice:
        print "job", job_id, item