Skip to content

Instantly share code, notes, and snippets.

View ashrithr's full-sized avatar
🎯
Focusing

Ashrith Mekala ashrithr

🎯
Focusing
137 followers · 9 following
View GitHub Profile
@ashrithr
ashrithr / centos7_static_ip.bash
Last active August 29, 2015 14:04
Centos 7 Static IP
## Get a list of NIC names using `ifconfig -a` or `ip -a`
#
# vi /etc/sysconfig/network-scripts/ifcfg-en0xxxxxxx
NAME=en0xxxxxx
NM_CONTROLLED=no
ONBOOT=yes
HWADDR=A4:BA:DB:37:F1:04
TYPE=Ethernet
BOOTPROTO=static
@ashrithr
ashrithr / cdh5_mr2_installation.md
Last active February 16, 2017 09:27
Installing YARN (MR2) On CentOS

###Pre-Req's:

Installing Cloudera Repo on all nodes that you'll install hadoop daemons on:

sudo cat > /etc/yum.repos.d/cloudera.repo <<EOF
[cloudera-cdh5]
# Packages for Cloudera's Distribution for Hadoop, Version 5, on RedHat	or CentOS 6 x86_64
name=Cloudera's Distribution for Hadoop, Version 5
baseurl=http://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5/
@ashrithr
ashrithr / benchmark-commands.txt
Created June 26, 2014 22:47 — forked from jkreps/benchmark-commands.txt
Producer
Setup
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test-rep-one --partitions 6 --replication-factor 1
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test --partitions 6 --replication-factor 3
Single thread, no replication
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 100 -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196
@ashrithr
ashrithr / ssh_tunneling.md
Last active July 22, 2022 21:25
ssh tunneling and port forwarding

###Single hop tunelling:

ssh -f -N -L 9906:127.0.0.1:3306 user@dev.example.com

where,

  • -f puts ssh in background
  • -N makes it not execute a remote command
@ashrithr
ashrithr / spark_on_yarn.md
Last active September 3, 2015 12:08
spark 0.9 on yarn (hadoop-2.2)

##Using yarn as the resource manager you can deploy spark application in two modes:

  1. yarn-standalone mode, in which your driver program is running as a thread of the yarn application master, which itself runs on one of the node managers in the cluster. The Yarn client just pulls status from the application master. This mode is same as a mapreduce job, where the MR application master coordinates the containers to run the map/reduce tasks.

With this mode, your application is actually run on the remote machine where the Application Master is run upon. Thus application that involve local interaction will not work well, e.g. spark-shell.

  1. yarn-client mode, in which your driver program is running on the yarn client where you type the command to submit the spark application (may not be a machine in the yarn cluster). In this mode, although the drive program is running on the client machine, the tasks are executed on the executors in the node managers of the YARN cluster.

Simply putting to gether:

@ashrithr
ashrithr / Usage.md
Created April 21, 2014 19:57
csshx usage

config file:

cat > aws_hosts <<EOF
clusters = kafka_cluster spark_cluster cass_cluster all
kafka_cluster = ec2-54-219-114-64.us-west-1.compute.amazonaws.com ec2-204-236-188-149.us-west-1.compute.amazonaws.com ec2-54-193-135-104.us-west-1.compute.amazonaws.com
spark_cluster = ec2-54-219-87-178.us-west-1.compute.amazonaws.com ec2-204-236-137-42.us-west-1.compute.amazonaws.com ec2-54-219-117-253.us-west-1.compute.amazonaws.com
cass_cluster = ec2-50-18-10-150.us-west-1.compute.amazonaws.com ec2-54-219-230-45.us-west-1.compute.amazonaws.com ec2-50-18-12-57.us-west-1.compute.amazonaws.com
all = ec2-54-219-114-64.us-west-1.compute.amazonaws.com ec2-204-236-188-149.us-west-1.compute.amazonaws.com ec2-54-193-135-104.us-west-1.compute.amazonaws.com ec2-54-219-87-178.us-west-1.compute.amazonaws.com ec2-204-236-137-42.us-west-1.compute.amazonaws.com ec2-54-219-117-253.us-west-1.compute.amazonaws.com ec2-50-18-10-150.us-west-1.compute.amazonaws.com ec2-54-219-230-45.us-west-1.compute.amazonaws.com ec2-50-18-12-57.us-
@ashrithr
ashrithr / Usage.md
Last active August 29, 2015 14:00
i2cssh

Config file:

cat > ~/.i2csshrc <<EOF
---
version: 2
clusters:
  kafka_cluster:
    login: ec2-user
 hosts:
@ashrithr
ashrithr / pm.scala
Last active August 29, 2015 14:00
scala pattern match on instances of the class
def test[T](t: T) = t match {
case _: Int => println("Got int" + t)
case _: String => println("Got string" + t)
case _ => println("I dont know what that is" + t)
}
@ashrithr
ashrithr / pom.xml
Created April 16, 2014 23:40
YCSB Hbase CDH5 pom YCSB/hbase/pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.yahoo.ycsb</groupId>
<artifactId>root</artifactId>
<version>0.1.4</version>
</parent>
<artifactId>hbase-binding</artifactId>
@ashrithr
ashrithr / detach_process.md
Created April 5, 2014 01:11
Send foreground process to background

Using the Job Control of bash to send the process into the background:

  1. ctrl+z to stop (pause) the program and get back to the shell
  2. bg to run it in the background
  3. disown -h [job-spec] where [job-spec] is the job number (like %1 for the first running job; find about your number with the jobs command) so that the job isn't killed when the terminal closes