slgithub

GIT Everyday Command Brushup

http://aayushtuladhar.wordpress.com/2012/01/19/git-brushup/

####Git Setup - Configuring Username/Email

git config --global user.name "Aayush Tuladhar"'
git config --global user.email "aayush.tuladhar@gmail.com"'

####Creating an empty GIT repository

slgithub

DAEMON_PATH=/opt/kafka/bin
DAEMON_NAME=kafka
# Check that networking is up.
#[ ${NETWORKING} = "no" ] && exit 0

PATH=$PATH:$DAEMON_PATH

# See how we were called.
case "$1" in
  start)

slgithub

Mac Shortcut	Windows Shortcut	Description
⌘ + ⇧ + R	Ctrl + Shift + R	Open / Search for Resources
⌘ + ⇧ + T	Ctrl + Shift + T	Open / Search for types (Useful in finding classes)
⌘ + O	Ctrl + O	Shows Quick Outline of the Java Class
⌘ + T	CTRL-T	Show type hierarchy
ALT – ↑ or ↓	ALT – ↑ or ↓	Move line/block
⌃ + Space	Ctrl + Space	Content assist and Code completion
⌘ + ⇧ + F	Ctrl + Shift + F	Format source code
⌘ + L	Ctrl + L	Go to Line Number

slgithub

Introduction to Kafka

Kafka acts as a kind of write-ahead log (WAL) that records messages to a persistent store (disk) and allows subscribers to read and apply these changes to their own stores in a system appropriate time-frame.

Terminology:

• Producers send messages to brokers
• Consumers read messages from brokers
• Messages are sent to a topic

slgithub

#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0

slgithub

#!/bin/sh
exec scala -savecompiled "$0" "$@"

!#
import scala.collection.mutable.Map
import scala.util.Random
import scala.collection.mutable.ArrayBuffer
import java.io._

class IPGenerator(var sessionCount: Int, var sessionLength: Int) {

slgithub

import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.StatusLine;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
import org.json.simple.JSONValue;

slgithub

Storm

Storm is a distributed and fault-tolerant realtime computation system

###Features of storm:

• Scalable and robust
• Fault-tolrant (automatic reassigning of tasks)
• Reliable (all messages are processed at least once)
• Fast

slgithub

#Mongo UseCase:

Installing mongodb on 5 machines with the following deamon configurations:

Host	Mongo Role
router.mongo.cw.com	Router(mongos), Application Server, Arbiter (Shard1), Arbiter (Shard2), Config1, Config2, Config3
shard1r1.mongo.cw.com	shard1 replica primary
shard1r2.mongo.cw.com	shard1 replica secondary
shard2r1.mongo.cw.com	shard2 replica primary

slgithub

##Using yarn as the resource manager you can deploy spark application in two modes:

1. yarn-standalone mode, in which your driver program is running as a thread of the yarn application master, which itself runs on one of the node managers in the cluster. The Yarn client just pulls status from the application master. This mode is same as a mapreduce job, where the MR application master coordinates the containers to run the map/reduce tasks.

With this mode, your application is actually run on the remote machine where the Application Master is run upon. Thus application that involve local interaction will not work well, e.g. spark-shell.

2. yarn-client mode, in which your driver program is running on the yarn client where you type the command to submit the spark application (may not be a machine in the yarn cluster). In this mode, although the drive program is running on the client machine, the tasks are executed on the executors in the node managers of the YARN cluster.

Simply putting to gether: