lalyos

Keybase proof

I hereby claim:

• I am lalyos on github.
• I am lalyos (https://keybase.io/lalyos) on keybase.
• I have a public key whose fingerprint is 6CDF 8D01 D55D 18B8 5F12 AF64 7949 F304 320E F43D

To claim this, I am signing this object:

lalyos

[
  {
    "description": "HBase is used widely at Facebook and one of the biggest usecase is Facebook Messages. With a billion users there are a lot of reliability and performance challenges on both HBase and HDFS. HDFS was originally designed for a batch processing system like MapReduce/Hive. A realtime usecase like Facebook Messages where the p99 latency can`t be more than a couple hundreds of milliseconds poses a lot of challenges for HDFS. In this talk we will share the work the HDFS team at Facebook has done to support a realtime usecase like Facebook Messages : (1) Using system calls to tune performance; (2) Inline checksums to reduce iops by 40%; (3) Reducing the p99 for read and write latencies by about 10x; (4) Tools used to determine root cause of outliers. We will discuss the details of each technique, the challenges we faced, lessons learned and results showing the impact of each improvement.",
    "speaker": "Pritam Damania",
    "title": "Putting wings on the Elephant",
    "type": "operating-ha

lalyos

INFO 2014-03-14 09:38:52,505 main.py:83 - loglevel=logging.INFO
INFO 2014-03-14 09:38:52,505 DataCleaner.py:36 - Data cleanup thread started
INFO 2014-03-14 09:38:52,506 DataCleaner.py:71 - Data cleanup started
INFO 2014-03-14 09:38:52,507 DataCleaner.py:73 - Data cleanup finished
INFO 2014-03-14 09:38:52,508 PingPortListener.py:48 - Ping port listener started on port: 8670
INFO 2014-03-14 09:38:52,508 main.py:218 - Connecting to the server at: https://ambari.vmati.com:8440
INFO 2014-03-14 09:38:52,508 NetUtil.py:65 - DEBUG: Trying to connect to the server at https://ambari.vmati.com:8440
INFO 2014-03-14 09:38:52,508 NetUtil.py:41 - Connecting to the following url https://ambari.vmati.com:8440/cert/ca
INFO 2014-03-14 09:38:52,579 NetUtil.py:48 - Calling url received 200
INFO 2014-03-14 09:38:54,694 HostCheckReportFileHandler.py:43 - Host check report at /var/lib/ambari-agent/data/hostcheck.result

lalyos

netstat -na|grep LISTE|grep -v "127.0.0.1"|sed -n "/0.0.0.0\|:::/  {s/^[^:]*:*//; s/ .*//p}"|xargs echo

lalyos

[defaults]
hostfile = ~/.ansible.hosts

[ssh_connection]
#ssh_args =

lalyos

#!/bin/bash
# until the pull request get merged into main repo use my fork
set -e

#brew upgrade https://raw.githubusercontent.com/lalyos/homebrew/master/Library/Formula/boot2docker.rb
# meghivlak egy marek pisztaciara ha ezt olvasod ;)

brew update
brew upgrade boot2docker
boot2docker stop

lalyos

Ambari on Vagrant

Begin by starting up 3 nodes via quick start on ambari wiki

start 3 nodes

git clone https://github.com/u39kun/ambari-vagrant.git
cd ambari-vagrant/centos6.4/
./up.sh 3

lalyos

[AMBARI.dev-1.x]
name=Ambari 1.x
baseurl= http://s3.amazonaws.com/dev.hortonworks.com/AMBARI.dev-1.x/repos/centos6
gpgcheck=1
gpgkey= http://s3.amazonaws.com/dev.hortonworks.com/AMBARI.dev-1.x/repos/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.16]
name=Hortonworks Data Platform Utils Version - HDP-UTILS-1.1.0.16

lalyos

7 Deadly Hadoop Missconfgiuration

by Kathleen Ting

MR2 motivation

mr1:

• jobtracker: single point of failure

mr2:

• app mgr

lalyos

#!/bin/bash
JUMPER_LIST=$(curl -Ls https://gist.githubusercontent.com/lalyos/10364962/raw/jumper.list)

list() {
cat <<EOF

known jumpers:

$JUMPER_LIST