Skip to content

Instantly share code, notes, and snippets.

View hex108's full-sized avatar

Jun Gong hex108

  • Tencent
View GitHub Profile
@hex108
hex108 / get_ip.sh
Last active August 29, 2015 14:10
Get IP for specified host using 'http://ping.eu/ping', then write it to hosts.
#!/bin/bash
# Get IP for specified host using 'http://ping.eu/ping', then write it to hosts.
if [ $# -ne 1 ]; then
echo "Usage: $0 host"
echo " e.g. # $0 baidu.com"
exit 0
fi
@hex108
hex108 / tmx.sh
Last active August 29, 2015 14:10 — forked from chenchun/.tmux.conf
#! /bin/sh
tmux attach -t chen || tmux new -s chen
@hex108
hex108 / config_and_start_hadoop.sh
Created November 27, 2014 03:40
Configure Hadoop so that we could start Hadoop in a local machine. Then it will start Hadoop which includes HDFS and Yarn.
#!/bin/bash
set -u
set -e
if [ $# -ne 1 ]; then
echo "Usage: $0 hadoop_home_directory"
echo " e.g. $0 /home/test/hadoop-2.4.1"
exit -1
fi
@hex108
hex108 / config_redis.sh
Created November 27, 2014 03:46
Config redis automatically. The configure method is from http://redis.io/topics/quickstart.
#!/bin/bash
set -u
set -e
if [ $# -ne 2 ]; then
echo "Usage : $0 redis_src_directory port"
echo " e.g. $0 /home/test/redis-stable 6379"
exit -1
fi
@hex108
hex108 / docker_registry_tool.go
Last active August 29, 2015 14:19
A tool for two purposes: 1. Check repositories to find those broken ones. 2. Draw repository's image dependence using dot
package main
import (
"flag"
"fmt"
"io/ioutil"
"net/http"
"encoding/json"
"strings"
"container/list"
@hex108
hex108 / YARN.md
Last active October 31, 2015 03:25

通过一些问题引导读者了解、熟悉YARN,主要分为以下几大块:

#1. RM

##调度器 内容:FariScheduler, 调度器性能,SLS,Ganglia, 队列,抢占,调度原理, node label

问题1. 怎么保证用户的最低资源使用量

问题2. 如果集群中有多余的资源,已达到最低资源使用量的用户是否可以使用这些资源,此时如果有别的队列有app请求资源并且该队列没有达到最低资源使用量,如何处理

Contribution to open source by hex108:

  • YARN-4497 RM might fail to restart when recovering apps whose attempts are missing
  • YARN-3480 Remove attempts that are beyond max-attempt limit from state store
  • YARN-4256 YARN fair scheduler vcores with decimal values
  • YARN-4201 AMBlacklist does not work for minicluster
  • HADOOP-12388 Fix components' version information in the web page About the Cluster
  • YARN-3896 RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously
  • YARN-4057 If ContainersMonitor is not enabled, only print related log info one
#!/bin/bash
set -u
set -e
run_cmd() {
echo "$ $@"
eval "$@"
if [ $? -ne 0 ] ; then
echo "Failed to run $@, exiting..."
@hex108
hex108 / vps
Last active February 28, 2020 04:46
Manage(start/stop/status) vps on DigitalOcean. You could install shadowsocks automatically when creating a new vps.
#!/bin/bash
set -e
set -u
#set -x
SSH_KEY="05:8b:8f:f5:09:35:ae:61:a9:3b:21:ea:1c:36:bf:0c"
USER_DATA_FILE="$(dirname $0)/vps.userdata"
### sub functions
@hex108
hex108 / vagrantfile-for-vxlan-test-env
Created December 17, 2018 14:19
Vagrantfile for vxlan test env.
Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/xenial64"
config.vm.define "node0" do |node0|
node0.vm.provision :shell, inline: "hostname node0"
end
config.vm.define "node1" do |node1|
node1.vm.provision :shell, inline: "hostname node1"
end