feitang0

# From: https://github.com/docker/docker/issues/8791
cgroupfs_mount() {
        # see also https://github.com/tianon/cgroupfs-mount/blob/master/cgroupfs-mount
        if grep -v '^#' /etc/fstab | grep -q cgroup \
                || [ ! -e /proc/cgroups ] \
                || [ ! -d /sys/fs/cgroup ]; then
                return
        fi
        if ! mountpoint -q /sys/fs/cgroup; then
                mount -t tmpfs -o uid=0,gid=0,mode=0755 cgroup /sys/fs/cgroup

feitang0

# add lines to the beginning of file
sed -i '1s/^/<added text>\n/' file

feitang0

deb mirror://mirrors.ubuntu.com/mirrors.txt precise main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt precise-updates main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt precise-backports main restricted universe multiverse
deb mirror://mirrors.ubuntu.com/mirrors.txt precise-security main restricted universe multiverse

add above on the top in /etc/apt/sources.list

feitang0

Spark on Yarn

yarn-client

yarn-cluster

feitang0

Spark学习笔记

集群环境的挑战

1. 并行化
2. 单点失败的处理(节点宕机,个别节点缓慢)
3. 多用户共享,动态进行计算资源的分配

MapReduce--简单通用和自动容错的批处理计算模型,不适合交互计算,迭代计算和流计算