Skip to content

Instantly share code, notes, and snippets.

Sunny SunnyBingoMe

Block or report user

Report or block SunnyBingoMe

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View Spark high availability.md

Spark - High availability

Components in play

As a reminder, here are the components in play to run an application:

  • The cluster:
    • Spark Master: coordinates the resources
    • Spark Workers: offer resources to run the applications
  • The application:
View Dockerfile--spark8s-pyspark
# I was having problems to create docker images for spark-on-k8s using docker-image-tool.sh in spark (it always gives spark-r).
# Thus here is the docker file to creat for running pyspark.
# The image is available on sunnybingome/spark8s:pyspark240py368 with spark 2.4.0
# For clie-mode (interactive mode), please use pipenv to ensure py version. The image is in using py 3.6(.8).
# Sunny modified from spark-2.4.0-bin-hadoop2.7/kubernetes/dockerfiles/spark/Dockerfile
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
@SunnyBingoMe
SunnyBingoMe / Simple Spark Tests - README.md
Last active Mar 5, 2019
Simple Spark Tests: simple demos to test if spark is installed well and working
View Simple Spark Tests - README.md

This is a gist of collection of simple spark tests for Jupyter and spark-submit.

@SunnyBingoMe
SunnyBingoMe / get_ip.py
Last active Feb 10, 2019
get a non-127 ip address by python
View get_ip.py
import socket
print([l for l in ([ip for ip in socket.gethostbyname_ex(socket.gethostname())[2]
if not ip.startswith("127.")][:1], [[(s.connect(('8.8.8.8', 53)),
s.getsockname()[0], s.close()) for s in [socket.socket(socket.AF_INET,
socket.SOCK_DGRAM)]][0][1]]) if l][0][0])
# limit: only one IP, not getting multi-NIC IPs
@SunnyBingoMe
SunnyBingoMe / PS_promot_wo_git.ps1
Last active Dec 12, 2018
MS windows powershell promt, (without git)
View PS_promot_wo_git.ps1
# Import module from previous step
Import-Module -Name posh-git
function Test-Administrator {
$user = [Security.Principal.WindowsIdentity]::GetCurrent();
(New-Object Security.Principal.WindowsPrincipal $user).IsInRole([Security.Principal.WindowsBuiltinRole]::Administrator)
}
function mklink { cmd /c mklink $args }
View simple_tensorflow_serving-requirements-pip2.txt
# pip2
absl-py==0.2.2
adium-theme-ubuntu==0.3.4
astor==0.6.2
backports.functools-lru-cache==1.5
backports.shutil-get-terminal-size==1.0.0
backports.weakref==1.0.post1
bleach==1.5.0
certifi==2018.4.16
chardet==3.0.4
@SunnyBingoMe
SunnyBingoMe / cuda90.sh
Last active Aug 10, 2018
re-install cuda to v9.0 from v9.1, due to tf 1.10.0 does not know recognize 9.1 yet
View cuda90.sh
apt-get purge cuda && \
apt-get purge libcudnn6 && \
apt-get purge libcudnn6-dev && \
wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.0.176-1_amd64.deb && \
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb && \
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libcudnn7-dev_7.0.5.15-1+cuda9.0_amd64.deb && \
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libnccl2_2.1.4-1+cuda9.0_amd64.deb && \
wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libnccl-dev_2.1.4-1+cuda9.0_amd64.deb && \
dpkg -i cuda-repo-ubuntu1604_9.0.176-1_amd64.deb && \
dpkg -i libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb && \
@SunnyBingoMe
SunnyBingoMe / external_links.js
Created Aug 9, 2018
open external links in new tabs, i.e. target="_blank"
View external_links.js
@SunnyBingoMe
SunnyBingoMe / install_mosh_server_non_root.sh
Last active Aug 8, 2019 — forked from lazywei/install_mosh_locally.sh
Install mosh server without root permission
View install_mosh_server_non_root.sh
#!/bin/sh
# tested in Ubuntu 14.04 manually.
# this script does absolutely ZERO error checking. however, it worked
# for me on a RHEL 6.3 machine on 2012-08-08. clearly, the version numbers
# and/or URLs should be made variables. cheers, zmil...@cs.wisc.edu
mkdir mosh
cd mosh
@SunnyBingoMe
SunnyBingoMe / caffe.install.sh
Last active Apr 17, 2018
Caffe installing script for ubuntu 16.04 support Cuda 8
View caffe.install.sh
# ref http://t.cn/RYlajoc
# sunny: needs hdf5, protobuf
sudo apt-get install -y libhdf5-10 libhdf5-serial-dev libhdf5-dev libhdf5-cpp-11 && \
export CPATH="/usr/include/hdf5/serial/"
apt-get install -y libprotobuf-dev protobuf-compiler libsqlite3-dev
# sunny: needs gflags
#apt-get install -y cmake
You can’t perform that action at this time.