Skip to content

Instantly share code, notes, and snippets.

@liyuqi

liyuqi/jupyter_with_python_R.md

Last active January 7, 2016 03:53
Show Gist options
  • Save liyuqi/487320c7c0d2dd7cedb2 to your computer and use it in GitHub Desktop.
Save liyuqi/487320c7c0d2dd7cedb2 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
jupyter_with_python_R.md

1 jupyter with python and R

透過jupyter互動式網頁寫腳本，有助於git分享結果。

本篇將jupter安裝在ubuntu 14.04 server 上， 可透過https 方式連線，編寫python、R程式。

1.1 安裝虛擬機

1.1.1 準備軟體與工具

注意相依套件

  • curl
  • git
  • gcc
  • g++
  • libcurl
  • libxml
  • libxslt
  • libffi
  • libssl

1.1.2 虛擬機設定

開二張網卡，1用bridge連外網；2用Host-only連內網，cluster間的連線。

Adpter 1 下圖為例，連接Wifi的網卡。

Adapter 2下圖為例，連接VirtualBox Host-Only Ethernet的網卡。

English Hostname:hadoop01 Software selection: OpenSSH server User/Password: test LVM Apt-get: Security updates automaticall

1.1.3 線上安裝套件JAVA

安裝oracle-java7

$ sudo apt-get update
$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java7-installer

安裝完的路徑在

$ ls /usr/lib/jvm/java-7-oracle

檢查安裝是否成功

$ java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
$ sudo vi ~/.bashrc 	# 編輯環境變數
#...最下方
export JAVA_HOME=/usr/lib/jvm/java-7-oracle
export PATH=$PATH:$JAVA_HOME/bin:/home/test/hadoop-2.7.1/bin:/home/test/hadoop-2.7.1/sbin
$ source ~/.bashrc 	# 重載環境變數

1.1.4 修改網路設定

$ sudo vi /etc/network/interfaces
# 新增
auto eth1
iface eth1 inet static
address 192.168.56.101
netmask 255.255.255.0
$ sudo vi /etc/hosts
127.0.0.1       localhost
192.168.56.101  hadoop01
192.168.56.102  hadoop02
192.168.56.103  hadoop03
 
# The following lines are desirable for IPv6 capable hosts
#::1     localhost ip6-localhost ip6-loopback
#ff02::1 ip6-allnodes
#ff02::2 ip6-allrouters
$ ifconfig   # 確認設定
eth1      Link encap:Ethernet  HWaddr 08:00:27:7d:a5:23
          inet addr:192.168.56.101  Bcast:192.168.56.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1403 errors:0 dropped:0 overruns:0 frame:0
          TX packets:68 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:130322 (130.3 KB)  TX bytes:11731 (11.7 KB)

$ sudo reboot # 重新啟動

1.1.5 SSH連線設定

$ sudo apt-get install openssh-server openssh-client     	# 安裝
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa				# 產生密鑰
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys 		# 公鑰授權

1.3 程式環境準備

1.3.1 Ubuntu 上的 jupyter相依套件

Ubuntu (debian-based)

$ sudo apt-get install libxml2-dev libxslt1-dev libffi-dev libssl-dev
$ sudo apt-get install python-dev

CentOS (red-hat-based)

$ yum install libxslt-devel libxml2-devel

1.3.1.1 安裝jupyter notebook

# setup pip
$ wget https://bootstrap.pypa.io/get-pip.py
$ sudo python get-pip.py
# make a virtualenv
$ sudo pip install virtualenv
$ cd ~
$ virtualenv local/python/jupyter
$ source local/python/jupyter/bin/activate
## get the cutting edge ipython version
$ git clone --recursive https://github.com/ipython/ipython.git
# and install all its dependencies
$ cd ipython
$ sudo pip install -e ".[notebook]"
$ sudo pip install jupyter

安裝python出現InsecurePlatformWarning錯誤時

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning

$ sudo pip install requests[security]
$ sudo pip install requests[security] --upgrade
$ sudo pip install numpy

1.3.1.2 建立jupyter profile SSL設定

建立notebook config

$ jupyter notebook --generate-config

Writing default config to: ~/.jupyter/jupyter_notebook_config.py

$ jupyter console

In [1]: from notebook.auth import passwd
In [2]: passwd()
Enter password:
Verify password:
Out[2]: 'sha1:7733a7fdb0f7:d9995f4e04c90a887c40dbd9a4de7542a2e7bb10'

$ vi ~/.jupyter/jupyter_notebook_config.py

# Notebook config

# c = get_config()
c.IPKernelApp.pylab = 'inline' 								# allow plotting support
c.NotebookApp.certfile = u'/home/btserver/workspace/mycert.pem'
c.NotebookApp.keyfile = u'/home/btserver/workspace/mykey.key'
c.NotebookApp.ip = '*' 										# allow remote login
c.NotebookApp.open_browser = False
c.NotebookApp.password = u'sha1:7733a7fdb0f7:d9995f4e04c90a887c40dbd9a4de7542a2e7bb10'
# It is a good idea to put it on a known, fixed port
c.NotebookApp.port = 9999
$ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mykey.key -out mycert.pem
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:
$ ls
mycert.pem
mykey.key

1.3.2 IRkernel 設定

Reference:

安裝相依套件 non-zero exit

$ sudo apt-get install libzmq3-dev libcurl4-openssl-dev

check ubuntu 版本 $ lsb_release -a 

No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.2 LTS
Release:	14.04
Codename:	trust

準備rzmq (jupyter 與 R互動介面資訊)

$ cd workspace
$ git clone https://github.com/armstrtw/rzmq.git --recursive

安裝 R

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9
$ sudo add-apt-repository "deb http://cran.csie.ntu.edu.tw/bin/linux/ubuntu trusty/"
$ sudo apt-get update
$ sudo apt-get install r-base r-base-dev

$ sudo R

>
install.packages('devtools') 		# 10 min
# 編譯與安裝...
install.packages('RCurl')  
library(devtools)  
install_local('./rzmq')                # workspace 下
install_github('IRkernel/repr')
install_github('IRkernel/IRdisplay')  
install_github('IRkernel/IRkernel')

IRkernel::installspec()
[InstallKernelSpec] Installed kernelspec ir in /home/btserver/.local/share/jupyter/kernels/ir

1.3.3 Jupyter browser

網址鍵入 [https://192.168.56.102:9999/ ]

登入ipython，輸入password

編輯ipython

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment