使用maven构建spark程序，借助IDEA。
maven-archetype-quickstart ；Add Framework Support > scala，然后编辑 pom.xml 添加相应依赖；SubmitExample.scala；.setMaster("local")代码，并用spark-submit提交任务。
| from __future__ import division | |
| from __future__ import print_function | |
| """ | |
| ------------ Follow The Regularized Leader - Proximal ------------ | |
| FTRL-P is an online classification algorithm that combines both L1 and L2 | |
| norms, particularly suited for large data sets with extremely high dimensionality. | |
| This implementation follow the algorithm by H. B. McMahan et. al. It minimizes |
| """ | |
| Calculate AUC of a sample | |
| Reference: | |
| http://binf.gmu.edu/mmasso/ROC101.pdf | |
| """ | |
| import matplotlib.pyplot as plt | |
| # each element is a tuple(positive, score) | |
| sample = [ | |
| (True, 0.63), |
| ******************************** | |
| Gist | |
| ******************************** | |
| Motivation | |
| ----------- | |
| The typical mapreduce job creates files with the prefix "part-"..and then the "m" or "r" depending | |
| on whether it is a map or a reduce output, and then the part number. There are scenarios where we | |
| may want to create separate files based on criteria-data keys and/or values. Enter the "MultipleOutputs" | |
| functionality. |
| ''' | |
| DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE | |
| Version 2, December 2004 | |
| Copyright (C) 2004 Sam Hocevar <sam@hocevar.net> | |
| Everyone is permitted to copy and distribute verbatim or modified | |
| copies of this license document, and changing it is allowed as long | |
| as the name is changed. |
| SLIDES := $(patsubst %.md,%.slides.pdf,$(wildcard *.md)) | |
| HANDOUTS := $(patsubst %.md,%.handout.pdf,$(wildcard *.md)) | |
| all : $(SLIDES) | |
| %.slides.pdf : %.md | |
| pandoc --latex-engine=xelatex --toc $^ -t beamer --slide-level 2 -o $@ | |
| %.handout.pdf : %.md | |
| pandoc $^ -t beamer --slide-level 2 -V handout -o $@ |
| #!/usr/bin/expect | |
| # Name: autoexpect - generate an Expect script from watching a session | |
| # | |
| # Description: | |
| # | |
| # Given a program name, autoexpect will run that program. Otherwise | |
| # autoexpect will start a shell. Interact as desired. When done, exit | |
| # the program or shell. Autoexpect will create a script that reproduces | |
| # your interactions. By default, the script is named script.exp. | |
| # See the man page for more info. |
| import numpy as np | |
| from math import exp, log | |
| """ | |
| SGD for logistic loss + factorization machines | |
| The code follows this paper: | |
| [1] http://www.ics.uci.edu/~smyth/courses/cs277/papers/factorization_machines_with_libFM.pdf | |
| """ | |
| def sigmoid(x): |
| /* LXC demo: | |
| * 1. UTC, isolate hostname | |
| * 2. IPC, pipe | |
| * 3. PID, chroot process tree | |
| * 4. NS, mount /proc to make `top` works well | |
| * 5. NET, veth from OpenVZ | |
| */ | |
| #define _GNU_SOURCE |