Tested with Apache Spark 2.1.0, Python 2.7.13 and Java 1.8.0_112
For older versions of Spark and ipython, please, see also previous version of text.
osemer01
osemer01
/ Spark+ipython_on_MacOS.md
Created
April 13, 2017 20:26
— forked from ololobus/Spark+ipython_on_MacOS.md
Apache Spark installation + ipython/jupyter notebook integration guide for macOS
osemer01
/ pspark_config.py
Created
May 23, 2017 15:52
— forked from robenalt/pspark_config.py
Sample pyspark context setting with configs params
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Set up spark configuration | |
| conf = SparkConf().setMaster("yarn-client").setAppName("sparK-mer") | |
| #conf = SparkConf().setMaster("local[16]").setAppName("sparK-mer") | |
| conf.set("yarn.nodemanager.resource.cpu_vcores",args.C) | |
| # Saturate with executors | |
| conf.set("spark.executor.instances",executorInstances) | |
| conf.set("spark.executor.heartbeatInterval","5s") | |
| # cores per executor | |
| conf.set("spark.executor.cores",args.E) | |
| # set driver cores |
osemer01
/ seminal_papers.txt
Last active
May 28, 2020 18:25
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Ranking with Ordered Weighted Pairwise Classification | |
| https://icml.cc/Conferences/2009/papers/163.pdf | |
| WARP Loss USed by Google Before Deep Learning | |
| https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37180.pdf | |
| Deep neural networks for youtube recommendations | |
| https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45530.pdf | |