deroneriksson

Apache Committer Notes

Deploying to Apache Snapshot Repository

To deploy project artifacts to the Apache Snapshot Repository takes a little set-up. Here are the steps that I followed.

Maven Password Encryption

Follow the instructions here: https://maven.apache.org/guides/mini/guide-encryption.html

deroneriksson

Deron Eriksson SystemML Git Notes

Useful Links

• Luciano's Development Guidelines - https://github.com/SparkTC/development-guidelines
• Mike's SystemML Git Guide - https://gist.github.com/dusenberrymw/78eb31b101c1b1b236e5
• Dev Mailing List - http://mail-archives.apache.org/mod_mbox/systemml-dev/
• SystemML Website - http://systemml.apache.org/
• SystemML on GitHub - https://github.com/apache/systemml
• SystemML Documentation - https://apache.github.io/systemml
• Scott Chacon's Pro Git - https://progit.org/

deroneriksson

Python Stuff

NumPy array - print without cutoffs

from scipy.ndimage import measurements
im = array(...)
labels, num_features = measurements.label(im)
set_printoptions(threshold=nan, linewidth=nan)
print(str(labels[0:100,0:100]))

deroneriksson

MNIST LeNet

import org.apache.sysml.api.mlcontext._
val ml = new MLContext(spark)
val clf = ml.nn.examples.Mnist_lenet
val dummy = clf.generate_dummy_data
val dummyVal = clf.generate_dummy_data
val params = clf.train(dummy.X, dummy.Y, dummyVal.X, dummyVal.Y, dummy.C, dummy.Hin, dummy.Win, 1)
val probs = clf.predict(dummy.X, dummy.C, dummy.Hin, dummy.Win, params.W1, params.b1, params.W2, params.b2, params.W3, params.b3, params.W4, params.b4)

deroneriksson

Useful Git Commands

Create a local branch based on a remote branch

$ git fetch --all

Fetching origin
Fetching apache-github
From https://github.com/apache/incubator-systemml
 * [new branch]      branch-0.9 -> apache-github/branch-0.9
 * [new branch] gh-pages -> apache-github/gh-pages

deroneriksson

Handy Regular Expressions

Javadoc match: /\*\*(?s:(?!\*/).)*\*/

Javadoc match public methods: /\*\*(?s:(?!\*/).)*\*/(\s*)public

Trailing whitespace: [ \t]+$

Four spaces at beginning of line: ^[ ]{4}

deroneriksson

Recursively remove .DS_Store files
find . -name '.DS_Store' -type f -delete

deroneriksson

SystemML Stuff

Run SystemML via MLContext in Spark Shell with SystemML jar file

spark-shell --executor-memory 4G --driver-memory 4G --driver-class-path=target/SystemML.jar

Run SystemML via MLContext in Spark Shell without SystemML jar file

spark-shell --executor-memory 4G --driver-memory 4G --driver-class-path="target/classes:target/lib/antlr4-runtime-4.5.3.jar:target/lib/wink-json4j-1.4.jar"

deroneriksson

SystemML Python Notes

Running SystemML Python Tests

pip3 install -U -e src/main/python
mvn clean package
PYSPARK_PYTHON=python3 spark-submit --master local[*] --driver-class-path target/SystemML.jar src/main/python/tests/test_mllearn.py
PYSPARK_PYTHON=python3 spark-submit --master local[*] --driver-class-path target/SystemML.jar src/main/python/tests/test_mlcontext.py

deroneriksson

Jupyter Notes for SystemML Cancer Project

Install Python 3

brew update
brew install python3

Install Python Packages including Jupyter