kepricon

WARNING: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.mapred.JobConf
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:95)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
	at org.apache.hadoop.security.Groups.<init>(Groups.java:79)
	at org.apache.hadoop.security.Groups.<init>(Groups.java:74)

kepricon

xiaoyzhu@hn0-xiaoyz:~/kepricon/git/lagom-skil-api/skildistro/target/skil-distro-1.1-SNAPSHOT-dist/skil-distro-1.1-SNAPSHOT$ python sbin/agent_request.py --host localhost --port 9000 spark --deployMode client --trainingMasterPath "/tmp/parameteraveraging.json" --modelPath "/tmp/neuralnet.json"
{'status': '', 'totalExecutorCores': 1, 'driverClassPath': '', 'uiUrl': '', 'driverMemory': '8g', 'kill': '', 'principal': '', 'deployMode': 'client', 'driverLibraryPath': '', 'numExecutors': 2, 'modelPath': '', 'mainClass': 'io.skymind.skil.train.spark.SKILSparkMain', 'master': 'local[*]', 'driverCores': 1, 'jars': '', 'executorMemory': '', 'files': '', 'keyTab': '', 'propertiesFile': '', 'trainingMasterPath': '', 'supervise': '', 'yarnQueue': 'default', 'packages': '', 'excludePackages': '', 'name': 'SKILTrain', 'proxyUser': '', 'repositories': '', 'driverJavaOptions': ''}
Traceback (most recent call last):
  File "sbin/agent_request.py", line 177, in <module>
    route=endpoint)
  File "sbin/agent_request.py", line 97

kepricon

- test3 HDInsight cluster 
NOTE: this cluster(10 worker nodes) will be deleted soon.

  Use admin/Pa$$word123 combination.
  Ambari : https://xiaoyzhusparktest3.azurehdinsight.net/ 
  YARN UI: https://xiaoyzhusparktest3.azurehdinsight.net/yarnui 
  spark history: https://xiaoyzhusparktest3.azurehdinsight.net/sparkhistory. 

  Username: xiaoyzhu password: 8czLCaM8ec)un6M'

kepricon

FROM ubuntu:14.04
MAINTAINER DH Skymind.io daehyun@skymind.io

# install first necessary things and java8
RUN apt-get update && \
    echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | debconf-set-selections && \
    apt-get install -y make g++ wget tar software-properties-common && \
    add-apt-repository -y ppa:webupd8team/java && \
    apt-get update && \
    apt-get install -y oracle-java8-installer && \