Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
ChristophersMBP:Spark-Wiki cmeiklejohn$ ~/Downloads/spark-1.4.0-bin-hadoop2.4/bin/spark-submit --class WikiCount --master local --packages com.amazonaws:aws-java-sdk-pom:1.10.34,org.apache.hadoop:hadoop-aws:2.6.0 target/scala-2.11/wikicount_2.11-1.0.jar s3://cmeiklejohn-test-bucket $AWS_ACCESS_KEY_ID $AWS_SECRET_ACCESS_KEY
Ivy Default Cache set to: /Users/cmeiklejohn/.ivy2/cache
The jars for the packages stored in: /Users/cmeiklejohn/.ivy2/jars
:: loading settings :: url = jar:file:/Users/cmeiklejohn/Downloads/spark-1.4.0-bin-hadoop2.4/lib/spark-assembly-1.4.0-hadoop2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
com.amazonaws#aws-java-sdk-pom added as a dependency
org.apache.hadoop#hadoop-aws added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found com.amazonaws#aws-java-sdk-pom;1.10.34 in central
found org.apache.hadoop#hadoop-aws;2.6.0 in local-m2-cache
found org.apache.hadoop#hadoop-common;2.6.0 in local-m2-cache
found org.apache.hadoop#hadoop-annotations;2.6.0 in local-m2-cache
found com.google.guava#guava;11.0.2 in list
found com.google.code.findbugs#jsr305;1.3.9 in list
found commons-cli#commons-cli;1.2 in list
found org.apache.commons#commons-math3;3.1.1 in list
found xmlenc#xmlenc;0.52 in list
found commons-httpclient#commons-httpclient;3.1 in list
found commons-logging#commons-logging;1.1.3 in list
found commons-codec#commons-codec;1.4 in list
found commons-io#commons-io;2.4 in list
found commons-net#commons-net;3.1 in list
found commons-collections#commons-collections;3.2.1 in list
found javax.servlet#servlet-api;2.5 in list
found org.mortbay.jetty#jetty;6.1.26 in list
found org.mortbay.jetty#jetty-util;6.1.26 in list
found com.sun.jersey#jersey-core;1.9 in list
found com.sun.jersey#jersey-json;1.9 in list
found org.codehaus.jettison#jettison;1.1 in list
found com.sun.xml.bind#jaxb-impl;2.2.3-1 in list
found javax.xml.bind#jaxb-api;2.2.2 in list
found javax.xml.stream#stax-api;1.0-2 in list
found javax.activation#activation;1.1 in list
found org.codehaus.jackson#jackson-core-asl;1.9.13 in list
found org.codehaus.jackson#jackson-mapper-asl;1.9.13 in list
found org.codehaus.jackson#jackson-jaxrs;1.9.13 in central
found org.codehaus.jackson#jackson-xc;1.9.13 in central
found com.sun.jersey#jersey-server;1.9 in list
found asm#asm;3.2 in central
found log4j#log4j;1.2.17 in list
found net.java.dev.jets3t#jets3t;0.9.0 in list
found org.apache.httpcomponents#httpclient;4.2.5 in list
found org.apache.httpcomponents#httpcore;4.2.5 in central
found com.jamesmurty.utils#java-xmlbuilder;0.4 in list
found commons-lang#commons-lang;2.6 in list
found commons-configuration#commons-configuration;1.6 in list
found commons-digester#commons-digester;1.8 in list
found commons-beanutils#commons-beanutils;1.7.0 in list
found commons-beanutils#commons-beanutils-core;1.8.0 in list
found org.slf4j#slf4j-api;1.7.5 in local-m2-cache
found org.apache.avro#avro;1.7.4 in list
found com.thoughtworks.paranamer#paranamer;2.3 in list
found org.xerial.snappy#snappy-java;1.0.4.1 in list
found org.apache.commons#commons-compress;1.4.1 in list
found org.tukaani#xz;1.0 in list
found com.google.protobuf#protobuf-java;2.5.0 in list
found com.google.code.gson#gson;2.2.4 in list
found org.apache.hadoop#hadoop-auth;2.6.0 in local-m2-cache
found org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15 in list
found org.apache.directory.server#apacheds-i18n;2.0.0-M15 in list
found org.apache.directory.api#api-asn1-api;1.0.0-M20 in list
found org.apache.directory.api#api-util;1.0.0-M20 in list
found org.apache.curator#curator-framework;2.6.0 in local-m2-cache
found org.apache.curator#curator-client;2.6.0 in local-m2-cache
found org.apache.zookeeper#zookeeper;3.4.6 in list
found jline#jline;0.9.94 in list
found io.netty#netty;3.6.2.Final in central
found com.jcraft#jsch;0.1.42 in list
found org.apache.curator#curator-recipes;2.6.0 in local-m2-cache
found org.htrace#htrace-core;3.0.4 in local-m2-cache
found tomcat#jasper-compiler;5.5.23 in local-m2-cache
found tomcat#jasper-runtime;5.5.23 in local-m2-cache
found commons-el#commons-el;1.0 in local-m2-cache
found javax.servlet.jsp#jsp-api;2.1 in list
found org.slf4j#slf4j-log4j12;1.7.5 in local-m2-cache
found com.fasterxml.jackson.core#jackson-databind;2.2.3 in list
found com.fasterxml.jackson.core#jackson-annotations;2.2.3 in list
found com.fasterxml.jackson.core#jackson-core;2.2.3 in list
found com.amazonaws#aws-java-sdk;1.7.4 in list
found joda-time#joda-time;2.9.6 in list
[2.9.6] joda-time#joda-time;[2.2,)
downloading file:/Users/cmeiklejohn/.m2/repository/org/apache/hadoop/hadoop-aws/2.6.0/hadoop-aws-2.6.0.jar ...
[SUCCESSFUL ] org.apache.hadoop#hadoop-aws;2.6.0!hadoop-aws.jar (3ms)
downloading file:/Users/cmeiklejohn/.m2/repository/org/apache/hadoop/hadoop-common/2.6.0/hadoop-common-2.6.0.jar ...
[SUCCESSFUL ] org.apache.hadoop#hadoop-common;2.6.0!hadoop-common.jar (48ms)
downloading file:/Users/cmeiklejohn/.m2/repository/org/apache/hadoop/hadoop-auth/2.6.0/hadoop-auth-2.6.0.jar ...
[SUCCESSFUL ] org.apache.hadoop#hadoop-auth;2.6.0!hadoop-auth.jar (3ms)
downloading file:/Users/cmeiklejohn/.m2/repository/org/apache/curator/curator-client/2.6.0/curator-client-2.6.0.jar ...
[SUCCESSFUL ] org.apache.curator#curator-client;2.6.0!curator-client.jar(bundle) (6ms)
downloading file:/Users/cmeiklejohn/.m2/repository/org/htrace/htrace-core/3.0.4/htrace-core-3.0.4.jar ...
[SUCCESSFUL ] org.htrace#htrace-core;3.0.4!htrace-core.jar (2ms)
downloading https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar ...
[SUCCESSFUL ] org.codehaus.jackson#jackson-jaxrs;1.9.13!jackson-jaxrs.jar (34ms)
downloading https://repo1.maven.org/maven2/org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar ...
[SUCCESSFUL ] org.codehaus.jackson#jackson-xc;1.9.13!jackson-xc.jar (52ms)
downloading https://repo1.maven.org/maven2/asm/asm/3.2/asm-3.2.jar ...
[SUCCESSFUL ] asm#asm;3.2!asm.jar (61ms)
downloading https://repo1.maven.org/maven2/org/apache/httpcomponents/httpcore/4.2.5/httpcore-4.2.5.jar ...
[SUCCESSFUL ] org.apache.httpcomponents#httpcore;4.2.5!httpcore.jar (214ms)
downloading https://repo1.maven.org/maven2/io/netty/netty/3.6.2.Final/netty-3.6.2.Final.jar ...
[SUCCESSFUL ] io.netty#netty;3.6.2.Final!netty.jar(bundle) (616ms)
downloading file:/Users/cmeiklejohn/.m2/repository/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar ...
[SUCCESSFUL ] tomcat#jasper-compiler;5.5.23!jasper-compiler.jar (20ms)
downloading file:/Users/cmeiklejohn/.m2/repository/tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar ...
[SUCCESSFUL ] tomcat#jasper-runtime;5.5.23!jasper-runtime.jar (3ms)
downloading file:/Users/cmeiklejohn/.m2/repository/commons-el/commons-el/1.0/commons-el-1.0.jar ...
[SUCCESSFUL ] commons-el#commons-el;1.0!commons-el.jar (3ms)
:: resolution report :: resolve 4878ms :: artifacts dl 1134ms
:: modules in use:
asm#asm;3.2 from central in [default]
com.amazonaws#aws-java-sdk;1.7.4 from list in [default]
com.amazonaws#aws-java-sdk-pom;1.10.34 from central in [default]
com.fasterxml.jackson.core#jackson-annotations;2.2.3 from list in [default]
com.fasterxml.jackson.core#jackson-core;2.2.3 from list in [default]
com.fasterxml.jackson.core#jackson-databind;2.2.3 from list in [default]
com.google.code.findbugs#jsr305;1.3.9 from list in [default]
com.google.code.gson#gson;2.2.4 from list in [default]
com.google.guava#guava;11.0.2 from list in [default]
com.google.protobuf#protobuf-java;2.5.0 from list in [default]
com.jamesmurty.utils#java-xmlbuilder;0.4 from list in [default]
com.jcraft#jsch;0.1.42 from list in [default]
com.sun.jersey#jersey-core;1.9 from list in [default]
com.sun.jersey#jersey-json;1.9 from list in [default]
com.sun.jersey#jersey-server;1.9 from list in [default]
com.sun.xml.bind#jaxb-impl;2.2.3-1 from list in [default]
com.thoughtworks.paranamer#paranamer;2.3 from list in [default]
commons-beanutils#commons-beanutils;1.7.0 from list in [default]
commons-beanutils#commons-beanutils-core;1.8.0 from list in [default]
commons-cli#commons-cli;1.2 from list in [default]
commons-codec#commons-codec;1.4 from list in [default]
commons-collections#commons-collections;3.2.1 from list in [default]
commons-configuration#commons-configuration;1.6 from list in [default]
commons-digester#commons-digester;1.8 from list in [default]
commons-el#commons-el;1.0 from local-m2-cache in [default]
commons-httpclient#commons-httpclient;3.1 from list in [default]
commons-io#commons-io;2.4 from list in [default]
commons-lang#commons-lang;2.6 from list in [default]
commons-logging#commons-logging;1.1.3 from list in [default]
commons-net#commons-net;3.1 from list in [default]
io.netty#netty;3.6.2.Final from central in [default]
javax.activation#activation;1.1 from list in [default]
javax.servlet#servlet-api;2.5 from list in [default]
javax.servlet.jsp#jsp-api;2.1 from list in [default]
javax.xml.bind#jaxb-api;2.2.2 from list in [default]
javax.xml.stream#stax-api;1.0-2 from list in [default]
jline#jline;0.9.94 from list in [default]
joda-time#joda-time;2.9.6 from list in [default]
log4j#log4j;1.2.17 from list in [default]
net.java.dev.jets3t#jets3t;0.9.0 from list in [default]
org.apache.avro#avro;1.7.4 from list in [default]
org.apache.commons#commons-compress;1.4.1 from list in [default]
org.apache.commons#commons-math3;3.1.1 from list in [default]
org.apache.curator#curator-client;2.6.0 from local-m2-cache in [default]
org.apache.curator#curator-framework;2.6.0 from local-m2-cache in [default]
org.apache.curator#curator-recipes;2.6.0 from local-m2-cache in [default]
org.apache.directory.api#api-asn1-api;1.0.0-M20 from list in [default]
org.apache.directory.api#api-util;1.0.0-M20 from list in [default]
org.apache.directory.server#apacheds-i18n;2.0.0-M15 from list in [default]
org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15 from list in [default]
org.apache.hadoop#hadoop-annotations;2.6.0 from local-m2-cache in [default]
org.apache.hadoop#hadoop-auth;2.6.0 from local-m2-cache in [default]
org.apache.hadoop#hadoop-aws;2.6.0 from local-m2-cache in [default]
org.apache.hadoop#hadoop-common;2.6.0 from local-m2-cache in [default]
org.apache.httpcomponents#httpclient;4.2.5 from list in [default]
org.apache.httpcomponents#httpcore;4.2.5 from central in [default]
org.apache.zookeeper#zookeeper;3.4.6 from list in [default]
org.codehaus.jackson#jackson-core-asl;1.9.13 from list in [default]
org.codehaus.jackson#jackson-jaxrs;1.9.13 from central in [default]
org.codehaus.jackson#jackson-mapper-asl;1.9.13 from list in [default]
org.codehaus.jackson#jackson-xc;1.9.13 from central in [default]
org.codehaus.jettison#jettison;1.1 from list in [default]
org.htrace#htrace-core;3.0.4 from local-m2-cache in [default]
org.mortbay.jetty#jetty;6.1.26 from list in [default]
org.mortbay.jetty#jetty-util;6.1.26 from list in [default]
org.slf4j#slf4j-api;1.7.5 from local-m2-cache in [default]
org.slf4j#slf4j-log4j12;1.7.5 from local-m2-cache in [default]
org.tukaani#xz;1.0 from list in [default]
org.xerial.snappy#snappy-java;1.0.4.1 from list in [default]
tomcat#jasper-compiler;5.5.23 from local-m2-cache in [default]
tomcat#jasper-runtime;5.5.23 from local-m2-cache in [default]
xmlenc#xmlenc;0.52 from list in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 72 | 20 | 19 | 0 || 71 | 13 |
---------------------------------------------------------------------
:: problems summary ::
:::: WARNINGS
[NOT FOUND ] org.apache.hadoop#hadoop-annotations;2.6.0!hadoop-annotations.jar (0ms)
==== local-m2-cache: tried
file:/Users/cmeiklejohn/.m2/repository/org/apache/hadoop/hadoop-annotations/2.6.0/hadoop-annotations-2.6.0.jar
[NOT FOUND ] org.slf4j#slf4j-api;1.7.5!slf4j-api.jar (1ms)
==== local-m2-cache: tried
file:/Users/cmeiklejohn/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[NOT FOUND ] org.apache.curator#curator-recipes;2.6.0!curator-recipes.jar(bundle) (0ms)
==== local-m2-cache: tried
file:/Users/cmeiklejohn/.m2/repository/org/apache/curator/curator-recipes/2.6.0/curator-recipes-2.6.0.jar
[NOT FOUND ] org.apache.curator#curator-framework;2.6.0!curator-framework.jar(bundle) (0ms)
==== local-m2-cache: tried
file:/Users/cmeiklejohn/.m2/repository/org/apache/curator/curator-framework/2.6.0/curator-framework-2.6.0.jar
[NOT FOUND ] org.slf4j#slf4j-log4j12;1.7.5!slf4j-log4j12.jar (0ms)
==== local-m2-cache: tried
file:/Users/cmeiklejohn/.m2/repository/org/slf4j/slf4j-log4j12/1.7.5/slf4j-log4j12-1.7.5.jar
::::::::::::::::::::::::::::::::::::::::::::::
:: FAILED DOWNLOADS ::
:: ^ see resolution messages for details ^ ::
::::::::::::::::::::::::::::::::::::::::::::::
:: org.apache.hadoop#hadoop-annotations;2.6.0!hadoop-annotations.jar
:: org.slf4j#slf4j-api;1.7.5!slf4j-api.jar
:: org.apache.curator#curator-framework;2.6.0!curator-framework.jar(bundle)
:: org.apache.curator#curator-recipes;2.6.0!curator-recipes.jar(bundle)
:: org.slf4j#slf4j-log4j12;1.7.5!slf4j-log4j12.jar
::::::::::::::::::::::::::::::::::::::::::::::
:::: ERRORS
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver sbt-chain
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
unknown resolver null
:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed: org.apache.hadoop#hadoop-annotations;2.6.0!hadoop-annotations.jar, download failed: org.slf4j#slf4j-api;1.7.5!slf4j-api.jar, download failed: org.apache.curator#curator-framework;2.6.0!curator-framework.jar(bundle), download failed: org.apache.curator#curator-recipes;2.6.0!curator-recipes.jar(bundle), download failed: org.slf4j#slf4j-log4j12;1.7.5!slf4j-log4j12.jar]
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:978)
at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:262)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:144)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/11/21 13:26:10 INFO Utils: Shutdown hook called
ChristophersMBP:Spark-Wiki cmeiklejohn$
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment