Last active
May 15, 2019 20:36
-
-
Save skonto/c618c7d2a3f699cddce5bc6c697bbc16 to your computer and use it in GitHub Desktop.
works
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
./bin/spark-shell --conf spark.jars.packages=com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.6 | |
Ivy Default Cache set to: /home/ubuntu/.ivy2/cache | |
The jars for the packages stored in: /home/ubuntu/.ivy2/jars | |
:: loading settings :: url = jar:file:/home/ubuntu/workspace/spark-2.4.3-bin-hadoop2.7/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml | |
com.amazonaws#aws-java-sdk added as a dependency | |
org.apache.hadoop#hadoop-aws added as a dependency | |
:: resolving dependencies :: org.apache.spark#spark-submit-parent-d3c9d1bd-404e-405d-9bb5-b4419657a1a2;1.0 | |
confs: [default] | |
found com.amazonaws#aws-java-sdk;1.7.4 in central | |
found commons-logging#commons-logging;1.1.1 in central | |
found org.apache.httpcomponents#httpclient;4.2 in central | |
found org.apache.httpcomponents#httpcore;4.2 in central | |
found commons-codec#commons-codec;1.3 in central | |
found com.fasterxml.jackson.core#jackson-core;2.1.1 in central | |
found com.fasterxml.jackson.core#jackson-databind;2.1.1 in central | |
found com.fasterxml.jackson.core#jackson-annotations;2.1.1 in central | |
found joda-time#joda-time;2.10.2 in central | |
[2.10.2] joda-time#joda-time;[2.2,) | |
found org.apache.hadoop#hadoop-aws;2.7.6 in central | |
found org.apache.hadoop#hadoop-common;2.7.6 in central | |
found org.apache.hadoop#hadoop-annotations;2.7.6 in central | |
found com.google.guava#guava;11.0.2 in central | |
found com.google.code.findbugs#jsr305;3.0.0 in central | |
found commons-cli#commons-cli;1.2 in central | |
found org.apache.commons#commons-math3;3.1.1 in central | |
found xmlenc#xmlenc;0.52 in central | |
found commons-httpclient#commons-httpclient;3.1 in central | |
found commons-logging#commons-logging;1.1.3 in central | |
found commons-codec#commons-codec;1.4 in central | |
found commons-io#commons-io;2.4 in central | |
found commons-net#commons-net;3.1 in central | |
found commons-collections#commons-collections;3.2.2 in central | |
found javax.servlet#servlet-api;2.5 in central | |
found org.mortbay.jetty#jetty;6.1.26 in central | |
found org.mortbay.jetty#jetty-util;6.1.26 in central | |
found org.mortbay.jetty#jetty-sslengine;6.1.26 in central | |
found com.sun.jersey#jersey-core;1.9 in central | |
found com.sun.jersey#jersey-json;1.9 in central | |
found org.codehaus.jettison#jettison;1.1 in central | |
found com.sun.xml.bind#jaxb-impl;2.2.3-1 in central | |
found javax.xml.bind#jaxb-api;2.2.2 in central | |
found javax.xml.stream#stax-api;1.0-2 in central | |
found javax.activation#activation;1.1 in central | |
found org.codehaus.jackson#jackson-core-asl;1.9.13 in central | |
found org.codehaus.jackson#jackson-mapper-asl;1.9.13 in central | |
found org.codehaus.jackson#jackson-jaxrs;1.9.13 in central | |
found org.codehaus.jackson#jackson-xc;1.9.13 in central | |
found com.sun.jersey#jersey-server;1.9 in central | |
found asm#asm;3.2 in central | |
found log4j#log4j;1.2.17 in central | |
found net.java.dev.jets3t#jets3t;0.9.0 in central | |
found org.apache.httpcomponents#httpclient;4.2.5 in central | |
found org.apache.httpcomponents#httpcore;4.2.5 in central | |
found com.jamesmurty.utils#java-xmlbuilder;0.4 in central | |
found commons-lang#commons-lang;2.6 in central | |
found commons-configuration#commons-configuration;1.6 in central | |
found commons-digester#commons-digester;1.8 in central | |
found commons-beanutils#commons-beanutils;1.7.0 in central | |
found commons-beanutils#commons-beanutils-core;1.8.0 in central | |
found org.slf4j#slf4j-api;1.7.10 in central | |
found org.apache.avro#avro;1.7.4 in central | |
found com.thoughtworks.paranamer#paranamer;2.3 in central | |
found org.xerial.snappy#snappy-java;1.0.4.1 in central | |
found org.apache.commons#commons-compress;1.4.1 in central | |
found org.tukaani#xz;1.0 in central | |
found com.google.protobuf#protobuf-java;2.5.0 in central | |
found com.google.code.gson#gson;2.2.4 in central | |
found org.apache.hadoop#hadoop-auth;2.7.6 in central | |
found org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15 in central | |
found org.apache.directory.server#apacheds-i18n;2.0.0-M15 in central | |
found org.apache.directory.api#api-asn1-api;1.0.0-M20 in central | |
found org.apache.directory.api#api-util;1.0.0-M20 in central | |
found org.apache.zookeeper#zookeeper;3.4.6 in central | |
found org.slf4j#slf4j-log4j12;1.7.10 in central | |
found io.netty#netty;3.6.2.Final in central | |
found org.apache.curator#curator-framework;2.7.1 in central | |
found org.apache.curator#curator-client;2.7.1 in central | |
found com.jcraft#jsch;0.1.54 in central | |
found org.apache.curator#curator-recipes;2.7.1 in central | |
found org.apache.htrace#htrace-core;3.1.0-incubating in central | |
found org.mortbay.jetty#servlet-api;2.5-20081211 in central | |
found javax.servlet.jsp#jsp-api;2.1 in central | |
found jline#jline;0.9.94 in central | |
found junit#junit;4.11 in central | |
found org.hamcrest#hamcrest-core;1.3 in central | |
found com.fasterxml.jackson.core#jackson-databind;2.2.3 in central | |
found com.fasterxml.jackson.core#jackson-annotations;2.2.3 in central | |
found com.fasterxml.jackson.core#jackson-core;2.2.3 in central | |
:: resolution report :: resolve 1884ms :: artifacts dl 21ms | |
:: modules in use: | |
asm#asm;3.2 from central in [default] | |
com.amazonaws#aws-java-sdk;1.7.4 from central in [default] | |
com.fasterxml.jackson.core#jackson-annotations;2.2.3 from central in [default] | |
com.fasterxml.jackson.core#jackson-core;2.2.3 from central in [default] | |
com.fasterxml.jackson.core#jackson-databind;2.2.3 from central in [default] | |
com.google.code.findbugs#jsr305;3.0.0 from central in [default] | |
com.google.code.gson#gson;2.2.4 from central in [default] | |
com.google.guava#guava;11.0.2 from central in [default] | |
com.google.protobuf#protobuf-java;2.5.0 from central in [default] | |
com.jamesmurty.utils#java-xmlbuilder;0.4 from central in [default] | |
com.jcraft#jsch;0.1.54 from central in [default] | |
com.sun.jersey#jersey-core;1.9 from central in [default] | |
com.sun.jersey#jersey-json;1.9 from central in [default] | |
com.sun.jersey#jersey-server;1.9 from central in [default] | |
com.sun.xml.bind#jaxb-impl;2.2.3-1 from central in [default] | |
com.thoughtworks.paranamer#paranamer;2.3 from central in [default] | |
commons-beanutils#commons-beanutils;1.7.0 from central in [default] | |
commons-beanutils#commons-beanutils-core;1.8.0 from central in [default] | |
commons-cli#commons-cli;1.2 from central in [default] | |
commons-codec#commons-codec;1.4 from central in [default] | |
commons-collections#commons-collections;3.2.2 from central in [default] | |
commons-configuration#commons-configuration;1.6 from central in [default] | |
commons-digester#commons-digester;1.8 from central in [default] | |
commons-httpclient#commons-httpclient;3.1 from central in [default] | |
commons-io#commons-io;2.4 from central in [default] | |
commons-lang#commons-lang;2.6 from central in [default] | |
commons-logging#commons-logging;1.1.3 from central in [default] | |
commons-net#commons-net;3.1 from central in [default] | |
io.netty#netty;3.6.2.Final from central in [default] | |
javax.activation#activation;1.1 from central in [default] | |
javax.servlet#servlet-api;2.5 from central in [default] | |
javax.servlet.jsp#jsp-api;2.1 from central in [default] | |
javax.xml.bind#jaxb-api;2.2.2 from central in [default] | |
javax.xml.stream#stax-api;1.0-2 from central in [default] | |
jline#jline;0.9.94 from central in [default] | |
joda-time#joda-time;2.10.2 from central in [default] | |
junit#junit;4.11 from central in [default] | |
log4j#log4j;1.2.17 from central in [default] | |
net.java.dev.jets3t#jets3t;0.9.0 from central in [default] | |
org.apache.avro#avro;1.7.4 from central in [default] | |
org.apache.commons#commons-compress;1.4.1 from central in [default] | |
org.apache.commons#commons-math3;3.1.1 from central in [default] | |
org.apache.curator#curator-client;2.7.1 from central in [default] | |
org.apache.curator#curator-framework;2.7.1 from central in [default] | |
org.apache.curator#curator-recipes;2.7.1 from central in [default] | |
org.apache.directory.api#api-asn1-api;1.0.0-M20 from central in [default] | |
org.apache.directory.api#api-util;1.0.0-M20 from central in [default] | |
org.apache.directory.server#apacheds-i18n;2.0.0-M15 from central in [default] | |
org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15 from central in [default] | |
org.apache.hadoop#hadoop-annotations;2.7.6 from central in [default] | |
org.apache.hadoop#hadoop-auth;2.7.6 from central in [default] | |
org.apache.hadoop#hadoop-aws;2.7.6 from central in [default] | |
org.apache.hadoop#hadoop-common;2.7.6 from central in [default] | |
org.apache.htrace#htrace-core;3.1.0-incubating from central in [default] | |
org.apache.httpcomponents#httpclient;4.2.5 from central in [default] | |
org.apache.httpcomponents#httpcore;4.2.5 from central in [default] | |
org.apache.zookeeper#zookeeper;3.4.6 from central in [default] | |
org.codehaus.jackson#jackson-core-asl;1.9.13 from central in [default] | |
org.codehaus.jackson#jackson-jaxrs;1.9.13 from central in [default] | |
org.codehaus.jackson#jackson-mapper-asl;1.9.13 from central in [default] | |
org.codehaus.jackson#jackson-xc;1.9.13 from central in [default] | |
org.codehaus.jettison#jettison;1.1 from central in [default] | |
org.hamcrest#hamcrest-core;1.3 from central in [default] | |
org.mortbay.jetty#jetty;6.1.26 from central in [default] | |
org.mortbay.jetty#jetty-sslengine;6.1.26 from central in [default] | |
org.mortbay.jetty#jetty-util;6.1.26 from central in [default] | |
org.mortbay.jetty#servlet-api;2.5-20081211 from central in [default] | |
org.slf4j#slf4j-api;1.7.10 from central in [default] | |
org.slf4j#slf4j-log4j12;1.7.10 from central in [default] | |
org.tukaani#xz;1.0 from central in [default] | |
org.xerial.snappy#snappy-java;1.0.4.1 from central in [default] | |
xmlenc#xmlenc;0.52 from central in [default] | |
:: evicted modules: | |
commons-logging#commons-logging;1.1.1 by [commons-logging#commons-logging;1.1.3] in [default] | |
org.apache.httpcomponents#httpclient;4.2 by [org.apache.httpcomponents#httpclient;4.2.5] in [default] | |
org.apache.httpcomponents#httpcore;4.2 by [org.apache.httpcomponents#httpcore;4.2.5] in [default] | |
commons-codec#commons-codec;1.6 by [commons-codec#commons-codec;1.3] in [default] | |
commons-codec#commons-codec;1.3 by [commons-codec#commons-codec;1.4] in [default] | |
com.fasterxml.jackson.core#jackson-core;2.1.1 by [com.fasterxml.jackson.core#jackson-core;2.2.3] in [default] | |
com.fasterxml.jackson.core#jackson-databind;2.1.1 by [com.fasterxml.jackson.core#jackson-databind;2.2.3] in [default] | |
com.fasterxml.jackson.core#jackson-annotations;2.1.1 by [com.fasterxml.jackson.core#jackson-annotations;2.2.3] in [default] | |
--------------------------------------------------------------------- | |
| | modules || artifacts | | |
| conf | number| search|dwnlded|evicted|| number|dwnlded| | |
--------------------------------------------------------------------- | |
| default | 80 | 1 | 0 | 8 || 72 | 0 | | |
--------------------------------------------------------------------- | |
:: retrieving :: org.apache.spark#spark-submit-parent-d3c9d1bd-404e-405d-9bb5-b4419657a1a2 | |
confs: [default] | |
0 artifacts copied, 72 already retrieved (0kB/19ms) | |
19/05/15 20:33:21 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable | |
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties | |
Setting default log level to "WARN". | |
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). | |
Spark context Web UI available at http://xxxxxx:4040 | |
Spark context available as 'sc' (master = local[*], app id = local-1557952407937). | |
Spark session available as 'spark'. | |
Welcome to | |
____ __ | |
/ __/__ ___ _____/ /__ | |
_\ \/ _ \/ _ `/ __/ '_/ | |
/___/ .__/\_,_/_/ /_/\_\ version 2.4.3 | |
/_/ | |
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_212) | |
Type in expressions to have them evaluated. | |
Type :help for more information. | |
scala> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment