-
-
Save jimbobhickville/1f10b3508ef946eccb92 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| spark-submit --properties-file "../conf/app.conf" --packages "com.databricks:spark-csv_2.11:1.2.0" --jars "hdfs://master-1.local:8020/apps/twitter_sentiment/lucene-core-5.3.0.jar,hdfs://master-1.local:8020/apps/twitter_sentiment/lucene-analyzers-common-5.3.0.jar" --class "com.rackspace.spark.Sentiment" --supervise --deploy-mode "cluster" --master spark://master-1.local:7077 hdfs://master-1.local:8020/apps/twitter_sentiment/sentiment-project_2.10-1.0.jar |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Launch Command: "/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.60-2.b27.el7_1.x86_64/jre/bin/java" "-cp" "/usr/lib/spark/lib/spark-examples-1.5.0-hadoop2.7.1.jar:/etc/spark/conf/:/usr/lib/spark/lib/spark-assembly-1.5.0-hadoop2.7.1.jar" "-Xms1024M" "-Xmx1024M" "-Dspark.sentimentApp.numKafkaPartitions=3" "-Dakka.loglevel=WARNING" "-Dspark.sentimentApp.accessToken=[redacted]" "-Dspark.sentimentApp.twitterRefresh=10" "-Dspark.submit.deployMode=cluster" "-Dspark.sentimentApp.clientRefreshInterval=30" "-Dspark.sentimentApp.zookeeperHosts=zookeeper-1.local:2181,zookeeper-2.local:2181,zookeeper-3.local:2181" "-Dspark.sentimentApp.apiSecret=[redacted]" "-Dspark.rpc.askTimeout=10" "-Dspark.sentimentApp.apiKey=[redacted]" "-Dspark.jars=hdfs://master-1.local:8020/apps/twitter_sentiment/lucene-core-5.3.0.jar,hdfs://master-1.local:8020/apps/twitter_sentiment/lucene-analyzers-common-5.3.0.jar,file:/root/.ivy2/jars/com.databricks_spark-csv_2.11-1.2.0.jar,file:/root/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar,file:/root/.ivy2/jars/com.univocity_univocity-parsers-1.5.1.jar,file:/root/.ivy2/jars/com.databricks_spark-csv_2.11-1.2.0.jar,file:/root/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar,file:/root/.ivy2/jars/com.univocity_univocity-parsers-1.5.1.jar,hdfs://master-1.local:8020/apps/twitter_sentiment/sentiment-project_2.10-1.0.jar" "-Dspark.sentimentApp.kafkaQueue=scored-tweets" "-Dspark.driver.supervise=true" "-Dspark.sentimentApp.keywords=[redacted]" "-Dspark.sentimentApp.clientQueueCache=/tmp/tweet_cache.json" "-Dspark.sentimentApp.accessTokenSecret=[redacted]" "-Dspark.sentimentApp.clientQueueSize=150" "-Dspark.sentimentApp.kafkaBrokers=slave-1.local:6667,slave-2.local:6667,slave-3.local:6667" "-Dspark.master=spark://master-1.local:7077" "-Dspark.app.name=com.rackspace.spark.Sentiment" "org.apache.spark.deploy.worker.DriverWrapper" "akka.tcp://sparkWorker@10.223.112.235:59117/user/Worker" "/var/lib/spark/driver-20151210153926-0009/sentiment-project_2.10-1.0.jar" "com.rackspace.spark.Sentiment" | |
| ======================================== | |
| SLF4J: Class path contains multiple SLF4J bindings. | |
| SLF4J: Found binding in [jar:file:/usr/lib/spark/lib/spark-examples-1.5.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
| SLF4J: Found binding in [jar:file:/usr/lib/spark/lib/spark-assembly-1.5.0-hadoop2.7.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
| SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. | |
| SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] | |
| 15/12/10 15:50:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable | |
| 15/12/10 15:50:50 INFO SecurityManager: Changing view acls to: spark | |
| 15/12/10 15:50:50 INFO SecurityManager: Changing modify acls to: spark | |
| 15/12/10 15:50:50 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark) | |
| 15/12/10 15:50:51 INFO Slf4jLogger: Slf4jLogger started | |
| 15/12/10 15:50:51 INFO Utils: Successfully started service 'Driver' on port 33551. | |
| 15/12/10 15:50:51 INFO WorkerWatcher: Connecting to worker akka.tcp://sparkWorker@10.223.112.235:59117/user/Worker | |
| 15/12/10 15:50:51 INFO SparkContext: Running Spark version 1.5.0 | |
| 15/12/10 15:50:51 WARN SparkConf: | |
| SPARK_CLASSPATH was detected (set to '/usr/lib/spark/lib/spark-examples-1.5.0-hadoop2.7.1.jar'). | |
| This is deprecated in Spark 1.0+. | |
| Please instead use: | |
| - ./spark-submit with --driver-class-path to augment the driver classpath | |
| - spark.executor.extraClassPath to augment the executor classpath | |
| 15/12/10 15:50:51 WARN SparkConf: Setting 'spark.executor.extraClassPath' to '/usr/lib/spark/lib/spark-examples-1.5.0-hadoop2.7.1.jar' as a work-around. | |
| 15/12/10 15:50:51 WARN SparkConf: Setting 'spark.driver.extraClassPath' to '/usr/lib/spark/lib/spark-examples-1.5.0-hadoop2.7.1.jar' as a work-around. | |
| 15/12/10 15:50:51 INFO SecurityManager: Changing view acls to: spark | |
| 15/12/10 15:50:51 INFO SecurityManager: Changing modify acls to: spark | |
| 15/12/10 15:50:51 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark) | |
| 15/12/10 15:50:51 INFO WorkerWatcher: Successfully connected to akka.tcp://sparkWorker@10.223.112.235:59117/user/Worker | |
| 15/12/10 15:50:51 INFO Slf4jLogger: Slf4jLogger started | |
| 15/12/10 15:50:51 INFO Utils: Successfully started service 'sparkDriver' on port 57077. | |
| 15/12/10 15:50:51 INFO SparkEnv: Registering MapOutputTracker | |
| 15/12/10 15:50:51 INFO SparkEnv: Registering BlockManagerMaster | |
| 15/12/10 15:50:51 INFO DiskBlockManager: Created local directory at /data1/spark/blockmgr-e7cc1f76-ad3d-4f50-9337-6d34b29cf952 | |
| 15/12/10 15:50:51 INFO MemoryStore: MemoryStore started with capacity 530.0 MB | |
| 15/12/10 15:50:51 INFO HttpFileServer: HTTP File server directory is /data1/spark/spark-ee667b90-7be7-49f0-b542-9349c6f06224/httpd-192c340c-2242-4c87-ba7b-80af868ebbc3 | |
| 15/12/10 15:50:51 INFO HttpServer: Starting HTTP Server | |
| 15/12/10 15:50:52 INFO Utils: Successfully started service 'HTTP file server' on port 32802. | |
| 15/12/10 15:50:52 INFO SparkEnv: Registering OutputCommitCoordinator | |
| 15/12/10 15:50:52 INFO Utils: Successfully started service 'SparkUI' on port 4040. | |
| 15/12/10 15:50:52 INFO SparkUI: Started SparkUI at http://gateway-1.local:4040 | |
| 15/12/10 15:50:52 INFO SparkContext: Added JAR hdfs://master-1.local:8020/apps/twitter_sentiment/lucene-core-5.3.0.jar at hdfs://master-1.local:8020/apps/twitter_sentiment/lucene-core-5.3.0.jar with timestamp 1449762652181 | |
| 15/12/10 15:50:52 INFO SparkContext: Added JAR hdfs://master-1.local:8020/apps/twitter_sentiment/lucene-analyzers-common-5.3.0.jar at hdfs://master-1.local:8020/apps/twitter_sentiment/lucene-analyzers-common-5.3.0.jar with timestamp 1449762652185 | |
| 15/12/10 15:50:52 ERROR SparkContext: Jar not found at file:/root/.ivy2/jars/com.databricks_spark-csv_2.11-1.2.0.jar | |
| 15/12/10 15:50:52 ERROR SparkContext: Jar not found at file:/root/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar | |
| 15/12/10 15:50:52 ERROR SparkContext: Jar not found at file:/root/.ivy2/jars/com.univocity_univocity-parsers-1.5.1.jar | |
| 15/12/10 15:50:52 ERROR SparkContext: Jar not found at file:/root/.ivy2/jars/com.databricks_spark-csv_2.11-1.2.0.jar | |
| 15/12/10 15:50:52 ERROR SparkContext: Jar not found at file:/root/.ivy2/jars/org.apache.commons_commons-csv-1.1.jar | |
| 15/12/10 15:50:52 ERROR SparkContext: Jar not found at file:/root/.ivy2/jars/com.univocity_univocity-parsers-1.5.1.jar | |
| 15/12/10 15:50:52 INFO SparkContext: Added JAR hdfs://master-1.local:8020/apps/twitter_sentiment/sentiment-project_2.10-1.0.jar at hdfs://master-1.local:8020/apps/twitter_sentiment/sentiment-project_2.10-1.0.jar with timestamp 1449762652198 | |
| 15/12/10 15:50:52 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set. | |
| 15/12/10 15:50:52 INFO AppClient$ClientEndpoint: Connecting to master spark://master-1.local:7077... | |
| 15/12/10 15:50:52 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20151210155052-0096 | |
| 15/12/10 15:50:52 INFO AppClient$ClientEndpoint: Executor added: app-20151210155052-0096/0 on worker-20151029032924-10.223.112.199-48093 (10.223.112.199:48093) with 2 cores | |
| 15/12/10 15:50:52 INFO SparkDeploySchedulerBackend: Granted executor ID app-20151210155052-0096/0 on hostPort 10.223.112.199:48093 with 2 cores, 1024.0 MB RAM | |
| 15/12/10 15:50:52 INFO AppClient$ClientEndpoint: Executor added: app-20151210155052-0096/1 on worker-20151029032656-10.223.112.209-39515 (10.223.112.209:39515) with 2 cores | |
| 15/12/10 15:50:52 INFO SparkDeploySchedulerBackend: Granted executor ID app-20151210155052-0096/1 on hostPort 10.223.112.209:39515 with 2 cores, 1024.0 MB RAM | |
| 15/12/10 15:50:52 INFO AppClient$ClientEndpoint: Executor added: app-20151210155052-0096/2 on worker-20151029033207-10.223.112.235-59117 (10.223.112.235:59117) with 1 cores | |
| 15/12/10 15:50:52 INFO SparkDeploySchedulerBackend: Granted executor ID app-20151210155052-0096/2 on hostPort 10.223.112.235:59117 with 1 cores, 1024.0 MB RAM | |
| 15/12/10 15:50:52 INFO AppClient$ClientEndpoint: Executor updated: app-20151210155052-0096/1 is now LOADING | |
| 15/12/10 15:50:52 INFO AppClient$ClientEndpoint: Executor updated: app-20151210155052-0096/0 is now LOADING | |
| 15/12/10 15:50:52 INFO AppClient$ClientEndpoint: Executor updated: app-20151210155052-0096/2 is now LOADING | |
| 15/12/10 15:50:52 INFO AppClient$ClientEndpoint: Executor updated: app-20151210155052-0096/0 is now RUNNING | |
| 15/12/10 15:50:52 INFO AppClient$ClientEndpoint: Executor updated: app-20151210155052-0096/1 is now RUNNING | |
| 15/12/10 15:50:52 INFO AppClient$ClientEndpoint: Executor updated: app-20151210155052-0096/2 is now RUNNING | |
| 15/12/10 15:50:52 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43519. | |
| 15/12/10 15:50:52 INFO NettyBlockTransferService: Server created on 43519 | |
| 15/12/10 15:50:52 INFO BlockManagerMaster: Trying to register BlockManager | |
| 15/12/10 15:50:52 INFO BlockManagerMasterEndpoint: Registering block manager 10.223.112.235:43519 with 530.0 MB RAM, BlockManagerId(driver, 10.223.112.235, 43519) | |
| 15/12/10 15:50:52 INFO BlockManagerMaster: Registered BlockManager | |
| 15/12/10 15:50:53 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 | |
| 15/12/10 15:50:55 INFO SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@10.223.112.199:43065/user/Executor#734925163]) with ID 0 | |
| Exception in thread "main" java.lang.reflect.InvocationTargetException | |
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:497) | |
| at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58) | |
| at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) | |
| Caused by: java.lang.ClassNotFoundException: Failed to load class for data source: com.databricks.spark.csv. | |
| at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:67) | |
| at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:87) | |
| at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114) | |
| at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:104) | |
| at com.rackspace.spark.Sentiment$.prepareNBModel(Sentiment.scala:72) | |
| at com.rackspace.spark.Sentiment$.main(Sentiment.scala:189) | |
| at com.rackspace.spark.Sentiment.main(Sentiment.scala) | |
| ... 6 more | |
| Caused by: java.lang.ClassNotFoundException: com.databricks.spark.csv.DefaultSource | |
| at java.net.URLClassLoader.findClass(URLClassLoader.java:381) | |
| at java.lang.ClassLoader.loadClass(ClassLoader.java:424) | |
| at java.lang.ClassLoader.loadClass(ClassLoader.java:357) | |
| at org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:60) | |
| at org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4$$anonfun$apply$1.apply(ResolvedDataSource.scala:60) | |
| at scala.util.Try$.apply(Try.scala:161) | |
| at org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:60) | |
| at org.apache.spark.sql.execution.datasources.ResolvedDataSource$$anonfun$4.apply(ResolvedDataSource.scala:60) | |
| at scala.util.Try.orElse(Try.scala:82) | |
| at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.lookupDataSource(ResolvedDataSource.scala:60) | |
| ... 12 more | |
| 15/12/10 15:50:55 INFO SparkContext: Invoking stop() from shutdown hook | |
| 15/12/10 15:50:55 INFO BlockManagerMasterEndpoint: Registering block manager 10.223.112.199:42121 with 530.0 MB RAM, BlockManagerId(0, 10.223.112.199, 42121) | |
| 15/12/10 15:50:55 INFO SparkUI: Stopped Spark web UI at http://gateway-1.local:4040 | |
| 15/12/10 15:50:55 INFO DAGScheduler: Stopping DAGScheduler | |
| 15/12/10 15:50:55 INFO SparkDeploySchedulerBackend: Shutting down all executors | |
| 15/12/10 15:50:55 INFO SparkDeploySchedulerBackend: Asking each executor to shut down | |
| 15/12/10 15:50:55 INFO SparkDeploySchedulerBackend: Registered executor: AkkaRpcEndpointRef(Actor[akka.tcp://sparkExecutor@10.223.112.209:46567/user/Executor#1038335329]) with ID 1 | |
| 15/12/10 15:50:55 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! | |
| 15/12/10 15:50:55 INFO MemoryStore: MemoryStore cleared | |
| 15/12/10 15:50:55 INFO BlockManager: BlockManager stopped | |
| 15/12/10 15:50:55 INFO BlockManagerMaster: BlockManagerMaster stopped | |
| 15/12/10 15:50:55 INFO SparkContext: Successfully stopped SparkContext | |
| 15/12/10 15:50:55 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! | |
| 15/12/10 15:50:55 INFO ShutdownHookManager: Shutdown hook called | |
| 15/12/10 15:50:55 INFO ShutdownHookManager: Deleting directory /data1/spark/spark-ee667b90-7be7-49f0-b542-9349c6f06224 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment