Created
June 14, 2017 11:43
-
-
Save anonymous/86df1936e6d9de791c61a6df04844b74 to your computer and use it in GitHub Desktop.
Shred logs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| SLF4J: Class path contains multiple SLF4J bindings. | |
| SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/filecache/10/__spark_libs__1774640650339734057.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
| SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] | |
| SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. | |
| SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] | |
| 17/06/14 04:09:57 INFO SignalUtils: Registered signal handler for TERM | |
| 17/06/14 04:09:57 INFO SignalUtils: Registered signal handler for HUP | |
| 17/06/14 04:09:57 INFO SignalUtils: Registered signal handler for INT | |
| 17/06/14 04:09:57 INFO ApplicationMaster: Preparing Local resources | |
| 17/06/14 04:09:58 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1497410974650_0004_000001 | |
| 17/06/14 04:09:58 INFO SecurityManager: Changing view acls to: yarn,hadoop | |
| 17/06/14 04:09:58 INFO SecurityManager: Changing modify acls to: yarn,hadoop | |
| 17/06/14 04:09:58 INFO SecurityManager: Changing view acls groups to: | |
| 17/06/14 04:09:58 INFO SecurityManager: Changing modify acls groups to: | |
| 17/06/14 04:09:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set() | |
| 17/06/14 04:09:58 INFO ApplicationMaster: Starting the user application in a separate Thread | |
| 17/06/14 04:09:58 INFO ApplicationMaster: Waiting for spark context initialization... | |
| 17/06/14 04:09:58 INFO SparkContext: Running Spark version 2.1.0 | |
| 17/06/14 04:09:58 INFO SecurityManager: Changing view acls to: yarn,hadoop | |
| 17/06/14 04:09:58 INFO SecurityManager: Changing modify acls to: yarn,hadoop | |
| 17/06/14 04:09:58 INFO SecurityManager: Changing view acls groups to: | |
| 17/06/14 04:09:58 INFO SecurityManager: Changing modify acls groups to: | |
| 17/06/14 04:09:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set() | |
| 17/06/14 04:09:58 INFO Utils: Successfully started service 'sparkDriver' on port 43899. | |
| 17/06/14 04:09:58 INFO SparkEnv: Registering MapOutputTracker | |
| 17/06/14 04:09:58 INFO SparkEnv: Registering BlockManagerMaster | |
| 17/06/14 04:09:58 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information | |
| 17/06/14 04:09:58 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up | |
| 17/06/14 04:09:58 INFO DiskBlockManager: Created local directory at /mnt/yarn/usercache/hadoop/appcache/application_1497410974650_0004/blockmgr-ce0152b0-7861-4375-9adc-b66a2cb5fb0b | |
| 17/06/14 04:09:58 INFO MemoryStore: MemoryStore started with capacity 413.9 MB | |
| 17/06/14 04:09:58 INFO SparkEnv: Registering OutputCommitCoordinator | |
| 17/06/14 04:09:59 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter | |
| 17/06/14 04:09:59 INFO Utils: Successfully started service 'SparkUI' on port 32886. | |
| 17/06/14 04:09:59 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.0.49.229:32886 | |
| 17/06/14 04:09:59 INFO YarnClusterScheduler: Created YarnClusterScheduler | |
| 17/06/14 04:09:59 INFO SchedulerExtensionServices: Starting Yarn extension services with app application_1497410974650_0004 and attemptId Some(appattempt_1497410974650_0004_000001) | |
| 17/06/14 04:09:59 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances | |
| 17/06/14 04:09:59 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33721. | |
| 17/06/14 04:09:59 INFO NettyBlockTransferService: Server created on 10.0.49.229:33721 | |
| 17/06/14 04:09:59 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy | |
| 17/06/14 04:09:59 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.0.49.229, 33721, None) | |
| 17/06/14 04:09:59 INFO BlockManagerMasterEndpoint: Registering block manager 10.0.49.229:33721 with 413.9 MB RAM, BlockManagerId(driver, 10.0.49.229, 33721, None) | |
| 17/06/14 04:09:59 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.0.49.229, 33721, None) | |
| 17/06/14 04:09:59 INFO BlockManager: external shuffle service port = 7337 | |
| 17/06/14 04:09:59 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.0.49.229, 33721, None) | |
| 17/06/14 04:09:59 INFO EventLoggingListener: Logging events to hdfs:///var/log/spark/apps/application_1497410974650_0004_1 | |
| 17/06/14 04:09:59 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances | |
| 17/06/14 04:09:59 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered! | |
| 17/06/14 04:09:59 INFO YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 | |
| 17/06/14 04:09:59 INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done | |
| 17/06/14 04:09:59 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@10.0.49.229:43899) | |
| 17/06/14 04:09:59 INFO SharedState: Warehouse path is 'hdfs:///user/spark/warehouse'. | |
| 17/06/14 04:09:59 INFO ApplicationMaster: | |
| =============================================================================== | |
| YARN executor launch context: | |
| env: | |
| CLASSPATH -> /usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*<CPS>{{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>/usr/lib/hadoop-lzo/lib/*<CPS>/usr/share/aws/emr/emrfs/conf<CPS>/usr/share/aws/emr/emrfs/lib/*<CPS>/usr/share/aws/emr/emrfs/auxlib/*<CPS>/usr/share/aws/emr/lib/*<CPS>/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar<CPS>/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar<CPS>/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar<CPS>/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar<CPS>/usr/lib/spark/yarn/lib/datanucleus-core.jar<CPS>/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar<CPS>/usr/share/aws/emr/cloudwatch-sink/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*<CPS>/usr/lib/hadoop-lzo/lib/*<CPS>/usr/share/aws/emr/emrfs/conf<CPS>/usr/share/aws/emr/emrfs/lib/*<CPS>/usr/share/aws/emr/emrfs/auxlib/*<CPS>/usr/share/aws/emr/lib/*<CPS>/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar<CPS>/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar<CPS>/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar<CPS>/usr/share/aws/emr/cloudwatch-sink/lib/* | |
| SPARK_YARN_STAGING_DIR -> hdfs://ip-10-0-51-196.ec2.internal:8020/user/hadoop/.sparkStaging/application_1497410974650_0004 | |
| SPARK_USER -> hadoop | |
| SPARK_YARN_MODE -> true | |
| command: | |
| LD_LIBRARY_PATH="/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:$LD_LIBRARY_PATH" \ | |
| {{JAVA_HOME}}/bin/java \ | |
| -server \ | |
| -Xmx5120m \ | |
| '-verbose:gc' \ | |
| '-XX:+PrintGCDetails' \ | |
| '-XX:+PrintGCDateStamps' \ | |
| '-XX:+UseConcMarkSweepGC' \ | |
| '-XX:CMSInitiatingOccupancyFraction=70' \ | |
| '-XX:MaxHeapFreeRatio=70' \ | |
| '-XX:+CMSClassUnloadingEnabled' \ | |
| '-XX:OnOutOfMemoryError=kill -9 %p' \ | |
| -Djava.io.tmpdir={{PWD}}/tmp \ | |
| '-Dspark.history.ui.port=18080' \ | |
| -Dspark.yarn.app.container.log.dir=<LOG_DIR> \ | |
| org.apache.spark.executor.CoarseGrainedExecutorBackend \ | |
| --driver-url \ | |
| spark://CoarseGrainedScheduler@10.0.49.229:43899 \ | |
| --executor-id \ | |
| <executorId> \ | |
| --hostname \ | |
| <hostname> \ | |
| --cores \ | |
| 4 \ | |
| --app-id \ | |
| application_1497410974650_0004 \ | |
| --user-class-path \ | |
| file:$PWD/__app__.jar \ | |
| 1><LOG_DIR>/stdout \ | |
| 2><LOG_DIR>/stderr | |
| resources: | |
| __app__.jar -> resource { scheme: "hdfs" host: "ip-10-0-51-196.ec2.internal" port: 8020 file: "/user/hadoop/.sparkStaging/application_1497410974650_0004/snowplow-rdb-shredder-0.12.0.jar" } size: 76600934 timestamp: 1497413395326 type: FILE visibility: PRIVATE | |
| __spark_libs__ -> resource { scheme: "hdfs" host: "ip-10-0-51-196.ec2.internal" port: 8020 file: "/user/hadoop/.sparkStaging/application_1497410974650_0004/__spark_libs__1774640650339734057.zip" } size: 196696785 timestamp: 1497413385131 type: ARCHIVE visibility: PRIVATE | |
| __spark_conf__ -> resource { scheme: "hdfs" host: "ip-10-0-51-196.ec2.internal" port: 8020 file: "/user/hadoop/.sparkStaging/application_1497410974650_0004/__spark_conf__.zip" } size: 74219 timestamp: 1497413395576 type: ARCHIVE visibility: PRIVATE | |
| =============================================================================== | |
| 17/06/14 04:09:59 INFO RMProxy: Connecting to ResourceManager at ip-10-0-51-196.ec2.internal/10.0.51.196:8030 | |
| 17/06/14 04:09:59 INFO YarnRMClient: Registering the ApplicationMaster | |
| 17/06/14 04:09:59 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances | |
| 17/06/14 04:09:59 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals | |
| 17/06/14 04:10:01 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 299.5 KB, free 413.6 MB) | |
| 17/06/14 04:10:01 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 26.1 KB, free 413.6 MB) | |
| 17/06/14 04:10:01 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.0.49.229:33721 (size: 26.1 KB, free: 413.9 MB) | |
| 17/06/14 04:10:01 INFO SparkContext: Created broadcast 0 from textFile at ShredJob.scala:340 | |
| 17/06/14 04:10:02 INFO GPLNativeCodeLoader: Loaded native gpl library | |
| 17/06/14 04:10:02 INFO LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 0338e6621a7a828d8224b10e83a4d72395c41c29] | |
| 17/06/14 04:10:02 INFO FileInputFormat: Total input paths to process : 24 | |
| 17/06/14 04:10:03 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id | |
| 17/06/14 04:10:03 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id | |
| 17/06/14 04:10:03 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap | |
| 17/06/14 04:10:03 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition | |
| 17/06/14 04:10:03 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id | |
| 17/06/14 04:10:03 INFO DirectFileOutputCommitter: Nothing to setup since the outputs are written directly. | |
| 17/06/14 04:10:03 INFO SparkContext: Starting job: saveAsTextFile at ShredJob.scala:400 | |
| 17/06/14 04:10:03 INFO DAGScheduler: Registering RDD 7 (groupBy at ShredJob.scala:356) | |
| 17/06/14 04:10:03 INFO DAGScheduler: Got job 0 (saveAsTextFile at ShredJob.scala:400) with 270 output partitions | |
| 17/06/14 04:10:03 INFO DAGScheduler: Final stage: ResultStage 1 (saveAsTextFile at ShredJob.scala:400) | |
| 17/06/14 04:10:03 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) | |
| 17/06/14 04:10:03 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0) | |
| 17/06/14 04:10:03 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[7] at groupBy at ShredJob.scala:356), which has no missing parents | |
| 17/06/14 04:10:03 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 9.3 KB, free 413.6 MB) | |
| 17/06/14 04:10:03 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 5.0 KB, free 413.6 MB) | |
| 17/06/14 04:10:03 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.49.229:33721 (size: 5.0 KB, free: 413.9 MB) | |
| 17/06/14 04:10:03 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:997 | |
| 17/06/14 04:10:03 INFO DAGScheduler: Submitting 135 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[7] at groupBy at ShredJob.scala:356) | |
| 17/06/14 04:10:03 INFO YarnClusterScheduler: Adding task set 0.0 with 135 tasks | |
| 17/06/14 04:10:04 INFO YarnAllocator: Driver requested a total number of 1 executor(s). | |
| 17/06/14 04:10:04 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 1) | |
| 17/06/14 04:10:04 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:10:04 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:05 INFO AMRMClientImpl: Received new token for : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 04:10:05 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000002 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 04:10:05 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:10:05 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:05 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 04:10:05 INFO YarnAllocator: Driver requested a total number of 3 executor(s). | |
| 17/06/14 04:10:05 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:10:05 INFO ExecutorAllocationManager: Requesting 2 new executors because tasks are backlogged (new desired total will be 3) | |
| 17/06/14 04:10:05 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:05 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:06 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000003 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 04:10:06 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000004 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 04:10:06 INFO YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them. | |
| 17/06/14 04:10:06 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:06 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:06 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 04:10:06 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 04:10:06 INFO YarnAllocator: Driver requested a total number of 7 executor(s). | |
| 17/06/14 04:10:06 INFO YarnAllocator: Will request 4 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:10:06 INFO ExecutorAllocationManager: Requesting 4 new executors because tasks are backlogged (new desired total will be 7) | |
| 17/06/14 04:10:06 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:06 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:06 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:06 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:07 INFO AMRMClientImpl: Received new token for : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 04:10:07 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000005 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 04:10:07 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000006 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 04:10:07 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000007 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 04:10:07 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000008 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 04:10:07 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:07 INFO YarnAllocator: Received 4 containers from YARN, launching executors on 4 of them. | |
| 17/06/14 04:10:07 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:07 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:07 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:07 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 04:10:07 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 04:10:07 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 04:10:07 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 04:10:07 INFO YarnAllocator: Driver requested a total number of 15 executor(s). | |
| 17/06/14 04:10:07 INFO YarnAllocator: Will request 8 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:10:07 INFO ExecutorAllocationManager: Requesting 8 new executors because tasks are backlogged (new desired total will be 15) | |
| 17/06/14 04:10:07 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:07 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:07 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:07 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:07 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:07 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:10:07 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:10:07 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:10:07 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:34210) with ID 1 | |
| 17/06/14 04:10:07 INFO ExecutorAllocationManager: New executor 1 has registered (new total is 1) | |
| 17/06/14 04:10:07 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, ip-10-0-51-185.ec2.internal, executor 1, partition 0, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:07 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, ip-10-0-51-185.ec2.internal, executor 1, partition 1, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:07 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, ip-10-0-51-185.ec2.internal, executor 1, partition 2, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:07 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 3, ip-10-0-51-185.ec2.internal, executor 1, partition 5, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:07 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:32987 with 2.8 GB RAM, BlockManagerId(1, ip-10-0-51-185.ec2.internal, 32987, None) | |
| 17/06/14 04:10:07 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:34214) with ID 3 | |
| 17/06/14 04:10:07 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 4, ip-10-0-51-185.ec2.internal, executor 3, partition 11, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:07 INFO ExecutorAllocationManager: New executor 3 has registered (new total is 2) | |
| 17/06/14 04:10:07 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 5, ip-10-0-51-185.ec2.internal, executor 3, partition 14, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:07 INFO TaskSetManager: Starting task 15.0 in stage 0.0 (TID 6, ip-10-0-51-185.ec2.internal, executor 3, partition 15, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:07 INFO TaskSetManager: Starting task 16.0 in stage 0.0 (TID 7, ip-10-0-51-185.ec2.internal, executor 3, partition 16, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:08 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:34392 with 2.8 GB RAM, BlockManagerId(3, ip-10-0-51-185.ec2.internal, 34392, None) | |
| 17/06/14 04:10:08 INFO AMRMClientImpl: Received new token for : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 04:10:08 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000010 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 04:10:08 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000012 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 04:10:08 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000013 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 04:10:08 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000014 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:08 INFO YarnAllocator: Received 4 containers from YARN, launching executors on 4 of them. | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 04:10:08 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:34216) with ID 2 | |
| 17/06/14 04:10:08 INFO TaskSetManager: Starting task 17.0 in stage 0.0 (TID 8, ip-10-0-51-185.ec2.internal, executor 2, partition 17, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:08 INFO TaskSetManager: Starting task 18.0 in stage 0.0 (TID 9, ip-10-0-51-185.ec2.internal, executor 2, partition 18, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:08 INFO ExecutorAllocationManager: New executor 2 has registered (new total is 3) | |
| 17/06/14 04:10:08 INFO TaskSetManager: Starting task 19.0 in stage 0.0 (TID 10, ip-10-0-51-185.ec2.internal, executor 2, partition 19, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:08 INFO TaskSetManager: Starting task 20.0 in stage 0.0 (TID 11, ip-10-0-51-185.ec2.internal, executor 2, partition 20, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:08 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:36390 with 2.8 GB RAM, BlockManagerId(2, ip-10-0-51-185.ec2.internal, 36390, None) | |
| 17/06/14 04:10:08 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-51-185.ec2.internal:32987 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:08 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-51-185.ec2.internal:34392 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:08 INFO YarnAllocator: Driver requested a total number of 31 executor(s). | |
| 17/06/14 04:10:08 INFO ExecutorAllocationManager: Requesting 16 new executors because tasks are backlogged (new desired total will be 31) | |
| 17/06/14 04:10:08 INFO YarnAllocator: Will request 16 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:10:08 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:10:08 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-51-185.ec2.internal:36390 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:08 INFO AMRMClientImpl: Received new token for : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 04:10:08 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000016 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 04:10:08 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000017 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 04:10:08 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000018 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:08 INFO YarnAllocator: Launching container container_1497410974650_0004_01_000019 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 04:10:08 INFO YarnAllocator: Received 4 containers from YARN, launching executors on 4 of them. | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 04:10:08 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 04:10:08 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-51-185.ec2.internal:32987 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:08 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-51-185.ec2.internal:34392 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:08 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-51-185.ec2.internal:36390 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:09 INFO YarnAllocator: Driver requested a total number of 34 executor(s). | |
| 17/06/14 04:10:09 INFO YarnAllocator: Will request 3 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:10:09 INFO ExecutorAllocationManager: Requesting 3 new executors because tasks are backlogged (new desired total will be 34) | |
| 17/06/14 04:10:09 INFO YarnAllocator: Submitted container request for host ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:09 INFO YarnAllocator: Submitted container request for host ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:10:09 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:10:09 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:44698) with ID 5 | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 12, ip-10-0-49-157.ec2.internal, executor 5, partition 3, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO ExecutorAllocationManager: New executor 5 has registered (new total is 4) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 13, ip-10-0-49-157.ec2.internal, executor 5, partition 4, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 14, ip-10-0-49-157.ec2.internal, executor 5, partition 6, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 15, ip-10-0-49-157.ec2.internal, executor 5, partition 7, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:44700) with ID 6 | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 16, ip-10-0-49-157.ec2.internal, executor 6, partition 8, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 17, ip-10-0-49-157.ec2.internal, executor 6, partition 9, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO ExecutorAllocationManager: New executor 6 has registered (new total is 5) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 18, ip-10-0-49-157.ec2.internal, executor 6, partition 10, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 12.0 in stage 0.0 (TID 19, ip-10-0-49-157.ec2.internal, executor 6, partition 12, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:44702) with ID 7 | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 13.0 in stage 0.0 (TID 20, ip-10-0-49-157.ec2.internal, executor 7, partition 13, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 23.0 in stage 0.0 (TID 21, ip-10-0-49-157.ec2.internal, executor 7, partition 23, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 25.0 in stage 0.0 (TID 22, ip-10-0-49-157.ec2.internal, executor 7, partition 25, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 26.0 in stage 0.0 (TID 23, ip-10-0-49-157.ec2.internal, executor 7, partition 26, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO ExecutorAllocationManager: New executor 7 has registered (new total is 6) | |
| 17/06/14 04:10:09 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:44704) with ID 4 | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 27.0 in stage 0.0 (TID 24, ip-10-0-49-157.ec2.internal, executor 4, partition 27, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 28.0 in stage 0.0 (TID 25, ip-10-0-49-157.ec2.internal, executor 4, partition 28, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO ExecutorAllocationManager: New executor 4 has registered (new total is 7) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 29.0 in stage 0.0 (TID 26, ip-10-0-49-157.ec2.internal, executor 4, partition 29, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 30.0 in stage 0.0 (TID 27, ip-10-0-49-157.ec2.internal, executor 4, partition 30, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:43518 with 2.8 GB RAM, BlockManagerId(7, ip-10-0-49-157.ec2.internal, 43518, None) | |
| 17/06/14 04:10:09 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:48778) with ID 10 | |
| 17/06/14 04:10:09 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:45398 with 2.8 GB RAM, BlockManagerId(5, ip-10-0-49-157.ec2.internal, 45398, None) | |
| 17/06/14 04:10:09 INFO ExecutorAllocationManager: New executor 10 has registered (new total is 8) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 21.0 in stage 0.0 (TID 28, ip-10-0-49-229.ec2.internal, executor 10, partition 21, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 22.0 in stage 0.0 (TID 29, ip-10-0-49-229.ec2.internal, executor 10, partition 22, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 32.0 in stage 0.0 (TID 30, ip-10-0-49-229.ec2.internal, executor 10, partition 32, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 35.0 in stage 0.0 (TID 31, ip-10-0-49-229.ec2.internal, executor 10, partition 35, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:48780) with ID 11 | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 38.0 in stage 0.0 (TID 32, ip-10-0-49-229.ec2.internal, executor 11, partition 38, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 42.0 in stage 0.0 (TID 33, ip-10-0-49-229.ec2.internal, executor 11, partition 42, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 43.0 in stage 0.0 (TID 34, ip-10-0-49-229.ec2.internal, executor 11, partition 43, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO TaskSetManager: Starting task 46.0 in stage 0.0 (TID 35, ip-10-0-49-229.ec2.internal, executor 11, partition 46, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:09 INFO ExecutorAllocationManager: New executor 11 has registered (new total is 9) | |
| 17/06/14 04:10:09 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:34052 with 2.8 GB RAM, BlockManagerId(6, ip-10-0-49-157.ec2.internal, 34052, None) | |
| 17/06/14 04:10:10 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:34337 with 2.8 GB RAM, BlockManagerId(4, ip-10-0-49-157.ec2.internal, 34337, None) | |
| 17/06/14 04:10:10 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:46261 with 2.8 GB RAM, BlockManagerId(10, ip-10-0-49-229.ec2.internal, 46261, None) | |
| 17/06/14 04:10:10 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:37760 with 2.8 GB RAM, BlockManagerId(11, ip-10-0-49-229.ec2.internal, 37760, None) | |
| 17/06/14 04:10:10 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:48782) with ID 9 | |
| 17/06/14 04:10:10 INFO TaskSetManager: Starting task 51.0 in stage 0.0 (TID 36, ip-10-0-49-229.ec2.internal, executor 9, partition 51, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:10 INFO TaskSetManager: Starting task 54.0 in stage 0.0 (TID 37, ip-10-0-49-229.ec2.internal, executor 9, partition 54, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:10 INFO TaskSetManager: Starting task 65.0 in stage 0.0 (TID 38, ip-10-0-49-229.ec2.internal, executor 9, partition 65, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:10 INFO TaskSetManager: Starting task 70.0 in stage 0.0 (TID 39, ip-10-0-49-229.ec2.internal, executor 9, partition 70, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:10 INFO ExecutorAllocationManager: New executor 9 has registered (new total is 10) | |
| 17/06/14 04:10:10 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:44338 with 2.8 GB RAM, BlockManagerId(9, ip-10-0-49-229.ec2.internal, 44338, None) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-49-157.ec2.internal:34052 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-49-157.ec2.internal:45398 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-49-157.ec2.internal:43518 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-49-157.ec2.internal:34337 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-49-229.ec2.internal:37760 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-49-229.ec2.internal:44338 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-49-229.ec2.internal:46261 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:10 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:34268) with ID 8 | |
| 17/06/14 04:10:10 INFO TaskSetManager: Starting task 24.0 in stage 0.0 (TID 40, ip-10-0-51-185.ec2.internal, executor 8, partition 24, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:10 INFO TaskSetManager: Starting task 31.0 in stage 0.0 (TID 41, ip-10-0-51-185.ec2.internal, executor 8, partition 31, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:10 INFO TaskSetManager: Starting task 33.0 in stage 0.0 (TID 42, ip-10-0-51-185.ec2.internal, executor 8, partition 33, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:10 INFO ExecutorAllocationManager: New executor 8 has registered (new total is 11) | |
| 17/06/14 04:10:10 INFO TaskSetManager: Starting task 34.0 in stage 0.0 (TID 43, ip-10-0-51-185.ec2.internal, executor 8, partition 34, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:10 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:43229 with 2.8 GB RAM, BlockManagerId(8, ip-10-0-51-185.ec2.internal, 43229, None) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-49-157.ec2.internal:34052 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-49-157.ec2.internal:34337 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-51-185.ec2.internal:43229 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-49-157.ec2.internal:45398 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-49-229.ec2.internal:44338 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-49-157.ec2.internal:43518 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-49-229.ec2.internal:37760 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:10 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-49-229.ec2.internal:46261 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:35516) with ID 14 | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 36.0 in stage 0.0 (TID 44, ip-10-0-50-96.ec2.internal, executor 14, partition 36, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 37.0 in stage 0.0 (TID 45, ip-10-0-50-96.ec2.internal, executor 14, partition 37, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 44.0 in stage 0.0 (TID 46, ip-10-0-50-96.ec2.internal, executor 14, partition 44, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 49.0 in stage 0.0 (TID 47, ip-10-0-50-96.ec2.internal, executor 14, partition 49, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO ExecutorAllocationManager: New executor 14 has registered (new total is 12) | |
| 17/06/14 04:10:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:35520) with ID 13 | |
| 17/06/14 04:10:11 INFO ExecutorAllocationManager: New executor 13 has registered (new total is 13) | |
| 17/06/14 04:10:11 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-51-185.ec2.internal:43229 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 50.0 in stage 0.0 (TID 48, ip-10-0-50-96.ec2.internal, executor 13, partition 50, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 52.0 in stage 0.0 (TID 49, ip-10-0-50-96.ec2.internal, executor 13, partition 52, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 55.0 in stage 0.0 (TID 50, ip-10-0-50-96.ec2.internal, executor 13, partition 55, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 57.0 in stage 0.0 (TID 51, ip-10-0-50-96.ec2.internal, executor 13, partition 57, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:35518) with ID 12 | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 58.0 in stage 0.0 (TID 52, ip-10-0-50-96.ec2.internal, executor 12, partition 58, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 60.0 in stage 0.0 (TID 53, ip-10-0-50-96.ec2.internal, executor 12, partition 60, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO ExecutorAllocationManager: New executor 12 has registered (new total is 14) | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 61.0 in stage 0.0 (TID 54, ip-10-0-50-96.ec2.internal, executor 12, partition 61, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 62.0 in stage 0.0 (TID 55, ip-10-0-50-96.ec2.internal, executor 12, partition 62, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:38815 with 2.8 GB RAM, BlockManagerId(14, ip-10-0-50-96.ec2.internal, 38815, None) | |
| 17/06/14 04:10:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:35522) with ID 15 | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 63.0 in stage 0.0 (TID 56, ip-10-0-50-96.ec2.internal, executor 15, partition 63, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO ExecutorAllocationManager: New executor 15 has registered (new total is 15) | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 71.0 in stage 0.0 (TID 57, ip-10-0-50-96.ec2.internal, executor 15, partition 71, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 74.0 in stage 0.0 (TID 58, ip-10-0-50-96.ec2.internal, executor 15, partition 74, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO TaskSetManager: Starting task 77.0 in stage 0.0 (TID 59, ip-10-0-50-96.ec2.internal, executor 15, partition 77, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:11 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:41521 with 2.8 GB RAM, BlockManagerId(13, ip-10-0-50-96.ec2.internal, 41521, None) | |
| 17/06/14 04:10:11 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:46753 with 2.8 GB RAM, BlockManagerId(12, ip-10-0-50-96.ec2.internal, 46753, None) | |
| 17/06/14 04:10:11 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:32956 with 2.8 GB RAM, BlockManagerId(15, ip-10-0-50-96.ec2.internal, 32956, None) | |
| 17/06/14 04:10:11 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-50-96.ec2.internal:41521 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:11 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-50-96.ec2.internal:38815 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:11 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-50-96.ec2.internal:46753 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:11 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-10-0-50-96.ec2.internal:32956 (size: 5.0 KB, free: 2.8 GB) | |
| 17/06/14 04:10:12 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-50-96.ec2.internal:38815 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:12 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-50-96.ec2.internal:41521 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:12 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-50-96.ec2.internal:46753 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:12 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-50-96.ec2.internal:32956 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:10:36 INFO BlockManagerInfo: Added rdd_2_10 in memory on ip-10-0-49-157.ec2.internal:34052 (size: 154.5 MB, free: 2.6 GB) | |
| 17/06/14 04:10:39 INFO TaskSetManager: Starting task 39.0 in stage 0.0 (TID 60, ip-10-0-49-157.ec2.internal, executor 6, partition 39, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:39 INFO TaskSetManager: Finished task 10.0 in stage 0.0 (TID 18) in 29953 ms on ip-10-0-49-157.ec2.internal (executor 6) (1/135) | |
| 17/06/14 04:10:44 INFO BlockManagerInfo: Added rdd_2_32 in memory on ip-10-0-49-229.ec2.internal:46261 (size: 273.5 MB, free: 2.5 GB) | |
| 17/06/14 04:10:48 INFO BlockManagerInfo: Added rdd_2_4 in memory on ip-10-0-49-157.ec2.internal:45398 (size: 202.3 MB, free: 2.6 GB) | |
| 17/06/14 04:10:48 INFO TaskSetManager: Starting task 76.0 in stage 0.0 (TID 61, ip-10-0-49-229.ec2.internal, executor 10, partition 76, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:48 INFO TaskSetManager: Finished task 32.0 in stage 0.0 (TID 30) in 38492 ms on ip-10-0-49-229.ec2.internal (executor 10) (2/135) | |
| 17/06/14 04:10:50 INFO BlockManagerInfo: Added rdd_2_42 in memory on ip-10-0-49-229.ec2.internal:37760 (size: 329.1 MB, free: 2.4 GB) | |
| 17/06/14 04:10:51 INFO BlockManagerInfo: Added rdd_2_18 in memory on ip-10-0-51-185.ec2.internal:36390 (size: 261.7 MB, free: 2.5 GB) | |
| 17/06/14 04:10:51 INFO TaskSetManager: Starting task 40.0 in stage 0.0 (TID 62, ip-10-0-49-157.ec2.internal, executor 5, partition 40, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:51 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 13) in 41962 ms on ip-10-0-49-157.ec2.internal (executor 5) (3/135) | |
| 17/06/14 04:10:51 INFO YarnAllocator: Driver requested a total number of 33 executor(s). | |
| 17/06/14 04:10:51 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 33 executors. | |
| 17/06/14 04:10:52 INFO BlockManagerInfo: Added rdd_2_51 in memory on ip-10-0-49-229.ec2.internal:44338 (size: 346.8 MB, free: 2.4 GB) | |
| 17/06/14 04:10:54 INFO TaskSetManager: Starting task 78.0 in stage 0.0 (TID 63, ip-10-0-49-229.ec2.internal, executor 11, partition 78, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:54 INFO TaskSetManager: Finished task 42.0 in stage 0.0 (TID 33) in 44768 ms on ip-10-0-49-229.ec2.internal (executor 11) (4/135) | |
| 17/06/14 04:10:55 INFO TaskSetManager: Starting task 41.0 in stage 0.0 (TID 64, ip-10-0-51-185.ec2.internal, executor 2, partition 41, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:55 INFO TaskSetManager: Finished task 18.0 in stage 0.0 (TID 9) in 47264 ms on ip-10-0-51-185.ec2.internal (executor 2) (5/135) | |
| 17/06/14 04:10:56 INFO TaskSetManager: Starting task 79.0 in stage 0.0 (TID 65, ip-10-0-49-229.ec2.internal, executor 9, partition 79, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:10:56 INFO TaskSetManager: Finished task 51.0 in stage 0.0 (TID 36) in 46668 ms on ip-10-0-49-229.ec2.internal (executor 9) (6/135) | |
| 17/06/14 04:11:00 INFO BlockManagerInfo: Added rdd_2_6 in memory on ip-10-0-49-157.ec2.internal:45398 (size: 300.5 MB, free: 2.3 GB) | |
| 17/06/14 04:11:00 INFO BlockManagerInfo: Added rdd_2_5 in memory on ip-10-0-51-185.ec2.internal:32987 (size: 307.2 MB, free: 2.5 GB) | |
| 17/06/14 04:11:00 INFO BlockManagerInfo: Added rdd_2_7 in memory on ip-10-0-49-157.ec2.internal:45398 (size: 313.2 MB, free: 2011.1 MB) | |
| 17/06/14 04:11:03 INFO TaskSetManager: Starting task 45.0 in stage 0.0 (TID 66, ip-10-0-49-157.ec2.internal, executor 5, partition 45, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:03 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 14) in 53795 ms on ip-10-0-49-157.ec2.internal (executor 5) (7/135) | |
| 17/06/14 04:11:03 INFO YarnAllocator: Driver requested a total number of 32 executor(s). | |
| 17/06/14 04:11:03 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 32 executors. | |
| 17/06/14 04:11:04 INFO TaskSetManager: Starting task 47.0 in stage 0.0 (TID 67, ip-10-0-49-157.ec2.internal, executor 5, partition 47, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:04 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 15) in 54464 ms on ip-10-0-49-157.ec2.internal (executor 5) (8/135) | |
| 17/06/14 04:11:05 INFO TaskSetManager: Starting task 48.0 in stage 0.0 (TID 68, ip-10-0-51-185.ec2.internal, executor 1, partition 48, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:05 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 3) in 57203 ms on ip-10-0-51-185.ec2.internal (executor 1) (9/135) | |
| 17/06/14 04:11:07 INFO BlockManagerInfo: Added rdd_2_8 in memory on ip-10-0-49-157.ec2.internal:34052 (size: 368.8 MB, free: 2.2 GB) | |
| 17/06/14 04:11:11 INFO TaskSetManager: Starting task 53.0 in stage 0.0 (TID 69, ip-10-0-49-157.ec2.internal, executor 6, partition 53, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:11 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 16) in 61593 ms on ip-10-0-49-157.ec2.internal (executor 6) (10/135) | |
| 17/06/14 04:11:15 INFO BlockManagerInfo: Added rdd_2_2 in memory on ip-10-0-51-185.ec2.internal:32987 (size: 436.2 MB, free: 2.0 GB) | |
| 17/06/14 04:11:17 INFO BlockManagerInfo: Added rdd_2_60 in memory on ip-10-0-50-96.ec2.internal:46753 (size: 421.9 MB, free: 2.3 GB) | |
| 17/06/14 04:11:20 INFO TaskSetManager: Starting task 56.0 in stage 0.0 (TID 70, ip-10-0-51-185.ec2.internal, executor 1, partition 56, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:20 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 72483 ms on ip-10-0-51-185.ec2.internal (executor 1) (11/135) | |
| 17/06/14 04:11:20 INFO YarnAllocator: Driver requested a total number of 31 executor(s). | |
| 17/06/14 04:11:20 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 31 executors. | |
| 17/06/14 04:11:22 INFO TaskSetManager: Starting task 80.0 in stage 0.0 (TID 71, ip-10-0-50-96.ec2.internal, executor 12, partition 80, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:22 INFO TaskSetManager: Finished task 60.0 in stage 0.0 (TID 53) in 71322 ms on ip-10-0-50-96.ec2.internal (executor 12) (12/135) | |
| 17/06/14 04:11:34 INFO BlockManagerInfo: Added rdd_2_54 in memory on ip-10-0-49-229.ec2.internal:44338 (size: 811.5 MB, free: 1668.8 MB) | |
| 17/06/14 04:11:35 INFO BlockManagerInfo: Added rdd_2_38 in memory on ip-10-0-49-229.ec2.internal:37760 (size: 772.7 MB, free: 1725.3 MB) | |
| 17/06/14 04:11:35 INFO BlockManagerInfo: Added rdd_2_24 in memory on ip-10-0-51-185.ec2.internal:43229 (size: 579.6 MB, free: 2.2 GB) | |
| 17/06/14 04:11:36 INFO BlockManagerInfo: Added rdd_2_70 in memory on ip-10-0-49-229.ec2.internal:44338 (size: 834.3 MB, free: 834.5 MB) | |
| 17/06/14 04:11:36 INFO BlockManagerInfo: Added rdd_2_43 in memory on ip-10-0-49-229.ec2.internal:37760 (size: 759.2 MB, free: 966.1 MB) | |
| 17/06/14 04:11:38 INFO BlockManagerInfo: Added rdd_2_21 in memory on ip-10-0-49-229.ec2.internal:46261 (size: 778.4 MB, free: 1775.2 MB) | |
| 17/06/14 04:11:40 INFO TaskSetManager: Starting task 83.0 in stage 0.0 (TID 72, ip-10-0-49-229.ec2.internal, executor 9, partition 83, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:40 INFO TaskSetManager: Finished task 54.0 in stage 0.0 (TID 37) in 90679 ms on ip-10-0-49-229.ec2.internal (executor 9) (13/135) | |
| 17/06/14 04:11:40 INFO TaskSetManager: Starting task 86.0 in stage 0.0 (TID 73, ip-10-0-49-229.ec2.internal, executor 11, partition 86, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:40 INFO TaskSetManager: Finished task 38.0 in stage 0.0 (TID 32) in 90923 ms on ip-10-0-49-229.ec2.internal (executor 11) (14/135) | |
| 17/06/14 04:11:41 INFO TaskSetManager: Starting task 87.0 in stage 0.0 (TID 74, ip-10-0-49-229.ec2.internal, executor 9, partition 87, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:41 INFO TaskSetManager: Finished task 65.0 in stage 0.0 (TID 38) in 91746 ms on ip-10-0-49-229.ec2.internal (executor 9) (15/135) | |
| 17/06/14 04:11:41 INFO YarnAllocator: Driver requested a total number of 30 executor(s). | |
| 17/06/14 04:11:41 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 30 executors. | |
| 17/06/14 04:11:42 INFO TaskSetManager: Starting task 88.0 in stage 0.0 (TID 75, ip-10-0-49-229.ec2.internal, executor 10, partition 88, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:42 INFO TaskSetManager: Finished task 35.0 in stage 0.0 (TID 31) in 92035 ms on ip-10-0-49-229.ec2.internal (executor 10) (16/135) | |
| 17/06/14 04:11:42 INFO TaskSetManager: Starting task 90.0 in stage 0.0 (TID 76, ip-10-0-49-229.ec2.internal, executor 11, partition 90, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:42 INFO TaskSetManager: Finished task 43.0 in stage 0.0 (TID 34) in 92697 ms on ip-10-0-49-229.ec2.internal (executor 11) (17/135) | |
| 17/06/14 04:11:42 INFO TaskSetManager: Starting task 97.0 in stage 0.0 (TID 77, ip-10-0-49-229.ec2.internal, executor 10, partition 97, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:42 INFO TaskSetManager: Finished task 22.0 in stage 0.0 (TID 29) in 92739 ms on ip-10-0-49-229.ec2.internal (executor 10) (18/135) | |
| 17/06/14 04:11:42 INFO TaskSetManager: Starting task 101.0 in stage 0.0 (TID 78, ip-10-0-49-229.ec2.internal, executor 9, partition 101, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:42 INFO TaskSetManager: Finished task 70.0 in stage 0.0 (TID 39) in 92692 ms on ip-10-0-49-229.ec2.internal (executor 9) (19/135) | |
| 17/06/14 04:11:42 INFO YarnAllocator: Driver requested a total number of 29 executor(s). | |
| 17/06/14 04:11:42 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 29 executors. | |
| 17/06/14 04:11:42 INFO TaskSetManager: Starting task 59.0 in stage 0.0 (TID 79, ip-10-0-51-185.ec2.internal, executor 8, partition 59, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:42 INFO TaskSetManager: Finished task 24.0 in stage 0.0 (TID 40) in 92353 ms on ip-10-0-51-185.ec2.internal (executor 8) (20/135) | |
| 17/06/14 04:11:45 INFO TaskSetManager: Starting task 102.0 in stage 0.0 (TID 80, ip-10-0-49-229.ec2.internal, executor 10, partition 102, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:45 INFO TaskSetManager: Finished task 21.0 in stage 0.0 (TID 28) in 95275 ms on ip-10-0-49-229.ec2.internal (executor 10) (21/135) | |
| 17/06/14 04:11:46 INFO TaskSetManager: Starting task 103.0 in stage 0.0 (TID 81, ip-10-0-49-229.ec2.internal, executor 11, partition 103, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:46 INFO TaskSetManager: Finished task 46.0 in stage 0.0 (TID 35) in 97016 ms on ip-10-0-49-229.ec2.internal (executor 11) (22/135) | |
| 17/06/14 04:11:47 INFO TaskSetManager: Starting task 64.0 in stage 0.0 (TID 82, ip-10-0-49-157.ec2.internal, executor 7, partition 64, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:47 INFO TaskSetManager: Finished task 13.0 in stage 0.0 (TID 20) in 97321 ms on ip-10-0-49-157.ec2.internal (executor 7) (23/135) | |
| 17/06/14 04:11:47 INFO YarnAllocator: Driver requested a total number of 28 executor(s). | |
| 17/06/14 04:11:47 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 28 executors. | |
| 17/06/14 04:11:52 INFO BlockManagerInfo: Added rdd_2_76 in memory on ip-10-0-49-229.ec2.internal:46261 (size: 579.2 MB, free: 1196.0 MB) | |
| 17/06/14 04:11:58 INFO TaskSetManager: Starting task 104.0 in stage 0.0 (TID 83, ip-10-0-49-229.ec2.internal, executor 10, partition 104, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:11:58 INFO TaskSetManager: Finished task 76.0 in stage 0.0 (TID 61) in 69702 ms on ip-10-0-49-229.ec2.internal (executor 10) (24/135) | |
| 17/06/14 04:11:59 INFO BlockManagerInfo: Added rdd_2_1 in memory on ip-10-0-51-185.ec2.internal:32987 (size: 794.8 MB, free: 1288.9 MB) | |
| 17/06/14 04:12:00 INFO BlockManagerInfo: Added rdd_2_50 in memory on ip-10-0-50-96.ec2.internal:41521 (size: 805.2 MB, free: 2021.9 MB) | |
| 17/06/14 04:12:00 INFO BlockManagerInfo: Added rdd_2_58 in memory on ip-10-0-50-96.ec2.internal:46753 (size: 827.4 MB, free: 1577.7 MB) | |
| 17/06/14 04:12:01 INFO BlockManagerInfo: Added rdd_2_57 in memory on ip-10-0-50-96.ec2.internal:41521 (size: 786.4 MB, free: 1235.4 MB) | |
| 17/06/14 04:12:01 INFO BlockManagerInfo: Added rdd_2_63 in memory on ip-10-0-50-96.ec2.internal:32956 (size: 783.8 MB, free: 2043.3 MB) | |
| 17/06/14 04:12:01 INFO BlockManagerInfo: Added rdd_2_36 in memory on ip-10-0-50-96.ec2.internal:38815 (size: 763.3 MB, free: 2.0 GB) | |
| 17/06/14 04:12:02 INFO BlockManagerInfo: Added rdd_2_74 in memory on ip-10-0-50-96.ec2.internal:32956 (size: 800.1 MB, free: 1243.2 MB) | |
| 17/06/14 04:12:02 INFO BlockManagerInfo: Added rdd_2_15 in memory on ip-10-0-51-185.ec2.internal:34392 (size: 825.1 MB, free: 2002.0 MB) | |
| 17/06/14 04:12:03 INFO BlockManagerInfo: Added rdd_2_37 in memory on ip-10-0-50-96.ec2.internal:38815 (size: 793.2 MB, free: 1270.6 MB) | |
| 17/06/14 04:12:04 INFO BlockManagerInfo: Added rdd_2_33 in memory on ip-10-0-51-185.ec2.internal:43229 (size: 831.2 MB, free: 1416.2 MB) | |
| 17/06/14 04:12:04 INFO BlockManagerInfo: Added rdd_2_62 in memory on ip-10-0-50-96.ec2.internal:46753 (size: 825.9 MB, free: 751.8 MB) | |
| 17/06/14 04:12:05 INFO BlockManagerInfo: Added rdd_2_34 in memory on ip-10-0-51-185.ec2.internal:43229 (size: 775.9 MB, free: 640.4 MB) | |
| 17/06/14 04:12:05 INFO TaskSetManager: Starting task 66.0 in stage 0.0 (TID 84, ip-10-0-51-185.ec2.internal, executor 1, partition 66, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:05 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 118012 ms on ip-10-0-51-185.ec2.internal (executor 1) (25/135) | |
| 17/06/14 04:12:06 INFO TaskSetManager: Starting task 67.0 in stage 0.0 (TID 85, ip-10-0-51-185.ec2.internal, executor 1, partition 67, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:06 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 118757 ms on ip-10-0-51-185.ec2.internal (executor 1) (26/135) | |
| 17/06/14 04:12:06 INFO TaskSetManager: Starting task 68.0 in stage 0.0 (TID 86, ip-10-0-51-185.ec2.internal, executor 2, partition 68, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:06 INFO TaskSetManager: Finished task 20.0 in stage 0.0 (TID 11) in 118680 ms on ip-10-0-51-185.ec2.internal (executor 2) (27/135) | |
| 17/06/14 04:12:06 INFO YarnAllocator: Driver requested a total number of 27 executor(s). | |
| 17/06/14 04:12:06 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 27 executors. | |
| 17/06/14 04:12:08 INFO TaskSetManager: Starting task 81.0 in stage 0.0 (TID 87, ip-10-0-50-96.ec2.internal, executor 13, partition 81, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:08 INFO TaskSetManager: Finished task 50.0 in stage 0.0 (TID 48) in 116936 ms on ip-10-0-50-96.ec2.internal (executor 13) (28/135) | |
| 17/06/14 04:12:08 INFO TaskSetManager: Starting task 69.0 in stage 0.0 (TID 88, ip-10-0-49-157.ec2.internal, executor 7, partition 69, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:08 INFO TaskSetManager: Finished task 23.0 in stage 0.0 (TID 21) in 118424 ms on ip-10-0-49-157.ec2.internal (executor 7) (29/135) | |
| 17/06/14 04:12:08 INFO TaskSetManager: Starting task 72.0 in stage 0.0 (TID 89, ip-10-0-51-185.ec2.internal, executor 3, partition 72, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:08 INFO TaskSetManager: Finished task 16.0 in stage 0.0 (TID 7) in 120368 ms on ip-10-0-51-185.ec2.internal (executor 3) (30/135) | |
| 17/06/14 04:12:08 INFO TaskSetManager: Starting task 73.0 in stage 0.0 (TID 90, ip-10-0-51-185.ec2.internal, executor 3, partition 73, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:08 INFO TaskSetManager: Finished task 14.0 in stage 0.0 (TID 5) in 120564 ms on ip-10-0-51-185.ec2.internal (executor 3) (31/135) | |
| 17/06/14 04:12:08 INFO YarnAllocator: Driver requested a total number of 26 executor(s). | |
| 17/06/14 04:12:08 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 26 executors. | |
| 17/06/14 04:12:08 INFO TaskSetManager: Starting task 82.0 in stage 0.0 (TID 91, ip-10-0-50-96.ec2.internal, executor 12, partition 82, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:08 INFO TaskSetManager: Finished task 58.0 in stage 0.0 (TID 52) in 117546 ms on ip-10-0-50-96.ec2.internal (executor 12) (32/135) | |
| 17/06/14 04:12:09 INFO BlockManagerInfo: Added rdd_2_29 in memory on ip-10-0-49-157.ec2.internal:34337 (size: 865.0 MB, free: 1962.0 MB) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Starting task 84.0 in stage 0.0 (TID 92, ip-10-0-50-96.ec2.internal, executor 13, partition 84, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Finished task 57.0 in stage 0.0 (TID 51) in 117828 ms on ip-10-0-50-96.ec2.internal (executor 13) (33/135) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Starting task 94.0 in stage 0.0 (TID 93, ip-10-0-50-96.ec2.internal, executor 15, partition 94, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Finished task 71.0 in stage 0.0 (TID 57) in 117841 ms on ip-10-0-50-96.ec2.internal (executor 15) (34/135) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Starting task 75.0 in stage 0.0 (TID 94, ip-10-0-51-185.ec2.internal, executor 2, partition 75, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Finished task 19.0 in stage 0.0 (TID 10) in 121194 ms on ip-10-0-51-185.ec2.internal (executor 2) (35/135) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Starting task 85.0 in stage 0.0 (TID 95, ip-10-0-51-185.ec2.internal, executor 2, partition 85, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Finished task 17.0 in stage 0.0 (TID 8) in 121214 ms on ip-10-0-51-185.ec2.internal (executor 2) (36/135) | |
| 17/06/14 04:12:09 INFO YarnAllocator: Driver requested a total number of 25 executor(s). | |
| 17/06/14 04:12:09 INFO TaskSetManager: Starting task 96.0 in stage 0.0 (TID 96, ip-10-0-50-96.ec2.internal, executor 15, partition 96, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Finished task 63.0 in stage 0.0 (TID 56) in 118073 ms on ip-10-0-50-96.ec2.internal (executor 15) (37/135) | |
| 17/06/14 04:12:09 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 25 executors. | |
| 17/06/14 04:12:09 INFO TaskSetManager: Starting task 98.0 in stage 0.0 (TID 97, ip-10-0-50-96.ec2.internal, executor 15, partition 98, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Finished task 74.0 in stage 0.0 (TID 58) in 118299 ms on ip-10-0-50-96.ec2.internal (executor 15) (38/135) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Starting task 89.0 in stage 0.0 (TID 98, ip-10-0-51-185.ec2.internal, executor 3, partition 89, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Finished task 15.0 in stage 0.0 (TID 6) in 121723 ms on ip-10-0-51-185.ec2.internal (executor 3) (39/135) | |
| 17/06/14 04:12:09 INFO YarnAllocator: Driver requested a total number of 24 executor(s). | |
| 17/06/14 04:12:09 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 24 executors. | |
| 17/06/14 04:12:09 INFO TaskSetManager: Starting task 100.0 in stage 0.0 (TID 99, ip-10-0-50-96.ec2.internal, executor 14, partition 100, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Finished task 36.0 in stage 0.0 (TID 44) in 118701 ms on ip-10-0-50-96.ec2.internal (executor 14) (40/135) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Starting task 91.0 in stage 0.0 (TID 100, ip-10-0-49-157.ec2.internal, executor 7, partition 91, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Finished task 26.0 in stage 0.0 (TID 23) in 120126 ms on ip-10-0-49-157.ec2.internal (executor 7) (41/135) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Starting task 92.0 in stage 0.0 (TID 101, ip-10-0-49-157.ec2.internal, executor 7, partition 92, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:09 INFO TaskSetManager: Finished task 25.0 in stage 0.0 (TID 22) in 120152 ms on ip-10-0-49-157.ec2.internal (executor 7) (42/135) | |
| 17/06/14 04:12:10 INFO TaskSetManager: Starting task 108.0 in stage 0.0 (TID 102, ip-10-0-50-96.ec2.internal, executor 13, partition 108, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:10 INFO TaskSetManager: Finished task 55.0 in stage 0.0 (TID 50) in 118812 ms on ip-10-0-50-96.ec2.internal (executor 13) (43/135) | |
| 17/06/14 04:12:10 INFO YarnAllocator: Driver requested a total number of 23 executor(s). | |
| 17/06/14 04:12:10 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 23 executors. | |
| 17/06/14 04:12:10 INFO TaskSetManager: Starting task 112.0 in stage 0.0 (TID 103, ip-10-0-50-96.ec2.internal, executor 13, partition 112, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:10 INFO TaskSetManager: Finished task 52.0 in stage 0.0 (TID 49) in 119052 ms on ip-10-0-50-96.ec2.internal (executor 13) (44/135) | |
| 17/06/14 04:12:10 INFO TaskSetManager: Starting task 93.0 in stage 0.0 (TID 104, ip-10-0-51-185.ec2.internal, executor 8, partition 93, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:10 INFO TaskSetManager: Finished task 31.0 in stage 0.0 (TID 41) in 120163 ms on ip-10-0-51-185.ec2.internal (executor 8) (45/135) | |
| 17/06/14 04:12:10 INFO TaskSetManager: Starting task 118.0 in stage 0.0 (TID 105, ip-10-0-50-96.ec2.internal, executor 14, partition 118, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:10 INFO TaskSetManager: Finished task 49.0 in stage 0.0 (TID 47) in 119678 ms on ip-10-0-50-96.ec2.internal (executor 14) (46/135) | |
| 17/06/14 04:12:10 INFO TaskSetManager: Starting task 119.0 in stage 0.0 (TID 106, ip-10-0-50-96.ec2.internal, executor 14, partition 119, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:10 INFO TaskSetManager: Finished task 44.0 in stage 0.0 (TID 46) in 119768 ms on ip-10-0-50-96.ec2.internal (executor 14) (47/135) | |
| 17/06/14 04:12:10 INFO TaskSetManager: Starting task 121.0 in stage 0.0 (TID 107, ip-10-0-50-96.ec2.internal, executor 12, partition 121, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:10 INFO TaskSetManager: Finished task 61.0 in stage 0.0 (TID 54) in 119689 ms on ip-10-0-50-96.ec2.internal (executor 12) (48/135) | |
| 17/06/14 04:12:10 INFO YarnAllocator: Driver requested a total number of 22 executor(s). | |
| 17/06/14 04:12:11 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 22 executors. | |
| 17/06/14 04:12:11 INFO TaskSetManager: Starting task 124.0 in stage 0.0 (TID 108, ip-10-0-50-96.ec2.internal, executor 14, partition 124, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:11 INFO TaskSetManager: Finished task 37.0 in stage 0.0 (TID 45) in 120321 ms on ip-10-0-50-96.ec2.internal (executor 14) (49/135) | |
| 17/06/14 04:12:12 INFO TaskSetManager: Starting task 95.0 in stage 0.0 (TID 109, ip-10-0-51-185.ec2.internal, executor 8, partition 95, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:12 INFO TaskSetManager: Finished task 33.0 in stage 0.0 (TID 42) in 121488 ms on ip-10-0-51-185.ec2.internal (executor 8) (50/135) | |
| 17/06/14 04:12:12 INFO TaskSetManager: Starting task 99.0 in stage 0.0 (TID 110, ip-10-0-51-185.ec2.internal, executor 3, partition 99, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:12 INFO TaskSetManager: Finished task 11.0 in stage 0.0 (TID 4) in 124091 ms on ip-10-0-51-185.ec2.internal (executor 3) (51/135) | |
| 17/06/14 04:12:12 INFO YarnAllocator: Driver requested a total number of 21 executor(s). | |
| 17/06/14 04:12:12 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 21 executors. | |
| 17/06/14 04:12:12 INFO TaskSetManager: Starting task 125.0 in stage 0.0 (TID 111, ip-10-0-50-96.ec2.internal, executor 12, partition 125, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:12 INFO TaskSetManager: Finished task 62.0 in stage 0.0 (TID 55) in 121294 ms on ip-10-0-50-96.ec2.internal (executor 12) (52/135) | |
| 17/06/14 04:12:12 INFO TaskSetManager: Starting task 105.0 in stage 0.0 (TID 112, ip-10-0-51-185.ec2.internal, executor 8, partition 105, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:12 INFO TaskSetManager: Finished task 34.0 in stage 0.0 (TID 43) in 122211 ms on ip-10-0-51-185.ec2.internal (executor 8) (53/135) | |
| 17/06/14 04:12:12 INFO TaskSetManager: Starting task 107.0 in stage 0.0 (TID 113, ip-10-0-49-157.ec2.internal, executor 4, partition 107, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:12 INFO TaskSetManager: Finished task 30.0 in stage 0.0 (TID 27) in 122929 ms on ip-10-0-49-157.ec2.internal (executor 4) (54/135) | |
| 17/06/14 04:12:12 INFO TaskSetManager: Starting task 109.0 in stage 0.0 (TID 114, ip-10-0-49-157.ec2.internal, executor 5, partition 109, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:12 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 12) in 123100 ms on ip-10-0-49-157.ec2.internal (executor 5) (55/135) | |
| 17/06/14 04:12:13 INFO YarnAllocator: Driver requested a total number of 20 executor(s). | |
| 17/06/14 04:12:13 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 20 executors. | |
| 17/06/14 04:12:13 INFO TaskSetManager: Starting task 131.0 in stage 0.0 (TID 115, ip-10-0-50-96.ec2.internal, executor 15, partition 131, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:13 INFO TaskSetManager: Finished task 77.0 in stage 0.0 (TID 59) in 121804 ms on ip-10-0-50-96.ec2.internal (executor 15) (56/135) | |
| 17/06/14 04:12:13 INFO TaskSetManager: Starting task 110.0 in stage 0.0 (TID 116, ip-10-0-49-157.ec2.internal, executor 6, partition 110, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:13 INFO TaskSetManager: Finished task 12.0 in stage 0.0 (TID 19) in 123601 ms on ip-10-0-49-157.ec2.internal (executor 6) (57/135) | |
| 17/06/14 04:12:13 INFO TaskSetManager: Starting task 111.0 in stage 0.0 (TID 117, ip-10-0-49-157.ec2.internal, executor 4, partition 111, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:13 INFO TaskSetManager: Finished task 28.0 in stage 0.0 (TID 25) in 123893 ms on ip-10-0-49-157.ec2.internal (executor 4) (58/135) | |
| 17/06/14 04:12:14 INFO TaskSetManager: Starting task 113.0 in stage 0.0 (TID 118, ip-10-0-49-157.ec2.internal, executor 4, partition 113, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:14 INFO TaskSetManager: Finished task 27.0 in stage 0.0 (TID 24) in 124309 ms on ip-10-0-49-157.ec2.internal (executor 4) (59/135) | |
| 17/06/14 04:12:14 INFO YarnAllocator: Driver requested a total number of 19 executor(s). | |
| 17/06/14 04:12:14 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 19 executors. | |
| 17/06/14 04:12:16 INFO TaskSetManager: Starting task 114.0 in stage 0.0 (TID 119, ip-10-0-49-157.ec2.internal, executor 6, partition 114, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:16 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 17) in 126549 ms on ip-10-0-49-157.ec2.internal (executor 6) (60/135) | |
| 17/06/14 04:12:16 INFO TaskSetManager: Starting task 115.0 in stage 0.0 (TID 120, ip-10-0-49-157.ec2.internal, executor 4, partition 115, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:16 INFO TaskSetManager: Finished task 29.0 in stage 0.0 (TID 26) in 126787 ms on ip-10-0-49-157.ec2.internal (executor 4) (61/135) | |
| 17/06/14 04:12:21 INFO TaskSetManager: Starting task 106.0 in stage 0.0 (TID 121, ip-10-0-49-229.ec2.internal, executor 11, partition 106, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:21 INFO TaskSetManager: Finished task 78.0 in stage 0.0 (TID 63) in 86639 ms on ip-10-0-49-229.ec2.internal (executor 11) (62/135) | |
| 17/06/14 04:12:21 INFO TaskSetManager: Starting task 126.0 in stage 0.0 (TID 122, ip-10-0-49-229.ec2.internal, executor 9, partition 126, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:21 INFO TaskSetManager: Finished task 79.0 in stage 0.0 (TID 65) in 84780 ms on ip-10-0-49-229.ec2.internal (executor 9) (63/135) | |
| 17/06/14 04:12:21 INFO YarnAllocator: Driver requested a total number of 18 executor(s). | |
| 17/06/14 04:12:21 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 18 executors. | |
| 17/06/14 04:12:22 INFO BlockManagerInfo: Added rdd_2_68 in memory on ip-10-0-51-185.ec2.internal:36390 (size: 95.2 MB, free: 2.4 GB) | |
| 17/06/14 04:12:23 INFO TaskSetManager: Starting task 116.0 in stage 0.0 (TID 123, ip-10-0-51-185.ec2.internal, executor 2, partition 116, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:23 INFO TaskSetManager: Finished task 68.0 in stage 0.0 (TID 86) in 17162 ms on ip-10-0-51-185.ec2.internal (executor 2) (64/135) | |
| 17/06/14 04:12:29 INFO TaskSetManager: Starting task 117.0 in stage 0.0 (TID 124, ip-10-0-49-157.ec2.internal, executor 6, partition 117, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:29 INFO TaskSetManager: Finished task 39.0 in stage 0.0 (TID 60) in 110030 ms on ip-10-0-49-157.ec2.internal (executor 6) (65/135) | |
| 17/06/14 04:12:38 INFO BlockManagerInfo: Added rdd_2_115 in memory on ip-10-0-49-157.ec2.internal:34337 (size: 156.1 MB, free: 1805.9 MB) | |
| 17/06/14 04:12:40 INFO TaskSetManager: Starting task 120.0 in stage 0.0 (TID 125, ip-10-0-49-157.ec2.internal, executor 4, partition 120, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:40 INFO TaskSetManager: Finished task 115.0 in stage 0.0 (TID 120) in 23629 ms on ip-10-0-49-157.ec2.internal (executor 4) (66/135) | |
| 17/06/14 04:12:42 INFO BlockManagerInfo: Added rdd_2_41 in memory on ip-10-0-51-185.ec2.internal:36390 (size: 800.6 MB, free: 1669.7 MB) | |
| 17/06/14 04:12:43 INFO TaskSetManager: Starting task 122.0 in stage 0.0 (TID 126, ip-10-0-49-157.ec2.internal, executor 5, partition 122, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:43 INFO TaskSetManager: Finished task 40.0 in stage 0.0 (TID 62) in 111508 ms on ip-10-0-49-157.ec2.internal (executor 5) (67/135) | |
| 17/06/14 04:12:43 INFO YarnAllocator: Driver requested a total number of 17 executor(s). | |
| 17/06/14 04:12:43 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 17 executors. | |
| 17/06/14 04:12:45 INFO BlockManagerInfo: Added rdd_2_106 in memory on ip-10-0-49-229.ec2.internal:37760 (size: 206.0 MB, free: 760.1 MB) | |
| 17/06/14 04:12:47 INFO TaskSetManager: Starting task 127.0 in stage 0.0 (TID 127, ip-10-0-49-229.ec2.internal, executor 11, partition 127, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:47 INFO TaskSetManager: Finished task 106.0 in stage 0.0 (TID 121) in 25676 ms on ip-10-0-49-229.ec2.internal (executor 11) (68/135) | |
| 17/06/14 04:12:49 INFO BlockManagerInfo: Added rdd_2_47 in memory on ip-10-0-49-157.ec2.internal:45398 (size: 739.9 MB, free: 1271.2 MB) | |
| 17/06/14 04:12:51 INFO TaskSetManager: Starting task 123.0 in stage 0.0 (TID 128, ip-10-0-51-185.ec2.internal, executor 2, partition 123, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:51 INFO TaskSetManager: Finished task 41.0 in stage 0.0 (TID 64) in 116385 ms on ip-10-0-51-185.ec2.internal (executor 2) (69/135) | |
| 17/06/14 04:12:53 INFO BlockManagerInfo: Added rdd_2_99 in memory on ip-10-0-51-185.ec2.internal:34392 (size: 294.8 MB, free: 1707.2 MB) | |
| 17/06/14 04:12:54 INFO TaskSetManager: Starting task 128.0 in stage 0.0 (TID 129, ip-10-0-49-157.ec2.internal, executor 5, partition 128, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:54 INFO TaskSetManager: Finished task 45.0 in stage 0.0 (TID 66) in 110619 ms on ip-10-0-49-157.ec2.internal (executor 5) (70/135) | |
| 17/06/14 04:12:56 INFO TaskSetManager: Starting task 129.0 in stage 0.0 (TID 130, ip-10-0-49-157.ec2.internal, executor 5, partition 129, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:56 INFO TaskSetManager: Finished task 47.0 in stage 0.0 (TID 67) in 112588 ms on ip-10-0-49-157.ec2.internal (executor 5) (71/135) | |
| 17/06/14 04:12:56 INFO YarnAllocator: Driver requested a total number of 16 executor(s). | |
| 17/06/14 04:12:56 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 16 executors. | |
| 17/06/14 04:12:57 INFO TaskSetManager: Starting task 130.0 in stage 0.0 (TID 131, ip-10-0-51-185.ec2.internal, executor 3, partition 130, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:57 INFO TaskSetManager: Finished task 99.0 in stage 0.0 (TID 110) in 45415 ms on ip-10-0-51-185.ec2.internal (executor 3) (72/135) | |
| 17/06/14 04:12:58 INFO TaskSetManager: Starting task 134.0 in stage 0.0 (TID 132, ip-10-0-51-185.ec2.internal, executor 1, partition 134, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:12:58 INFO TaskSetManager: Finished task 48.0 in stage 0.0 (TID 68) in 113018 ms on ip-10-0-51-185.ec2.internal (executor 1) (73/135) | |
| 17/06/14 04:12:59 INFO BlockManagerInfo: Added rdd_2_88 in memory on ip-10-0-49-229.ec2.internal:46261 (size: 783.8 MB, free: 412.2 MB) | |
| 17/06/14 04:13:00 INFO TaskSetManager: Starting task 132.0 in stage 0.0 (TID 133, ip-10-0-49-157.ec2.internal, executor 6, partition 132, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:13:00 INFO TaskSetManager: Finished task 53.0 in stage 0.0 (TID 69) in 108659 ms on ip-10-0-49-157.ec2.internal (executor 6) (74/135) | |
| 17/06/14 04:13:00 INFO TaskSetManager: Starting task 133.0 in stage 0.0 (TID 134, ip-10-0-49-229.ec2.internal, executor 9, partition 133, NODE_LOCAL, 6083 bytes) | |
| 17/06/14 04:13:00 INFO TaskSetManager: Finished task 87.0 in stage 0.0 (TID 74) in 78656 ms on ip-10-0-49-229.ec2.internal (executor 9) (75/135) | |
| 17/06/14 04:13:00 INFO YarnAllocator: Driver requested a total number of 15 executor(s). | |
| 17/06/14 04:13:00 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 15 executors. | |
| 17/06/14 04:13:01 INFO TaskSetManager: Finished task 83.0 in stage 0.0 (TID 72) in 80259 ms on ip-10-0-49-229.ec2.internal (executor 9) (76/135) | |
| 17/06/14 04:13:01 INFO TaskSetManager: Finished task 101.0 in stage 0.0 (TID 78) in 78748 ms on ip-10-0-49-229.ec2.internal (executor 9) (77/135) | |
| 17/06/14 04:13:04 INFO TaskSetManager: Finished task 86.0 in stage 0.0 (TID 73) in 83110 ms on ip-10-0-49-229.ec2.internal (executor 11) (78/135) | |
| 17/06/14 04:13:05 INFO TaskSetManager: Finished task 90.0 in stage 0.0 (TID 76) in 82497 ms on ip-10-0-49-229.ec2.internal (executor 11) (79/135) | |
| 17/06/14 04:13:05 INFO YarnAllocator: Driver requested a total number of 14 executor(s). | |
| 17/06/14 04:13:05 INFO TaskSetManager: Finished task 88.0 in stage 0.0 (TID 75) in 83707 ms on ip-10-0-49-229.ec2.internal (executor 10) (80/135) | |
| 17/06/14 04:13:06 INFO TaskSetManager: Finished task 97.0 in stage 0.0 (TID 77) in 84045 ms on ip-10-0-49-229.ec2.internal (executor 10) (81/135) | |
| 17/06/14 04:13:07 INFO TaskSetManager: Finished task 103.0 in stage 0.0 (TID 81) in 80031 ms on ip-10-0-49-229.ec2.internal (executor 11) (82/135) | |
| 17/06/14 04:13:08 INFO TaskSetManager: Finished task 102.0 in stage 0.0 (TID 80) in 82886 ms on ip-10-0-49-229.ec2.internal (executor 10) (83/135) | |
| 17/06/14 04:13:08 INFO YarnAllocator: Driver requested a total number of 13 executor(s). | |
| 17/06/14 04:13:12 INFO TaskSetManager: Finished task 80.0 in stage 0.0 (TID 71) in 110037 ms on ip-10-0-50-96.ec2.internal (executor 12) (84/135) | |
| 17/06/14 04:13:13 INFO TaskSetManager: Finished task 104.0 in stage 0.0 (TID 83) in 75774 ms on ip-10-0-49-229.ec2.internal (executor 10) (85/135) | |
| 17/06/14 04:13:15 INFO TaskSetManager: Finished task 56.0 in stage 0.0 (TID 70) in 114832 ms on ip-10-0-51-185.ec2.internal (executor 1) (86/135) | |
| 17/06/14 04:13:21 INFO BlockManagerInfo: Added rdd_2_134 in memory on ip-10-0-51-185.ec2.internal:32987 (size: 139.8 MB, free: 1149.1 MB) | |
| 17/06/14 04:13:23 INFO TaskSetManager: Finished task 134.0 in stage 0.0 (TID 132) in 25265 ms on ip-10-0-51-185.ec2.internal (executor 1) (87/135) | |
| 17/06/14 04:13:23 INFO YarnAllocator: Driver requested a total number of 12 executor(s). | |
| 17/06/14 04:13:25 INFO TaskSetManager: Finished task 126.0 in stage 0.0 (TID 122) in 63474 ms on ip-10-0-49-229.ec2.internal (executor 9) (88/135) | |
| 17/06/14 04:13:25 INFO TaskSetManager: Finished task 84.0 in stage 0.0 (TID 92) in 76475 ms on ip-10-0-50-96.ec2.internal (executor 13) (89/135) | |
| 17/06/14 04:13:27 INFO BlockManagerInfo: Added rdd_2_92 in memory on ip-10-0-49-157.ec2.internal:43518 (size: 590.0 MB, free: 2.2 GB) | |
| 17/06/14 04:13:30 INFO BlockManagerInfo: Added rdd_2_64 in memory on ip-10-0-49-157.ec2.internal:43518 (size: 791.6 MB, free: 1445.4 MB) | |
| 17/06/14 04:13:34 INFO TaskSetManager: Finished task 92.0 in stage 0.0 (TID 101) in 84079 ms on ip-10-0-49-157.ec2.internal (executor 7) (90/135) | |
| 17/06/14 04:13:35 INFO TaskSetManager: Finished task 59.0 in stage 0.0 (TID 79) in 112976 ms on ip-10-0-51-185.ec2.internal (executor 8) (91/135) | |
| 17/06/14 04:13:35 INFO YarnAllocator: Driver requested a total number of 11 executor(s). | |
| 17/06/14 04:13:36 INFO TaskSetManager: Finished task 127.0 in stage 0.0 (TID 127) in 49260 ms on ip-10-0-49-229.ec2.internal (executor 11) (92/135) | |
| 17/06/14 04:13:38 INFO TaskSetManager: Finished task 64.0 in stage 0.0 (TID 82) in 111513 ms on ip-10-0-49-157.ec2.internal (executor 7) (93/135) | |
| 17/06/14 04:13:44 INFO TaskSetManager: Finished task 133.0 in stage 0.0 (TID 134) in 43861 ms on ip-10-0-49-229.ec2.internal (executor 9) (94/135) | |
| 17/06/14 04:13:45 INFO BlockManagerInfo: Added rdd_2_66 in memory on ip-10-0-51-185.ec2.internal:32987 (size: 789.2 MB, free: 359.8 MB) | |
| 17/06/14 04:13:46 INFO BlockManagerInfo: Added rdd_2_69 in memory on ip-10-0-49-157.ec2.internal:43518 (size: 792.2 MB, free: 653.3 MB) | |
| 17/06/14 04:13:46 INFO TaskSetManager: Finished task 82.0 in stage 0.0 (TID 91) in 97914 ms on ip-10-0-50-96.ec2.internal (executor 12) (95/135) | |
| 17/06/14 04:13:46 INFO YarnAllocator: Driver requested a total number of 10 executor(s). | |
| 17/06/14 04:13:48 INFO BlockManagerInfo: Added rdd_2_114 in memory on ip-10-0-49-157.ec2.internal:34052 (size: 767.1 MB, free: 1536.7 MB) | |
| 17/06/14 04:13:48 INFO TaskSetManager: Finished task 125.0 in stage 0.0 (TID 111) in 95834 ms on ip-10-0-50-96.ec2.internal (executor 12) (96/135) | |
| 17/06/14 04:13:49 INFO TaskSetManager: Finished task 67.0 in stage 0.0 (TID 85) in 102391 ms on ip-10-0-51-185.ec2.internal (executor 1) (97/135) | |
| 17/06/14 04:13:49 INFO TaskSetManager: Finished task 121.0 in stage 0.0 (TID 107) in 98497 ms on ip-10-0-50-96.ec2.internal (executor 12) (98/135) | |
| 17/06/14 04:13:49 INFO TaskSetManager: Finished task 81.0 in stage 0.0 (TID 87) in 101264 ms on ip-10-0-50-96.ec2.internal (executor 13) (99/135) | |
| 17/06/14 04:13:49 INFO YarnAllocator: Driver requested a total number of 9 executor(s). | |
| 17/06/14 04:13:49 INFO BlockManagerInfo: Added rdd_2_108 in memory on ip-10-0-50-96.ec2.internal:41521 (size: 803.8 MB, free: 431.7 MB) | |
| 17/06/14 04:13:50 INFO TaskSetManager: Finished task 66.0 in stage 0.0 (TID 84) in 104371 ms on ip-10-0-51-185.ec2.internal (executor 1) (100/135) | |
| 17/06/14 04:13:50 INFO BlockManagerInfo: Added rdd_2_124 in memory on ip-10-0-50-96.ec2.internal:38815 (size: 823.7 MB, free: 446.9 MB) | |
| 17/06/14 04:13:50 INFO BlockManagerInfo: Added rdd_2_94 in memory on ip-10-0-50-96.ec2.internal:32956 (size: 798.9 MB, free: 444.4 MB) | |
| 17/06/14 04:13:51 INFO TaskSetManager: Finished task 69.0 in stage 0.0 (TID 88) in 102734 ms on ip-10-0-49-157.ec2.internal (executor 7) (101/135) | |
| 17/06/14 04:13:51 INFO TaskSetManager: Finished task 118.0 in stage 0.0 (TID 105) in 100360 ms on ip-10-0-50-96.ec2.internal (executor 14) (102/135) | |
| 17/06/14 04:13:52 INFO TaskSetManager: Finished task 91.0 in stage 0.0 (TID 100) in 102085 ms on ip-10-0-49-157.ec2.internal (executor 7) (103/135) | |
| 17/06/14 04:13:52 INFO YarnAllocator: Driver requested a total number of 8 executor(s). | |
| 17/06/14 04:13:52 INFO TaskSetManager: Finished task 112.0 in stage 0.0 (TID 103) in 102695 ms on ip-10-0-50-96.ec2.internal (executor 13) (104/135) | |
| 17/06/14 04:13:53 INFO TaskSetManager: Finished task 105.0 in stage 0.0 (TID 112) in 100497 ms on ip-10-0-51-185.ec2.internal (executor 8) (105/135) | |
| 17/06/14 04:13:53 INFO TaskSetManager: Finished task 95.0 in stage 0.0 (TID 109) in 101282 ms on ip-10-0-51-185.ec2.internal (executor 8) (106/135) | |
| 17/06/14 04:13:53 INFO TaskSetManager: Finished task 93.0 in stage 0.0 (TID 104) in 102717 ms on ip-10-0-51-185.ec2.internal (executor 8) (107/135) | |
| 17/06/14 04:13:53 INFO YarnAllocator: Driver requested a total number of 7 executor(s). | |
| 17/06/14 04:13:53 INFO TaskSetManager: Finished task 108.0 in stage 0.0 (TID 102) in 103547 ms on ip-10-0-50-96.ec2.internal (executor 13) (108/135) | |
| 17/06/14 04:13:53 INFO BlockManagerInfo: Added rdd_2_89 in memory on ip-10-0-51-185.ec2.internal:34392 (size: 761.2 MB, free: 946.0 MB) | |
| 17/06/14 04:13:53 INFO TaskSetManager: Finished task 119.0 in stage 0.0 (TID 106) in 102785 ms on ip-10-0-50-96.ec2.internal (executor 14) (109/135) | |
| 17/06/14 04:13:53 INFO TaskSetManager: Finished task 114.0 in stage 0.0 (TID 119) in 97352 ms on ip-10-0-49-157.ec2.internal (executor 6) (110/135) | |
| 17/06/14 04:13:53 INFO TaskSetManager: Finished task 96.0 in stage 0.0 (TID 96) in 104570 ms on ip-10-0-50-96.ec2.internal (executor 15) (111/135) | |
| 17/06/14 04:13:53 INFO TaskSetManager: Finished task 100.0 in stage 0.0 (TID 99) in 104115 ms on ip-10-0-50-96.ec2.internal (executor 14) (112/135) | |
| 17/06/14 04:13:54 INFO YarnAllocator: Driver requested a total number of 6 executor(s). | |
| 17/06/14 04:13:54 INFO TaskSetManager: Finished task 113.0 in stage 0.0 (TID 118) in 99833 ms on ip-10-0-49-157.ec2.internal (executor 4) (113/135) | |
| 17/06/14 04:13:54 INFO TaskSetManager: Finished task 124.0 in stage 0.0 (TID 108) in 103266 ms on ip-10-0-50-96.ec2.internal (executor 14) (114/135) | |
| 17/06/14 04:13:54 INFO TaskSetManager: Finished task 98.0 in stage 0.0 (TID 97) in 105194 ms on ip-10-0-50-96.ec2.internal (executor 15) (115/135) | |
| 17/06/14 04:13:54 INFO YarnAllocator: Driver requested a total number of 5 executor(s). | |
| 17/06/14 04:13:55 INFO BlockManagerInfo: Added rdd_2_107 in memory on ip-10-0-49-157.ec2.internal:34337 (size: 840.3 MB, free: 965.6 MB) | |
| 17/06/14 04:13:55 INFO TaskSetManager: Finished task 109.0 in stage 0.0 (TID 114) in 102333 ms on ip-10-0-49-157.ec2.internal (executor 5) (116/135) | |
| 17/06/14 04:13:55 INFO TaskSetManager: Finished task 94.0 in stage 0.0 (TID 93) in 106141 ms on ip-10-0-50-96.ec2.internal (executor 15) (117/135) | |
| 17/06/14 04:13:55 INFO TaskSetManager: Finished task 131.0 in stage 0.0 (TID 115) in 102462 ms on ip-10-0-50-96.ec2.internal (executor 15) (118/135) | |
| 17/06/14 04:13:56 INFO TaskSetManager: Finished task 72.0 in stage 0.0 (TID 89) in 107874 ms on ip-10-0-51-185.ec2.internal (executor 3) (119/135) | |
| 17/06/14 04:13:56 INFO YarnAllocator: Driver requested a total number of 4 executor(s). | |
| 17/06/14 04:13:56 INFO BlockManagerInfo: Added rdd_2_116 in memory on ip-10-0-51-185.ec2.internal:36390 (size: 738.8 MB, free: 930.9 MB) | |
| 17/06/14 04:13:56 INFO TaskSetManager: Finished task 73.0 in stage 0.0 (TID 90) in 108326 ms on ip-10-0-51-185.ec2.internal (executor 3) (120/135) | |
| 17/06/14 04:13:57 INFO TaskSetManager: Finished task 110.0 in stage 0.0 (TID 116) in 103635 ms on ip-10-0-49-157.ec2.internal (executor 6) (121/135) | |
| 17/06/14 04:13:57 INFO BlockManagerInfo: Added rdd_2_129 in memory on ip-10-0-49-157.ec2.internal:45398 (size: 522.1 MB, free: 749.1 MB) | |
| 17/06/14 04:13:57 INFO TaskSetManager: Finished task 75.0 in stage 0.0 (TID 94) in 108513 ms on ip-10-0-51-185.ec2.internal (executor 2) (122/135) | |
| 17/06/14 04:13:58 INFO TaskSetManager: Finished task 89.0 in stage 0.0 (TID 98) in 108329 ms on ip-10-0-51-185.ec2.internal (executor 3) (123/135) | |
| 17/06/14 04:13:58 INFO YarnAllocator: Driver requested a total number of 3 executor(s). | |
| 17/06/14 04:13:58 INFO TaskSetManager: Finished task 85.0 in stage 0.0 (TID 95) in 108856 ms on ip-10-0-51-185.ec2.internal (executor 2) (124/135) | |
| 17/06/14 04:13:58 INFO TaskSetManager: Finished task 111.0 in stage 0.0 (TID 117) in 105123 ms on ip-10-0-49-157.ec2.internal (executor 4) (125/135) | |
| 17/06/14 04:14:00 INFO TaskSetManager: Finished task 129.0 in stage 0.0 (TID 130) in 63129 ms on ip-10-0-49-157.ec2.internal (executor 5) (126/135) | |
| 17/06/14 04:14:00 INFO TaskSetManager: Finished task 116.0 in stage 0.0 (TID 123) in 96226 ms on ip-10-0-51-185.ec2.internal (executor 2) (127/135) | |
| 17/06/14 04:14:00 INFO YarnAllocator: Driver requested a total number of 2 executor(s). | |
| 17/06/14 04:14:00 INFO TaskSetManager: Finished task 107.0 in stage 0.0 (TID 113) in 107289 ms on ip-10-0-49-157.ec2.internal (executor 4) (128/135) | |
| 17/06/14 04:14:00 INFO TaskSetManager: Finished task 117.0 in stage 0.0 (TID 124) in 90621 ms on ip-10-0-49-157.ec2.internal (executor 6) (129/135) | |
| 17/06/14 04:14:02 INFO BlockManagerInfo: Added rdd_2_123 in memory on ip-10-0-51-185.ec2.internal:36390 (size: 658.9 MB, free: 271.9 MB) | |
| 17/06/14 04:14:05 INFO TaskSetManager: Finished task 123.0 in stage 0.0 (TID 128) in 73568 ms on ip-10-0-51-185.ec2.internal (executor 2) (130/135) | |
| 17/06/14 04:14:06 INFO BlockManagerInfo: Added rdd_2_120 in memory on ip-10-0-49-157.ec2.internal:34337 (size: 815.9 MB, free: 149.8 MB) | |
| 17/06/14 04:14:08 INFO TaskSetManager: Finished task 122.0 in stage 0.0 (TID 126) in 85176 ms on ip-10-0-49-157.ec2.internal (executor 5) (131/135) | |
| 17/06/14 04:14:08 INFO YarnAllocator: Driver requested a total number of 1 executor(s). | |
| 17/06/14 04:14:09 INFO TaskSetManager: Finished task 120.0 in stage 0.0 (TID 125) in 88921 ms on ip-10-0-49-157.ec2.internal (executor 4) (132/135) | |
| 17/06/14 04:14:11 INFO TaskSetManager: Finished task 128.0 in stage 0.0 (TID 129) in 77388 ms on ip-10-0-49-157.ec2.internal (executor 5) (133/135) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Finished task 132.0 in stage 0.0 (TID 133) in 73126 ms on ip-10-0-49-157.ec2.internal (executor 6) (134/135) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Finished task 130.0 in stage 0.0 (TID 131) in 75875 ms on ip-10-0-51-185.ec2.internal (executor 3) (135/135) | |
| 17/06/14 04:14:13 INFO DAGScheduler: ShuffleMapStage 0 (groupBy at ShredJob.scala:356) finished in 249.884 s | |
| 17/06/14 04:14:13 INFO YarnClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool | |
| 17/06/14 04:14:13 INFO DAGScheduler: looking for newly runnable stages | |
| 17/06/14 04:14:13 INFO DAGScheduler: running: Set() | |
| 17/06/14 04:14:13 INFO DAGScheduler: waiting: Set(ResultStage 1) | |
| 17/06/14 04:14:13 INFO DAGScheduler: failed: Set() | |
| 17/06/14 04:14:13 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[26] at saveAsTextFile at ShredJob.scala:400), which has no missing parents | |
| 17/06/14 04:14:13 INFO YarnAllocator: Driver requested a total number of 0 executor(s). | |
| 17/06/14 04:14:13 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 86.3 KB, free 413.5 MB) | |
| 17/06/14 04:14:13 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 33.3 KB, free 413.5 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 10.0.49.229:33721 (size: 33.3 KB, free: 413.9 MB) | |
| 17/06/14 04:14:13 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:997 | |
| 17/06/14 04:14:13 INFO DAGScheduler: Submitting 270 missing tasks from ResultStage 1 (MapPartitionsRDD[26] at saveAsTextFile at ShredJob.scala:400) | |
| 17/06/14 04:14:13 INFO YarnClusterScheduler: Adding task set 1.0 with 270 tasks | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 50.0 in stage 1.0 (TID 135, ip-10-0-50-96.ec2.internal, executor 13, partition 50, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 8.0 in stage 1.0 (TID 136, ip-10-0-49-157.ec2.internal, executor 6, partition 8, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 29.0 in stage 1.0 (TID 137, ip-10-0-49-157.ec2.internal, executor 4, partition 29, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 24.0 in stage 1.0 (TID 138, ip-10-0-51-185.ec2.internal, executor 8, partition 24, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 51.0 in stage 1.0 (TID 139, ip-10-0-49-229.ec2.internal, executor 9, partition 51, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 58.0 in stage 1.0 (TID 140, ip-10-0-50-96.ec2.internal, executor 12, partition 58, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 15.0 in stage 1.0 (TID 141, ip-10-0-51-185.ec2.internal, executor 3, partition 15, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 21.0 in stage 1.0 (TID 142, ip-10-0-49-229.ec2.internal, executor 10, partition 21, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 64.0 in stage 1.0 (TID 143, ip-10-0-49-157.ec2.internal, executor 7, partition 64, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 144, ip-10-0-51-185.ec2.internal, executor 1, partition 1, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 36.0 in stage 1.0 (TID 145, ip-10-0-50-96.ec2.internal, executor 14, partition 36, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 18.0 in stage 1.0 (TID 146, ip-10-0-51-185.ec2.internal, executor 2, partition 18, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 147, ip-10-0-49-157.ec2.internal, executor 5, partition 4, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 63.0 in stage 1.0 (TID 148, ip-10-0-50-96.ec2.internal, executor 15, partition 63, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 38.0 in stage 1.0 (TID 149, ip-10-0-49-229.ec2.internal, executor 11, partition 38, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 57.0 in stage 1.0 (TID 150, ip-10-0-50-96.ec2.internal, executor 13, partition 57, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 10.0 in stage 1.0 (TID 151, ip-10-0-49-157.ec2.internal, executor 6, partition 10, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 107.0 in stage 1.0 (TID 152, ip-10-0-49-157.ec2.internal, executor 4, partition 107, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 33.0 in stage 1.0 (TID 153, ip-10-0-51-185.ec2.internal, executor 8, partition 33, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 54.0 in stage 1.0 (TID 154, ip-10-0-49-229.ec2.internal, executor 9, partition 54, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 60.0 in stage 1.0 (TID 155, ip-10-0-50-96.ec2.internal, executor 12, partition 60, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 89.0 in stage 1.0 (TID 156, ip-10-0-51-185.ec2.internal, executor 3, partition 89, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 32.0 in stage 1.0 (TID 157, ip-10-0-49-229.ec2.internal, executor 10, partition 32, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 69.0 in stage 1.0 (TID 158, ip-10-0-49-157.ec2.internal, executor 7, partition 69, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 159, ip-10-0-51-185.ec2.internal, executor 1, partition 2, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 37.0 in stage 1.0 (TID 160, ip-10-0-50-96.ec2.internal, executor 14, partition 37, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 41.0 in stage 1.0 (TID 161, ip-10-0-51-185.ec2.internal, executor 2, partition 41, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 6.0 in stage 1.0 (TID 162, ip-10-0-49-157.ec2.internal, executor 5, partition 6, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 74.0 in stage 1.0 (TID 163, ip-10-0-50-96.ec2.internal, executor 15, partition 74, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 42.0 in stage 1.0 (TID 164, ip-10-0-49-229.ec2.internal, executor 11, partition 42, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 108.0 in stage 1.0 (TID 165, ip-10-0-50-96.ec2.internal, executor 13, partition 108, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 114.0 in stage 1.0 (TID 166, ip-10-0-49-157.ec2.internal, executor 6, partition 114, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 115.0 in stage 1.0 (TID 167, ip-10-0-49-157.ec2.internal, executor 4, partition 115, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 34.0 in stage 1.0 (TID 168, ip-10-0-51-185.ec2.internal, executor 8, partition 34, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 70.0 in stage 1.0 (TID 169, ip-10-0-49-229.ec2.internal, executor 9, partition 70, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 62.0 in stage 1.0 (TID 170, ip-10-0-50-96.ec2.internal, executor 12, partition 62, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 99.0 in stage 1.0 (TID 171, ip-10-0-51-185.ec2.internal, executor 3, partition 99, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 76.0 in stage 1.0 (TID 172, ip-10-0-49-229.ec2.internal, executor 10, partition 76, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 92.0 in stage 1.0 (TID 173, ip-10-0-49-157.ec2.internal, executor 7, partition 92, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 174, ip-10-0-51-185.ec2.internal, executor 1, partition 5, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 124.0 in stage 1.0 (TID 175, ip-10-0-50-96.ec2.internal, executor 14, partition 124, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 68.0 in stage 1.0 (TID 176, ip-10-0-51-185.ec2.internal, executor 2, partition 68, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 7.0 in stage 1.0 (TID 177, ip-10-0-49-157.ec2.internal, executor 5, partition 7, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 94.0 in stage 1.0 (TID 178, ip-10-0-50-96.ec2.internal, executor 15, partition 94, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 43.0 in stage 1.0 (TID 179, ip-10-0-49-229.ec2.internal, executor 11, partition 43, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 120.0 in stage 1.0 (TID 180, ip-10-0-49-157.ec2.internal, executor 4, partition 120, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 88.0 in stage 1.0 (TID 181, ip-10-0-49-229.ec2.internal, executor 10, partition 88, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 66.0 in stage 1.0 (TID 182, ip-10-0-51-185.ec2.internal, executor 1, partition 66, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 116.0 in stage 1.0 (TID 183, ip-10-0-51-185.ec2.internal, executor 2, partition 116, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 47.0 in stage 1.0 (TID 184, ip-10-0-49-157.ec2.internal, executor 5, partition 47, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 106.0 in stage 1.0 (TID 185, ip-10-0-49-229.ec2.internal, executor 11, partition 106, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 186, ip-10-0-50-96.ec2.internal, executor 13, partition 0, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 187, ip-10-0-49-157.ec2.internal, executor 6, partition 3, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 11.0 in stage 1.0 (TID 188, ip-10-0-51-185.ec2.internal, executor 8, partition 11, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 9.0 in stage 1.0 (TID 189, ip-10-0-49-229.ec2.internal, executor 9, partition 9, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 13.0 in stage 1.0 (TID 190, ip-10-0-50-96.ec2.internal, executor 12, partition 13, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 14.0 in stage 1.0 (TID 191, ip-10-0-51-185.ec2.internal, executor 3, partition 14, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 12.0 in stage 1.0 (TID 192, ip-10-0-49-157.ec2.internal, executor 7, partition 12, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 16.0 in stage 1.0 (TID 193, ip-10-0-50-96.ec2.internal, executor 14, partition 16, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO TaskSetManager: Starting task 20.0 in stage 1.0 (TID 194, ip-10-0-50-96.ec2.internal, executor 15, partition 20, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-229.ec2.internal:46261 (size: 33.3 KB, free: 412.2 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-157.ec2.internal:34052 (size: 33.3 KB, free: 1536.6 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-51-185.ec2.internal:32987 (size: 33.3 KB, free: 359.8 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-50-96.ec2.internal:46753 (size: 33.3 KB, free: 751.8 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-229.ec2.internal:44338 (size: 33.3 KB, free: 834.5 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-51-185.ec2.internal:36390 (size: 33.3 KB, free: 271.9 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-229.ec2.internal:37760 (size: 33.3 KB, free: 760.0 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-50-96.ec2.internal:41521 (size: 33.3 KB, free: 431.6 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-157.ec2.internal:34337 (size: 33.3 KB, free: 149.7 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-157.ec2.internal:45398 (size: 33.3 KB, free: 749.1 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on 10.0.49.229:33721 in memory (size: 5.0 KB, free: 413.9 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-50-96.ec2.internal:46753 in memory (size: 5.0 KB, free: 751.8 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-51-185.ec2.internal:43229 in memory (size: 5.0 KB, free: 640.4 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-50-96.ec2.internal:38815 in memory (size: 5.0 KB, free: 446.9 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-50-96.ec2.internal:38815 (size: 33.3 KB, free: 446.9 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-51-185.ec2.internal:34392 in memory (size: 5.0 KB, free: 946.0 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-157.ec2.internal:43518 (size: 33.3 KB, free: 653.2 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-51-185.ec2.internal:32987 in memory (size: 5.0 KB, free: 359.8 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-50-96.ec2.internal:32956 (size: 33.3 KB, free: 444.4 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-51-185.ec2.internal:43229 (size: 33.3 KB, free: 640.3 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-49-229.ec2.internal:46261 in memory (size: 5.0 KB, free: 412.2 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-51-185.ec2.internal:34392 (size: 33.3 KB, free: 945.9 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-50-96.ec2.internal:41521 in memory (size: 5.0 KB, free: 431.6 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-50-96.ec2.internal:32956 in memory (size: 5.0 KB, free: 444.4 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-49-157.ec2.internal:45398 in memory (size: 5.0 KB, free: 749.1 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-49-229.ec2.internal:37760 in memory (size: 5.0 KB, free: 760.0 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-49-157.ec2.internal:34052 in memory (size: 5.0 KB, free: 1536.6 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-51-185.ec2.internal:36390 in memory (size: 5.0 KB, free: 271.9 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-49-229.ec2.internal:44338 in memory (size: 5.0 KB, free: 834.5 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-49-157.ec2.internal:43518 in memory (size: 5.0 KB, free: 653.2 MB) | |
| 17/06/14 04:14:13 INFO BlockManagerInfo: Removed broadcast_1_piece0 on ip-10-0-49-157.ec2.internal:34337 in memory (size: 5.0 KB, free: 149.7 MB) | |
| 17/06/14 04:14:14 INFO YarnAllocator: Driver requested a total number of 16 executor(s). | |
| 17/06/14 04:14:14 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:14:14 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:14 INFO ExecutorAllocationManager: Requesting 16 new executors because tasks are backlogged (new desired total will be 16) | |
| 17/06/14 04:14:15 INFO YarnAllocator: Driver requested a total number of 17 executor(s). | |
| 17/06/14 04:14:15 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:14:15 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 17) | |
| 17/06/14 04:14:15 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 17.0 in stage 1.0 (TID 195, ip-10-0-49-229.ec2.internal, executor 10, partition 17, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 21.0 in stage 1.0 (TID 142) in 2096 ms on ip-10-0-49-229.ec2.internal (executor 10) (1/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 22.0 in stage 1.0 (TID 196, ip-10-0-49-229.ec2.internal, executor 10, partition 22, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 32.0 in stage 1.0 (TID 157) in 2093 ms on ip-10-0-49-229.ec2.internal (executor 10) (2/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 25.0 in stage 1.0 (TID 197, ip-10-0-49-229.ec2.internal, executor 10, partition 25, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 76.0 in stage 1.0 (TID 172) in 2090 ms on ip-10-0-49-229.ec2.internal (executor 10) (3/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 35.0 in stage 1.0 (TID 198, ip-10-0-49-229.ec2.internal, executor 10, partition 35, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 88.0 in stage 1.0 (TID 181) in 2090 ms on ip-10-0-49-229.ec2.internal (executor 10) (4/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 46.0 in stage 1.0 (TID 199, ip-10-0-49-229.ec2.internal, executor 11, partition 46, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 42.0 in stage 1.0 (TID 164) in 2115 ms on ip-10-0-49-229.ec2.internal (executor 11) (5/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 65.0 in stage 1.0 (TID 200, ip-10-0-49-229.ec2.internal, executor 11, partition 65, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 38.0 in stage 1.0 (TID 149) in 2120 ms on ip-10-0-49-229.ec2.internal (executor 11) (6/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 78.0 in stage 1.0 (TID 201, ip-10-0-49-229.ec2.internal, executor 11, partition 78, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 79.0 in stage 1.0 (TID 202, ip-10-0-49-229.ec2.internal, executor 11, partition 79, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 106.0 in stage 1.0 (TID 185) in 2115 ms on ip-10-0-49-229.ec2.internal (executor 11) (7/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 43.0 in stage 1.0 (TID 179) in 2116 ms on ip-10-0-49-229.ec2.internal (executor 11) (8/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 83.0 in stage 1.0 (TID 203, ip-10-0-49-229.ec2.internal, executor 9, partition 83, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 86.0 in stage 1.0 (TID 204, ip-10-0-49-229.ec2.internal, executor 9, partition 86, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 87.0 in stage 1.0 (TID 205, ip-10-0-49-229.ec2.internal, executor 9, partition 87, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 70.0 in stage 1.0 (TID 169) in 2311 ms on ip-10-0-49-229.ec2.internal (executor 9) (9/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 51.0 in stage 1.0 (TID 139) in 2319 ms on ip-10-0-49-229.ec2.internal (executor 9) (10/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 54.0 in stage 1.0 (TID 154) in 2316 ms on ip-10-0-49-229.ec2.internal (executor 9) (11/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 19.0 in stage 1.0 (TID 206, ip-10-0-49-157.ec2.internal, executor 6, partition 19, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 10.0 in stage 1.0 (TID 151) in 2395 ms on ip-10-0-49-157.ec2.internal (executor 6) (12/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 23.0 in stage 1.0 (TID 207, ip-10-0-49-157.ec2.internal, executor 6, partition 23, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 114.0 in stage 1.0 (TID 166) in 2400 ms on ip-10-0-49-157.ec2.internal (executor 6) (13/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 26.0 in stage 1.0 (TID 208, ip-10-0-49-157.ec2.internal, executor 6, partition 26, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID 136) in 2417 ms on ip-10-0-49-157.ec2.internal (executor 6) (14/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 129.0 in stage 1.0 (TID 209, ip-10-0-49-157.ec2.internal, executor 5, partition 129, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 6.0 in stage 1.0 (TID 162) in 2441 ms on ip-10-0-49-157.ec2.internal (executor 5) (15/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 134.0 in stage 1.0 (TID 210, ip-10-0-51-185.ec2.internal, executor 1, partition 134, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 144) in 2466 ms on ip-10-0-51-185.ec2.internal (executor 1) (16/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 30.0 in stage 1.0 (TID 211, ip-10-0-51-185.ec2.internal, executor 1, partition 30, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 159) in 2467 ms on ip-10-0-51-185.ec2.internal (executor 1) (17/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 27.0 in stage 1.0 (TID 212, ip-10-0-49-157.ec2.internal, executor 5, partition 27, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 31.0 in stage 1.0 (TID 213, ip-10-0-51-185.ec2.internal, executor 1, partition 31, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 28.0 in stage 1.0 (TID 214, ip-10-0-49-157.ec2.internal, executor 5, partition 28, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 66.0 in stage 1.0 (TID 182) in 2473 ms on ip-10-0-51-185.ec2.internal (executor 1) (18/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 7.0 in stage 1.0 (TID 177) in 2474 ms on ip-10-0-49-157.ec2.internal (executor 5) (19/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 47.0 in stage 1.0 (TID 184) in 2472 ms on ip-10-0-49-157.ec2.internal (executor 5) (20/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 39.0 in stage 1.0 (TID 215, ip-10-0-51-185.ec2.internal, executor 1, partition 39, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 174) in 2489 ms on ip-10-0-51-185.ec2.internal (executor 1) (21/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 40.0 in stage 1.0 (TID 216, ip-10-0-49-157.ec2.internal, executor 5, partition 40, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Starting task 45.0 in stage 1.0 (TID 217, ip-10-0-49-157.ec2.internal, executor 5, partition 45, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 129.0 in stage 1.0 (TID 209) in 112 ms on ip-10-0-49-157.ec2.internal (executor 5) (22/270) | |
| 17/06/14 04:14:15 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 147) in 2556 ms on ip-10-0-49-157.ec2.internal (executor 5) (23/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 44.0 in stage 1.0 (TID 218, ip-10-0-51-185.ec2.internal, executor 1, partition 44, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 134.0 in stage 1.0 (TID 210) in 148 ms on ip-10-0-51-185.ec2.internal (executor 1) (24/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 48.0 in stage 1.0 (TID 219, ip-10-0-49-157.ec2.internal, executor 4, partition 48, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 53.0 in stage 1.0 (TID 220, ip-10-0-49-157.ec2.internal, executor 4, partition 53, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 56.0 in stage 1.0 (TID 221, ip-10-0-49-157.ec2.internal, executor 4, partition 56, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 59.0 in stage 1.0 (TID 222, ip-10-0-49-157.ec2.internal, executor 4, partition 59, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 29.0 in stage 1.0 (TID 137) in 2703 ms on ip-10-0-49-157.ec2.internal (executor 4) (25/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 115.0 in stage 1.0 (TID 167) in 2696 ms on ip-10-0-49-157.ec2.internal (executor 4) (26/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 107.0 in stage 1.0 (TID 152) in 2699 ms on ip-10-0-49-157.ec2.internal (executor 4) (27/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 120.0 in stage 1.0 (TID 180) in 2693 ms on ip-10-0-49-157.ec2.internal (executor 4) (28/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 123.0 in stage 1.0 (TID 223, ip-10-0-51-185.ec2.internal, executor 2, partition 123, PROCESS_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 49.0 in stage 1.0 (TID 224, ip-10-0-51-185.ec2.internal, executor 2, partition 49, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 41.0 in stage 1.0 (TID 161) in 2736 ms on ip-10-0-51-185.ec2.internal (executor 2) (29/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 18.0 in stage 1.0 (TID 146) in 2740 ms on ip-10-0-51-185.ec2.internal (executor 2) (30/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 52.0 in stage 1.0 (TID 225, ip-10-0-51-185.ec2.internal, executor 2, partition 52, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 55.0 in stage 1.0 (TID 226, ip-10-0-51-185.ec2.internal, executor 2, partition 55, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 68.0 in stage 1.0 (TID 176) in 2748 ms on ip-10-0-51-185.ec2.internal (executor 2) (31/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 116.0 in stage 1.0 (TID 183) in 2746 ms on ip-10-0-51-185.ec2.internal (executor 2) (32/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 67.0 in stage 1.0 (TID 227, ip-10-0-49-157.ec2.internal, executor 7, partition 67, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 92.0 in stage 1.0 (TID 173) in 2809 ms on ip-10-0-49-157.ec2.internal (executor 7) (33/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 61.0 in stage 1.0 (TID 228, ip-10-0-51-185.ec2.internal, executor 8, partition 61, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 34.0 in stage 1.0 (TID 168) in 2813 ms on ip-10-0-51-185.ec2.internal (executor 8) (34/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 71.0 in stage 1.0 (TID 229, ip-10-0-51-185.ec2.internal, executor 8, partition 71, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 72.0 in stage 1.0 (TID 230, ip-10-0-51-185.ec2.internal, executor 8, partition 72, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 24.0 in stage 1.0 (TID 138) in 2821 ms on ip-10-0-51-185.ec2.internal (executor 8) (35/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 33.0 in stage 1.0 (TID 153) in 2817 ms on ip-10-0-51-185.ec2.internal (executor 8) (36/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 73.0 in stage 1.0 (TID 231, ip-10-0-49-157.ec2.internal, executor 7, partition 73, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 64.0 in stage 1.0 (TID 143) in 2851 ms on ip-10-0-49-157.ec2.internal (executor 7) (37/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 75.0 in stage 1.0 (TID 232, ip-10-0-51-185.ec2.internal, executor 2, partition 75, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 123.0 in stage 1.0 (TID 223) in 122 ms on ip-10-0-51-185.ec2.internal (executor 2) (38/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 77.0 in stage 1.0 (TID 233, ip-10-0-51-185.ec2.internal, executor 3, partition 77, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 15.0 in stage 1.0 (TID 141) in 2923 ms on ip-10-0-51-185.ec2.internal (executor 3) (39/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 80.0 in stage 1.0 (TID 234, ip-10-0-51-185.ec2.internal, executor 3, partition 80, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 89.0 in stage 1.0 (TID 156) in 2924 ms on ip-10-0-51-185.ec2.internal (executor 3) (40/270) | |
| 17/06/14 04:14:16 INFO YarnAllocator: Driver requested a total number of 19 executor(s). | |
| 17/06/14 04:14:16 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:14:16 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:16 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:16 INFO ExecutorAllocationManager: Requesting 2 new executors because tasks are backlogged (new desired total will be 19) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 81.0 in stage 1.0 (TID 235, ip-10-0-51-185.ec2.internal, executor 3, partition 81, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 99.0 in stage 1.0 (TID 171) in 3013 ms on ip-10-0-51-185.ec2.internal (executor 3) (41/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 82.0 in stage 1.0 (TID 236, ip-10-0-50-96.ec2.internal, executor 12, partition 82, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 62.0 in stage 1.0 (TID 170) in 3378 ms on ip-10-0-50-96.ec2.internal (executor 12) (42/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 84.0 in stage 1.0 (TID 237, ip-10-0-50-96.ec2.internal, executor 12, partition 84, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 60.0 in stage 1.0 (TID 155) in 3383 ms on ip-10-0-50-96.ec2.internal (executor 12) (43/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 96.0 in stage 1.0 (TID 238, ip-10-0-50-96.ec2.internal, executor 12, partition 96, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 58.0 in stage 1.0 (TID 140) in 3402 ms on ip-10-0-50-96.ec2.internal (executor 12) (44/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 98.0 in stage 1.0 (TID 239, ip-10-0-50-96.ec2.internal, executor 13, partition 98, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 108.0 in stage 1.0 (TID 165) in 3481 ms on ip-10-0-50-96.ec2.internal (executor 13) (45/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 100.0 in stage 1.0 (TID 240, ip-10-0-50-96.ec2.internal, executor 13, partition 100, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 57.0 in stage 1.0 (TID 150) in 3489 ms on ip-10-0-50-96.ec2.internal (executor 13) (46/270) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Starting task 112.0 in stage 1.0 (TID 241, ip-10-0-50-96.ec2.internal, executor 13, partition 112, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:16 INFO TaskSetManager: Finished task 50.0 in stage 1.0 (TID 135) in 3503 ms on ip-10-0-50-96.ec2.internal (executor 13) (47/270) | |
| 17/06/14 04:14:17 INFO TaskSetManager: Starting task 118.0 in stage 1.0 (TID 242, ip-10-0-50-96.ec2.internal, executor 15, partition 118, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:17 INFO TaskSetManager: Finished task 63.0 in stage 1.0 (TID 148) in 3689 ms on ip-10-0-50-96.ec2.internal (executor 15) (48/270) | |
| 17/06/14 04:14:17 INFO TaskSetManager: Starting task 119.0 in stage 1.0 (TID 243, ip-10-0-50-96.ec2.internal, executor 15, partition 119, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:17 INFO TaskSetManager: Finished task 94.0 in stage 1.0 (TID 178) in 3690 ms on ip-10-0-50-96.ec2.internal (executor 15) (49/270) | |
| 17/06/14 04:14:17 INFO TaskSetManager: Starting task 121.0 in stage 1.0 (TID 244, ip-10-0-50-96.ec2.internal, executor 15, partition 121, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:17 INFO TaskSetManager: Finished task 74.0 in stage 1.0 (TID 163) in 3699 ms on ip-10-0-50-96.ec2.internal (executor 15) (50/270) | |
| 17/06/14 04:14:17 INFO TaskSetManager: Starting task 85.0 in stage 1.0 (TID 245, ip-10-0-49-157.ec2.internal, executor 7, partition 85, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:17 INFO TaskSetManager: Finished task 69.0 in stage 1.0 (TID 158) in 3813 ms on ip-10-0-49-157.ec2.internal (executor 7) (51/270) | |
| 17/06/14 04:14:17 INFO TaskSetManager: Starting task 125.0 in stage 1.0 (TID 246, ip-10-0-50-96.ec2.internal, executor 14, partition 125, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:17 INFO TaskSetManager: Starting task 131.0 in stage 1.0 (TID 247, ip-10-0-50-96.ec2.internal, executor 14, partition 131, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:17 INFO TaskSetManager: Starting task 132.0 in stage 1.0 (TID 248, ip-10-0-50-96.ec2.internal, executor 14, partition 132, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:14:17 INFO TaskSetManager: Finished task 37.0 in stage 1.0 (TID 160) in 3865 ms on ip-10-0-50-96.ec2.internal (executor 14) (52/270) | |
| 17/06/14 04:14:17 INFO TaskSetManager: Finished task 36.0 in stage 1.0 (TID 145) in 3870 ms on ip-10-0-50-96.ec2.internal (executor 14) (53/270) | |
| 17/06/14 04:14:17 INFO TaskSetManager: Finished task 124.0 in stage 1.0 (TID 175) in 3862 ms on ip-10-0-50-96.ec2.internal (executor 14) (54/270) | |
| 17/06/14 04:14:17 INFO YarnAllocator: Driver requested a total number of 23 executor(s). | |
| 17/06/14 04:14:17 INFO ExecutorAllocationManager: Requesting 4 new executors because tasks are backlogged (new desired total will be 23) | |
| 17/06/14 04:14:17 INFO YarnAllocator: Will request 4 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:14:17 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:17 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:17 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:17 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:14:18 INFO YarnAllocator: Driver requested a total number of 31 executor(s). | |
| 17/06/14 04:14:18 INFO YarnAllocator: Will request 8 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:14:18 INFO ExecutorAllocationManager: Requesting 8 new executors because tasks are backlogged (new desired total will be 31) | |
| 17/06/14 04:14:18 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:18 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:18 INFO YarnAllocator: Submitted container request for host ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:18 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:18 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:18 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:18 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:14:18 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:14:19 INFO YarnAllocator: Driver requested a total number of 47 executor(s). | |
| 17/06/14 04:14:19 INFO YarnAllocator: Will request 16 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:14:19 INFO YarnAllocator: Submitted 10 unlocalized container requests. | |
| 17/06/14 04:14:19 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:19 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:14:19 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:14:19 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:14:19 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:14:19 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:14:19 INFO ExecutorAllocationManager: Requesting 16 new executors because tasks are backlogged (new desired total will be 47) | |
| 17/06/14 04:14:20 INFO YarnAllocator: Driver requested a total number of 54 executor(s). | |
| 17/06/14 04:14:20 INFO YarnAllocator: Will request 7 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:14:20 INFO YarnAllocator: Submitted 7 unlocalized container requests. | |
| 17/06/14 04:14:20 INFO ExecutorAllocationManager: Requesting 7 new executors because tasks are backlogged (new desired total will be 54) | |
| 17/06/14 04:15:28 INFO TaskSetManager: Starting task 90.0 in stage 1.0 (TID 249, ip-10-0-49-229.ec2.internal, executor 10, partition 90, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:28 INFO TaskSetManager: Finished task 17.0 in stage 1.0 (TID 195) in 73087 ms on ip-10-0-49-229.ec2.internal (executor 10) (55/270) | |
| 17/06/14 04:15:28 INFO TaskSetManager: Starting task 97.0 in stage 1.0 (TID 250, ip-10-0-49-229.ec2.internal, executor 10, partition 97, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:28 INFO TaskSetManager: Finished task 35.0 in stage 1.0 (TID 198) in 73166 ms on ip-10-0-49-229.ec2.internal (executor 10) (56/270) | |
| 17/06/14 04:15:29 INFO TaskSetManager: Starting task 101.0 in stage 1.0 (TID 251, ip-10-0-49-229.ec2.internal, executor 10, partition 101, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:29 INFO TaskSetManager: Finished task 22.0 in stage 1.0 (TID 196) in 73997 ms on ip-10-0-49-229.ec2.internal (executor 10) (57/270) | |
| 17/06/14 04:15:29 INFO TaskSetManager: Starting task 135.0 in stage 1.0 (TID 252, ip-10-0-50-96.ec2.internal, executor 12, partition 135, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:29 INFO TaskSetManager: Finished task 84.0 in stage 1.0 (TID 237) in 72829 ms on ip-10-0-50-96.ec2.internal (executor 12) (58/270) | |
| 17/06/14 04:15:29 INFO YarnAllocator: Driver requested a total number of 53 executor(s). | |
| 17/06/14 04:15:29 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 53 executors. | |
| 17/06/14 04:15:29 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:35518 | |
| 17/06/14 04:15:29 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 6257 bytes | |
| 17/06/14 04:15:30 INFO TaskSetManager: Starting task 102.0 in stage 1.0 (TID 253, ip-10-0-49-229.ec2.internal, executor 10, partition 102, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:30 INFO TaskSetManager: Finished task 25.0 in stage 1.0 (TID 197) in 75272 ms on ip-10-0-49-229.ec2.internal (executor 10) (59/270) | |
| 17/06/14 04:15:31 INFO TaskSetManager: Starting task 103.0 in stage 1.0 (TID 254, ip-10-0-49-229.ec2.internal, executor 11, partition 103, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:31 INFO TaskSetManager: Finished task 65.0 in stage 1.0 (TID 200) in 76114 ms on ip-10-0-49-229.ec2.internal (executor 11) (60/270) | |
| 17/06/14 04:15:32 INFO TaskSetManager: Starting task 104.0 in stage 1.0 (TID 255, ip-10-0-49-229.ec2.internal, executor 11, partition 104, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:32 INFO TaskSetManager: Finished task 78.0 in stage 1.0 (TID 201) in 77101 ms on ip-10-0-49-229.ec2.internal (executor 11) (61/270) | |
| 17/06/14 04:15:32 INFO TaskSetManager: Starting task 105.0 in stage 1.0 (TID 256, ip-10-0-49-229.ec2.internal, executor 11, partition 105, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:32 INFO TaskSetManager: Finished task 46.0 in stage 1.0 (TID 199) in 77208 ms on ip-10-0-49-229.ec2.internal (executor 11) (62/270) | |
| 17/06/14 04:15:32 INFO YarnAllocator: Driver requested a total number of 52 executor(s). | |
| 17/06/14 04:15:32 INFO TaskSetManager: Starting task 110.0 in stage 1.0 (TID 257, ip-10-0-49-229.ec2.internal, executor 9, partition 110, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:32 INFO TaskSetManager: Finished task 83.0 in stage 1.0 (TID 203) in 77161 ms on ip-10-0-49-229.ec2.internal (executor 9) (63/270) | |
| 17/06/14 04:15:32 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 52 executors. | |
| 17/06/14 04:15:34 INFO TaskSetManager: Starting task 126.0 in stage 1.0 (TID 258, ip-10-0-49-229.ec2.internal, executor 11, partition 126, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:34 INFO TaskSetManager: Finished task 79.0 in stage 1.0 (TID 202) in 78879 ms on ip-10-0-49-229.ec2.internal (executor 11) (64/270) | |
| 17/06/14 04:15:34 INFO TaskSetManager: Starting task 127.0 in stage 1.0 (TID 259, ip-10-0-49-229.ec2.internal, executor 9, partition 127, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:34 INFO TaskSetManager: Finished task 87.0 in stage 1.0 (TID 205) in 78938 ms on ip-10-0-49-229.ec2.internal (executor 9) (65/270) | |
| 17/06/14 04:15:34 INFO TaskSetManager: Starting task 133.0 in stage 1.0 (TID 260, ip-10-0-49-229.ec2.internal, executor 9, partition 133, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:34 INFO TaskSetManager: Finished task 86.0 in stage 1.0 (TID 204) in 79180 ms on ip-10-0-49-229.ec2.internal (executor 9) (66/270) | |
| 17/06/14 04:15:35 INFO YarnAllocator: Driver requested a total number of 51 executor(s). | |
| 17/06/14 04:15:35 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 51 executors. | |
| 17/06/14 04:15:35 INFO TaskSetManager: Starting task 136.0 in stage 1.0 (TID 261, ip-10-0-49-229.ec2.internal, executor 9, partition 136, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:35 INFO TaskSetManager: Finished task 9.0 in stage 1.0 (TID 189) in 82403 ms on ip-10-0-49-229.ec2.internal (executor 9) (67/270) | |
| 17/06/14 04:15:35 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.229:48782 | |
| 17/06/14 04:15:37 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-50-96.ec2.internal:46753 (size: 26.1 KB) | |
| 17/06/14 04:15:37 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-50-96.ec2.internal:46753 (size: 33.3 KB) | |
| 17/06/14 04:15:37 INFO BlockManagerInfo: Removed rdd_2_60 on ip-10-0-50-96.ec2.internal:46753 in memory (size: 421.9 MB, free: 1173.8 MB) | |
| 17/06/14 04:15:38 INFO TaskSetManager: Starting task 137.0 in stage 1.0 (TID 262, ip-10-0-50-96.ec2.internal, executor 12, partition 137, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:38 INFO TaskSetManager: Finished task 13.0 in stage 1.0 (TID 190) in 84831 ms on ip-10-0-50-96.ec2.internal (executor 12) (68/270) | |
| 17/06/14 04:15:41 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-49-229.ec2.internal:44338 (size: 26.1 KB) | |
| 17/06/14 04:15:41 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-49-229.ec2.internal:44338 (size: 33.3 KB) | |
| 17/06/14 04:15:41 INFO BlockManagerInfo: Removed rdd_2_51 on ip-10-0-49-229.ec2.internal:44338 in memory (size: 346.8 MB, free: 1181.4 MB) | |
| 17/06/14 04:15:44 INFO BlockManagerInfo: Removed rdd_2_58 on ip-10-0-50-96.ec2.internal:46753 in memory (size: 827.4 MB, free: 2001.2 MB) | |
| 17/06/14 04:15:48 INFO TaskSetManager: Starting task 138.0 in stage 1.0 (TID 263, ip-10-0-50-96.ec2.internal, executor 13, partition 138, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:48 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 186) in 95403 ms on ip-10-0-50-96.ec2.internal (executor 13) (69/270) | |
| 17/06/14 04:15:48 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:35520 | |
| 17/06/14 04:15:50 INFO TaskSetManager: Starting task 139.0 in stage 1.0 (TID 264, ip-10-0-50-96.ec2.internal, executor 15, partition 139, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:50 INFO TaskSetManager: Finished task 118.0 in stage 1.0 (TID 242) in 93375 ms on ip-10-0-50-96.ec2.internal (executor 15) (70/270) | |
| 17/06/14 04:15:50 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:35522 | |
| 17/06/14 04:15:50 INFO YarnAllocator: Driver requested a total number of 50 executor(s). | |
| 17/06/14 04:15:50 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 50 executors. | |
| 17/06/14 04:15:50 INFO TaskSetManager: Starting task 91.0 in stage 1.0 (TID 265, ip-10-0-49-157.ec2.internal, executor 4, partition 91, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:50 INFO TaskSetManager: Finished task 48.0 in stage 1.0 (TID 219) in 94838 ms on ip-10-0-49-157.ec2.internal (executor 4) (71/270) | |
| 17/06/14 04:15:51 INFO TaskSetManager: Starting task 93.0 in stage 1.0 (TID 266, ip-10-0-49-157.ec2.internal, executor 4, partition 93, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:51 INFO TaskSetManager: Finished task 56.0 in stage 1.0 (TID 221) in 95408 ms on ip-10-0-49-157.ec2.internal (executor 4) (72/270) | |
| 17/06/14 04:15:52 INFO TaskSetManager: Starting task 140.0 in stage 1.0 (TID 267, ip-10-0-50-96.ec2.internal, executor 14, partition 140, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:52 INFO TaskSetManager: Finished task 16.0 in stage 1.0 (TID 193) in 98706 ms on ip-10-0-50-96.ec2.internal (executor 14) (73/270) | |
| 17/06/14 04:15:52 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:35516 | |
| 17/06/14 04:15:52 INFO TaskSetManager: Starting task 141.0 in stage 1.0 (TID 268, ip-10-0-50-96.ec2.internal, executor 15, partition 141, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:52 INFO TaskSetManager: Finished task 20.0 in stage 1.0 (TID 194) in 98814 ms on ip-10-0-50-96.ec2.internal (executor 15) (74/270) | |
| 17/06/14 04:15:52 INFO YarnAllocator: Driver requested a total number of 49 executor(s). | |
| 17/06/14 04:15:52 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 49 executors. | |
| 17/06/14 04:15:53 INFO TaskSetManager: Starting task 142.0 in stage 1.0 (TID 269, ip-10-0-50-96.ec2.internal, executor 14, partition 142, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:53 INFO TaskSetManager: Finished task 131.0 in stage 1.0 (TID 247) in 96092 ms on ip-10-0-50-96.ec2.internal (executor 14) (75/270) | |
| 17/06/14 04:15:53 INFO TaskSetManager: Starting task 143.0 in stage 1.0 (TID 270, ip-10-0-50-96.ec2.internal, executor 14, partition 143, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:53 INFO TaskSetManager: Finished task 132.0 in stage 1.0 (TID 248) in 96543 ms on ip-10-0-50-96.ec2.internal (executor 14) (76/270) | |
| 17/06/14 04:15:54 INFO TaskSetManager: Starting task 95.0 in stage 1.0 (TID 271, ip-10-0-49-157.ec2.internal, executor 4, partition 95, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:54 INFO TaskSetManager: Finished task 59.0 in stage 1.0 (TID 222) in 98198 ms on ip-10-0-49-157.ec2.internal (executor 4) (77/270) | |
| 17/06/14 04:15:54 INFO TaskSetManager: Starting task 144.0 in stage 1.0 (TID 272, ip-10-0-50-96.ec2.internal, executor 13, partition 144, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:54 INFO TaskSetManager: Finished task 98.0 in stage 1.0 (TID 239) in 97548 ms on ip-10-0-50-96.ec2.internal (executor 13) (78/270) | |
| 17/06/14 04:15:54 INFO YarnAllocator: Driver requested a total number of 48 executor(s). | |
| 17/06/14 04:15:54 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 48 executors. | |
| 17/06/14 04:15:54 INFO TaskSetManager: Starting task 145.0 in stage 1.0 (TID 273, ip-10-0-50-96.ec2.internal, executor 14, partition 145, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:54 INFO TaskSetManager: Finished task 125.0 in stage 1.0 (TID 246) in 97309 ms on ip-10-0-50-96.ec2.internal (executor 14) (79/270) | |
| 17/06/14 04:15:55 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-50-96.ec2.internal:32956 (size: 26.1 KB) | |
| 17/06/14 04:15:55 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-50-96.ec2.internal:32956 (size: 33.3 KB) | |
| 17/06/14 04:15:55 INFO BlockManagerInfo: Removed rdd_2_63 on ip-10-0-50-96.ec2.internal:32956 in memory (size: 783.8 MB, free: 1228.2 MB) | |
| 17/06/14 04:15:55 INFO TaskSetManager: Starting task 109.0 in stage 1.0 (TID 274, ip-10-0-51-185.ec2.internal, executor 2, partition 109, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:55 INFO TaskSetManager: Finished task 49.0 in stage 1.0 (TID 224) in 99633 ms on ip-10-0-51-185.ec2.internal (executor 2) (80/270) | |
| 17/06/14 04:15:55 INFO TaskSetManager: Starting task 111.0 in stage 1.0 (TID 275, ip-10-0-51-185.ec2.internal, executor 2, partition 111, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:55 INFO TaskSetManager: Finished task 55.0 in stage 1.0 (TID 226) in 99703 ms on ip-10-0-51-185.ec2.internal (executor 2) (81/270) | |
| 17/06/14 04:15:55 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-50-96.ec2.internal:41521 (size: 26.1 KB) | |
| 17/06/14 04:15:55 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-50-96.ec2.internal:41521 (size: 33.3 KB) | |
| 17/06/14 04:15:55 INFO BlockManagerInfo: Removed rdd_2_57 on ip-10-0-50-96.ec2.internal:41521 in memory (size: 786.4 MB, free: 1218.1 MB) | |
| 17/06/14 04:15:56 INFO TaskSetManager: Starting task 146.0 in stage 1.0 (TID 276, ip-10-0-50-96.ec2.internal, executor 13, partition 146, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:56 INFO TaskSetManager: Finished task 100.0 in stage 1.0 (TID 240) in 99141 ms on ip-10-0-50-96.ec2.internal (executor 13) (82/270) | |
| 17/06/14 04:15:56 INFO YarnAllocator: Driver requested a total number of 47 executor(s). | |
| 17/06/14 04:15:56 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 47 executors. | |
| 17/06/14 04:15:56 INFO TaskSetManager: Starting task 113.0 in stage 1.0 (TID 277, ip-10-0-49-157.ec2.internal, executor 4, partition 113, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:56 INFO TaskSetManager: Finished task 53.0 in stage 1.0 (TID 220) in 100099 ms on ip-10-0-49-157.ec2.internal (executor 4) (83/270) | |
| 17/06/14 04:15:56 INFO TaskSetManager: Starting task 117.0 in stage 1.0 (TID 278, ip-10-0-51-185.ec2.internal, executor 2, partition 117, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:56 INFO TaskSetManager: Finished task 52.0 in stage 1.0 (TID 225) in 100170 ms on ip-10-0-51-185.ec2.internal (executor 2) (84/270) | |
| 17/06/14 04:15:56 INFO TaskSetManager: Starting task 147.0 in stage 1.0 (TID 279, ip-10-0-50-96.ec2.internal, executor 15, partition 147, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:56 INFO TaskSetManager: Finished task 121.0 in stage 1.0 (TID 244) in 99412 ms on ip-10-0-50-96.ec2.internal (executor 15) (85/270) | |
| 17/06/14 04:15:56 INFO TaskSetManager: Starting task 122.0 in stage 1.0 (TID 280, ip-10-0-51-185.ec2.internal, executor 1, partition 122, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:56 INFO TaskSetManager: Finished task 31.0 in stage 1.0 (TID 213) in 100713 ms on ip-10-0-51-185.ec2.internal (executor 1) (86/270) | |
| 17/06/14 04:15:56 INFO YarnAllocator: Driver requested a total number of 46 executor(s). | |
| 17/06/14 04:15:56 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 46 executors. | |
| 17/06/14 04:15:56 INFO TaskSetManager: Starting task 128.0 in stage 1.0 (TID 281, ip-10-0-51-185.ec2.internal, executor 2, partition 128, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:56 INFO TaskSetManager: Finished task 75.0 in stage 1.0 (TID 232) in 100375 ms on ip-10-0-51-185.ec2.internal (executor 2) (87/270) | |
| 17/06/14 04:15:56 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-50-96.ec2.internal:38815 (size: 26.1 KB) | |
| 17/06/14 04:15:56 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-50-96.ec2.internal:38815 (size: 33.3 KB) | |
| 17/06/14 04:15:56 INFO BlockManagerInfo: Removed rdd_2_37 on ip-10-0-50-96.ec2.internal:38815 in memory (size: 793.2 MB, free: 1240.1 MB) | |
| 17/06/14 04:15:57 INFO TaskSetManager: Starting task 130.0 in stage 1.0 (TID 282, ip-10-0-51-185.ec2.internal, executor 1, partition 130, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:57 INFO TaskSetManager: Finished task 39.0 in stage 1.0 (TID 215) in 101891 ms on ip-10-0-51-185.ec2.internal (executor 1) (88/270) | |
| 17/06/14 04:15:57 INFO TaskSetManager: Starting task 148.0 in stage 1.0 (TID 283, ip-10-0-50-96.ec2.internal, executor 15, partition 148, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:57 INFO TaskSetManager: Finished task 119.0 in stage 1.0 (TID 243) in 100864 ms on ip-10-0-50-96.ec2.internal (executor 15) (89/270) | |
| 17/06/14 04:15:58 INFO TaskSetManager: Starting task 149.0 in stage 1.0 (TID 284, ip-10-0-50-96.ec2.internal, executor 13, partition 149, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:15:58 INFO TaskSetManager: Finished task 112.0 in stage 1.0 (TID 241) in 101175 ms on ip-10-0-50-96.ec2.internal (executor 13) (90/270) | |
| 17/06/14 04:15:58 INFO YarnAllocator: Driver requested a total number of 45 executor(s). | |
| 17/06/14 04:15:58 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 45 executors. | |
| 17/06/14 04:15:58 INFO YarnAllocator: Launching container container_1497410974650_0004_01_002432 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 04:15:58 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:15:58 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:15:58 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000016 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:15:58 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:15:58 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 04:15:58 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:15:58 ERROR YarnClusterScheduler: Lost executor 12 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:15:58 WARN TaskSetManager: Lost task 82.0 in stage 1.0 (TID 236, ip-10-0-50-96.ec2.internal, executor 12): ExecutorLostFailure (executor 12 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:15:58 WARN TaskSetManager: Lost task 137.0 in stage 1.0 (TID 262, ip-10-0-50-96.ec2.internal, executor 12): ExecutorLostFailure (executor 12 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:15:58 WARN TaskSetManager: Lost task 96.0 in stage 1.0 (TID 238, ip-10-0-50-96.ec2.internal, executor 12): ExecutorLostFailure (executor 12 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:15:58 WARN TaskSetManager: Lost task 135.0 in stage 1.0 (TID 252, ip-10-0-50-96.ec2.internal, executor 12): ExecutorLostFailure (executor 12 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:15:58 INFO DAGScheduler: Executor lost: 12 (epoch 1) | |
| 17/06/14 04:15:58 INFO BlockManagerMasterEndpoint: Trying to remove executor 12 from BlockManagerMaster. | |
| 17/06/14 04:15:58 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(12, ip-10-0-50-96.ec2.internal, 46753, None) | |
| 17/06/14 04:15:58 INFO BlockManagerMaster: Removed 12 successfully in removeExecutor | |
| 17/06/14 04:15:58 INFO ExecutorAllocationManager: Existing executor 12 has been removed (new total is 14) | |
| 17/06/14 04:15:58 INFO BlockManagerInfo: Removed rdd_2_94 on ip-10-0-50-96.ec2.internal:32956 in memory (size: 798.9 MB, free: 2027.1 MB) | |
| 17/06/14 04:15:58 INFO TaskSetManager: Starting task 96.1 in stage 1.0 (TID 285, ip-10-0-51-185.ec2.internal, executor 1, partition 96, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:15:58 INFO TaskSetManager: Finished task 30.0 in stage 1.0 (TID 211) in 102863 ms on ip-10-0-51-185.ec2.internal (executor 1) (91/270) | |
| 17/06/14 04:15:58 WARN TransportChannelHandler: Exception in connection from /10.0.50.96:35518 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 04:15:58 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 12. | |
| 17/06/14 04:15:58 ERROR YarnClusterScheduler: Lost an executor 12 (already removed): Pending loss reason. | |
| 17/06/14 04:15:58 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:15:58 INFO BlockManagerMaster: Removal of executor 12 requested | |
| 17/06/14 04:15:58 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 12 | |
| 17/06/14 04:15:58 INFO BlockManagerMasterEndpoint: Trying to remove executor 12 from BlockManagerMaster. | |
| 17/06/14 04:15:58 INFO YarnAllocator: Canceled 9 unlocalized container requests to resubmit with locality | |
| 17/06/14 04:15:58 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:15:58 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:15:58 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:15:58 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:15:58 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:15:58 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:15:58 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:15:58 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:15:58 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:15:58 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:15:59 INFO BlockManagerInfo: Removed rdd_2_36 on ip-10-0-50-96.ec2.internal:38815 in memory (size: 763.3 MB, free: 2003.4 MB) | |
| 17/06/14 04:16:01 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:36208) with ID 16 | |
| 17/06/14 04:16:01 INFO TaskSetManager: Starting task 82.1 in stage 1.0 (TID 286, ip-10-0-50-96.ec2.internal, executor 16, partition 82, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:16:01 INFO TaskSetManager: Starting task 135.1 in stage 1.0 (TID 287, ip-10-0-50-96.ec2.internal, executor 16, partition 135, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:01 INFO TaskSetManager: Starting task 137.1 in stage 1.0 (TID 288, ip-10-0-50-96.ec2.internal, executor 16, partition 137, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:01 INFO TaskSetManager: Starting task 150.0 in stage 1.0 (TID 289, ip-10-0-50-96.ec2.internal, executor 16, partition 150, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:01 INFO ExecutorAllocationManager: New executor 16 has registered (new total is 15) | |
| 17/06/14 04:16:01 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:41603 with 2.8 GB RAM, BlockManagerId(16, ip-10-0-50-96.ec2.internal, 41603, None) | |
| 17/06/14 04:16:01 INFO TaskSetManager: Starting task 151.0 in stage 1.0 (TID 290, ip-10-0-51-185.ec2.internal, executor 1, partition 151, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:01 INFO TaskSetManager: Finished task 44.0 in stage 1.0 (TID 218) in 105389 ms on ip-10-0-51-185.ec2.internal (executor 1) (92/270) | |
| 17/06/14 04:16:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:34210 | |
| 17/06/14 04:16:02 INFO TaskSetManager: Starting task 152.0 in stage 1.0 (TID 291, ip-10-0-51-185.ec2.internal, executor 8, partition 152, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:02 INFO TaskSetManager: Finished task 61.0 in stage 1.0 (TID 228) in 106529 ms on ip-10-0-51-185.ec2.internal (executor 8) (93/270) | |
| 17/06/14 04:16:02 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:34268 | |
| 17/06/14 04:16:02 INFO TaskSetManager: Starting task 153.0 in stage 1.0 (TID 292, ip-10-0-49-157.ec2.internal, executor 5, partition 153, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:02 INFO TaskSetManager: Finished task 45.0 in stage 1.0 (TID 217) in 106839 ms on ip-10-0-49-157.ec2.internal (executor 5) (94/270) | |
| 17/06/14 04:16:02 INFO YarnAllocator: Driver requested a total number of 44 executor(s). | |
| 17/06/14 04:16:02 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 44 executors. | |
| 17/06/14 04:16:02 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.157:44698 | |
| 17/06/14 04:16:02 INFO TaskSetManager: Starting task 154.0 in stage 1.0 (TID 293, ip-10-0-51-185.ec2.internal, executor 8, partition 154, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:02 INFO TaskSetManager: Finished task 71.0 in stage 1.0 (TID 229) in 106738 ms on ip-10-0-51-185.ec2.internal (executor 8) (95/270) | |
| 17/06/14 04:16:03 INFO TaskSetManager: Starting task 155.0 in stage 1.0 (TID 294, ip-10-0-49-157.ec2.internal, executor 7, partition 155, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:03 INFO TaskSetManager: Finished task 73.0 in stage 1.0 (TID 231) in 106823 ms on ip-10-0-49-157.ec2.internal (executor 7) (96/270) | |
| 17/06/14 04:16:03 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.157:44702 | |
| 17/06/14 04:16:03 INFO TaskSetManager: Starting task 156.0 in stage 1.0 (TID 295, ip-10-0-51-185.ec2.internal, executor 3, partition 156, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:03 INFO TaskSetManager: Finished task 14.0 in stage 1.0 (TID 191) in 109943 ms on ip-10-0-51-185.ec2.internal (executor 3) (97/270) | |
| 17/06/14 04:16:03 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:34214 | |
| 17/06/14 04:16:03 INFO TaskSetManager: Starting task 157.0 in stage 1.0 (TID 296, ip-10-0-49-157.ec2.internal, executor 5, partition 157, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:03 INFO TaskSetManager: Finished task 27.0 in stage 1.0 (TID 212) in 107783 ms on ip-10-0-49-157.ec2.internal (executor 5) (98/270) | |
| 17/06/14 04:16:03 INFO YarnAllocator: Driver requested a total number of 43 executor(s). | |
| 17/06/14 04:16:03 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 43 executors. | |
| 17/06/14 04:16:03 INFO TaskSetManager: Starting task 158.0 in stage 1.0 (TID 297, ip-10-0-49-157.ec2.internal, executor 5, partition 158, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:03 INFO TaskSetManager: Finished task 28.0 in stage 1.0 (TID 214) in 108081 ms on ip-10-0-49-157.ec2.internal (executor 5) (99/270) | |
| 17/06/14 04:16:04 INFO TaskSetManager: Starting task 159.0 in stage 1.0 (TID 298, ip-10-0-51-185.ec2.internal, executor 3, partition 159, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:04 INFO TaskSetManager: Finished task 81.0 in stage 1.0 (TID 235) in 107723 ms on ip-10-0-51-185.ec2.internal (executor 3) (100/270) | |
| 17/06/14 04:16:04 INFO TaskSetManager: Starting task 160.0 in stage 1.0 (TID 299, ip-10-0-51-185.ec2.internal, executor 8, partition 160, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:04 INFO TaskSetManager: Finished task 11.0 in stage 1.0 (TID 188) in 110997 ms on ip-10-0-51-185.ec2.internal (executor 8) (101/270) | |
| 17/06/14 04:16:04 INFO TaskSetManager: Starting task 161.0 in stage 1.0 (TID 300, ip-10-0-49-157.ec2.internal, executor 7, partition 161, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:04 INFO TaskSetManager: Finished task 67.0 in stage 1.0 (TID 227) in 108307 ms on ip-10-0-49-157.ec2.internal (executor 7) (102/270) | |
| 17/06/14 04:16:04 INFO YarnAllocator: Driver requested a total number of 42 executor(s). | |
| 17/06/14 04:16:04 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 42 executors. | |
| 17/06/14 04:16:04 INFO TaskSetManager: Starting task 162.0 in stage 1.0 (TID 301, ip-10-0-49-157.ec2.internal, executor 5, partition 162, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:04 INFO TaskSetManager: Finished task 40.0 in stage 1.0 (TID 216) in 108914 ms on ip-10-0-49-157.ec2.internal (executor 5) (103/270) | |
| 17/06/14 04:16:05 INFO BlockManagerInfo: Added rdd_2_3 in memory on ip-10-0-49-157.ec2.internal:34052 (size: 764.4 MB, free: 772.3 MB) | |
| 17/06/14 04:16:05 INFO TaskSetManager: Starting task 163.0 in stage 1.0 (TID 302, ip-10-0-49-157.ec2.internal, executor 6, partition 163, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:05 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 187) in 112131 ms on ip-10-0-49-157.ec2.internal (executor 6) (104/270) | |
| 17/06/14 04:16:05 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.157:44700 | |
| 17/06/14 04:16:05 INFO TaskSetManager: Starting task 164.0 in stage 1.0 (TID 303, ip-10-0-49-157.ec2.internal, executor 7, partition 164, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:05 INFO TaskSetManager: Finished task 85.0 in stage 1.0 (TID 245) in 108573 ms on ip-10-0-49-157.ec2.internal (executor 7) (105/270) | |
| 17/06/14 04:16:06 INFO TaskSetManager: Starting task 165.0 in stage 1.0 (TID 304, ip-10-0-51-185.ec2.internal, executor 8, partition 165, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:06 INFO TaskSetManager: Finished task 72.0 in stage 1.0 (TID 230) in 110443 ms on ip-10-0-51-185.ec2.internal (executor 8) (106/270) | |
| 17/06/14 04:16:06 INFO YarnAllocator: Driver requested a total number of 41 executor(s). | |
| 17/06/14 04:16:06 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 41 executors. | |
| 17/06/14 04:16:07 INFO TaskSetManager: Starting task 166.0 in stage 1.0 (TID 305, ip-10-0-51-185.ec2.internal, executor 3, partition 166, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:07 INFO TaskSetManager: Finished task 77.0 in stage 1.0 (TID 233) in 110926 ms on ip-10-0-51-185.ec2.internal (executor 3) (107/270) | |
| 17/06/14 04:16:07 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-51-185.ec2.internal:32987 (size: 26.1 KB) | |
| 17/06/14 04:16:07 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-51-185.ec2.internal:32987 (size: 33.3 KB) | |
| 17/06/14 04:16:07 INFO BlockManagerInfo: Removed rdd_2_1 on ip-10-0-51-185.ec2.internal:32987 in memory (size: 794.8 MB, free: 1154.7 MB) | |
| 17/06/14 04:16:07 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 14. | |
| 17/06/14 04:16:07 INFO DAGScheduler: Executor lost: 14 (epoch 1) | |
| 17/06/14 04:16:07 INFO BlockManagerMasterEndpoint: Trying to remove executor 14 from BlockManagerMaster. | |
| 17/06/14 04:16:07 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(14, ip-10-0-50-96.ec2.internal, 38815, None) | |
| 17/06/14 04:16:07 INFO BlockManagerMaster: Removed 14 successfully in removeExecutor | |
| 17/06/14 04:16:07 INFO YarnAllocator: Launching container container_1497410974650_0004_01_002509 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 04:16:07 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:16:07 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000018 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:16:07 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:07 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:16:07 ERROR YarnClusterScheduler: Lost executor 14 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:07 WARN TaskSetManager: Lost task 140.0 in stage 1.0 (TID 267, ip-10-0-50-96.ec2.internal, executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:07 WARN TaskSetManager: Lost task 143.0 in stage 1.0 (TID 270, ip-10-0-50-96.ec2.internal, executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:07 WARN TaskSetManager: Lost task 145.0 in stage 1.0 (TID 273, ip-10-0-50-96.ec2.internal, executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:07 WARN TaskSetManager: Lost task 142.0 in stage 1.0 (TID 269, ip-10-0-50-96.ec2.internal, executor 14): ExecutorLostFailure (executor 14 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:07 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:07 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 04:16:07 INFO BlockManagerMasterEndpoint: Trying to remove executor 14 from BlockManagerMaster. | |
| 17/06/14 04:16:07 INFO ExecutorAllocationManager: Existing executor 14 has been removed (new total is 14) | |
| 17/06/14 04:16:07 INFO BlockManagerMaster: Removal of executor 14 requested | |
| 17/06/14 04:16:07 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 14 | |
| 17/06/14 04:16:08 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:16:08 INFO YarnAllocator: Canceled 3 unlocalized container requests to resubmit with locality | |
| 17/06/14 04:16:08 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:16:08 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:16:08 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:16:08 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:16:08 INFO TaskSetManager: Starting task 142.1 in stage 1.0 (TID 306, ip-10-0-49-157.ec2.internal, executor 6, partition 142, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:08 INFO TaskSetManager: Finished task 23.0 in stage 1.0 (TID 207) in 112613 ms on ip-10-0-49-157.ec2.internal (executor 6) (108/270) | |
| 17/06/14 04:16:08 INFO TaskSetManager: Starting task 145.1 in stage 1.0 (TID 307, ip-10-0-49-157.ec2.internal, executor 7, partition 145, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:08 INFO TaskSetManager: Finished task 12.0 in stage 1.0 (TID 192) in 115455 ms on ip-10-0-49-157.ec2.internal (executor 7) (109/270) | |
| 17/06/14 04:16:09 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-50-96.ec2.internal:41603 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:16:09 INFO BlockManagerInfo: Removed rdd_2_108 on ip-10-0-50-96.ec2.internal:41521 in memory (size: 803.8 MB, free: 2021.9 MB) | |
| 17/06/14 04:16:09 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-51-185.ec2.internal:43229 (size: 26.1 KB) | |
| 17/06/14 04:16:09 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-51-185.ec2.internal:43229 (size: 33.3 KB) | |
| 17/06/14 04:16:09 INFO BlockManagerInfo: Removed rdd_2_34 on ip-10-0-51-185.ec2.internal:43229 in memory (size: 775.9 MB, free: 1416.3 MB) | |
| 17/06/14 04:16:09 INFO TaskSetManager: Starting task 143.1 in stage 1.0 (TID 308, ip-10-0-49-157.ec2.internal, executor 6, partition 143, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:09 INFO TaskSetManager: Finished task 26.0 in stage 1.0 (TID 208) in 113550 ms on ip-10-0-49-157.ec2.internal (executor 6) (110/270) | |
| 17/06/14 04:16:09 INFO YarnAllocator: Driver requested a total number of 40 executor(s). | |
| 17/06/14 04:16:09 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 40 executors. | |
| 17/06/14 04:16:09 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-49-157.ec2.internal:45398 (size: 26.1 KB) | |
| 17/06/14 04:16:09 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-49-157.ec2.internal:45398 (size: 33.3 KB) | |
| 17/06/14 04:16:09 INFO BlockManagerInfo: Removed rdd_2_4 on ip-10-0-49-157.ec2.internal:45398 in memory (size: 202.3 MB, free: 951.4 MB) | |
| 17/06/14 04:16:09 INFO TaskSetManager: Starting task 140.1 in stage 1.0 (TID 309, ip-10-0-51-185.ec2.internal, executor 3, partition 140, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:09 INFO TaskSetManager: Finished task 80.0 in stage 1.0 (TID 234) in 113405 ms on ip-10-0-51-185.ec2.internal (executor 3) (111/270) | |
| 17/06/14 04:16:10 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:36208 | |
| 17/06/14 04:16:10 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-49-157.ec2.internal:43518 (size: 26.1 KB) | |
| 17/06/14 04:16:10 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-49-157.ec2.internal:43518 (size: 33.3 KB) | |
| 17/06/14 04:16:10 INFO BlockManagerInfo: Removed rdd_2_64 on ip-10-0-49-157.ec2.internal:43518 in memory (size: 791.6 MB, free: 1444.9 MB) | |
| 17/06/14 04:16:11 INFO YarnAllocator: Launching container container_1497410974650_0004_01_002535 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 04:16:11 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:16:11 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000017 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:16:11 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:11 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:16:11 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 04:16:11 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:11 ERROR YarnClusterScheduler: Lost executor 13 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:11 WARN TaskSetManager: Lost task 146.0 in stage 1.0 (TID 276, ip-10-0-50-96.ec2.internal, executor 13): ExecutorLostFailure (executor 13 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:11 WARN TaskSetManager: Lost task 144.0 in stage 1.0 (TID 272, ip-10-0-50-96.ec2.internal, executor 13): ExecutorLostFailure (executor 13 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:11 WARN TaskSetManager: Lost task 138.0 in stage 1.0 (TID 263, ip-10-0-50-96.ec2.internal, executor 13): ExecutorLostFailure (executor 13 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:11 WARN TaskSetManager: Lost task 149.0 in stage 1.0 (TID 284, ip-10-0-50-96.ec2.internal, executor 13): ExecutorLostFailure (executor 13 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:11 INFO DAGScheduler: Executor lost: 13 (epoch 1) | |
| 17/06/14 04:16:11 INFO BlockManagerMasterEndpoint: Trying to remove executor 13 from BlockManagerMaster. | |
| 17/06/14 04:16:11 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(13, ip-10-0-50-96.ec2.internal, 41521, None) | |
| 17/06/14 04:16:11 INFO ExecutorAllocationManager: Existing executor 13 has been removed (new total is 13) | |
| 17/06/14 04:16:11 WARN ExecutorAllocationManager: Attempted to mark unknown executor 13 idle | |
| 17/06/14 04:16:11 INFO BlockManagerMaster: Removed 13 successfully in removeExecutor | |
| 17/06/14 04:16:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:36224) with ID 17 | |
| 17/06/14 04:16:11 INFO TaskSetManager: Starting task 149.1 in stage 1.0 (TID 310, ip-10-0-50-96.ec2.internal, executor 17, partition 149, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:11 INFO TaskSetManager: Starting task 138.1 in stage 1.0 (TID 311, ip-10-0-50-96.ec2.internal, executor 17, partition 138, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:11 INFO TaskSetManager: Starting task 144.1 in stage 1.0 (TID 312, ip-10-0-50-96.ec2.internal, executor 17, partition 144, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:11 INFO TaskSetManager: Starting task 146.1 in stage 1.0 (TID 313, ip-10-0-50-96.ec2.internal, executor 17, partition 146, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:11 INFO ExecutorAllocationManager: New executor 17 has registered (new total is 14) | |
| 17/06/14 04:16:11 WARN TransportChannelHandler: Exception in connection from /10.0.50.96:35520 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 04:16:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 13. | |
| 17/06/14 04:16:11 ERROR YarnClusterScheduler: Lost an executor 13 (already removed): Pending loss reason. | |
| 17/06/14 04:16:11 INFO BlockManagerMaster: Removal of executor 13 requested | |
| 17/06/14 04:16:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 13 | |
| 17/06/14 04:16:11 INFO BlockManagerMasterEndpoint: Trying to remove executor 13 from BlockManagerMaster. | |
| 17/06/14 04:16:11 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:16:11 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:16:11 INFO TaskSetManager: Starting task 167.0 in stage 1.0 (TID 314, ip-10-0-49-157.ec2.internal, executor 6, partition 167, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:11 INFO TaskSetManager: Finished task 19.0 in stage 1.0 (TID 206) in 115531 ms on ip-10-0-49-157.ec2.internal (executor 6) (112/270) | |
| 17/06/14 04:16:11 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:33453 with 2.8 GB RAM, BlockManagerId(17, ip-10-0-50-96.ec2.internal, 33453, None) | |
| 17/06/14 04:16:11 INFO BlockManagerInfo: Removed rdd_2_7 on ip-10-0-49-157.ec2.internal:45398 in memory (size: 313.2 MB, free: 1264.7 MB) | |
| 17/06/14 04:16:11 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-51-185.ec2.internal:34392 (size: 26.1 KB) | |
| 17/06/14 04:16:11 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-50-96.ec2.internal:33453 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:16:12 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-51-185.ec2.internal:34392 (size: 33.3 KB) | |
| 17/06/14 04:16:12 INFO BlockManagerInfo: Removed rdd_2_15 on ip-10-0-51-185.ec2.internal:34392 in memory (size: 825.1 MB, free: 1771.1 MB) | |
| 17/06/14 04:16:12 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:36224 | |
| 17/06/14 04:16:12 INFO BlockManagerInfo: Removed rdd_2_47 on ip-10-0-49-157.ec2.internal:45398 in memory (size: 739.9 MB, free: 2004.6 MB) | |
| 17/06/14 04:16:13 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:36240) with ID 18 | |
| 17/06/14 04:16:13 INFO TaskSetManager: Starting task 168.0 in stage 1.0 (TID 315, ip-10-0-50-96.ec2.internal, executor 18, partition 168, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:13 INFO TaskSetManager: Starting task 169.0 in stage 1.0 (TID 316, ip-10-0-50-96.ec2.internal, executor 18, partition 169, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:13 INFO TaskSetManager: Starting task 170.0 in stage 1.0 (TID 317, ip-10-0-50-96.ec2.internal, executor 18, partition 170, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:13 INFO TaskSetManager: Starting task 171.0 in stage 1.0 (TID 318, ip-10-0-50-96.ec2.internal, executor 18, partition 171, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:14 INFO ExecutorAllocationManager: New executor 18 has registered (new total is 15) | |
| 17/06/14 04:16:14 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:43860 with 2.8 GB RAM, BlockManagerId(18, ip-10-0-50-96.ec2.internal, 43860, None) | |
| 17/06/14 04:16:14 INFO YarnAllocator: Launching container container_1497410974650_0004_01_002561 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 04:16:14 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:16:14 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000019 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:16:14 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:14 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:14 ERROR YarnClusterScheduler: Lost executor 15 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:14 WARN TaskSetManager: Lost task 148.0 in stage 1.0 (TID 283, ip-10-0-50-96.ec2.internal, executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:14 WARN TaskSetManager: Lost task 141.0 in stage 1.0 (TID 268, ip-10-0-50-96.ec2.internal, executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:14 WARN TaskSetManager: Lost task 147.0 in stage 1.0 (TID 279, ip-10-0-50-96.ec2.internal, executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:14 WARN TaskSetManager: Lost task 139.0 in stage 1.0 (TID 264, ip-10-0-50-96.ec2.internal, executor 15): ExecutorLostFailure (executor 15 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:14 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:16:14 INFO DAGScheduler: Executor lost: 15 (epoch 1) | |
| 17/06/14 04:16:14 INFO BlockManagerMasterEndpoint: Trying to remove executor 15 from BlockManagerMaster. | |
| 17/06/14 04:16:14 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(15, ip-10-0-50-96.ec2.internal, 32956, None) | |
| 17/06/14 04:16:14 INFO BlockManagerMaster: Removed 15 successfully in removeExecutor | |
| 17/06/14 04:16:14 INFO ExecutorAllocationManager: Existing executor 15 has been removed (new total is 14) | |
| 17/06/14 04:16:14 WARN ExecutorAllocationManager: Attempted to mark unknown executor 15 idle | |
| 17/06/14 04:16:14 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 04:16:14 WARN TransportChannelHandler: Exception in connection from /10.0.50.96:35522 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 04:16:14 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 15. | |
| 17/06/14 04:16:14 ERROR YarnClusterScheduler: Lost an executor 15 (already removed): Pending loss reason. | |
| 17/06/14 04:16:14 INFO BlockManagerMaster: Removal of executor 15 requested | |
| 17/06/14 04:16:14 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 15 | |
| 17/06/14 04:16:14 INFO BlockManagerMasterEndpoint: Trying to remove executor 15 from BlockManagerMaster. | |
| 17/06/14 04:16:14 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-49-157.ec2.internal:34052 (size: 26.1 KB) | |
| 17/06/14 04:16:14 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-49-157.ec2.internal:34052 (size: 33.3 KB) | |
| 17/06/14 04:16:14 INFO BlockManagerInfo: Removed rdd_2_10 on ip-10-0-49-157.ec2.internal:34052 in memory (size: 154.5 MB, free: 926.8 MB) | |
| 17/06/14 04:16:14 INFO BlockManagerInfo: Removed rdd_2_8 on ip-10-0-49-157.ec2.internal:34052 in memory (size: 368.8 MB, free: 1295.6 MB) | |
| 17/06/14 04:16:14 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:16:14 INFO YarnAllocator: Canceled 1 unlocalized container requests to resubmit with locality | |
| 17/06/14 04:16:14 INFO YarnAllocator: Submitted container request for host ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:16:14 INFO YarnAllocator: Submitted container request for host ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:16:14 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-50-96.ec2.internal:43860 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:16:15 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:36240 | |
| 17/06/14 04:16:16 INFO BlockManagerInfo: Removed rdd_2_114 on ip-10-0-49-157.ec2.internal:34052 in memory (size: 767.1 MB, free: 2.0 GB) | |
| 17/06/14 04:16:17 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:36258) with ID 19 | |
| 17/06/14 04:16:17 INFO TaskSetManager: Starting task 139.1 in stage 1.0 (TID 319, ip-10-0-50-96.ec2.internal, executor 19, partition 139, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:17 INFO TaskSetManager: Starting task 147.1 in stage 1.0 (TID 320, ip-10-0-50-96.ec2.internal, executor 19, partition 147, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:17 INFO TaskSetManager: Starting task 141.1 in stage 1.0 (TID 321, ip-10-0-50-96.ec2.internal, executor 19, partition 141, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:17 INFO TaskSetManager: Starting task 148.1 in stage 1.0 (TID 322, ip-10-0-50-96.ec2.internal, executor 19, partition 148, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:17 INFO ExecutorAllocationManager: New executor 19 has registered (new total is 15) | |
| 17/06/14 04:16:17 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:35803 with 2.8 GB RAM, BlockManagerId(19, ip-10-0-50-96.ec2.internal, 35803, None) | |
| 17/06/14 04:16:17 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-50-96.ec2.internal:35803 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:16:18 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:36258 | |
| 17/06/14 04:16:25 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 8. | |
| 17/06/14 04:16:25 INFO DAGScheduler: Executor lost: 8 (epoch 1) | |
| 17/06/14 04:16:25 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster. | |
| 17/06/14 04:16:25 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(8, ip-10-0-51-185.ec2.internal, 43229, None) | |
| 17/06/14 04:16:25 INFO BlockManagerMaster: Removed 8 successfully in removeExecutor | |
| 17/06/14 04:16:25 INFO YarnAllocator: Launching container container_1497410974650_0004_01_002656 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 04:16:25 INFO YarnAllocator: Launching container container_1497410974650_0004_01_002658 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 04:16:25 INFO YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them. | |
| 17/06/14 04:16:25 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000010 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:16:25 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:25 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:16:25 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000002 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:16:25 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:25 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:16:25 ERROR YarnClusterScheduler: Lost executor 8 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:25 WARN TaskSetManager: Lost task 165.0 in stage 1.0 (TID 304, ip-10-0-51-185.ec2.internal, executor 8): ExecutorLostFailure (executor 8 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:25 WARN TaskSetManager: Lost task 152.0 in stage 1.0 (TID 291, ip-10-0-51-185.ec2.internal, executor 8): ExecutorLostFailure (executor 8 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:25 WARN TaskSetManager: Lost task 160.0 in stage 1.0 (TID 299, ip-10-0-51-185.ec2.internal, executor 8): ExecutorLostFailure (executor 8 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:25 WARN TaskSetManager: Lost task 154.0 in stage 1.0 (TID 293, ip-10-0-51-185.ec2.internal, executor 8): ExecutorLostFailure (executor 8 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:25 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:25 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:25 INFO BlockManagerMaster: Removal of executor 8 requested | |
| 17/06/14 04:16:25 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 8 | |
| 17/06/14 04:16:25 ERROR YarnClusterScheduler: Lost executor 1 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:25 WARN TaskSetManager: Lost task 122.0 in stage 1.0 (TID 280, ip-10-0-51-185.ec2.internal, executor 1): ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:25 WARN TaskSetManager: Lost task 96.1 in stage 1.0 (TID 285, ip-10-0-51-185.ec2.internal, executor 1): ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:25 WARN TaskSetManager: Lost task 130.0 in stage 1.0 (TID 282, ip-10-0-51-185.ec2.internal, executor 1): ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:25 WARN TaskSetManager: Lost task 151.0 in stage 1.0 (TID 290, ip-10-0-51-185.ec2.internal, executor 1): ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:25 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster. | |
| 17/06/14 04:16:25 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 04:16:25 INFO DAGScheduler: Executor lost: 1 (epoch 1) | |
| 17/06/14 04:16:25 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster. | |
| 17/06/14 04:16:25 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(1, ip-10-0-51-185.ec2.internal, 32987, None) | |
| 17/06/14 04:16:25 INFO BlockManagerMaster: Removed 1 successfully in removeExecutor | |
| 17/06/14 04:16:25 INFO ExecutorAllocationManager: Existing executor 8 has been removed (new total is 14) | |
| 17/06/14 04:16:26 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 04:16:26 INFO ExecutorAllocationManager: Existing executor 1 has been removed (new total is 13) | |
| 17/06/14 04:16:26 WARN ExecutorAllocationManager: Attempted to mark unknown executor 8 idle | |
| 17/06/14 04:16:26 WARN ExecutorAllocationManager: Attempted to mark unknown executor 1 idle | |
| 17/06/14 04:16:26 WARN TransportChannelHandler: Exception in connection from /10.0.51.185:34210 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 04:16:26 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 1. | |
| 17/06/14 04:16:26 ERROR YarnClusterScheduler: Lost an executor 1 (already removed): Pending loss reason. | |
| 17/06/14 04:16:26 INFO BlockManagerMaster: Removal of executor 1 requested | |
| 17/06/14 04:16:26 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 1 | |
| 17/06/14 04:16:26 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster. | |
| 17/06/14 04:16:26 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:16:26 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:16:26 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:16:26 INFO YarnAllocator: Launching container container_1497410974650_0004_01_002662 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 04:16:26 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:16:26 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000006 on host: ip-10-0-49-157.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:16:26 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:26 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:26 ERROR YarnClusterScheduler: Lost executor 5 on ip-10-0-49-157.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:26 WARN TaskSetManager: Lost task 162.0 in stage 1.0 (TID 301, ip-10-0-49-157.ec2.internal, executor 5): ExecutorLostFailure (executor 5 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:26 WARN TaskSetManager: Lost task 153.0 in stage 1.0 (TID 292, ip-10-0-49-157.ec2.internal, executor 5): ExecutorLostFailure (executor 5 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:26 WARN TaskSetManager: Lost task 158.0 in stage 1.0 (TID 297, ip-10-0-49-157.ec2.internal, executor 5): ExecutorLostFailure (executor 5 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:26 WARN TaskSetManager: Lost task 157.0 in stage 1.0 (TID 296, ip-10-0-49-157.ec2.internal, executor 5): ExecutorLostFailure (executor 5 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:26 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:16:26 INFO DAGScheduler: Executor lost: 5 (epoch 1) | |
| 17/06/14 04:16:26 INFO BlockManagerMasterEndpoint: Trying to remove executor 5 from BlockManagerMaster. | |
| 17/06/14 04:16:26 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(5, ip-10-0-49-157.ec2.internal, 45398, None) | |
| 17/06/14 04:16:26 INFO ExecutorAllocationManager: Existing executor 5 has been removed (new total is 12) | |
| 17/06/14 04:16:26 INFO BlockManagerMaster: Removed 5 successfully in removeExecutor | |
| 17/06/14 04:16:26 WARN ExecutorAllocationManager: Attempted to mark unknown executor 5 idle | |
| 17/06/14 04:16:26 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 04:16:26 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:16:26 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal. | |
| 17/06/14 04:16:26 WARN TransportChannelHandler: Exception in connection from /10.0.49.157:44698 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 04:16:26 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 5. | |
| 17/06/14 04:16:26 ERROR YarnClusterScheduler: Lost an executor 5 (already removed): Pending loss reason. | |
| 17/06/14 04:16:26 INFO BlockManagerMaster: Removal of executor 5 requested | |
| 17/06/14 04:16:26 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 5 | |
| 17/06/14 04:16:26 INFO BlockManagerMasterEndpoint: Trying to remove executor 5 from BlockManagerMaster. | |
| 17/06/14 04:16:28 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:34928) with ID 21 | |
| 17/06/14 04:16:28 INFO TaskSetManager: Starting task 130.1 in stage 1.0 (TID 323, ip-10-0-51-185.ec2.internal, executor 21, partition 130, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:16:28 INFO TaskSetManager: Starting task 96.2 in stage 1.0 (TID 324, ip-10-0-51-185.ec2.internal, executor 21, partition 96, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:16:28 INFO TaskSetManager: Starting task 122.1 in stage 1.0 (TID 325, ip-10-0-51-185.ec2.internal, executor 21, partition 122, NODE_LOCAL, 6203 bytes) | |
| 17/06/14 04:16:28 INFO TaskSetManager: Starting task 157.1 in stage 1.0 (TID 326, ip-10-0-51-185.ec2.internal, executor 21, partition 157, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:28 INFO ExecutorAllocationManager: New executor 21 has registered (new total is 13) | |
| 17/06/14 04:16:28 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:34930) with ID 20 | |
| 17/06/14 04:16:28 INFO TaskSetManager: Starting task 158.1 in stage 1.0 (TID 327, ip-10-0-51-185.ec2.internal, executor 20, partition 158, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:28 INFO TaskSetManager: Starting task 153.1 in stage 1.0 (TID 328, ip-10-0-51-185.ec2.internal, executor 20, partition 153, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:28 INFO TaskSetManager: Starting task 162.1 in stage 1.0 (TID 329, ip-10-0-51-185.ec2.internal, executor 20, partition 162, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:28 INFO ExecutorAllocationManager: New executor 20 has registered (new total is 14) | |
| 17/06/14 04:16:28 INFO TaskSetManager: Starting task 151.1 in stage 1.0 (TID 330, ip-10-0-51-185.ec2.internal, executor 20, partition 151, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:28 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:46185 with 2.8 GB RAM, BlockManagerId(21, ip-10-0-51-185.ec2.internal, 46185, None) | |
| 17/06/14 04:16:28 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:37170 with 2.8 GB RAM, BlockManagerId(20, ip-10-0-51-185.ec2.internal, 37170, None) | |
| 17/06/14 04:16:28 INFO BlockManagerInfo: Removed rdd_2_89 on ip-10-0-51-185.ec2.internal:34392 in memory (size: 761.2 MB, free: 2.5 GB) | |
| 17/06/14 04:16:28 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-51-185.ec2.internal:46185 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:16:28 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-51-185.ec2.internal:37170 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:16:29 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:34928 | |
| 17/06/14 04:16:29 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-51-185.ec2.internal:46185 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:16:29 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 7. | |
| 17/06/14 04:16:29 INFO DAGScheduler: Executor lost: 7 (epoch 1) | |
| 17/06/14 04:16:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 7 from BlockManagerMaster. | |
| 17/06/14 04:16:29 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(7, ip-10-0-49-157.ec2.internal, 43518, None) | |
| 17/06/14 04:16:29 INFO BlockManagerMaster: Removed 7 successfully in removeExecutor | |
| 17/06/14 04:16:29 INFO YarnAllocator: Launching container container_1497410974650_0004_01_002682 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 04:16:29 INFO YarnAllocator: Launching container container_1497410974650_0004_01_002684 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 04:16:29 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:16:29 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 04:16:29 INFO YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them. | |
| 17/06/14 04:16:29 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000008 on host: ip-10-0-49-157.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:16:29 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:29 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000007 on host: ip-10-0-49-157.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:16:29 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:29 ERROR YarnClusterScheduler: Lost executor 7 on ip-10-0-49-157.ec2.internal: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:29 WARN TaskSetManager: Lost task 145.1 in stage 1.0 (TID 307, ip-10-0-49-157.ec2.internal, executor 7): ExecutorLostFailure (executor 7 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:29 WARN TaskSetManager: Lost task 161.0 in stage 1.0 (TID 300, ip-10-0-49-157.ec2.internal, executor 7): ExecutorLostFailure (executor 7 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:29 WARN TaskSetManager: Lost task 155.0 in stage 1.0 (TID 294, ip-10-0-49-157.ec2.internal, executor 7): ExecutorLostFailure (executor 7 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:29 WARN TaskSetManager: Lost task 164.0 in stage 1.0 (TID 303, ip-10-0-49-157.ec2.internal, executor 7): ExecutorLostFailure (executor 7 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:29 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:29 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:29 INFO BlockManagerMaster: Removal of executor 7 requested | |
| 17/06/14 04:16:29 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 7 | |
| 17/06/14 04:16:29 ERROR YarnClusterScheduler: Lost executor 6 on ip-10-0-49-157.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:29 WARN TaskSetManager: Lost task 142.1 in stage 1.0 (TID 306, ip-10-0-49-157.ec2.internal, executor 6): ExecutorLostFailure (executor 6 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:29 WARN TaskSetManager: Lost task 143.1 in stage 1.0 (TID 308, ip-10-0-49-157.ec2.internal, executor 6): ExecutorLostFailure (executor 6 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:29 WARN TaskSetManager: Lost task 163.0 in stage 1.0 (TID 302, ip-10-0-49-157.ec2.internal, executor 6): ExecutorLostFailure (executor 6 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:29 WARN TaskSetManager: Lost task 167.0 in stage 1.0 (TID 314, ip-10-0-49-157.ec2.internal, executor 6): ExecutorLostFailure (executor 6 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:29 INFO DAGScheduler: Executor lost: 6 (epoch 1) | |
| 17/06/14 04:16:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 7 from BlockManagerMaster. | |
| 17/06/14 04:16:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 6 from BlockManagerMaster. | |
| 17/06/14 04:16:29 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(6, ip-10-0-49-157.ec2.internal, 34052, None) | |
| 17/06/14 04:16:29 INFO BlockManagerMaster: Removed 6 successfully in removeExecutor | |
| 17/06/14 04:16:29 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:16:29 INFO ExecutorAllocationManager: Existing executor 7 has been removed (new total is 13) | |
| 17/06/14 04:16:29 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 04:16:29 INFO ExecutorAllocationManager: Existing executor 6 has been removed (new total is 12) | |
| 17/06/14 04:16:29 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:34930 | |
| 17/06/14 04:16:29 WARN TransportChannelHandler: Exception in connection from /10.0.49.157:44700 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 04:16:29 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 6. | |
| 17/06/14 04:16:29 ERROR YarnClusterScheduler: Lost an executor 6 (already removed): Pending loss reason. | |
| 17/06/14 04:16:29 INFO BlockManagerMaster: Removal of executor 6 requested | |
| 17/06/14 04:16:29 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 6 | |
| 17/06/14 04:16:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 6 from BlockManagerMaster. | |
| 17/06/14 04:16:29 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:45392) with ID 22 | |
| 17/06/14 04:16:29 INFO ExecutorAllocationManager: New executor 22 has registered (new total is 13) | |
| 17/06/14 04:16:29 INFO TaskSetManager: Starting task 167.1 in stage 1.0 (TID 331, ip-10-0-49-157.ec2.internal, executor 22, partition 167, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:29 INFO TaskSetManager: Starting task 163.1 in stage 1.0 (TID 332, ip-10-0-49-157.ec2.internal, executor 22, partition 163, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:29 INFO TaskSetManager: Starting task 143.2 in stage 1.0 (TID 333, ip-10-0-49-157.ec2.internal, executor 22, partition 143, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:29 INFO TaskSetManager: Starting task 142.2 in stage 1.0 (TID 334, ip-10-0-49-157.ec2.internal, executor 22, partition 142, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:29 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:16:29 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal. | |
| 17/06/14 04:16:29 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal. | |
| 17/06/14 04:16:29 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:44837 with 2.8 GB RAM, BlockManagerId(22, ip-10-0-49-157.ec2.internal, 44837, None) | |
| 17/06/14 04:16:30 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-157.ec2.internal:44837 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:16:30 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.157:45392 | |
| 17/06/14 04:16:32 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 3. | |
| 17/06/14 04:16:32 INFO DAGScheduler: Executor lost: 3 (epoch 1) | |
| 17/06/14 04:16:32 INFO BlockManagerMasterEndpoint: Trying to remove executor 3 from BlockManagerMaster. | |
| 17/06/14 04:16:32 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(3, ip-10-0-51-185.ec2.internal, 34392, None) | |
| 17/06/14 04:16:32 INFO YarnAllocator: Launching container container_1497410974650_0004_01_002701 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 04:16:32 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:16:32 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000004 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:16:32 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:32 INFO BlockManagerMaster: Removed 3 successfully in removeExecutor | |
| 17/06/14 04:16:32 ERROR YarnClusterScheduler: Lost executor 3 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:32 WARN TaskSetManager: Lost task 156.0 in stage 1.0 (TID 295, ip-10-0-51-185.ec2.internal, executor 3): ExecutorLostFailure (executor 3 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:32 WARN TaskSetManager: Lost task 159.0 in stage 1.0 (TID 298, ip-10-0-51-185.ec2.internal, executor 3): ExecutorLostFailure (executor 3 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:32 WARN TaskSetManager: Lost task 140.1 in stage 1.0 (TID 309, ip-10-0-51-185.ec2.internal, executor 3): ExecutorLostFailure (executor 3 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:32 WARN TaskSetManager: Lost task 166.0 in stage 1.0 (TID 305, ip-10-0-51-185.ec2.internal, executor 3): ExecutorLostFailure (executor 3 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:32 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:16:32 INFO BlockManagerMaster: Removal of executor 3 requested | |
| 17/06/14 04:16:32 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 3 | |
| 17/06/14 04:16:32 INFO BlockManagerMasterEndpoint: Trying to remove executor 3 from BlockManagerMaster. | |
| 17/06/14 04:16:32 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:16:32 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 04:16:32 INFO ExecutorAllocationManager: Existing executor 3 has been removed (new total is 12) | |
| 17/06/14 04:16:32 WARN ExecutorAllocationManager: Attempted to mark unknown executor 3 idle | |
| 17/06/14 04:16:32 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:45410) with ID 23 | |
| 17/06/14 04:16:32 INFO TaskSetManager: Starting task 166.1 in stage 1.0 (TID 335, ip-10-0-49-157.ec2.internal, executor 23, partition 166, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:32 INFO TaskSetManager: Starting task 140.2 in stage 1.0 (TID 336, ip-10-0-49-157.ec2.internal, executor 23, partition 140, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:32 INFO TaskSetManager: Starting task 159.1 in stage 1.0 (TID 337, ip-10-0-49-157.ec2.internal, executor 23, partition 159, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:32 INFO TaskSetManager: Starting task 156.1 in stage 1.0 (TID 338, ip-10-0-49-157.ec2.internal, executor 23, partition 156, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:32 INFO ExecutorAllocationManager: New executor 23 has registered (new total is 13) | |
| 17/06/14 04:16:32 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:16:32 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:16:32 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:46111 with 2.8 GB RAM, BlockManagerId(23, ip-10-0-49-157.ec2.internal, 46111, None) | |
| 17/06/14 04:16:32 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:45412) with ID 24 | |
| 17/06/14 04:16:32 INFO TaskSetManager: Starting task 164.1 in stage 1.0 (TID 339, ip-10-0-49-157.ec2.internal, executor 24, partition 164, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:32 INFO TaskSetManager: Starting task 155.1 in stage 1.0 (TID 340, ip-10-0-49-157.ec2.internal, executor 24, partition 155, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:32 INFO TaskSetManager: Starting task 161.1 in stage 1.0 (TID 341, ip-10-0-49-157.ec2.internal, executor 24, partition 161, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:32 INFO TaskSetManager: Starting task 145.2 in stage 1.0 (TID 342, ip-10-0-49-157.ec2.internal, executor 24, partition 145, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:32 INFO ExecutorAllocationManager: New executor 24 has registered (new total is 14) | |
| 17/06/14 04:16:32 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:43440 with 2.8 GB RAM, BlockManagerId(24, ip-10-0-49-157.ec2.internal, 43440, None) | |
| 17/06/14 04:16:32 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-157.ec2.internal:46111 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:16:32 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-157.ec2.internal:43440 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:16:33 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.157:45410 | |
| 17/06/14 04:16:33 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.157:45412 | |
| 17/06/14 04:16:34 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:34966) with ID 25 | |
| 17/06/14 04:16:34 INFO TaskSetManager: Starting task 154.1 in stage 1.0 (TID 343, ip-10-0-51-185.ec2.internal, executor 25, partition 154, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:34 INFO TaskSetManager: Starting task 160.1 in stage 1.0 (TID 344, ip-10-0-51-185.ec2.internal, executor 25, partition 160, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:34 INFO TaskSetManager: Starting task 152.1 in stage 1.0 (TID 345, ip-10-0-51-185.ec2.internal, executor 25, partition 152, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:34 INFO TaskSetManager: Starting task 165.1 in stage 1.0 (TID 346, ip-10-0-51-185.ec2.internal, executor 25, partition 165, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:34 INFO ExecutorAllocationManager: New executor 25 has registered (new total is 15) | |
| 17/06/14 04:16:35 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:43680 with 2.8 GB RAM, BlockManagerId(25, ip-10-0-51-185.ec2.internal, 43680, None) | |
| 17/06/14 04:16:35 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-51-185.ec2.internal:43680 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:16:35 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:34966 | |
| 17/06/14 04:16:42 INFO TaskSetManager: Starting task 172.0 in stage 1.0 (TID 347, ip-10-0-49-229.ec2.internal, executor 10, partition 172, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:42 INFO TaskSetManager: Finished task 97.0 in stage 1.0 (TID 250) in 73589 ms on ip-10-0-49-229.ec2.internal (executor 10) (113/270) | |
| 17/06/14 04:16:42 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.229:48778 | |
| 17/06/14 04:16:44 INFO TaskSetManager: Starting task 173.0 in stage 1.0 (TID 348, ip-10-0-49-229.ec2.internal, executor 9, partition 173, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:44 INFO TaskSetManager: Finished task 110.0 in stage 1.0 (TID 257) in 71486 ms on ip-10-0-49-229.ec2.internal (executor 9) (114/270) | |
| 17/06/14 04:16:44 INFO YarnAllocator: Driver requested a total number of 39 executor(s). | |
| 17/06/14 04:16:44 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 39 executors. | |
| 17/06/14 04:16:44 INFO TaskSetManager: Starting task 174.0 in stage 1.0 (TID 349, ip-10-0-49-229.ec2.internal, executor 9, partition 174, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:44 INFO TaskSetManager: Finished task 127.0 in stage 1.0 (TID 259) in 69741 ms on ip-10-0-49-229.ec2.internal (executor 9) (115/270) | |
| 17/06/14 04:16:44 INFO TaskSetManager: Starting task 175.0 in stage 1.0 (TID 350, ip-10-0-49-229.ec2.internal, executor 10, partition 175, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:44 INFO TaskSetManager: Finished task 90.0 in stage 1.0 (TID 249) in 75911 ms on ip-10-0-49-229.ec2.internal (executor 10) (116/270) | |
| 17/06/14 04:16:47 INFO BlockManagerInfo: Removed rdd_2_54 on ip-10-0-49-229.ec2.internal:44338 in memory (size: 811.5 MB, free: 1992.9 MB) | |
| 17/06/14 04:16:47 INFO TaskSetManager: Starting task 176.0 in stage 1.0 (TID 351, ip-10-0-49-229.ec2.internal, executor 10, partition 176, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:47 INFO TaskSetManager: Finished task 102.0 in stage 1.0 (TID 253) in 76317 ms on ip-10-0-49-229.ec2.internal (executor 10) (117/270) | |
| 17/06/14 04:16:47 INFO TaskSetManager: Starting task 177.0 in stage 1.0 (TID 352, ip-10-0-49-229.ec2.internal, executor 10, partition 177, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:47 INFO TaskSetManager: Finished task 101.0 in stage 1.0 (TID 251) in 77678 ms on ip-10-0-49-229.ec2.internal (executor 10) (118/270) | |
| 17/06/14 04:16:47 INFO YarnAllocator: Driver requested a total number of 38 executor(s). | |
| 17/06/14 04:16:47 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 38 executors. | |
| 17/06/14 04:16:47 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-49-229.ec2.internal:46261 (size: 26.1 KB) | |
| 17/06/14 04:16:47 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-49-229.ec2.internal:46261 (size: 33.3 KB) | |
| 17/06/14 04:16:47 INFO BlockManagerInfo: Removed rdd_2_21 on ip-10-0-49-229.ec2.internal:46261 in memory (size: 778.4 MB, free: 1190.7 MB) | |
| 17/06/14 04:16:49 INFO TaskSetManager: Starting task 178.0 in stage 1.0 (TID 353, ip-10-0-49-229.ec2.internal, executor 9, partition 178, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:49 INFO TaskSetManager: Finished task 133.0 in stage 1.0 (TID 260) in 74436 ms on ip-10-0-49-229.ec2.internal (executor 9) (119/270) | |
| 17/06/14 04:16:50 INFO TaskSetManager: Starting task 179.0 in stage 1.0 (TID 354, ip-10-0-49-229.ec2.internal, executor 11, partition 179, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:50 INFO TaskSetManager: Finished task 103.0 in stage 1.0 (TID 254) in 78706 ms on ip-10-0-49-229.ec2.internal (executor 11) (120/270) | |
| 17/06/14 04:16:50 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.229:48780 | |
| 17/06/14 04:16:51 INFO TaskSetManager: Starting task 180.0 in stage 1.0 (TID 355, ip-10-0-49-229.ec2.internal, executor 11, partition 180, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:51 INFO TaskSetManager: Finished task 104.0 in stage 1.0 (TID 255) in 78774 ms on ip-10-0-49-229.ec2.internal (executor 11) (121/270) | |
| 17/06/14 04:16:51 INFO BlockManagerInfo: Removed rdd_2_88 on ip-10-0-49-229.ec2.internal:46261 in memory (size: 783.8 MB, free: 1974.5 MB) | |
| 17/06/14 04:16:52 INFO TaskSetManager: Starting task 181.0 in stage 1.0 (TID 356, ip-10-0-49-229.ec2.internal, executor 11, partition 181, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:52 INFO TaskSetManager: Finished task 126.0 in stage 1.0 (TID 258) in 78467 ms on ip-10-0-49-229.ec2.internal (executor 11) (122/270) | |
| 17/06/14 04:16:52 INFO YarnAllocator: Driver requested a total number of 37 executor(s). | |
| 17/06/14 04:16:52 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 37 executors. | |
| 17/06/14 04:16:53 INFO TaskSetManager: Starting task 182.0 in stage 1.0 (TID 357, ip-10-0-49-229.ec2.internal, executor 11, partition 182, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:16:53 INFO TaskSetManager: Finished task 105.0 in stage 1.0 (TID 256) in 80393 ms on ip-10-0-49-229.ec2.internal (executor 11) (123/270) | |
| 17/06/14 04:16:54 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-49-229.ec2.internal:37760 (size: 26.1 KB) | |
| 17/06/14 04:16:54 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-49-229.ec2.internal:37760 (size: 33.3 KB) | |
| 17/06/14 04:16:54 INFO BlockManagerInfo: Removed rdd_2_43 on ip-10-0-49-229.ec2.internal:37760 in memory (size: 759.2 MB, free: 1519.3 MB) | |
| 17/06/14 04:17:01 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 10. | |
| 17/06/14 04:17:01 INFO DAGScheduler: Executor lost: 10 (epoch 1) | |
| 17/06/14 04:17:01 INFO BlockManagerMasterEndpoint: Trying to remove executor 10 from BlockManagerMaster. | |
| 17/06/14 04:17:01 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(10, ip-10-0-49-229.ec2.internal, 46261, None) | |
| 17/06/14 04:17:01 INFO BlockManagerMaster: Removed 10 successfully in removeExecutor | |
| 17/06/14 04:17:01 INFO YarnAllocator: Launching container container_1497410974650_0004_01_001503 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 04:17:01 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:17:01 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000013 on host: ip-10-0-49-229.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:17:01 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:01 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:01 ERROR YarnClusterScheduler: Lost executor 10 on ip-10-0-49-229.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:01 WARN TaskSetManager: Lost task 177.0 in stage 1.0 (TID 352, ip-10-0-49-229.ec2.internal, executor 10): ExecutorLostFailure (executor 10 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:01 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:17:01 WARN TaskSetManager: Lost task 176.0 in stage 1.0 (TID 351, ip-10-0-49-229.ec2.internal, executor 10): ExecutorLostFailure (executor 10 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:01 WARN TaskSetManager: Lost task 175.0 in stage 1.0 (TID 350, ip-10-0-49-229.ec2.internal, executor 10): ExecutorLostFailure (executor 10 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:01 WARN TaskSetManager: Lost task 172.0 in stage 1.0 (TID 347, ip-10-0-49-229.ec2.internal, executor 10): ExecutorLostFailure (executor 10 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:01 INFO BlockManagerMaster: Removal of executor 10 requested | |
| 17/06/14 04:17:01 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 10 | |
| 17/06/14 04:17:01 INFO BlockManagerMasterEndpoint: Trying to remove executor 10 from BlockManagerMaster. | |
| 17/06/14 04:17:01 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 04:17:01 INFO ExecutorAllocationManager: Existing executor 10 has been removed (new total is 14) | |
| 17/06/14 04:17:01 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:17:01 INFO YarnAllocator: Canceled 2 unlocalized container requests to resubmit with locality | |
| 17/06/14 04:17:01 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:17:01 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:17:01 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:17:03 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:49416) with ID 26 | |
| 17/06/14 04:17:03 INFO TaskSetManager: Starting task 172.1 in stage 1.0 (TID 358, ip-10-0-49-229.ec2.internal, executor 26, partition 172, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:03 INFO TaskSetManager: Starting task 175.1 in stage 1.0 (TID 359, ip-10-0-49-229.ec2.internal, executor 26, partition 175, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:03 INFO TaskSetManager: Starting task 176.1 in stage 1.0 (TID 360, ip-10-0-49-229.ec2.internal, executor 26, partition 176, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:03 INFO TaskSetManager: Starting task 177.1 in stage 1.0 (TID 361, ip-10-0-49-229.ec2.internal, executor 26, partition 177, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:03 INFO ExecutorAllocationManager: New executor 26 has registered (new total is 15) | |
| 17/06/14 04:17:03 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:42403 with 2.8 GB RAM, BlockManagerId(26, ip-10-0-49-229.ec2.internal, 42403, None) | |
| 17/06/14 04:17:03 INFO BlockManagerInfo: Removed rdd_2_106 on ip-10-0-49-229.ec2.internal:37760 in memory (size: 206.0 MB, free: 1725.4 MB) | |
| 17/06/14 04:17:03 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-229.ec2.internal:42403 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:17:03 INFO TaskSetManager: Starting task 183.0 in stage 1.0 (TID 362, ip-10-0-49-157.ec2.internal, executor 4, partition 183, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:03 INFO TaskSetManager: Finished task 113.0 in stage 1.0 (TID 277) in 67793 ms on ip-10-0-49-157.ec2.internal (executor 4) (124/270) | |
| 17/06/14 04:17:04 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.157:44704 | |
| 17/06/14 04:17:04 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.229:49416 | |
| 17/06/14 04:17:05 INFO TaskSetManager: Starting task 184.0 in stage 1.0 (TID 363, ip-10-0-49-157.ec2.internal, executor 4, partition 184, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:05 INFO TaskSetManager: Finished task 91.0 in stage 1.0 (TID 265) in 74370 ms on ip-10-0-49-157.ec2.internal (executor 4) (125/270) | |
| 17/06/14 04:17:06 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-49-157.ec2.internal:34337 (size: 26.1 KB) | |
| 17/06/14 04:17:06 INFO BlockManagerInfo: Removed rdd_2_115 on ip-10-0-49-157.ec2.internal:34337 in memory (size: 156.1 MB, free: 305.9 MB) | |
| 17/06/14 04:17:07 INFO YarnAllocator: Launching container container_1497410974650_0004_01_002955 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 04:17:07 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:17:07 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000014 on host: ip-10-0-49-229.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:17:07 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:07 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:07 ERROR YarnClusterScheduler: Lost executor 11 on ip-10-0-49-229.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:07 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:17:07 WARN TaskSetManager: Lost task 180.0 in stage 1.0 (TID 355, ip-10-0-49-229.ec2.internal, executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:07 WARN TaskSetManager: Lost task 179.0 in stage 1.0 (TID 354, ip-10-0-49-229.ec2.internal, executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:07 WARN TaskSetManager: Lost task 182.0 in stage 1.0 (TID 357, ip-10-0-49-229.ec2.internal, executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:07 WARN TaskSetManager: Lost task 181.0 in stage 1.0 (TID 356, ip-10-0-49-229.ec2.internal, executor 11): ExecutorLostFailure (executor 11 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:07 INFO DAGScheduler: Executor lost: 11 (epoch 1) | |
| 17/06/14 04:17:07 INFO BlockManagerMasterEndpoint: Trying to remove executor 11 from BlockManagerMaster. | |
| 17/06/14 04:17:07 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(11, ip-10-0-49-229.ec2.internal, 37760, None) | |
| 17/06/14 04:17:07 INFO BlockManagerMaster: Removed 11 successfully in removeExecutor | |
| 17/06/14 04:17:07 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 04:17:07 INFO ExecutorAllocationManager: Existing executor 11 has been removed (new total is 14) | |
| 17/06/14 04:17:07 WARN ExecutorAllocationManager: Attempted to mark unknown executor 11 idle | |
| 17/06/14 04:17:07 WARN TransportChannelHandler: Exception in connection from /10.0.49.229:48780 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 04:17:07 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 11. | |
| 17/06/14 04:17:07 ERROR YarnClusterScheduler: Lost an executor 11 (already removed): Pending loss reason. | |
| 17/06/14 04:17:07 INFO BlockManagerMaster: Removal of executor 11 requested | |
| 17/06/14 04:17:07 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 11 | |
| 17/06/14 04:17:07 INFO BlockManagerMasterEndpoint: Trying to remove executor 11 from BlockManagerMaster. | |
| 17/06/14 04:17:07 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 9. | |
| 17/06/14 04:17:07 INFO DAGScheduler: Executor lost: 9 (epoch 1) | |
| 17/06/14 04:17:07 INFO BlockManagerMasterEndpoint: Trying to remove executor 9 from BlockManagerMaster. | |
| 17/06/14 04:17:07 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(9, ip-10-0-49-229.ec2.internal, 44338, None) | |
| 17/06/14 04:17:07 INFO BlockManagerMaster: Removed 9 successfully in removeExecutor | |
| 17/06/14 04:17:07 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:17:07 INFO YarnAllocator: Canceled 1 unlocalized container requests to resubmit with locality | |
| 17/06/14 04:17:07 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal. | |
| 17/06/14 04:17:07 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal. | |
| 17/06/14 04:17:07 INFO TaskSetManager: Starting task 181.1 in stage 1.0 (TID 364, ip-10-0-49-157.ec2.internal, executor 4, partition 181, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:07 INFO TaskSetManager: Finished task 93.0 in stage 1.0 (TID 266) in 75833 ms on ip-10-0-49-157.ec2.internal (executor 4) (126/270) | |
| 17/06/14 04:17:07 INFO YarnAllocator: Driver requested a total number of 36 executor(s). | |
| 17/06/14 04:17:07 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 36 executors. | |
| 17/06/14 04:17:07 INFO BlockManagerInfo: Removed rdd_2_29 on ip-10-0-49-157.ec2.internal:34337 in memory (size: 865.0 MB, free: 1171.0 MB) | |
| 17/06/14 04:17:08 INFO YarnAllocator: Launching container container_1497410974650_0004_01_003005 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 04:17:08 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:17:08 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000012 on host: ip-10-0-49-229.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:17:08 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:08 ERROR YarnClusterScheduler: Lost executor 9 on ip-10-0-49-229.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:08 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:17:08 WARN TaskSetManager: Lost task 174.0 in stage 1.0 (TID 349, ip-10-0-49-229.ec2.internal, executor 9): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:08 WARN TaskSetManager: Lost task 173.0 in stage 1.0 (TID 348, ip-10-0-49-229.ec2.internal, executor 9): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:08 WARN TaskSetManager: Lost task 136.0 in stage 1.0 (TID 261, ip-10-0-49-229.ec2.internal, executor 9): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:08 WARN TaskSetManager: Lost task 178.0 in stage 1.0 (TID 353, ip-10-0-49-229.ec2.internal, executor 9): ExecutorLostFailure (executor 9 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:08 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:08 INFO BlockManagerMaster: Removal of executor 9 requested | |
| 17/06/14 04:17:08 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 9 | |
| 17/06/14 04:17:08 INFO BlockManagerMasterEndpoint: Trying to remove executor 9 from BlockManagerMaster. | |
| 17/06/14 04:17:08 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 04:17:08 INFO ExecutorAllocationManager: Existing executor 9 has been removed (new total is 13) | |
| 17/06/14 04:17:08 WARN ExecutorAllocationManager: Attempted to mark unknown executor 9 idle | |
| 17/06/14 04:17:08 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:17:08 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal. | |
| 17/06/14 04:17:09 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:49436) with ID 27 | |
| 17/06/14 04:17:09 INFO TaskSetManager: Starting task 178.1 in stage 1.0 (TID 365, ip-10-0-49-229.ec2.internal, executor 27, partition 178, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:09 INFO TaskSetManager: Starting task 136.1 in stage 1.0 (TID 366, ip-10-0-49-229.ec2.internal, executor 27, partition 136, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:09 INFO TaskSetManager: Starting task 173.1 in stage 1.0 (TID 367, ip-10-0-49-229.ec2.internal, executor 27, partition 173, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:09 INFO TaskSetManager: Starting task 174.1 in stage 1.0 (TID 368, ip-10-0-49-229.ec2.internal, executor 27, partition 174, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:09 INFO ExecutorAllocationManager: New executor 27 has registered (new total is 14) | |
| 17/06/14 04:17:09 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:33294 with 2.8 GB RAM, BlockManagerId(27, ip-10-0-49-229.ec2.internal, 33294, None) | |
| 17/06/14 04:17:09 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-229.ec2.internal:33294 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:17:10 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.229:49436 | |
| 17/06/14 04:17:10 INFO TaskSetManager: Starting task 182.1 in stage 1.0 (TID 369, ip-10-0-51-185.ec2.internal, executor 2, partition 182, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:10 INFO TaskSetManager: Finished task 117.0 in stage 1.0 (TID 278) in 73817 ms on ip-10-0-51-185.ec2.internal (executor 2) (127/270) | |
| 17/06/14 04:17:10 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:34216 | |
| 17/06/14 04:17:10 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:49444) with ID 28 | |
| 17/06/14 04:17:10 INFO TaskSetManager: Starting task 179.1 in stage 1.0 (TID 370, ip-10-0-49-229.ec2.internal, executor 28, partition 179, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:10 INFO TaskSetManager: Starting task 180.1 in stage 1.0 (TID 371, ip-10-0-49-229.ec2.internal, executor 28, partition 180, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:10 INFO TaskSetManager: Starting task 185.0 in stage 1.0 (TID 372, ip-10-0-49-229.ec2.internal, executor 28, partition 185, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:10 INFO TaskSetManager: Starting task 186.0 in stage 1.0 (TID 373, ip-10-0-49-229.ec2.internal, executor 28, partition 186, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:10 INFO ExecutorAllocationManager: New executor 28 has registered (new total is 15) | |
| 17/06/14 04:17:10 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:37408 with 2.8 GB RAM, BlockManagerId(28, ip-10-0-49-229.ec2.internal, 37408, None) | |
| 17/06/14 04:17:10 INFO BlockManagerInfo: Removed rdd_2_120 on ip-10-0-49-157.ec2.internal:34337 in memory (size: 815.9 MB, free: 1986.8 MB) | |
| 17/06/14 04:17:10 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-229.ec2.internal:37408 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:17:11 INFO TaskSetManager: Starting task 187.0 in stage 1.0 (TID 374, ip-10-0-49-157.ec2.internal, executor 4, partition 187, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:11 INFO TaskSetManager: Finished task 95.0 in stage 1.0 (TID 271) in 76788 ms on ip-10-0-49-157.ec2.internal (executor 4) (128/270) | |
| 17/06/14 04:17:11 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.229:49444 | |
| 17/06/14 04:17:13 INFO TaskSetManager: Starting task 188.0 in stage 1.0 (TID 375, ip-10-0-51-185.ec2.internal, executor 2, partition 188, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:13 INFO TaskSetManager: Finished task 109.0 in stage 1.0 (TID 274) in 77975 ms on ip-10-0-51-185.ec2.internal (executor 2) (129/270) | |
| 17/06/14 04:17:13 INFO TaskSetManager: Starting task 189.0 in stage 1.0 (TID 376, ip-10-0-51-185.ec2.internal, executor 2, partition 189, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:13 INFO TaskSetManager: Finished task 128.0 in stage 1.0 (TID 281) in 77266 ms on ip-10-0-51-185.ec2.internal (executor 2) (130/270) | |
| 17/06/14 04:17:13 INFO YarnAllocator: Driver requested a total number of 35 executor(s). | |
| 17/06/14 04:17:13 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 35 executors. | |
| 17/06/14 04:17:14 INFO BlockManagerInfo: Added broadcast_0_piece0 on disk on ip-10-0-51-185.ec2.internal:36390 (size: 26.1 KB) | |
| 17/06/14 04:17:14 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-51-185.ec2.internal:36390 (size: 33.3 KB) | |
| 17/06/14 04:17:14 INFO BlockManagerInfo: Removed rdd_2_116 on ip-10-0-51-185.ec2.internal:36390 in memory (size: 738.8 MB, free: 1010.8 MB) | |
| 17/06/14 04:17:16 INFO BlockManagerInfo: Removed rdd_2_41 on ip-10-0-51-185.ec2.internal:36390 in memory (size: 800.6 MB, free: 1811.3 MB) | |
| 17/06/14 04:17:17 INFO TaskSetManager: Starting task 190.0 in stage 1.0 (TID 377, ip-10-0-51-185.ec2.internal, executor 2, partition 190, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:17 INFO TaskSetManager: Finished task 111.0 in stage 1.0 (TID 275) in 81546 ms on ip-10-0-51-185.ec2.internal (executor 2) (131/270) | |
| 17/06/14 04:17:21 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 4. | |
| 17/06/14 04:17:21 INFO DAGScheduler: Executor lost: 4 (epoch 1) | |
| 17/06/14 04:17:21 INFO BlockManagerMasterEndpoint: Trying to remove executor 4 from BlockManagerMaster. | |
| 17/06/14 04:17:21 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(4, ip-10-0-49-157.ec2.internal, 34337, None) | |
| 17/06/14 04:17:21 INFO BlockManagerMaster: Removed 4 successfully in removeExecutor | |
| 17/06/14 04:17:21 INFO YarnAllocator: Launching container container_1497410974650_0004_01_003122 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 04:17:21 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:17:21 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000005 on host: ip-10-0-49-157.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:17:21 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:21 ERROR YarnClusterScheduler: Lost executor 4 on ip-10-0-49-157.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:21 WARN TaskSetManager: Lost task 181.1 in stage 1.0 (TID 364, ip-10-0-49-157.ec2.internal, executor 4): ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:21 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:17:21 WARN TaskSetManager: Lost task 184.0 in stage 1.0 (TID 363, ip-10-0-49-157.ec2.internal, executor 4): ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:21 WARN TaskSetManager: Lost task 183.0 in stage 1.0 (TID 362, ip-10-0-49-157.ec2.internal, executor 4): ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:21 WARN TaskSetManager: Lost task 187.0 in stage 1.0 (TID 374, ip-10-0-49-157.ec2.internal, executor 4): ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:21 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:21 INFO BlockManagerMaster: Removal of executor 4 requested | |
| 17/06/14 04:17:21 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 4 | |
| 17/06/14 04:17:21 INFO BlockManagerMasterEndpoint: Trying to remove executor 4 from BlockManagerMaster. | |
| 17/06/14 04:17:21 INFO ExecutorAllocationManager: Existing executor 4 has been removed (new total is 14) | |
| 17/06/14 04:17:21 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 04:17:21 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:17:21 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:17:23 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:45518) with ID 29 | |
| 17/06/14 04:17:23 INFO TaskSetManager: Starting task 187.1 in stage 1.0 (TID 378, ip-10-0-49-157.ec2.internal, executor 29, partition 187, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:23 INFO TaskSetManager: Starting task 183.1 in stage 1.0 (TID 379, ip-10-0-49-157.ec2.internal, executor 29, partition 183, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:23 INFO TaskSetManager: Starting task 184.1 in stage 1.0 (TID 380, ip-10-0-49-157.ec2.internal, executor 29, partition 184, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:23 INFO TaskSetManager: Starting task 181.2 in stage 1.0 (TID 381, ip-10-0-49-157.ec2.internal, executor 29, partition 181, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:23 INFO ExecutorAllocationManager: New executor 29 has registered (new total is 15) | |
| 17/06/14 04:17:23 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:46402 with 2.8 GB RAM, BlockManagerId(29, ip-10-0-49-157.ec2.internal, 46402, None) | |
| 17/06/14 04:17:23 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-157.ec2.internal:46402 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:17:23 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.157:45518 | |
| 17/06/14 04:17:26 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 2. | |
| 17/06/14 04:17:26 INFO DAGScheduler: Executor lost: 2 (epoch 1) | |
| 17/06/14 04:17:26 INFO BlockManagerMasterEndpoint: Trying to remove executor 2 from BlockManagerMaster. | |
| 17/06/14 04:17:26 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(2, ip-10-0-51-185.ec2.internal, 36390, None) | |
| 17/06/14 04:17:26 INFO BlockManagerMaster: Removed 2 successfully in removeExecutor | |
| 17/06/14 04:17:26 INFO YarnAllocator: Launching container container_1497410974650_0004_01_003163 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 04:17:26 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:17:26 INFO YarnAllocator: Completed container container_1497410974650_0004_01_000003 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:17:26 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:26 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:26 ERROR YarnClusterScheduler: Lost executor 2 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:26 WARN TaskSetManager: Lost task 189.0 in stage 1.0 (TID 376, ip-10-0-51-185.ec2.internal, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:26 WARN TaskSetManager: Lost task 182.1 in stage 1.0 (TID 369, ip-10-0-51-185.ec2.internal, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:26 WARN TaskSetManager: Lost task 188.0 in stage 1.0 (TID 375, ip-10-0-51-185.ec2.internal, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:26 WARN TaskSetManager: Lost task 190.0 in stage 1.0 (TID 377, ip-10-0-51-185.ec2.internal, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:17:26 INFO BlockManagerMaster: Removal of executor 2 requested | |
| 17/06/14 04:17:26 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 2 | |
| 17/06/14 04:17:26 INFO BlockManagerMasterEndpoint: Trying to remove executor 2 from BlockManagerMaster. | |
| 17/06/14 04:17:26 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:17:26 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 04:17:26 INFO ExecutorAllocationManager: Existing executor 2 has been removed (new total is 14) | |
| 17/06/14 04:17:26 WARN ExecutorAllocationManager: Attempted to mark unknown executor 2 idle | |
| 17/06/14 04:17:27 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-10-0-50-96.ec2.internal:41603 (size: 26.1 KB, free: 2.8 GB) | |
| 17/06/14 04:17:27 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:17:27 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:17:29 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:35094) with ID 30 | |
| 17/06/14 04:17:29 INFO TaskSetManager: Starting task 190.1 in stage 1.0 (TID 382, ip-10-0-51-185.ec2.internal, executor 30, partition 190, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:29 INFO TaskSetManager: Starting task 188.1 in stage 1.0 (TID 383, ip-10-0-51-185.ec2.internal, executor 30, partition 188, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:29 INFO TaskSetManager: Starting task 182.2 in stage 1.0 (TID 384, ip-10-0-51-185.ec2.internal, executor 30, partition 182, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:29 INFO TaskSetManager: Starting task 189.1 in stage 1.0 (TID 385, ip-10-0-51-185.ec2.internal, executor 30, partition 189, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:29 INFO ExecutorAllocationManager: New executor 30 has registered (new total is 15) | |
| 17/06/14 04:17:29 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:46768 with 2.8 GB RAM, BlockManagerId(30, ip-10-0-51-185.ec2.internal, 46768, None) | |
| 17/06/14 04:17:29 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-51-185.ec2.internal:46768 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:17:29 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:35094 | |
| 17/06/14 04:17:48 INFO BlockManagerInfo: Added rdd_2_96 in memory on ip-10-0-51-185.ec2.internal:46185 (size: 805.3 MB, free: 2021.7 MB) | |
| 17/06/14 04:17:48 INFO TaskSetManager: Starting task 191.0 in stage 1.0 (TID 386, ip-10-0-51-185.ec2.internal, executor 21, partition 191, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:48 INFO TaskSetManager: Finished task 122.1 in stage 1.0 (TID 325) in 80172 ms on ip-10-0-51-185.ec2.internal (executor 21) (132/270) | |
| 17/06/14 04:17:48 INFO TaskSetManager: Starting task 192.0 in stage 1.0 (TID 387, ip-10-0-51-185.ec2.internal, executor 21, partition 192, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:48 INFO TaskSetManager: Finished task 96.2 in stage 1.0 (TID 324) in 80415 ms on ip-10-0-51-185.ec2.internal (executor 21) (133/270) | |
| 17/06/14 04:17:49 INFO TaskSetManager: Starting task 193.0 in stage 1.0 (TID 388, ip-10-0-51-185.ec2.internal, executor 21, partition 193, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:17:49 INFO TaskSetManager: Finished task 130.1 in stage 1.0 (TID 323) in 80881 ms on ip-10-0-51-185.ec2.internal (executor 21) (134/270) | |
| 17/06/14 04:17:49 INFO YarnAllocator: Driver requested a total number of 34 executor(s). | |
| 17/06/14 04:17:49 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 34 executors. | |
| 17/06/14 04:18:05 INFO TaskSetManager: Starting task 194.0 in stage 1.0 (TID 389, ip-10-0-50-96.ec2.internal, executor 16, partition 194, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:18:05 INFO TaskSetManager: Finished task 82.1 in stage 1.0 (TID 286) in 124555 ms on ip-10-0-50-96.ec2.internal (executor 16) (135/270) | |
| 17/06/14 04:18:12 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 21. | |
| 17/06/14 04:18:12 INFO DAGScheduler: Executor lost: 21 (epoch 1) | |
| 17/06/14 04:18:12 INFO BlockManagerMasterEndpoint: Trying to remove executor 21 from BlockManagerMaster. | |
| 17/06/14 04:18:12 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(21, ip-10-0-51-185.ec2.internal, 46185, None) | |
| 17/06/14 04:18:12 INFO BlockManagerMaster: Removed 21 successfully in removeExecutor | |
| 17/06/14 04:18:12 INFO YarnAllocator: Launching container container_1497410974650_0004_01_003564 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 04:18:12 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:18:12 INFO YarnAllocator: Completed container container_1497410974650_0004_01_002658 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:18:12 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:18:12 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:18:12 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:18:12 ERROR YarnClusterScheduler: Lost executor 21 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:18:12 WARN TaskSetManager: Lost task 193.0 in stage 1.0 (TID 388, ip-10-0-51-185.ec2.internal, executor 21): ExecutorLostFailure (executor 21 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:18:12 WARN TaskSetManager: Lost task 192.0 in stage 1.0 (TID 387, ip-10-0-51-185.ec2.internal, executor 21): ExecutorLostFailure (executor 21 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:18:12 WARN TaskSetManager: Lost task 191.0 in stage 1.0 (TID 386, ip-10-0-51-185.ec2.internal, executor 21): ExecutorLostFailure (executor 21 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:18:12 WARN TaskSetManager: Lost task 157.1 in stage 1.0 (TID 326, ip-10-0-51-185.ec2.internal, executor 21): ExecutorLostFailure (executor 21 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:18:12 INFO BlockManagerMaster: Removal of executor 21 requested | |
| 17/06/14 04:18:12 INFO BlockManagerMasterEndpoint: Trying to remove executor 21 from BlockManagerMaster. | |
| 17/06/14 04:18:12 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 21 | |
| 17/06/14 04:18:12 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 04:18:12 INFO ExecutorAllocationManager: Existing executor 21 has been removed (new total is 14) | |
| 17/06/14 04:18:12 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:18:12 INFO YarnAllocator: Submitted container request for host ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:18:14 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:35188) with ID 31 | |
| 17/06/14 04:18:14 INFO ExecutorAllocationManager: New executor 31 has registered (new total is 15) | |
| 17/06/14 04:18:14 INFO TaskSetManager: Starting task 157.2 in stage 1.0 (TID 390, ip-10-0-51-185.ec2.internal, executor 31, partition 157, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:18:14 INFO TaskSetManager: Starting task 191.1 in stage 1.0 (TID 391, ip-10-0-51-185.ec2.internal, executor 31, partition 191, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:18:14 INFO TaskSetManager: Starting task 192.1 in stage 1.0 (TID 392, ip-10-0-51-185.ec2.internal, executor 31, partition 192, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:18:14 INFO TaskSetManager: Starting task 193.1 in stage 1.0 (TID 393, ip-10-0-51-185.ec2.internal, executor 31, partition 193, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:18:14 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:35857 with 2.8 GB RAM, BlockManagerId(31, ip-10-0-51-185.ec2.internal, 35857, None) | |
| 17/06/14 04:18:14 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-51-185.ec2.internal:35857 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:18:14 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:35188 | |
| 17/06/14 04:30:25 INFO TaskSetManager: Starting task 195.0 in stage 1.0 (TID 394, ip-10-0-50-96.ec2.internal, executor 19, partition 195, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:30:25 INFO TaskSetManager: Finished task 148.1 in stage 1.0 (TID 322) in 848389 ms on ip-10-0-50-96.ec2.internal (executor 19) (136/270) | |
| 17/06/14 04:30:29 INFO TaskSetManager: Starting task 196.0 in stage 1.0 (TID 395, ip-10-0-50-96.ec2.internal, executor 16, partition 196, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:30:29 INFO TaskSetManager: Finished task 150.0 in stage 1.0 (TID 289) in 868564 ms on ip-10-0-50-96.ec2.internal (executor 16) (137/270) | |
| 17/06/14 04:30:38 INFO BlockManagerInfo: Added rdd_10_4 in memory on ip-10-0-50-96.ec2.internal:35803 (size: 841.0 MB, free: 1986.1 MB) | |
| 17/06/14 04:30:39 INFO TaskSetManager: Starting task 197.0 in stage 1.0 (TID 396, ip-10-0-50-96.ec2.internal, executor 19, partition 197, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:30:39 INFO TaskSetManager: Finished task 139.1 in stage 1.0 (TID 319) in 861821 ms on ip-10-0-50-96.ec2.internal (executor 19) (138/270) | |
| 17/06/14 04:30:39 INFO YarnAllocator: Driver requested a total number of 33 executor(s). | |
| 17/06/14 04:30:39 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 33 executors. | |
| 17/06/14 04:30:41 INFO TaskSetManager: Starting task 198.0 in stage 1.0 (TID 397, ip-10-0-50-96.ec2.internal, executor 19, partition 198, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:30:41 INFO TaskSetManager: Finished task 141.1 in stage 1.0 (TID 321) in 864017 ms on ip-10-0-50-96.ec2.internal (executor 19) (139/270) | |
| 17/06/14 04:30:42 INFO BlockManagerInfo: Added rdd_10_18 in memory on ip-10-0-51-185.ec2.internal:37170 (size: 849.0 MB, free: 1978.1 MB) | |
| 17/06/14 04:30:42 INFO TaskSetManager: Starting task 199.0 in stage 1.0 (TID 398, ip-10-0-51-185.ec2.internal, executor 20, partition 199, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:30:42 INFO TaskSetManager: Finished task 153.1 in stage 1.0 (TID 328) in 854420 ms on ip-10-0-51-185.ec2.internal (executor 20) (140/270) | |
| 17/06/14 04:30:49 INFO TaskSetManager: Starting task 200.0 in stage 1.0 (TID 399, ip-10-0-51-185.ec2.internal, executor 20, partition 200, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:30:49 INFO TaskSetManager: Finished task 162.1 in stage 1.0 (TID 329) in 860748 ms on ip-10-0-51-185.ec2.internal (executor 20) (141/270) | |
| 17/06/14 04:30:52 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 16. | |
| 17/06/14 04:30:52 INFO DAGScheduler: Executor lost: 16 (epoch 1) | |
| 17/06/14 04:30:52 INFO BlockManagerMasterEndpoint: Trying to remove executor 16 from BlockManagerMaster. | |
| 17/06/14 04:30:52 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(16, ip-10-0-50-96.ec2.internal, 41603, None) | |
| 17/06/14 04:30:52 INFO BlockManagerMaster: Removed 16 successfully in removeExecutor | |
| 17/06/14 04:30:52 INFO YarnAllocator: Launching container container_1497410974650_0004_01_010388 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 04:30:52 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:30:52 INFO YarnAllocator: Completed container container_1497410974650_0004_01_002432 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:30:52 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:30:52 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:30:52 ERROR YarnClusterScheduler: Lost executor 16 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:30:52 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:30:52 WARN TaskSetManager: Lost task 137.1 in stage 1.0 (TID 288, ip-10-0-50-96.ec2.internal, executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:30:52 WARN TaskSetManager: Lost task 196.0 in stage 1.0 (TID 395, ip-10-0-50-96.ec2.internal, executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:30:52 WARN TaskSetManager: Lost task 194.0 in stage 1.0 (TID 389, ip-10-0-50-96.ec2.internal, executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:30:52 WARN TaskSetManager: Lost task 135.1 in stage 1.0 (TID 287, ip-10-0-50-96.ec2.internal, executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:30:52 INFO BlockManagerMaster: Removal of executor 16 requested | |
| 17/06/14 04:30:52 INFO BlockManagerMasterEndpoint: Trying to remove executor 16 from BlockManagerMaster. | |
| 17/06/14 04:30:52 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 16 | |
| 17/06/14 04:30:52 INFO ExecutorAllocationManager: Existing executor 16 has been removed (new total is 14) | |
| 17/06/14 04:30:52 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 04:30:52 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:30:52 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:30:52 INFO TaskSetManager: Starting task 135.2 in stage 1.0 (TID 400, ip-10-0-51-185.ec2.internal, executor 20, partition 135, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:30:52 INFO TaskSetManager: Finished task 151.1 in stage 1.0 (TID 330) in 864422 ms on ip-10-0-51-185.ec2.internal (executor 20) (142/270) | |
| 17/06/14 04:30:52 INFO YarnAllocator: Driver requested a total number of 32 executor(s). | |
| 17/06/14 04:30:53 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 32 executors. | |
| 17/06/14 04:30:54 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:36538) with ID 32 | |
| 17/06/14 04:30:54 INFO TaskSetManager: Starting task 194.1 in stage 1.0 (TID 401, ip-10-0-50-96.ec2.internal, executor 32, partition 194, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:30:54 INFO TaskSetManager: Starting task 196.1 in stage 1.0 (TID 402, ip-10-0-50-96.ec2.internal, executor 32, partition 196, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:30:54 INFO TaskSetManager: Starting task 137.2 in stage 1.0 (TID 403, ip-10-0-50-96.ec2.internal, executor 32, partition 137, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:30:54 INFO TaskSetManager: Starting task 201.0 in stage 1.0 (TID 404, ip-10-0-50-96.ec2.internal, executor 32, partition 201, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:30:54 INFO ExecutorAllocationManager: New executor 32 has registered (new total is 15) | |
| 17/06/14 04:30:54 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:45474 with 2.8 GB RAM, BlockManagerId(32, ip-10-0-50-96.ec2.internal, 45474, None) | |
| 17/06/14 04:30:54 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-50-96.ec2.internal:45474 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:30:55 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:36538 | |
| 17/06/14 04:31:05 INFO TaskSetManager: Starting task 202.0 in stage 1.0 (TID 405, ip-10-0-50-96.ec2.internal, executor 19, partition 202, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:31:05 INFO TaskSetManager: Finished task 147.1 in stage 1.0 (TID 320) in 887809 ms on ip-10-0-50-96.ec2.internal (executor 19) (143/270) | |
| 17/06/14 04:31:07 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 19. | |
| 17/06/14 04:31:07 INFO DAGScheduler: Executor lost: 19 (epoch 1) | |
| 17/06/14 04:31:07 INFO BlockManagerMasterEndpoint: Trying to remove executor 19 from BlockManagerMaster. | |
| 17/06/14 04:31:07 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(19, ip-10-0-50-96.ec2.internal, 35803, None) | |
| 17/06/14 04:31:07 INFO BlockManagerMaster: Removed 19 successfully in removeExecutor | |
| 17/06/14 04:31:07 INFO YarnAllocator: Launching container container_1497410974650_0004_01_010519 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 04:31:07 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:31:07 INFO YarnAllocator: Completed container container_1497410974650_0004_01_002561 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:31:07 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:31:07 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:31:07 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:31:07 ERROR YarnClusterScheduler: Lost executor 19 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:31:07 WARN TaskSetManager: Lost task 195.0 in stage 1.0 (TID 394, ip-10-0-50-96.ec2.internal, executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:31:07 WARN TaskSetManager: Lost task 198.0 in stage 1.0 (TID 397, ip-10-0-50-96.ec2.internal, executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:31:07 WARN TaskSetManager: Lost task 197.0 in stage 1.0 (TID 396, ip-10-0-50-96.ec2.internal, executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:31:07 WARN TaskSetManager: Lost task 202.0 in stage 1.0 (TID 405, ip-10-0-50-96.ec2.internal, executor 19): ExecutorLostFailure (executor 19 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:31:07 INFO BlockManagerMaster: Removal of executor 19 requested | |
| 17/06/14 04:31:07 INFO BlockManagerMasterEndpoint: Trying to remove executor 19 from BlockManagerMaster. | |
| 17/06/14 04:31:07 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 19 | |
| 17/06/14 04:31:07 INFO ExecutorAllocationManager: Existing executor 19 has been removed (new total is 14) | |
| 17/06/14 04:31:07 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 04:31:07 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:31:07 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:31:09 INFO TaskSetManager: Starting task 202.1 in stage 1.0 (TID 406, ip-10-0-50-96.ec2.internal, executor 18, partition 202, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:31:09 INFO TaskSetManager: Finished task 169.0 in stage 1.0 (TID 316) in 895956 ms on ip-10-0-50-96.ec2.internal (executor 18) (144/270) | |
| 17/06/14 04:31:10 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:36566) with ID 33 | |
| 17/06/14 04:31:10 INFO TaskSetManager: Starting task 197.1 in stage 1.0 (TID 407, ip-10-0-50-96.ec2.internal, executor 33, partition 197, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:31:10 INFO TaskSetManager: Starting task 198.1 in stage 1.0 (TID 408, ip-10-0-50-96.ec2.internal, executor 33, partition 198, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:31:10 INFO TaskSetManager: Starting task 195.1 in stage 1.0 (TID 409, ip-10-0-50-96.ec2.internal, executor 33, partition 195, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:31:10 INFO TaskSetManager: Starting task 203.0 in stage 1.0 (TID 410, ip-10-0-50-96.ec2.internal, executor 33, partition 203, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:31:10 INFO ExecutorAllocationManager: New executor 33 has registered (new total is 15) | |
| 17/06/14 04:31:10 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:32833 with 2.8 GB RAM, BlockManagerId(33, ip-10-0-50-96.ec2.internal, 32833, None) | |
| 17/06/14 04:31:10 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-50-96.ec2.internal:32833 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:31:10 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:36566 | |
| 17/06/14 04:31:10 INFO TaskSetManager: Starting task 204.0 in stage 1.0 (TID 411, ip-10-0-50-96.ec2.internal, executor 18, partition 204, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:31:10 INFO TaskSetManager: Finished task 170.0 in stage 1.0 (TID 317) in 896815 ms on ip-10-0-50-96.ec2.internal (executor 18) (145/270) | |
| 17/06/14 04:31:22 INFO TaskSetManager: Starting task 205.0 in stage 1.0 (TID 412, ip-10-0-50-96.ec2.internal, executor 18, partition 205, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:31:22 INFO TaskSetManager: Finished task 171.0 in stage 1.0 (TID 318) in 908443 ms on ip-10-0-50-96.ec2.internal (executor 18) (146/270) | |
| 17/06/14 04:31:22 INFO YarnAllocator: Driver requested a total number of 31 executor(s). | |
| 17/06/14 04:31:22 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 31 executors. | |
| 17/06/14 04:31:31 INFO BlockManagerInfo: Added rdd_10_33 in memory on ip-10-0-50-96.ec2.internal:43860 (size: 843.9 MB, free: 1983.2 MB) | |
| 17/06/14 04:31:31 INFO TaskSetManager: Starting task 206.0 in stage 1.0 (TID 413, ip-10-0-50-96.ec2.internal, executor 18, partition 206, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:31:31 INFO TaskSetManager: Finished task 168.0 in stage 1.0 (TID 315) in 917459 ms on ip-10-0-50-96.ec2.internal (executor 18) (147/270) | |
| 17/06/14 04:31:32 INFO TaskSetManager: Starting task 207.0 in stage 1.0 (TID 414, ip-10-0-51-185.ec2.internal, executor 20, partition 207, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:31:32 INFO TaskSetManager: Finished task 158.1 in stage 1.0 (TID 327) in 904062 ms on ip-10-0-51-185.ec2.internal (executor 20) (148/270) | |
| 17/06/14 04:31:57 INFO TaskSetManager: Starting task 208.0 in stage 1.0 (TID 415, ip-10-0-49-157.ec2.internal, executor 22, partition 208, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:31:57 INFO TaskSetManager: Finished task 167.1 in stage 1.0 (TID 331) in 928039 ms on ip-10-0-49-157.ec2.internal (executor 22) (149/270) | |
| 17/06/14 04:32:04 INFO TaskSetManager: Starting task 209.0 in stage 1.0 (TID 416, ip-10-0-49-157.ec2.internal, executor 24, partition 209, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:32:04 INFO TaskSetManager: Finished task 155.1 in stage 1.0 (TID 340) in 932221 ms on ip-10-0-49-157.ec2.internal (executor 24) (150/270) | |
| 17/06/14 04:32:04 INFO YarnAllocator: Driver requested a total number of 30 executor(s). | |
| 17/06/14 04:32:04 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 30 executors. | |
| 17/06/14 04:32:14 INFO TaskSetManager: Starting task 210.0 in stage 1.0 (TID 417, ip-10-0-49-229.ec2.internal, executor 26, partition 210, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:32:14 INFO TaskSetManager: Finished task 175.1 in stage 1.0 (TID 359) in 911217 ms on ip-10-0-49-229.ec2.internal (executor 26) (151/270) | |
| 17/06/14 04:32:22 INFO TaskSetManager: Starting task 211.0 in stage 1.0 (TID 418, ip-10-0-49-229.ec2.internal, executor 26, partition 211, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:32:22 INFO TaskSetManager: Finished task 177.1 in stage 1.0 (TID 361) in 919365 ms on ip-10-0-49-229.ec2.internal (executor 26) (152/270) | |
| 17/06/14 04:32:24 INFO TaskSetManager: Starting task 212.0 in stage 1.0 (TID 419, ip-10-0-49-229.ec2.internal, executor 26, partition 212, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:32:24 INFO TaskSetManager: Finished task 176.1 in stage 1.0 (TID 360) in 921321 ms on ip-10-0-49-229.ec2.internal (executor 26) (153/270) | |
| 17/06/14 04:32:29 INFO TaskSetManager: Starting task 213.0 in stage 1.0 (TID 420, ip-10-0-49-157.ec2.internal, executor 24, partition 213, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:32:29 INFO TaskSetManager: Finished task 161.1 in stage 1.0 (TID 341) in 956647 ms on ip-10-0-49-157.ec2.internal (executor 24) (154/270) | |
| 17/06/14 04:32:29 INFO YarnAllocator: Driver requested a total number of 29 executor(s). | |
| 17/06/14 04:32:29 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 29 executors. | |
| 17/06/14 04:32:36 INFO TaskSetManager: Starting task 214.0 in stage 1.0 (TID 421, ip-10-0-49-157.ec2.internal, executor 29, partition 214, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:32:36 INFO TaskSetManager: Finished task 184.1 in stage 1.0 (TID 380) in 913529 ms on ip-10-0-49-157.ec2.internal (executor 29) (155/270) | |
| 17/06/14 04:32:44 INFO TaskSetManager: Starting task 215.0 in stage 1.0 (TID 422, ip-10-0-51-185.ec2.internal, executor 30, partition 215, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:32:44 INFO TaskSetManager: Finished task 190.1 in stage 1.0 (TID 382) in 915820 ms on ip-10-0-51-185.ec2.internal (executor 30) (156/270) | |
| 17/06/14 04:32:49 INFO TaskSetManager: Starting task 216.0 in stage 1.0 (TID 423, ip-10-0-49-157.ec2.internal, executor 22, partition 216, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:32:49 INFO TaskSetManager: Finished task 163.1 in stage 1.0 (TID 332) in 979979 ms on ip-10-0-49-157.ec2.internal (executor 22) (157/270) | |
| 17/06/14 04:32:50 INFO TaskSetManager: Starting task 217.0 in stage 1.0 (TID 424, ip-10-0-50-96.ec2.internal, executor 17, partition 217, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:32:50 INFO TaskSetManager: Finished task 138.1 in stage 1.0 (TID 311) in 999608 ms on ip-10-0-50-96.ec2.internal (executor 17) (158/270) | |
| 17/06/14 04:32:50 INFO YarnAllocator: Driver requested a total number of 28 executor(s). | |
| 17/06/14 04:32:50 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 28 executors. | |
| 17/06/14 04:32:50 INFO BlockManagerInfo: Added rdd_10_10 in memory on ip-10-0-49-157.ec2.internal:43440 (size: 824.6 MB, free: 2002.5 MB) | |
| 17/06/14 04:32:51 INFO TaskSetManager: Starting task 218.0 in stage 1.0 (TID 425, ip-10-0-49-157.ec2.internal, executor 24, partition 218, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:32:51 INFO TaskSetManager: Finished task 145.2 in stage 1.0 (TID 342) in 978615 ms on ip-10-0-49-157.ec2.internal (executor 24) (159/270) | |
| 17/06/14 04:32:54 INFO TaskSetManager: Starting task 219.0 in stage 1.0 (TID 426, ip-10-0-49-157.ec2.internal, executor 22, partition 219, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:32:54 INFO TaskSetManager: Finished task 142.2 in stage 1.0 (TID 334) in 984823 ms on ip-10-0-49-157.ec2.internal (executor 22) (160/270) | |
| 17/06/14 04:32:56 INFO TaskSetManager: Starting task 220.0 in stage 1.0 (TID 427, ip-10-0-49-157.ec2.internal, executor 24, partition 220, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:32:56 INFO TaskSetManager: Finished task 164.1 in stage 1.0 (TID 339) in 983541 ms on ip-10-0-49-157.ec2.internal (executor 24) (161/270) | |
| 17/06/14 04:33:00 INFO BlockManagerInfo: Added rdd_10_9 in memory on ip-10-0-50-96.ec2.internal:33453 (size: 831.6 MB, free: 1995.5 MB) | |
| 17/06/14 04:33:00 INFO TaskSetManager: Starting task 221.0 in stage 1.0 (TID 428, ip-10-0-50-96.ec2.internal, executor 17, partition 221, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:00 INFO TaskSetManager: Finished task 144.1 in stage 1.0 (TID 312) in 1009736 ms on ip-10-0-50-96.ec2.internal (executor 17) (162/270) | |
| 17/06/14 04:33:00 INFO YarnAllocator: Driver requested a total number of 27 executor(s). | |
| 17/06/14 04:33:00 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 27 executors. | |
| 17/06/14 04:33:02 INFO TaskSetManager: Starting task 222.0 in stage 1.0 (TID 429, ip-10-0-50-96.ec2.internal, executor 17, partition 222, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:02 INFO TaskSetManager: Finished task 146.1 in stage 1.0 (TID 313) in 1011139 ms on ip-10-0-50-96.ec2.internal (executor 17) (163/270) | |
| 17/06/14 04:33:03 INFO BlockManagerInfo: Added rdd_10_48 in memory on ip-10-0-49-157.ec2.internal:46402 (size: 838.4 MB, free: 1988.7 MB) | |
| 17/06/14 04:33:03 INFO TaskSetManager: Starting task 223.0 in stage 1.0 (TID 430, ip-10-0-49-157.ec2.internal, executor 29, partition 223, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:03 INFO TaskSetManager: Finished task 183.1 in stage 1.0 (TID 379) in 940187 ms on ip-10-0-49-157.ec2.internal (executor 29) (164/270) | |
| 17/06/14 04:33:09 INFO TaskSetManager: Starting task 224.0 in stage 1.0 (TID 431, ip-10-0-49-157.ec2.internal, executor 29, partition 224, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:09 INFO TaskSetManager: Finished task 181.2 in stage 1.0 (TID 381) in 945686 ms on ip-10-0-49-157.ec2.internal (executor 29) (165/270) | |
| 17/06/14 04:33:09 INFO TaskSetManager: Starting task 225.0 in stage 1.0 (TID 432, ip-10-0-50-96.ec2.internal, executor 17, partition 225, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:09 INFO TaskSetManager: Finished task 149.1 in stage 1.0 (TID 310) in 1018025 ms on ip-10-0-50-96.ec2.internal (executor 17) (166/270) | |
| 17/06/14 04:33:09 INFO YarnAllocator: Driver requested a total number of 26 executor(s). | |
| 17/06/14 04:33:09 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 26 executors. | |
| 17/06/14 04:33:11 INFO TaskSetManager: Starting task 226.0 in stage 1.0 (TID 433, ip-10-0-49-157.ec2.internal, executor 22, partition 226, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:11 INFO TaskSetManager: Finished task 143.2 in stage 1.0 (TID 333) in 1001890 ms on ip-10-0-49-157.ec2.internal (executor 22) (167/270) | |
| 17/06/14 04:33:18 INFO TaskSetManager: Starting task 227.0 in stage 1.0 (TID 434, ip-10-0-49-229.ec2.internal, executor 26, partition 227, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:18 INFO TaskSetManager: Finished task 172.1 in stage 1.0 (TID 358) in 974815 ms on ip-10-0-49-229.ec2.internal (executor 26) (168/270) | |
| 17/06/14 04:33:18 INFO TaskSetManager: Starting task 228.0 in stage 1.0 (TID 435, ip-10-0-51-185.ec2.internal, executor 30, partition 228, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:18 INFO TaskSetManager: Finished task 182.2 in stage 1.0 (TID 384) in 949101 ms on ip-10-0-51-185.ec2.internal (executor 30) (169/270) | |
| 17/06/14 04:33:24 INFO TaskSetManager: Starting task 229.0 in stage 1.0 (TID 436, ip-10-0-51-185.ec2.internal, executor 30, partition 229, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:24 INFO TaskSetManager: Finished task 189.1 in stage 1.0 (TID 385) in 955244 ms on ip-10-0-51-185.ec2.internal (executor 30) (170/270) | |
| 17/06/14 04:33:24 INFO YarnAllocator: Driver requested a total number of 25 executor(s). | |
| 17/06/14 04:33:24 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 25 executors. | |
| 17/06/14 04:33:25 INFO YarnAllocator: Launching container container_1497410974650_0004_01_003009 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 04:33:25 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 04:33:25 INFO YarnAllocator: Completed container container_1497410974650_0004_01_001503 on host: ip-10-0-49-229.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 04:33:25 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:33:25 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:33:25 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 04:33:25 ERROR YarnClusterScheduler: Lost executor 26 on ip-10-0-49-229.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:33:25 WARN TaskSetManager: Lost task 211.0 in stage 1.0 (TID 418, ip-10-0-49-229.ec2.internal, executor 26): ExecutorLostFailure (executor 26 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:33:25 WARN TaskSetManager: Lost task 210.0 in stage 1.0 (TID 417, ip-10-0-49-229.ec2.internal, executor 26): ExecutorLostFailure (executor 26 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:33:25 WARN TaskSetManager: Lost task 212.0 in stage 1.0 (TID 419, ip-10-0-49-229.ec2.internal, executor 26): ExecutorLostFailure (executor 26 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:33:25 WARN TaskSetManager: Lost task 227.0 in stage 1.0 (TID 434, ip-10-0-49-229.ec2.internal, executor 26): ExecutorLostFailure (executor 26 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 04:33:25 INFO DAGScheduler: Executor lost: 26 (epoch 1) | |
| 17/06/14 04:33:25 INFO BlockManagerMasterEndpoint: Trying to remove executor 26 from BlockManagerMaster. | |
| 17/06/14 04:33:25 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(26, ip-10-0-49-229.ec2.internal, 42403, None) | |
| 17/06/14 04:33:25 INFO BlockManagerMaster: Removed 26 successfully in removeExecutor | |
| 17/06/14 04:33:25 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 04:33:25 INFO ExecutorAllocationManager: Existing executor 26 has been removed (new total is 14) | |
| 17/06/14 04:33:25 WARN TransportChannelHandler: Exception in connection from /10.0.49.229:49416 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 04:33:25 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 26. | |
| 17/06/14 04:33:25 ERROR YarnClusterScheduler: Lost an executor 26 (already removed): Pending loss reason. | |
| 17/06/14 04:33:25 INFO BlockManagerMaster: Removal of executor 26 requested | |
| 17/06/14 04:33:25 INFO BlockManagerMasterEndpoint: Trying to remove executor 26 from BlockManagerMaster. | |
| 17/06/14 04:33:25 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 26 | |
| 17/06/14 04:33:25 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 04:33:25 INFO YarnAllocator: Submitted container request for host ip-10-0-50-96.ec2.internal,ip-10-0-49-229.ec2.internal,ip-10-0-51-185.ec2.internal,ip-10-0-49-157.ec2.internal. | |
| 17/06/14 04:33:27 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:49748) with ID 34 | |
| 17/06/14 04:33:27 INFO TaskSetManager: Starting task 227.1 in stage 1.0 (TID 437, ip-10-0-49-229.ec2.internal, executor 34, partition 227, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:27 INFO TaskSetManager: Starting task 212.1 in stage 1.0 (TID 438, ip-10-0-49-229.ec2.internal, executor 34, partition 212, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:27 INFO TaskSetManager: Starting task 210.1 in stage 1.0 (TID 439, ip-10-0-49-229.ec2.internal, executor 34, partition 210, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:27 INFO TaskSetManager: Starting task 211.1 in stage 1.0 (TID 440, ip-10-0-49-229.ec2.internal, executor 34, partition 211, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:27 INFO ExecutorAllocationManager: New executor 34 has registered (new total is 15) | |
| 17/06/14 04:33:27 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:38148 with 2.8 GB RAM, BlockManagerId(34, ip-10-0-49-229.ec2.internal, 38148, None) | |
| 17/06/14 04:33:27 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-10-0-49-229.ec2.internal:38148 (size: 33.3 KB, free: 2.8 GB) | |
| 17/06/14 04:33:27 INFO TaskSetManager: Starting task 230.0 in stage 1.0 (TID 441, ip-10-0-51-185.ec2.internal, executor 25, partition 230, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:27 INFO TaskSetManager: Finished task 165.1 in stage 1.0 (TID 346) in 1012741 ms on ip-10-0-51-185.ec2.internal (executor 25) (171/270) | |
| 17/06/14 04:33:27 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.229:49748 | |
| 17/06/14 04:33:29 INFO TaskSetManager: Starting task 231.0 in stage 1.0 (TID 442, ip-10-0-49-157.ec2.internal, executor 23, partition 231, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:29 INFO TaskSetManager: Finished task 166.1 in stage 1.0 (TID 335) in 1017033 ms on ip-10-0-49-157.ec2.internal (executor 23) (172/270) | |
| 17/06/14 04:33:30 INFO TaskSetManager: Starting task 232.0 in stage 1.0 (TID 443, ip-10-0-49-157.ec2.internal, executor 29, partition 232, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:30 INFO TaskSetManager: Finished task 187.1 in stage 1.0 (TID 378) in 967381 ms on ip-10-0-49-157.ec2.internal (executor 29) (173/270) | |
| 17/06/14 04:33:31 INFO TaskSetManager: Starting task 233.0 in stage 1.0 (TID 444, ip-10-0-49-157.ec2.internal, executor 23, partition 233, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:31 INFO TaskSetManager: Finished task 156.1 in stage 1.0 (TID 338) in 1018925 ms on ip-10-0-49-157.ec2.internal (executor 23) (174/270) | |
| 17/06/14 04:33:31 INFO YarnAllocator: Driver requested a total number of 24 executor(s). | |
| 17/06/14 04:33:31 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 24 executors. | |
| 17/06/14 04:33:35 INFO TaskSetManager: Starting task 234.0 in stage 1.0 (TID 445, ip-10-0-51-185.ec2.internal, executor 30, partition 234, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:35 INFO TaskSetManager: Finished task 188.1 in stage 1.0 (TID 383) in 966725 ms on ip-10-0-51-185.ec2.internal (executor 30) (175/270) | |
| 17/06/14 04:33:36 INFO BlockManagerInfo: Added rdd_10_5 in memory on ip-10-0-49-157.ec2.internal:46111 (size: 828.8 MB, free: 1998.3 MB) | |
| 17/06/14 04:33:36 INFO TaskSetManager: Starting task 235.0 in stage 1.0 (TID 446, ip-10-0-49-157.ec2.internal, executor 23, partition 235, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:36 INFO TaskSetManager: Finished task 140.2 in stage 1.0 (TID 336) in 1024281 ms on ip-10-0-49-157.ec2.internal (executor 23) (176/270) | |
| 17/06/14 04:33:43 INFO TaskSetManager: Starting task 236.0 in stage 1.0 (TID 447, ip-10-0-49-229.ec2.internal, executor 28, partition 236, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:43 INFO TaskSetManager: Finished task 185.0 in stage 1.0 (TID 372) in 993225 ms on ip-10-0-49-229.ec2.internal (executor 28) (177/270) | |
| 17/06/14 04:33:59 INFO TaskSetManager: Starting task 237.0 in stage 1.0 (TID 448, ip-10-0-51-185.ec2.internal, executor 25, partition 237, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:33:59 INFO TaskSetManager: Finished task 152.1 in stage 1.0 (TID 345) in 1044318 ms on ip-10-0-51-185.ec2.internal (executor 25) (178/270) | |
| 17/06/14 04:33:59 INFO YarnAllocator: Driver requested a total number of 23 executor(s). | |
| 17/06/14 04:33:59 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 23 executors. | |
| 17/06/14 04:34:01 INFO TaskSetManager: Starting task 238.0 in stage 1.0 (TID 449, ip-10-0-51-185.ec2.internal, executor 25, partition 238, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:34:01 INFO TaskSetManager: Finished task 154.1 in stage 1.0 (TID 343) in 1046715 ms on ip-10-0-51-185.ec2.internal (executor 25) (179/270) | |
| 17/06/14 04:34:06 INFO TaskSetManager: Starting task 239.0 in stage 1.0 (TID 450, ip-10-0-51-185.ec2.internal, executor 31, partition 239, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:34:06 INFO TaskSetManager: Finished task 193.1 in stage 1.0 (TID 393) in 951829 ms on ip-10-0-51-185.ec2.internal (executor 31) (180/270) | |
| 17/06/14 04:34:09 INFO BlockManagerInfo: Added rdd_10_25 in memory on ip-10-0-51-185.ec2.internal:43680 (size: 809.9 MB, free: 2017.2 MB) | |
| 17/06/14 04:34:09 INFO TaskSetManager: Starting task 240.0 in stage 1.0 (TID 451, ip-10-0-51-185.ec2.internal, executor 25, partition 240, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:34:09 INFO TaskSetManager: Finished task 160.1 in stage 1.0 (TID 344) in 1054361 ms on ip-10-0-51-185.ec2.internal (executor 25) (181/270) | |
| 17/06/14 04:34:21 INFO TaskSetManager: Starting task 241.0 in stage 1.0 (TID 452, ip-10-0-49-229.ec2.internal, executor 27, partition 241, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:34:21 INFO TaskSetManager: Finished task 178.1 in stage 1.0 (TID 365) in 1032046 ms on ip-10-0-49-229.ec2.internal (executor 27) (182/270) | |
| 17/06/14 04:34:21 INFO YarnAllocator: Driver requested a total number of 22 executor(s). | |
| 17/06/14 04:34:21 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 22 executors. | |
| 17/06/14 04:34:23 INFO TaskSetManager: Starting task 242.0 in stage 1.0 (TID 453, ip-10-0-49-157.ec2.internal, executor 23, partition 242, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:34:23 INFO TaskSetManager: Finished task 159.1 in stage 1.0 (TID 337) in 1070786 ms on ip-10-0-49-157.ec2.internal (executor 23) (183/270) | |
| 17/06/14 04:34:32 INFO TaskSetManager: Starting task 243.0 in stage 1.0 (TID 454, ip-10-0-49-229.ec2.internal, executor 27, partition 243, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:34:32 INFO TaskSetManager: Finished task 174.1 in stage 1.0 (TID 368) in 1043310 ms on ip-10-0-49-229.ec2.internal (executor 27) (184/270) | |
| 17/06/14 04:34:34 INFO TaskSetManager: Starting task 244.0 in stage 1.0 (TID 455, ip-10-0-51-185.ec2.internal, executor 31, partition 244, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:34:34 INFO TaskSetManager: Finished task 192.1 in stage 1.0 (TID 392) in 979781 ms on ip-10-0-51-185.ec2.internal (executor 31) (185/270) | |
| 17/06/14 04:34:41 INFO TaskSetManager: Starting task 245.0 in stage 1.0 (TID 456, ip-10-0-49-229.ec2.internal, executor 27, partition 245, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:34:41 INFO TaskSetManager: Finished task 173.1 in stage 1.0 (TID 367) in 1052025 ms on ip-10-0-49-229.ec2.internal (executor 27) (186/270) | |
| 17/06/14 04:34:41 INFO YarnAllocator: Driver requested a total number of 21 executor(s). | |
| 17/06/14 04:34:41 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 21 executors. | |
| 17/06/14 04:34:43 INFO TaskSetManager: Starting task 246.0 in stage 1.0 (TID 457, ip-10-0-49-229.ec2.internal, executor 28, partition 246, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:34:43 INFO TaskSetManager: Finished task 186.0 in stage 1.0 (TID 373) in 1052849 ms on ip-10-0-49-229.ec2.internal (executor 28) (187/270) | |
| 17/06/14 04:34:48 INFO TaskSetManager: Starting task 247.0 in stage 1.0 (TID 458, ip-10-0-49-229.ec2.internal, executor 28, partition 247, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:34:48 INFO TaskSetManager: Finished task 180.1 in stage 1.0 (TID 371) in 1058181 ms on ip-10-0-49-229.ec2.internal (executor 28) (188/270) | |
| 17/06/14 04:34:59 INFO TaskSetManager: Starting task 248.0 in stage 1.0 (TID 459, ip-10-0-51-185.ec2.internal, executor 31, partition 248, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:34:59 INFO TaskSetManager: Finished task 191.1 in stage 1.0 (TID 391) in 1005522 ms on ip-10-0-51-185.ec2.internal (executor 31) (189/270) | |
| 17/06/14 04:35:10 INFO BlockManagerInfo: Added rdd_10_22 in memory on ip-10-0-51-185.ec2.internal:35857 (size: 850.6 MB, free: 1976.5 MB) | |
| 17/06/14 04:35:10 INFO TaskSetManager: Starting task 249.0 in stage 1.0 (TID 460, ip-10-0-51-185.ec2.internal, executor 31, partition 249, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:35:10 INFO TaskSetManager: Finished task 157.2 in stage 1.0 (TID 390) in 1016044 ms on ip-10-0-51-185.ec2.internal (executor 31) (190/270) | |
| 17/06/14 04:35:10 INFO YarnAllocator: Driver requested a total number of 20 executor(s). | |
| 17/06/14 04:35:10 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 20 executors. | |
| 17/06/14 04:35:25 INFO TaskSetManager: Starting task 250.0 in stage 1.0 (TID 461, ip-10-0-49-229.ec2.internal, executor 27, partition 250, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:35:25 INFO TaskSetManager: Finished task 136.1 in stage 1.0 (TID 366) in 1096299 ms on ip-10-0-49-229.ec2.internal (executor 27) (191/270) | |
| 17/06/14 04:35:31 INFO TaskSetManager: Starting task 251.0 in stage 1.0 (TID 462, ip-10-0-49-229.ec2.internal, executor 28, partition 251, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:35:31 INFO TaskSetManager: Finished task 179.1 in stage 1.0 (TID 370) in 1100700 ms on ip-10-0-49-229.ec2.internal (executor 28) (192/270) | |
| 17/06/14 04:50:03 INFO TaskSetManager: Starting task 252.0 in stage 1.0 (TID 463, ip-10-0-51-185.ec2.internal, executor 20, partition 252, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:50:03 INFO TaskSetManager: Finished task 135.2 in stage 1.0 (TID 400) in 1150751 ms on ip-10-0-51-185.ec2.internal (executor 20) (193/270) | |
| 17/06/14 04:50:41 INFO TaskSetManager: Starting task 253.0 in stage 1.0 (TID 464, ip-10-0-51-185.ec2.internal, executor 20, partition 253, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:50:41 INFO TaskSetManager: Finished task 200.0 in stage 1.0 (TID 399) in 1192322 ms on ip-10-0-51-185.ec2.internal (executor 20) (194/270) | |
| 17/06/14 04:50:41 INFO YarnAllocator: Driver requested a total number of 19 executor(s). | |
| 17/06/14 04:50:41 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 19 executors. | |
| 17/06/14 04:50:45 INFO TaskSetManager: Starting task 254.0 in stage 1.0 (TID 465, ip-10-0-51-185.ec2.internal, executor 20, partition 254, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:50:45 INFO TaskSetManager: Finished task 199.0 in stage 1.0 (TID 398) in 1202338 ms on ip-10-0-51-185.ec2.internal (executor 20) (195/270) | |
| 17/06/14 04:51:24 INFO TaskSetManager: Starting task 255.0 in stage 1.0 (TID 466, ip-10-0-50-96.ec2.internal, executor 18, partition 255, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:51:24 INFO TaskSetManager: Finished task 204.0 in stage 1.0 (TID 411) in 1214051 ms on ip-10-0-50-96.ec2.internal (executor 18) (196/270) | |
| 17/06/14 04:51:52 INFO TaskSetManager: Starting task 256.0 in stage 1.0 (TID 467, ip-10-0-50-96.ec2.internal, executor 18, partition 256, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:51:52 INFO TaskSetManager: Finished task 206.0 in stage 1.0 (TID 413) in 1220638 ms on ip-10-0-50-96.ec2.internal (executor 18) (197/270) | |
| 17/06/14 04:52:04 INFO TaskSetManager: Starting task 257.0 in stage 1.0 (TID 468, ip-10-0-50-96.ec2.internal, executor 18, partition 257, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:52:04 INFO TaskSetManager: Finished task 202.1 in stage 1.0 (TID 406) in 1254302 ms on ip-10-0-50-96.ec2.internal (executor 18) (198/270) | |
| 17/06/14 04:52:04 INFO YarnAllocator: Driver requested a total number of 18 executor(s). | |
| 17/06/14 04:52:04 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 18 executors. | |
| 17/06/14 04:52:12 INFO TaskSetManager: Starting task 258.0 in stage 1.0 (TID 469, ip-10-0-50-96.ec2.internal, executor 32, partition 258, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:52:12 INFO TaskSetManager: Finished task 196.1 in stage 1.0 (TID 402) in 1277763 ms on ip-10-0-50-96.ec2.internal (executor 32) (199/270) | |
| 17/06/14 04:52:12 INFO TaskSetManager: Starting task 259.0 in stage 1.0 (TID 470, ip-10-0-50-96.ec2.internal, executor 33, partition 259, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:52:12 INFO TaskSetManager: Finished task 197.1 in stage 1.0 (TID 407) in 1262623 ms on ip-10-0-50-96.ec2.internal (executor 33) (200/270) | |
| 17/06/14 04:52:14 INFO TaskSetManager: Starting task 260.0 in stage 1.0 (TID 471, ip-10-0-51-185.ec2.internal, executor 20, partition 260, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:52:14 INFO TaskSetManager: Finished task 207.0 in stage 1.0 (TID 414) in 1242480 ms on ip-10-0-51-185.ec2.internal (executor 20) (201/270) | |
| 17/06/14 04:52:29 INFO TaskSetManager: Starting task 261.0 in stage 1.0 (TID 472, ip-10-0-50-96.ec2.internal, executor 18, partition 261, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:52:29 INFO TaskSetManager: Finished task 205.0 in stage 1.0 (TID 412) in 1266908 ms on ip-10-0-50-96.ec2.internal (executor 18) (202/270) | |
| 17/06/14 04:52:29 INFO YarnAllocator: Driver requested a total number of 17 executor(s). | |
| 17/06/14 04:52:29 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 17 executors. | |
| 17/06/14 04:52:29 INFO TaskSetManager: Starting task 262.0 in stage 1.0 (TID 473, ip-10-0-50-96.ec2.internal, executor 33, partition 262, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:52:29 INFO TaskSetManager: Finished task 195.1 in stage 1.0 (TID 409) in 1279355 ms on ip-10-0-50-96.ec2.internal (executor 33) (203/270) | |
| 17/06/14 04:52:53 INFO BlockManagerInfo: Added rdd_10_63 in memory on ip-10-0-50-96.ec2.internal:32833 (size: 824.7 MB, free: 2002.4 MB) | |
| 17/06/14 04:52:53 INFO TaskSetManager: Starting task 263.0 in stage 1.0 (TID 474, ip-10-0-50-96.ec2.internal, executor 33, partition 263, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:52:53 INFO TaskSetManager: Finished task 198.1 in stage 1.0 (TID 408) in 1303647 ms on ip-10-0-50-96.ec2.internal (executor 33) (204/270) | |
| 17/06/14 04:53:03 INFO BlockManagerInfo: Added rdd_10_2 in memory on ip-10-0-50-96.ec2.internal:45474 (size: 832.5 MB, free: 1994.6 MB) | |
| 17/06/14 04:53:03 INFO TaskSetManager: Starting task 264.0 in stage 1.0 (TID 475, ip-10-0-50-96.ec2.internal, executor 32, partition 264, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:53:03 INFO TaskSetManager: Finished task 137.2 in stage 1.0 (TID 403) in 1328467 ms on ip-10-0-50-96.ec2.internal (executor 32) (205/270) | |
| 17/06/14 04:53:05 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-50-96.ec2.internal:45474 (size: 33.3 KB) | |
| 17/06/14 04:53:05 INFO BlockManagerInfo: Removed rdd_10_2 on ip-10-0-50-96.ec2.internal:45474 in memory (size: 832.5 MB, free: 2.8 GB) | |
| 17/06/14 04:53:12 INFO TaskSetManager: Starting task 265.0 in stage 1.0 (TID 476, ip-10-0-50-96.ec2.internal, executor 33, partition 265, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:53:12 INFO TaskSetManager: Finished task 203.0 in stage 1.0 (TID 410) in 1322370 ms on ip-10-0-50-96.ec2.internal (executor 33) (206/270) | |
| 17/06/14 04:53:12 INFO YarnAllocator: Driver requested a total number of 16 executor(s). | |
| 17/06/14 04:53:12 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 16 executors. | |
| 17/06/14 04:53:27 INFO TaskSetManager: Starting task 266.0 in stage 1.0 (TID 477, ip-10-0-49-157.ec2.internal, executor 24, partition 266, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:53:27 INFO TaskSetManager: Finished task 220.0 in stage 1.0 (TID 427) in 1231574 ms on ip-10-0-49-157.ec2.internal (executor 24) (207/270) | |
| 17/06/14 04:53:32 INFO BlockManagerInfo: Added rdd_10_66 in memory on ip-10-0-50-96.ec2.internal:45474 (size: 840.6 MB, free: 1986.6 MB) | |
| 17/06/14 04:53:32 INFO TaskSetManager: Starting task 267.0 in stage 1.0 (TID 478, ip-10-0-50-96.ec2.internal, executor 32, partition 267, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:53:32 INFO TaskSetManager: Finished task 201.0 in stage 1.0 (TID 404) in 1358144 ms on ip-10-0-50-96.ec2.internal (executor 32) (208/270) | |
| 17/06/14 04:53:33 INFO TaskSetManager: Starting task 268.0 in stage 1.0 (TID 479, ip-10-0-49-157.ec2.internal, executor 24, partition 268, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:53:33 INFO TaskSetManager: Finished task 209.0 in stage 1.0 (TID 416) in 1288883 ms on ip-10-0-49-157.ec2.internal (executor 24) (209/270) | |
| 17/06/14 04:53:35 INFO TaskSetManager: Starting task 269.0 in stage 1.0 (TID 480, ip-10-0-49-157.ec2.internal, executor 29, partition 269, PROCESS_LOCAL, 5883 bytes) | |
| 17/06/14 04:53:35 INFO TaskSetManager: Finished task 214.0 in stage 1.0 (TID 421) in 1258231 ms on ip-10-0-49-157.ec2.internal (executor 29) (210/270) | |
| 17/06/14 04:53:35 INFO YarnAllocator: Driver requested a total number of 15 executor(s). | |
| 17/06/14 04:53:35 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 15 executors. | |
| 17/06/14 04:53:36 INFO TaskSetManager: Finished task 229.0 in stage 1.0 (TID 436) in 1211831 ms on ip-10-0-51-185.ec2.internal (executor 30) (211/270) | |
| 17/06/14 04:53:39 INFO TaskSetManager: Finished task 194.1 in stage 1.0 (TID 401) in 1364522 ms on ip-10-0-50-96.ec2.internal (executor 32) (212/270) | |
| 17/06/14 04:53:39 INFO BlockManagerInfo: Added rdd_10_73 in memory on ip-10-0-49-157.ec2.internal:44837 (size: 815.7 MB, free: 2011.4 MB) | |
| 17/06/14 04:53:39 INFO TaskSetManager: Finished task 208.0 in stage 1.0 (TID 415) in 1301490 ms on ip-10-0-49-157.ec2.internal (executor 22) (213/270) | |
| 17/06/14 04:53:49 INFO TaskSetManager: Finished task 218.0 in stage 1.0 (TID 425) in 1258560 ms on ip-10-0-49-157.ec2.internal (executor 24) (214/270) | |
| 17/06/14 04:53:49 INFO YarnAllocator: Driver requested a total number of 14 executor(s). | |
| 17/06/14 04:53:50 INFO TaskSetManager: Finished task 224.0 in stage 1.0 (TID 431) in 1241574 ms on ip-10-0-49-157.ec2.internal (executor 29) (215/270) | |
| 17/06/14 04:53:53 INFO TaskSetManager: Finished task 223.0 in stage 1.0 (TID 430) in 1250039 ms on ip-10-0-49-157.ec2.internal (executor 29) (216/270) | |
| 17/06/14 04:53:56 INFO TaskSetManager: Finished task 228.0 in stage 1.0 (TID 435) in 1238733 ms on ip-10-0-51-185.ec2.internal (executor 30) (217/270) | |
| 17/06/14 04:54:09 INFO TaskSetManager: Finished task 219.0 in stage 1.0 (TID 426) in 1274402 ms on ip-10-0-49-157.ec2.internal (executor 22) (218/270) | |
| 17/06/14 04:54:09 INFO YarnAllocator: Driver requested a total number of 13 executor(s). | |
| 17/06/14 04:54:10 INFO TaskSetManager: Finished task 213.0 in stage 1.0 (TID 420) in 1301033 ms on ip-10-0-49-157.ec2.internal (executor 24) (219/270) | |
| 17/06/14 04:54:13 INFO BlockManagerInfo: Added rdd_10_99 in memory on ip-10-0-51-185.ec2.internal:46768 (size: 844.3 MB, free: 1982.8 MB) | |
| 17/06/14 04:54:13 INFO TaskSetManager: Finished task 234.0 in stage 1.0 (TID 445) in 1237775 ms on ip-10-0-51-185.ec2.internal (executor 30) (220/270) | |
| 17/06/14 04:54:18 INFO TaskSetManager: Finished task 215.0 in stage 1.0 (TID 422) in 1293634 ms on ip-10-0-51-185.ec2.internal (executor 30) (221/270) | |
| 17/06/14 04:54:21 INFO TaskSetManager: Finished task 226.0 in stage 1.0 (TID 433) in 1270053 ms on ip-10-0-49-157.ec2.internal (executor 22) (222/270) | |
| 17/06/14 04:54:21 INFO YarnAllocator: Driver requested a total number of 12 executor(s). | |
| 17/06/14 04:54:26 INFO TaskSetManager: Finished task 235.0 in stage 1.0 (TID 446) in 1250260 ms on ip-10-0-49-157.ec2.internal (executor 23) (223/270) | |
| 17/06/14 04:54:33 INFO BlockManagerInfo: Added rdd_10_75 in memory on ip-10-0-49-229.ec2.internal:38148 (size: 851.0 MB, free: 1976.1 MB) | |
| 17/06/14 04:54:33 INFO TaskSetManager: Finished task 210.1 in stage 1.0 (TID 439) in 1266454 ms on ip-10-0-49-229.ec2.internal (executor 34) (224/270) | |
| 17/06/14 04:54:36 INFO BlockManagerInfo: Added rdd_10_81 in memory on ip-10-0-49-157.ec2.internal:44837 (size: 824.4 MB, free: 1187.0 MB) | |
| 17/06/14 04:54:36 INFO TaskSetManager: Finished task 216.0 in stage 1.0 (TID 423) in 1306937 ms on ip-10-0-49-157.ec2.internal (executor 22) (225/270) | |
| 17/06/14 04:54:39 INFO TaskSetManager: Finished task 231.0 in stage 1.0 (TID 442) in 1270142 ms on ip-10-0-49-157.ec2.internal (executor 23) (226/270) | |
| 17/06/14 04:54:39 INFO YarnAllocator: Driver requested a total number of 11 executor(s). | |
| 17/06/14 04:54:43 INFO TaskSetManager: Finished task 232.0 in stage 1.0 (TID 443) in 1272884 ms on ip-10-0-49-157.ec2.internal (executor 29) (227/270) | |
| 17/06/14 04:54:47 INFO TaskSetManager: Finished task 239.0 in stage 1.0 (TID 450) in 1241562 ms on ip-10-0-51-185.ec2.internal (executor 31) (228/270) | |
| 17/06/14 04:54:51 INFO TaskSetManager: Finished task 244.0 in stage 1.0 (TID 455) in 1217469 ms on ip-10-0-51-185.ec2.internal (executor 31) (229/270) | |
| 17/06/14 04:54:52 INFO TaskSetManager: Finished task 221.0 in stage 1.0 (TID 428) in 1311824 ms on ip-10-0-50-96.ec2.internal (executor 17) (230/270) | |
| 17/06/14 04:54:52 INFO YarnAllocator: Driver requested a total number of 10 executor(s). | |
| 17/06/14 04:54:53 INFO TaskSetManager: Finished task 230.0 in stage 1.0 (TID 441) in 1286115 ms on ip-10-0-51-185.ec2.internal (executor 25) (231/270) | |
| 17/06/14 04:54:55 INFO TaskSetManager: Finished task 225.0 in stage 1.0 (TID 432) in 1305776 ms on ip-10-0-50-96.ec2.internal (executor 17) (232/270) | |
| 17/06/14 04:54:59 INFO TaskSetManager: Finished task 211.1 in stage 1.0 (TID 440) in 1292085 ms on ip-10-0-49-229.ec2.internal (executor 34) (233/270) | |
| 17/06/14 04:55:03 INFO TaskSetManager: Finished task 238.0 in stage 1.0 (TID 449) in 1261494 ms on ip-10-0-51-185.ec2.internal (executor 25) (234/270) | |
| 17/06/14 04:55:03 INFO YarnAllocator: Driver requested a total number of 9 executor(s). | |
| 17/06/14 04:55:03 INFO TaskSetManager: Finished task 217.0 in stage 1.0 (TID 424) in 1332599 ms on ip-10-0-50-96.ec2.internal (executor 17) (235/270) | |
| 17/06/14 04:55:03 INFO TaskSetManager: Finished task 236.0 in stage 1.0 (TID 447) in 1279957 ms on ip-10-0-49-229.ec2.internal (executor 28) (236/270) | |
| 17/06/14 04:55:06 INFO TaskSetManager: Finished task 222.0 in stage 1.0 (TID 429) in 1324058 ms on ip-10-0-50-96.ec2.internal (executor 17) (237/270) | |
| 17/06/14 04:55:06 INFO TaskSetManager: Finished task 249.0 in stage 1.0 (TID 460) in 1196497 ms on ip-10-0-51-185.ec2.internal (executor 31) (238/270) | |
| 17/06/14 04:55:06 INFO YarnAllocator: Driver requested a total number of 8 executor(s). | |
| 17/06/14 04:55:07 INFO TaskSetManager: Finished task 227.1 in stage 1.0 (TID 437) in 1300862 ms on ip-10-0-49-229.ec2.internal (executor 34) (239/270) | |
| 17/06/14 04:55:10 INFO TaskSetManager: Finished task 237.0 in stage 1.0 (TID 448) in 1271680 ms on ip-10-0-51-185.ec2.internal (executor 25) (240/270) | |
| 17/06/14 04:55:20 INFO TaskSetManager: Finished task 240.0 in stage 1.0 (TID 451) in 1271493 ms on ip-10-0-51-185.ec2.internal (executor 25) (241/270) | |
| 17/06/14 04:55:22 INFO TaskSetManager: Finished task 242.0 in stage 1.0 (TID 453) in 1259232 ms on ip-10-0-49-157.ec2.internal (executor 23) (242/270) | |
| 17/06/14 04:55:22 INFO YarnAllocator: Driver requested a total number of 7 executor(s). | |
| 17/06/14 04:55:26 INFO TaskSetManager: Finished task 233.0 in stage 1.0 (TID 444) in 1314892 ms on ip-10-0-49-157.ec2.internal (executor 23) (243/270) | |
| 17/06/14 04:55:34 INFO TaskSetManager: Finished task 241.0 in stage 1.0 (TID 452) in 1273259 ms on ip-10-0-49-229.ec2.internal (executor 27) (244/270) | |
| 17/06/14 04:55:34 INFO TaskSetManager: Finished task 251.0 in stage 1.0 (TID 462) in 1203659 ms on ip-10-0-49-229.ec2.internal (executor 28) (245/270) | |
| 17/06/14 04:55:37 INFO TaskSetManager: Finished task 212.1 in stage 1.0 (TID 438) in 1330740 ms on ip-10-0-49-229.ec2.internal (executor 34) (246/270) | |
| 17/06/14 04:55:37 INFO YarnAllocator: Driver requested a total number of 6 executor(s). | |
| 17/06/14 04:55:39 INFO TaskSetManager: Finished task 243.0 in stage 1.0 (TID 454) in 1267179 ms on ip-10-0-49-229.ec2.internal (executor 27) (247/270) | |
| 17/06/14 04:55:41 INFO TaskSetManager: Finished task 248.0 in stage 1.0 (TID 459) in 1241750 ms on ip-10-0-51-185.ec2.internal (executor 31) (248/270) | |
| 17/06/14 04:55:46 INFO BlockManagerInfo: Added rdd_10_111 in memory on ip-10-0-49-229.ec2.internal:37408 (size: 807.1 MB, free: 2020.0 MB) | |
| 17/06/14 04:55:47 INFO TaskSetManager: Finished task 246.0 in stage 1.0 (TID 457) in 1263813 ms on ip-10-0-49-229.ec2.internal (executor 28) (249/270) | |
| 17/06/14 04:55:49 INFO BlockManagerInfo: Added rdd_10_112 in memory on ip-10-0-49-229.ec2.internal:37408 (size: 829.0 MB, free: 1191.0 MB) | |
| 17/06/14 04:55:49 INFO TaskSetManager: Finished task 247.0 in stage 1.0 (TID 458) in 1260602 ms on ip-10-0-49-229.ec2.internal (executor 28) (250/270) | |
| 17/06/14 04:55:49 INFO YarnAllocator: Driver requested a total number of 5 executor(s). | |
| 17/06/14 04:55:53 INFO TaskSetManager: Finished task 245.0 in stage 1.0 (TID 456) in 1272139 ms on ip-10-0-49-229.ec2.internal (executor 27) (251/270) | |
| 17/06/14 04:55:59 INFO TaskSetManager: Finished task 250.0 in stage 1.0 (TID 461) in 1233611 ms on ip-10-0-49-229.ec2.internal (executor 27) (252/270) | |
| 17/06/14 04:56:59 INFO ExecutorAllocationManager: Request to remove executorIds: 27 | |
| 17/06/14 04:56:59 INFO YarnClusterSchedulerBackend: Requesting to kill executor(s) 27 | |
| 17/06/14 04:56:59 INFO YarnClusterSchedulerBackend: Actual list of executor(s) to be killed is 27 | |
| 17/06/14 04:56:59 INFO YarnAllocator: Driver requested a total number of 14 executor(s). | |
| 17/06/14 04:56:59 INFO ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 27. | |
| 17/06/14 04:56:59 INFO ExecutorAllocationManager: Removing executor 27 because it has been idle for 60 seconds (new desired total will be 14) | |
| 17/06/14 04:57:03 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 27. | |
| 17/06/14 04:57:03 INFO DAGScheduler: Executor lost: 27 (epoch 1) | |
| 17/06/14 04:57:03 INFO BlockManagerMasterEndpoint: Trying to remove executor 27 from BlockManagerMaster. | |
| 17/06/14 04:57:03 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(27, ip-10-0-49-229.ec2.internal, 33294, None) | |
| 17/06/14 04:57:03 INFO BlockManagerMaster: Removed 27 successfully in removeExecutor | |
| 17/06/14 04:57:03 INFO YarnClusterScheduler: Executor 27 on ip-10-0-49-229.ec2.internal killed by driver. | |
| 17/06/14 04:57:03 INFO ExecutorAllocationManager: Existing executor 27 has been removed (new total is 14) | |
| 17/06/14 05:00:08 INFO TaskSetManager: Finished task 252.0 in stage 1.0 (TID 463) in 605080 ms on ip-10-0-51-185.ec2.internal (executor 20) (253/270) | |
| 17/06/14 05:00:15 INFO TaskSetManager: Finished task 253.0 in stage 1.0 (TID 464) in 573523 ms on ip-10-0-51-185.ec2.internal (executor 20) (254/270) | |
| 17/06/14 05:00:15 INFO YarnAllocator: Driver requested a total number of 4 executor(s). | |
| 17/06/14 05:00:19 INFO TaskSetManager: Finished task 254.0 in stage 1.0 (TID 465) in 573943 ms on ip-10-0-51-185.ec2.internal (executor 20) (255/270) | |
| 17/06/14 05:00:21 INFO TaskSetManager: Finished task 255.0 in stage 1.0 (TID 466) in 536177 ms on ip-10-0-50-96.ec2.internal (executor 18) (256/270) | |
| 17/06/14 05:00:26 INFO BlockManagerInfo: Added rdd_10_131 in memory on ip-10-0-49-157.ec2.internal:43440 (size: 821.5 MB, free: 1181.0 MB) | |
| 17/06/14 05:00:26 INFO TaskSetManager: Finished task 266.0 in stage 1.0 (TID 477) in 418896 ms on ip-10-0-49-157.ec2.internal (executor 24) (257/270) | |
| 17/06/14 05:00:30 INFO TaskSetManager: Finished task 268.0 in stage 1.0 (TID 479) in 416527 ms on ip-10-0-49-157.ec2.internal (executor 24) (258/270) | |
| 17/06/14 05:00:30 INFO YarnAllocator: Driver requested a total number of 3 executor(s). | |
| 17/06/14 05:00:30 INFO TaskSetManager: Finished task 257.0 in stage 1.0 (TID 468) in 506663 ms on ip-10-0-50-96.ec2.internal (executor 18) (259/270) | |
| 17/06/14 05:00:35 INFO TaskSetManager: Finished task 256.0 in stage 1.0 (TID 467) in 523892 ms on ip-10-0-50-96.ec2.internal (executor 18) (260/270) | |
| 17/06/14 05:00:41 INFO TaskSetManager: Finished task 261.0 in stage 1.0 (TID 472) in 492319 ms on ip-10-0-50-96.ec2.internal (executor 18) (261/270) | |
| 17/06/14 05:00:45 INFO TaskSetManager: Finished task 260.0 in stage 1.0 (TID 471) in 510795 ms on ip-10-0-51-185.ec2.internal (executor 20) (262/270) | |
| 17/06/14 05:00:45 INFO YarnAllocator: Driver requested a total number of 2 executor(s). | |
| 17/06/14 05:00:59 INFO TaskSetManager: Finished task 258.0 in stage 1.0 (TID 469) in 527214 ms on ip-10-0-50-96.ec2.internal (executor 32) (263/270) | |
| 17/06/14 05:01:13 INFO TaskSetManager: Finished task 259.0 in stage 1.0 (TID 470) in 540865 ms on ip-10-0-50-96.ec2.internal (executor 33) (264/270) | |
| 17/06/14 05:01:15 INFO TaskSetManager: Finished task 262.0 in stage 1.0 (TID 473) in 526475 ms on ip-10-0-50-96.ec2.internal (executor 33) (265/270) | |
| 17/06/14 05:01:20 INFO BlockManagerInfo: Added rdd_10_129 in memory on ip-10-0-50-96.ec2.internal:45474 (size: 788.4 MB, free: 1198.1 MB) | |
| 17/06/14 05:01:20 INFO TaskSetManager: Finished task 264.0 in stage 1.0 (TID 475) in 497320 ms on ip-10-0-50-96.ec2.internal (executor 32) (266/270) | |
| 17/06/14 05:01:20 INFO YarnAllocator: Driver requested a total number of 1 executor(s). | |
| 17/06/14 05:01:20 INFO TaskSetManager: Finished task 263.0 in stage 1.0 (TID 474) in 507283 ms on ip-10-0-50-96.ec2.internal (executor 33) (267/270) | |
| 17/06/14 05:01:23 INFO BlockManagerInfo: Added rdd_10_134 in memory on ip-10-0-49-157.ec2.internal:46402 (size: 828.0 MB, free: 1160.6 MB) | |
| 17/06/14 05:01:23 INFO TaskSetManager: Finished task 269.0 in stage 1.0 (TID 480) in 468583 ms on ip-10-0-49-157.ec2.internal (executor 29) (268/270) | |
| 17/06/14 05:01:27 INFO TaskSetManager: Finished task 267.0 in stage 1.0 (TID 478) in 475165 ms on ip-10-0-50-96.ec2.internal (executor 32) (269/270) | |
| 17/06/14 05:01:32 INFO TaskSetManager: Finished task 265.0 in stage 1.0 (TID 476) in 500332 ms on ip-10-0-50-96.ec2.internal (executor 33) (270/270) | |
| 17/06/14 05:01:32 INFO YarnClusterScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool | |
| 17/06/14 05:01:32 INFO DAGScheduler: ResultStage 1 (saveAsTextFile at ShredJob.scala:400) finished in 2839.341 s | |
| 17/06/14 05:01:32 INFO DAGScheduler: Job 0 finished: saveAsTextFile at ShredJob.scala:400, took 3089.427851 s | |
| 17/06/14 05:01:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1 | |
| 17/06/14 05:01:32 INFO DirectFileOutputCommitter: Nothing to clean up since no temporary files were written. | |
| 17/06/14 05:01:32 INFO MultipartUploadOutputStream: close closed:false s3://gn-snowplow-data-bucket/shredded/bad/run=2017-06-14-03-25-42/_SUCCESS | |
| 17/06/14 05:01:32 INFO YarnAllocator: Driver requested a total number of 0 executor(s). | |
| 17/06/14 05:01:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1 | |
| 17/06/14 05:01:33 INFO SparkContext: Starting job: saveAsTextFile at ShredJob.scala:412 | |
| 17/06/14 05:01:33 INFO DAGScheduler: Registering RDD 13 (groupBy at ShredJob.scala:376) | |
| 17/06/14 05:01:33 INFO DAGScheduler: Registering RDD 16 (map at ShredJob.scala:378) | |
| 17/06/14 05:01:33 INFO DAGScheduler: Registering RDD 18 (map at ShredJob.scala:384) | |
| 17/06/14 05:01:33 INFO DAGScheduler: Got job 1 (saveAsTextFile at ShredJob.scala:412) with 135 output partitions | |
| 17/06/14 05:01:33 INFO DAGScheduler: Final stage: ResultStage 6 (saveAsTextFile at ShredJob.scala:412) | |
| 17/06/14 05:01:33 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 5, ShuffleMapStage 4) | |
| 17/06/14 05:01:33 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 5, ShuffleMapStage 4) | |
| 17/06/14 05:01:33 INFO DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[13] at groupBy at ShredJob.scala:376), which has no missing parents | |
| 17/06/14 05:01:33 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 10.9 KB, free 413.5 MB) | |
| 17/06/14 05:01:33 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 5.5 KB, free 413.5 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 10.0.49.229:33721 (size: 5.5 KB, free: 413.9 MB) | |
| 17/06/14 05:01:33 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:997 | |
| 17/06/14 05:01:33 INFO DAGScheduler: Submitting 135 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[13] at groupBy at ShredJob.scala:376) | |
| 17/06/14 05:01:33 INFO YarnClusterScheduler: Adding task set 3.0 with 135 tasks | |
| 17/06/14 05:01:33 INFO DAGScheduler: Submitting ShuffleMapStage 5 (MapPartitionsRDD[18] at map at ShredJob.scala:384), which has no missing parents | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 75.0 in stage 3.0 (TID 481, ip-10-0-49-229.ec2.internal, executor 34, partition 75, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 99.0 in stage 3.0 (TID 482, ip-10-0-51-185.ec2.internal, executor 30, partition 99, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 73.0 in stage 3.0 (TID 483, ip-10-0-49-157.ec2.internal, executor 22, partition 73, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 5.0 in stage 3.0 (TID 484, ip-10-0-49-157.ec2.internal, executor 23, partition 5, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 33.0 in stage 3.0 (TID 485, ip-10-0-50-96.ec2.internal, executor 18, partition 33, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 18.0 in stage 3.0 (TID 486, ip-10-0-51-185.ec2.internal, executor 20, partition 18, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 22.0 in stage 3.0 (TID 487, ip-10-0-51-185.ec2.internal, executor 31, partition 22, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 111.0 in stage 3.0 (TID 488, ip-10-0-49-229.ec2.internal, executor 28, partition 111, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 66.0 in stage 3.0 (TID 489, ip-10-0-50-96.ec2.internal, executor 32, partition 66, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 10.0 in stage 3.0 (TID 490, ip-10-0-49-157.ec2.internal, executor 24, partition 10, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 9.0 in stage 3.0 (TID 491, ip-10-0-50-96.ec2.internal, executor 17, partition 9, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 25.0 in stage 3.0 (TID 492, ip-10-0-51-185.ec2.internal, executor 25, partition 25, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 48.0 in stage 3.0 (TID 493, ip-10-0-49-157.ec2.internal, executor 29, partition 48, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 63.0 in stage 3.0 (TID 494, ip-10-0-50-96.ec2.internal, executor 33, partition 63, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 81.0 in stage 3.0 (TID 495, ip-10-0-49-157.ec2.internal, executor 22, partition 81, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 112.0 in stage 3.0 (TID 496, ip-10-0-49-229.ec2.internal, executor 28, partition 112, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 129.0 in stage 3.0 (TID 497, ip-10-0-50-96.ec2.internal, executor 32, partition 129, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 131.0 in stage 3.0 (TID 498, ip-10-0-49-157.ec2.internal, executor 24, partition 131, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 134.0 in stage 3.0 (TID 499, ip-10-0-49-157.ec2.internal, executor 29, partition 134, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 500, ip-10-0-49-229.ec2.internal, executor 34, partition 0, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 501, ip-10-0-51-185.ec2.internal, executor 30, partition 1, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 2.0 in stage 3.0 (TID 502, ip-10-0-49-157.ec2.internal, executor 22, partition 2, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 3.0 in stage 3.0 (TID 503, ip-10-0-49-157.ec2.internal, executor 23, partition 3, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 4.0 in stage 3.0 (TID 504, ip-10-0-50-96.ec2.internal, executor 18, partition 4, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 6.0 in stage 3.0 (TID 505, ip-10-0-51-185.ec2.internal, executor 20, partition 6, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 7.0 in stage 3.0 (TID 506, ip-10-0-51-185.ec2.internal, executor 31, partition 7, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 8.0 in stage 3.0 (TID 507, ip-10-0-49-229.ec2.internal, executor 28, partition 8, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 11.0 in stage 3.0 (TID 508, ip-10-0-50-96.ec2.internal, executor 32, partition 11, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 12.0 in stage 3.0 (TID 509, ip-10-0-49-157.ec2.internal, executor 24, partition 12, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 13.0 in stage 3.0 (TID 510, ip-10-0-50-96.ec2.internal, executor 17, partition 13, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 14.0 in stage 3.0 (TID 511, ip-10-0-51-185.ec2.internal, executor 25, partition 14, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 15.0 in stage 3.0 (TID 512, ip-10-0-49-157.ec2.internal, executor 29, partition 15, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 16.0 in stage 3.0 (TID 513, ip-10-0-50-96.ec2.internal, executor 33, partition 16, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 17.0 in stage 3.0 (TID 514, ip-10-0-49-229.ec2.internal, executor 34, partition 17, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 19.0 in stage 3.0 (TID 515, ip-10-0-51-185.ec2.internal, executor 30, partition 19, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 20.0 in stage 3.0 (TID 516, ip-10-0-49-157.ec2.internal, executor 22, partition 20, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 21.0 in stage 3.0 (TID 517, ip-10-0-49-157.ec2.internal, executor 23, partition 21, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 23.0 in stage 3.0 (TID 518, ip-10-0-50-96.ec2.internal, executor 18, partition 23, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 24.0 in stage 3.0 (TID 519, ip-10-0-51-185.ec2.internal, executor 20, partition 24, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 26.0 in stage 3.0 (TID 520, ip-10-0-51-185.ec2.internal, executor 31, partition 26, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 27.0 in stage 3.0 (TID 521, ip-10-0-49-229.ec2.internal, executor 28, partition 27, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 28.0 in stage 3.0 (TID 522, ip-10-0-50-96.ec2.internal, executor 32, partition 28, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 29.0 in stage 3.0 (TID 523, ip-10-0-49-157.ec2.internal, executor 24, partition 29, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 30.0 in stage 3.0 (TID 524, ip-10-0-50-96.ec2.internal, executor 17, partition 30, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 31.0 in stage 3.0 (TID 525, ip-10-0-51-185.ec2.internal, executor 25, partition 31, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 32.0 in stage 3.0 (TID 526, ip-10-0-49-157.ec2.internal, executor 29, partition 32, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 34.0 in stage 3.0 (TID 527, ip-10-0-50-96.ec2.internal, executor 33, partition 34, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 35.0 in stage 3.0 (TID 528, ip-10-0-49-229.ec2.internal, executor 34, partition 35, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 36.0 in stage 3.0 (TID 529, ip-10-0-51-185.ec2.internal, executor 30, partition 36, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 37.0 in stage 3.0 (TID 530, ip-10-0-49-157.ec2.internal, executor 23, partition 37, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 38.0 in stage 3.0 (TID 531, ip-10-0-50-96.ec2.internal, executor 18, partition 38, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 39.0 in stage 3.0 (TID 532, ip-10-0-51-185.ec2.internal, executor 20, partition 39, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 40.0 in stage 3.0 (TID 533, ip-10-0-51-185.ec2.internal, executor 31, partition 40, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 41.0 in stage 3.0 (TID 534, ip-10-0-50-96.ec2.internal, executor 17, partition 41, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 42.0 in stage 3.0 (TID 535, ip-10-0-51-185.ec2.internal, executor 25, partition 42, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO TaskSetManager: Starting task 43.0 in stage 3.0 (TID 536, ip-10-0-50-96.ec2.internal, executor 33, partition 43, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:33 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 10.9 KB, free 413.5 MB) | |
| 17/06/14 05:01:33 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 5.5 KB, free 413.5 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 10.0.49.229:33721 (size: 5.5 KB, free: 413.9 MB) | |
| 17/06/14 05:01:33 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:997 | |
| 17/06/14 05:01:33 INFO DAGScheduler: Submitting 135 missing tasks from ShuffleMapStage 5 (MapPartitionsRDD[18] at map at ShredJob.scala:384) | |
| 17/06/14 05:01:33 INFO YarnClusterScheduler: Adding task set 5.0 with 135 tasks | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-51-185.ec2.internal:35857 (size: 5.5 KB, free: 1976.5 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-51-185.ec2.internal:46768 (size: 5.5 KB, free: 1982.8 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-157.ec2.internal:46111 (size: 5.5 KB, free: 1998.3 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-50-96.ec2.internal:43860 (size: 5.5 KB, free: 1983.2 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-229.ec2.internal:38148 (size: 5.5 KB, free: 1976.1 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-51-185.ec2.internal:37170 (size: 5.5 KB, free: 1978.1 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-157.ec2.internal:44837 (size: 5.5 KB, free: 1187.0 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-157.ec2.internal:43440 (size: 5.5 KB, free: 1181.0 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-229.ec2.internal:37408 (size: 5.5 KB, free: 1191.0 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-50-96.ec2.internal:32833 (size: 5.5 KB, free: 2002.4 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-50-96.ec2.internal:45474 (size: 5.5 KB, free: 1198.1 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-50-96.ec2.internal:33453 (size: 5.5 KB, free: 1995.5 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-157.ec2.internal:46402 (size: 5.5 KB, free: 1160.6 MB) | |
| 17/06/14 05:01:33 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-51-185.ec2.internal:43680 (size: 5.5 KB, free: 2017.2 MB) | |
| 17/06/14 05:01:34 INFO YarnAllocator: Driver requested a total number of 15 executor(s). | |
| 17/06/14 05:01:34 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:34 INFO ExecutorAllocationManager: Requesting 15 new executors because tasks are backlogged (new desired total will be 15) | |
| 17/06/14 05:01:34 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:01:34 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022610 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 05:01:34 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:01:34 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:01:34 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 05:01:35 INFO YarnAllocator: Driver requested a total number of 16 executor(s). | |
| 17/06/14 05:01:35 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:35 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 16) | |
| 17/06/14 05:01:35 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:01:35 INFO YarnAllocator: Completed container container_1497410974650_0004_01_002662 on host: ip-10-0-49-157.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:01:35 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:35 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:35 ERROR YarnClusterScheduler: Lost executor 22 on ip-10-0-49-157.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:35 WARN TaskSetManager: Lost task 20.0 in stage 3.0 (TID 516, ip-10-0-49-157.ec2.internal, executor 22): ExecutorLostFailure (executor 22 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:35 WARN TaskSetManager: Lost task 73.0 in stage 3.0 (TID 483, ip-10-0-49-157.ec2.internal, executor 22): ExecutorLostFailure (executor 22 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:35 WARN TaskSetManager: Lost task 81.0 in stage 3.0 (TID 495, ip-10-0-49-157.ec2.internal, executor 22): ExecutorLostFailure (executor 22 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:35 WARN TaskSetManager: Lost task 2.0 in stage 3.0 (TID 502, ip-10-0-49-157.ec2.internal, executor 22): ExecutorLostFailure (executor 22 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:35 INFO DAGScheduler: Executor lost: 22 (epoch 1) | |
| 17/06/14 05:01:35 INFO ExecutorAllocationManager: Existing executor 22 has been removed (new total is 13) | |
| 17/06/14 05:01:35 INFO BlockManagerMasterEndpoint: Trying to remove executor 22 from BlockManagerMaster. | |
| 17/06/14 05:01:35 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(22, ip-10-0-49-157.ec2.internal, 44837, None) | |
| 17/06/14 05:01:35 INFO BlockManagerMaster: Removed 22 successfully in removeExecutor | |
| 17/06/14 05:01:35 WARN ExecutorAllocationManager: Attempted to mark unknown executor 22 idle | |
| 17/06/14 05:01:35 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:35 INFO YarnAllocator: Submitted container request for host ip-10-0-49-157.ec2.internal. | |
| 17/06/14 05:01:35 WARN TransportChannelHandler: Exception in connection from /10.0.49.157:45392 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 05:01:35 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 22. | |
| 17/06/14 05:01:35 ERROR YarnClusterScheduler: Lost an executor 22 (already removed): Pending loss reason. | |
| 17/06/14 05:01:35 INFO BlockManagerMaster: Removal of executor 22 requested | |
| 17/06/14 05:01:35 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 22 | |
| 17/06/14 05:01:35 INFO BlockManagerMasterEndpoint: Trying to remove executor 22 from BlockManagerMaster. | |
| 17/06/14 05:01:36 INFO YarnAllocator: Driver requested a total number of 18 executor(s). | |
| 17/06/14 05:01:36 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:36 INFO YarnAllocator: Submitted 2 unlocalized container requests. | |
| 17/06/14 05:01:36 INFO ExecutorAllocationManager: Requesting 2 new executors because tasks are backlogged (new desired total will be 18) | |
| 17/06/14 05:01:36 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022614 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 05:01:36 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:01:36 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:01:36 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 05:01:36 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:50066) with ID 35 | |
| 17/06/14 05:01:36 INFO TaskSetManager: Starting task 2.1 in stage 3.0 (TID 537, ip-10-0-49-229.ec2.internal, executor 35, partition 2, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:36 INFO TaskSetManager: Starting task 20.1 in stage 3.0 (TID 538, ip-10-0-49-229.ec2.internal, executor 35, partition 20, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:36 INFO TaskSetManager: Starting task 44.0 in stage 3.0 (TID 539, ip-10-0-49-229.ec2.internal, executor 35, partition 44, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:36 INFO TaskSetManager: Starting task 45.0 in stage 3.0 (TID 540, ip-10-0-49-229.ec2.internal, executor 35, partition 45, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:36 INFO ExecutorAllocationManager: New executor 35 has registered (new total is 14) | |
| 17/06/14 05:01:36 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:45629 with 2.8 GB RAM, BlockManagerId(35, ip-10-0-49-229.ec2.internal, 45629, None) | |
| 17/06/14 05:01:37 INFO YarnAllocator: Driver requested a total number of 22 executor(s). | |
| 17/06/14 05:01:37 INFO YarnAllocator: Will request 4 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:37 INFO YarnAllocator: Submitted 4 unlocalized container requests. | |
| 17/06/14 05:01:37 INFO ExecutorAllocationManager: Requesting 4 new executors because tasks are backlogged (new desired total will be 22) | |
| 17/06/14 05:01:37 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-229.ec2.internal:45629 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:01:37 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.229:50066 | |
| 17/06/14 05:01:38 INFO YarnAllocator: Driver requested a total number of 30 executor(s). | |
| 17/06/14 05:01:38 INFO YarnAllocator: Will request 8 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:38 INFO YarnAllocator: Submitted 8 unlocalized container requests. | |
| 17/06/14 05:01:38 INFO ExecutorAllocationManager: Requesting 8 new executors because tasks are backlogged (new desired total will be 30) | |
| 17/06/14 05:01:39 INFO YarnAllocator: Driver requested a total number of 46 executor(s). | |
| 17/06/14 05:01:39 INFO ExecutorAllocationManager: Requesting 16 new executors because tasks are backlogged (new desired total will be 46) | |
| 17/06/14 05:01:39 INFO YarnAllocator: Will request 16 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:39 INFO YarnAllocator: Submitted 16 unlocalized container requests. | |
| 17/06/14 05:01:39 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:45932) with ID 36 | |
| 17/06/14 05:01:39 INFO TaskSetManager: Starting task 81.1 in stage 3.0 (TID 541, ip-10-0-49-157.ec2.internal, executor 36, partition 81, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:39 INFO TaskSetManager: Starting task 73.1 in stage 3.0 (TID 542, ip-10-0-49-157.ec2.internal, executor 36, partition 73, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:39 INFO TaskSetManager: Starting task 46.0 in stage 3.0 (TID 543, ip-10-0-49-157.ec2.internal, executor 36, partition 46, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:39 INFO TaskSetManager: Starting task 47.0 in stage 3.0 (TID 544, ip-10-0-49-157.ec2.internal, executor 36, partition 47, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:39 INFO ExecutorAllocationManager: New executor 36 has registered (new total is 15) | |
| 17/06/14 05:01:39 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:39668 with 2.8 GB RAM, BlockManagerId(36, ip-10-0-49-157.ec2.internal, 39668, None) | |
| 17/06/14 05:01:40 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-157.ec2.internal:39668 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:01:40 INFO YarnAllocator: Driver requested a total number of 68 executor(s). | |
| 17/06/14 05:01:40 INFO YarnAllocator: Will request 22 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:40 INFO YarnAllocator: Submitted 22 unlocalized container requests. | |
| 17/06/14 05:01:40 INFO ExecutorAllocationManager: Requesting 22 new executors because tasks are backlogged (new desired total will be 68) | |
| 17/06/14 05:01:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.157:45932 | |
| 17/06/14 05:01:43 INFO TaskSetManager: Starting task 49.0 in stage 3.0 (TID 545, ip-10-0-49-229.ec2.internal, executor 28, partition 49, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:43 INFO TaskSetManager: Finished task 112.0 in stage 3.0 (TID 496) in 10488 ms on ip-10-0-49-229.ec2.internal (executor 28) (1/135) | |
| 17/06/14 05:01:43 INFO TaskSetManager: Starting task 50.0 in stage 3.0 (TID 546, ip-10-0-49-229.ec2.internal, executor 28, partition 50, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:43 INFO TaskSetManager: Finished task 111.0 in stage 3.0 (TID 488) in 10829 ms on ip-10-0-49-229.ec2.internal (executor 28) (2/135) | |
| 17/06/14 05:01:43 INFO YarnAllocator: Driver requested a total number of 67 executor(s). | |
| 17/06/14 05:01:43 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 67 executors. | |
| 17/06/14 05:01:45 INFO TaskSetManager: Starting task 51.0 in stage 3.0 (TID 547, ip-10-0-50-96.ec2.internal, executor 32, partition 51, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:45 INFO TaskSetManager: Finished task 129.0 in stage 3.0 (TID 497) in 11986 ms on ip-10-0-50-96.ec2.internal (executor 32) (3/135) | |
| 17/06/14 05:01:45 INFO TaskSetManager: Starting task 52.0 in stage 3.0 (TID 548, ip-10-0-50-96.ec2.internal, executor 32, partition 52, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:45 INFO TaskSetManager: Finished task 66.0 in stage 3.0 (TID 489) in 12340 ms on ip-10-0-50-96.ec2.internal (executor 32) (4/135) | |
| 17/06/14 05:01:45 INFO TaskSetManager: Starting task 53.0 in stage 3.0 (TID 549, ip-10-0-49-229.ec2.internal, executor 34, partition 53, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:45 INFO TaskSetManager: Finished task 75.0 in stage 3.0 (TID 481) in 12922 ms on ip-10-0-49-229.ec2.internal (executor 34) (5/135) | |
| 17/06/14 05:01:45 INFO TaskSetManager: Starting task 54.0 in stage 3.0 (TID 550, ip-10-0-49-157.ec2.internal, executor 29, partition 54, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:45 INFO TaskSetManager: Finished task 134.0 in stage 3.0 (TID 499) in 12935 ms on ip-10-0-49-157.ec2.internal (executor 29) (6/135) | |
| 17/06/14 05:01:46 INFO YarnAllocator: Driver requested a total number of 66 executor(s). | |
| 17/06/14 05:01:46 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 66 executors. | |
| 17/06/14 05:01:46 INFO TaskSetManager: Starting task 55.0 in stage 3.0 (TID 551, ip-10-0-49-157.ec2.internal, executor 29, partition 55, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:46 INFO TaskSetManager: Finished task 48.0 in stage 3.0 (TID 493) in 13032 ms on ip-10-0-49-157.ec2.internal (executor 29) (7/135) | |
| 17/06/14 05:01:46 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-50-96.ec2.internal:45474 (size: 5.5 KB) | |
| 17/06/14 05:01:46 INFO BlockManagerInfo: Removed rdd_10_129 on ip-10-0-50-96.ec2.internal:45474 in memory (size: 788.4 MB, free: 1986.6 MB) | |
| 17/06/14 05:01:46 INFO TaskSetManager: Starting task 56.0 in stage 3.0 (TID 552, ip-10-0-51-185.ec2.internal, executor 30, partition 56, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:46 INFO TaskSetManager: Finished task 99.0 in stage 3.0 (TID 482) in 13671 ms on ip-10-0-51-185.ec2.internal (executor 30) (8/135) | |
| 17/06/14 05:01:46 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-49-157.ec2.internal:46402 (size: 33.3 KB) | |
| 17/06/14 05:01:46 INFO BlockManagerInfo: Removed rdd_10_134 on ip-10-0-49-157.ec2.internal:46402 in memory (size: 828.0 MB, free: 1988.7 MB) | |
| 17/06/14 05:01:47 INFO BlockManagerInfo: Added broadcast_2_piece0 on disk on ip-10-0-49-229.ec2.internal:37408 (size: 33.3 KB) | |
| 17/06/14 05:01:47 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-49-229.ec2.internal:37408 (size: 5.5 KB) | |
| 17/06/14 05:01:47 INFO BlockManagerInfo: Removed rdd_10_112 on ip-10-0-49-229.ec2.internal:37408 in memory (size: 829.0 MB, free: 2020.0 MB) | |
| 17/06/14 05:01:51 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 20. | |
| 17/06/14 05:01:51 INFO DAGScheduler: Executor lost: 20 (epoch 1) | |
| 17/06/14 05:01:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 20 from BlockManagerMaster. | |
| 17/06/14 05:01:51 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(20, ip-10-0-51-185.ec2.internal, 37170, None) | |
| 17/06/14 05:01:51 INFO BlockManagerMaster: Removed 20 successfully in removeExecutor | |
| 17/06/14 05:01:51 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022659 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 05:01:51 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:01:51 INFO YarnAllocator: Completed container container_1497410974650_0004_01_002656 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:01:51 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:51 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:01:51 ERROR YarnClusterScheduler: Lost executor 20 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:51 WARN TaskSetManager: Lost task 6.0 in stage 3.0 (TID 505, ip-10-0-51-185.ec2.internal, executor 20): ExecutorLostFailure (executor 20 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:51 WARN TaskSetManager: Lost task 39.0 in stage 3.0 (TID 532, ip-10-0-51-185.ec2.internal, executor 20): ExecutorLostFailure (executor 20 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:51 WARN TaskSetManager: Lost task 24.0 in stage 3.0 (TID 519, ip-10-0-51-185.ec2.internal, executor 20): ExecutorLostFailure (executor 20 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:51 WARN TaskSetManager: Lost task 18.0 in stage 3.0 (TID 486, ip-10-0-51-185.ec2.internal, executor 20): ExecutorLostFailure (executor 20 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:51 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:51 INFO BlockManagerMaster: Removal of executor 20 requested | |
| 17/06/14 05:01:51 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 20 | |
| 17/06/14 05:01:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 20 from BlockManagerMaster. | |
| 17/06/14 05:01:51 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 05:01:51 INFO ExecutorAllocationManager: Existing executor 20 has been removed (new total is 14) | |
| 17/06/14 05:01:51 WARN ExecutorAllocationManager: Attempted to mark unknown executor 20 idle | |
| 17/06/14 05:01:51 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 31. | |
| 17/06/14 05:01:51 INFO DAGScheduler: Executor lost: 31 (epoch 1) | |
| 17/06/14 05:01:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 31 from BlockManagerMaster. | |
| 17/06/14 05:01:51 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(31, ip-10-0-51-185.ec2.internal, 35857, None) | |
| 17/06/14 05:01:51 INFO BlockManagerMaster: Removed 31 successfully in removeExecutor | |
| 17/06/14 05:01:51 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 25. | |
| 17/06/14 05:01:51 INFO DAGScheduler: Executor lost: 25 (epoch 1) | |
| 17/06/14 05:01:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 25 from BlockManagerMaster. | |
| 17/06/14 05:01:51 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(25, ip-10-0-51-185.ec2.internal, 43680, None) | |
| 17/06/14 05:01:51 INFO BlockManagerMaster: Removed 25 successfully in removeExecutor | |
| 17/06/14 05:01:51 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:51 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:01:52 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022662 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 05:01:52 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:01:52 INFO YarnAllocator: Completed container container_1497410974650_0004_01_003564 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:01:52 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:52 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:52 INFO YarnAllocator: Completed container container_1497410974650_0004_01_002701 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:01:52 ERROR YarnClusterScheduler: Lost executor 31 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:52 WARN TaskSetManager: Lost task 22.0 in stage 3.0 (TID 487, ip-10-0-51-185.ec2.internal, executor 31): ExecutorLostFailure (executor 31 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:52 WARN TaskSetManager: Lost task 7.0 in stage 3.0 (TID 506, ip-10-0-51-185.ec2.internal, executor 31): ExecutorLostFailure (executor 31 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:52 WARN TaskSetManager: Lost task 40.0 in stage 3.0 (TID 533, ip-10-0-51-185.ec2.internal, executor 31): ExecutorLostFailure (executor 31 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:52 WARN TaskSetManager: Lost task 26.0 in stage 3.0 (TID 520, ip-10-0-51-185.ec2.internal, executor 31): ExecutorLostFailure (executor 31 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:52 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:01:52 INFO BlockManagerMaster: Removal of executor 31 requested | |
| 17/06/14 05:01:52 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 31 | |
| 17/06/14 05:01:52 INFO BlockManagerMasterEndpoint: Trying to remove executor 31 from BlockManagerMaster. | |
| 17/06/14 05:01:52 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:52 ERROR YarnClusterScheduler: Lost executor 25 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:52 WARN TaskSetManager: Lost task 31.0 in stage 3.0 (TID 525, ip-10-0-51-185.ec2.internal, executor 25): ExecutorLostFailure (executor 25 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:52 WARN TaskSetManager: Lost task 25.0 in stage 3.0 (TID 492, ip-10-0-51-185.ec2.internal, executor 25): ExecutorLostFailure (executor 25 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:52 WARN TaskSetManager: Lost task 42.0 in stage 3.0 (TID 535, ip-10-0-51-185.ec2.internal, executor 25): ExecutorLostFailure (executor 25 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:52 WARN TaskSetManager: Lost task 14.0 in stage 3.0 (TID 511, ip-10-0-51-185.ec2.internal, executor 25): ExecutorLostFailure (executor 25 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:52 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:52 INFO BlockManagerMaster: Removal of executor 25 requested | |
| 17/06/14 05:01:52 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 25 | |
| 17/06/14 05:01:52 INFO BlockManagerMasterEndpoint: Trying to remove executor 25 from BlockManagerMaster. | |
| 17/06/14 05:01:52 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 05:01:52 INFO ExecutorAllocationManager: Existing executor 31 has been removed (new total is 13) | |
| 17/06/14 05:01:52 INFO ExecutorAllocationManager: Existing executor 25 has been removed (new total is 12) | |
| 17/06/14 05:01:52 WARN ExecutorAllocationManager: Attempted to mark unknown executor 25 idle | |
| 17/06/14 05:01:52 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 17. | |
| 17/06/14 05:01:52 INFO DAGScheduler: Executor lost: 17 (epoch 1) | |
| 17/06/14 05:01:52 INFO BlockManagerMasterEndpoint: Trying to remove executor 17 from BlockManagerMaster. | |
| 17/06/14 05:01:52 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:52 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(17, ip-10-0-50-96.ec2.internal, 33453, None) | |
| 17/06/14 05:01:52 INFO BlockManagerMaster: Removed 17 successfully in removeExecutor | |
| 17/06/14 05:01:52 INFO YarnAllocator: Submitted 2 unlocalized container requests. | |
| 17/06/14 05:01:52 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 18. | |
| 17/06/14 05:01:52 INFO DAGScheduler: Executor lost: 18 (epoch 1) | |
| 17/06/14 05:01:52 INFO BlockManagerMasterEndpoint: Trying to remove executor 18 from BlockManagerMaster. | |
| 17/06/14 05:01:52 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(18, ip-10-0-50-96.ec2.internal, 43860, None) | |
| 17/06/14 05:01:52 INFO BlockManagerMaster: Removed 18 successfully in removeExecutor | |
| 17/06/14 05:01:52 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022665 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 05:01:52 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:01:52 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:01:52 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 05:01:53 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022666 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 05:01:53 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:01:53 INFO YarnAllocator: Completed container container_1497410974650_0004_01_002682 on host: ip-10-0-49-157.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:01:53 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 ERROR YarnClusterScheduler: Lost executor 23 on ip-10-0-49-157.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 WARN TaskSetManager: Lost task 21.0 in stage 3.0 (TID 517, ip-10-0-49-157.ec2.internal, executor 23): ExecutorLostFailure (executor 23 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 WARN TaskSetManager: Lost task 37.0 in stage 3.0 (TID 530, ip-10-0-49-157.ec2.internal, executor 23): ExecutorLostFailure (executor 23 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 WARN TaskSetManager: Lost task 3.0 in stage 3.0 (TID 503, ip-10-0-49-157.ec2.internal, executor 23): ExecutorLostFailure (executor 23 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:01:53 WARN TaskSetManager: Lost task 5.0 in stage 3.0 (TID 484, ip-10-0-49-157.ec2.internal, executor 23): ExecutorLostFailure (executor 23 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 INFO DAGScheduler: Executor lost: 23 (epoch 1) | |
| 17/06/14 05:01:53 INFO BlockManagerMasterEndpoint: Trying to remove executor 23 from BlockManagerMaster. | |
| 17/06/14 05:01:53 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(23, ip-10-0-49-157.ec2.internal, 46111, None) | |
| 17/06/14 05:01:53 INFO BlockManagerMaster: Removed 23 successfully in removeExecutor | |
| 17/06/14 05:01:53 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 05:01:53 INFO ExecutorAllocationManager: Existing executor 23 has been removed (new total is 11) | |
| 17/06/14 05:01:53 WARN ExecutorAllocationManager: Attempted to mark unknown executor 23 idle | |
| 17/06/14 05:01:53 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:35570) with ID 37 | |
| 17/06/14 05:01:53 INFO TaskSetManager: Starting task 25.1 in stage 3.0 (TID 553, ip-10-0-51-185.ec2.internal, executor 37, partition 25, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:53 INFO TaskSetManager: Starting task 22.1 in stage 3.0 (TID 554, ip-10-0-51-185.ec2.internal, executor 37, partition 22, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:53 INFO TaskSetManager: Starting task 18.1 in stage 3.0 (TID 555, ip-10-0-51-185.ec2.internal, executor 37, partition 18, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:53 INFO TaskSetManager: Starting task 3.1 in stage 3.0 (TID 556, ip-10-0-51-185.ec2.internal, executor 37, partition 3, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:53 INFO ExecutorAllocationManager: New executor 37 has registered (new total is 12) | |
| 17/06/14 05:01:53 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:33788 with 2.8 GB RAM, BlockManagerId(37, ip-10-0-51-185.ec2.internal, 33788, None) | |
| 17/06/14 05:01:53 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:53 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:01:53 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022667 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 05:01:53 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:01:53 INFO YarnAllocator: Completed container container_1497410974650_0004_01_002535 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:01:53 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:01:53 INFO YarnAllocator: Completed container container_1497410974650_0004_01_002509 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:01:53 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 05:01:53 ERROR YarnClusterScheduler: Lost executor 18 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 WARN TaskSetManager: Lost task 38.0 in stage 3.0 (TID 531, ip-10-0-50-96.ec2.internal, executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 WARN TaskSetManager: Lost task 4.0 in stage 3.0 (TID 504, ip-10-0-50-96.ec2.internal, executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 WARN TaskSetManager: Lost task 23.0 in stage 3.0 (TID 518, ip-10-0-50-96.ec2.internal, executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 WARN TaskSetManager: Lost task 33.0 in stage 3.0 (TID 485, ip-10-0-50-96.ec2.internal, executor 18): ExecutorLostFailure (executor 18 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 ERROR YarnClusterScheduler: Lost executor 17 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 WARN TaskSetManager: Lost task 41.0 in stage 3.0 (TID 534, ip-10-0-50-96.ec2.internal, executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 WARN TaskSetManager: Lost task 13.0 in stage 3.0 (TID 510, ip-10-0-50-96.ec2.internal, executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 WARN TaskSetManager: Lost task 30.0 in stage 3.0 (TID 524, ip-10-0-50-96.ec2.internal, executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 WARN TaskSetManager: Lost task 9.0 in stage 3.0 (TID 491, ip-10-0-50-96.ec2.internal, executor 17): ExecutorLostFailure (executor 17 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:53 INFO BlockManagerMaster: Removal of executor 18 requested | |
| 17/06/14 05:01:53 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 18 | |
| 17/06/14 05:01:53 INFO BlockManagerMaster: Removal of executor 17 requested | |
| 17/06/14 05:01:53 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 17 | |
| 17/06/14 05:01:53 INFO BlockManagerMasterEndpoint: Trying to remove executor 18 from BlockManagerMaster. | |
| 17/06/14 05:01:53 INFO BlockManagerMasterEndpoint: Trying to remove executor 17 from BlockManagerMaster. | |
| 17/06/14 05:01:53 INFO ExecutorAllocationManager: Existing executor 18 has been removed (new total is 11) | |
| 17/06/14 05:01:53 INFO ExecutorAllocationManager: Existing executor 17 has been removed (new total is 10) | |
| 17/06/14 05:01:53 WARN ExecutorAllocationManager: Attempted to mark unknown executor 18 idle | |
| 17/06/14 05:01:53 WARN ExecutorAllocationManager: Attempted to mark unknown executor 17 idle | |
| 17/06/14 05:01:53 WARN TransportChannelHandler: Exception in connection from /10.0.49.157:45410 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 05:01:53 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 23. | |
| 17/06/14 05:01:53 ERROR YarnClusterScheduler: Lost an executor 23 (already removed): Pending loss reason. | |
| 17/06/14 05:01:53 INFO BlockManagerMaster: Removal of executor 23 requested | |
| 17/06/14 05:01:53 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 23 | |
| 17/06/14 05:01:53 INFO BlockManagerMasterEndpoint: Trying to remove executor 23 from BlockManagerMaster. | |
| 17/06/14 05:01:53 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:53 INFO YarnAllocator: Submitted 2 unlocalized container requests. | |
| 17/06/14 05:01:54 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:35578) with ID 38 | |
| 17/06/14 05:01:54 INFO TaskSetManager: Starting task 30.1 in stage 3.0 (TID 557, ip-10-0-51-185.ec2.internal, executor 38, partition 30, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:54 INFO TaskSetManager: Starting task 13.1 in stage 3.0 (TID 558, ip-10-0-51-185.ec2.internal, executor 38, partition 13, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:54 INFO TaskSetManager: Starting task 41.1 in stage 3.0 (TID 559, ip-10-0-51-185.ec2.internal, executor 38, partition 41, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:54 INFO TaskSetManager: Starting task 23.1 in stage 3.0 (TID 560, ip-10-0-51-185.ec2.internal, executor 38, partition 23, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:54 INFO ExecutorAllocationManager: New executor 38 has registered (new total is 11) | |
| 17/06/14 05:01:54 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:37560 with 2.8 GB RAM, BlockManagerId(38, ip-10-0-51-185.ec2.internal, 37560, None) | |
| 17/06/14 05:01:54 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022670 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 05:01:54 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:01:54 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:01:54 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 05:01:54 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:35586) with ID 39 | |
| 17/06/14 05:01:54 INFO TaskSetManager: Starting task 4.1 in stage 3.0 (TID 561, ip-10-0-51-185.ec2.internal, executor 39, partition 4, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:54 INFO TaskSetManager: Starting task 38.1 in stage 3.0 (TID 562, ip-10-0-51-185.ec2.internal, executor 39, partition 38, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:54 INFO TaskSetManager: Starting task 37.1 in stage 3.0 (TID 563, ip-10-0-51-185.ec2.internal, executor 39, partition 37, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:54 INFO TaskSetManager: Starting task 21.1 in stage 3.0 (TID 564, ip-10-0-51-185.ec2.internal, executor 39, partition 21, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:54 INFO ExecutorAllocationManager: New executor 39 has registered (new total is 12) | |
| 17/06/14 05:01:55 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:46252 with 2.8 GB RAM, BlockManagerId(39, ip-10-0-51-185.ec2.internal, 46252, None) | |
| 17/06/14 05:01:55 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-51-185.ec2.internal:46252 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:01:55 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-51-185.ec2.internal:33788 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:01:55 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-51-185.ec2.internal:37560 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:01:55 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 32. | |
| 17/06/14 05:01:55 INFO DAGScheduler: Executor lost: 32 (epoch 1) | |
| 17/06/14 05:01:55 INFO BlockManagerMasterEndpoint: Trying to remove executor 32 from BlockManagerMaster. | |
| 17/06/14 05:01:55 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(32, ip-10-0-50-96.ec2.internal, 45474, None) | |
| 17/06/14 05:01:55 INFO BlockManagerMaster: Removed 32 successfully in removeExecutor | |
| 17/06/14 05:01:55 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022674 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 05:01:55 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:01:55 INFO YarnAllocator: Completed container container_1497410974650_0004_01_010388 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:01:55 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:55 ERROR YarnClusterScheduler: Lost executor 32 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:55 WARN TaskSetManager: Lost task 11.0 in stage 3.0 (TID 508, ip-10-0-50-96.ec2.internal, executor 32): ExecutorLostFailure (executor 32 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:55 WARN TaskSetManager: Lost task 28.0 in stage 3.0 (TID 522, ip-10-0-50-96.ec2.internal, executor 32): ExecutorLostFailure (executor 32 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:55 WARN TaskSetManager: Lost task 52.0 in stage 3.0 (TID 548, ip-10-0-50-96.ec2.internal, executor 32): ExecutorLostFailure (executor 32 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:55 WARN TaskSetManager: Lost task 51.0 in stage 3.0 (TID 547, ip-10-0-50-96.ec2.internal, executor 32): ExecutorLostFailure (executor 32 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:55 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:01:55 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:55 INFO BlockManagerMaster: Removal of executor 32 requested | |
| 17/06/14 05:01:55 INFO BlockManagerMasterEndpoint: Trying to remove executor 32 from BlockManagerMaster. | |
| 17/06/14 05:01:55 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 32 | |
| 17/06/14 05:01:55 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 05:01:55 INFO ExecutorAllocationManager: Existing executor 32 has been removed (new total is 11) | |
| 17/06/14 05:01:56 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:36892) with ID 41 | |
| 17/06/14 05:01:56 INFO TaskSetManager: Starting task 9.1 in stage 3.0 (TID 565, ip-10-0-50-96.ec2.internal, executor 41, partition 9, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:56 INFO TaskSetManager: Starting task 33.1 in stage 3.0 (TID 566, ip-10-0-50-96.ec2.internal, executor 41, partition 33, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:56 INFO TaskSetManager: Starting task 51.1 in stage 3.0 (TID 567, ip-10-0-50-96.ec2.internal, executor 41, partition 51, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:56 INFO TaskSetManager: Starting task 52.1 in stage 3.0 (TID 568, ip-10-0-50-96.ec2.internal, executor 41, partition 52, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:56 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:56 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:01:56 INFO ExecutorAllocationManager: New executor 41 has registered (new total is 12) | |
| 17/06/14 05:01:56 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:37989 with 2.8 GB RAM, BlockManagerId(41, ip-10-0-50-96.ec2.internal, 37989, None) | |
| 17/06/14 05:01:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:35578 | |
| 17/06/14 05:01:56 INFO TaskSetManager: Starting task 5.1 in stage 3.0 (TID 569, ip-10-0-49-157.ec2.internal, executor 24, partition 5, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:56 INFO TaskSetManager: Finished task 131.0 in stage 3.0 (TID 498) in 23297 ms on ip-10-0-49-157.ec2.internal (executor 24) (9/135) | |
| 17/06/14 05:01:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:35586 | |
| 17/06/14 05:01:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:35570 | |
| 17/06/14 05:01:56 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022677 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 05:01:56 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:01:56 INFO YarnAllocator: Completed container container_1497410974650_0004_01_003122 on host: ip-10-0-49-157.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:01:56 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:56 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:56 ERROR YarnClusterScheduler: Lost executor 29 on ip-10-0-49-157.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:56 WARN TaskSetManager: Lost task 54.0 in stage 3.0 (TID 550, ip-10-0-49-157.ec2.internal, executor 29): ExecutorLostFailure (executor 29 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:56 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:01:56 WARN TaskSetManager: Lost task 15.0 in stage 3.0 (TID 512, ip-10-0-49-157.ec2.internal, executor 29): ExecutorLostFailure (executor 29 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:56 WARN TaskSetManager: Lost task 55.0 in stage 3.0 (TID 551, ip-10-0-49-157.ec2.internal, executor 29): ExecutorLostFailure (executor 29 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:56 WARN TaskSetManager: Lost task 32.0 in stage 3.0 (TID 526, ip-10-0-49-157.ec2.internal, executor 29): ExecutorLostFailure (executor 29 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:56 INFO DAGScheduler: Executor lost: 29 (epoch 1) | |
| 17/06/14 05:01:56 INFO BlockManagerMasterEndpoint: Trying to remove executor 29 from BlockManagerMaster. | |
| 17/06/14 05:01:56 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(29, ip-10-0-49-157.ec2.internal, 46402, None) | |
| 17/06/14 05:01:56 INFO BlockManagerMaster: Removed 29 successfully in removeExecutor | |
| 17/06/14 05:01:56 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 05:01:56 INFO ExecutorAllocationManager: Existing executor 29 has been removed (new total is 11) | |
| 17/06/14 05:01:56 WARN TransportChannelHandler: Exception in connection from /10.0.49.157:45518 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 05:01:56 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 29. | |
| 17/06/14 05:01:56 ERROR YarnClusterScheduler: Lost an executor 29 (already removed): Pending loss reason. | |
| 17/06/14 05:01:56 INFO BlockManagerMaster: Removal of executor 29 requested | |
| 17/06/14 05:01:56 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 29 | |
| 17/06/14 05:01:56 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:56 INFO BlockManagerMasterEndpoint: Trying to remove executor 29 from BlockManagerMaster. | |
| 17/06/14 05:01:56 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:01:56 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:36900) with ID 42 | |
| 17/06/14 05:01:56 INFO TaskSetManager: Starting task 32.1 in stage 3.0 (TID 570, ip-10-0-50-96.ec2.internal, executor 42, partition 32, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:56 INFO TaskSetManager: Starting task 55.1 in stage 3.0 (TID 571, ip-10-0-50-96.ec2.internal, executor 42, partition 55, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:56 INFO TaskSetManager: Starting task 15.1 in stage 3.0 (TID 572, ip-10-0-50-96.ec2.internal, executor 42, partition 15, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:56 INFO TaskSetManager: Starting task 54.1 in stage 3.0 (TID 573, ip-10-0-50-96.ec2.internal, executor 42, partition 54, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:56 INFO ExecutorAllocationManager: New executor 42 has registered (new total is 12) | |
| 17/06/14 05:01:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-50-96.ec2.internal:37989 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:01:56 INFO TaskSetManager: Starting task 28.1 in stage 3.0 (TID 574, ip-10-0-50-96.ec2.internal, executor 33, partition 28, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:56 INFO TaskSetManager: Finished task 63.0 in stage 3.0 (TID 494) in 23928 ms on ip-10-0-50-96.ec2.internal (executor 33) (10/135) | |
| 17/06/14 05:01:56 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:42160 with 2.8 GB RAM, BlockManagerId(42, ip-10-0-50-96.ec2.internal, 42160, None) | |
| 17/06/14 05:01:57 INFO YarnAllocator: Driver requested a total number of 65 executor(s). | |
| 17/06/14 05:01:57 INFO TaskSetManager: Starting task 11.1 in stage 3.0 (TID 575, ip-10-0-49-157.ec2.internal, executor 24, partition 11, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:57 INFO TaskSetManager: Finished task 10.0 in stage 3.0 (TID 490) in 24040 ms on ip-10-0-49-157.ec2.internal (executor 24) (11/135) | |
| 17/06/14 05:01:57 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:45956) with ID 40 | |
| 17/06/14 05:01:57 INFO TaskSetManager: Starting task 14.1 in stage 3.0 (TID 576, ip-10-0-49-157.ec2.internal, executor 40, partition 14, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:57 INFO TaskSetManager: Starting task 42.1 in stage 3.0 (TID 577, ip-10-0-49-157.ec2.internal, executor 40, partition 42, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:57 INFO TaskSetManager: Starting task 31.1 in stage 3.0 (TID 578, ip-10-0-49-157.ec2.internal, executor 40, partition 31, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:57 INFO TaskSetManager: Starting task 26.1 in stage 3.0 (TID 579, ip-10-0-49-157.ec2.internal, executor 40, partition 26, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:57 INFO ExecutorAllocationManager: New executor 40 has registered (new total is 13) | |
| 17/06/14 05:01:57 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 65 executors. | |
| 17/06/14 05:01:57 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:42968 with 2.8 GB RAM, BlockManagerId(40, ip-10-0-49-157.ec2.internal, 42968, None) | |
| 17/06/14 05:01:57 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-50-96.ec2.internal:42160 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:01:57 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:36892 | |
| 17/06/14 05:01:57 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-157.ec2.internal:42968 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:01:58 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:36900 | |
| 17/06/14 05:01:58 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022685 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 05:01:58 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:01:58 INFO YarnAllocator: Completed container container_1497410974650_0004_01_010519 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:01:58 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:58 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:58 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:01:58 ERROR YarnClusterScheduler: Lost executor 33 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:58 WARN TaskSetManager: Lost task 16.0 in stage 3.0 (TID 513, ip-10-0-50-96.ec2.internal, executor 33): ExecutorLostFailure (executor 33 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:58 WARN TaskSetManager: Lost task 34.0 in stage 3.0 (TID 527, ip-10-0-50-96.ec2.internal, executor 33): ExecutorLostFailure (executor 33 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:58 WARN TaskSetManager: Lost task 43.0 in stage 3.0 (TID 536, ip-10-0-50-96.ec2.internal, executor 33): ExecutorLostFailure (executor 33 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:58 WARN TaskSetManager: Lost task 28.1 in stage 3.0 (TID 574, ip-10-0-50-96.ec2.internal, executor 33): ExecutorLostFailure (executor 33 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:58 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 05:01:58 INFO DAGScheduler: Executor lost: 33 (epoch 1) | |
| 17/06/14 05:01:58 INFO BlockManagerMasterEndpoint: Trying to remove executor 33 from BlockManagerMaster. | |
| 17/06/14 05:01:58 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(33, ip-10-0-50-96.ec2.internal, 32833, None) | |
| 17/06/14 05:01:58 INFO BlockManagerMaster: Removed 33 successfully in removeExecutor | |
| 17/06/14 05:01:58 INFO ExecutorAllocationManager: Existing executor 33 has been removed (new total is 12) | |
| 17/06/14 05:01:58 WARN ExecutorAllocationManager: Attempted to mark unknown executor 33 idle | |
| 17/06/14 05:01:58 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.157:45956 | |
| 17/06/14 05:01:58 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:36924) with ID 43 | |
| 17/06/14 05:01:58 INFO TaskSetManager: Starting task 28.2 in stage 3.0 (TID 580, ip-10-0-50-96.ec2.internal, executor 43, partition 28, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:58 INFO TaskSetManager: Starting task 43.1 in stage 3.0 (TID 581, ip-10-0-50-96.ec2.internal, executor 43, partition 43, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:58 INFO TaskSetManager: Starting task 34.1 in stage 3.0 (TID 582, ip-10-0-50-96.ec2.internal, executor 43, partition 34, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:58 INFO TaskSetManager: Starting task 16.1 in stage 3.0 (TID 583, ip-10-0-50-96.ec2.internal, executor 43, partition 16, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:01:58 INFO ExecutorAllocationManager: New executor 43 has registered (new total is 13) | |
| 17/06/14 05:01:59 WARN TransportChannelHandler: Exception in connection from /10.0.50.96:36566 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 05:01:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 33. | |
| 17/06/14 05:01:59 ERROR YarnClusterScheduler: Lost an executor 33 (already removed): Pending loss reason. | |
| 17/06/14 05:01:59 INFO BlockManagerMaster: Removal of executor 33 requested | |
| 17/06/14 05:01:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 33 | |
| 17/06/14 05:01:59 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:59 INFO BlockManagerMasterEndpoint: Trying to remove executor 33 from BlockManagerMaster. | |
| 17/06/14 05:01:59 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:01:59 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:39293 with 2.8 GB RAM, BlockManagerId(43, ip-10-0-50-96.ec2.internal, 39293, None) | |
| 17/06/14 05:01:59 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022611 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 05:01:59 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:01:59 INFO YarnAllocator: Completed container container_1497410974650_0004_01_003009 on host: ip-10-0-49-229.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:01:59 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:59 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:59 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:01:59 ERROR YarnClusterScheduler: Lost executor 34 on ip-10-0-49-229.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:59 WARN TaskSetManager: Lost task 17.0 in stage 3.0 (TID 514, ip-10-0-49-229.ec2.internal, executor 34): ExecutorLostFailure (executor 34 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:59 WARN TaskSetManager: Lost task 53.0 in stage 3.0 (TID 549, ip-10-0-49-229.ec2.internal, executor 34): ExecutorLostFailure (executor 34 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:59 WARN TaskSetManager: Lost task 35.0 in stage 3.0 (TID 528, ip-10-0-49-229.ec2.internal, executor 34): ExecutorLostFailure (executor 34 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:59 WARN TaskSetManager: Lost task 0.0 in stage 3.0 (TID 500, ip-10-0-49-229.ec2.internal, executor 34): ExecutorLostFailure (executor 34 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:59 INFO DAGScheduler: Executor lost: 34 (epoch 1) | |
| 17/06/14 05:01:59 INFO BlockManagerMasterEndpoint: Trying to remove executor 34 from BlockManagerMaster. | |
| 17/06/14 05:01:59 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(34, ip-10-0-49-229.ec2.internal, 38148, None) | |
| 17/06/14 05:01:59 INFO BlockManagerMaster: Removed 34 successfully in removeExecutor | |
| 17/06/14 05:01:59 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 05:01:59 INFO ExecutorAllocationManager: Existing executor 34 has been removed (new total is 12) | |
| 17/06/14 05:01:59 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-50-96.ec2.internal:39293 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:01:59 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:59 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:01:59 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022689 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 05:01:59 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:01:59 INFO YarnAllocator: Completed container container_1497410974650_0004_01_002684 on host: ip-10-0-49-157.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:01:59 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:59 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:59 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:01:59 ERROR YarnClusterScheduler: Lost executor 24 on ip-10-0-49-157.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:59 WARN TaskSetManager: Lost task 29.0 in stage 3.0 (TID 523, ip-10-0-49-157.ec2.internal, executor 24): ExecutorLostFailure (executor 24 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:59 WARN TaskSetManager: Lost task 11.1 in stage 3.0 (TID 575, ip-10-0-49-157.ec2.internal, executor 24): ExecutorLostFailure (executor 24 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:59 WARN TaskSetManager: Lost task 5.1 in stage 3.0 (TID 569, ip-10-0-49-157.ec2.internal, executor 24): ExecutorLostFailure (executor 24 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:59 WARN TaskSetManager: Lost task 12.0 in stage 3.0 (TID 509, ip-10-0-49-157.ec2.internal, executor 24): ExecutorLostFailure (executor 24 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:01:59 INFO DAGScheduler: Executor lost: 24 (epoch 1) | |
| 17/06/14 05:01:59 INFO BlockManagerMasterEndpoint: Trying to remove executor 24 from BlockManagerMaster. | |
| 17/06/14 05:01:59 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(24, ip-10-0-49-157.ec2.internal, 43440, None) | |
| 17/06/14 05:01:59 INFO BlockManagerMaster: Removed 24 successfully in removeExecutor | |
| 17/06/14 05:01:59 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 05:01:59 INFO ExecutorAllocationManager: Existing executor 24 has been removed (new total is 11) | |
| 17/06/14 05:01:59 WARN TransportChannelHandler: Exception in connection from /10.0.49.229:49748 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 05:01:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 34. | |
| 17/06/14 05:01:59 ERROR YarnClusterScheduler: Lost an executor 34 (already removed): Pending loss reason. | |
| 17/06/14 05:01:59 INFO BlockManagerMaster: Removal of executor 34 requested | |
| 17/06/14 05:01:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 34 | |
| 17/06/14 05:01:59 INFO BlockManagerMasterEndpoint: Trying to remove executor 34 from BlockManagerMaster. | |
| 17/06/14 05:01:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 28. | |
| 17/06/14 05:01:59 INFO DAGScheduler: Executor lost: 28 (epoch 1) | |
| 17/06/14 05:01:59 INFO BlockManagerMasterEndpoint: Trying to remove executor 28 from BlockManagerMaster. | |
| 17/06/14 05:01:59 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(28, ip-10-0-49-229.ec2.internal, 37408, None) | |
| 17/06/14 05:01:59 INFO BlockManagerMaster: Removed 28 successfully in removeExecutor | |
| 17/06/14 05:01:59 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:01:59 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:01:59 WARN TransportChannelHandler: Exception in connection from /10.0.49.157:45412 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 05:01:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 24. | |
| 17/06/14 05:01:59 ERROR YarnClusterScheduler: Lost an executor 24 (already removed): Pending loss reason. | |
| 17/06/14 05:01:59 INFO BlockManagerMaster: Removal of executor 24 requested | |
| 17/06/14 05:01:59 INFO BlockManagerMasterEndpoint: Trying to remove executor 24 from BlockManagerMaster. | |
| 17/06/14 05:01:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 24 | |
| 17/06/14 05:02:00 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:36924 | |
| 17/06/14 05:02:00 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022692 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 05:02:00 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:02:00 INFO YarnAllocator: Completed container container_1497410974650_0004_01_003163 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:02:00 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:02:00 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:02:00 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:02:00 ERROR YarnClusterScheduler: Lost executor 30 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:02:00 WARN TaskSetManager: Lost task 56.0 in stage 3.0 (TID 552, ip-10-0-51-185.ec2.internal, executor 30): ExecutorLostFailure (executor 30 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:02:00 WARN TaskSetManager: Lost task 1.0 in stage 3.0 (TID 501, ip-10-0-51-185.ec2.internal, executor 30): ExecutorLostFailure (executor 30 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:02:00 WARN TaskSetManager: Lost task 19.0 in stage 3.0 (TID 515, ip-10-0-51-185.ec2.internal, executor 30): ExecutorLostFailure (executor 30 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:02:00 WARN TaskSetManager: Lost task 36.0 in stage 3.0 (TID 529, ip-10-0-51-185.ec2.internal, executor 30): ExecutorLostFailure (executor 30 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:02:00 INFO DAGScheduler: Executor lost: 30 (epoch 1) | |
| 17/06/14 05:02:00 INFO BlockManagerMasterEndpoint: Trying to remove executor 30 from BlockManagerMaster. | |
| 17/06/14 05:02:00 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 05:02:00 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(30, ip-10-0-51-185.ec2.internal, 46768, None) | |
| 17/06/14 05:02:00 INFO BlockManagerMaster: Removed 30 successfully in removeExecutor | |
| 17/06/14 05:02:00 INFO ExecutorAllocationManager: Existing executor 30 has been removed (new total is 10) | |
| 17/06/14 05:02:00 WARN ExecutorAllocationManager: Attempted to mark unknown executor 30 idle | |
| 17/06/14 05:02:00 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:02:00 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:02:00 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022687 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 05:02:00 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:02:00 INFO YarnAllocator: Completed container container_1497410974650_0004_01_003005 on host: ip-10-0-49-229.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:02:00 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:02:00 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:02:00 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:02:00 ERROR YarnClusterScheduler: Lost executor 28 on ip-10-0-49-229.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:02:00 WARN TaskSetManager: Lost task 8.0 in stage 3.0 (TID 507, ip-10-0-49-229.ec2.internal, executor 28): ExecutorLostFailure (executor 28 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:02:00 WARN TaskSetManager: Lost task 50.0 in stage 3.0 (TID 546, ip-10-0-49-229.ec2.internal, executor 28): ExecutorLostFailure (executor 28 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:02:00 WARN TaskSetManager: Lost task 27.0 in stage 3.0 (TID 521, ip-10-0-49-229.ec2.internal, executor 28): ExecutorLostFailure (executor 28 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:02:00 WARN TaskSetManager: Lost task 49.0 in stage 3.0 (TID 545, ip-10-0-49-229.ec2.internal, executor 28): ExecutorLostFailure (executor 28 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:02:00 INFO BlockManagerMaster: Removal of executor 28 requested | |
| 17/06/14 05:02:00 INFO BlockManagerMasterEndpoint: Trying to remove executor 28 from BlockManagerMaster. | |
| 17/06/14 05:02:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 28 | |
| 17/06/14 05:02:00 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 05:02:00 INFO ExecutorAllocationManager: Existing executor 28 has been removed (new total is 9) | |
| 17/06/14 05:02:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:45972) with ID 44 | |
| 17/06/14 05:02:00 INFO TaskSetManager: Starting task 5.2 in stage 3.0 (TID 584, ip-10-0-49-157.ec2.internal, executor 44, partition 5, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:00 INFO TaskSetManager: Starting task 49.1 in stage 3.0 (TID 585, ip-10-0-49-157.ec2.internal, executor 44, partition 49, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:00 INFO TaskSetManager: Starting task 27.1 in stage 3.0 (TID 586, ip-10-0-49-157.ec2.internal, executor 44, partition 27, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:00 INFO TaskSetManager: Starting task 50.1 in stage 3.0 (TID 587, ip-10-0-49-157.ec2.internal, executor 44, partition 50, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:00 INFO ExecutorAllocationManager: New executor 44 has registered (new total is 10) | |
| 17/06/14 05:02:00 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:45984 with 2.8 GB RAM, BlockManagerId(44, ip-10-0-49-157.ec2.internal, 45984, None) | |
| 17/06/14 05:02:00 WARN TransportChannelHandler: Exception in connection from /10.0.51.185:35094 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 05:02:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 30. | |
| 17/06/14 05:02:00 ERROR YarnClusterScheduler: Lost an executor 30 (already removed): Pending loss reason. | |
| 17/06/14 05:02:00 INFO BlockManagerMaster: Removal of executor 30 requested | |
| 17/06/14 05:02:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 30 | |
| 17/06/14 05:02:00 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:02:00 INFO BlockManagerMasterEndpoint: Trying to remove executor 30 from BlockManagerMaster. | |
| 17/06/14 05:02:00 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:02:00 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-157.ec2.internal:45984 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:02:01 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:50128) with ID 46 | |
| 17/06/14 05:02:01 INFO TaskSetManager: Starting task 8.1 in stage 3.0 (TID 588, ip-10-0-49-229.ec2.internal, executor 46, partition 8, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:01 INFO TaskSetManager: Starting task 36.1 in stage 3.0 (TID 589, ip-10-0-49-229.ec2.internal, executor 46, partition 36, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:01 INFO TaskSetManager: Starting task 19.1 in stage 3.0 (TID 590, ip-10-0-49-229.ec2.internal, executor 46, partition 19, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:01 INFO TaskSetManager: Starting task 1.1 in stage 3.0 (TID 591, ip-10-0-49-229.ec2.internal, executor 46, partition 1, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:01 INFO ExecutorAllocationManager: New executor 46 has registered (new total is 11) | |
| 17/06/14 05:02:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.157:45972 | |
| 17/06/14 05:02:01 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:40160 with 2.8 GB RAM, BlockManagerId(46, ip-10-0-49-229.ec2.internal, 40160, None) | |
| 17/06/14 05:02:01 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-229.ec2.internal:40160 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:02:01 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:36940) with ID 45 | |
| 17/06/14 05:02:01 INFO TaskSetManager: Starting task 56.1 in stage 3.0 (TID 592, ip-10-0-50-96.ec2.internal, executor 45, partition 56, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:01 INFO TaskSetManager: Starting task 12.1 in stage 3.0 (TID 593, ip-10-0-50-96.ec2.internal, executor 45, partition 12, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:01 INFO TaskSetManager: Starting task 11.2 in stage 3.0 (TID 594, ip-10-0-50-96.ec2.internal, executor 45, partition 11, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:01 INFO TaskSetManager: Starting task 29.1 in stage 3.0 (TID 595, ip-10-0-50-96.ec2.internal, executor 45, partition 29, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:01 INFO ExecutorAllocationManager: New executor 45 has registered (new total is 12) | |
| 17/06/14 05:02:01 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:40866 with 2.8 GB RAM, BlockManagerId(45, ip-10-0-50-96.ec2.internal, 40866, None) | |
| 17/06/14 05:02:01 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.229:50128 | |
| 17/06/14 05:02:02 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-50-96.ec2.internal:40866 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:02:02 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:45994) with ID 47 | |
| 17/06/14 05:02:02 INFO TaskSetManager: Starting task 0.1 in stage 3.0 (TID 596, ip-10-0-49-157.ec2.internal, executor 47, partition 0, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:02 INFO TaskSetManager: Starting task 35.1 in stage 3.0 (TID 597, ip-10-0-49-157.ec2.internal, executor 47, partition 35, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:02 INFO TaskSetManager: Starting task 53.1 in stage 3.0 (TID 598, ip-10-0-49-157.ec2.internal, executor 47, partition 53, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:02 INFO TaskSetManager: Starting task 17.1 in stage 3.0 (TID 599, ip-10-0-49-157.ec2.internal, executor 47, partition 17, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:02 INFO ExecutorAllocationManager: New executor 47 has registered (new total is 13) | |
| 17/06/14 05:02:02 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:50144) with ID 49 | |
| 17/06/14 05:02:02 INFO TaskSetManager: Starting task 40.1 in stage 3.0 (TID 600, ip-10-0-49-229.ec2.internal, executor 49, partition 40, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:02 INFO TaskSetManager: Starting task 7.1 in stage 3.0 (TID 601, ip-10-0-49-229.ec2.internal, executor 49, partition 7, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:02 INFO TaskSetManager: Starting task 24.1 in stage 3.0 (TID 602, ip-10-0-49-229.ec2.internal, executor 49, partition 24, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:02 INFO TaskSetManager: Starting task 39.1 in stage 3.0 (TID 603, ip-10-0-49-229.ec2.internal, executor 49, partition 39, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:02 INFO ExecutorAllocationManager: New executor 49 has registered (new total is 14) | |
| 17/06/14 05:02:02 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:32802 with 2.8 GB RAM, BlockManagerId(47, ip-10-0-49-157.ec2.internal, 32802, None) | |
| 17/06/14 05:02:02 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:45654 with 2.8 GB RAM, BlockManagerId(49, ip-10-0-49-229.ec2.internal, 45654, None) | |
| 17/06/14 05:02:02 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:36940 | |
| 17/06/14 05:02:02 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-229.ec2.internal:45654 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:02:02 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:35620) with ID 48 | |
| 17/06/14 05:02:02 INFO TaskSetManager: Starting task 6.1 in stage 3.0 (TID 604, ip-10-0-51-185.ec2.internal, executor 48, partition 6, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:02 INFO TaskSetManager: Starting task 57.0 in stage 3.0 (TID 605, ip-10-0-51-185.ec2.internal, executor 48, partition 57, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:02 INFO TaskSetManager: Starting task 58.0 in stage 3.0 (TID 606, ip-10-0-51-185.ec2.internal, executor 48, partition 58, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:02 INFO TaskSetManager: Starting task 59.0 in stage 3.0 (TID 607, ip-10-0-51-185.ec2.internal, executor 48, partition 59, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:02:02 INFO ExecutorAllocationManager: New executor 48 has registered (new total is 15) | |
| 17/06/14 05:02:03 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-157.ec2.internal:32802 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:02:03 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:34029 with 2.8 GB RAM, BlockManagerId(48, ip-10-0-51-185.ec2.internal, 34029, None) | |
| 17/06/14 05:02:03 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.229:50144 | |
| 17/06/14 05:02:03 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-51-185.ec2.internal:34029 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:02:03 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.157:45994 | |
| 17/06/14 05:02:03 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:35620 | |
| 17/06/14 05:09:59 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 10.0.49.229:33721 in memory (size: 33.3 KB, free: 413.9 MB) | |
| 17/06/14 05:19:16 INFO TaskSetManager: Starting task 60.0 in stage 3.0 (TID 608, ip-10-0-49-229.ec2.internal, executor 35, partition 60, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:19:16 INFO TaskSetManager: Finished task 20.1 in stage 3.0 (TID 538) in 1059259 ms on ip-10-0-49-229.ec2.internal (executor 35) (12/135) | |
| 17/06/14 05:19:26 INFO TaskSetManager: Starting task 61.0 in stage 3.0 (TID 609, ip-10-0-49-229.ec2.internal, executor 35, partition 61, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:19:26 INFO TaskSetManager: Finished task 44.0 in stage 3.0 (TID 539) in 1069864 ms on ip-10-0-49-229.ec2.internal (executor 35) (13/135) | |
| 17/06/14 05:19:53 INFO TaskSetManager: Starting task 62.0 in stage 3.0 (TID 610, ip-10-0-49-157.ec2.internal, executor 36, partition 62, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:19:53 INFO TaskSetManager: Finished task 81.1 in stage 3.0 (TID 541) in 1094309 ms on ip-10-0-49-157.ec2.internal (executor 36) (14/135) | |
| 17/06/14 05:19:53 INFO YarnAllocator: Driver requested a total number of 64 executor(s). | |
| 17/06/14 05:19:53 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 64 executors. | |
| 17/06/14 05:20:06 INFO TaskSetManager: Starting task 64.0 in stage 3.0 (TID 611, ip-10-0-49-229.ec2.internal, executor 35, partition 64, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:20:06 INFO TaskSetManager: Finished task 45.0 in stage 3.0 (TID 540) in 1109230 ms on ip-10-0-49-229.ec2.internal (executor 35) (15/135) | |
| 17/06/14 05:20:07 INFO BlockManagerInfo: Added rdd_10_2 in memory on ip-10-0-49-229.ec2.internal:45629 (size: 826.5 MB, free: 2000.6 MB) | |
| 17/06/14 05:20:08 INFO TaskSetManager: Starting task 65.0 in stage 3.0 (TID 612, ip-10-0-49-157.ec2.internal, executor 36, partition 65, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:20:08 INFO TaskSetManager: Finished task 46.0 in stage 3.0 (TID 543) in 1108588 ms on ip-10-0-49-157.ec2.internal (executor 36) (16/135) | |
| 17/06/14 05:20:14 INFO TaskSetManager: Starting task 67.0 in stage 3.0 (TID 613, ip-10-0-49-229.ec2.internal, executor 35, partition 67, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:20:14 INFO TaskSetManager: Finished task 2.1 in stage 3.0 (TID 537) in 1117540 ms on ip-10-0-49-229.ec2.internal (executor 35) (17/135) | |
| 17/06/14 05:20:29 INFO BlockManagerInfo: Added rdd_10_47 in memory on ip-10-0-49-157.ec2.internal:39668 (size: 836.2 MB, free: 1990.9 MB) | |
| 17/06/14 05:20:36 INFO TaskSetManager: Starting task 68.0 in stage 3.0 (TID 614, ip-10-0-49-157.ec2.internal, executor 36, partition 68, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:20:36 INFO TaskSetManager: Finished task 47.0 in stage 3.0 (TID 544) in 1136887 ms on ip-10-0-49-157.ec2.internal (executor 36) (18/135) | |
| 17/06/14 05:20:36 INFO YarnAllocator: Driver requested a total number of 63 executor(s). | |
| 17/06/14 05:20:36 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 63 executors. | |
| 17/06/14 05:21:00 INFO TaskSetManager: Starting task 69.0 in stage 3.0 (TID 615, ip-10-0-49-157.ec2.internal, executor 36, partition 69, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:21:00 INFO TaskSetManager: Finished task 73.1 in stage 3.0 (TID 542) in 1161119 ms on ip-10-0-49-157.ec2.internal (executor 36) (19/135) | |
| 17/06/14 05:21:02 INFO TaskSetManager: Starting task 70.0 in stage 3.0 (TID 616, ip-10-0-50-96.ec2.internal, executor 41, partition 70, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:21:02 INFO TaskSetManager: Finished task 52.1 in stage 3.0 (TID 568) in 1146897 ms on ip-10-0-50-96.ec2.internal (executor 41) (20/135) | |
| 17/06/14 05:21:19 INFO TaskSetManager: Starting task 71.0 in stage 3.0 (TID 617, ip-10-0-50-96.ec2.internal, executor 42, partition 71, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:21:19 INFO TaskSetManager: Finished task 15.1 in stage 3.0 (TID 572) in 1162514 ms on ip-10-0-50-96.ec2.internal (executor 42) (21/135) | |
| 17/06/14 05:21:28 INFO TaskSetManager: Starting task 72.0 in stage 3.0 (TID 618, ip-10-0-49-229.ec2.internal, executor 49, partition 72, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:21:28 INFO TaskSetManager: Finished task 7.1 in stage 3.0 (TID 601) in 1165836 ms on ip-10-0-49-229.ec2.internal (executor 49) (22/135) | |
| 17/06/14 05:21:28 INFO YarnAllocator: Driver requested a total number of 62 executor(s). | |
| 17/06/14 05:21:28 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 62 executors. | |
| 17/06/14 05:21:50 INFO TaskSetManager: Starting task 74.0 in stage 3.0 (TID 619, ip-10-0-50-96.ec2.internal, executor 42, partition 74, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:21:50 INFO TaskSetManager: Finished task 55.1 in stage 3.0 (TID 571) in 1193299 ms on ip-10-0-50-96.ec2.internal (executor 42) (23/135) | |
| 17/06/14 05:21:51 INFO TaskSetManager: Starting task 76.0 in stage 3.0 (TID 620, ip-10-0-50-96.ec2.internal, executor 41, partition 76, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:21:51 INFO TaskSetManager: Finished task 33.1 in stage 3.0 (TID 566) in 1195709 ms on ip-10-0-50-96.ec2.internal (executor 41) (24/135) | |
| 17/06/14 05:21:52 INFO TaskSetManager: Starting task 77.0 in stage 3.0 (TID 621, ip-10-0-49-229.ec2.internal, executor 49, partition 77, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:21:52 INFO TaskSetManager: Finished task 40.1 in stage 3.0 (TID 600) in 1190092 ms on ip-10-0-49-229.ec2.internal (executor 49) (25/135) | |
| 17/06/14 05:21:53 INFO BlockManagerInfo: Added rdd_10_9 in memory on ip-10-0-50-96.ec2.internal:37989 (size: 831.6 MB, free: 1995.5 MB) | |
| 17/06/14 05:21:59 INFO TaskSetManager: Starting task 78.0 in stage 3.0 (TID 622, ip-10-0-50-96.ec2.internal, executor 41, partition 78, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:21:59 INFO TaskSetManager: Finished task 51.1 in stage 3.0 (TID 567) in 1203425 ms on ip-10-0-50-96.ec2.internal (executor 41) (26/135) | |
| 17/06/14 05:21:59 INFO YarnAllocator: Driver requested a total number of 61 executor(s). | |
| 17/06/14 05:21:59 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 61 executors. | |
| 17/06/14 05:22:00 INFO TaskSetManager: Starting task 79.0 in stage 3.0 (TID 623, ip-10-0-50-96.ec2.internal, executor 41, partition 79, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:00 INFO TaskSetManager: Finished task 9.1 in stage 3.0 (TID 565) in 1204934 ms on ip-10-0-50-96.ec2.internal (executor 41) (27/135) | |
| 17/06/14 05:22:02 INFO TaskSetManager: Starting task 80.0 in stage 3.0 (TID 624, ip-10-0-50-96.ec2.internal, executor 42, partition 80, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:02 INFO TaskSetManager: Finished task 32.1 in stage 3.0 (TID 570) in 1205509 ms on ip-10-0-50-96.ec2.internal (executor 42) (28/135) | |
| 17/06/14 05:22:03 INFO TaskSetManager: Starting task 82.0 in stage 3.0 (TID 625, ip-10-0-49-229.ec2.internal, executor 46, partition 82, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:03 INFO TaskSetManager: Finished task 8.1 in stage 3.0 (TID 588) in 1202555 ms on ip-10-0-49-229.ec2.internal (executor 46) (29/135) | |
| 17/06/14 05:22:09 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-49-157.ec2.internal:39668 (size: 5.5 KB) | |
| 17/06/14 05:22:09 INFO BlockManagerInfo: Removed rdd_10_47 on ip-10-0-49-157.ec2.internal:39668 in memory (size: 836.2 MB, free: 2.8 GB) | |
| 17/06/14 05:22:12 INFO TaskSetManager: Starting task 83.0 in stage 3.0 (TID 626, ip-10-0-49-229.ec2.internal, executor 46, partition 83, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:12 INFO TaskSetManager: Finished task 36.1 in stage 3.0 (TID 589) in 1211697 ms on ip-10-0-49-229.ec2.internal (executor 46) (30/135) | |
| 17/06/14 05:22:12 INFO YarnAllocator: Driver requested a total number of 60 executor(s). | |
| 17/06/14 05:22:12 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 60 executors. | |
| 17/06/14 05:22:14 INFO TaskSetManager: Starting task 84.0 in stage 3.0 (TID 627, ip-10-0-50-96.ec2.internal, executor 43, partition 84, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:14 INFO TaskSetManager: Finished task 28.2 in stage 3.0 (TID 580) in 1216095 ms on ip-10-0-50-96.ec2.internal (executor 43) (31/135) | |
| 17/06/14 05:22:15 INFO BlockManagerInfo: Added rdd_10_5 in memory on ip-10-0-49-157.ec2.internal:45984 (size: 828.8 MB, free: 1998.3 MB) | |
| 17/06/14 05:22:16 INFO TaskSetManager: Starting task 85.0 in stage 3.0 (TID 628, ip-10-0-51-185.ec2.internal, executor 38, partition 85, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:16 INFO TaskSetManager: Finished task 30.1 in stage 3.0 (TID 557) in 1221871 ms on ip-10-0-51-185.ec2.internal (executor 38) (32/135) | |
| 17/06/14 05:22:19 INFO BlockManagerInfo: Added rdd_11_5 in memory on ip-10-0-49-157.ec2.internal:45984 (size: 833.4 MB, free: 1164.9 MB) | |
| 17/06/14 05:22:19 INFO TaskSetManager: Starting task 86.0 in stage 3.0 (TID 629, ip-10-0-49-229.ec2.internal, executor 49, partition 86, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:19 INFO TaskSetManager: Finished task 24.1 in stage 3.0 (TID 602) in 1217447 ms on ip-10-0-49-229.ec2.internal (executor 49) (33/135) | |
| 17/06/14 05:22:21 INFO TaskSetManager: Starting task 87.0 in stage 3.0 (TID 630, ip-10-0-50-96.ec2.internal, executor 45, partition 87, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:21 INFO TaskSetManager: Finished task 29.1 in stage 3.0 (TID 595) in 1219882 ms on ip-10-0-50-96.ec2.internal (executor 45) (34/135) | |
| 17/06/14 05:22:21 INFO YarnAllocator: Driver requested a total number of 59 executor(s). | |
| 17/06/14 05:22:21 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 59 executors. | |
| 17/06/14 05:22:22 INFO TaskSetManager: Starting task 88.0 in stage 3.0 (TID 631, ip-10-0-49-157.ec2.internal, executor 44, partition 88, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:22 INFO TaskSetManager: Finished task 5.2 in stage 3.0 (TID 584) in 1222216 ms on ip-10-0-49-157.ec2.internal (executor 44) (35/135) | |
| 17/06/14 05:22:23 INFO TaskSetManager: Starting task 89.0 in stage 3.0 (TID 632, ip-10-0-49-157.ec2.internal, executor 47, partition 89, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:23 INFO TaskSetManager: Finished task 53.1 in stage 3.0 (TID 598) in 1221189 ms on ip-10-0-49-157.ec2.internal (executor 47) (36/135) | |
| 17/06/14 05:22:24 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-49-157.ec2.internal:45984 (size: 5.5 KB) | |
| 17/06/14 05:22:24 INFO BlockManagerInfo: Removed rdd_10_5 on ip-10-0-49-157.ec2.internal:45984 in memory (size: 828.8 MB, free: 1993.7 MB) | |
| 17/06/14 05:22:29 INFO TaskSetManager: Starting task 90.0 in stage 3.0 (TID 633, ip-10-0-49-229.ec2.internal, executor 49, partition 90, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:29 INFO TaskSetManager: Finished task 39.1 in stage 3.0 (TID 603) in 1226841 ms on ip-10-0-49-229.ec2.internal (executor 49) (37/135) | |
| 17/06/14 05:22:34 INFO TaskSetManager: Starting task 91.0 in stage 3.0 (TID 634, ip-10-0-50-96.ec2.internal, executor 43, partition 91, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:34 INFO TaskSetManager: Finished task 34.1 in stage 3.0 (TID 582) in 1235234 ms on ip-10-0-50-96.ec2.internal (executor 43) (38/135) | |
| 17/06/14 05:22:34 INFO YarnAllocator: Driver requested a total number of 58 executor(s). | |
| 17/06/14 05:22:34 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 58 executors. | |
| 17/06/14 05:22:34 INFO TaskSetManager: Starting task 92.0 in stage 3.0 (TID 635, ip-10-0-51-185.ec2.internal, executor 39, partition 92, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:34 INFO TaskSetManager: Finished task 37.1 in stage 3.0 (TID 563) in 1239381 ms on ip-10-0-51-185.ec2.internal (executor 39) (39/135) | |
| 17/06/14 05:22:36 INFO BlockManagerInfo: Added rdd_10_54 in memory on ip-10-0-50-96.ec2.internal:42160 (size: 824.2 MB, free: 2002.9 MB) | |
| 17/06/14 05:22:38 INFO TaskSetManager: Starting task 93.0 in stage 3.0 (TID 636, ip-10-0-49-157.ec2.internal, executor 44, partition 93, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:38 INFO TaskSetManager: Finished task 27.1 in stage 3.0 (TID 586) in 1238544 ms on ip-10-0-49-157.ec2.internal (executor 44) (40/135) | |
| 17/06/14 05:22:40 INFO BlockManagerInfo: Added rdd_11_54 in memory on ip-10-0-50-96.ec2.internal:42160 (size: 802.7 MB, free: 1200.3 MB) | |
| 17/06/14 05:22:41 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 43. | |
| 17/06/14 05:22:41 INFO DAGScheduler: Executor lost: 43 (epoch 1) | |
| 17/06/14 05:22:41 INFO BlockManagerMasterEndpoint: Trying to remove executor 43 from BlockManagerMaster. | |
| 17/06/14 05:22:41 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(43, ip-10-0-50-96.ec2.internal, 39293, None) | |
| 17/06/14 05:22:41 INFO BlockManagerMaster: Removed 43 successfully in removeExecutor | |
| 17/06/14 05:22:41 INFO YarnAllocator: Launching container container_1497410974650_0004_01_026409 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 05:22:41 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:22:41 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022674 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:22:41 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:22:41 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:22:41 ERROR YarnClusterScheduler: Lost executor 43 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:22:41 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:22:41 WARN TaskSetManager: Lost task 84.0 in stage 3.0 (TID 627, ip-10-0-50-96.ec2.internal, executor 43): ExecutorLostFailure (executor 43 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:22:41 WARN TaskSetManager: Lost task 43.1 in stage 3.0 (TID 581, ip-10-0-50-96.ec2.internal, executor 43): ExecutorLostFailure (executor 43 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:22:41 WARN TaskSetManager: Lost task 91.0 in stage 3.0 (TID 634, ip-10-0-50-96.ec2.internal, executor 43): ExecutorLostFailure (executor 43 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:22:41 WARN TaskSetManager: Lost task 16.1 in stage 3.0 (TID 583, ip-10-0-50-96.ec2.internal, executor 43): ExecutorLostFailure (executor 43 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:22:41 INFO BlockManagerMaster: Removal of executor 43 requested | |
| 17/06/14 05:22:41 INFO BlockManagerMasterEndpoint: Trying to remove executor 43 from BlockManagerMaster. | |
| 17/06/14 05:22:41 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 43 | |
| 17/06/14 05:22:41 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 05:22:41 INFO ExecutorAllocationManager: Existing executor 43 has been removed (new total is 14) | |
| 17/06/14 05:22:41 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:22:41 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:22:41 INFO TaskSetManager: Starting task 16.2 in stage 3.0 (TID 637, ip-10-0-51-185.ec2.internal, executor 38, partition 16, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:41 INFO TaskSetManager: Finished task 41.1 in stage 3.0 (TID 559) in 1247507 ms on ip-10-0-51-185.ec2.internal (executor 38) (41/135) | |
| 17/06/14 05:22:42 INFO BlockManagerInfo: Added rdd_10_21 in memory on ip-10-0-51-185.ec2.internal:46252 (size: 827.8 MB, free: 1999.3 MB) | |
| 17/06/14 05:22:42 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:37112) with ID 50 | |
| 17/06/14 05:22:42 INFO TaskSetManager: Starting task 91.1 in stage 3.0 (TID 638, ip-10-0-50-96.ec2.internal, executor 50, partition 91, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:42 INFO TaskSetManager: Starting task 43.2 in stage 3.0 (TID 639, ip-10-0-50-96.ec2.internal, executor 50, partition 43, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:42 INFO TaskSetManager: Starting task 84.1 in stage 3.0 (TID 640, ip-10-0-50-96.ec2.internal, executor 50, partition 84, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:42 INFO TaskSetManager: Starting task 94.0 in stage 3.0 (TID 641, ip-10-0-50-96.ec2.internal, executor 50, partition 94, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:42 INFO ExecutorAllocationManager: New executor 50 has registered (new total is 15) | |
| 17/06/14 05:22:42 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:34711 with 2.8 GB RAM, BlockManagerId(50, ip-10-0-50-96.ec2.internal, 34711, None) | |
| 17/06/14 05:22:43 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-50-96.ec2.internal:34711 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:22:43 INFO TaskSetManager: Starting task 95.0 in stage 3.0 (TID 642, ip-10-0-49-157.ec2.internal, executor 40, partition 95, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:43 INFO TaskSetManager: Finished task 42.1 in stage 3.0 (TID 577) in 1246002 ms on ip-10-0-49-157.ec2.internal (executor 40) (42/135) | |
| 17/06/14 05:22:43 INFO YarnAllocator: Driver requested a total number of 57 executor(s). | |
| 17/06/14 05:22:43 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 57 executors. | |
| 17/06/14 05:22:43 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:37112 | |
| 17/06/14 05:22:43 INFO TaskSetManager: Starting task 96.0 in stage 3.0 (TID 643, ip-10-0-50-96.ec2.internal, executor 42, partition 96, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:43 INFO TaskSetManager: Finished task 54.1 in stage 3.0 (TID 573) in 1246572 ms on ip-10-0-50-96.ec2.internal (executor 42) (43/135) | |
| 17/06/14 05:22:45 INFO TaskSetManager: Starting task 97.0 in stage 3.0 (TID 644, ip-10-0-50-96.ec2.internal, executor 45, partition 97, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:45 INFO TaskSetManager: Finished task 11.2 in stage 3.0 (TID 594) in 1243526 ms on ip-10-0-50-96.ec2.internal (executor 45) (44/135) | |
| 17/06/14 05:22:45 INFO BlockManagerInfo: Added rdd_10_23 in memory on ip-10-0-51-185.ec2.internal:37560 (size: 836.0 MB, free: 1991.1 MB) | |
| 17/06/14 05:22:45 INFO BlockManagerInfo: Removed rdd_10_54 on ip-10-0-50-96.ec2.internal:42160 in memory (size: 824.2 MB, free: 2024.4 MB) | |
| 17/06/14 05:22:46 INFO BlockManagerInfo: Added rdd_11_21 in memory on ip-10-0-51-185.ec2.internal:46252 (size: 860.1 MB, free: 1139.3 MB) | |
| 17/06/14 05:22:49 INFO TaskSetManager: Starting task 98.0 in stage 3.0 (TID 645, ip-10-0-51-185.ec2.internal, executor 39, partition 98, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:49 INFO TaskSetManager: Finished task 21.1 in stage 3.0 (TID 564) in 1254282 ms on ip-10-0-51-185.ec2.internal (executor 39) (45/135) | |
| 17/06/14 05:22:49 INFO TaskSetManager: Starting task 100.0 in stage 3.0 (TID 646, ip-10-0-51-185.ec2.internal, executor 39, partition 100, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:49 INFO TaskSetManager: Finished task 38.1 in stage 3.0 (TID 562) in 1254651 ms on ip-10-0-51-185.ec2.internal (executor 39) (46/135) | |
| 17/06/14 05:22:49 INFO YarnAllocator: Driver requested a total number of 56 executor(s). | |
| 17/06/14 05:22:49 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 56 executors. | |
| 17/06/14 05:22:49 INFO TaskSetManager: Starting task 101.0 in stage 3.0 (TID 647, ip-10-0-49-157.ec2.internal, executor 44, partition 101, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:49 INFO TaskSetManager: Finished task 50.1 in stage 3.0 (TID 587) in 1249487 ms on ip-10-0-49-157.ec2.internal (executor 44) (47/135) | |
| 17/06/14 05:22:50 INFO YarnAllocator: Launching container container_1497410974650_0004_01_026437 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 05:22:50 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:22:50 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022689 on host: ip-10-0-49-157.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:22:50 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:22:50 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:22:50 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:22:50 ERROR YarnClusterScheduler: Lost executor 47 on ip-10-0-49-157.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:22:50 WARN TaskSetManager: Lost task 35.1 in stage 3.0 (TID 597, ip-10-0-49-157.ec2.internal, executor 47): ExecutorLostFailure (executor 47 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:22:50 WARN TaskSetManager: Lost task 89.0 in stage 3.0 (TID 632, ip-10-0-49-157.ec2.internal, executor 47): ExecutorLostFailure (executor 47 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:22:50 WARN TaskSetManager: Lost task 17.1 in stage 3.0 (TID 599, ip-10-0-49-157.ec2.internal, executor 47): ExecutorLostFailure (executor 47 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:22:50 WARN TaskSetManager: Lost task 0.1 in stage 3.0 (TID 596, ip-10-0-49-157.ec2.internal, executor 47): ExecutorLostFailure (executor 47 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:22:50 INFO DAGScheduler: Executor lost: 47 (epoch 1) | |
| 17/06/14 05:22:50 INFO BlockManagerMasterEndpoint: Trying to remove executor 47 from BlockManagerMaster. | |
| 17/06/14 05:22:50 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(47, ip-10-0-49-157.ec2.internal, 32802, None) | |
| 17/06/14 05:22:50 INFO BlockManagerMaster: Removed 47 successfully in removeExecutor | |
| 17/06/14 05:22:50 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 05:22:50 INFO ExecutorAllocationManager: Existing executor 47 has been removed (new total is 14) | |
| 17/06/14 05:22:50 WARN ExecutorAllocationManager: Attempted to mark unknown executor 47 idle | |
| 17/06/14 05:22:50 WARN TransportChannelHandler: Exception in connection from /10.0.49.157:45994 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 05:22:50 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 47. | |
| 17/06/14 05:22:50 ERROR YarnClusterScheduler: Lost an executor 47 (already removed): Pending loss reason. | |
| 17/06/14 05:22:50 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:22:50 INFO BlockManagerMaster: Removal of executor 47 requested | |
| 17/06/14 05:22:50 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 47 | |
| 17/06/14 05:22:50 INFO BlockManagerMasterEndpoint: Trying to remove executor 47 from BlockManagerMaster. | |
| 17/06/14 05:22:50 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:22:51 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-51-185.ec2.internal:46252 (size: 5.5 KB) | |
| 17/06/14 05:22:51 INFO BlockManagerInfo: Removed rdd_10_21 on ip-10-0-51-185.ec2.internal:46252 in memory (size: 827.8 MB, free: 1967.0 MB) | |
| 17/06/14 05:22:52 INFO BlockManagerInfo: Added rdd_10_56 in memory on ip-10-0-50-96.ec2.internal:40866 (size: 840.1 MB, free: 1987.0 MB) | |
| 17/06/14 05:22:52 INFO TaskSetManager: Starting task 0.2 in stage 3.0 (TID 648, ip-10-0-51-185.ec2.internal, executor 38, partition 0, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:52 INFO TaskSetManager: Finished task 23.1 in stage 3.0 (TID 560) in 1258157 ms on ip-10-0-51-185.ec2.internal (executor 38) (48/135) | |
| 17/06/14 05:22:52 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:46182) with ID 51 | |
| 17/06/14 05:22:52 INFO TaskSetManager: Starting task 17.2 in stage 3.0 (TID 649, ip-10-0-49-157.ec2.internal, executor 51, partition 17, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:52 INFO TaskSetManager: Starting task 89.1 in stage 3.0 (TID 650, ip-10-0-49-157.ec2.internal, executor 51, partition 89, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:52 INFO TaskSetManager: Starting task 35.2 in stage 3.0 (TID 651, ip-10-0-49-157.ec2.internal, executor 51, partition 35, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:52 INFO TaskSetManager: Starting task 102.0 in stage 3.0 (TID 652, ip-10-0-49-157.ec2.internal, executor 51, partition 102, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:52 INFO ExecutorAllocationManager: New executor 51 has registered (new total is 15) | |
| 17/06/14 05:22:52 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:44407 with 2.8 GB RAM, BlockManagerId(51, ip-10-0-49-157.ec2.internal, 44407, None) | |
| 17/06/14 05:22:52 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-157.ec2.internal:44407 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:22:53 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.157:46182 | |
| 17/06/14 05:22:53 INFO TaskSetManager: Starting task 103.0 in stage 3.0 (TID 653, ip-10-0-51-185.ec2.internal, executor 48, partition 103, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:53 INFO TaskSetManager: Finished task 59.0 in stage 3.0 (TID 607) in 1250340 ms on ip-10-0-51-185.ec2.internal (executor 48) (49/135) | |
| 17/06/14 05:22:53 INFO BlockManagerInfo: Added rdd_10_57 in memory on ip-10-0-51-185.ec2.internal:34029 (size: 844.9 MB, free: 1982.2 MB) | |
| 17/06/14 05:22:55 INFO TaskSetManager: Starting task 104.0 in stage 3.0 (TID 654, ip-10-0-51-185.ec2.internal, executor 37, partition 104, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:55 INFO TaskSetManager: Finished task 3.1 in stage 3.0 (TID 556) in 1262590 ms on ip-10-0-51-185.ec2.internal (executor 37) (50/135) | |
| 17/06/14 05:22:55 INFO YarnAllocator: Driver requested a total number of 55 executor(s). | |
| 17/06/14 05:22:55 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 55 executors. | |
| 17/06/14 05:22:56 INFO TaskSetManager: Starting task 105.0 in stage 3.0 (TID 655, ip-10-0-49-229.ec2.internal, executor 46, partition 105, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:56 INFO TaskSetManager: Finished task 1.1 in stage 3.0 (TID 591) in 1255167 ms on ip-10-0-49-229.ec2.internal (executor 46) (51/135) | |
| 17/06/14 05:22:56 INFO TaskSetManager: Starting task 106.0 in stage 3.0 (TID 656, ip-10-0-49-157.ec2.internal, executor 40, partition 106, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:56 INFO TaskSetManager: Finished task 31.1 in stage 3.0 (TID 578) in 1259460 ms on ip-10-0-49-157.ec2.internal (executor 40) (52/135) | |
| 17/06/14 05:22:56 INFO BlockManagerInfo: Added rdd_11_56 in memory on ip-10-0-50-96.ec2.internal:40866 (size: 828.4 MB, free: 1158.5 MB) | |
| 17/06/14 05:22:58 INFO TaskSetManager: Starting task 107.0 in stage 3.0 (TID 657, ip-10-0-51-185.ec2.internal, executor 38, partition 107, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:58 INFO TaskSetManager: Finished task 13.1 in stage 3.0 (TID 558) in 1264318 ms on ip-10-0-51-185.ec2.internal (executor 38) (53/135) | |
| 17/06/14 05:22:59 INFO TaskSetManager: Starting task 108.0 in stage 3.0 (TID 658, ip-10-0-51-185.ec2.internal, executor 37, partition 108, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:59 INFO TaskSetManager: Finished task 22.1 in stage 3.0 (TID 554) in 1266486 ms on ip-10-0-51-185.ec2.internal (executor 37) (54/135) | |
| 17/06/14 05:22:59 INFO TaskSetManager: Starting task 109.0 in stage 3.0 (TID 659, ip-10-0-50-96.ec2.internal, executor 45, partition 109, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:22:59 INFO TaskSetManager: Finished task 56.1 in stage 3.0 (TID 592) in 1258321 ms on ip-10-0-50-96.ec2.internal (executor 45) (55/135) | |
| 17/06/14 05:22:59 INFO YarnAllocator: Driver requested a total number of 54 executor(s). | |
| 17/06/14 05:22:59 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 54 executors. | |
| 17/06/14 05:23:00 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-49-229.ec2.internal:45629 (size: 5.5 KB) | |
| 17/06/14 05:23:00 INFO BlockManagerInfo: Removed rdd_10_2 on ip-10-0-49-229.ec2.internal:45629 in memory (size: 826.5 MB, free: 2.8 GB) | |
| 17/06/14 05:23:01 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-50-96.ec2.internal:40866 (size: 5.5 KB) | |
| 17/06/14 05:23:01 INFO BlockManagerInfo: Removed rdd_10_56 on ip-10-0-50-96.ec2.internal:40866 in memory (size: 840.1 MB, free: 1998.7 MB) | |
| 17/06/14 05:23:02 INFO TaskSetManager: Starting task 110.0 in stage 3.0 (TID 660, ip-10-0-49-157.ec2.internal, executor 44, partition 110, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:23:02 INFO TaskSetManager: Finished task 49.1 in stage 3.0 (TID 585) in 1262649 ms on ip-10-0-49-157.ec2.internal (executor 44) (56/135) | |
| 17/06/14 05:23:03 INFO TaskSetManager: Starting task 113.0 in stage 3.0 (TID 661, ip-10-0-51-185.ec2.internal, executor 48, partition 113, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:23:03 INFO TaskSetManager: Finished task 57.0 in stage 3.0 (TID 605) in 1260140 ms on ip-10-0-51-185.ec2.internal (executor 48) (57/135) | |
| 17/06/14 05:23:03 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-51-185.ec2.internal:37560 (size: 5.5 KB) | |
| 17/06/14 05:23:03 INFO BlockManagerInfo: Removed rdd_10_23 on ip-10-0-51-185.ec2.internal:37560 in memory (size: 836.0 MB, free: 2.8 GB) | |
| 17/06/14 05:23:06 INFO TaskSetManager: Starting task 114.0 in stage 3.0 (TID 662, ip-10-0-49-157.ec2.internal, executor 40, partition 114, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:23:06 INFO TaskSetManager: Finished task 14.1 in stage 3.0 (TID 576) in 1268926 ms on ip-10-0-49-157.ec2.internal (executor 40) (58/135) | |
| 17/06/14 05:23:06 INFO YarnAllocator: Driver requested a total number of 53 executor(s). | |
| 17/06/14 05:23:06 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 53 executors. | |
| 17/06/14 05:23:12 INFO TaskSetManager: Starting task 115.0 in stage 3.0 (TID 663, ip-10-0-49-229.ec2.internal, executor 46, partition 115, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:23:12 INFO TaskSetManager: Finished task 19.1 in stage 3.0 (TID 590) in 1271490 ms on ip-10-0-49-229.ec2.internal (executor 46) (59/135) | |
| 17/06/14 05:23:13 INFO TaskSetManager: Starting task 116.0 in stage 3.0 (TID 664, ip-10-0-51-185.ec2.internal, executor 48, partition 116, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:23:13 INFO TaskSetManager: Finished task 6.1 in stage 3.0 (TID 604) in 1270080 ms on ip-10-0-51-185.ec2.internal (executor 48) (60/135) | |
| 17/06/14 05:23:13 INFO TaskSetManager: Starting task 117.0 in stage 3.0 (TID 665, ip-10-0-51-185.ec2.internal, executor 48, partition 117, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:23:13 INFO TaskSetManager: Finished task 58.0 in stage 3.0 (TID 606) in 1270807 ms on ip-10-0-51-185.ec2.internal (executor 48) (61/135) | |
| 17/06/14 05:23:14 INFO TaskSetManager: Starting task 118.0 in stage 3.0 (TID 666, ip-10-0-50-96.ec2.internal, executor 45, partition 118, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:23:14 INFO TaskSetManager: Finished task 12.1 in stage 3.0 (TID 593) in 1273239 ms on ip-10-0-50-96.ec2.internal (executor 45) (62/135) | |
| 17/06/14 05:23:14 INFO YarnAllocator: Driver requested a total number of 52 executor(s). | |
| 17/06/14 05:23:14 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 52 executors. | |
| 17/06/14 05:23:16 INFO TaskSetManager: Starting task 119.0 in stage 3.0 (TID 667, ip-10-0-51-185.ec2.internal, executor 39, partition 119, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:23:16 INFO TaskSetManager: Finished task 4.1 in stage 3.0 (TID 561) in 1281212 ms on ip-10-0-51-185.ec2.internal (executor 39) (63/135) | |
| 17/06/14 05:23:42 INFO TaskSetManager: Starting task 120.0 in stage 3.0 (TID 668, ip-10-0-49-157.ec2.internal, executor 40, partition 120, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:23:42 INFO TaskSetManager: Finished task 26.1 in stage 3.0 (TID 579) in 1305196 ms on ip-10-0-49-157.ec2.internal (executor 40) (64/135) | |
| 17/06/14 05:23:51 INFO TaskSetManager: Starting task 121.0 in stage 3.0 (TID 669, ip-10-0-51-185.ec2.internal, executor 37, partition 121, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:23:51 INFO TaskSetManager: Finished task 25.1 in stage 3.0 (TID 553) in 1318148 ms on ip-10-0-51-185.ec2.internal (executor 37) (65/135) | |
| 17/06/14 05:24:02 INFO TaskSetManager: Starting task 122.0 in stage 3.0 (TID 670, ip-10-0-51-185.ec2.internal, executor 37, partition 122, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:24:02 INFO TaskSetManager: Finished task 18.1 in stage 3.0 (TID 555) in 1329243 ms on ip-10-0-51-185.ec2.internal (executor 37) (66/135) | |
| 17/06/14 05:24:02 INFO YarnAllocator: Driver requested a total number of 51 executor(s). | |
| 17/06/14 05:24:02 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 51 executors. | |
| 17/06/14 05:24:13 INFO BlockManagerInfo: Removed rdd_11_5 on ip-10-0-49-157.ec2.internal:45984 in memory (size: 833.4 MB, free: 2.8 GB) | |
| 17/06/14 05:24:42 INFO BlockManagerInfo: Removed rdd_11_56 on ip-10-0-50-96.ec2.internal:40866 in memory (size: 828.4 MB, free: 2.8 GB) | |
| 17/06/14 05:24:47 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-51-185.ec2.internal:34029 (size: 5.5 KB) | |
| 17/06/14 05:24:47 INFO BlockManagerInfo: Removed rdd_10_57 on ip-10-0-51-185.ec2.internal:34029 in memory (size: 844.9 MB, free: 2.8 GB) | |
| 17/06/14 05:25:05 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-50-96.ec2.internal:37989 (size: 5.5 KB) | |
| 17/06/14 05:25:05 INFO BlockManagerInfo: Removed rdd_10_9 on ip-10-0-50-96.ec2.internal:37989 in memory (size: 831.6 MB, free: 2.8 GB) | |
| 17/06/14 05:26:07 INFO BlockManagerInfo: Removed rdd_11_21 on ip-10-0-51-185.ec2.internal:46252 in memory (size: 860.1 MB, free: 2.8 GB) | |
| 17/06/14 05:27:14 INFO BlockManagerInfo: Removed rdd_11_54 on ip-10-0-50-96.ec2.internal:42160 in memory (size: 802.7 MB, free: 2.8 GB) | |
| 17/06/14 05:40:28 INFO TaskSetManager: Starting task 123.0 in stage 3.0 (TID 671, ip-10-0-49-157.ec2.internal, executor 36, partition 123, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:40:28 INFO TaskSetManager: Finished task 65.0 in stage 3.0 (TID 612) in 1220706 ms on ip-10-0-49-157.ec2.internal (executor 36) (67/135) | |
| 17/06/14 05:40:29 INFO TaskSetManager: Starting task 124.0 in stage 3.0 (TID 672, ip-10-0-49-229.ec2.internal, executor 35, partition 124, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:40:29 INFO TaskSetManager: Finished task 60.0 in stage 3.0 (TID 608) in 1273456 ms on ip-10-0-49-229.ec2.internal (executor 35) (68/135) | |
| 17/06/14 05:40:54 INFO BlockManagerInfo: Added rdd_10_68 in memory on ip-10-0-49-157.ec2.internal:39668 (size: 840.7 MB, free: 1986.4 MB) | |
| 17/06/14 05:40:57 INFO TaskSetManager: Starting task 125.0 in stage 3.0 (TID 673, ip-10-0-50-96.ec2.internal, executor 41, partition 125, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:40:57 INFO TaskSetManager: Finished task 70.0 in stage 3.0 (TID 616) in 1194198 ms on ip-10-0-50-96.ec2.internal (executor 41) (69/135) | |
| 17/06/14 05:41:00 INFO TaskSetManager: Starting task 126.0 in stage 3.0 (TID 674, ip-10-0-49-157.ec2.internal, executor 36, partition 126, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:41:00 INFO TaskSetManager: Finished task 68.0 in stage 3.0 (TID 614) in 1224027 ms on ip-10-0-49-157.ec2.internal (executor 36) (70/135) | |
| 17/06/14 05:41:00 INFO YarnAllocator: Driver requested a total number of 50 executor(s). | |
| 17/06/14 05:41:00 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 50 executors. | |
| 17/06/14 05:41:26 INFO TaskSetManager: Starting task 127.0 in stage 3.0 (TID 675, ip-10-0-49-157.ec2.internal, executor 36, partition 127, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:41:26 INFO TaskSetManager: Finished task 62.0 in stage 3.0 (TID 610) in 1292329 ms on ip-10-0-49-157.ec2.internal (executor 36) (71/135) | |
| 17/06/14 05:41:46 INFO TaskSetManager: Starting task 128.0 in stage 3.0 (TID 676, ip-10-0-49-157.ec2.internal, executor 36, partition 128, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:41:46 INFO TaskSetManager: Finished task 69.0 in stage 3.0 (TID 615) in 1245825 ms on ip-10-0-49-157.ec2.internal (executor 36) (72/135) | |
| 17/06/14 05:41:50 INFO BlockManagerInfo: Added rdd_10_64 in memory on ip-10-0-49-229.ec2.internal:45629 (size: 790.9 MB, free: 2036.2 MB) | |
| 17/06/14 05:41:50 INFO TaskSetManager: Starting task 130.0 in stage 3.0 (TID 677, ip-10-0-50-96.ec2.internal, executor 42, partition 130, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:41:50 INFO TaskSetManager: Finished task 71.0 in stage 3.0 (TID 617) in 1231248 ms on ip-10-0-50-96.ec2.internal (executor 42) (73/135) | |
| 17/06/14 05:41:54 INFO TaskSetManager: Starting task 132.0 in stage 3.0 (TID 678, ip-10-0-49-229.ec2.internal, executor 35, partition 132, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:41:54 INFO TaskSetManager: Finished task 61.0 in stage 3.0 (TID 609) in 1348134 ms on ip-10-0-49-229.ec2.internal (executor 35) (74/135) | |
| 17/06/14 05:41:54 INFO YarnAllocator: Driver requested a total number of 49 executor(s). | |
| 17/06/14 05:41:54 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 49 executors. | |
| 17/06/14 05:41:56 INFO TaskSetManager: Starting task 133.0 in stage 3.0 (TID 679, ip-10-0-49-229.ec2.internal, executor 35, partition 133, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:41:56 INFO TaskSetManager: Finished task 64.0 in stage 3.0 (TID 611) in 1310565 ms on ip-10-0-49-229.ec2.internal (executor 35) (75/135) | |
| 17/06/14 05:41:57 INFO TaskSetManager: Starting task 75.0 in stage 5.0 (TID 680, ip-10-0-49-229.ec2.internal, executor 35, partition 75, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:41:57 INFO TaskSetManager: Finished task 67.0 in stage 3.0 (TID 613) in 1303116 ms on ip-10-0-49-229.ec2.internal (executor 35) (76/135) | |
| 17/06/14 05:41:57 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-49-229.ec2.internal:45629 (size: 5.5 KB, free: 2036.2 MB) | |
| 17/06/14 05:42:14 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 35. | |
| 17/06/14 05:42:14 INFO DAGScheduler: Executor lost: 35 (epoch 1) | |
| 17/06/14 05:42:14 INFO BlockManagerMasterEndpoint: Trying to remove executor 35 from BlockManagerMaster. | |
| 17/06/14 05:42:14 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(35, ip-10-0-49-229.ec2.internal, 45629, None) | |
| 17/06/14 05:42:14 INFO BlockManagerMaster: Removed 35 successfully in removeExecutor | |
| 17/06/14 05:42:14 INFO YarnAllocator: Launching container container_1497410974650_0004_01_022693 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 05:42:14 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:42:14 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022610 on host: ip-10-0-49-229.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:42:14 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:42:14 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:42:14 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:42:14 ERROR YarnClusterScheduler: Lost executor 35 on ip-10-0-49-229.ec2.internal: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:42:14 WARN TaskSetManager: Lost task 132.0 in stage 3.0 (TID 678, ip-10-0-49-229.ec2.internal, executor 35): ExecutorLostFailure (executor 35 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:42:14 WARN TaskSetManager: Lost task 124.0 in stage 3.0 (TID 672, ip-10-0-49-229.ec2.internal, executor 35): ExecutorLostFailure (executor 35 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:42:14 WARN TaskSetManager: Lost task 133.0 in stage 3.0 (TID 679, ip-10-0-49-229.ec2.internal, executor 35): ExecutorLostFailure (executor 35 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:42:14 WARN TaskSetManager: Lost task 75.0 in stage 5.0 (TID 680, ip-10-0-49-229.ec2.internal, executor 35): ExecutorLostFailure (executor 35 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:42:14 INFO BlockManagerMaster: Removal of executor 35 requested | |
| 17/06/14 05:42:14 INFO BlockManagerMasterEndpoint: Trying to remove executor 35 from BlockManagerMaster. | |
| 17/06/14 05:42:14 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 35 | |
| 17/06/14 05:42:14 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 05:42:14 INFO ExecutorAllocationManager: Existing executor 35 has been removed (new total is 14) | |
| 17/06/14 05:42:14 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:42:14 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:42:15 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:50552) with ID 52 | |
| 17/06/14 05:42:15 INFO TaskSetManager: Starting task 133.1 in stage 3.0 (TID 681, ip-10-0-49-229.ec2.internal, executor 52, partition 133, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:42:15 INFO TaskSetManager: Starting task 124.1 in stage 3.0 (TID 682, ip-10-0-49-229.ec2.internal, executor 52, partition 124, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:42:15 INFO TaskSetManager: Starting task 132.1 in stage 3.0 (TID 683, ip-10-0-49-229.ec2.internal, executor 52, partition 132, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:42:15 INFO TaskSetManager: Starting task 75.1 in stage 5.0 (TID 684, ip-10-0-49-229.ec2.internal, executor 52, partition 75, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:42:15 INFO ExecutorAllocationManager: New executor 52 has registered (new total is 15) | |
| 17/06/14 05:42:15 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:44395 with 2.8 GB RAM, BlockManagerId(52, ip-10-0-49-229.ec2.internal, 44395, None) | |
| 17/06/14 05:42:16 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-229.ec2.internal:44395 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:42:16 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-49-229.ec2.internal:44395 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:42:16 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.229:50552 | |
| 17/06/14 05:42:39 INFO TaskSetManager: Starting task 111.0 in stage 5.0 (TID 685, ip-10-0-49-229.ec2.internal, executor 49, partition 111, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:42:39 INFO TaskSetManager: Finished task 72.0 in stage 3.0 (TID 618) in 1271144 ms on ip-10-0-49-229.ec2.internal (executor 49) (77/135) | |
| 17/06/14 05:42:39 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-49-229.ec2.internal:45654 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:42:41 INFO TaskSetManager: Starting task 112.0 in stage 5.0 (TID 686, ip-10-0-49-229.ec2.internal, executor 49, partition 112, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:42:41 INFO TaskSetManager: Finished task 77.0 in stage 3.0 (TID 621) in 1248892 ms on ip-10-0-49-229.ec2.internal (executor 49) (78/135) | |
| 17/06/14 05:42:41 INFO YarnAllocator: Driver requested a total number of 48 executor(s). | |
| 17/06/14 05:42:41 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 48 executors. | |
| 17/06/14 05:42:43 INFO TaskSetManager: Starting task 9.0 in stage 5.0 (TID 687, ip-10-0-50-96.ec2.internal, executor 41, partition 9, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:42:43 INFO TaskSetManager: Finished task 78.0 in stage 3.0 (TID 622) in 1243795 ms on ip-10-0-50-96.ec2.internal (executor 41) (79/135) | |
| 17/06/14 05:42:43 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-50-96.ec2.internal:37989 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:42:57 INFO BlockManagerInfo: Added rdd_10_86 in memory on ip-10-0-49-229.ec2.internal:45654 (size: 837.3 MB, free: 1989.8 MB) | |
| 17/06/14 05:43:02 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-49-157.ec2.internal:39668 (size: 5.5 KB) | |
| 17/06/14 05:43:02 INFO BlockManagerInfo: Removed rdd_10_68 on ip-10-0-49-157.ec2.internal:39668 in memory (size: 840.7 MB, free: 2.8 GB) | |
| 17/06/14 05:43:03 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 688, ip-10-0-49-229.ec2.internal, executor 49, partition 0, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:03 INFO TaskSetManager: Finished task 86.0 in stage 3.0 (TID 629) in 1243557 ms on ip-10-0-49-229.ec2.internal (executor 49) (80/135) | |
| 17/06/14 05:43:05 INFO TaskSetManager: Starting task 33.0 in stage 5.0 (TID 689, ip-10-0-50-96.ec2.internal, executor 41, partition 33, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:05 INFO TaskSetManager: Finished task 76.0 in stage 3.0 (TID 620) in 1273260 ms on ip-10-0-50-96.ec2.internal (executor 41) (81/135) | |
| 17/06/14 05:43:17 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID 690, ip-10-0-49-229.ec2.internal, executor 46, partition 1, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:17 INFO TaskSetManager: Finished task 82.0 in stage 3.0 (TID 625) in 1274096 ms on ip-10-0-49-229.ec2.internal (executor 46) (82/135) | |
| 17/06/14 05:43:17 INFO YarnAllocator: Driver requested a total number of 47 executor(s). | |
| 17/06/14 05:43:17 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 47 executors. | |
| 17/06/14 05:43:17 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-49-229.ec2.internal:40160 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:43:17 INFO TaskSetManager: Starting task 2.0 in stage 5.0 (TID 691, ip-10-0-49-229.ec2.internal, executor 49, partition 2, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:17 INFO TaskSetManager: Finished task 90.0 in stage 3.0 (TID 633) in 1248639 ms on ip-10-0-49-229.ec2.internal (executor 49) (83/135) | |
| 17/06/14 05:43:21 INFO BlockManagerInfo: Added rdd_10_83 in memory on ip-10-0-49-229.ec2.internal:40160 (size: 831.9 MB, free: 1995.2 MB) | |
| 17/06/14 05:43:23 INFO TaskSetManager: Starting task 63.0 in stage 5.0 (TID 692, ip-10-0-50-96.ec2.internal, executor 42, partition 63, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:23 INFO TaskSetManager: Finished task 74.0 in stage 3.0 (TID 619) in 1292968 ms on ip-10-0-50-96.ec2.internal (executor 42) (84/135) | |
| 17/06/14 05:43:23 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-50-96.ec2.internal:42160 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:43:24 INFO BlockManagerInfo: Added rdd_10_79 in memory on ip-10-0-50-96.ec2.internal:37989 (size: 847.8 MB, free: 1979.3 MB) | |
| 17/06/14 05:43:24 INFO TaskSetManager: Starting task 66.0 in stage 5.0 (TID 693, ip-10-0-50-96.ec2.internal, executor 45, partition 66, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:24 INFO TaskSetManager: Finished task 97.0 in stage 3.0 (TID 644) in 1239430 ms on ip-10-0-50-96.ec2.internal (executor 45) (85/135) | |
| 17/06/14 05:43:24 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-50-96.ec2.internal:40866 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:43:25 INFO TaskSetManager: Starting task 3.0 in stage 5.0 (TID 694, ip-10-0-49-229.ec2.internal, executor 46, partition 3, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:25 INFO TaskSetManager: Finished task 83.0 in stage 3.0 (TID 626) in 1272705 ms on ip-10-0-49-229.ec2.internal (executor 46) (86/135) | |
| 17/06/14 05:43:25 INFO YarnAllocator: Driver requested a total number of 46 executor(s). | |
| 17/06/14 05:43:25 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 46 executors. | |
| 17/06/14 05:43:27 INFO TaskSetManager: Starting task 5.0 in stage 5.0 (TID 695, ip-10-0-49-157.ec2.internal, executor 44, partition 5, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:27 INFO TaskSetManager: Finished task 93.0 in stage 3.0 (TID 636) in 1248403 ms on ip-10-0-49-157.ec2.internal (executor 44) (87/135) | |
| 17/06/14 05:43:27 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-49-157.ec2.internal:45984 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:43:28 INFO BlockManagerInfo: Added rdd_11_79 in memory on ip-10-0-50-96.ec2.internal:37989 (size: 836.0 MB, free: 1143.3 MB) | |
| 17/06/14 05:43:28 INFO BlockManagerInfo: Added rdd_10_88 in memory on ip-10-0-49-157.ec2.internal:45984 (size: 827.7 MB, free: 1999.4 MB) | |
| 17/06/14 05:43:31 INFO TaskSetManager: Starting task 18.0 in stage 5.0 (TID 696, ip-10-0-51-185.ec2.internal, executor 39, partition 18, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:31 INFO TaskSetManager: Finished task 98.0 in stage 3.0 (TID 645) in 1241776 ms on ip-10-0-51-185.ec2.internal (executor 39) (88/135) | |
| 17/06/14 05:43:31 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-51-185.ec2.internal:46252 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:43:31 INFO TaskSetManager: Starting task 129.0 in stage 5.0 (TID 697, ip-10-0-50-96.ec2.internal, executor 41, partition 129, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:31 INFO TaskSetManager: Finished task 79.0 in stage 3.0 (TID 623) in 1290661 ms on ip-10-0-50-96.ec2.internal (executor 41) (89/135) | |
| 17/06/14 05:43:32 INFO BlockManagerInfo: Removed rdd_10_79 on ip-10-0-50-96.ec2.internal:37989 in memory (size: 847.8 MB, free: 1991.1 MB) | |
| 17/06/14 05:43:35 INFO TaskSetManager: Starting task 10.0 in stage 5.0 (TID 698, ip-10-0-49-157.ec2.internal, executor 44, partition 10, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:35 INFO TaskSetManager: Finished task 88.0 in stage 3.0 (TID 631) in 1272928 ms on ip-10-0-49-157.ec2.internal (executor 44) (90/135) | |
| 17/06/14 05:43:35 INFO YarnAllocator: Driver requested a total number of 45 executor(s). | |
| 17/06/14 05:43:35 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 45 executors. | |
| 17/06/14 05:43:36 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-49-229.ec2.internal:40160 (size: 5.5 KB) | |
| 17/06/14 05:43:36 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-49-229.ec2.internal:40160 (size: 5.5 KB) | |
| 17/06/14 05:43:36 INFO BlockManagerInfo: Removed rdd_10_83 on ip-10-0-49-229.ec2.internal:40160 in memory (size: 831.9 MB, free: 2.8 GB) | |
| 17/06/14 05:43:37 INFO TaskSetManager: Starting task 4.0 in stage 5.0 (TID 699, ip-10-0-50-96.ec2.internal, executor 45, partition 4, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:37 INFO TaskSetManager: Finished task 87.0 in stage 3.0 (TID 630) in 1276137 ms on ip-10-0-50-96.ec2.internal (executor 45) (91/135) | |
| 17/06/14 05:43:40 INFO TaskSetManager: Starting task 6.0 in stage 5.0 (TID 700, ip-10-0-50-96.ec2.internal, executor 42, partition 6, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:40 INFO TaskSetManager: Finished task 80.0 in stage 3.0 (TID 624) in 1298184 ms on ip-10-0-50-96.ec2.internal (executor 42) (92/135) | |
| 17/06/14 05:43:54 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 46. | |
| 17/06/14 05:43:54 INFO DAGScheduler: Executor lost: 46 (epoch 1) | |
| 17/06/14 05:43:54 INFO BlockManagerMasterEndpoint: Trying to remove executor 46 from BlockManagerMaster. | |
| 17/06/14 05:43:54 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(46, ip-10-0-49-229.ec2.internal, 40160, None) | |
| 17/06/14 05:43:54 INFO BlockManagerMaster: Removed 46 successfully in removeExecutor | |
| 17/06/14 05:43:54 INFO YarnAllocator: Launching container container_1497410974650_0004_01_029923 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 05:43:54 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:43:54 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022611 on host: ip-10-0-49-229.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:43:54 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:43:54 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:43:54 ERROR YarnClusterScheduler: Lost executor 46 on ip-10-0-49-229.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:43:54 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:43:54 WARN TaskSetManager: Lost task 115.0 in stage 3.0 (TID 663, ip-10-0-49-229.ec2.internal, executor 46): ExecutorLostFailure (executor 46 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:43:54 WARN TaskSetManager: Lost task 105.0 in stage 3.0 (TID 655, ip-10-0-49-229.ec2.internal, executor 46): ExecutorLostFailure (executor 46 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:43:54 WARN TaskSetManager: Lost task 1.0 in stage 5.0 (TID 690, ip-10-0-49-229.ec2.internal, executor 46): ExecutorLostFailure (executor 46 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:43:54 WARN TaskSetManager: Lost task 3.0 in stage 5.0 (TID 694, ip-10-0-49-229.ec2.internal, executor 46): ExecutorLostFailure (executor 46 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:43:54 INFO BlockManagerMaster: Removal of executor 46 requested | |
| 17/06/14 05:43:54 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 46 | |
| 17/06/14 05:43:54 INFO BlockManagerMasterEndpoint: Trying to remove executor 46 from BlockManagerMaster. | |
| 17/06/14 05:43:54 INFO ExecutorAllocationManager: Existing executor 46 has been removed (new total is 14) | |
| 17/06/14 05:43:54 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 05:43:54 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:43:54 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:43:54 INFO TaskSetManager: Starting task 105.1 in stage 3.0 (TID 701, ip-10-0-49-157.ec2.internal, executor 51, partition 105, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:54 INFO TaskSetManager: Finished task 35.2 in stage 3.0 (TID 651) in 1262334 ms on ip-10-0-49-157.ec2.internal (executor 51) (93/135) | |
| 17/06/14 05:43:55 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:50616) with ID 53 | |
| 17/06/14 05:43:55 INFO TaskSetManager: Starting task 115.1 in stage 3.0 (TID 702, ip-10-0-49-229.ec2.internal, executor 53, partition 115, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:55 INFO TaskSetManager: Starting task 3.1 in stage 5.0 (TID 703, ip-10-0-49-229.ec2.internal, executor 53, partition 3, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:55 INFO ExecutorAllocationManager: New executor 53 has registered (new total is 15) | |
| 17/06/14 05:43:55 INFO TaskSetManager: Starting task 1.1 in stage 5.0 (TID 704, ip-10-0-49-229.ec2.internal, executor 53, partition 1, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:55 INFO TaskSetManager: Starting task 7.0 in stage 5.0 (TID 705, ip-10-0-49-229.ec2.internal, executor 53, partition 7, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:56 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:44599 with 2.8 GB RAM, BlockManagerId(53, ip-10-0-49-229.ec2.internal, 44599, None) | |
| 17/06/14 05:43:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-49-229.ec2.internal:44599 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:43:56 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-49-229.ec2.internal:44599 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:43:56 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.49.229:50616 | |
| 17/06/14 05:43:58 INFO TaskSetManager: Starting task 22.0 in stage 5.0 (TID 706, ip-10-0-51-185.ec2.internal, executor 38, partition 22, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:58 INFO TaskSetManager: Finished task 0.2 in stage 3.0 (TID 648) in 1265621 ms on ip-10-0-51-185.ec2.internal (executor 38) (94/135) | |
| 17/06/14 05:43:58 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-51-185.ec2.internal:37560 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:43:58 INFO YarnAllocator: Driver requested a total number of 44 executor(s). | |
| 17/06/14 05:43:58 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 44 executors. | |
| 17/06/14 05:43:58 INFO TaskSetManager: Starting task 48.0 in stage 5.0 (TID 707, ip-10-0-49-157.ec2.internal, executor 51, partition 48, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:43:58 INFO TaskSetManager: Finished task 89.1 in stage 3.0 (TID 650) in 1266399 ms on ip-10-0-49-157.ec2.internal (executor 51) (95/135) | |
| 17/06/14 05:43:58 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-49-157.ec2.internal:44407 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:44:00 INFO BlockManagerInfo: Added rdd_10_103 in memory on ip-10-0-51-185.ec2.internal:34029 (size: 813.6 MB, free: 2013.6 MB) | |
| 17/06/14 05:44:02 INFO TaskSetManager: Starting task 25.0 in stage 5.0 (TID 708, ip-10-0-51-185.ec2.internal, executor 37, partition 25, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:02 INFO TaskSetManager: Finished task 104.0 in stage 3.0 (TID 654) in 1266181 ms on ip-10-0-51-185.ec2.internal (executor 37) (96/135) | |
| 17/06/14 05:44:02 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-51-185.ec2.internal:33788 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:44:03 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-49-157.ec2.internal:44407 (size: 5.5 KB) | |
| 17/06/14 05:44:03 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-49-157.ec2.internal:44407 (size: 5.5 KB) | |
| 17/06/14 05:44:06 INFO TaskSetManager: Starting task 73.0 in stage 5.0 (TID 709, ip-10-0-49-157.ec2.internal, executor 40, partition 73, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:06 INFO TaskSetManager: Finished task 106.0 in stage 3.0 (TID 656) in 1269795 ms on ip-10-0-49-157.ec2.internal (executor 40) (97/135) | |
| 17/06/14 05:44:06 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-49-157.ec2.internal:42968 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:44:06 INFO TaskSetManager: Starting task 99.0 in stage 5.0 (TID 710, ip-10-0-51-185.ec2.internal, executor 48, partition 99, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:06 INFO TaskSetManager: Finished task 103.0 in stage 3.0 (TID 653) in 1273464 ms on ip-10-0-51-185.ec2.internal (executor 48) (98/135) | |
| 17/06/14 05:44:06 INFO YarnAllocator: Driver requested a total number of 43 executor(s). | |
| 17/06/14 05:44:06 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 43 executors. | |
| 17/06/14 05:44:06 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-51-185.ec2.internal:34029 (size: 5.5 KB, free: 2013.6 MB) | |
| 17/06/14 05:44:06 INFO BlockManagerInfo: Added rdd_10_92 in memory on ip-10-0-51-185.ec2.internal:46252 (size: 818.5 MB, free: 2008.7 MB) | |
| 17/06/14 05:44:07 INFO TaskSetManager: Starting task 81.0 in stage 5.0 (TID 711, ip-10-0-49-157.ec2.internal, executor 51, partition 81, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:07 INFO TaskSetManager: Finished task 102.0 in stage 3.0 (TID 652) in 1274727 ms on ip-10-0-49-157.ec2.internal (executor 51) (99/135) | |
| 17/06/14 05:44:08 INFO TaskSetManager: Starting task 131.0 in stage 5.0 (TID 712, ip-10-0-49-157.ec2.internal, executor 44, partition 131, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:08 INFO TaskSetManager: Finished task 110.0 in stage 3.0 (TID 660) in 1265421 ms on ip-10-0-49-157.ec2.internal (executor 44) (100/135) | |
| 17/06/14 05:44:09 INFO TaskSetManager: Starting task 8.0 in stage 5.0 (TID 713, ip-10-0-50-96.ec2.internal, executor 45, partition 8, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:09 INFO TaskSetManager: Finished task 118.0 in stage 3.0 (TID 666) in 1254784 ms on ip-10-0-50-96.ec2.internal (executor 45) (101/135) | |
| 17/06/14 05:44:11 INFO TaskSetManager: Starting task 134.0 in stage 5.0 (TID 714, ip-10-0-49-157.ec2.internal, executor 51, partition 134, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:11 INFO TaskSetManager: Finished task 17.2 in stage 3.0 (TID 649) in 1278892 ms on ip-10-0-49-157.ec2.internal (executor 51) (102/135) | |
| 17/06/14 05:44:11 INFO YarnAllocator: Driver requested a total number of 42 executor(s). | |
| 17/06/14 05:44:11 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 42 executors. | |
| 17/06/14 05:44:12 INFO TaskSetManager: Starting task 11.0 in stage 5.0 (TID 715, ip-10-0-51-185.ec2.internal, executor 39, partition 11, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:12 INFO TaskSetManager: Finished task 92.0 in stage 3.0 (TID 635) in 1298472 ms on ip-10-0-51-185.ec2.internal (executor 39) (103/135) | |
| 17/06/14 05:44:17 INFO TaskSetManager: Starting task 12.0 in stage 5.0 (TID 716, ip-10-0-51-185.ec2.internal, executor 38, partition 12, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:17 INFO TaskSetManager: Finished task 85.0 in stage 3.0 (TID 628) in 1321494 ms on ip-10-0-51-185.ec2.internal (executor 38) (104/135) | |
| 17/06/14 05:44:17 INFO TaskSetManager: Starting task 13.0 in stage 5.0 (TID 717, ip-10-0-51-185.ec2.internal, executor 48, partition 13, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:17 INFO TaskSetManager: Finished task 113.0 in stage 3.0 (TID 661) in 1274835 ms on ip-10-0-51-185.ec2.internal (executor 48) (105/135) | |
| 17/06/14 05:44:19 INFO BlockManagerInfo: Added rdd_10_16 in memory on ip-10-0-51-185.ec2.internal:37560 (size: 823.3 MB, free: 2003.8 MB) | |
| 17/06/14 05:44:24 INFO BlockManagerInfo: Added rdd_11_16 in memory on ip-10-0-51-185.ec2.internal:37560 (size: 818.1 MB, free: 1185.7 MB) | |
| 17/06/14 05:44:25 INFO BlockManagerInfo: Added rdd_10_96 in memory on ip-10-0-50-96.ec2.internal:42160 (size: 836.2 MB, free: 1990.9 MB) | |
| 17/06/14 05:44:26 INFO BlockManagerInfo: Added rdd_10_109 in memory on ip-10-0-50-96.ec2.internal:40866 (size: 833.7 MB, free: 1993.4 MB) | |
| 17/06/14 05:44:27 INFO TaskSetManager: Starting task 14.0 in stage 5.0 (TID 718, ip-10-0-51-185.ec2.internal, executor 38, partition 14, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:27 INFO TaskSetManager: Finished task 16.2 in stage 3.0 (TID 637) in 1306104 ms on ip-10-0-51-185.ec2.internal (executor 38) (106/135) | |
| 17/06/14 05:44:27 INFO YarnAllocator: Driver requested a total number of 41 executor(s). | |
| 17/06/14 05:44:27 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 41 executors. | |
| 17/06/14 05:44:30 INFO BlockManagerInfo: Added rdd_11_109 in memory on ip-10-0-50-96.ec2.internal:40866 (size: 798.3 MB, free: 1195.1 MB) | |
| 17/06/14 05:44:30 INFO TaskSetManager: Starting task 15.0 in stage 5.0 (TID 719, ip-10-0-51-185.ec2.internal, executor 37, partition 15, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:30 INFO TaskSetManager: Finished task 108.0 in stage 3.0 (TID 658) in 1290520 ms on ip-10-0-51-185.ec2.internal (executor 37) (107/135) | |
| 17/06/14 05:44:31 INFO TaskSetManager: Starting task 16.0 in stage 5.0 (TID 720, ip-10-0-50-96.ec2.internal, executor 42, partition 16, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:31 INFO TaskSetManager: Finished task 96.0 in stage 3.0 (TID 643) in 1308455 ms on ip-10-0-50-96.ec2.internal (executor 42) (108/135) | |
| 17/06/14 05:44:32 INFO TaskSetManager: Starting task 17.0 in stage 5.0 (TID 721, ip-10-0-50-96.ec2.internal, executor 45, partition 17, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:32 INFO TaskSetManager: Finished task 109.0 in stage 3.0 (TID 659) in 1292611 ms on ip-10-0-50-96.ec2.internal (executor 45) (109/135) | |
| 17/06/14 05:44:34 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-50-96.ec2.internal:40866 (size: 5.5 KB) | |
| 17/06/14 05:44:34 INFO BlockManagerInfo: Removed rdd_10_109 on ip-10-0-50-96.ec2.internal:40866 in memory (size: 833.7 MB, free: 2028.8 MB) | |
| 17/06/14 05:44:34 INFO BlockManagerInfo: Added rdd_10_114 in memory on ip-10-0-49-157.ec2.internal:42968 (size: 790.1 MB, free: 2037.0 MB) | |
| 17/06/14 05:44:36 INFO TaskSetManager: Starting task 19.0 in stage 5.0 (TID 722, ip-10-0-51-185.ec2.internal, executor 48, partition 19, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:36 INFO TaskSetManager: Finished task 117.0 in stage 3.0 (TID 665) in 1283030 ms on ip-10-0-51-185.ec2.internal (executor 48) (110/135) | |
| 17/06/14 05:44:36 INFO YarnAllocator: Driver requested a total number of 40 executor(s). | |
| 17/06/14 05:44:36 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 40 executors. | |
| 17/06/14 05:44:39 INFO TaskSetManager: Starting task 20.0 in stage 5.0 (TID 723, ip-10-0-49-157.ec2.internal, executor 40, partition 20, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:39 INFO TaskSetManager: Finished task 114.0 in stage 3.0 (TID 662) in 1293877 ms on ip-10-0-49-157.ec2.internal (executor 40) (111/135) | |
| 17/06/14 05:44:40 INFO TaskSetManager: Starting task 21.0 in stage 5.0 (TID 724, ip-10-0-50-96.ec2.internal, executor 50, partition 21, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:40 INFO TaskSetManager: Finished task 43.2 in stage 3.0 (TID 639) in 1318130 ms on ip-10-0-50-96.ec2.internal (executor 50) (112/135) | |
| 17/06/14 05:44:41 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-50-96.ec2.internal:34711 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:44:42 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-49-157.ec2.internal:42968 (size: 5.5 KB) | |
| 17/06/14 05:44:42 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-49-157.ec2.internal:42968 (size: 5.5 KB) | |
| 17/06/14 05:44:42 INFO BlockManagerInfo: Removed rdd_10_114 on ip-10-0-49-157.ec2.internal:42968 in memory (size: 790.1 MB, free: 2.8 GB) | |
| 17/06/14 05:44:42 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 38. | |
| 17/06/14 05:44:42 INFO DAGScheduler: Executor lost: 38 (epoch 1) | |
| 17/06/14 05:44:42 INFO BlockManagerMasterEndpoint: Trying to remove executor 38 from BlockManagerMaster. | |
| 17/06/14 05:44:42 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(38, ip-10-0-51-185.ec2.internal, 37560, None) | |
| 17/06/14 05:44:42 INFO BlockManagerMaster: Removed 38 successfully in removeExecutor | |
| 17/06/14 05:44:42 INFO YarnAllocator: Launching container container_1497410974650_0004_01_030366 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 05:44:42 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:44:42 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022662 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:44:42 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:44:42 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:44:42 ERROR YarnClusterScheduler: Lost executor 38 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:44:42 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:44:42 WARN TaskSetManager: Lost task 107.0 in stage 3.0 (TID 657, ip-10-0-51-185.ec2.internal, executor 38): ExecutorLostFailure (executor 38 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:44:42 WARN TaskSetManager: Lost task 12.0 in stage 5.0 (TID 716, ip-10-0-51-185.ec2.internal, executor 38): ExecutorLostFailure (executor 38 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:44:42 WARN TaskSetManager: Lost task 22.0 in stage 5.0 (TID 706, ip-10-0-51-185.ec2.internal, executor 38): ExecutorLostFailure (executor 38 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:44:42 WARN TaskSetManager: Lost task 14.0 in stage 5.0 (TID 718, ip-10-0-51-185.ec2.internal, executor 38): ExecutorLostFailure (executor 38 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:44:42 INFO BlockManagerMaster: Removal of executor 38 requested | |
| 17/06/14 05:44:42 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 38 | |
| 17/06/14 05:44:42 INFO BlockManagerMasterEndpoint: Trying to remove executor 38 from BlockManagerMaster. | |
| 17/06/14 05:44:42 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 05:44:42 INFO ExecutorAllocationManager: Existing executor 38 has been removed (new total is 14) | |
| 17/06/14 05:44:42 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:44:42 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:44:44 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:35966) with ID 54 | |
| 17/06/14 05:44:44 INFO TaskSetManager: Starting task 107.1 in stage 3.0 (TID 725, ip-10-0-51-185.ec2.internal, executor 54, partition 107, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:44 INFO TaskSetManager: Starting task 22.1 in stage 5.0 (TID 726, ip-10-0-51-185.ec2.internal, executor 54, partition 22, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:44 INFO TaskSetManager: Starting task 14.1 in stage 5.0 (TID 727, ip-10-0-51-185.ec2.internal, executor 54, partition 14, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:44 INFO TaskSetManager: Starting task 12.1 in stage 5.0 (TID 728, ip-10-0-51-185.ec2.internal, executor 54, partition 12, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:44 INFO ExecutorAllocationManager: New executor 54 has registered (new total is 15) | |
| 17/06/14 05:44:44 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:45572 with 2.8 GB RAM, BlockManagerId(54, ip-10-0-51-185.ec2.internal, 45572, None) | |
| 17/06/14 05:44:44 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-51-185.ec2.internal:45572 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:44:44 INFO TaskSetManager: Starting task 23.0 in stage 5.0 (TID 729, ip-10-0-49-157.ec2.internal, executor 40, partition 23, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:44 INFO TaskSetManager: Finished task 95.0 in stage 3.0 (TID 642) in 1321592 ms on ip-10-0-49-157.ec2.internal (executor 40) (113/135) | |
| 17/06/14 05:44:44 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-51-185.ec2.internal:45572 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:44:44 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:35966 | |
| 17/06/14 05:44:47 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-49-229.ec2.internal:45654 (size: 5.5 KB) | |
| 17/06/14 05:44:47 INFO BlockManagerInfo: Removed rdd_10_86 on ip-10-0-49-229.ec2.internal:45654 in memory (size: 837.3 MB, free: 2.8 GB) | |
| 17/06/14 05:44:48 INFO TaskSetManager: Starting task 24.0 in stage 5.0 (TID 730, ip-10-0-49-157.ec2.internal, executor 44, partition 24, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:48 INFO TaskSetManager: Finished task 101.0 in stage 3.0 (TID 647) in 1318330 ms on ip-10-0-49-157.ec2.internal (executor 44) (114/135) | |
| 17/06/14 05:44:48 INFO YarnAllocator: Driver requested a total number of 39 executor(s). | |
| 17/06/14 05:44:48 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 39 executors. | |
| 17/06/14 05:44:48 INFO TaskSetManager: Starting task 26.0 in stage 5.0 (TID 731, ip-10-0-50-96.ec2.internal, executor 42, partition 26, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:48 INFO TaskSetManager: Finished task 16.0 in stage 5.0 (TID 720) in 16861 ms on ip-10-0-50-96.ec2.internal (executor 42) (1/135) | |
| 17/06/14 05:44:49 INFO TaskSetManager: Starting task 27.0 in stage 5.0 (TID 732, ip-10-0-51-185.ec2.internal, executor 39, partition 27, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:44:49 INFO TaskSetManager: Finished task 100.0 in stage 3.0 (TID 646) in 1319751 ms on ip-10-0-51-185.ec2.internal (executor 39) (115/135) | |
| 17/06/14 05:45:00 INFO BlockManagerInfo: Added rdd_10_120 in memory on ip-10-0-49-157.ec2.internal:42968 (size: 855.3 MB, free: 1971.8 MB) | |
| 17/06/14 05:45:00 INFO TaskSetManager: Starting task 28.0 in stage 5.0 (TID 733, ip-10-0-51-185.ec2.internal, executor 48, partition 28, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:45:00 INFO TaskSetManager: Finished task 116.0 in stage 3.0 (TID 664) in 1307887 ms on ip-10-0-51-185.ec2.internal (executor 48) (116/135) | |
| 17/06/14 05:45:04 INFO BlockManagerInfo: Added rdd_11_120 in memory on ip-10-0-49-157.ec2.internal:42968 (size: 821.1 MB, free: 1150.7 MB) | |
| 17/06/14 05:45:06 INFO TaskSetManager: Starting task 29.0 in stage 5.0 (TID 734, ip-10-0-49-157.ec2.internal, executor 40, partition 29, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:45:06 INFO TaskSetManager: Finished task 120.0 in stage 3.0 (TID 668) in 1284102 ms on ip-10-0-49-157.ec2.internal (executor 40) (117/135) | |
| 17/06/14 05:45:06 INFO YarnAllocator: Driver requested a total number of 38 executor(s). | |
| 17/06/14 05:45:06 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 38 executors. | |
| 17/06/14 05:45:07 INFO BlockManagerInfo: Removed rdd_10_120 on ip-10-0-49-157.ec2.internal:42968 in memory (size: 855.3 MB, free: 2006.0 MB) | |
| 17/06/14 05:45:24 INFO BlockManagerInfo: Added rdd_10_121 in memory on ip-10-0-51-185.ec2.internal:33788 (size: 841.3 MB, free: 1985.8 MB) | |
| 17/06/14 05:45:27 INFO BlockManagerInfo: Added rdd_11_121 in memory on ip-10-0-51-185.ec2.internal:33788 (size: 827.0 MB, free: 1158.8 MB) | |
| 17/06/14 05:45:29 INFO TaskSetManager: Starting task 30.0 in stage 5.0 (TID 735, ip-10-0-50-96.ec2.internal, executor 50, partition 30, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:45:29 INFO TaskSetManager: Finished task 91.1 in stage 3.0 (TID 638) in 1366539 ms on ip-10-0-50-96.ec2.internal (executor 50) (118/135) | |
| 17/06/14 05:45:30 INFO TaskSetManager: Starting task 31.0 in stage 5.0 (TID 736, ip-10-0-51-185.ec2.internal, executor 37, partition 31, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:45:30 INFO TaskSetManager: Finished task 121.0 in stage 3.0 (TID 669) in 1298885 ms on ip-10-0-51-185.ec2.internal (executor 37) (119/135) | |
| 17/06/14 05:45:31 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-51-185.ec2.internal:33788 (size: 5.5 KB) | |
| 17/06/14 05:45:31 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-51-185.ec2.internal:33788 (size: 5.5 KB) | |
| 17/06/14 05:45:31 INFO BlockManagerInfo: Removed rdd_10_121 on ip-10-0-51-185.ec2.internal:33788 in memory (size: 841.3 MB, free: 2000.1 MB) | |
| 17/06/14 05:45:32 INFO BlockManagerInfo: Added rdd_10_94 in memory on ip-10-0-50-96.ec2.internal:34711 (size: 816.4 MB, free: 2010.7 MB) | |
| 17/06/14 05:45:34 INFO TaskSetManager: Starting task 32.0 in stage 5.0 (TID 737, ip-10-0-51-185.ec2.internal, executor 39, partition 32, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:45:34 INFO TaskSetManager: Finished task 119.0 in stage 3.0 (TID 667) in 1338208 ms on ip-10-0-51-185.ec2.internal (executor 39) (120/135) | |
| 17/06/14 05:45:38 INFO TaskSetManager: Starting task 34.0 in stage 5.0 (TID 738, ip-10-0-50-96.ec2.internal, executor 50, partition 34, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:45:38 INFO TaskSetManager: Finished task 94.0 in stage 3.0 (TID 641) in 1375411 ms on ip-10-0-50-96.ec2.internal (executor 50) (121/135) | |
| 17/06/14 05:45:38 INFO YarnAllocator: Driver requested a total number of 37 executor(s). | |
| 17/06/14 05:45:38 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 37 executors. | |
| 17/06/14 05:45:40 INFO TaskSetManager: Starting task 35.0 in stage 5.0 (TID 739, ip-10-0-51-185.ec2.internal, executor 37, partition 35, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:45:40 INFO TaskSetManager: Finished task 122.0 in stage 3.0 (TID 670) in 1297537 ms on ip-10-0-51-185.ec2.internal (executor 37) (122/135) | |
| 17/06/14 05:45:50 INFO TaskSetManager: Starting task 36.0 in stage 5.0 (TID 740, ip-10-0-50-96.ec2.internal, executor 50, partition 36, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:45:50 INFO TaskSetManager: Finished task 84.1 in stage 3.0 (TID 640) in 1387329 ms on ip-10-0-50-96.ec2.internal (executor 50) (123/135) | |
| 17/06/14 05:45:54 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 42. | |
| 17/06/14 05:45:54 INFO DAGScheduler: Executor lost: 42 (epoch 1) | |
| 17/06/14 05:45:54 INFO BlockManagerMasterEndpoint: Trying to remove executor 42 from BlockManagerMaster. | |
| 17/06/14 05:45:54 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(42, ip-10-0-50-96.ec2.internal, 42160, None) | |
| 17/06/14 05:45:54 INFO BlockManagerMaster: Removed 42 successfully in removeExecutor | |
| 17/06/14 05:45:54 INFO YarnAllocator: Launching container container_1497410974650_0004_01_030582 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 05:45:54 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 05:45:54 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022670 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 05:45:54 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:45:54 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:45:54 ERROR YarnClusterScheduler: Lost executor 42 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:45:54 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 05:45:54 WARN TaskSetManager: Lost task 130.0 in stage 3.0 (TID 677, ip-10-0-50-96.ec2.internal, executor 42): ExecutorLostFailure (executor 42 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:45:54 WARN TaskSetManager: Lost task 63.0 in stage 5.0 (TID 692, ip-10-0-50-96.ec2.internal, executor 42): ExecutorLostFailure (executor 42 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:45:54 WARN TaskSetManager: Lost task 26.0 in stage 5.0 (TID 731, ip-10-0-50-96.ec2.internal, executor 42): ExecutorLostFailure (executor 42 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:45:54 WARN TaskSetManager: Lost task 6.0 in stage 5.0 (TID 700, ip-10-0-50-96.ec2.internal, executor 42): ExecutorLostFailure (executor 42 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 05:45:54 INFO BlockManagerMaster: Removal of executor 42 requested | |
| 17/06/14 05:45:54 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 42 | |
| 17/06/14 05:45:54 INFO BlockManagerMasterEndpoint: Trying to remove executor 42 from BlockManagerMaster. | |
| 17/06/14 05:45:54 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 05:45:54 INFO ExecutorAllocationManager: Existing executor 42 has been removed (new total is 14) | |
| 17/06/14 05:45:54 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 05:45:54 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 05:45:56 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:37288) with ID 55 | |
| 17/06/14 05:45:56 INFO ExecutorAllocationManager: New executor 55 has registered (new total is 15) | |
| 17/06/14 05:45:56 INFO TaskSetManager: Starting task 130.1 in stage 3.0 (TID 741, ip-10-0-50-96.ec2.internal, executor 55, partition 130, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:45:56 INFO TaskSetManager: Starting task 63.1 in stage 5.0 (TID 742, ip-10-0-50-96.ec2.internal, executor 55, partition 63, NODE_LOCAL, 5763 bytes) | |
| 17/06/14 05:45:56 INFO TaskSetManager: Starting task 6.1 in stage 5.0 (TID 743, ip-10-0-50-96.ec2.internal, executor 55, partition 6, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:45:56 INFO TaskSetManager: Starting task 26.1 in stage 5.0 (TID 744, ip-10-0-50-96.ec2.internal, executor 55, partition 26, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 05:45:56 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:40971 with 2.8 GB RAM, BlockManagerId(55, ip-10-0-50-96.ec2.internal, 40971, None) | |
| 17/06/14 05:45:56 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-50-96.ec2.internal:40971 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:45:57 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-10-0-50-96.ec2.internal:40971 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 05:45:57 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:37288 | |
| 17/06/14 05:46:00 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-50-96.ec2.internal:37989 (size: 5.5 KB) | |
| 17/06/14 05:46:00 INFO BlockManagerInfo: Removed rdd_11_79 on ip-10-0-50-96.ec2.internal:37989 in memory (size: 836.0 MB, free: 2.8 GB) | |
| 17/06/14 05:46:08 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-50-96.ec2.internal:34711 (size: 5.5 KB) | |
| 17/06/14 05:46:08 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-50-96.ec2.internal:34711 (size: 5.5 KB) | |
| 17/06/14 05:46:08 INFO BlockManagerInfo: Removed rdd_10_94 on ip-10-0-50-96.ec2.internal:34711 in memory (size: 816.4 MB, free: 2.8 GB) | |
| 17/06/14 05:46:24 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-51-185.ec2.internal:46252 (size: 5.5 KB) | |
| 17/06/14 05:46:24 INFO BlockManagerInfo: Removed rdd_10_92 on ip-10-0-51-185.ec2.internal:46252 in memory (size: 818.5 MB, free: 2.8 GB) | |
| 17/06/14 05:47:05 INFO BlockManagerInfo: Removed rdd_10_103 on ip-10-0-51-185.ec2.internal:34029 in memory (size: 813.6 MB, free: 2.8 GB) | |
| 17/06/14 05:47:15 INFO BlockManagerInfo: Removed rdd_11_121 on ip-10-0-51-185.ec2.internal:33788 in memory (size: 827.0 MB, free: 2.8 GB) | |
| 17/06/14 05:48:05 INFO BlockManagerInfo: Removed rdd_11_109 on ip-10-0-50-96.ec2.internal:40866 in memory (size: 798.3 MB, free: 2.8 GB) | |
| 17/06/14 05:48:11 INFO BlockManagerInfo: Removed rdd_11_120 on ip-10-0-49-157.ec2.internal:42968 in memory (size: 821.1 MB, free: 2.8 GB) | |
| 17/06/14 05:49:54 INFO BlockManagerInfo: Removed rdd_10_88 on ip-10-0-49-157.ec2.internal:45984 in memory (size: 827.7 MB, free: 2.8 GB) | |
| 17/06/14 06:01:50 INFO TaskSetManager: Starting task 37.0 in stage 5.0 (TID 745, ip-10-0-50-96.ec2.internal, executor 41, partition 37, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:01:50 INFO TaskSetManager: Finished task 125.0 in stage 3.0 (TID 673) in 1253625 ms on ip-10-0-50-96.ec2.internal (executor 41) (124/135) | |
| 17/06/14 06:01:59 INFO TaskSetManager: Starting task 38.0 in stage 5.0 (TID 746, ip-10-0-49-157.ec2.internal, executor 36, partition 38, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:01:59 INFO TaskSetManager: Finished task 123.0 in stage 3.0 (TID 671) in 1290345 ms on ip-10-0-49-157.ec2.internal (executor 36) (125/135) | |
| 17/06/14 06:01:59 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on ip-10-0-49-157.ec2.internal:39668 (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 06:01:59 INFO YarnAllocator: Driver requested a total number of 36 executor(s). | |
| 17/06/14 06:01:59 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 36 executors. | |
| 17/06/14 06:02:18 INFO TaskSetManager: Starting task 39.0 in stage 5.0 (TID 747, ip-10-0-49-157.ec2.internal, executor 36, partition 39, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:02:18 INFO TaskSetManager: Finished task 127.0 in stage 3.0 (TID 675) in 1252834 ms on ip-10-0-49-157.ec2.internal (executor 36) (126/135) | |
| 17/06/14 06:02:54 INFO BlockManagerInfo: Added rdd_11_126 in memory on ip-10-0-49-157.ec2.internal:39668 (size: 860.3 MB, free: 1966.9 MB) | |
| 17/06/14 06:02:57 INFO TaskSetManager: Starting task 40.0 in stage 5.0 (TID 748, ip-10-0-49-157.ec2.internal, executor 36, partition 40, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:02:57 INFO TaskSetManager: Finished task 126.0 in stage 3.0 (TID 674) in 1316962 ms on ip-10-0-49-157.ec2.internal (executor 36) (127/135) | |
| 17/06/14 06:03:04 INFO TaskSetManager: Starting task 41.0 in stage 5.0 (TID 749, ip-10-0-50-96.ec2.internal, executor 41, partition 41, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:03:04 INFO TaskSetManager: Finished task 9.0 in stage 5.0 (TID 687) in 1220841 ms on ip-10-0-50-96.ec2.internal (executor 41) (2/135) | |
| 17/06/14 06:03:33 INFO BlockManagerInfo: Added rdd_10_75 in memory on ip-10-0-49-229.ec2.internal:44395 (size: 851.0 MB, free: 1976.2 MB) | |
| 17/06/14 06:03:39 INFO TaskSetManager: Starting task 42.0 in stage 5.0 (TID 750, ip-10-0-49-229.ec2.internal, executor 52, partition 42, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:03:39 INFO TaskSetManager: Finished task 75.1 in stage 5.0 (TID 684) in 1283872 ms on ip-10-0-49-229.ec2.internal (executor 52) (3/135) | |
| 17/06/14 06:03:39 INFO YarnAllocator: Driver requested a total number of 35 executor(s). | |
| 17/06/14 06:03:39 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 35 executors. | |
| 17/06/14 06:03:42 INFO TaskSetManager: Starting task 43.0 in stage 5.0 (TID 751, ip-10-0-49-157.ec2.internal, executor 36, partition 43, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:03:42 INFO TaskSetManager: Finished task 128.0 in stage 3.0 (TID 676) in 1316564 ms on ip-10-0-49-157.ec2.internal (executor 36) (128/135) | |
| 17/06/14 06:03:45 INFO TaskSetManager: Starting task 44.0 in stage 5.0 (TID 752, ip-10-0-49-229.ec2.internal, executor 52, partition 44, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:03:45 INFO TaskSetManager: Finished task 124.1 in stage 3.0 (TID 682) in 1289956 ms on ip-10-0-49-229.ec2.internal (executor 52) (129/135) | |
| 17/06/14 06:03:52 INFO TaskSetManager: Starting task 45.0 in stage 5.0 (TID 753, ip-10-0-49-229.ec2.internal, executor 49, partition 45, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:03:52 INFO TaskSetManager: Finished task 112.0 in stage 5.0 (TID 686) in 1270731 ms on ip-10-0-49-229.ec2.internal (executor 49) (4/135) | |
| 17/06/14 06:03:59 INFO TaskSetManager: Starting task 46.0 in stage 5.0 (TID 754, ip-10-0-50-96.ec2.internal, executor 41, partition 46, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:03:59 INFO TaskSetManager: Finished task 33.0 in stage 5.0 (TID 689) in 1254669 ms on ip-10-0-50-96.ec2.internal (executor 41) (5/135) | |
| 17/06/14 06:03:59 INFO YarnAllocator: Driver requested a total number of 34 executor(s). | |
| 17/06/14 06:03:59 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 34 executors. | |
| 17/06/14 06:04:00 INFO TaskSetManager: Starting task 47.0 in stage 5.0 (TID 755, ip-10-0-49-229.ec2.internal, executor 52, partition 47, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:04:00 INFO TaskSetManager: Finished task 132.1 in stage 3.0 (TID 683) in 1304648 ms on ip-10-0-49-229.ec2.internal (executor 52) (130/135) | |
| 17/06/14 06:04:03 INFO TaskSetManager: Starting task 49.0 in stage 5.0 (TID 756, ip-10-0-49-229.ec2.internal, executor 52, partition 49, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:04:03 INFO TaskSetManager: Finished task 133.1 in stage 3.0 (TID 681) in 1307218 ms on ip-10-0-49-229.ec2.internal (executor 52) (131/135) | |
| 17/06/14 06:04:06 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-49-157.ec2.internal:39668 (size: 5.5 KB) | |
| 17/06/14 06:04:06 INFO BlockManagerInfo: Removed rdd_11_126 on ip-10-0-49-157.ec2.internal:39668 in memory (size: 860.3 MB, free: 2.8 GB) | |
| 17/06/14 06:04:08 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-49-229.ec2.internal:44395 (size: 5.5 KB) | |
| 17/06/14 06:04:08 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-49-229.ec2.internal:44395 (size: 5.5 KB) | |
| 17/06/14 06:04:08 INFO BlockManagerInfo: Removed rdd_10_75 on ip-10-0-49-229.ec2.internal:44395 in memory (size: 851.0 MB, free: 2.8 GB) | |
| 17/06/14 06:04:26 INFO TaskSetManager: Starting task 50.0 in stage 5.0 (TID 757, ip-10-0-50-96.ec2.internal, executor 45, partition 50, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:04:26 INFO TaskSetManager: Finished task 4.0 in stage 5.0 (TID 699) in 1248763 ms on ip-10-0-50-96.ec2.internal (executor 45) (6/135) | |
| 17/06/14 06:04:29 INFO BlockManagerInfo: Added rdd_10_129 in memory on ip-10-0-50-96.ec2.internal:37989 (size: 788.4 MB, free: 2038.7 MB) | |
| 17/06/14 06:04:30 INFO TaskSetManager: Starting task 51.0 in stage 5.0 (TID 758, ip-10-0-49-229.ec2.internal, executor 49, partition 51, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:04:30 INFO TaskSetManager: Finished task 2.0 in stage 5.0 (TID 691) in 1272945 ms on ip-10-0-49-229.ec2.internal (executor 49) (7/135) | |
| 17/06/14 06:04:30 INFO YarnAllocator: Driver requested a total number of 33 executor(s). | |
| 17/06/14 06:04:30 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 33 executors. | |
| 17/06/14 06:04:35 INFO TaskSetManager: Starting task 52.0 in stage 5.0 (TID 759, ip-10-0-50-96.ec2.internal, executor 41, partition 52, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:04:35 INFO TaskSetManager: Finished task 129.0 in stage 5.0 (TID 697) in 1264123 ms on ip-10-0-50-96.ec2.internal (executor 41) (8/135) | |
| 17/06/14 06:04:38 INFO TaskSetManager: Starting task 53.0 in stage 5.0 (TID 760, ip-10-0-49-229.ec2.internal, executor 53, partition 53, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:04:38 INFO TaskSetManager: Finished task 3.1 in stage 5.0 (TID 703) in 1242869 ms on ip-10-0-49-229.ec2.internal (executor 53) (9/135) | |
| 17/06/14 06:04:39 INFO BlockManagerInfo: Added rdd_10_25 in memory on ip-10-0-51-185.ec2.internal:33788 (size: 809.9 MB, free: 2017.2 MB) | |
| 17/06/14 06:04:45 INFO TaskSetManager: Starting task 54.0 in stage 5.0 (TID 761, ip-10-0-51-185.ec2.internal, executor 37, partition 54, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:04:45 INFO TaskSetManager: Finished task 25.0 in stage 5.0 (TID 708) in 1243029 ms on ip-10-0-51-185.ec2.internal (executor 37) (10/135) | |
| 17/06/14 06:04:52 INFO BlockManagerInfo: Added rdd_10_0 in memory on ip-10-0-49-229.ec2.internal:45654 (size: 890.4 MB, free: 1936.7 MB) | |
| 17/06/14 06:04:57 INFO BlockManagerInfo: Added rdd_10_18 in memory on ip-10-0-51-185.ec2.internal:46252 (size: 848.9 MB, free: 1978.2 MB) | |
| 17/06/14 06:04:58 INFO TaskSetManager: Starting task 55.0 in stage 5.0 (TID 762, ip-10-0-49-229.ec2.internal, executor 49, partition 55, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:04:58 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 688) in 1315175 ms on ip-10-0-49-229.ec2.internal (executor 49) (11/135) | |
| 17/06/14 06:04:58 INFO YarnAllocator: Driver requested a total number of 32 executor(s). | |
| 17/06/14 06:04:58 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 32 executors. | |
| 17/06/14 06:05:03 INFO TaskSetManager: Starting task 56.0 in stage 5.0 (TID 763, ip-10-0-51-185.ec2.internal, executor 39, partition 56, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:03 INFO TaskSetManager: Finished task 18.0 in stage 5.0 (TID 696) in 1292721 ms on ip-10-0-51-185.ec2.internal (executor 39) (12/135) | |
| 17/06/14 06:05:15 INFO TaskSetManager: Starting task 57.0 in stage 5.0 (TID 764, ip-10-0-50-96.ec2.internal, executor 45, partition 57, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:15 INFO TaskSetManager: Finished task 17.0 in stage 5.0 (TID 721) in 1243403 ms on ip-10-0-50-96.ec2.internal (executor 45) (13/135) | |
| 17/06/14 06:05:16 INFO TaskSetManager: Starting task 58.0 in stage 5.0 (TID 765, ip-10-0-49-157.ec2.internal, executor 51, partition 58, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:16 INFO TaskSetManager: Finished task 48.0 in stage 5.0 (TID 707) in 1277236 ms on ip-10-0-49-157.ec2.internal (executor 51) (14/135) | |
| 17/06/14 06:05:16 INFO TaskSetManager: Starting task 59.0 in stage 5.0 (TID 766, ip-10-0-51-185.ec2.internal, executor 48, partition 59, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:16 INFO TaskSetManager: Finished task 28.0 in stage 5.0 (TID 733) in 1215917 ms on ip-10-0-51-185.ec2.internal (executor 48) (15/135) | |
| 17/06/14 06:05:16 INFO YarnAllocator: Driver requested a total number of 31 executor(s). | |
| 17/06/14 06:05:16 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 31 executors. | |
| 17/06/14 06:05:17 INFO BlockManagerInfo: Added rdd_10_19 in memory on ip-10-0-51-185.ec2.internal:34029 (size: 830.9 MB, free: 1996.3 MB) | |
| 17/06/14 06:05:20 INFO TaskSetManager: Starting task 60.0 in stage 5.0 (TID 767, ip-10-0-49-229.ec2.internal, executor 49, partition 60, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:20 INFO TaskSetManager: Finished task 111.0 in stage 5.0 (TID 685) in 1360963 ms on ip-10-0-49-229.ec2.internal (executor 49) (16/135) | |
| 17/06/14 06:05:22 INFO TaskSetManager: Starting task 61.0 in stage 5.0 (TID 768, ip-10-0-50-96.ec2.internal, executor 45, partition 61, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:22 INFO TaskSetManager: Finished task 8.0 in stage 5.0 (TID 713) in 1272702 ms on ip-10-0-50-96.ec2.internal (executor 45) (17/135) | |
| 17/06/14 06:05:23 INFO TaskSetManager: Starting task 62.0 in stage 5.0 (TID 769, ip-10-0-51-185.ec2.internal, executor 39, partition 62, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:23 INFO TaskSetManager: Finished task 11.0 in stage 5.0 (TID 715) in 1270686 ms on ip-10-0-51-185.ec2.internal (executor 39) (18/135) | |
| 17/06/14 06:05:23 INFO TaskSetManager: Starting task 64.0 in stage 5.0 (TID 770, ip-10-0-49-157.ec2.internal, executor 51, partition 64, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:23 INFO TaskSetManager: Finished task 105.1 in stage 3.0 (TID 701) in 1289134 ms on ip-10-0-49-157.ec2.internal (executor 51) (132/135) | |
| 17/06/14 06:05:23 INFO YarnAllocator: Driver requested a total number of 30 executor(s). | |
| 17/06/14 06:05:23 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 30 executors. | |
| 17/06/14 06:05:24 INFO TaskSetManager: Starting task 65.0 in stage 5.0 (TID 771, ip-10-0-49-229.ec2.internal, executor 53, partition 65, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:24 INFO TaskSetManager: Finished task 1.1 in stage 5.0 (TID 704) in 1288296 ms on ip-10-0-49-229.ec2.internal (executor 53) (19/135) | |
| 17/06/14 06:05:24 INFO TaskSetManager: Starting task 67.0 in stage 5.0 (TID 772, ip-10-0-51-185.ec2.internal, executor 48, partition 67, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:24 INFO TaskSetManager: Finished task 13.0 in stage 5.0 (TID 717) in 1266388 ms on ip-10-0-51-185.ec2.internal (executor 48) (20/135) | |
| 17/06/14 06:05:24 INFO TaskSetManager: Starting task 68.0 in stage 5.0 (TID 773, ip-10-0-51-185.ec2.internal, executor 48, partition 68, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:24 INFO TaskSetManager: Finished task 19.0 in stage 5.0 (TID 722) in 1247534 ms on ip-10-0-51-185.ec2.internal (executor 48) (21/135) | |
| 17/06/14 06:05:27 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-51-185.ec2.internal:34029 (size: 5.5 KB) | |
| 17/06/14 06:05:27 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-51-185.ec2.internal:34029 (size: 5.5 KB) | |
| 17/06/14 06:05:27 INFO BlockManagerInfo: Removed rdd_10_19 on ip-10-0-51-185.ec2.internal:34029 in memory (size: 830.9 MB, free: 2.8 GB) | |
| 17/06/14 06:05:28 INFO TaskSetManager: Starting task 69.0 in stage 5.0 (TID 774, ip-10-0-49-157.ec2.internal, executor 51, partition 69, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:28 INFO TaskSetManager: Finished task 81.0 in stage 5.0 (TID 711) in 1281257 ms on ip-10-0-49-157.ec2.internal (executor 51) (22/135) | |
| 17/06/14 06:05:28 INFO YarnAllocator: Driver requested a total number of 29 executor(s). | |
| 17/06/14 06:05:28 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 29 executors. | |
| 17/06/14 06:05:34 INFO TaskSetManager: Starting task 70.0 in stage 5.0 (TID 775, ip-10-0-49-229.ec2.internal, executor 53, partition 70, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:34 INFO TaskSetManager: Finished task 115.1 in stage 3.0 (TID 702) in 1298660 ms on ip-10-0-49-229.ec2.internal (executor 53) (133/135) | |
| 17/06/14 06:05:35 INFO TaskSetManager: Starting task 71.0 in stage 5.0 (TID 776, ip-10-0-49-157.ec2.internal, executor 44, partition 71, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:35 INFO TaskSetManager: Finished task 5.0 in stage 5.0 (TID 695) in 1328163 ms on ip-10-0-49-157.ec2.internal (executor 44) (23/135) | |
| 17/06/14 06:05:36 INFO TaskSetManager: Starting task 72.0 in stage 5.0 (TID 777, ip-10-0-51-185.ec2.internal, executor 48, partition 72, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:36 INFO TaskSetManager: Finished task 99.0 in stage 5.0 (TID 710) in 1290115 ms on ip-10-0-51-185.ec2.internal (executor 48) (24/135) | |
| 17/06/14 06:05:42 INFO BlockManagerInfo: Added rdd_10_66 in memory on ip-10-0-50-96.ec2.internal:40866 (size: 840.6 MB, free: 1986.6 MB) | |
| 17/06/14 06:05:43 INFO TaskSetManager: Starting task 74.0 in stage 5.0 (TID 778, ip-10-0-51-185.ec2.internal, executor 37, partition 74, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:43 INFO TaskSetManager: Finished task 15.0 in stage 5.0 (TID 719) in 1273243 ms on ip-10-0-51-185.ec2.internal (executor 37) (25/135) | |
| 17/06/14 06:05:43 INFO YarnAllocator: Driver requested a total number of 28 executor(s). | |
| 17/06/14 06:05:43 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 28 executors. | |
| 17/06/14 06:05:48 INFO BlockManagerInfo: Added rdd_10_10 in memory on ip-10-0-49-157.ec2.internal:45984 (size: 824.6 MB, free: 2002.6 MB) | |
| 17/06/14 06:05:48 INFO TaskSetManager: Starting task 76.0 in stage 5.0 (TID 779, ip-10-0-50-96.ec2.internal, executor 45, partition 76, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:48 INFO TaskSetManager: Finished task 66.0 in stage 5.0 (TID 693) in 1343783 ms on ip-10-0-50-96.ec2.internal (executor 45) (26/135) | |
| 17/06/14 06:05:51 INFO BlockManagerInfo: Added rdd_11_10 in memory on ip-10-0-49-157.ec2.internal:45984 (size: 801.2 MB, free: 1201.3 MB) | |
| 17/06/14 06:05:54 INFO TaskSetManager: Starting task 77.0 in stage 5.0 (TID 780, ip-10-0-49-157.ec2.internal, executor 44, partition 77, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:54 INFO TaskSetManager: Finished task 10.0 in stage 5.0 (TID 698) in 1338820 ms on ip-10-0-49-157.ec2.internal (executor 44) (27/135) | |
| 17/06/14 06:05:55 INFO TaskSetManager: Starting task 78.0 in stage 5.0 (TID 781, ip-10-0-51-185.ec2.internal, executor 39, partition 78, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:55 INFO TaskSetManager: Finished task 27.0 in stage 5.0 (TID 732) in 1265956 ms on ip-10-0-51-185.ec2.internal (executor 39) (28/135) | |
| 17/06/14 06:05:55 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-49-157.ec2.internal:45984 (size: 5.5 KB) | |
| 17/06/14 06:05:55 INFO BlockManagerInfo: Removed rdd_10_10 on ip-10-0-49-157.ec2.internal:45984 in memory (size: 824.6 MB, free: 2025.9 MB) | |
| 17/06/14 06:05:56 INFO TaskSetManager: Starting task 79.0 in stage 5.0 (TID 782, ip-10-0-49-229.ec2.internal, executor 53, partition 79, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:56 INFO TaskSetManager: Finished task 7.0 in stage 5.0 (TID 705) in 1320558 ms on ip-10-0-49-229.ec2.internal (executor 53) (29/135) | |
| 17/06/14 06:05:56 INFO YarnAllocator: Driver requested a total number of 27 executor(s). | |
| 17/06/14 06:05:56 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 27 executors. | |
| 17/06/14 06:05:59 INFO TaskSetManager: Starting task 80.0 in stage 5.0 (TID 783, ip-10-0-50-96.ec2.internal, executor 50, partition 80, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:59 INFO TaskSetManager: Finished task 21.0 in stage 5.0 (TID 724) in 1278175 ms on ip-10-0-50-96.ec2.internal (executor 50) (30/135) | |
| 17/06/14 06:05:59 INFO TaskSetManager: Starting task 82.0 in stage 5.0 (TID 784, ip-10-0-51-185.ec2.internal, executor 37, partition 82, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:05:59 INFO TaskSetManager: Finished task 31.0 in stage 5.0 (TID 736) in 1229282 ms on ip-10-0-51-185.ec2.internal (executor 37) (31/135) | |
| 17/06/14 06:06:10 INFO TaskSetManager: Starting task 83.0 in stage 5.0 (TID 785, ip-10-0-49-157.ec2.internal, executor 40, partition 83, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:06:10 INFO TaskSetManager: Finished task 20.0 in stage 5.0 (TID 723) in 1290525 ms on ip-10-0-49-157.ec2.internal (executor 40) (32/135) | |
| 17/06/14 06:06:11 INFO TaskSetManager: Starting task 84.0 in stage 5.0 (TID 786, ip-10-0-51-185.ec2.internal, executor 37, partition 84, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:06:11 INFO TaskSetManager: Finished task 35.0 in stage 5.0 (TID 739) in 1231475 ms on ip-10-0-51-185.ec2.internal (executor 37) (33/135) | |
| 17/06/14 06:06:11 INFO YarnAllocator: Driver requested a total number of 26 executor(s). | |
| 17/06/14 06:06:11 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 26 executors. | |
| 17/06/14 06:06:14 INFO BlockManagerInfo: Added rdd_10_134 in memory on ip-10-0-49-157.ec2.internal:44407 (size: 828.0 MB, free: 1999.1 MB) | |
| 17/06/14 06:06:18 INFO BlockManagerInfo: Added rdd_11_134 in memory on ip-10-0-49-157.ec2.internal:44407 (size: 806.4 MB, free: 1192.7 MB) | |
| 17/06/14 06:06:20 INFO TaskSetManager: Starting task 85.0 in stage 5.0 (TID 787, ip-10-0-49-157.ec2.internal, executor 51, partition 85, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:06:20 INFO TaskSetManager: Finished task 134.0 in stage 5.0 (TID 714) in 1329097 ms on ip-10-0-49-157.ec2.internal (executor 51) (34/135) | |
| 17/06/14 06:06:21 INFO BlockManagerInfo: Removed rdd_10_134 on ip-10-0-49-157.ec2.internal:44407 in memory (size: 828.0 MB, free: 2020.7 MB) | |
| 17/06/14 06:06:22 INFO TaskSetManager: Starting task 86.0 in stage 5.0 (TID 788, ip-10-0-51-185.ec2.internal, executor 54, partition 86, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:06:22 INFO TaskSetManager: Finished task 22.1 in stage 5.0 (TID 726) in 1298465 ms on ip-10-0-51-185.ec2.internal (executor 54) (35/135) | |
| 17/06/14 06:06:23 INFO TaskSetManager: Starting task 87.0 in stage 5.0 (TID 789, ip-10-0-49-157.ec2.internal, executor 40, partition 87, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:06:23 INFO TaskSetManager: Finished task 73.0 in stage 5.0 (TID 709) in 1336737 ms on ip-10-0-49-157.ec2.internal (executor 40) (36/135) | |
| 17/06/14 06:06:29 INFO TaskSetManager: Starting task 88.0 in stage 5.0 (TID 790, ip-10-0-51-185.ec2.internal, executor 54, partition 88, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:06:29 INFO TaskSetManager: Finished task 12.1 in stage 5.0 (TID 728) in 1304849 ms on ip-10-0-51-185.ec2.internal (executor 54) (37/135) | |
| 17/06/14 06:06:29 INFO YarnAllocator: Driver requested a total number of 25 executor(s). | |
| 17/06/14 06:06:29 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 25 executors. | |
| 17/06/14 06:06:32 INFO TaskSetManager: Starting task 89.0 in stage 5.0 (TID 791, ip-10-0-50-96.ec2.internal, executor 50, partition 89, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:06:32 INFO TaskSetManager: Finished task 30.0 in stage 5.0 (TID 735) in 1263022 ms on ip-10-0-50-96.ec2.internal (executor 50) (38/135) | |
| 17/06/14 06:06:38 INFO TaskSetManager: Starting task 90.0 in stage 5.0 (TID 792, ip-10-0-49-157.ec2.internal, executor 40, partition 90, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:06:38 INFO TaskSetManager: Finished task 23.0 in stage 5.0 (TID 729) in 1314007 ms on ip-10-0-49-157.ec2.internal (executor 40) (39/135) | |
| 17/06/14 06:06:39 INFO BlockManagerInfo: Removed rdd_10_25 on ip-10-0-51-185.ec2.internal:33788 in memory (size: 809.9 MB, free: 2.8 GB) | |
| 17/06/14 06:06:41 INFO TaskSetManager: Starting task 91.0 in stage 5.0 (TID 793, ip-10-0-50-96.ec2.internal, executor 50, partition 91, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:06:41 INFO TaskSetManager: Finished task 34.0 in stage 5.0 (TID 738) in 1262858 ms on ip-10-0-50-96.ec2.internal (executor 50) (40/135) | |
| 17/06/14 06:06:41 INFO TaskSetManager: Starting task 92.0 in stage 5.0 (TID 794, ip-10-0-49-157.ec2.internal, executor 44, partition 92, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:06:41 INFO TaskSetManager: Finished task 131.0 in stage 5.0 (TID 712) in 1353109 ms on ip-10-0-49-157.ec2.internal (executor 44) (41/135) | |
| 17/06/14 06:06:41 INFO YarnAllocator: Driver requested a total number of 24 executor(s). | |
| 17/06/14 06:06:41 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 24 executors. | |
| 17/06/14 06:06:49 INFO TaskSetManager: Starting task 93.0 in stage 5.0 (TID 795, ip-10-0-50-96.ec2.internal, executor 55, partition 93, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:06:49 INFO TaskSetManager: Finished task 63.1 in stage 5.0 (TID 742) in 1252701 ms on ip-10-0-50-96.ec2.internal (executor 55) (42/135) | |
| 17/06/14 06:06:54 INFO TaskSetManager: Starting task 94.0 in stage 5.0 (TID 796, ip-10-0-51-185.ec2.internal, executor 39, partition 94, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:06:54 INFO TaskSetManager: Finished task 32.0 in stage 5.0 (TID 737) in 1280283 ms on ip-10-0-51-185.ec2.internal (executor 39) (43/135) | |
| 17/06/14 06:06:54 INFO TaskSetManager: Starting task 95.0 in stage 5.0 (TID 797, ip-10-0-51-185.ec2.internal, executor 54, partition 95, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:06:54 INFO TaskSetManager: Finished task 107.1 in stage 3.0 (TID 725) in 1330517 ms on ip-10-0-51-185.ec2.internal (executor 54) (134/135) | |
| 17/06/14 06:06:56 INFO BlockManagerInfo: Added rdd_10_29 in memory on ip-10-0-49-157.ec2.internal:42968 (size: 846.0 MB, free: 1981.2 MB) | |
| 17/06/14 06:06:58 INFO BlockManagerInfo: Added rdd_10_26 in memory on ip-10-0-50-96.ec2.internal:40971 (size: 806.9 MB, free: 2020.2 MB) | |
| 17/06/14 06:07:02 INFO TaskSetManager: Starting task 96.0 in stage 5.0 (TID 798, ip-10-0-49-157.ec2.internal, executor 40, partition 96, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:02 INFO TaskSetManager: Finished task 29.0 in stage 5.0 (TID 734) in 1316057 ms on ip-10-0-49-157.ec2.internal (executor 40) (44/135) | |
| 17/06/14 06:07:02 INFO YarnAllocator: Driver requested a total number of 23 executor(s). | |
| 17/06/14 06:07:02 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 23 executors. | |
| 17/06/14 06:07:04 INFO TaskSetManager: Starting task 97.0 in stage 5.0 (TID 799, ip-10-0-50-96.ec2.internal, executor 55, partition 97, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:04 INFO TaskSetManager: Finished task 26.1 in stage 5.0 (TID 744) in 1268171 ms on ip-10-0-50-96.ec2.internal (executor 55) (45/135) | |
| 17/06/14 06:07:06 INFO TaskSetManager: Starting task 98.0 in stage 5.0 (TID 800, ip-10-0-50-96.ec2.internal, executor 55, partition 98, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:06 INFO TaskSetManager: Finished task 6.1 in stage 5.0 (TID 743) in 1270060 ms on ip-10-0-50-96.ec2.internal (executor 55) (46/135) | |
| 17/06/14 06:07:13 INFO TaskSetManager: Starting task 100.0 in stage 5.0 (TID 801, ip-10-0-49-157.ec2.internal, executor 44, partition 100, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:13 INFO TaskSetManager: Finished task 24.0 in stage 5.0 (TID 730) in 1345184 ms on ip-10-0-49-157.ec2.internal (executor 44) (47/135) | |
| 17/06/14 06:07:20 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-50-96.ec2.internal:40866 (size: 5.5 KB) | |
| 17/06/14 06:07:20 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-50-96.ec2.internal:40866 (size: 5.5 KB) | |
| 17/06/14 06:07:20 INFO BlockManagerInfo: Removed rdd_10_66 on ip-10-0-50-96.ec2.internal:40866 in memory (size: 840.6 MB, free: 2.8 GB) | |
| 17/06/14 06:07:22 INFO BlockManagerInfo: Added rdd_10_14 in memory on ip-10-0-51-185.ec2.internal:45572 (size: 831.8 MB, free: 1995.4 MB) | |
| 17/06/14 06:07:28 INFO TaskSetManager: Starting task 101.0 in stage 5.0 (TID 802, ip-10-0-50-96.ec2.internal, executor 55, partition 101, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:28 INFO TaskSetManager: Finished task 130.1 in stage 3.0 (TID 741) in 1291439 ms on ip-10-0-50-96.ec2.internal (executor 55) (135/135) | |
| 17/06/14 06:07:28 INFO YarnClusterScheduler: Removed TaskSet 3.0, whose tasks have all completed, from pool | |
| 17/06/14 06:07:28 INFO DAGScheduler: ShuffleMapStage 3 (groupBy at ShredJob.scala:376) finished in 3954.973 s | |
| 17/06/14 06:07:28 INFO DAGScheduler: looking for newly runnable stages | |
| 17/06/14 06:07:28 INFO DAGScheduler: running: Set(ShuffleMapStage 5) | |
| 17/06/14 06:07:28 INFO DAGScheduler: waiting: Set(ResultStage 6, ShuffleMapStage 4) | |
| 17/06/14 06:07:28 INFO DAGScheduler: failed: Set() | |
| 17/06/14 06:07:28 INFO DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[16] at map at ShredJob.scala:378), which has no missing parents | |
| 17/06/14 06:07:28 INFO YarnAllocator: Driver requested a total number of 24 executor(s). | |
| 17/06/14 06:07:28 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:07:28 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 24) | |
| 17/06/14 06:07:28 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 06:07:28 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 11.5 KB, free 413.6 MB) | |
| 17/06/14 06:07:28 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 5.7 KB, free 413.6 MB) | |
| 17/06/14 06:07:28 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on 10.0.49.229:33721 (size: 5.7 KB, free: 413.9 MB) | |
| 17/06/14 06:07:28 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:997 | |
| 17/06/14 06:07:28 INFO DAGScheduler: Submitting 135 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[16] at map at ShredJob.scala:378) | |
| 17/06/14 06:07:28 INFO YarnClusterScheduler: Adding task set 4.0 with 135 tasks | |
| 17/06/14 06:07:28 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 803, ip-10-0-51-185.ec2.internal, executor 54, partition 0, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:28 INFO TaskSetManager: Finished task 14.1 in stage 5.0 (TID 727) in 1364486 ms on ip-10-0-51-185.ec2.internal (executor 54) (48/135) | |
| 17/06/14 06:07:28 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on ip-10-0-51-185.ec2.internal:45572 (size: 5.7 KB, free: 1995.3 MB) | |
| 17/06/14 06:07:28 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 3 to 10.0.51.185:35966 | |
| 17/06/14 06:07:28 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 3 is 5184 bytes | |
| 17/06/14 06:07:29 INFO YarnAllocator: Driver requested a total number of 26 executor(s). | |
| 17/06/14 06:07:29 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:07:29 INFO ExecutorAllocationManager: Requesting 2 new executors because tasks are backlogged (new desired total will be 26) | |
| 17/06/14 06:07:29 INFO YarnAllocator: Submitted 2 unlocalized container requests. | |
| 17/06/14 06:07:30 INFO YarnAllocator: Driver requested a total number of 30 executor(s). | |
| 17/06/14 06:07:30 INFO YarnAllocator: Will request 4 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:07:30 INFO ExecutorAllocationManager: Requesting 4 new executors because tasks are backlogged (new desired total will be 30) | |
| 17/06/14 06:07:30 INFO YarnAllocator: Submitted 4 unlocalized container requests. | |
| 17/06/14 06:07:31 INFO YarnAllocator: Driver requested a total number of 38 executor(s). | |
| 17/06/14 06:07:31 INFO ExecutorAllocationManager: Requesting 8 new executors because tasks are backlogged (new desired total will be 38) | |
| 17/06/14 06:07:31 INFO YarnAllocator: Will request 8 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:07:31 INFO YarnAllocator: Submitted 8 unlocalized container requests. | |
| 17/06/14 06:07:32 INFO YarnAllocator: Driver requested a total number of 54 executor(s). | |
| 17/06/14 06:07:32 INFO ExecutorAllocationManager: Requesting 16 new executors because tasks are backlogged (new desired total will be 54) | |
| 17/06/14 06:07:32 INFO YarnAllocator: Will request 16 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:07:32 INFO YarnAllocator: Submitted 16 unlocalized container requests. | |
| 17/06/14 06:07:33 INFO YarnAllocator: Driver requested a total number of 56 executor(s). | |
| 17/06/14 06:07:33 INFO ExecutorAllocationManager: Requesting 2 new executors because tasks are backlogged (new desired total will be 56) | |
| 17/06/14 06:07:33 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:07:33 INFO YarnAllocator: Submitted 2 unlocalized container requests. | |
| 17/06/14 06:07:35 INFO TaskSetManager: Starting task 1.0 in stage 4.0 (TID 804, ip-10-0-51-185.ec2.internal, executor 54, partition 1, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:35 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 803) in 6670 ms on ip-10-0-51-185.ec2.internal (executor 54) (1/135) | |
| 17/06/14 06:07:42 INFO TaskSetManager: Starting task 2.0 in stage 4.0 (TID 805, ip-10-0-51-185.ec2.internal, executor 54, partition 2, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:42 INFO TaskSetManager: Finished task 1.0 in stage 4.0 (TID 804) in 6783 ms on ip-10-0-51-185.ec2.internal (executor 54) (2/135) | |
| 17/06/14 06:07:42 INFO YarnAllocator: Driver requested a total number of 55 executor(s). | |
| 17/06/14 06:07:42 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 55 executors. | |
| 17/06/14 06:07:42 INFO TaskSetManager: Starting task 3.0 in stage 4.0 (TID 806, ip-10-0-50-96.ec2.internal, executor 50, partition 3, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:42 INFO TaskSetManager: Finished task 36.0 in stage 5.0 (TID 740) in 1312120 ms on ip-10-0-50-96.ec2.internal (executor 50) (49/135) | |
| 17/06/14 06:07:42 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on ip-10-0-50-96.ec2.internal:34711 (size: 5.7 KB, free: 2.8 GB) | |
| 17/06/14 06:07:42 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 3 to 10.0.50.96:37112 | |
| 17/06/14 06:07:45 INFO TaskSetManager: Starting task 4.0 in stage 4.0 (TID 807, ip-10-0-50-96.ec2.internal, executor 50, partition 4, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:45 INFO TaskSetManager: Finished task 3.0 in stage 4.0 (TID 806) in 3539 ms on ip-10-0-50-96.ec2.internal (executor 50) (3/135) | |
| 17/06/14 06:07:49 INFO TaskSetManager: Starting task 5.0 in stage 4.0 (TID 808, ip-10-0-50-96.ec2.internal, executor 50, partition 5, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:49 INFO TaskSetManager: Finished task 4.0 in stage 4.0 (TID 807) in 3281 ms on ip-10-0-50-96.ec2.internal (executor 50) (4/135) | |
| 17/06/14 06:07:49 INFO TaskSetManager: Starting task 6.0 in stage 4.0 (TID 809, ip-10-0-51-185.ec2.internal, executor 54, partition 6, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:49 INFO TaskSetManager: Finished task 2.0 in stage 4.0 (TID 805) in 7064 ms on ip-10-0-51-185.ec2.internal (executor 54) (5/135) | |
| 17/06/14 06:07:49 INFO YarnAllocator: Driver requested a total number of 54 executor(s). | |
| 17/06/14 06:07:49 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 54 executors. | |
| 17/06/14 06:07:52 INFO TaskSetManager: Starting task 7.0 in stage 4.0 (TID 810, ip-10-0-50-96.ec2.internal, executor 50, partition 7, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:52 INFO TaskSetManager: Finished task 5.0 in stage 4.0 (TID 808) in 3702 ms on ip-10-0-50-96.ec2.internal (executor 50) (6/135) | |
| 17/06/14 06:07:56 INFO TaskSetManager: Starting task 8.0 in stage 4.0 (TID 811, ip-10-0-50-96.ec2.internal, executor 50, partition 8, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:56 INFO TaskSetManager: Finished task 7.0 in stage 4.0 (TID 810) in 3240 ms on ip-10-0-50-96.ec2.internal (executor 50) (7/135) | |
| 17/06/14 06:07:56 INFO TaskSetManager: Starting task 9.0 in stage 4.0 (TID 812, ip-10-0-51-185.ec2.internal, executor 54, partition 9, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:07:56 INFO TaskSetManager: Finished task 6.0 in stage 4.0 (TID 809) in 7046 ms on ip-10-0-51-185.ec2.internal (executor 54) (8/135) | |
| 17/06/14 06:08:03 INFO TaskSetManager: Starting task 10.0 in stage 4.0 (TID 813, ip-10-0-50-96.ec2.internal, executor 50, partition 10, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:03 INFO TaskSetManager: Finished task 8.0 in stage 4.0 (TID 811) in 7275 ms on ip-10-0-50-96.ec2.internal (executor 50) (9/135) | |
| 17/06/14 06:08:03 INFO YarnAllocator: Driver requested a total number of 53 executor(s). | |
| 17/06/14 06:08:03 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 53 executors. | |
| 17/06/14 06:08:05 INFO TaskSetManager: Starting task 11.0 in stage 4.0 (TID 814, ip-10-0-51-185.ec2.internal, executor 54, partition 11, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:05 INFO TaskSetManager: Finished task 9.0 in stage 4.0 (TID 812) in 8782 ms on ip-10-0-51-185.ec2.internal (executor 54) (10/135) | |
| 17/06/14 06:08:10 INFO TaskSetManager: Starting task 12.0 in stage 4.0 (TID 815, ip-10-0-50-96.ec2.internal, executor 50, partition 12, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:10 INFO TaskSetManager: Finished task 10.0 in stage 4.0 (TID 813) in 7044 ms on ip-10-0-50-96.ec2.internal (executor 50) (11/135) | |
| 17/06/14 06:08:11 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-49-229.ec2.internal:45654 (size: 5.5 KB) | |
| 17/06/14 06:08:11 INFO BlockManagerInfo: Removed rdd_10_0 on ip-10-0-49-229.ec2.internal:45654 in memory (size: 890.4 MB, free: 2.8 GB) | |
| 17/06/14 06:08:12 INFO TaskSetManager: Starting task 13.0 in stage 4.0 (TID 816, ip-10-0-51-185.ec2.internal, executor 54, partition 13, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:12 INFO TaskSetManager: Finished task 11.0 in stage 4.0 (TID 814) in 6957 ms on ip-10-0-51-185.ec2.internal (executor 54) (12/135) | |
| 17/06/14 06:08:13 INFO TaskSetManager: Starting task 14.0 in stage 4.0 (TID 817, ip-10-0-50-96.ec2.internal, executor 50, partition 14, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:13 INFO TaskSetManager: Finished task 12.0 in stage 4.0 (TID 815) in 3334 ms on ip-10-0-50-96.ec2.internal (executor 50) (13/135) | |
| 17/06/14 06:08:13 INFO YarnAllocator: Driver requested a total number of 52 executor(s). | |
| 17/06/14 06:08:13 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 52 executors. | |
| 17/06/14 06:08:15 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-50-96.ec2.internal:40971 (size: 5.5 KB) | |
| 17/06/14 06:08:15 INFO BlockManagerInfo: Added broadcast_3_piece0 on disk on ip-10-0-50-96.ec2.internal:40971 (size: 5.5 KB) | |
| 17/06/14 06:08:15 INFO BlockManagerInfo: Removed rdd_10_26 on ip-10-0-50-96.ec2.internal:40971 in memory (size: 806.9 MB, free: 2.8 GB) | |
| 17/06/14 06:08:18 INFO TaskSetManager: Starting task 15.0 in stage 4.0 (TID 818, ip-10-0-51-185.ec2.internal, executor 54, partition 15, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:18 INFO TaskSetManager: Finished task 13.0 in stage 4.0 (TID 816) in 6728 ms on ip-10-0-51-185.ec2.internal (executor 54) (14/135) | |
| 17/06/14 06:08:21 INFO TaskSetManager: Starting task 16.0 in stage 4.0 (TID 819, ip-10-0-50-96.ec2.internal, executor 50, partition 16, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:21 INFO TaskSetManager: Finished task 14.0 in stage 4.0 (TID 817) in 7849 ms on ip-10-0-50-96.ec2.internal (executor 50) (15/135) | |
| 17/06/14 06:08:26 INFO TaskSetManager: Starting task 17.0 in stage 4.0 (TID 820, ip-10-0-51-185.ec2.internal, executor 54, partition 17, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:26 INFO TaskSetManager: Finished task 15.0 in stage 4.0 (TID 818) in 7554 ms on ip-10-0-51-185.ec2.internal (executor 54) (16/135) | |
| 17/06/14 06:08:27 INFO BlockManagerInfo: Removed rdd_11_10 on ip-10-0-49-157.ec2.internal:45984 in memory (size: 801.2 MB, free: 2.8 GB) | |
| 17/06/14 06:08:29 INFO TaskSetManager: Starting task 18.0 in stage 4.0 (TID 821, ip-10-0-50-96.ec2.internal, executor 50, partition 18, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:29 INFO TaskSetManager: Finished task 16.0 in stage 4.0 (TID 819) in 7489 ms on ip-10-0-50-96.ec2.internal (executor 50) (17/135) | |
| 17/06/14 06:08:29 INFO YarnAllocator: Driver requested a total number of 51 executor(s). | |
| 17/06/14 06:08:29 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 51 executors. | |
| 17/06/14 06:08:32 INFO TaskSetManager: Starting task 19.0 in stage 4.0 (TID 822, ip-10-0-50-96.ec2.internal, executor 50, partition 19, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:32 INFO TaskSetManager: Finished task 18.0 in stage 4.0 (TID 821) in 3010 ms on ip-10-0-50-96.ec2.internal (executor 50) (18/135) | |
| 17/06/14 06:08:33 INFO TaskSetManager: Starting task 20.0 in stage 4.0 (TID 823, ip-10-0-51-185.ec2.internal, executor 54, partition 20, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:33 INFO TaskSetManager: Finished task 17.0 in stage 4.0 (TID 820) in 6697 ms on ip-10-0-51-185.ec2.internal (executor 54) (19/135) | |
| 17/06/14 06:08:39 INFO BlockManagerInfo: Removed rdd_10_18 on ip-10-0-51-185.ec2.internal:46252 in memory (size: 848.9 MB, free: 2.8 GB) | |
| 17/06/14 06:08:40 INFO TaskSetManager: Starting task 21.0 in stage 4.0 (TID 824, ip-10-0-51-185.ec2.internal, executor 54, partition 21, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:40 INFO TaskSetManager: Finished task 20.0 in stage 4.0 (TID 823) in 7533 ms on ip-10-0-51-185.ec2.internal (executor 54) (20/135) | |
| 17/06/14 06:08:40 INFO TaskSetManager: Starting task 22.0 in stage 4.0 (TID 825, ip-10-0-50-96.ec2.internal, executor 50, partition 22, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:40 INFO TaskSetManager: Finished task 19.0 in stage 4.0 (TID 822) in 8591 ms on ip-10-0-50-96.ec2.internal (executor 50) (21/135) | |
| 17/06/14 06:08:40 INFO YarnAllocator: Driver requested a total number of 50 executor(s). | |
| 17/06/14 06:08:40 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 50 executors. | |
| 17/06/14 06:08:47 INFO TaskSetManager: Starting task 23.0 in stage 4.0 (TID 826, ip-10-0-51-185.ec2.internal, executor 54, partition 23, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:47 INFO TaskSetManager: Finished task 21.0 in stage 4.0 (TID 824) in 7135 ms on ip-10-0-51-185.ec2.internal (executor 54) (22/135) | |
| 17/06/14 06:08:47 INFO TaskSetManager: Starting task 24.0 in stage 4.0 (TID 827, ip-10-0-50-96.ec2.internal, executor 50, partition 24, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:47 INFO TaskSetManager: Finished task 22.0 in stage 4.0 (TID 825) in 7297 ms on ip-10-0-50-96.ec2.internal (executor 50) (23/135) | |
| 17/06/14 06:08:48 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-49-157.ec2.internal:42968 (size: 5.5 KB) | |
| 17/06/14 06:08:48 INFO BlockManagerInfo: Removed rdd_10_29 on ip-10-0-49-157.ec2.internal:42968 in memory (size: 846.0 MB, free: 2.8 GB) | |
| 17/06/14 06:08:55 INFO TaskSetManager: Starting task 25.0 in stage 4.0 (TID 828, ip-10-0-50-96.ec2.internal, executor 50, partition 25, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:55 INFO TaskSetManager: Finished task 24.0 in stage 4.0 (TID 827) in 7524 ms on ip-10-0-50-96.ec2.internal (executor 50) (24/135) | |
| 17/06/14 06:08:57 INFO TaskSetManager: Starting task 26.0 in stage 4.0 (TID 829, ip-10-0-51-185.ec2.internal, executor 54, partition 26, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:08:57 INFO TaskSetManager: Finished task 23.0 in stage 4.0 (TID 826) in 9317 ms on ip-10-0-51-185.ec2.internal (executor 54) (25/135) | |
| 17/06/14 06:08:57 INFO YarnAllocator: Driver requested a total number of 49 executor(s). | |
| 17/06/14 06:08:57 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 49 executors. | |
| 17/06/14 06:09:03 INFO TaskSetManager: Starting task 27.0 in stage 4.0 (TID 830, ip-10-0-50-96.ec2.internal, executor 50, partition 27, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:03 INFO TaskSetManager: Finished task 25.0 in stage 4.0 (TID 828) in 7574 ms on ip-10-0-50-96.ec2.internal (executor 50) (26/135) | |
| 17/06/14 06:09:03 INFO BlockManagerInfo: Removed rdd_10_129 on ip-10-0-50-96.ec2.internal:37989 in memory (size: 788.4 MB, free: 2.8 GB) | |
| 17/06/14 06:09:05 INFO TaskSetManager: Starting task 28.0 in stage 4.0 (TID 831, ip-10-0-51-185.ec2.internal, executor 54, partition 28, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:05 INFO TaskSetManager: Finished task 26.0 in stage 4.0 (TID 829) in 8992 ms on ip-10-0-51-185.ec2.internal (executor 54) (27/135) | |
| 17/06/14 06:09:10 INFO TaskSetManager: Starting task 29.0 in stage 4.0 (TID 832, ip-10-0-50-96.ec2.internal, executor 50, partition 29, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:10 INFO TaskSetManager: Finished task 27.0 in stage 4.0 (TID 830) in 7642 ms on ip-10-0-50-96.ec2.internal (executor 50) (28/135) | |
| 17/06/14 06:09:12 INFO TaskSetManager: Starting task 30.0 in stage 4.0 (TID 833, ip-10-0-51-185.ec2.internal, executor 54, partition 30, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:12 INFO TaskSetManager: Finished task 28.0 in stage 4.0 (TID 831) in 6764 ms on ip-10-0-51-185.ec2.internal (executor 54) (29/135) | |
| 17/06/14 06:09:12 INFO YarnAllocator: Driver requested a total number of 48 executor(s). | |
| 17/06/14 06:09:12 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 48 executors. | |
| 17/06/14 06:09:13 INFO TaskSetManager: Starting task 31.0 in stage 4.0 (TID 834, ip-10-0-50-96.ec2.internal, executor 50, partition 31, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:13 INFO TaskSetManager: Finished task 29.0 in stage 4.0 (TID 832) in 3294 ms on ip-10-0-50-96.ec2.internal (executor 50) (30/135) | |
| 17/06/14 06:09:19 INFO TaskSetManager: Starting task 32.0 in stage 4.0 (TID 835, ip-10-0-51-185.ec2.internal, executor 54, partition 32, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:19 INFO TaskSetManager: Finished task 30.0 in stage 4.0 (TID 833) in 6519 ms on ip-10-0-51-185.ec2.internal (executor 54) (31/135) | |
| 17/06/14 06:09:20 INFO TaskSetManager: Starting task 33.0 in stage 4.0 (TID 836, ip-10-0-50-96.ec2.internal, executor 50, partition 33, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:20 INFO TaskSetManager: Finished task 31.0 in stage 4.0 (TID 834) in 6839 ms on ip-10-0-50-96.ec2.internal (executor 50) (32/135) | |
| 17/06/14 06:09:25 INFO TaskSetManager: Starting task 34.0 in stage 4.0 (TID 837, ip-10-0-51-185.ec2.internal, executor 54, partition 34, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:25 INFO TaskSetManager: Finished task 32.0 in stage 4.0 (TID 835) in 6598 ms on ip-10-0-51-185.ec2.internal (executor 54) (33/135) | |
| 17/06/14 06:09:25 INFO YarnAllocator: Driver requested a total number of 47 executor(s). | |
| 17/06/14 06:09:25 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 47 executors. | |
| 17/06/14 06:09:28 INFO TaskSetManager: Starting task 35.0 in stage 4.0 (TID 838, ip-10-0-50-96.ec2.internal, executor 50, partition 35, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:28 INFO TaskSetManager: Finished task 33.0 in stage 4.0 (TID 836) in 7246 ms on ip-10-0-50-96.ec2.internal (executor 50) (34/135) | |
| 17/06/14 06:09:32 INFO TaskSetManager: Starting task 36.0 in stage 4.0 (TID 839, ip-10-0-51-185.ec2.internal, executor 54, partition 36, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:32 INFO TaskSetManager: Finished task 34.0 in stage 4.0 (TID 837) in 6717 ms on ip-10-0-51-185.ec2.internal (executor 54) (35/135) | |
| 17/06/14 06:09:34 INFO TaskSetManager: Starting task 37.0 in stage 4.0 (TID 840, ip-10-0-50-96.ec2.internal, executor 50, partition 37, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:34 INFO TaskSetManager: Finished task 35.0 in stage 4.0 (TID 838) in 6466 ms on ip-10-0-50-96.ec2.internal (executor 50) (36/135) | |
| 17/06/14 06:09:41 INFO TaskSetManager: Starting task 38.0 in stage 4.0 (TID 841, ip-10-0-51-185.ec2.internal, executor 54, partition 38, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:41 INFO TaskSetManager: Finished task 36.0 in stage 4.0 (TID 839) in 8770 ms on ip-10-0-51-185.ec2.internal (executor 54) (37/135) | |
| 17/06/14 06:09:41 INFO YarnAllocator: Driver requested a total number of 46 executor(s). | |
| 17/06/14 06:09:41 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 46 executors. | |
| 17/06/14 06:09:41 INFO TaskSetManager: Starting task 39.0 in stage 4.0 (TID 842, ip-10-0-50-96.ec2.internal, executor 50, partition 39, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:41 INFO TaskSetManager: Finished task 37.0 in stage 4.0 (TID 840) in 7372 ms on ip-10-0-50-96.ec2.internal (executor 50) (38/135) | |
| 17/06/14 06:09:47 INFO TaskSetManager: Starting task 40.0 in stage 4.0 (TID 843, ip-10-0-51-185.ec2.internal, executor 54, partition 40, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:47 INFO TaskSetManager: Finished task 38.0 in stage 4.0 (TID 841) in 6257 ms on ip-10-0-51-185.ec2.internal (executor 54) (39/135) | |
| 17/06/14 06:09:48 INFO TaskSetManager: Starting task 41.0 in stage 4.0 (TID 844, ip-10-0-50-96.ec2.internal, executor 50, partition 41, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:48 INFO TaskSetManager: Finished task 39.0 in stage 4.0 (TID 842) in 6519 ms on ip-10-0-50-96.ec2.internal (executor 50) (40/135) | |
| 17/06/14 06:09:54 INFO TaskSetManager: Starting task 42.0 in stage 4.0 (TID 845, ip-10-0-51-185.ec2.internal, executor 54, partition 42, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:54 INFO TaskSetManager: Finished task 40.0 in stage 4.0 (TID 843) in 6705 ms on ip-10-0-51-185.ec2.internal (executor 54) (41/135) | |
| 17/06/14 06:09:54 INFO YarnAllocator: Driver requested a total number of 45 executor(s). | |
| 17/06/14 06:09:54 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 45 executors. | |
| 17/06/14 06:09:55 INFO TaskSetManager: Starting task 43.0 in stage 4.0 (TID 846, ip-10-0-50-96.ec2.internal, executor 50, partition 43, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:09:55 INFO TaskSetManager: Finished task 41.0 in stage 4.0 (TID 844) in 7124 ms on ip-10-0-50-96.ec2.internal (executor 50) (42/135) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on 10.0.49.229:33721 in memory (size: 5.5 KB, free: 413.9 MB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-51-185.ec2.internal:33788 on disk (size: 5.5 KB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-49-157.ec2.internal:45984 on disk (size: 5.5 KB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-50-96.ec2.internal:37989 on disk (size: 5.5 KB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-49-229.ec2.internal:45654 on disk (size: 5.5 KB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-51-185.ec2.internal:34029 on disk (size: 5.5 KB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-50-96.ec2.internal:40866 on disk (size: 5.5 KB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-49-157.ec2.internal:39668 on disk (size: 5.5 KB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-51-185.ec2.internal:46252 on disk (size: 5.5 KB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-49-157.ec2.internal:42968 on disk (size: 5.5 KB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-49-229.ec2.internal:44395 on disk (size: 5.5 KB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-50-96.ec2.internal:34711 on disk (size: 5.5 KB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-49-157.ec2.internal:44407 on disk (size: 5.5 KB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-49-229.ec2.internal:44599 in memory (size: 5.5 KB, free: 2.8 GB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-50-96.ec2.internal:40971 on disk (size: 5.5 KB) | |
| 17/06/14 06:09:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on ip-10-0-51-185.ec2.internal:45572 in memory (size: 5.5 KB, free: 1995.4 MB) | |
| 17/06/14 06:10:00 INFO TaskSetManager: Starting task 44.0 in stage 4.0 (TID 847, ip-10-0-51-185.ec2.internal, executor 54, partition 44, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:00 INFO TaskSetManager: Finished task 42.0 in stage 4.0 (TID 845) in 6490 ms on ip-10-0-51-185.ec2.internal (executor 54) (43/135) | |
| 17/06/14 06:10:02 INFO TaskSetManager: Starting task 45.0 in stage 4.0 (TID 848, ip-10-0-50-96.ec2.internal, executor 50, partition 45, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:02 INFO TaskSetManager: Finished task 43.0 in stage 4.0 (TID 846) in 7403 ms on ip-10-0-50-96.ec2.internal (executor 50) (44/135) | |
| 17/06/14 06:10:09 INFO TaskSetManager: Starting task 46.0 in stage 4.0 (TID 849, ip-10-0-51-185.ec2.internal, executor 54, partition 46, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:09 INFO TaskSetManager: Finished task 44.0 in stage 4.0 (TID 847) in 8547 ms on ip-10-0-51-185.ec2.internal (executor 54) (45/135) | |
| 17/06/14 06:10:09 INFO YarnAllocator: Driver requested a total number of 44 executor(s). | |
| 17/06/14 06:10:09 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 44 executors. | |
| 17/06/14 06:10:10 INFO TaskSetManager: Starting task 47.0 in stage 4.0 (TID 850, ip-10-0-50-96.ec2.internal, executor 50, partition 47, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:10 INFO TaskSetManager: Finished task 45.0 in stage 4.0 (TID 848) in 7127 ms on ip-10-0-50-96.ec2.internal (executor 50) (46/135) | |
| 17/06/14 06:10:16 INFO TaskSetManager: Starting task 48.0 in stage 4.0 (TID 851, ip-10-0-51-185.ec2.internal, executor 54, partition 48, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:16 INFO TaskSetManager: Finished task 46.0 in stage 4.0 (TID 849) in 6937 ms on ip-10-0-51-185.ec2.internal (executor 54) (47/135) | |
| 17/06/14 06:10:16 INFO TaskSetManager: Starting task 49.0 in stage 4.0 (TID 852, ip-10-0-50-96.ec2.internal, executor 50, partition 49, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:16 INFO TaskSetManager: Finished task 47.0 in stage 4.0 (TID 850) in 6861 ms on ip-10-0-50-96.ec2.internal (executor 50) (48/135) | |
| 17/06/14 06:10:23 INFO TaskSetManager: Starting task 50.0 in stage 4.0 (TID 853, ip-10-0-50-96.ec2.internal, executor 50, partition 50, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:23 INFO TaskSetManager: Finished task 49.0 in stage 4.0 (TID 852) in 6722 ms on ip-10-0-50-96.ec2.internal (executor 50) (49/135) | |
| 17/06/14 06:10:23 INFO YarnAllocator: Driver requested a total number of 43 executor(s). | |
| 17/06/14 06:10:23 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 43 executors. | |
| 17/06/14 06:10:25 INFO TaskSetManager: Starting task 51.0 in stage 4.0 (TID 854, ip-10-0-51-185.ec2.internal, executor 54, partition 51, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:25 INFO TaskSetManager: Finished task 48.0 in stage 4.0 (TID 851) in 9006 ms on ip-10-0-51-185.ec2.internal (executor 54) (50/135) | |
| 17/06/14 06:10:30 INFO TaskSetManager: Starting task 52.0 in stage 4.0 (TID 855, ip-10-0-50-96.ec2.internal, executor 50, partition 52, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:30 INFO TaskSetManager: Finished task 50.0 in stage 4.0 (TID 853) in 7028 ms on ip-10-0-50-96.ec2.internal (executor 50) (51/135) | |
| 17/06/14 06:10:33 INFO TaskSetManager: Starting task 53.0 in stage 4.0 (TID 856, ip-10-0-51-185.ec2.internal, executor 54, partition 53, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:33 INFO TaskSetManager: Finished task 51.0 in stage 4.0 (TID 854) in 8614 ms on ip-10-0-51-185.ec2.internal (executor 54) (52/135) | |
| 17/06/14 06:10:38 INFO TaskSetManager: Starting task 54.0 in stage 4.0 (TID 857, ip-10-0-50-96.ec2.internal, executor 50, partition 54, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:38 INFO TaskSetManager: Finished task 52.0 in stage 4.0 (TID 855) in 7366 ms on ip-10-0-50-96.ec2.internal (executor 50) (53/135) | |
| 17/06/14 06:10:38 INFO YarnAllocator: Driver requested a total number of 42 executor(s). | |
| 17/06/14 06:10:38 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 42 executors. | |
| 17/06/14 06:10:38 INFO BlockManagerInfo: Removed rdd_11_134 on ip-10-0-49-157.ec2.internal:44407 in memory (size: 806.4 MB, free: 2.8 GB) | |
| 17/06/14 06:10:41 INFO TaskSetManager: Starting task 55.0 in stage 4.0 (TID 858, ip-10-0-50-96.ec2.internal, executor 50, partition 55, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:41 INFO TaskSetManager: Finished task 54.0 in stage 4.0 (TID 857) in 3094 ms on ip-10-0-50-96.ec2.internal (executor 50) (54/135) | |
| 17/06/14 06:10:42 INFO TaskSetManager: Starting task 56.0 in stage 4.0 (TID 859, ip-10-0-51-185.ec2.internal, executor 54, partition 56, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:42 INFO TaskSetManager: Finished task 53.0 in stage 4.0 (TID 856) in 8703 ms on ip-10-0-51-185.ec2.internal (executor 54) (55/135) | |
| 17/06/14 06:10:47 INFO TaskSetManager: Starting task 57.0 in stage 4.0 (TID 860, ip-10-0-50-96.ec2.internal, executor 50, partition 57, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:47 INFO TaskSetManager: Finished task 55.0 in stage 4.0 (TID 858) in 6485 ms on ip-10-0-50-96.ec2.internal (executor 50) (56/135) | |
| 17/06/14 06:10:51 INFO TaskSetManager: Starting task 58.0 in stage 4.0 (TID 861, ip-10-0-51-185.ec2.internal, executor 54, partition 58, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:51 INFO TaskSetManager: Finished task 56.0 in stage 4.0 (TID 859) in 8840 ms on ip-10-0-51-185.ec2.internal (executor 54) (57/135) | |
| 17/06/14 06:10:51 INFO YarnAllocator: Driver requested a total number of 41 executor(s). | |
| 17/06/14 06:10:51 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 41 executors. | |
| 17/06/14 06:10:54 INFO TaskSetManager: Starting task 59.0 in stage 4.0 (TID 862, ip-10-0-50-96.ec2.internal, executor 50, partition 59, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:10:54 INFO TaskSetManager: Finished task 57.0 in stage 4.0 (TID 860) in 7172 ms on ip-10-0-50-96.ec2.internal (executor 50) (58/135) | |
| 17/06/14 06:11:00 INFO TaskSetManager: Starting task 60.0 in stage 4.0 (TID 863, ip-10-0-51-185.ec2.internal, executor 54, partition 60, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:00 INFO TaskSetManager: Finished task 58.0 in stage 4.0 (TID 861) in 8793 ms on ip-10-0-51-185.ec2.internal (executor 54) (59/135) | |
| 17/06/14 06:11:01 INFO TaskSetManager: Starting task 61.0 in stage 4.0 (TID 864, ip-10-0-50-96.ec2.internal, executor 50, partition 61, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:01 INFO TaskSetManager: Finished task 59.0 in stage 4.0 (TID 862) in 6660 ms on ip-10-0-50-96.ec2.internal (executor 50) (60/135) | |
| 17/06/14 06:11:06 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-51-185.ec2.internal:45572 (size: 5.5 KB) | |
| 17/06/14 06:11:06 INFO BlockManagerInfo: Removed rdd_10_14 on ip-10-0-51-185.ec2.internal:45572 in memory (size: 831.8 MB, free: 2.8 GB) | |
| 17/06/14 06:11:08 INFO TaskSetManager: Starting task 62.0 in stage 4.0 (TID 865, ip-10-0-51-185.ec2.internal, executor 54, partition 62, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:08 INFO TaskSetManager: Finished task 60.0 in stage 4.0 (TID 863) in 7937 ms on ip-10-0-51-185.ec2.internal (executor 54) (61/135) | |
| 17/06/14 06:11:08 INFO YarnAllocator: Driver requested a total number of 40 executor(s). | |
| 17/06/14 06:11:08 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 40 executors. | |
| 17/06/14 06:11:08 INFO TaskSetManager: Starting task 63.0 in stage 4.0 (TID 866, ip-10-0-50-96.ec2.internal, executor 50, partition 63, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:08 INFO TaskSetManager: Finished task 61.0 in stage 4.0 (TID 864) in 6830 ms on ip-10-0-50-96.ec2.internal (executor 50) (62/135) | |
| 17/06/14 06:11:15 INFO TaskSetManager: Starting task 64.0 in stage 4.0 (TID 867, ip-10-0-51-185.ec2.internal, executor 54, partition 64, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:15 INFO TaskSetManager: Finished task 62.0 in stage 4.0 (TID 865) in 6876 ms on ip-10-0-51-185.ec2.internal (executor 54) (63/135) | |
| 17/06/14 06:11:15 INFO TaskSetManager: Starting task 65.0 in stage 4.0 (TID 868, ip-10-0-50-96.ec2.internal, executor 50, partition 65, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:15 INFO TaskSetManager: Finished task 63.0 in stage 4.0 (TID 866) in 6854 ms on ip-10-0-50-96.ec2.internal (executor 50) (64/135) | |
| 17/06/14 06:11:21 INFO TaskSetManager: Starting task 66.0 in stage 4.0 (TID 869, ip-10-0-50-96.ec2.internal, executor 50, partition 66, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:21 INFO TaskSetManager: Finished task 65.0 in stage 4.0 (TID 868) in 6662 ms on ip-10-0-50-96.ec2.internal (executor 50) (65/135) | |
| 17/06/14 06:11:21 INFO YarnAllocator: Driver requested a total number of 39 executor(s). | |
| 17/06/14 06:11:21 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 39 executors. | |
| 17/06/14 06:11:22 INFO TaskSetManager: Starting task 67.0 in stage 4.0 (TID 870, ip-10-0-51-185.ec2.internal, executor 54, partition 67, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:22 INFO TaskSetManager: Finished task 64.0 in stage 4.0 (TID 867) in 6977 ms on ip-10-0-51-185.ec2.internal (executor 54) (66/135) | |
| 17/06/14 06:11:28 INFO TaskSetManager: Starting task 68.0 in stage 4.0 (TID 871, ip-10-0-50-96.ec2.internal, executor 50, partition 68, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:28 INFO TaskSetManager: Finished task 66.0 in stage 4.0 (TID 869) in 6692 ms on ip-10-0-50-96.ec2.internal (executor 50) (67/135) | |
| 17/06/14 06:11:29 INFO TaskSetManager: Starting task 69.0 in stage 4.0 (TID 872, ip-10-0-51-185.ec2.internal, executor 54, partition 69, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:29 INFO TaskSetManager: Finished task 67.0 in stage 4.0 (TID 870) in 7130 ms on ip-10-0-51-185.ec2.internal (executor 54) (68/135) | |
| 17/06/14 06:11:34 INFO TaskSetManager: Starting task 70.0 in stage 4.0 (TID 873, ip-10-0-50-96.ec2.internal, executor 50, partition 70, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:34 INFO TaskSetManager: Finished task 68.0 in stage 4.0 (TID 871) in 6496 ms on ip-10-0-50-96.ec2.internal (executor 50) (69/135) | |
| 17/06/14 06:11:35 INFO YarnAllocator: Driver requested a total number of 38 executor(s). | |
| 17/06/14 06:11:35 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 38 executors. | |
| 17/06/14 06:11:35 INFO TaskSetManager: Starting task 71.0 in stage 4.0 (TID 874, ip-10-0-51-185.ec2.internal, executor 54, partition 71, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:35 INFO TaskSetManager: Finished task 69.0 in stage 4.0 (TID 872) in 6657 ms on ip-10-0-51-185.ec2.internal (executor 54) (70/135) | |
| 17/06/14 06:11:41 INFO TaskSetManager: Starting task 72.0 in stage 4.0 (TID 875, ip-10-0-50-96.ec2.internal, executor 50, partition 72, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:41 INFO TaskSetManager: Finished task 70.0 in stage 4.0 (TID 873) in 6609 ms on ip-10-0-50-96.ec2.internal (executor 50) (71/135) | |
| 17/06/14 06:11:43 INFO TaskSetManager: Starting task 73.0 in stage 4.0 (TID 876, ip-10-0-51-185.ec2.internal, executor 54, partition 73, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:43 INFO TaskSetManager: Finished task 71.0 in stage 4.0 (TID 874) in 7551 ms on ip-10-0-51-185.ec2.internal (executor 54) (72/135) | |
| 17/06/14 06:11:48 INFO TaskSetManager: Starting task 74.0 in stage 4.0 (TID 877, ip-10-0-50-96.ec2.internal, executor 50, partition 74, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:48 INFO TaskSetManager: Finished task 72.0 in stage 4.0 (TID 875) in 7099 ms on ip-10-0-50-96.ec2.internal (executor 50) (73/135) | |
| 17/06/14 06:11:48 INFO YarnAllocator: Driver requested a total number of 37 executor(s). | |
| 17/06/14 06:11:48 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 37 executors. | |
| 17/06/14 06:11:50 INFO TaskSetManager: Starting task 75.0 in stage 4.0 (TID 878, ip-10-0-51-185.ec2.internal, executor 54, partition 75, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:50 INFO TaskSetManager: Finished task 73.0 in stage 4.0 (TID 876) in 7431 ms on ip-10-0-51-185.ec2.internal (executor 54) (74/135) | |
| 17/06/14 06:11:55 INFO TaskSetManager: Starting task 76.0 in stage 4.0 (TID 879, ip-10-0-50-96.ec2.internal, executor 50, partition 76, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:55 INFO TaskSetManager: Finished task 74.0 in stage 4.0 (TID 877) in 6622 ms on ip-10-0-50-96.ec2.internal (executor 50) (75/135) | |
| 17/06/14 06:11:57 INFO TaskSetManager: Starting task 77.0 in stage 4.0 (TID 880, ip-10-0-51-185.ec2.internal, executor 54, partition 77, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:11:57 INFO TaskSetManager: Finished task 75.0 in stage 4.0 (TID 878) in 6797 ms on ip-10-0-51-185.ec2.internal (executor 54) (76/135) | |
| 17/06/14 06:12:02 INFO TaskSetManager: Starting task 78.0 in stage 4.0 (TID 881, ip-10-0-50-96.ec2.internal, executor 50, partition 78, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:02 INFO TaskSetManager: Finished task 76.0 in stage 4.0 (TID 879) in 6694 ms on ip-10-0-50-96.ec2.internal (executor 50) (77/135) | |
| 17/06/14 06:12:02 INFO YarnAllocator: Driver requested a total number of 36 executor(s). | |
| 17/06/14 06:12:02 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 36 executors. | |
| 17/06/14 06:12:04 INFO TaskSetManager: Starting task 79.0 in stage 4.0 (TID 882, ip-10-0-51-185.ec2.internal, executor 54, partition 79, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:04 INFO TaskSetManager: Finished task 77.0 in stage 4.0 (TID 880) in 7001 ms on ip-10-0-51-185.ec2.internal (executor 54) (78/135) | |
| 17/06/14 06:12:08 INFO TaskSetManager: Starting task 80.0 in stage 4.0 (TID 883, ip-10-0-50-96.ec2.internal, executor 50, partition 80, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:08 INFO TaskSetManager: Finished task 78.0 in stage 4.0 (TID 881) in 6784 ms on ip-10-0-50-96.ec2.internal (executor 50) (79/135) | |
| 17/06/14 06:12:11 INFO TaskSetManager: Starting task 81.0 in stage 4.0 (TID 884, ip-10-0-51-185.ec2.internal, executor 54, partition 81, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:11 INFO TaskSetManager: Finished task 79.0 in stage 4.0 (TID 882) in 6737 ms on ip-10-0-51-185.ec2.internal (executor 54) (80/135) | |
| 17/06/14 06:12:15 INFO TaskSetManager: Starting task 82.0 in stage 4.0 (TID 885, ip-10-0-50-96.ec2.internal, executor 50, partition 82, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:15 INFO TaskSetManager: Finished task 80.0 in stage 4.0 (TID 883) in 6246 ms on ip-10-0-50-96.ec2.internal (executor 50) (81/135) | |
| 17/06/14 06:12:15 INFO YarnAllocator: Driver requested a total number of 35 executor(s). | |
| 17/06/14 06:12:15 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 35 executors. | |
| 17/06/14 06:12:18 INFO TaskSetManager: Starting task 83.0 in stage 4.0 (TID 886, ip-10-0-51-185.ec2.internal, executor 54, partition 83, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:18 INFO TaskSetManager: Finished task 81.0 in stage 4.0 (TID 884) in 7057 ms on ip-10-0-51-185.ec2.internal (executor 54) (82/135) | |
| 17/06/14 06:12:21 INFO TaskSetManager: Starting task 84.0 in stage 4.0 (TID 887, ip-10-0-50-96.ec2.internal, executor 50, partition 84, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:21 INFO TaskSetManager: Finished task 82.0 in stage 4.0 (TID 885) in 6671 ms on ip-10-0-50-96.ec2.internal (executor 50) (83/135) | |
| 17/06/14 06:12:25 INFO TaskSetManager: Starting task 85.0 in stage 4.0 (TID 888, ip-10-0-51-185.ec2.internal, executor 54, partition 85, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:25 INFO TaskSetManager: Finished task 83.0 in stage 4.0 (TID 886) in 6739 ms on ip-10-0-51-185.ec2.internal (executor 54) (84/135) | |
| 17/06/14 06:12:28 INFO TaskSetManager: Starting task 86.0 in stage 4.0 (TID 889, ip-10-0-50-96.ec2.internal, executor 50, partition 86, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:28 INFO TaskSetManager: Finished task 84.0 in stage 4.0 (TID 887) in 6792 ms on ip-10-0-50-96.ec2.internal (executor 50) (85/135) | |
| 17/06/14 06:12:28 INFO YarnAllocator: Driver requested a total number of 34 executor(s). | |
| 17/06/14 06:12:28 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 34 executors. | |
| 17/06/14 06:12:31 INFO TaskSetManager: Starting task 87.0 in stage 4.0 (TID 890, ip-10-0-51-185.ec2.internal, executor 54, partition 87, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:31 INFO TaskSetManager: Finished task 85.0 in stage 4.0 (TID 888) in 6683 ms on ip-10-0-51-185.ec2.internal (executor 54) (86/135) | |
| 17/06/14 06:12:34 INFO TaskSetManager: Starting task 88.0 in stage 4.0 (TID 891, ip-10-0-50-96.ec2.internal, executor 50, partition 88, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:34 INFO TaskSetManager: Finished task 86.0 in stage 4.0 (TID 889) in 6187 ms on ip-10-0-50-96.ec2.internal (executor 50) (87/135) | |
| 17/06/14 06:12:38 INFO TaskSetManager: Starting task 89.0 in stage 4.0 (TID 892, ip-10-0-51-185.ec2.internal, executor 54, partition 89, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:38 INFO TaskSetManager: Finished task 87.0 in stage 4.0 (TID 890) in 6882 ms on ip-10-0-51-185.ec2.internal (executor 54) (88/135) | |
| 17/06/14 06:12:41 INFO TaskSetManager: Starting task 90.0 in stage 4.0 (TID 893, ip-10-0-50-96.ec2.internal, executor 50, partition 90, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:41 INFO TaskSetManager: Finished task 88.0 in stage 4.0 (TID 891) in 6609 ms on ip-10-0-50-96.ec2.internal (executor 50) (89/135) | |
| 17/06/14 06:12:41 INFO YarnAllocator: Driver requested a total number of 33 executor(s). | |
| 17/06/14 06:12:41 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 33 executors. | |
| 17/06/14 06:12:45 INFO TaskSetManager: Starting task 91.0 in stage 4.0 (TID 894, ip-10-0-51-185.ec2.internal, executor 54, partition 91, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:45 INFO TaskSetManager: Finished task 89.0 in stage 4.0 (TID 892) in 6977 ms on ip-10-0-51-185.ec2.internal (executor 54) (90/135) | |
| 17/06/14 06:12:47 INFO TaskSetManager: Starting task 92.0 in stage 4.0 (TID 895, ip-10-0-50-96.ec2.internal, executor 50, partition 92, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:47 INFO TaskSetManager: Finished task 90.0 in stage 4.0 (TID 893) in 6531 ms on ip-10-0-50-96.ec2.internal (executor 50) (91/135) | |
| 17/06/14 06:12:52 INFO TaskSetManager: Starting task 93.0 in stage 4.0 (TID 896, ip-10-0-51-185.ec2.internal, executor 54, partition 93, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:52 INFO TaskSetManager: Finished task 91.0 in stage 4.0 (TID 894) in 7086 ms on ip-10-0-51-185.ec2.internal (executor 54) (92/135) | |
| 17/06/14 06:12:53 INFO TaskSetManager: Starting task 94.0 in stage 4.0 (TID 897, ip-10-0-50-96.ec2.internal, executor 50, partition 94, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:53 INFO TaskSetManager: Finished task 92.0 in stage 4.0 (TID 895) in 6107 ms on ip-10-0-50-96.ec2.internal (executor 50) (93/135) | |
| 17/06/14 06:12:54 INFO YarnAllocator: Driver requested a total number of 32 executor(s). | |
| 17/06/14 06:12:54 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 32 executors. | |
| 17/06/14 06:12:59 INFO TaskSetManager: Starting task 95.0 in stage 4.0 (TID 898, ip-10-0-51-185.ec2.internal, executor 54, partition 95, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:12:59 INFO TaskSetManager: Finished task 93.0 in stage 4.0 (TID 896) in 6778 ms on ip-10-0-51-185.ec2.internal (executor 54) (94/135) | |
| 17/06/14 06:13:00 INFO TaskSetManager: Starting task 96.0 in stage 4.0 (TID 899, ip-10-0-50-96.ec2.internal, executor 50, partition 96, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:00 INFO TaskSetManager: Finished task 94.0 in stage 4.0 (TID 897) in 6937 ms on ip-10-0-50-96.ec2.internal (executor 50) (95/135) | |
| 17/06/14 06:13:06 INFO TaskSetManager: Starting task 97.0 in stage 4.0 (TID 900, ip-10-0-51-185.ec2.internal, executor 54, partition 97, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:06 INFO TaskSetManager: Finished task 95.0 in stage 4.0 (TID 898) in 7191 ms on ip-10-0-51-185.ec2.internal (executor 54) (96/135) | |
| 17/06/14 06:13:08 INFO TaskSetManager: Starting task 98.0 in stage 4.0 (TID 901, ip-10-0-50-96.ec2.internal, executor 50, partition 98, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:08 INFO TaskSetManager: Finished task 96.0 in stage 4.0 (TID 899) in 7520 ms on ip-10-0-50-96.ec2.internal (executor 50) (97/135) | |
| 17/06/14 06:13:08 INFO YarnAllocator: Driver requested a total number of 31 executor(s). | |
| 17/06/14 06:13:08 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 31 executors. | |
| 17/06/14 06:13:10 INFO TaskSetManager: Starting task 99.0 in stage 4.0 (TID 902, ip-10-0-51-185.ec2.internal, executor 54, partition 99, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:10 INFO TaskSetManager: Finished task 97.0 in stage 4.0 (TID 900) in 3375 ms on ip-10-0-51-185.ec2.internal (executor 54) (98/135) | |
| 17/06/14 06:13:14 INFO TaskSetManager: Starting task 100.0 in stage 4.0 (TID 903, ip-10-0-50-96.ec2.internal, executor 50, partition 100, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:14 INFO TaskSetManager: Finished task 98.0 in stage 4.0 (TID 901) in 6468 ms on ip-10-0-50-96.ec2.internal (executor 50) (99/135) | |
| 17/06/14 06:13:16 INFO TaskSetManager: Starting task 101.0 in stage 4.0 (TID 904, ip-10-0-51-185.ec2.internal, executor 54, partition 101, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:16 INFO TaskSetManager: Finished task 99.0 in stage 4.0 (TID 902) in 6754 ms on ip-10-0-51-185.ec2.internal (executor 54) (100/135) | |
| 17/06/14 06:13:21 INFO TaskSetManager: Starting task 102.0 in stage 4.0 (TID 905, ip-10-0-50-96.ec2.internal, executor 50, partition 102, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:21 INFO TaskSetManager: Finished task 100.0 in stage 4.0 (TID 903) in 6293 ms on ip-10-0-50-96.ec2.internal (executor 50) (101/135) | |
| 17/06/14 06:13:21 INFO YarnAllocator: Driver requested a total number of 30 executor(s). | |
| 17/06/14 06:13:21 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 30 executors. | |
| 17/06/14 06:13:23 INFO TaskSetManager: Starting task 103.0 in stage 4.0 (TID 906, ip-10-0-51-185.ec2.internal, executor 54, partition 103, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:23 INFO TaskSetManager: Finished task 101.0 in stage 4.0 (TID 904) in 6812 ms on ip-10-0-51-185.ec2.internal (executor 54) (102/135) | |
| 17/06/14 06:13:28 INFO TaskSetManager: Starting task 104.0 in stage 4.0 (TID 907, ip-10-0-50-96.ec2.internal, executor 50, partition 104, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:28 INFO TaskSetManager: Finished task 102.0 in stage 4.0 (TID 905) in 7085 ms on ip-10-0-50-96.ec2.internal (executor 50) (103/135) | |
| 17/06/14 06:13:30 INFO TaskSetManager: Starting task 105.0 in stage 4.0 (TID 908, ip-10-0-51-185.ec2.internal, executor 54, partition 105, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:30 INFO TaskSetManager: Finished task 103.0 in stage 4.0 (TID 906) in 6863 ms on ip-10-0-51-185.ec2.internal (executor 54) (104/135) | |
| 17/06/14 06:13:36 INFO TaskSetManager: Starting task 106.0 in stage 4.0 (TID 909, ip-10-0-50-96.ec2.internal, executor 50, partition 106, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:36 INFO TaskSetManager: Finished task 104.0 in stage 4.0 (TID 907) in 8180 ms on ip-10-0-50-96.ec2.internal (executor 50) (105/135) | |
| 17/06/14 06:13:36 INFO YarnAllocator: Driver requested a total number of 29 executor(s). | |
| 17/06/14 06:13:36 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 29 executors. | |
| 17/06/14 06:13:37 INFO TaskSetManager: Starting task 107.0 in stage 4.0 (TID 910, ip-10-0-51-185.ec2.internal, executor 54, partition 107, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:37 INFO TaskSetManager: Finished task 105.0 in stage 4.0 (TID 908) in 6500 ms on ip-10-0-51-185.ec2.internal (executor 54) (106/135) | |
| 17/06/14 06:13:42 INFO TaskSetManager: Starting task 108.0 in stage 4.0 (TID 911, ip-10-0-50-96.ec2.internal, executor 50, partition 108, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:42 INFO TaskSetManager: Finished task 106.0 in stage 4.0 (TID 909) in 6453 ms on ip-10-0-50-96.ec2.internal (executor 50) (107/135) | |
| 17/06/14 06:13:43 INFO TaskSetManager: Starting task 109.0 in stage 4.0 (TID 912, ip-10-0-51-185.ec2.internal, executor 54, partition 109, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:43 INFO TaskSetManager: Finished task 107.0 in stage 4.0 (TID 910) in 6758 ms on ip-10-0-51-185.ec2.internal (executor 54) (108/135) | |
| 17/06/14 06:13:49 INFO TaskSetManager: Starting task 110.0 in stage 4.0 (TID 913, ip-10-0-50-96.ec2.internal, executor 50, partition 110, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:49 INFO TaskSetManager: Finished task 108.0 in stage 4.0 (TID 911) in 6473 ms on ip-10-0-50-96.ec2.internal (executor 50) (109/135) | |
| 17/06/14 06:13:49 INFO YarnAllocator: Driver requested a total number of 28 executor(s). | |
| 17/06/14 06:13:49 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 28 executors. | |
| 17/06/14 06:13:50 INFO TaskSetManager: Starting task 111.0 in stage 4.0 (TID 914, ip-10-0-51-185.ec2.internal, executor 54, partition 111, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:50 INFO TaskSetManager: Finished task 109.0 in stage 4.0 (TID 912) in 6315 ms on ip-10-0-51-185.ec2.internal (executor 54) (110/135) | |
| 17/06/14 06:13:57 INFO TaskSetManager: Starting task 112.0 in stage 4.0 (TID 915, ip-10-0-51-185.ec2.internal, executor 54, partition 112, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:57 INFO TaskSetManager: Finished task 111.0 in stage 4.0 (TID 914) in 7010 ms on ip-10-0-51-185.ec2.internal (executor 54) (111/135) | |
| 17/06/14 06:13:57 INFO TaskSetManager: Starting task 113.0 in stage 4.0 (TID 916, ip-10-0-50-96.ec2.internal, executor 50, partition 113, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:13:57 INFO TaskSetManager: Finished task 110.0 in stage 4.0 (TID 913) in 8333 ms on ip-10-0-50-96.ec2.internal (executor 50) (112/135) | |
| 17/06/14 06:14:03 INFO TaskSetManager: Starting task 114.0 in stage 4.0 (TID 917, ip-10-0-50-96.ec2.internal, executor 50, partition 114, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:03 INFO TaskSetManager: Finished task 113.0 in stage 4.0 (TID 916) in 6018 ms on ip-10-0-50-96.ec2.internal (executor 50) (113/135) | |
| 17/06/14 06:14:03 INFO YarnAllocator: Driver requested a total number of 27 executor(s). | |
| 17/06/14 06:14:03 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 27 executors. | |
| 17/06/14 06:14:04 INFO TaskSetManager: Starting task 115.0 in stage 4.0 (TID 918, ip-10-0-51-185.ec2.internal, executor 54, partition 115, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:04 INFO TaskSetManager: Finished task 112.0 in stage 4.0 (TID 915) in 6963 ms on ip-10-0-51-185.ec2.internal (executor 54) (114/135) | |
| 17/06/14 06:14:09 INFO TaskSetManager: Starting task 116.0 in stage 4.0 (TID 919, ip-10-0-50-96.ec2.internal, executor 50, partition 116, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:09 INFO TaskSetManager: Finished task 114.0 in stage 4.0 (TID 917) in 6277 ms on ip-10-0-50-96.ec2.internal (executor 50) (115/135) | |
| 17/06/14 06:14:11 INFO TaskSetManager: Starting task 117.0 in stage 4.0 (TID 920, ip-10-0-51-185.ec2.internal, executor 54, partition 117, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:11 INFO TaskSetManager: Finished task 115.0 in stage 4.0 (TID 918) in 7115 ms on ip-10-0-51-185.ec2.internal (executor 54) (116/135) | |
| 17/06/14 06:14:16 INFO TaskSetManager: Starting task 118.0 in stage 4.0 (TID 921, ip-10-0-50-96.ec2.internal, executor 50, partition 118, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:16 INFO TaskSetManager: Finished task 116.0 in stage 4.0 (TID 919) in 6517 ms on ip-10-0-50-96.ec2.internal (executor 50) (117/135) | |
| 17/06/14 06:14:16 INFO YarnAllocator: Driver requested a total number of 26 executor(s). | |
| 17/06/14 06:14:16 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 26 executors. | |
| 17/06/14 06:14:18 INFO TaskSetManager: Starting task 119.0 in stage 4.0 (TID 922, ip-10-0-51-185.ec2.internal, executor 54, partition 119, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:18 INFO TaskSetManager: Finished task 117.0 in stage 4.0 (TID 920) in 6937 ms on ip-10-0-51-185.ec2.internal (executor 54) (118/135) | |
| 17/06/14 06:14:22 INFO TaskSetManager: Starting task 120.0 in stage 4.0 (TID 923, ip-10-0-50-96.ec2.internal, executor 50, partition 120, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:22 INFO TaskSetManager: Finished task 118.0 in stage 4.0 (TID 921) in 6263 ms on ip-10-0-50-96.ec2.internal (executor 50) (119/135) | |
| 17/06/14 06:14:24 INFO TaskSetManager: Starting task 121.0 in stage 4.0 (TID 924, ip-10-0-51-185.ec2.internal, executor 54, partition 121, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:24 INFO TaskSetManager: Finished task 119.0 in stage 4.0 (TID 922) in 6844 ms on ip-10-0-51-185.ec2.internal (executor 54) (120/135) | |
| 17/06/14 06:14:31 INFO TaskSetManager: Starting task 122.0 in stage 4.0 (TID 925, ip-10-0-50-96.ec2.internal, executor 50, partition 122, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:31 INFO TaskSetManager: Finished task 120.0 in stage 4.0 (TID 923) in 8379 ms on ip-10-0-50-96.ec2.internal (executor 50) (121/135) | |
| 17/06/14 06:14:31 INFO YarnAllocator: Driver requested a total number of 25 executor(s). | |
| 17/06/14 06:14:31 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 25 executors. | |
| 17/06/14 06:14:31 INFO TaskSetManager: Starting task 123.0 in stage 4.0 (TID 926, ip-10-0-51-185.ec2.internal, executor 54, partition 123, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:31 INFO TaskSetManager: Finished task 121.0 in stage 4.0 (TID 924) in 6738 ms on ip-10-0-51-185.ec2.internal (executor 54) (122/135) | |
| 17/06/14 06:14:38 INFO TaskSetManager: Starting task 124.0 in stage 4.0 (TID 927, ip-10-0-51-185.ec2.internal, executor 54, partition 124, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:38 INFO TaskSetManager: Finished task 123.0 in stage 4.0 (TID 926) in 6932 ms on ip-10-0-51-185.ec2.internal (executor 54) (123/135) | |
| 17/06/14 06:14:39 INFO TaskSetManager: Starting task 125.0 in stage 4.0 (TID 928, ip-10-0-50-96.ec2.internal, executor 50, partition 125, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:39 INFO TaskSetManager: Finished task 122.0 in stage 4.0 (TID 925) in 8551 ms on ip-10-0-50-96.ec2.internal (executor 50) (124/135) | |
| 17/06/14 06:14:45 INFO TaskSetManager: Starting task 126.0 in stage 4.0 (TID 929, ip-10-0-51-185.ec2.internal, executor 54, partition 126, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:45 INFO TaskSetManager: Finished task 124.0 in stage 4.0 (TID 927) in 6455 ms on ip-10-0-51-185.ec2.internal (executor 54) (125/135) | |
| 17/06/14 06:14:45 INFO YarnAllocator: Driver requested a total number of 24 executor(s). | |
| 17/06/14 06:14:45 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 24 executors. | |
| 17/06/14 06:14:46 INFO TaskSetManager: Starting task 127.0 in stage 4.0 (TID 930, ip-10-0-50-96.ec2.internal, executor 50, partition 127, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:46 INFO TaskSetManager: Finished task 125.0 in stage 4.0 (TID 928) in 6489 ms on ip-10-0-50-96.ec2.internal (executor 50) (126/135) | |
| 17/06/14 06:14:52 INFO TaskSetManager: Starting task 128.0 in stage 4.0 (TID 931, ip-10-0-51-185.ec2.internal, executor 54, partition 128, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:52 INFO TaskSetManager: Finished task 126.0 in stage 4.0 (TID 929) in 6965 ms on ip-10-0-51-185.ec2.internal (executor 54) (127/135) | |
| 17/06/14 06:14:54 INFO TaskSetManager: Starting task 129.0 in stage 4.0 (TID 932, ip-10-0-50-96.ec2.internal, executor 50, partition 129, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:54 INFO TaskSetManager: Finished task 127.0 in stage 4.0 (TID 930) in 8269 ms on ip-10-0-50-96.ec2.internal (executor 50) (128/135) | |
| 17/06/14 06:14:58 INFO TaskSetManager: Starting task 130.0 in stage 4.0 (TID 933, ip-10-0-51-185.ec2.internal, executor 54, partition 130, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:14:58 INFO TaskSetManager: Finished task 128.0 in stage 4.0 (TID 931) in 6688 ms on ip-10-0-51-185.ec2.internal (executor 54) (129/135) | |
| 17/06/14 06:14:58 INFO YarnAllocator: Driver requested a total number of 23 executor(s). | |
| 17/06/14 06:14:58 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 23 executors. | |
| 17/06/14 06:15:00 INFO TaskSetManager: Starting task 131.0 in stage 4.0 (TID 934, ip-10-0-50-96.ec2.internal, executor 50, partition 131, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:15:00 INFO TaskSetManager: Finished task 129.0 in stage 4.0 (TID 932) in 6276 ms on ip-10-0-50-96.ec2.internal (executor 50) (130/135) | |
| 17/06/14 06:15:05 INFO TaskSetManager: Starting task 132.0 in stage 4.0 (TID 935, ip-10-0-51-185.ec2.internal, executor 54, partition 132, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:15:05 INFO TaskSetManager: Finished task 130.0 in stage 4.0 (TID 933) in 6860 ms on ip-10-0-51-185.ec2.internal (executor 54) (131/135) | |
| 17/06/14 06:15:09 INFO TaskSetManager: Starting task 133.0 in stage 4.0 (TID 936, ip-10-0-50-96.ec2.internal, executor 50, partition 133, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:15:09 INFO TaskSetManager: Finished task 131.0 in stage 4.0 (TID 934) in 8358 ms on ip-10-0-50-96.ec2.internal (executor 50) (132/135) | |
| 17/06/14 06:15:12 INFO TaskSetManager: Starting task 134.0 in stage 4.0 (TID 937, ip-10-0-51-185.ec2.internal, executor 54, partition 134, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:15:12 INFO TaskSetManager: Finished task 132.0 in stage 4.0 (TID 935) in 6552 ms on ip-10-0-51-185.ec2.internal (executor 54) (133/135) | |
| 17/06/14 06:15:12 INFO YarnAllocator: Driver requested a total number of 22 executor(s). | |
| 17/06/14 06:15:12 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 22 executors. | |
| 17/06/14 06:15:17 INFO TaskSetManager: Starting task 102.0 in stage 5.0 (TID 938, ip-10-0-50-96.ec2.internal, executor 50, partition 102, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:15:17 INFO TaskSetManager: Finished task 133.0 in stage 4.0 (TID 936) in 8420 ms on ip-10-0-50-96.ec2.internal (executor 50) (134/135) | |
| 17/06/14 06:15:17 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.50.96:37112 | |
| 17/06/14 06:15:17 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 6257 bytes | |
| 17/06/14 06:15:20 INFO TaskSetManager: Starting task 103.0 in stage 5.0 (TID 939, ip-10-0-51-185.ec2.internal, executor 54, partition 103, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:15:20 INFO TaskSetManager: Finished task 134.0 in stage 4.0 (TID 937) in 8568 ms on ip-10-0-51-185.ec2.internal (executor 54) (135/135) | |
| 17/06/14 06:15:20 INFO YarnClusterScheduler: Removed TaskSet 4.0, whose tasks have all completed, from pool | |
| 17/06/14 06:15:20 INFO DAGScheduler: ShuffleMapStage 4 (map at ShredJob.scala:378) finished in 472.660 s | |
| 17/06/14 06:15:20 INFO DAGScheduler: looking for newly runnable stages | |
| 17/06/14 06:15:20 INFO DAGScheduler: running: Set(ShuffleMapStage 5) | |
| 17/06/14 06:15:20 INFO DAGScheduler: waiting: Set(ResultStage 6) | |
| 17/06/14 06:15:20 INFO DAGScheduler: failed: Set() | |
| 17/06/14 06:15:20 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.0.51.185:35966 | |
| 17/06/14 06:15:20 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 6257 bytes | |
| 17/06/14 06:22:36 INFO TaskSetManager: Starting task 104.0 in stage 5.0 (TID 940, ip-10-0-49-157.ec2.internal, executor 36, partition 104, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:22:36 INFO TaskSetManager: Finished task 38.0 in stage 5.0 (TID 746) in 1237066 ms on ip-10-0-49-157.ec2.internal (executor 36) (50/135) | |
| 17/06/14 06:22:52 INFO TaskSetManager: Starting task 105.0 in stage 5.0 (TID 941, ip-10-0-50-96.ec2.internal, executor 41, partition 105, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:22:52 INFO TaskSetManager: Finished task 37.0 in stage 5.0 (TID 745) in 1261382 ms on ip-10-0-50-96.ec2.internal (executor 41) (51/135) | |
| 17/06/14 06:22:52 INFO YarnAllocator: Driver requested a total number of 21 executor(s). | |
| 17/06/14 06:22:52 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 21 executors. | |
| 17/06/14 06:22:54 INFO TaskSetManager: Starting task 106.0 in stage 5.0 (TID 942, ip-10-0-49-157.ec2.internal, executor 36, partition 106, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:22:54 INFO TaskSetManager: Finished task 39.0 in stage 5.0 (TID 747) in 1235099 ms on ip-10-0-49-157.ec2.internal (executor 36) (52/135) | |
| 17/06/14 06:23:33 INFO BlockManagerInfo: Added rdd_10_40 in memory on ip-10-0-49-157.ec2.internal:39668 (size: 824.9 MB, free: 2002.2 MB) | |
| 17/06/14 06:23:40 INFO TaskSetManager: Starting task 107.0 in stage 5.0 (TID 943, ip-10-0-49-157.ec2.internal, executor 36, partition 107, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:23:40 INFO TaskSetManager: Finished task 40.0 in stage 5.0 (TID 748) in 1242759 ms on ip-10-0-49-157.ec2.internal (executor 36) (53/135) | |
| 17/06/14 06:23:48 INFO TaskSetManager: Starting task 108.0 in stage 5.0 (TID 944, ip-10-0-50-96.ec2.internal, executor 41, partition 108, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:23:48 INFO TaskSetManager: Finished task 41.0 in stage 5.0 (TID 749) in 1244518 ms on ip-10-0-50-96.ec2.internal (executor 41) (54/135) | |
| 17/06/14 06:24:38 INFO TaskSetManager: Starting task 109.0 in stage 5.0 (TID 945, ip-10-0-49-229.ec2.internal, executor 52, partition 109, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:24:38 INFO TaskSetManager: Finished task 44.0 in stage 5.0 (TID 752) in 1252377 ms on ip-10-0-49-229.ec2.internal (executor 52) (55/135) | |
| 17/06/14 06:24:38 INFO YarnAllocator: Driver requested a total number of 20 executor(s). | |
| 17/06/14 06:24:38 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 20 executors. | |
| 17/06/14 06:24:43 INFO TaskSetManager: Starting task 110.0 in stage 5.0 (TID 946, ip-10-0-49-229.ec2.internal, executor 52, partition 110, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:24:43 INFO TaskSetManager: Finished task 49.0 in stage 5.0 (TID 756) in 1240225 ms on ip-10-0-49-229.ec2.internal (executor 52) (56/135) | |
| 17/06/14 06:24:43 INFO BlockManagerInfo: Added broadcast_5_piece0 on disk on ip-10-0-51-185.ec2.internal:45572 (size: 5.7 KB) | |
| 17/06/14 06:25:07 INFO TaskSetManager: Starting task 113.0 in stage 5.0 (TID 947, ip-10-0-49-157.ec2.internal, executor 36, partition 113, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:25:07 INFO TaskSetManager: Finished task 43.0 in stage 5.0 (TID 751) in 1284473 ms on ip-10-0-49-157.ec2.internal (executor 36) (57/135) | |
| 17/06/14 06:25:08 INFO TaskSetManager: Starting task 114.0 in stage 5.0 (TID 948, ip-10-0-50-96.ec2.internal, executor 45, partition 114, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:25:08 INFO TaskSetManager: Finished task 50.0 in stage 5.0 (TID 757) in 1242073 ms on ip-10-0-50-96.ec2.internal (executor 45) (58/135) | |
| 17/06/14 06:25:16 INFO TaskSetManager: Starting task 115.0 in stage 5.0 (TID 949, ip-10-0-49-229.ec2.internal, executor 52, partition 115, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:25:16 INFO TaskSetManager: Finished task 42.0 in stage 5.0 (TID 750) in 1296410 ms on ip-10-0-49-229.ec2.internal (executor 52) (59/135) | |
| 17/06/14 06:25:16 INFO YarnAllocator: Driver requested a total number of 19 executor(s). | |
| 17/06/14 06:25:16 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 19 executors. | |
| 17/06/14 06:25:20 INFO TaskSetManager: Starting task 116.0 in stage 5.0 (TID 950, ip-10-0-51-185.ec2.internal, executor 48, partition 116, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:25:20 INFO TaskSetManager: Finished task 59.0 in stage 5.0 (TID 766) in 1204118 ms on ip-10-0-51-185.ec2.internal (executor 48) (60/135) | |
| 17/06/14 06:25:23 INFO TaskSetManager: Starting task 117.0 in stage 5.0 (TID 951, ip-10-0-49-229.ec2.internal, executor 49, partition 117, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:25:23 INFO TaskSetManager: Finished task 45.0 in stage 5.0 (TID 753) in 1291029 ms on ip-10-0-49-229.ec2.internal (executor 49) (61/135) | |
| 17/06/14 06:25:24 INFO TaskSetManager: Starting task 118.0 in stage 5.0 (TID 952, ip-10-0-49-229.ec2.internal, executor 53, partition 118, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:25:24 INFO TaskSetManager: Finished task 53.0 in stage 5.0 (TID 760) in 1245980 ms on ip-10-0-49-229.ec2.internal (executor 53) (62/135) | |
| 17/06/14 06:25:36 INFO TaskSetManager: Starting task 119.0 in stage 5.0 (TID 953, ip-10-0-50-96.ec2.internal, executor 41, partition 119, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:25:36 INFO TaskSetManager: Finished task 52.0 in stage 5.0 (TID 759) in 1261003 ms on ip-10-0-50-96.ec2.internal (executor 41) (63/135) | |
| 17/06/14 06:25:36 INFO YarnAllocator: Driver requested a total number of 18 executor(s). | |
| 17/06/14 06:25:36 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 18 executors. | |
| 17/06/14 06:25:40 INFO BlockManagerInfo: Added rdd_11_46 in memory on ip-10-0-50-96.ec2.internal:37989 (size: 823.0 MB, free: 2004.1 MB) | |
| 17/06/14 06:25:41 INFO TaskSetManager: Starting task 120.0 in stage 5.0 (TID 954, ip-10-0-50-96.ec2.internal, executor 45, partition 120, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:25:41 INFO TaskSetManager: Finished task 57.0 in stage 5.0 (TID 764) in 1225868 ms on ip-10-0-50-96.ec2.internal (executor 45) (64/135) | |
| 17/06/14 06:25:44 INFO TaskSetManager: Starting task 121.0 in stage 5.0 (TID 955, ip-10-0-51-185.ec2.internal, executor 48, partition 121, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:25:44 INFO TaskSetManager: Finished task 68.0 in stage 5.0 (TID 773) in 1219707 ms on ip-10-0-51-185.ec2.internal (executor 48) (65/135) | |
| 17/06/14 06:25:44 INFO TaskSetManager: Starting task 122.0 in stage 5.0 (TID 956, ip-10-0-50-96.ec2.internal, executor 41, partition 122, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:25:44 INFO TaskSetManager: Finished task 46.0 in stage 5.0 (TID 754) in 1304498 ms on ip-10-0-50-96.ec2.internal (executor 41) (66/135) | |
| 17/06/14 06:25:47 INFO BlockManagerInfo: Removed rdd_11_46 on ip-10-0-50-96.ec2.internal:37989 in memory (size: 823.0 MB, free: 2.8 GB) | |
| 17/06/14 06:25:54 INFO TaskSetManager: Starting task 123.0 in stage 5.0 (TID 957, ip-10-0-51-185.ec2.internal, executor 37, partition 123, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:25:54 INFO TaskSetManager: Finished task 54.0 in stage 5.0 (TID 761) in 1269309 ms on ip-10-0-51-185.ec2.internal (executor 37) (67/135) | |
| 17/06/14 06:25:54 INFO YarnAllocator: Driver requested a total number of 17 executor(s). | |
| 17/06/14 06:25:54 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 17 executors. | |
| 17/06/14 06:26:03 INFO TaskSetManager: Starting task 124.0 in stage 5.0 (TID 958, ip-10-0-51-185.ec2.internal, executor 48, partition 124, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:26:03 INFO TaskSetManager: Finished task 67.0 in stage 5.0 (TID 772) in 1239198 ms on ip-10-0-51-185.ec2.internal (executor 48) (68/135) | |
| 17/06/14 06:26:07 INFO TaskSetManager: Starting task 125.0 in stage 5.0 (TID 959, ip-10-0-49-229.ec2.internal, executor 49, partition 125, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:26:07 INFO TaskSetManager: Finished task 51.0 in stage 5.0 (TID 758) in 1296412 ms on ip-10-0-49-229.ec2.internal (executor 49) (69/135) | |
| 17/06/14 06:26:10 INFO TaskSetManager: Starting task 126.0 in stage 5.0 (TID 960, ip-10-0-51-185.ec2.internal, executor 39, partition 126, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:26:10 INFO TaskSetManager: Finished task 62.0 in stage 5.0 (TID 769) in 1246865 ms on ip-10-0-51-185.ec2.internal (executor 39) (70/135) | |
| 17/06/14 06:26:10 INFO BlockManagerInfo: Added rdd_10_55 in memory on ip-10-0-49-229.ec2.internal:45654 (size: 855.0 MB, free: 1972.1 MB) | |
| 17/06/14 06:26:17 INFO TaskSetManager: Starting task 127.0 in stage 5.0 (TID 961, ip-10-0-49-229.ec2.internal, executor 49, partition 127, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:26:17 INFO TaskSetManager: Finished task 55.0 in stage 5.0 (TID 762) in 1278490 ms on ip-10-0-49-229.ec2.internal (executor 49) (71/135) | |
| 17/06/14 06:26:17 INFO YarnAllocator: Driver requested a total number of 16 executor(s). | |
| 17/06/14 06:26:17 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 16 executors. | |
| 17/06/14 06:26:22 INFO TaskSetManager: Starting task 128.0 in stage 5.0 (TID 962, ip-10-0-49-229.ec2.internal, executor 52, partition 128, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:26:22 INFO TaskSetManager: Finished task 47.0 in stage 5.0 (TID 755) in 1341860 ms on ip-10-0-49-229.ec2.internal (executor 52) (72/135) | |
| 17/06/14 06:26:30 INFO TaskSetManager: Starting task 130.0 in stage 5.0 (TID 963, ip-10-0-51-185.ec2.internal, executor 37, partition 130, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:26:30 INFO TaskSetManager: Finished task 74.0 in stage 5.0 (TID 778) in 1247200 ms on ip-10-0-51-185.ec2.internal (executor 37) (73/135) | |
| 17/06/14 06:26:30 INFO TaskSetManager: Starting task 132.0 in stage 5.0 (TID 964, ip-10-0-51-185.ec2.internal, executor 39, partition 132, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:26:30 INFO TaskSetManager: Finished task 78.0 in stage 5.0 (TID 781) in 1235538 ms on ip-10-0-51-185.ec2.internal (executor 39) (74/135) | |
| 17/06/14 06:26:31 INFO TaskSetManager: Starting task 133.0 in stage 5.0 (TID 965, ip-10-0-51-185.ec2.internal, executor 37, partition 133, PROCESS_LOCAL, 5763 bytes) | |
| 17/06/14 06:26:31 INFO TaskSetManager: Finished task 82.0 in stage 5.0 (TID 784) in 1231371 ms on ip-10-0-51-185.ec2.internal (executor 37) (75/135) | |
| 17/06/14 06:26:31 INFO YarnAllocator: Driver requested a total number of 15 executor(s). | |
| 17/06/14 06:26:31 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 15 executors. | |
| 17/06/14 06:26:34 INFO TaskSetManager: Finished task 56.0 in stage 5.0 (TID 763) in 1291223 ms on ip-10-0-51-185.ec2.internal (executor 39) (76/135) | |
| 17/06/14 06:26:37 INFO TaskSetManager: Finished task 79.0 in stage 5.0 (TID 782) in 1240901 ms on ip-10-0-49-229.ec2.internal (executor 53) (77/135) | |
| 17/06/14 06:26:42 INFO BlockManagerInfo: Added rdd_10_58 in memory on ip-10-0-49-157.ec2.internal:44407 (size: 810.5 MB, free: 2016.7 MB) | |
| 17/06/14 06:26:45 INFO BlockManagerInfo: Added rdd_10_70 in memory on ip-10-0-49-229.ec2.internal:44599 (size: 820.0 MB, free: 2007.1 MB) | |
| 17/06/14 06:26:46 INFO TaskSetManager: Finished task 77.0 in stage 5.0 (TID 780) in 1252321 ms on ip-10-0-49-157.ec2.internal (executor 44) (78/135) | |
| 17/06/14 06:26:46 INFO BlockManagerInfo: Added rdd_11_58 in memory on ip-10-0-49-157.ec2.internal:44407 (size: 826.1 MB, free: 1190.5 MB) | |
| 17/06/14 06:26:47 INFO TaskSetManager: Finished task 87.0 in stage 5.0 (TID 789) in 1224183 ms on ip-10-0-49-157.ec2.internal (executor 40) (79/135) | |
| 17/06/14 06:26:47 INFO YarnAllocator: Driver requested a total number of 14 executor(s). | |
| 17/06/14 06:26:47 INFO BlockManagerInfo: Added rdd_10_65 in memory on ip-10-0-49-229.ec2.internal:44599 (size: 809.7 MB, free: 1197.4 MB) | |
| 17/06/14 06:26:49 INFO TaskSetManager: Finished task 58.0 in stage 5.0 (TID 765) in 1293000 ms on ip-10-0-49-157.ec2.internal (executor 51) (80/135) | |
| 17/06/14 06:26:51 INFO TaskSetManager: Finished task 70.0 in stage 5.0 (TID 775) in 1277145 ms on ip-10-0-49-229.ec2.internal (executor 53) (81/135) | |
| 17/06/14 06:26:53 INFO TaskSetManager: Finished task 65.0 in stage 5.0 (TID 771) in 1288877 ms on ip-10-0-49-229.ec2.internal (executor 53) (82/135) | |
| 17/06/14 06:26:54 INFO TaskSetManager: Finished task 92.0 in stage 5.0 (TID 794) in 1213566 ms on ip-10-0-49-157.ec2.internal (executor 44) (83/135) | |
| 17/06/14 06:26:55 INFO YarnAllocator: Driver requested a total number of 13 executor(s). | |
| 17/06/14 06:26:55 INFO BlockManagerInfo: Added rdd_10_94 in memory on ip-10-0-51-185.ec2.internal:46252 (size: 816.4 MB, free: 2010.7 MB) | |
| 17/06/14 06:26:57 INFO TaskSetManager: Finished task 85.0 in stage 5.0 (TID 787) in 1236841 ms on ip-10-0-49-157.ec2.internal (executor 51) (84/135) | |
| 17/06/14 06:26:58 INFO BlockManagerInfo: Added rdd_11_94 in memory on ip-10-0-51-185.ec2.internal:46252 (size: 838.0 MB, free: 1172.7 MB) | |
| 17/06/14 06:27:01 INFO TaskSetManager: Finished task 94.0 in stage 5.0 (TID 796) in 1206891 ms on ip-10-0-51-185.ec2.internal (executor 39) (85/135) | |
| 17/06/14 06:27:03 INFO BlockManagerInfo: Added rdd_10_72 in memory on ip-10-0-51-185.ec2.internal:34029 (size: 815.3 MB, free: 2011.8 MB) | |
| 17/06/14 06:27:07 INFO BlockManagerInfo: Added rdd_10_71 in memory on ip-10-0-49-157.ec2.internal:45984 (size: 853.6 MB, free: 1973.5 MB) | |
| 17/06/14 06:27:09 INFO TaskSetManager: Finished task 72.0 in stage 5.0 (TID 777) in 1292876 ms on ip-10-0-51-185.ec2.internal (executor 48) (86/135) | |
| 17/06/14 06:27:11 INFO BlockManagerInfo: Added rdd_11_71 in memory on ip-10-0-49-157.ec2.internal:45984 (size: 856.1 MB, free: 1117.4 MB) | |
| 17/06/14 06:27:12 INFO TaskSetManager: Finished task 61.0 in stage 5.0 (TID 768) in 1310425 ms on ip-10-0-50-96.ec2.internal (executor 45) (87/135) | |
| 17/06/14 06:27:12 INFO YarnAllocator: Driver requested a total number of 12 executor(s). | |
| 17/06/14 06:27:14 INFO TaskSetManager: Finished task 71.0 in stage 5.0 (TID 776) in 1298720 ms on ip-10-0-49-157.ec2.internal (executor 44) (88/135) | |
| 17/06/14 06:27:14 INFO TaskSetManager: Finished task 64.0 in stage 5.0 (TID 770) in 1310725 ms on ip-10-0-49-157.ec2.internal (executor 51) (89/135) | |
| 17/06/14 06:27:15 INFO BlockManagerInfo: Added rdd_10_90 in memory on ip-10-0-49-157.ec2.internal:42968 (size: 829.9 MB, free: 1997.3 MB) | |
| 17/06/14 06:27:15 INFO TaskSetManager: Finished task 84.0 in stage 5.0 (TID 786) in 1264254 ms on ip-10-0-51-185.ec2.internal (executor 37) (90/135) | |
| 17/06/14 06:27:18 INFO BlockManagerInfo: Added rdd_10_76 in memory on ip-10-0-50-96.ec2.internal:40866 (size: 808.4 MB, free: 2018.8 MB) | |
| 17/06/14 06:27:18 INFO BlockManagerInfo: Added rdd_11_90 in memory on ip-10-0-49-157.ec2.internal:42968 (size: 840.8 MB, free: 1156.5 MB) | |
| 17/06/14 06:27:19 INFO TaskSetManager: Finished task 69.0 in stage 5.0 (TID 774) in 1310927 ms on ip-10-0-49-157.ec2.internal (executor 51) (91/135) | |
| 17/06/14 06:27:19 INFO YarnAllocator: Driver requested a total number of 11 executor(s). | |
| 17/06/14 06:27:19 INFO TaskSetManager: Finished task 83.0 in stage 5.0 (TID 785) in 1269088 ms on ip-10-0-49-157.ec2.internal (executor 40) (92/135) | |
| 17/06/14 06:27:21 INFO TaskSetManager: Finished task 90.0 in stage 5.0 (TID 792) in 1242321 ms on ip-10-0-49-157.ec2.internal (executor 40) (93/135) | |
| 17/06/14 06:27:24 INFO TaskSetManager: Finished task 76.0 in stage 5.0 (TID 779) in 1295731 ms on ip-10-0-50-96.ec2.internal (executor 45) (94/135) | |
| 17/06/14 06:27:27 INFO TaskSetManager: Finished task 60.0 in stage 5.0 (TID 767) in 1327099 ms on ip-10-0-49-229.ec2.internal (executor 49) (95/135) | |
| 17/06/14 06:27:27 INFO YarnAllocator: Driver requested a total number of 10 executor(s). | |
| 17/06/14 06:27:31 INFO TaskSetManager: Finished task 98.0 in stage 5.0 (TID 800) in 1225135 ms on ip-10-0-50-96.ec2.internal (executor 55) (96/135) | |
| 17/06/14 06:27:42 INFO TaskSetManager: Finished task 80.0 in stage 5.0 (TID 783) in 1303800 ms on ip-10-0-50-96.ec2.internal (executor 50) (97/135) | |
| 17/06/14 06:27:44 INFO TaskSetManager: Finished task 97.0 in stage 5.0 (TID 799) in 1240012 ms on ip-10-0-50-96.ec2.internal (executor 55) (98/135) | |
| 17/06/14 06:27:48 INFO BlockManagerInfo: Removed rdd_10_40 on ip-10-0-49-157.ec2.internal:39668 in memory (size: 824.9 MB, free: 2.8 GB) | |
| 17/06/14 06:27:50 INFO BlockManagerInfo: Added rdd_10_89 in memory on ip-10-0-50-96.ec2.internal:34711 (size: 835.5 MB, free: 1991.7 MB) | |
| 17/06/14 06:27:54 INFO BlockManagerInfo: Added rdd_11_89 in memory on ip-10-0-50-96.ec2.internal:34711 (size: 848.9 MB, free: 1142.7 MB) | |
| 17/06/14 06:27:56 INFO TaskSetManager: Finished task 89.0 in stage 5.0 (TID 791) in 1284289 ms on ip-10-0-50-96.ec2.internal (executor 50) (99/135) | |
| 17/06/14 06:27:56 INFO YarnAllocator: Driver requested a total number of 9 executor(s). | |
| 17/06/14 06:27:57 INFO TaskSetManager: Finished task 96.0 in stage 5.0 (TID 798) in 1254733 ms on ip-10-0-49-157.ec2.internal (executor 40) (100/135) | |
| 17/06/14 06:28:02 INFO TaskSetManager: Finished task 93.0 in stage 5.0 (TID 795) in 1273644 ms on ip-10-0-50-96.ec2.internal (executor 55) (101/135) | |
| 17/06/14 06:28:04 INFO BlockManagerInfo: Added rdd_10_86 in memory on ip-10-0-51-185.ec2.internal:45572 (size: 837.3 MB, free: 1989.8 MB) | |
| 17/06/14 06:28:04 INFO TaskSetManager: Finished task 101.0 in stage 5.0 (TID 802) in 1236655 ms on ip-10-0-50-96.ec2.internal (executor 55) (102/135) | |
| 17/06/14 06:28:10 INFO TaskSetManager: Finished task 86.0 in stage 5.0 (TID 788) in 1308159 ms on ip-10-0-51-185.ec2.internal (executor 54) (103/135) | |
| 17/06/14 06:28:10 INFO YarnAllocator: Driver requested a total number of 8 executor(s). | |
| 17/06/14 06:28:17 INFO TaskSetManager: Finished task 95.0 in stage 5.0 (TID 797) in 1282684 ms on ip-10-0-51-185.ec2.internal (executor 54) (104/135) | |
| 17/06/14 06:28:18 INFO BlockManagerInfo: Added broadcast_5_piece0 on disk on ip-10-0-50-96.ec2.internal:34711 (size: 5.7 KB) | |
| 17/06/14 06:28:18 INFO BlockManagerInfo: Removed rdd_11_89 on ip-10-0-50-96.ec2.internal:34711 in memory (size: 848.9 MB, free: 1991.7 MB) | |
| 17/06/14 06:28:20 INFO TaskSetManager: Finished task 100.0 in stage 5.0 (TID 801) in 1267506 ms on ip-10-0-49-157.ec2.internal (executor 44) (105/135) | |
| 17/06/14 06:28:23 INFO TaskSetManager: Finished task 91.0 in stage 5.0 (TID 793) in 1302781 ms on ip-10-0-50-96.ec2.internal (executor 50) (106/135) | |
| 17/06/14 06:28:28 INFO TaskSetManager: Finished task 88.0 in stage 5.0 (TID 790) in 1319359 ms on ip-10-0-51-185.ec2.internal (executor 54) (107/135) | |
| 17/06/14 06:28:28 INFO YarnAllocator: Driver requested a total number of 7 executor(s). | |
| 17/06/14 06:28:56 INFO BlockManagerInfo: Removed rdd_10_72 on ip-10-0-51-185.ec2.internal:34029 in memory (size: 815.3 MB, free: 2.8 GB) | |
| 17/06/14 06:29:04 INFO ExecutorAllocationManager: Request to remove executorIds: 55 | |
| 17/06/14 06:29:04 INFO YarnClusterSchedulerBackend: Requesting to kill executor(s) 55 | |
| 17/06/14 06:29:04 INFO YarnClusterSchedulerBackend: Actual list of executor(s) to be killed is 55 | |
| 17/06/14 06:29:04 INFO YarnAllocator: Driver requested a total number of 14 executor(s). | |
| 17/06/14 06:29:04 INFO ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 55. | |
| 17/06/14 06:29:04 INFO ExecutorAllocationManager: Removing executor 55 because it has been idle for 60 seconds (new desired total will be 14) | |
| 17/06/14 06:29:05 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 55. | |
| 17/06/14 06:29:05 INFO DAGScheduler: Executor lost: 55 (epoch 3) | |
| 17/06/14 06:29:05 INFO BlockManagerMasterEndpoint: Trying to remove executor 55 from BlockManagerMaster. | |
| 17/06/14 06:29:05 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(55, ip-10-0-50-96.ec2.internal, 40971, None) | |
| 17/06/14 06:29:05 INFO BlockManagerMaster: Removed 55 successfully in removeExecutor | |
| 17/06/14 06:29:05 INFO YarnClusterScheduler: Executor 55 on ip-10-0-50-96.ec2.internal killed by driver. | |
| 17/06/14 06:29:05 INFO ExecutorAllocationManager: Existing executor 55 has been removed (new total is 14) | |
| 17/06/14 06:29:09 INFO BlockManagerInfo: Removed rdd_10_55 on ip-10-0-49-229.ec2.internal:45654 in memory (size: 855.0 MB, free: 2.8 GB) | |
| 17/06/14 06:30:04 INFO BlockManagerInfo: Removed rdd_11_94 on ip-10-0-51-185.ec2.internal:46252 in memory (size: 838.0 MB, free: 2010.7 MB) | |
| 17/06/14 06:31:17 INFO BlockManagerInfo: Removed rdd_10_76 on ip-10-0-50-96.ec2.internal:40866 in memory (size: 808.4 MB, free: 2.8 GB) | |
| 17/06/14 06:31:45 INFO BlockManagerInfo: Added rdd_10_102 in memory on ip-10-0-50-96.ec2.internal:34711 (size: 815.8 MB, free: 1175.9 MB) | |
| 17/06/14 06:31:48 INFO BlockManagerInfo: Added rdd_11_102 in memory on ip-10-0-50-96.ec2.internal:34711 (size: 801.0 MB, free: 374.8 MB) | |
| 17/06/14 06:31:51 INFO TaskSetManager: Finished task 102.0 in stage 5.0 (TID 938) in 993756 ms on ip-10-0-50-96.ec2.internal (executor 50) (108/135) | |
| 17/06/14 06:32:15 INFO TaskSetManager: Finished task 103.0 in stage 5.0 (TID 939) in 1014383 ms on ip-10-0-51-185.ec2.internal (executor 54) (109/135) | |
| 17/06/14 06:33:43 INFO BlockManagerInfo: Removed rdd_10_94 on ip-10-0-51-185.ec2.internal:46252 in memory (size: 816.4 MB, free: 2.8 GB) | |
| 17/06/14 06:34:46 INFO TaskSetManager: Finished task 104.0 in stage 5.0 (TID 940) in 730402 ms on ip-10-0-49-157.ec2.internal (executor 36) (110/135) | |
| 17/06/14 06:34:49 INFO TaskSetManager: Finished task 105.0 in stage 5.0 (TID 941) in 716971 ms on ip-10-0-50-96.ec2.internal (executor 41) (111/135) | |
| 17/06/14 06:34:49 INFO YarnAllocator: Driver requested a total number of 6 executor(s). | |
| 17/06/14 06:34:51 INFO BlockManagerInfo: Added rdd_10_106 in memory on ip-10-0-49-157.ec2.internal:39668 (size: 843.4 MB, free: 1983.7 MB) | |
| 17/06/14 06:34:54 INFO BlockManagerInfo: Added rdd_11_106 in memory on ip-10-0-49-157.ec2.internal:39668 (size: 785.7 MB, free: 1198.0 MB) | |
| 17/06/14 06:34:57 INFO TaskSetManager: Finished task 106.0 in stage 5.0 (TID 942) in 723192 ms on ip-10-0-49-157.ec2.internal (executor 36) (112/135) | |
| 17/06/14 06:35:18 INFO TaskSetManager: Finished task 108.0 in stage 5.0 (TID 944) in 690157 ms on ip-10-0-50-96.ec2.internal (executor 41) (113/135) | |
| 17/06/14 06:35:19 INFO TaskSetManager: Finished task 107.0 in stage 5.0 (TID 943) in 699380 ms on ip-10-0-49-157.ec2.internal (executor 36) (114/135) | |
| 17/06/14 06:35:27 INFO TaskSetManager: Finished task 113.0 in stage 5.0 (TID 947) in 620412 ms on ip-10-0-49-157.ec2.internal (executor 36) (115/135) | |
| 17/06/14 06:35:27 INFO YarnAllocator: Driver requested a total number of 5 executor(s). | |
| 17/06/14 06:35:32 INFO TaskSetManager: Finished task 109.0 in stage 5.0 (TID 945) in 654554 ms on ip-10-0-49-229.ec2.internal (executor 52) (116/135) | |
| 17/06/14 06:35:45 INFO TaskSetManager: Finished task 114.0 in stage 5.0 (TID 948) in 637161 ms on ip-10-0-50-96.ec2.internal (executor 45) (117/135) | |
| 17/06/14 06:35:46 INFO TaskSetManager: Finished task 115.0 in stage 5.0 (TID 949) in 630722 ms on ip-10-0-49-229.ec2.internal (executor 52) (118/135) | |
| 17/06/14 06:35:48 INFO TaskSetManager: Finished task 110.0 in stage 5.0 (TID 946) in 665530 ms on ip-10-0-49-229.ec2.internal (executor 52) (119/135) | |
| 17/06/14 06:35:48 INFO YarnAllocator: Driver requested a total number of 4 executor(s). | |
| 17/06/14 06:35:49 INFO TaskSetManager: Finished task 119.0 in stage 5.0 (TID 953) in 613187 ms on ip-10-0-50-96.ec2.internal (executor 41) (120/135) | |
| 17/06/14 06:35:55 INFO TaskSetManager: Finished task 122.0 in stage 5.0 (TID 956) in 610877 ms on ip-10-0-50-96.ec2.internal (executor 41) (121/135) | |
| 17/06/14 06:35:59 INFO TaskSetManager: Finished task 116.0 in stage 5.0 (TID 950) in 638489 ms on ip-10-0-51-185.ec2.internal (executor 48) (122/135) | |
| 17/06/14 06:36:03 INFO TaskSetManager: Finished task 117.0 in stage 5.0 (TID 951) in 640492 ms on ip-10-0-49-229.ec2.internal (executor 49) (123/135) | |
| 17/06/14 06:36:03 INFO YarnAllocator: Driver requested a total number of 3 executor(s). | |
| 17/06/14 06:36:04 INFO BlockManagerInfo: Added rdd_10_120 in memory on ip-10-0-50-96.ec2.internal:40866 (size: 855.3 MB, free: 1971.8 MB) | |
| 17/06/14 06:36:05 INFO TaskSetManager: Finished task 128.0 in stage 5.0 (TID 962) in 583397 ms on ip-10-0-49-229.ec2.internal (executor 52) (124/135) | |
| 17/06/14 06:36:07 INFO BlockManagerInfo: Added rdd_10_121 in memory on ip-10-0-51-185.ec2.internal:34029 (size: 841.3 MB, free: 1985.8 MB) | |
| 17/06/14 06:36:07 INFO BlockManagerInfo: Added rdd_11_120 in memory on ip-10-0-50-96.ec2.internal:40866 (size: 821.1 MB, free: 1150.7 MB) | |
| 17/06/14 06:36:08 INFO BlockManagerInfo: Added rdd_11_125 in memory on ip-10-0-49-229.ec2.internal:45654 (size: 856.8 MB, free: 1970.3 MB) | |
| 17/06/14 06:36:10 INFO TaskSetManager: Finished task 120.0 in stage 5.0 (TID 954) in 628543 ms on ip-10-0-50-96.ec2.internal (executor 45) (125/135) | |
| 17/06/14 06:36:10 INFO BlockManagerInfo: Added rdd_10_118 in memory on ip-10-0-49-229.ec2.internal:44599 (size: 801.1 MB, free: 396.3 MB) | |
| 17/06/14 06:36:10 INFO TaskSetManager: Finished task 125.0 in stage 5.0 (TID 959) in 603391 ms on ip-10-0-49-229.ec2.internal (executor 49) (126/135) | |
| 17/06/14 06:36:11 INFO BlockManagerInfo: Added rdd_11_121 in memory on ip-10-0-51-185.ec2.internal:34029 (size: 827.0 MB, free: 1158.8 MB) | |
| 17/06/14 06:36:11 INFO TaskSetManager: Finished task 127.0 in stage 5.0 (TID 961) in 594320 ms on ip-10-0-49-229.ec2.internal (executor 49) (127/135) | |
| 17/06/14 06:36:11 INFO YarnAllocator: Driver requested a total number of 2 executor(s). | |
| 17/06/14 06:36:13 INFO BlockManagerInfo: Added broadcast_4_piece0 on disk on ip-10-0-49-229.ec2.internal:44599 (size: 5.5 KB) | |
| 17/06/14 06:36:13 INFO BlockManagerInfo: Removed rdd_10_70 on ip-10-0-49-229.ec2.internal:44599 in memory (size: 820.0 MB, free: 1216.4 MB) | |
| 17/06/14 06:36:13 INFO TaskSetManager: Finished task 121.0 in stage 5.0 (TID 955) in 629810 ms on ip-10-0-51-185.ec2.internal (executor 48) (128/135) | |
| 17/06/14 06:36:14 INFO BlockManagerInfo: Added rdd_11_118 in memory on ip-10-0-49-229.ec2.internal:44599 (size: 839.9 MB, free: 376.5 MB) | |
| 17/06/14 06:36:15 INFO BlockManagerInfo: Added rdd_10_126 in memory on ip-10-0-51-185.ec2.internal:46252 (size: 838.8 MB, free: 1988.3 MB) | |
| 17/06/14 06:36:16 INFO TaskSetManager: Finished task 118.0 in stage 5.0 (TID 952) in 651615 ms on ip-10-0-49-229.ec2.internal (executor 53) (129/135) | |
| 17/06/14 06:36:16 INFO BlockManagerInfo: Added rdd_11_132 in memory on ip-10-0-51-185.ec2.internal:46252 (size: 816.0 MB, free: 1172.3 MB) | |
| 17/06/14 06:36:19 INFO BlockManagerInfo: Added rdd_11_126 in memory on ip-10-0-51-185.ec2.internal:46252 (size: 860.3 MB, free: 312.0 MB) | |
| 17/06/14 06:36:19 INFO TaskSetManager: Finished task 132.0 in stage 5.0 (TID 964) in 588286 ms on ip-10-0-51-185.ec2.internal (executor 39) (130/135) | |
| 17/06/14 06:36:19 INFO TaskSetManager: Finished task 124.0 in stage 5.0 (TID 958) in 616418 ms on ip-10-0-51-185.ec2.internal (executor 48) (131/135) | |
| 17/06/14 06:36:19 INFO YarnAllocator: Driver requested a total number of 1 executor(s). | |
| 17/06/14 06:36:21 INFO TaskSetManager: Finished task 126.0 in stage 5.0 (TID 960) in 611126 ms on ip-10-0-51-185.ec2.internal (executor 39) (132/135) | |
| 17/06/14 06:36:35 INFO TaskSetManager: Finished task 133.0 in stage 5.0 (TID 965) in 604343 ms on ip-10-0-51-185.ec2.internal (executor 37) (133/135) | |
| 17/06/14 06:36:36 INFO TaskSetManager: Finished task 123.0 in stage 5.0 (TID 957) in 641752 ms on ip-10-0-51-185.ec2.internal (executor 37) (134/135) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Finished task 130.0 in stage 5.0 (TID 963) in 610878 ms on ip-10-0-51-185.ec2.internal (executor 37) (135/135) | |
| 17/06/14 06:36:41 INFO YarnClusterScheduler: Removed TaskSet 5.0, whose tasks have all completed, from pool | |
| 17/06/14 06:36:41 INFO DAGScheduler: ShuffleMapStage 5 (map at ShredJob.scala:384) finished in 5708.617 s | |
| 17/06/14 06:36:41 INFO DAGScheduler: looking for newly runnable stages | |
| 17/06/14 06:36:41 INFO DAGScheduler: running: Set() | |
| 17/06/14 06:36:41 INFO DAGScheduler: waiting: Set(ResultStage 6) | |
| 17/06/14 06:36:41 INFO DAGScheduler: failed: Set() | |
| 17/06/14 06:36:41 INFO DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[29] at saveAsTextFile at ShredJob.scala:412), which has no missing parents | |
| 17/06/14 06:36:41 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 80.0 KB, free 413.5 MB) | |
| 17/06/14 06:36:41 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 30.1 KB, free 413.5 MB) | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 10.0.49.229:33721 (size: 30.1 KB, free: 413.9 MB) | |
| 17/06/14 06:36:41 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:997 | |
| 17/06/14 06:36:41 INFO DAGScheduler: Submitting 135 missing tasks from ResultStage 6 (MapPartitionsRDD[29] at saveAsTextFile at ShredJob.scala:412) | |
| 17/06/14 06:36:41 INFO YarnClusterScheduler: Adding task set 6.0 with 135 tasks | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 966, ip-10-0-49-229.ec2.internal, executor 52, partition 0, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 1.0 in stage 6.0 (TID 967, ip-10-0-49-157.ec2.internal, executor 51, partition 1, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 2.0 in stage 6.0 (TID 968, ip-10-0-50-96.ec2.internal, executor 41, partition 2, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 3.0 in stage 6.0 (TID 969, ip-10-0-51-185.ec2.internal, executor 39, partition 3, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 4.0 in stage 6.0 (TID 970, ip-10-0-50-96.ec2.internal, executor 50, partition 4, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 5.0 in stage 6.0 (TID 971, ip-10-0-49-157.ec2.internal, executor 44, partition 5, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 6.0 in stage 6.0 (TID 972, ip-10-0-51-185.ec2.internal, executor 54, partition 6, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 7.0 in stage 6.0 (TID 973, ip-10-0-49-229.ec2.internal, executor 49, partition 7, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 8.0 in stage 6.0 (TID 974, ip-10-0-49-157.ec2.internal, executor 36, partition 8, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 9.0 in stage 6.0 (TID 975, ip-10-0-49-157.ec2.internal, executor 40, partition 9, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 10.0 in stage 6.0 (TID 976, ip-10-0-51-185.ec2.internal, executor 48, partition 10, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 11.0 in stage 6.0 (TID 977, ip-10-0-49-229.ec2.internal, executor 53, partition 11, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 12.0 in stage 6.0 (TID 978, ip-10-0-51-185.ec2.internal, executor 37, partition 12, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 13.0 in stage 6.0 (TID 979, ip-10-0-50-96.ec2.internal, executor 45, partition 13, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 14.0 in stage 6.0 (TID 980, ip-10-0-49-229.ec2.internal, executor 52, partition 14, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 15.0 in stage 6.0 (TID 981, ip-10-0-49-157.ec2.internal, executor 51, partition 15, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 16.0 in stage 6.0 (TID 982, ip-10-0-50-96.ec2.internal, executor 41, partition 16, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 17.0 in stage 6.0 (TID 983, ip-10-0-51-185.ec2.internal, executor 39, partition 17, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 18.0 in stage 6.0 (TID 984, ip-10-0-50-96.ec2.internal, executor 50, partition 18, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 19.0 in stage 6.0 (TID 985, ip-10-0-49-157.ec2.internal, executor 44, partition 19, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 20.0 in stage 6.0 (TID 986, ip-10-0-51-185.ec2.internal, executor 54, partition 20, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 21.0 in stage 6.0 (TID 987, ip-10-0-49-229.ec2.internal, executor 49, partition 21, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 22.0 in stage 6.0 (TID 988, ip-10-0-49-157.ec2.internal, executor 36, partition 22, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 23.0 in stage 6.0 (TID 989, ip-10-0-49-157.ec2.internal, executor 40, partition 23, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 24.0 in stage 6.0 (TID 990, ip-10-0-51-185.ec2.internal, executor 48, partition 24, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 25.0 in stage 6.0 (TID 991, ip-10-0-49-229.ec2.internal, executor 53, partition 25, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 26.0 in stage 6.0 (TID 992, ip-10-0-51-185.ec2.internal, executor 37, partition 26, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 27.0 in stage 6.0 (TID 993, ip-10-0-50-96.ec2.internal, executor 45, partition 27, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 28.0 in stage 6.0 (TID 994, ip-10-0-49-229.ec2.internal, executor 52, partition 28, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 29.0 in stage 6.0 (TID 995, ip-10-0-49-157.ec2.internal, executor 51, partition 29, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 30.0 in stage 6.0 (TID 996, ip-10-0-50-96.ec2.internal, executor 41, partition 30, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 31.0 in stage 6.0 (TID 997, ip-10-0-51-185.ec2.internal, executor 39, partition 31, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 32.0 in stage 6.0 (TID 998, ip-10-0-50-96.ec2.internal, executor 50, partition 32, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 33.0 in stage 6.0 (TID 999, ip-10-0-49-157.ec2.internal, executor 44, partition 33, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 34.0 in stage 6.0 (TID 1000, ip-10-0-51-185.ec2.internal, executor 54, partition 34, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 35.0 in stage 6.0 (TID 1001, ip-10-0-49-229.ec2.internal, executor 49, partition 35, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 36.0 in stage 6.0 (TID 1002, ip-10-0-49-157.ec2.internal, executor 36, partition 36, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 37.0 in stage 6.0 (TID 1003, ip-10-0-49-157.ec2.internal, executor 40, partition 37, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 38.0 in stage 6.0 (TID 1004, ip-10-0-51-185.ec2.internal, executor 48, partition 38, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 39.0 in stage 6.0 (TID 1005, ip-10-0-49-229.ec2.internal, executor 53, partition 39, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 40.0 in stage 6.0 (TID 1006, ip-10-0-51-185.ec2.internal, executor 37, partition 40, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 41.0 in stage 6.0 (TID 1007, ip-10-0-50-96.ec2.internal, executor 45, partition 41, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 42.0 in stage 6.0 (TID 1008, ip-10-0-49-229.ec2.internal, executor 52, partition 42, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 43.0 in stage 6.0 (TID 1009, ip-10-0-49-157.ec2.internal, executor 51, partition 43, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 44.0 in stage 6.0 (TID 1010, ip-10-0-50-96.ec2.internal, executor 41, partition 44, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 45.0 in stage 6.0 (TID 1011, ip-10-0-51-185.ec2.internal, executor 39, partition 45, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 46.0 in stage 6.0 (TID 1012, ip-10-0-50-96.ec2.internal, executor 50, partition 46, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 47.0 in stage 6.0 (TID 1013, ip-10-0-49-157.ec2.internal, executor 44, partition 47, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 48.0 in stage 6.0 (TID 1014, ip-10-0-51-185.ec2.internal, executor 54, partition 48, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 49.0 in stage 6.0 (TID 1015, ip-10-0-49-229.ec2.internal, executor 49, partition 49, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 50.0 in stage 6.0 (TID 1016, ip-10-0-49-157.ec2.internal, executor 36, partition 50, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 51.0 in stage 6.0 (TID 1017, ip-10-0-49-157.ec2.internal, executor 40, partition 51, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 52.0 in stage 6.0 (TID 1018, ip-10-0-51-185.ec2.internal, executor 48, partition 52, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 53.0 in stage 6.0 (TID 1019, ip-10-0-49-229.ec2.internal, executor 53, partition 53, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 54.0 in stage 6.0 (TID 1020, ip-10-0-51-185.ec2.internal, executor 37, partition 54, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO TaskSetManager: Starting task 55.0 in stage 6.0 (TID 1021, ip-10-0-50-96.ec2.internal, executor 45, partition 55, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-51-185.ec2.internal:45572 (size: 30.1 KB, free: 1989.8 MB) | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-51-185.ec2.internal:33788 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-229.ec2.internal:44395 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-51-185.ec2.internal:46252 (size: 30.1 KB, free: 312.0 MB) | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-157.ec2.internal:39668 (size: 30.1 KB, free: 1198.0 MB) | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-157.ec2.internal:44407 (size: 30.1 KB, free: 1190.5 MB) | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-50-96.ec2.internal:34711 (size: 30.1 KB, free: 374.8 MB) | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-51-185.ec2.internal:34029 (size: 30.1 KB, free: 1158.8 MB) | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-50-96.ec2.internal:37989 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-50-96.ec2.internal:40866 (size: 30.1 KB, free: 1150.6 MB) | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-229.ec2.internal:45654 (size: 30.1 KB, free: 1970.3 MB) | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-157.ec2.internal:45984 (size: 30.1 KB, free: 1117.3 MB) | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-229.ec2.internal:44599 (size: 30.1 KB, free: 376.5 MB) | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.157:46182 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 5105 bytes | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.157:45932 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.51.185:35966 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.50.96:36892 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.51.185:35586 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.51.185:35966 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 475 bytes | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.51.185:35570 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.157:45972 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.51.185:35620 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.50.96:36940 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.50.96:36892 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.229:50552 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.229:50144 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.229:50616 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.51.185:35586 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.229:50552 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.229:50144 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.51.185:35620 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.157:45972 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.157:45932 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.50.96:36940 | |
| 17/06/14 06:36:41 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-157.ec2.internal:42968 (size: 30.1 KB, free: 1156.4 MB) | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.157:46182 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.229:50616 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.50.96:37112 | |
| 17/06/14 06:36:41 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.51.185:35570 | |
| 17/06/14 06:36:42 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.50.96:37112 | |
| 17/06/14 06:36:42 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.157:45956 | |
| 17/06/14 06:36:42 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.157:45956 | |
| 17/06/14 06:36:42 INFO YarnAllocator: Driver requested a total number of 15 executor(s). | |
| 17/06/14 06:36:42 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:36:42 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 06:36:42 INFO ExecutorAllocationManager: Requesting 14 new executors because tasks are backlogged (new desired total will be 15) | |
| 17/06/14 06:36:43 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037874 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 06:36:43 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 06:36:43 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:36:43 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 06:36:43 INFO YarnAllocator: Driver requested a total number of 16 executor(s). | |
| 17/06/14 06:36:43 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:36:43 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 06:36:43 INFO ExecutorAllocationManager: Requesting 1 new executor because tasks are backlogged (new desired total will be 16) | |
| 17/06/14 06:36:43 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022659 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:36:43 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:43 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:43 ERROR YarnClusterScheduler: Lost executor 37 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:43 WARN TaskSetManager: Lost task 26.0 in stage 6.0 (TID 992, ip-10-0-51-185.ec2.internal, executor 37): ExecutorLostFailure (executor 37 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:43 WARN TaskSetManager: Lost task 40.0 in stage 6.0 (TID 1006, ip-10-0-51-185.ec2.internal, executor 37): ExecutorLostFailure (executor 37 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:43 WARN TaskSetManager: Lost task 54.0 in stage 6.0 (TID 1020, ip-10-0-51-185.ec2.internal, executor 37): ExecutorLostFailure (executor 37 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:43 WARN TaskSetManager: Lost task 12.0 in stage 6.0 (TID 978, ip-10-0-51-185.ec2.internal, executor 37): ExecutorLostFailure (executor 37 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:43 INFO DAGScheduler: Executor lost: 37 (epoch 4) | |
| 17/06/14 06:36:43 INFO BlockManagerMasterEndpoint: Trying to remove executor 37 from BlockManagerMaster. | |
| 17/06/14 06:36:43 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(37, ip-10-0-51-185.ec2.internal, 33788, None) | |
| 17/06/14 06:36:43 INFO BlockManagerMaster: Removed 37 successfully in removeExecutor | |
| 17/06/14 06:36:43 INFO ExecutorAllocationManager: Existing executor 37 has been removed (new total is 13) | |
| 17/06/14 06:36:43 WARN ExecutorAllocationManager: Attempted to mark unknown executor 37 idle | |
| 17/06/14 06:36:43 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:36:43 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 06:36:44 WARN TransportChannelHandler: Exception in connection from /10.0.51.185:35570 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 06:36:44 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 37. | |
| 17/06/14 06:36:44 ERROR YarnClusterScheduler: Lost an executor 37 (already removed): Pending loss reason. | |
| 17/06/14 06:36:44 INFO BlockManagerMaster: Removal of executor 37 requested | |
| 17/06/14 06:36:44 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 37 | |
| 17/06/14 06:36:44 INFO BlockManagerMasterEndpoint: Trying to remove executor 37 from BlockManagerMaster. | |
| 17/06/14 06:36:44 INFO BlockManagerInfo: Removed rdd_10_89 on ip-10-0-50-96.ec2.internal:34711 in memory (size: 835.5 MB, free: 1210.3 MB) | |
| 17/06/14 06:36:44 INFO BlockManagerInfo: Removed rdd_10_126 on ip-10-0-51-185.ec2.internal:46252 in memory (size: 838.8 MB, free: 1150.8 MB) | |
| 17/06/14 06:36:44 INFO BlockManagerInfo: Removed rdd_10_65 on ip-10-0-49-229.ec2.internal:44599 in memory (size: 809.7 MB, free: 1186.2 MB) | |
| 17/06/14 06:36:44 INFO YarnAllocator: Driver requested a total number of 18 executor(s). | |
| 17/06/14 06:36:44 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:36:44 INFO YarnAllocator: Submitted 2 unlocalized container requests. | |
| 17/06/14 06:36:44 INFO ExecutorAllocationManager: Requesting 2 new executors because tasks are backlogged (new desired total will be 18) | |
| 17/06/14 06:36:44 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037877 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 06:36:44 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 06:36:44 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:36:44 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 06:36:45 INFO YarnAllocator: Driver requested a total number of 22 executor(s). | |
| 17/06/14 06:36:45 INFO YarnAllocator: Will request 4 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:36:45 INFO YarnAllocator: Submitted 4 unlocalized container requests. | |
| 17/06/14 06:36:45 INFO ExecutorAllocationManager: Requesting 4 new executors because tasks are backlogged (new desired total will be 22) | |
| 17/06/14 06:36:46 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:37630) with ID 56 | |
| 17/06/14 06:36:46 INFO TaskSetManager: Starting task 12.1 in stage 6.0 (TID 1022, ip-10-0-50-96.ec2.internal, executor 56, partition 12, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:46 INFO TaskSetManager: Starting task 54.1 in stage 6.0 (TID 1023, ip-10-0-50-96.ec2.internal, executor 56, partition 54, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:46 INFO TaskSetManager: Starting task 40.1 in stage 6.0 (TID 1024, ip-10-0-50-96.ec2.internal, executor 56, partition 40, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:46 INFO TaskSetManager: Starting task 26.1 in stage 6.0 (TID 1025, ip-10-0-50-96.ec2.internal, executor 56, partition 26, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:46 INFO ExecutorAllocationManager: New executor 56 has registered (new total is 14) | |
| 17/06/14 06:36:46 INFO BlockManagerInfo: Removed rdd_10_118 on ip-10-0-49-229.ec2.internal:44599 in memory (size: 801.1 MB, free: 1987.2 MB) | |
| 17/06/14 06:36:46 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:42816 with 2.8 GB RAM, BlockManagerId(56, ip-10-0-50-96.ec2.internal, 42816, None) | |
| 17/06/14 06:36:46 INFO BlockManagerInfo: Removed rdd_10_120 on ip-10-0-50-96.ec2.internal:40866 in memory (size: 855.3 MB, free: 2006.0 MB) | |
| 17/06/14 06:36:46 INFO BlockManagerInfo: Removed rdd_10_102 on ip-10-0-50-96.ec2.internal:34711 in memory (size: 815.8 MB, free: 2026.1 MB) | |
| 17/06/14 06:36:46 INFO BlockManagerInfo: Removed rdd_10_58 on ip-10-0-49-157.ec2.internal:44407 in memory (size: 810.5 MB, free: 2001.0 MB) | |
| 17/06/14 06:36:46 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-50-96.ec2.internal:42816 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:36:46 INFO YarnAllocator: Driver requested a total number of 30 executor(s). | |
| 17/06/14 06:36:46 INFO YarnAllocator: Will request 8 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:36:46 INFO YarnAllocator: Submitted 8 unlocalized container requests. | |
| 17/06/14 06:36:46 INFO ExecutorAllocationManager: Requesting 8 new executors because tasks are backlogged (new desired total will be 30) | |
| 17/06/14 06:36:46 INFO BlockManagerInfo: Removed rdd_10_71 on ip-10-0-49-157.ec2.internal:45984 in memory (size: 853.6 MB, free: 1971.0 MB) | |
| 17/06/14 06:36:47 INFO BlockManagerInfo: Removed rdd_11_132 on ip-10-0-51-185.ec2.internal:46252 in memory (size: 816.0 MB, free: 1966.8 MB) | |
| 17/06/14 06:36:47 INFO BlockManagerInfo: Removed rdd_10_121 on ip-10-0-51-185.ec2.internal:34029 in memory (size: 841.3 MB, free: 2000.1 MB) | |
| 17/06/14 06:36:47 INFO BlockManagerInfo: Removed rdd_10_106 on ip-10-0-49-157.ec2.internal:39668 in memory (size: 843.4 MB, free: 2041.4 MB) | |
| 17/06/14 06:36:47 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.50.96:37630 | |
| 17/06/14 06:36:47 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.50.96:37630 | |
| 17/06/14 06:36:47 INFO YarnAllocator: Driver requested a total number of 34 executor(s). | |
| 17/06/14 06:36:47 INFO YarnAllocator: Will request 4 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:36:47 INFO YarnAllocator: Submitted 4 unlocalized container requests. | |
| 17/06/14 06:36:47 INFO ExecutorAllocationManager: Requesting 4 new executors because tasks are backlogged (new desired total will be 34) | |
| 17/06/14 06:36:47 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:36336) with ID 57 | |
| 17/06/14 06:36:47 INFO TaskSetManager: Starting task 56.0 in stage 6.0 (TID 1026, ip-10-0-51-185.ec2.internal, executor 57, partition 56, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:47 INFO TaskSetManager: Starting task 57.0 in stage 6.0 (TID 1027, ip-10-0-51-185.ec2.internal, executor 57, partition 57, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:47 INFO TaskSetManager: Starting task 58.0 in stage 6.0 (TID 1028, ip-10-0-51-185.ec2.internal, executor 57, partition 58, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:47 INFO TaskSetManager: Starting task 59.0 in stage 6.0 (TID 1029, ip-10-0-51-185.ec2.internal, executor 57, partition 59, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:47 INFO ExecutorAllocationManager: New executor 57 has registered (new total is 15) | |
| 17/06/14 06:36:47 INFO BlockManagerInfo: Removed rdd_10_90 on ip-10-0-49-157.ec2.internal:42968 in memory (size: 829.9 MB, free: 1986.3 MB) | |
| 17/06/14 06:36:48 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:38110 with 2.8 GB RAM, BlockManagerId(57, ip-10-0-51-185.ec2.internal, 38110, None) | |
| 17/06/14 06:36:48 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-51-185.ec2.internal:38110 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:36:49 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.51.185:36336 | |
| 17/06/14 06:36:49 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.51.185:36336 | |
| 17/06/14 06:36:55 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 53. | |
| 17/06/14 06:36:55 INFO DAGScheduler: Executor lost: 53 (epoch 4) | |
| 17/06/14 06:36:55 INFO BlockManagerMasterEndpoint: Trying to remove executor 53 from BlockManagerMaster. | |
| 17/06/14 06:36:55 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(53, ip-10-0-49-229.ec2.internal, 44599, None) | |
| 17/06/14 06:36:55 INFO BlockManagerMaster: Removed 53 successfully in removeExecutor | |
| 17/06/14 06:36:55 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037876 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 06:36:55 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 06:36:55 INFO YarnAllocator: Completed container container_1497410974650_0004_01_029923 on host: ip-10-0-49-229.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:36:55 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:55 ERROR YarnClusterScheduler: Lost executor 53 on ip-10-0-49-229.ec2.internal: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:55 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:36:55 WARN TaskSetManager: Lost task 53.0 in stage 6.0 (TID 1019, ip-10-0-49-229.ec2.internal, executor 53): ExecutorLostFailure (executor 53 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:55 WARN TaskSetManager: Lost task 11.0 in stage 6.0 (TID 977, ip-10-0-49-229.ec2.internal, executor 53): ExecutorLostFailure (executor 53 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:55 WARN TaskSetManager: Lost task 25.0 in stage 6.0 (TID 991, ip-10-0-49-229.ec2.internal, executor 53): ExecutorLostFailure (executor 53 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:55 WARN TaskSetManager: Lost task 39.0 in stage 6.0 (TID 1005, ip-10-0-49-229.ec2.internal, executor 53): ExecutorLostFailure (executor 53 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:55 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:55 INFO BlockManagerMaster: Removal of executor 53 requested | |
| 17/06/14 06:36:55 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 53 | |
| 17/06/14 06:36:55 INFO BlockManagerMasterEndpoint: Trying to remove executor 53 from BlockManagerMaster. | |
| 17/06/14 06:36:55 INFO ExecutorAllocationManager: Existing executor 53 has been removed (new total is 14) | |
| 17/06/14 06:36:55 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 06:36:55 WARN ExecutorAllocationManager: Attempted to mark unknown executor 53 idle | |
| 17/06/14 06:36:55 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:36:55 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 06:36:58 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:51168) with ID 58 | |
| 17/06/14 06:36:58 INFO TaskSetManager: Starting task 39.1 in stage 6.0 (TID 1030, ip-10-0-49-229.ec2.internal, executor 58, partition 39, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:58 INFO TaskSetManager: Starting task 25.1 in stage 6.0 (TID 1031, ip-10-0-49-229.ec2.internal, executor 58, partition 25, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:58 INFO TaskSetManager: Starting task 11.1 in stage 6.0 (TID 1032, ip-10-0-49-229.ec2.internal, executor 58, partition 11, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:58 INFO TaskSetManager: Starting task 53.1 in stage 6.0 (TID 1033, ip-10-0-49-229.ec2.internal, executor 58, partition 53, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:36:58 INFO ExecutorAllocationManager: New executor 58 has registered (new total is 15) | |
| 17/06/14 06:36:58 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:35857 with 2.8 GB RAM, BlockManagerId(58, ip-10-0-49-229.ec2.internal, 35857, None) | |
| 17/06/14 06:36:58 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-229.ec2.internal:35857 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:36:58 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 49. | |
| 17/06/14 06:36:58 INFO DAGScheduler: Executor lost: 49 (epoch 4) | |
| 17/06/14 06:36:58 INFO BlockManagerMasterEndpoint: Trying to remove executor 49 from BlockManagerMaster. | |
| 17/06/14 06:36:58 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(49, ip-10-0-49-229.ec2.internal, 45654, None) | |
| 17/06/14 06:36:58 INFO BlockManagerMaster: Removed 49 successfully in removeExecutor | |
| 17/06/14 06:36:58 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037909 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 06:36:58 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 06:36:58 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022687 on host: ip-10-0-49-229.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:36:58 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 ERROR YarnClusterScheduler: Lost executor 49 on ip-10-0-49-229.ec2.internal: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:36:58 WARN TaskSetManager: Lost task 35.0 in stage 6.0 (TID 1001, ip-10-0-49-229.ec2.internal, executor 49): ExecutorLostFailure (executor 49 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 WARN TaskSetManager: Lost task 49.0 in stage 6.0 (TID 1015, ip-10-0-49-229.ec2.internal, executor 49): ExecutorLostFailure (executor 49 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 WARN TaskSetManager: Lost task 7.0 in stage 6.0 (TID 973, ip-10-0-49-229.ec2.internal, executor 49): ExecutorLostFailure (executor 49 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 WARN TaskSetManager: Lost task 21.0 in stage 6.0 (TID 987, ip-10-0-49-229.ec2.internal, executor 49): ExecutorLostFailure (executor 49 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 INFO BlockManagerMaster: Removal of executor 49 requested | |
| 17/06/14 06:36:58 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 49 | |
| 17/06/14 06:36:58 INFO BlockManagerMasterEndpoint: Trying to remove executor 49 from BlockManagerMaster. | |
| 17/06/14 06:36:58 INFO ExecutorAllocationManager: Existing executor 49 has been removed (new total is 14) | |
| 17/06/14 06:36:58 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 06:36:58 WARN ExecutorAllocationManager: Attempted to mark unknown executor 49 idle | |
| 17/06/14 06:36:58 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:36:58 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 06:36:58 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037923 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 06:36:58 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037924 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 06:36:58 INFO YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them. | |
| 17/06/14 06:36:58 INFO YarnAllocator: Completed container container_1497410974650_0004_01_030366 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:36:58 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:36:58 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022692 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:36:58 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:36:58 ERROR YarnClusterScheduler: Lost executor 54 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 WARN TaskSetManager: Lost task 20.0 in stage 6.0 (TID 986, ip-10-0-51-185.ec2.internal, executor 54): ExecutorLostFailure (executor 54 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 WARN TaskSetManager: Lost task 34.0 in stage 6.0 (TID 1000, ip-10-0-51-185.ec2.internal, executor 54): ExecutorLostFailure (executor 54 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 WARN TaskSetManager: Lost task 48.0 in stage 6.0 (TID 1014, ip-10-0-51-185.ec2.internal, executor 54): ExecutorLostFailure (executor 54 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 WARN TaskSetManager: Lost task 6.0 in stage 6.0 (TID 972, ip-10-0-51-185.ec2.internal, executor 54): ExecutorLostFailure (executor 54 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 ERROR YarnClusterScheduler: Lost executor 48 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 WARN TaskSetManager: Lost task 38.0 in stage 6.0 (TID 1004, ip-10-0-51-185.ec2.internal, executor 48): ExecutorLostFailure (executor 48 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 WARN TaskSetManager: Lost task 52.0 in stage 6.0 (TID 1018, ip-10-0-51-185.ec2.internal, executor 48): ExecutorLostFailure (executor 48 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 WARN TaskSetManager: Lost task 10.0 in stage 6.0 (TID 976, ip-10-0-51-185.ec2.internal, executor 48): ExecutorLostFailure (executor 48 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 WARN TaskSetManager: Lost task 24.0 in stage 6.0 (TID 990, ip-10-0-51-185.ec2.internal, executor 48): ExecutorLostFailure (executor 48 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:58 INFO DAGScheduler: Executor lost: 54 (epoch 4) | |
| 17/06/14 06:36:58 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 06:36:58 INFO BlockManagerMasterEndpoint: Trying to remove executor 54 from BlockManagerMaster. | |
| 17/06/14 06:36:58 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(54, ip-10-0-51-185.ec2.internal, 45572, None) | |
| 17/06/14 06:36:58 INFO BlockManagerMaster: Removed 54 successfully in removeExecutor | |
| 17/06/14 06:36:58 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 06:36:58 INFO ExecutorAllocationManager: Existing executor 54 has been removed (new total is 13) | |
| 17/06/14 06:36:58 INFO DAGScheduler: Executor lost: 48 (epoch 4) | |
| 17/06/14 06:36:58 INFO BlockManagerMasterEndpoint: Trying to remove executor 48 from BlockManagerMaster. | |
| 17/06/14 06:36:58 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(48, ip-10-0-51-185.ec2.internal, 34029, None) | |
| 17/06/14 06:36:58 INFO BlockManagerMaster: Removed 48 successfully in removeExecutor | |
| 17/06/14 06:36:58 INFO ExecutorAllocationManager: Existing executor 48 has been removed (new total is 12) | |
| 17/06/14 06:36:58 WARN ExecutorAllocationManager: Attempted to mark unknown executor 54 idle | |
| 17/06/14 06:36:58 WARN ExecutorAllocationManager: Attempted to mark unknown executor 48 idle | |
| 17/06/14 06:36:59 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.229:51168 | |
| 17/06/14 06:36:59 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.229:51168 | |
| 17/06/14 06:36:59 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:36:59 INFO YarnAllocator: Submitted 2 unlocalized container requests. | |
| 17/06/14 06:36:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 48. | |
| 17/06/14 06:36:59 ERROR YarnClusterScheduler: Lost an executor 48 (already removed): Pending loss reason. | |
| 17/06/14 06:36:59 INFO BlockManagerMaster: Removal of executor 48 requested | |
| 17/06/14 06:36:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 48 | |
| 17/06/14 06:36:59 INFO BlockManagerMasterEndpoint: Trying to remove executor 48 from BlockManagerMaster. | |
| 17/06/14 06:36:59 WARN TransportChannelHandler: Exception in connection from /10.0.51.185:35966 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 06:36:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 54. | |
| 17/06/14 06:36:59 ERROR YarnClusterScheduler: Lost an executor 54 (already removed): Pending loss reason. | |
| 17/06/14 06:36:59 INFO BlockManagerMaster: Removal of executor 54 requested | |
| 17/06/14 06:36:59 INFO BlockManagerMasterEndpoint: Trying to remove executor 54 from BlockManagerMaster. | |
| 17/06/14 06:36:59 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 54 | |
| 17/06/14 06:36:59 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037926 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 06:36:59 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 06:36:59 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022614 on host: ip-10-0-49-157.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:36:59 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:59 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:59 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:36:59 ERROR YarnClusterScheduler: Lost executor 36 on ip-10-0-49-157.ec2.internal: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:59 WARN TaskSetManager: Lost task 50.0 in stage 6.0 (TID 1016, ip-10-0-49-157.ec2.internal, executor 36): ExecutorLostFailure (executor 36 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:59 WARN TaskSetManager: Lost task 8.0 in stage 6.0 (TID 974, ip-10-0-49-157.ec2.internal, executor 36): ExecutorLostFailure (executor 36 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:59 WARN TaskSetManager: Lost task 22.0 in stage 6.0 (TID 988, ip-10-0-49-157.ec2.internal, executor 36): ExecutorLostFailure (executor 36 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:59 WARN TaskSetManager: Lost task 36.0 in stage 6.0 (TID 1002, ip-10-0-49-157.ec2.internal, executor 36): ExecutorLostFailure (executor 36 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:36:59 INFO DAGScheduler: Executor lost: 36 (epoch 4) | |
| 17/06/14 06:36:59 INFO BlockManagerMasterEndpoint: Trying to remove executor 36 from BlockManagerMaster. | |
| 17/06/14 06:36:59 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(36, ip-10-0-49-157.ec2.internal, 39668, None) | |
| 17/06/14 06:36:59 INFO BlockManagerMaster: Removed 36 successfully in removeExecutor | |
| 17/06/14 06:36:59 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 06:36:59 INFO ExecutorAllocationManager: Existing executor 36 has been removed (new total is 11) | |
| 17/06/14 06:36:59 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:36:59 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 06:37:00 WARN TransportChannelHandler: Exception in connection from /10.0.49.157:45932 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 06:37:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 36. | |
| 17/06/14 06:37:00 ERROR YarnClusterScheduler: Lost an executor 36 (already removed): Pending loss reason. | |
| 17/06/14 06:37:00 INFO BlockManagerMaster: Removal of executor 36 requested | |
| 17/06/14 06:37:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 36 | |
| 17/06/14 06:37:00 INFO BlockManagerMasterEndpoint: Trying to remove executor 36 from BlockManagerMaster. | |
| 17/06/14 06:37:01 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:51194) with ID 59 | |
| 17/06/14 06:37:01 INFO TaskSetManager: Starting task 36.1 in stage 6.0 (TID 1034, ip-10-0-49-229.ec2.internal, executor 59, partition 36, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:01 INFO TaskSetManager: Starting task 22.1 in stage 6.0 (TID 1035, ip-10-0-49-229.ec2.internal, executor 59, partition 22, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:01 INFO TaskSetManager: Starting task 8.1 in stage 6.0 (TID 1036, ip-10-0-49-229.ec2.internal, executor 59, partition 8, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:01 INFO TaskSetManager: Starting task 50.1 in stage 6.0 (TID 1037, ip-10-0-49-229.ec2.internal, executor 59, partition 50, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:01 INFO ExecutorAllocationManager: New executor 59 has registered (new total is 12) | |
| 17/06/14 06:37:01 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:38006 with 2.8 GB RAM, BlockManagerId(59, ip-10-0-49-229.ec2.internal, 38006, None) | |
| 17/06/14 06:37:01 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037919 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 06:37:01 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 06:37:01 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022693 on host: ip-10-0-49-229.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:37:01 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:01 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:01 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:37:01 ERROR YarnClusterScheduler: Lost executor 52 on ip-10-0-49-229.ec2.internal: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:01 WARN TaskSetManager: Lost task 14.0 in stage 6.0 (TID 980, ip-10-0-49-229.ec2.internal, executor 52): ExecutorLostFailure (executor 52 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:01 WARN TaskSetManager: Lost task 28.0 in stage 6.0 (TID 994, ip-10-0-49-229.ec2.internal, executor 52): ExecutorLostFailure (executor 52 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:01 WARN TaskSetManager: Lost task 42.0 in stage 6.0 (TID 1008, ip-10-0-49-229.ec2.internal, executor 52): ExecutorLostFailure (executor 52 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:01 WARN TaskSetManager: Lost task 0.0 in stage 6.0 (TID 966, ip-10-0-49-229.ec2.internal, executor 52): ExecutorLostFailure (executor 52 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:01 INFO DAGScheduler: Executor lost: 52 (epoch 4) | |
| 17/06/14 06:37:01 INFO BlockManagerMasterEndpoint: Trying to remove executor 52 from BlockManagerMaster. | |
| 17/06/14 06:37:01 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(52, ip-10-0-49-229.ec2.internal, 44395, None) | |
| 17/06/14 06:37:01 INFO BlockManagerMaster: Removed 52 successfully in removeExecutor | |
| 17/06/14 06:37:01 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 06:37:01 INFO ExecutorAllocationManager: Existing executor 52 has been removed (new total is 11) | |
| 17/06/14 06:37:01 WARN ExecutorAllocationManager: Attempted to mark unknown executor 52 idle | |
| 17/06/14 06:37:01 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-229.ec2.internal:38006 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:01 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 52. | |
| 17/06/14 06:37:01 ERROR YarnClusterScheduler: Lost an executor 52 (already removed): Pending loss reason. | |
| 17/06/14 06:37:01 INFO BlockManagerMaster: Removal of executor 52 requested | |
| 17/06/14 06:37:01 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 52 | |
| 17/06/14 06:37:01 INFO BlockManagerMasterEndpoint: Trying to remove executor 52 from BlockManagerMaster. | |
| 17/06/14 06:37:01 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:37:01 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 06:37:02 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:36370) with ID 60 | |
| 17/06/14 06:37:02 INFO TaskSetManager: Starting task 0.1 in stage 6.0 (TID 1038, ip-10-0-51-185.ec2.internal, executor 60, partition 0, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:02 INFO TaskSetManager: Starting task 42.1 in stage 6.0 (TID 1039, ip-10-0-51-185.ec2.internal, executor 60, partition 42, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:02 INFO TaskSetManager: Starting task 28.1 in stage 6.0 (TID 1040, ip-10-0-51-185.ec2.internal, executor 60, partition 28, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:02 INFO TaskSetManager: Starting task 14.1 in stage 6.0 (TID 1041, ip-10-0-51-185.ec2.internal, executor 60, partition 14, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:02 INFO ExecutorAllocationManager: New executor 60 has registered (new total is 12) | |
| 17/06/14 06:37:02 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.229:51194 | |
| 17/06/14 06:37:02 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:36372) with ID 61 | |
| 17/06/14 06:37:02 INFO TaskSetManager: Starting task 24.1 in stage 6.0 (TID 1042, ip-10-0-51-185.ec2.internal, executor 61, partition 24, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:02 INFO TaskSetManager: Starting task 10.1 in stage 6.0 (TID 1043, ip-10-0-51-185.ec2.internal, executor 61, partition 10, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:02 INFO TaskSetManager: Starting task 52.1 in stage 6.0 (TID 1044, ip-10-0-51-185.ec2.internal, executor 61, partition 52, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:02 INFO TaskSetManager: Starting task 38.1 in stage 6.0 (TID 1045, ip-10-0-51-185.ec2.internal, executor 61, partition 38, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:02 INFO ExecutorAllocationManager: New executor 61 has registered (new total is 13) | |
| 17/06/14 06:37:02 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.229:51194 | |
| 17/06/14 06:37:02 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:33361 with 2.8 GB RAM, BlockManagerId(60, ip-10-0-51-185.ec2.internal, 33361, None) | |
| 17/06/14 06:37:02 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:43408 with 2.8 GB RAM, BlockManagerId(61, ip-10-0-51-185.ec2.internal, 43408, None) | |
| 17/06/14 06:37:02 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037935 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 06:37:02 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037936 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 06:37:02 INFO YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them. | |
| 17/06/14 06:37:02 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022667 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:37:02 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 INFO YarnAllocator: Completed container container_1497410974650_0004_01_026409 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:37:02 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:37:02 ERROR YarnClusterScheduler: Lost executor 41 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 WARN TaskSetManager: Lost task 44.0 in stage 6.0 (TID 1010, ip-10-0-50-96.ec2.internal, executor 41): ExecutorLostFailure (executor 41 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 WARN TaskSetManager: Lost task 2.0 in stage 6.0 (TID 968, ip-10-0-50-96.ec2.internal, executor 41): ExecutorLostFailure (executor 41 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 WARN TaskSetManager: Lost task 16.0 in stage 6.0 (TID 982, ip-10-0-50-96.ec2.internal, executor 41): ExecutorLostFailure (executor 41 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 WARN TaskSetManager: Lost task 30.0 in stage 6.0 (TID 996, ip-10-0-50-96.ec2.internal, executor 41): ExecutorLostFailure (executor 41 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:37:02 ERROR YarnClusterScheduler: Lost executor 50 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 WARN TaskSetManager: Lost task 32.0 in stage 6.0 (TID 998, ip-10-0-50-96.ec2.internal, executor 50): ExecutorLostFailure (executor 50 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 WARN TaskSetManager: Lost task 46.0 in stage 6.0 (TID 1012, ip-10-0-50-96.ec2.internal, executor 50): ExecutorLostFailure (executor 50 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 WARN TaskSetManager: Lost task 4.0 in stage 6.0 (TID 970, ip-10-0-50-96.ec2.internal, executor 50): ExecutorLostFailure (executor 50 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 WARN TaskSetManager: Lost task 18.0 in stage 6.0 (TID 984, ip-10-0-50-96.ec2.internal, executor 50): ExecutorLostFailure (executor 50 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 INFO DAGScheduler: Executor lost: 41 (epoch 4) | |
| 17/06/14 06:37:02 INFO BlockManagerMasterEndpoint: Trying to remove executor 41 from BlockManagerMaster. | |
| 17/06/14 06:37:02 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(41, ip-10-0-50-96.ec2.internal, 37989, None) | |
| 17/06/14 06:37:02 INFO BlockManagerMaster: Removed 41 successfully in removeExecutor | |
| 17/06/14 06:37:02 INFO DAGScheduler: Executor lost: 50 (epoch 4) | |
| 17/06/14 06:37:02 INFO BlockManagerMasterEndpoint: Trying to remove executor 50 from BlockManagerMaster. | |
| 17/06/14 06:37:02 INFO ExecutorAllocationManager: Existing executor 41 has been removed (new total is 12) | |
| 17/06/14 06:37:02 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(50, ip-10-0-50-96.ec2.internal, 34711, None) | |
| 17/06/14 06:37:02 INFO BlockManagerMaster: Removed 50 successfully in removeExecutor | |
| 17/06/14 06:37:02 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 06:37:02 INFO ExecutorAllocationManager: Existing executor 50 has been removed (new total is 11) | |
| 17/06/14 06:37:02 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 06:37:02 WARN ExecutorAllocationManager: Attempted to mark unknown executor 41 idle | |
| 17/06/14 06:37:02 WARN ExecutorAllocationManager: Attempted to mark unknown executor 50 idle | |
| 17/06/14 06:37:02 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-51-185.ec2.internal:33361 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:02 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-51-185.ec2.internal:43408 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:02 WARN TransportChannelHandler: Exception in connection from /10.0.50.96:36892 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 06:37:02 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 41. | |
| 17/06/14 06:37:02 ERROR YarnClusterScheduler: Lost an executor 41 (already removed): Pending loss reason. | |
| 17/06/14 06:37:02 INFO BlockManagerMaster: Removal of executor 41 requested | |
| 17/06/14 06:37:02 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 41 | |
| 17/06/14 06:37:02 INFO BlockManagerMasterEndpoint: Trying to remove executor 41 from BlockManagerMaster. | |
| 17/06/14 06:37:02 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:37:02 INFO YarnAllocator: Submitted 2 unlocalized container requests. | |
| 17/06/14 06:37:02 WARN TransportChannelHandler: Exception in connection from /10.0.50.96:37112 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 06:37:02 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 50. | |
| 17/06/14 06:37:02 ERROR YarnClusterScheduler: Lost an executor 50 (already removed): Pending loss reason. | |
| 17/06/14 06:37:02 INFO BlockManagerMaster: Removal of executor 50 requested | |
| 17/06/14 06:37:02 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 50 | |
| 17/06/14 06:37:02 INFO BlockManagerMasterEndpoint: Trying to remove executor 50 from BlockManagerMaster. | |
| 17/06/14 06:37:02 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037938 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 06:37:02 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 06:37:02 INFO YarnAllocator: Completed container container_1497410974650_0004_01_026437 on host: ip-10-0-49-157.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:37:02 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 ERROR YarnClusterScheduler: Lost executor 51 on ip-10-0-49-157.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:37:02 WARN TaskSetManager: Lost task 29.0 in stage 6.0 (TID 995, ip-10-0-49-157.ec2.internal, executor 51): ExecutorLostFailure (executor 51 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 WARN TaskSetManager: Lost task 43.0 in stage 6.0 (TID 1009, ip-10-0-49-157.ec2.internal, executor 51): ExecutorLostFailure (executor 51 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 WARN TaskSetManager: Lost task 1.0 in stage 6.0 (TID 967, ip-10-0-49-157.ec2.internal, executor 51): ExecutorLostFailure (executor 51 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 WARN TaskSetManager: Lost task 15.0 in stage 6.0 (TID 981, ip-10-0-49-157.ec2.internal, executor 51): ExecutorLostFailure (executor 51 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:02 INFO DAGScheduler: Executor lost: 51 (epoch 4) | |
| 17/06/14 06:37:02 INFO BlockManagerMasterEndpoint: Trying to remove executor 51 from BlockManagerMaster. | |
| 17/06/14 06:37:02 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(51, ip-10-0-49-157.ec2.internal, 44407, None) | |
| 17/06/14 06:37:02 INFO BlockManagerMaster: Removed 51 successfully in removeExecutor | |
| 17/06/14 06:37:02 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 06:37:02 INFO ExecutorAllocationManager: Existing executor 51 has been removed (new total is 10) | |
| 17/06/14 06:37:02 WARN ExecutorAllocationManager: Attempted to mark unknown executor 51 idle | |
| 17/06/14 06:37:03 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:46710) with ID 62 | |
| 17/06/14 06:37:03 INFO TaskSetManager: Starting task 15.1 in stage 6.0 (TID 1046, ip-10-0-49-157.ec2.internal, executor 62, partition 15, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:03 INFO TaskSetManager: Starting task 1.1 in stage 6.0 (TID 1047, ip-10-0-49-157.ec2.internal, executor 62, partition 1, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:03 INFO TaskSetManager: Starting task 43.1 in stage 6.0 (TID 1048, ip-10-0-49-157.ec2.internal, executor 62, partition 43, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:03 INFO TaskSetManager: Starting task 29.1 in stage 6.0 (TID 1049, ip-10-0-49-157.ec2.internal, executor 62, partition 29, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:03 INFO ExecutorAllocationManager: New executor 62 has registered (new total is 11) | |
| 17/06/14 06:37:03 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:42597 with 2.8 GB RAM, BlockManagerId(62, ip-10-0-49-157.ec2.internal, 42597, None) | |
| 17/06/14 06:37:03 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:37:03 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 06:37:03 WARN TransportChannelHandler: Exception in connection from /10.0.49.157:46182 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 06:37:03 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 51. | |
| 17/06/14 06:37:03 ERROR YarnClusterScheduler: Lost an executor 51 (already removed): Pending loss reason. | |
| 17/06/14 06:37:03 INFO BlockManagerMaster: Removal of executor 51 requested | |
| 17/06/14 06:37:03 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 51 | |
| 17/06/14 06:37:03 INFO BlockManagerMasterEndpoint: Trying to remove executor 51 from BlockManagerMaster. | |
| 17/06/14 06:37:03 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 40. | |
| 17/06/14 06:37:03 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 44. | |
| 17/06/14 06:37:03 INFO DAGScheduler: Executor lost: 40 (epoch 4) | |
| 17/06/14 06:37:03 INFO BlockManagerMasterEndpoint: Trying to remove executor 40 from BlockManagerMaster. | |
| 17/06/14 06:37:03 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(40, ip-10-0-49-157.ec2.internal, 42968, None) | |
| 17/06/14 06:37:03 INFO BlockManagerMaster: Removed 40 successfully in removeExecutor | |
| 17/06/14 06:37:03 INFO DAGScheduler: Executor lost: 44 (epoch 4) | |
| 17/06/14 06:37:03 INFO BlockManagerMasterEndpoint: Trying to remove executor 44 from BlockManagerMaster. | |
| 17/06/14 06:37:03 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(44, ip-10-0-49-157.ec2.internal, 45984, None) | |
| 17/06/14 06:37:03 INFO BlockManagerMaster: Removed 44 successfully in removeExecutor | |
| 17/06/14 06:37:03 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.51.185:36370 | |
| 17/06/14 06:37:03 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.51.185:36372 | |
| 17/06/14 06:37:03 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.51.185:36372 | |
| 17/06/14 06:37:03 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.51.185:36370 | |
| 17/06/14 06:37:04 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:51228) with ID 63 | |
| 17/06/14 06:37:04 INFO TaskSetManager: Starting task 18.1 in stage 6.0 (TID 1050, ip-10-0-49-229.ec2.internal, executor 63, partition 18, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:04 INFO TaskSetManager: Starting task 4.1 in stage 6.0 (TID 1051, ip-10-0-49-229.ec2.internal, executor 63, partition 4, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:04 INFO TaskSetManager: Starting task 46.1 in stage 6.0 (TID 1052, ip-10-0-49-229.ec2.internal, executor 63, partition 46, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:04 INFO ExecutorAllocationManager: New executor 63 has registered (new total is 12) | |
| 17/06/14 06:37:04 INFO TaskSetManager: Starting task 32.1 in stage 6.0 (TID 1053, ip-10-0-49-229.ec2.internal, executor 63, partition 32, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:04 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:45131 with 2.8 GB RAM, BlockManagerId(63, ip-10-0-49-229.ec2.internal, 45131, None) | |
| 17/06/14 06:37:04 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037941 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 06:37:04 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 06:37:04 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022666 on host: ip-10-0-49-157.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:37:04 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:04 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022677 on host: ip-10-0-49-157.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:37:04 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:04 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:04 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:37:04 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:04 ERROR YarnClusterScheduler: Lost executor 40 on ip-10-0-49-157.ec2.internal: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:04 WARN TaskSetManager: Lost task 23.0 in stage 6.0 (TID 989, ip-10-0-49-157.ec2.internal, executor 40): ExecutorLostFailure (executor 40 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:04 WARN TaskSetManager: Lost task 37.0 in stage 6.0 (TID 1003, ip-10-0-49-157.ec2.internal, executor 40): ExecutorLostFailure (executor 40 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:04 WARN TaskSetManager: Lost task 51.0 in stage 6.0 (TID 1017, ip-10-0-49-157.ec2.internal, executor 40): ExecutorLostFailure (executor 40 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:04 WARN TaskSetManager: Lost task 9.0 in stage 6.0 (TID 975, ip-10-0-49-157.ec2.internal, executor 40): ExecutorLostFailure (executor 40 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:04 ERROR YarnClusterScheduler: Lost executor 44 on ip-10-0-49-157.ec2.internal: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:04 WARN TaskSetManager: Lost task 47.0 in stage 6.0 (TID 1013, ip-10-0-49-157.ec2.internal, executor 44): ExecutorLostFailure (executor 44 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:04 WARN TaskSetManager: Lost task 5.0 in stage 6.0 (TID 971, ip-10-0-49-157.ec2.internal, executor 44): ExecutorLostFailure (executor 44 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:04 WARN TaskSetManager: Lost task 19.0 in stage 6.0 (TID 985, ip-10-0-49-157.ec2.internal, executor 44): ExecutorLostFailure (executor 44 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:04 WARN TaskSetManager: Lost task 33.0 in stage 6.0 (TID 999, ip-10-0-49-157.ec2.internal, executor 44): ExecutorLostFailure (executor 44 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:04 INFO BlockManagerMaster: Removal of executor 40 requested | |
| 17/06/14 06:37:04 INFO BlockManagerMasterEndpoint: Trying to remove executor 40 from BlockManagerMaster. | |
| 17/06/14 06:37:04 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 40 | |
| 17/06/14 06:37:04 INFO BlockManagerMaster: Removal of executor 44 requested | |
| 17/06/14 06:37:04 INFO BlockManagerMasterEndpoint: Trying to remove executor 44 from BlockManagerMaster. | |
| 17/06/14 06:37:04 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 44 | |
| 17/06/14 06:37:04 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 06:37:04 INFO ExecutorAllocationManager: Existing executor 40 has been removed (new total is 11) | |
| 17/06/14 06:37:04 INFO ExecutorAllocationManager: Existing executor 44 has been removed (new total is 10) | |
| 17/06/14 06:37:04 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-229.ec2.internal:45131 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:04 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.229:51228 | |
| 17/06/14 06:37:04 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.229:51228 | |
| 17/06/14 06:37:04 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:37:04 INFO YarnAllocator: Submitted 2 unlocalized container requests. | |
| 17/06/14 06:37:04 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:46718) with ID 66 | |
| 17/06/14 06:37:04 INFO TaskSetManager: Starting task 33.1 in stage 6.0 (TID 1054, ip-10-0-49-157.ec2.internal, executor 66, partition 33, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:04 INFO TaskSetManager: Starting task 19.1 in stage 6.0 (TID 1055, ip-10-0-49-157.ec2.internal, executor 66, partition 19, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:04 INFO TaskSetManager: Starting task 5.1 in stage 6.0 (TID 1056, ip-10-0-49-157.ec2.internal, executor 66, partition 5, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:04 INFO TaskSetManager: Starting task 47.1 in stage 6.0 (TID 1057, ip-10-0-49-157.ec2.internal, executor 66, partition 47, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:04 INFO ExecutorAllocationManager: New executor 66 has registered (new total is 11) | |
| 17/06/14 06:37:04 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037944 on host ip-10-0-49-157.ec2.internal | |
| 17/06/14 06:37:04 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 06:37:04 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:37:04 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-157.ec2.internal:8041 | |
| 17/06/14 06:37:05 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:40656 with 2.8 GB RAM, BlockManagerId(66, ip-10-0-49-157.ec2.internal, 40656, None) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Starting task 9.1 in stage 6.0 (TID 1058, ip-10-0-51-185.ec2.internal, executor 57, partition 9, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Finished task 57.0 in stage 6.0 (TID 1027) in 17456 ms on ip-10-0-51-185.ec2.internal (executor 57) (1/135) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Starting task 51.1 in stage 6.0 (TID 1059, ip-10-0-50-96.ec2.internal, executor 56, partition 51, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Finished task 12.1 in stage 6.0 (TID 1022) in 19240 ms on ip-10-0-50-96.ec2.internal (executor 56) (2/135) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Starting task 37.1 in stage 6.0 (TID 1060, ip-10-0-50-96.ec2.internal, executor 56, partition 37, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Finished task 40.1 in stage 6.0 (TID 1024) in 19240 ms on ip-10-0-50-96.ec2.internal (executor 56) (3/135) | |
| 17/06/14 06:37:05 INFO YarnAllocator: Driver requested a total number of 33 executor(s). | |
| 17/06/14 06:37:05 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 33 executors. | |
| 17/06/14 06:37:05 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037946 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 06:37:05 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037947 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 06:37:05 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:37:05 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 06:37:05 INFO YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them. | |
| 17/06/14 06:37:05 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022685 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:37:05 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:05 INFO YarnAllocator: Completed container container_1497410974650_0004_01_022665 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:37:05 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.9 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:05 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:37:05 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:05 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.9 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:05 ERROR YarnClusterScheduler: Lost executor 45 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:05 WARN TaskSetManager: Lost task 41.0 in stage 6.0 (TID 1007, ip-10-0-50-96.ec2.internal, executor 45): ExecutorLostFailure (executor 45 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:05 WARN TaskSetManager: Lost task 55.0 in stage 6.0 (TID 1021, ip-10-0-50-96.ec2.internal, executor 45): ExecutorLostFailure (executor 45 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:05 WARN TaskSetManager: Lost task 13.0 in stage 6.0 (TID 979, ip-10-0-50-96.ec2.internal, executor 45): ExecutorLostFailure (executor 45 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:05 WARN TaskSetManager: Lost task 27.0 in stage 6.0 (TID 993, ip-10-0-50-96.ec2.internal, executor 45): ExecutorLostFailure (executor 45 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.7 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:05 ERROR YarnClusterScheduler: Lost executor 39 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.9 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:05 WARN TaskSetManager: Lost task 17.0 in stage 6.0 (TID 983, ip-10-0-51-185.ec2.internal, executor 39): ExecutorLostFailure (executor 39 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.9 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:05 WARN TaskSetManager: Lost task 31.0 in stage 6.0 (TID 997, ip-10-0-51-185.ec2.internal, executor 39): ExecutorLostFailure (executor 39 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.9 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:05 WARN TaskSetManager: Lost task 3.0 in stage 6.0 (TID 969, ip-10-0-51-185.ec2.internal, executor 39): ExecutorLostFailure (executor 39 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.9 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:05 WARN TaskSetManager: Lost task 45.0 in stage 6.0 (TID 1011, ip-10-0-51-185.ec2.internal, executor 39): ExecutorLostFailure (executor 39 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.9 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:05 INFO DAGScheduler: Executor lost: 45 (epoch 4) | |
| 17/06/14 06:37:05 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 06:37:05 INFO BlockManagerMasterEndpoint: Trying to remove executor 45 from BlockManagerMaster. | |
| 17/06/14 06:37:05 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(45, ip-10-0-50-96.ec2.internal, 40866, None) | |
| 17/06/14 06:37:05 INFO BlockManagerMaster: Removed 45 successfully in removeExecutor | |
| 17/06/14 06:37:05 INFO DAGScheduler: Executor lost: 39 (epoch 4) | |
| 17/06/14 06:37:05 INFO BlockManagerMasterEndpoint: Trying to remove executor 39 from BlockManagerMaster. | |
| 17/06/14 06:37:05 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(39, ip-10-0-51-185.ec2.internal, 46252, None) | |
| 17/06/14 06:37:05 INFO BlockManagerMaster: Removed 39 successfully in removeExecutor | |
| 17/06/14 06:37:05 INFO ExecutorAllocationManager: Existing executor 45 has been removed (new total is 10) | |
| 17/06/14 06:37:05 INFO ExecutorAllocationManager: Existing executor 39 has been removed (new total is 9) | |
| 17/06/14 06:37:05 WARN ExecutorAllocationManager: Attempted to mark unknown executor 45 idle | |
| 17/06/14 06:37:05 WARN ExecutorAllocationManager: Attempted to mark unknown executor 39 idle | |
| 17/06/14 06:37:05 WARN TransportChannelHandler: Exception in connection from /10.0.50.96:36940 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 06:37:05 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 45. | |
| 17/06/14 06:37:05 ERROR YarnClusterScheduler: Lost an executor 45 (already removed): Pending loss reason. | |
| 17/06/14 06:37:05 INFO BlockManagerMaster: Removal of executor 45 requested | |
| 17/06/14 06:37:05 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 45 | |
| 17/06/14 06:37:05 INFO BlockManagerMasterEndpoint: Trying to remove executor 45 from BlockManagerMaster. | |
| 17/06/14 06:37:05 INFO TaskSetManager: Starting task 45.1 in stage 6.0 (TID 1061, ip-10-0-50-96.ec2.internal, executor 56, partition 45, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Finished task 54.1 in stage 6.0 (TID 1023) in 19320 ms on ip-10-0-50-96.ec2.internal (executor 56) (4/135) | |
| 17/06/14 06:37:05 WARN TransportChannelHandler: Exception in connection from /10.0.51.185:35586 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 06:37:05 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 39. | |
| 17/06/14 06:37:05 ERROR YarnClusterScheduler: Lost an executor 39 (already removed): Pending loss reason. | |
| 17/06/14 06:37:05 INFO BlockManagerMaster: Removal of executor 39 requested | |
| 17/06/14 06:37:05 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 39 | |
| 17/06/14 06:37:05 INFO BlockManagerMasterEndpoint: Trying to remove executor 39 from BlockManagerMaster. | |
| 17/06/14 06:37:05 INFO TaskSetManager: Starting task 3.1 in stage 6.0 (TID 1062, ip-10-0-51-185.ec2.internal, executor 57, partition 3, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Finished task 58.0 in stage 6.0 (TID 1028) in 17687 ms on ip-10-0-51-185.ec2.internal (executor 57) (5/135) | |
| 17/06/14 06:37:05 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:37662) with ID 64 | |
| 17/06/14 06:37:05 INFO TaskSetManager: Starting task 31.1 in stage 6.0 (TID 1063, ip-10-0-50-96.ec2.internal, executor 64, partition 31, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Starting task 17.1 in stage 6.0 (TID 1064, ip-10-0-50-96.ec2.internal, executor 64, partition 17, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Starting task 27.1 in stage 6.0 (TID 1065, ip-10-0-50-96.ec2.internal, executor 64, partition 27, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Starting task 13.1 in stage 6.0 (TID 1066, ip-10-0-50-96.ec2.internal, executor 64, partition 13, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:05 INFO ExecutorAllocationManager: New executor 64 has registered (new total is 10) | |
| 17/06/14 06:37:05 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:37664) with ID 65 | |
| 17/06/14 06:37:05 INFO TaskSetManager: Starting task 55.1 in stage 6.0 (TID 1067, ip-10-0-50-96.ec2.internal, executor 65, partition 55, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Starting task 41.1 in stage 6.0 (TID 1068, ip-10-0-50-96.ec2.internal, executor 65, partition 41, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Starting task 23.1 in stage 6.0 (TID 1069, ip-10-0-50-96.ec2.internal, executor 65, partition 23, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Starting task 30.1 in stage 6.0 (TID 1070, ip-10-0-50-96.ec2.internal, executor 65, partition 30, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:05 INFO ExecutorAllocationManager: New executor 65 has registered (new total is 11) | |
| 17/06/14 06:37:05 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:37:05 INFO YarnAllocator: Submitted 2 unlocalized container requests. | |
| 17/06/14 06:37:05 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:44645 with 2.8 GB RAM, BlockManagerId(64, ip-10-0-50-96.ec2.internal, 44645, None) | |
| 17/06/14 06:37:05 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:37495 with 2.8 GB RAM, BlockManagerId(65, ip-10-0-50-96.ec2.internal, 37495, None) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Starting task 16.1 in stage 6.0 (TID 1071, ip-10-0-51-185.ec2.internal, executor 57, partition 16, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:05 INFO TaskSetManager: Finished task 56.0 in stage 6.0 (TID 1026) in 17894 ms on ip-10-0-51-185.ec2.internal (executor 57) (6/135) | |
| 17/06/14 06:37:06 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-50-96.ec2.internal:37495 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:06 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-50-96.ec2.internal:44645 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:06 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:46726) with ID 67 | |
| 17/06/14 06:37:06 INFO TaskSetManager: Starting task 2.1 in stage 6.0 (TID 1072, ip-10-0-49-157.ec2.internal, executor 67, partition 2, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:06 INFO TaskSetManager: Starting task 44.1 in stage 6.0 (TID 1073, ip-10-0-49-157.ec2.internal, executor 67, partition 44, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:06 INFO TaskSetManager: Starting task 6.1 in stage 6.0 (TID 1074, ip-10-0-49-157.ec2.internal, executor 67, partition 6, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:06 INFO TaskSetManager: Starting task 48.1 in stage 6.0 (TID 1075, ip-10-0-49-157.ec2.internal, executor 67, partition 48, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:06 INFO ExecutorAllocationManager: New executor 67 has registered (new total is 12) | |
| 17/06/14 06:37:06 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:45413 with 2.8 GB RAM, BlockManagerId(67, ip-10-0-49-157.ec2.internal, 45413, None) | |
| 17/06/14 06:37:06 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-157.ec2.internal:45413 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:06 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.50.96:37664 | |
| 17/06/14 06:37:06 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.50.96:37662 | |
| 17/06/14 06:37:06 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.157:46726 | |
| 17/06/14 06:37:06 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.157:46726 | |
| 17/06/14 06:37:06 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.50.96:37664 | |
| 17/06/14 06:37:06 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.50.96:37662 | |
| 17/06/14 06:37:06 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.157:46734) with ID 68 | |
| 17/06/14 06:37:06 INFO TaskSetManager: Starting task 34.1 in stage 6.0 (TID 1076, ip-10-0-49-157.ec2.internal, executor 68, partition 34, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:06 INFO TaskSetManager: Starting task 20.1 in stage 6.0 (TID 1077, ip-10-0-49-157.ec2.internal, executor 68, partition 20, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:06 INFO TaskSetManager: Starting task 21.1 in stage 6.0 (TID 1078, ip-10-0-49-157.ec2.internal, executor 68, partition 21, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:06 INFO TaskSetManager: Starting task 7.1 in stage 6.0 (TID 1079, ip-10-0-49-157.ec2.internal, executor 68, partition 7, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:06 INFO ExecutorAllocationManager: New executor 68 has registered (new total is 13) | |
| 17/06/14 06:37:07 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-157.ec2.internal:45574 with 2.8 GB RAM, BlockManagerId(68, ip-10-0-49-157.ec2.internal, 45574, None) | |
| 17/06/14 06:37:07 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-157.ec2.internal:45574 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:07 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.157:46734 | |
| 17/06/14 06:37:07 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.157:46734 | |
| 17/06/14 06:37:08 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.51.185:36400) with ID 69 | |
| 17/06/14 06:37:08 INFO TaskSetManager: Starting task 49.1 in stage 6.0 (TID 1080, ip-10-0-51-185.ec2.internal, executor 69, partition 49, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:08 INFO TaskSetManager: Starting task 35.1 in stage 6.0 (TID 1081, ip-10-0-51-185.ec2.internal, executor 69, partition 35, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:08 INFO TaskSetManager: Starting task 60.0 in stage 6.0 (TID 1082, ip-10-0-51-185.ec2.internal, executor 69, partition 60, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:08 INFO TaskSetManager: Starting task 61.0 in stage 6.0 (TID 1083, ip-10-0-51-185.ec2.internal, executor 69, partition 61, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:08 INFO ExecutorAllocationManager: New executor 69 has registered (new total is 14) | |
| 17/06/14 06:37:08 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:37692) with ID 70 | |
| 17/06/14 06:37:08 INFO TaskSetManager: Starting task 62.0 in stage 6.0 (TID 1084, ip-10-0-50-96.ec2.internal, executor 70, partition 62, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:08 INFO TaskSetManager: Starting task 63.0 in stage 6.0 (TID 1085, ip-10-0-50-96.ec2.internal, executor 70, partition 63, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:08 INFO TaskSetManager: Starting task 64.0 in stage 6.0 (TID 1086, ip-10-0-50-96.ec2.internal, executor 70, partition 64, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:08 INFO TaskSetManager: Starting task 65.0 in stage 6.0 (TID 1087, ip-10-0-50-96.ec2.internal, executor 70, partition 65, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:08 INFO ExecutorAllocationManager: New executor 70 has registered (new total is 15) | |
| 17/06/14 06:37:08 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-51-185.ec2.internal:39107 with 2.8 GB RAM, BlockManagerId(69, ip-10-0-51-185.ec2.internal, 39107, None) | |
| 17/06/14 06:37:08 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:42536 with 2.8 GB RAM, BlockManagerId(70, ip-10-0-50-96.ec2.internal, 42536, None) | |
| 17/06/14 06:37:08 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-51-185.ec2.internal:39107 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:09 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-50-96.ec2.internal:42536 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:09 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.51.185:36400 | |
| 17/06/14 06:37:09 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.51.185:36400 | |
| 17/06/14 06:37:09 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.50.96:37692 | |
| 17/06/14 06:37:09 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.50.96:37692 | |
| 17/06/14 06:37:17 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037984 on host ip-10-0-50-96.ec2.internal | |
| 17/06/14 06:37:17 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 06:37:17 INFO YarnAllocator: Completed container container_1497410974650_0004_01_037874 on host: ip-10-0-50-96.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:37:17 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:17 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:17 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:37:17 ERROR YarnClusterScheduler: Lost executor 56 on ip-10-0-50-96.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:17 WARN TaskSetManager: Lost task 45.1 in stage 6.0 (TID 1061, ip-10-0-50-96.ec2.internal, executor 56): ExecutorLostFailure (executor 56 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:17 WARN TaskSetManager: Lost task 26.1 in stage 6.0 (TID 1025, ip-10-0-50-96.ec2.internal, executor 56): ExecutorLostFailure (executor 56 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:17 WARN TaskSetManager: Lost task 37.1 in stage 6.0 (TID 1060, ip-10-0-50-96.ec2.internal, executor 56): ExecutorLostFailure (executor 56 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:17 WARN TaskSetManager: Lost task 51.1 in stage 6.0 (TID 1059, ip-10-0-50-96.ec2.internal, executor 56): ExecutorLostFailure (executor 56 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:17 INFO DAGScheduler: Executor lost: 56 (epoch 4) | |
| 17/06/14 06:37:17 INFO BlockManagerMasterEndpoint: Trying to remove executor 56 from BlockManagerMaster. | |
| 17/06/14 06:37:17 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(56, ip-10-0-50-96.ec2.internal, 42816, None) | |
| 17/06/14 06:37:17 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-50-96.ec2.internal:8041 | |
| 17/06/14 06:37:17 INFO BlockManagerMaster: Removed 56 successfully in removeExecutor | |
| 17/06/14 06:37:17 INFO ExecutorAllocationManager: Existing executor 56 has been removed (new total is 14) | |
| 17/06/14 06:37:17 WARN ExecutorAllocationManager: Attempted to mark unknown executor 56 idle | |
| 17/06/14 06:37:17 WARN TransportChannelHandler: Exception in connection from /10.0.50.96:37630 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 06:37:17 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 56. | |
| 17/06/14 06:37:17 ERROR YarnClusterScheduler: Lost an executor 56 (already removed): Pending loss reason. | |
| 17/06/14 06:37:17 INFO BlockManagerMaster: Removal of executor 56 requested | |
| 17/06/14 06:37:17 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 56 | |
| 17/06/14 06:37:17 INFO BlockManagerMasterEndpoint: Trying to remove executor 56 from BlockManagerMaster. | |
| 17/06/14 06:37:17 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:37:17 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 06:37:19 INFO TaskSetManager: Starting task 51.2 in stage 6.0 (TID 1088, ip-10-0-49-229.ec2.internal, executor 58, partition 51, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:19 INFO TaskSetManager: Starting task 37.2 in stage 6.0 (TID 1089, ip-10-0-49-229.ec2.internal, executor 58, partition 37, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:19 INFO TaskSetManager: Finished task 39.1 in stage 6.0 (TID 1030) in 21560 ms on ip-10-0-49-229.ec2.internal (executor 58) (7/135) | |
| 17/06/14 06:37:19 INFO TaskSetManager: Finished task 53.1 in stage 6.0 (TID 1033) in 21560 ms on ip-10-0-49-229.ec2.internal (executor 58) (8/135) | |
| 17/06/14 06:37:19 INFO YarnAllocator: Driver requested a total number of 32 executor(s). | |
| 17/06/14 06:37:19 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 32 executors. | |
| 17/06/14 06:37:19 INFO TaskSetManager: Starting task 26.2 in stage 6.0 (TID 1090, ip-10-0-49-229.ec2.internal, executor 58, partition 26, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:19 INFO TaskSetManager: Finished task 25.1 in stage 6.0 (TID 1031) in 21630 ms on ip-10-0-49-229.ec2.internal (executor 58) (9/135) | |
| 17/06/14 06:37:20 INFO TaskSetManager: Starting task 45.2 in stage 6.0 (TID 1091, ip-10-0-49-157.ec2.internal, executor 67, partition 45, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:20 INFO TaskSetManager: Finished task 6.1 in stage 6.0 (TID 1074) in 14145 ms on ip-10-0-49-157.ec2.internal (executor 67) (10/135) | |
| 17/06/14 06:37:20 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.50.96:37708) with ID 71 | |
| 17/06/14 06:37:20 INFO TaskSetManager: Starting task 66.0 in stage 6.0 (TID 1092, ip-10-0-50-96.ec2.internal, executor 71, partition 66, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:20 INFO TaskSetManager: Starting task 67.0 in stage 6.0 (TID 1093, ip-10-0-50-96.ec2.internal, executor 71, partition 67, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:20 INFO TaskSetManager: Starting task 68.0 in stage 6.0 (TID 1094, ip-10-0-50-96.ec2.internal, executor 71, partition 68, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:20 INFO TaskSetManager: Starting task 69.0 in stage 6.0 (TID 1095, ip-10-0-50-96.ec2.internal, executor 71, partition 69, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:20 INFO ExecutorAllocationManager: New executor 71 has registered (new total is 15) | |
| 17/06/14 06:37:20 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-50-96.ec2.internal:42878 with 2.8 GB RAM, BlockManagerId(71, ip-10-0-50-96.ec2.internal, 42878, None) | |
| 17/06/14 06:37:20 INFO TaskSetManager: Starting task 70.0 in stage 6.0 (TID 1096, ip-10-0-49-157.ec2.internal, executor 67, partition 70, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:20 INFO TaskSetManager: Finished task 2.1 in stage 6.0 (TID 1072) in 14560 ms on ip-10-0-49-157.ec2.internal (executor 67) (11/135) | |
| 17/06/14 06:37:20 INFO YarnAllocator: Driver requested a total number of 31 executor(s). | |
| 17/06/14 06:37:20 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 31 executors. | |
| 17/06/14 06:37:20 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-157.ec2.internal:40656 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:20 INFO TaskSetManager: Starting task 71.0 in stage 6.0 (TID 1097, ip-10-0-49-229.ec2.internal, executor 59, partition 71, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:20 INFO TaskSetManager: Finished task 22.1 in stage 6.0 (TID 1035) in 19645 ms on ip-10-0-49-229.ec2.internal (executor 59) (12/135) | |
| 17/06/14 06:37:21 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-50-96.ec2.internal:42878 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:21 INFO TaskSetManager: Starting task 72.0 in stage 6.0 (TID 1098, ip-10-0-49-229.ec2.internal, executor 59, partition 72, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:21 INFO TaskSetManager: Finished task 36.1 in stage 6.0 (TID 1034) in 20284 ms on ip-10-0-49-229.ec2.internal (executor 59) (13/135) | |
| 17/06/14 06:37:21 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.157:46718 | |
| 17/06/14 06:37:21 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.50.96:37708 | |
| 17/06/14 06:37:21 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.50.96:37708 | |
| 17/06/14 06:37:21 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.157:46718 | |
| 17/06/14 06:37:22 INFO TaskSetManager: Starting task 73.0 in stage 6.0 (TID 1099, ip-10-0-49-157.ec2.internal, executor 68, partition 73, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:22 INFO TaskSetManager: Finished task 7.1 in stage 6.0 (TID 1079) in 15419 ms on ip-10-0-49-157.ec2.internal (executor 68) (14/135) | |
| 17/06/14 06:37:22 INFO TaskSetManager: Starting task 74.0 in stage 6.0 (TID 1100, ip-10-0-49-157.ec2.internal, executor 68, partition 74, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:22 INFO TaskSetManager: Finished task 21.1 in stage 6.0 (TID 1078) in 15422 ms on ip-10-0-49-157.ec2.internal (executor 68) (15/135) | |
| 17/06/14 06:37:22 INFO YarnAllocator: Driver requested a total number of 30 executor(s). | |
| 17/06/14 06:37:22 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 30 executors. | |
| 17/06/14 06:37:24 INFO BlockManagerInfo: Added rdd_22_3 in memory on ip-10-0-51-185.ec2.internal:38110 (size: 833.5 MB, free: 1993.6 MB) | |
| 17/06/14 06:37:25 INFO TaskSetManager: Starting task 75.0 in stage 6.0 (TID 1101, ip-10-0-49-229.ec2.internal, executor 63, partition 75, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:25 INFO TaskSetManager: Starting task 76.0 in stage 6.0 (TID 1102, ip-10-0-49-229.ec2.internal, executor 63, partition 76, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:25 INFO TaskSetManager: Finished task 4.1 in stage 6.0 (TID 1051) in 21337 ms on ip-10-0-49-229.ec2.internal (executor 63) (16/135) | |
| 17/06/14 06:37:25 INFO TaskSetManager: Finished task 18.1 in stage 6.0 (TID 1050) in 21338 ms on ip-10-0-49-229.ec2.internal (executor 63) (17/135) | |
| 17/06/14 06:37:25 INFO TaskSetManager: Starting task 77.0 in stage 6.0 (TID 1103, ip-10-0-49-229.ec2.internal, executor 63, partition 77, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:25 INFO TaskSetManager: Finished task 46.1 in stage 6.0 (TID 1052) in 21338 ms on ip-10-0-49-229.ec2.internal (executor 63) (18/135) | |
| 17/06/14 06:37:29 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 58. | |
| 17/06/14 06:37:29 INFO DAGScheduler: Executor lost: 58 (epoch 4) | |
| 17/06/14 06:37:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 58 from BlockManagerMaster. | |
| 17/06/14 06:37:29 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(58, ip-10-0-49-229.ec2.internal, 35857, None) | |
| 17/06/14 06:37:29 INFO BlockManagerMaster: Removed 58 successfully in removeExecutor | |
| 17/06/14 06:37:29 INFO YarnAllocator: Launching container container_1497410974650_0004_01_037931 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 06:37:29 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 06:37:29 INFO YarnAllocator: Completed container container_1497410974650_0004_01_037876 on host: ip-10-0-49-229.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:37:29 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:29 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:29 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:37:29 ERROR YarnClusterScheduler: Lost executor 58 on ip-10-0-49-229.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:29 WARN TaskSetManager: Lost task 51.2 in stage 6.0 (TID 1088, ip-10-0-49-229.ec2.internal, executor 58): ExecutorLostFailure (executor 58 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:29 WARN TaskSetManager: Lost task 26.2 in stage 6.0 (TID 1090, ip-10-0-49-229.ec2.internal, executor 58): ExecutorLostFailure (executor 58 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:29 WARN TaskSetManager: Lost task 37.2 in stage 6.0 (TID 1089, ip-10-0-49-229.ec2.internal, executor 58): ExecutorLostFailure (executor 58 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:29 WARN TaskSetManager: Lost task 11.1 in stage 6.0 (TID 1032, ip-10-0-49-229.ec2.internal, executor 58): ExecutorLostFailure (executor 58 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:29 INFO BlockManagerMaster: Removal of executor 58 requested | |
| 17/06/14 06:37:29 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 58 | |
| 17/06/14 06:37:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 58 from BlockManagerMaster. | |
| 17/06/14 06:37:29 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 06:37:29 INFO ExecutorAllocationManager: Existing executor 58 has been removed (new total is 14) | |
| 17/06/14 06:37:29 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:37:29 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 06:37:30 INFO TaskSetManager: Starting task 11.2 in stage 6.0 (TID 1104, ip-10-0-51-185.ec2.internal, executor 60, partition 11, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:30 INFO TaskSetManager: Finished task 14.1 in stage 6.0 (TID 1041) in 28012 ms on ip-10-0-51-185.ec2.internal (executor 60) (19/135) | |
| 17/06/14 06:37:30 INFO YarnAllocator: Driver requested a total number of 29 executor(s). | |
| 17/06/14 06:37:30 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 29 executors. | |
| 17/06/14 06:37:30 INFO TaskSetManager: Starting task 37.3 in stage 6.0 (TID 1105, ip-10-0-50-96.ec2.internal, executor 64, partition 37, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:30 INFO TaskSetManager: Finished task 13.1 in stage 6.0 (TID 1066) in 24919 ms on ip-10-0-50-96.ec2.internal (executor 64) (20/135) | |
| 17/06/14 06:37:30 INFO TaskSetManager: Starting task 26.3 in stage 6.0 (TID 1106, ip-10-0-50-96.ec2.internal, executor 64, partition 26, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:30 INFO TaskSetManager: Finished task 27.1 in stage 6.0 (TID 1065) in 25062 ms on ip-10-0-50-96.ec2.internal (executor 64) (21/135) | |
| 17/06/14 06:37:31 INFO TaskSetManager: Starting task 51.3 in stage 6.0 (TID 1107, ip-10-0-51-185.ec2.internal, executor 57, partition 51, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:31 INFO TaskSetManager: Finished task 3.1 in stage 6.0 (TID 1062) in 25390 ms on ip-10-0-51-185.ec2.internal (executor 57) (22/135) | |
| 17/06/14 06:37:31 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:51294) with ID 72 | |
| 17/06/14 06:37:31 INFO TaskSetManager: Starting task 78.0 in stage 6.0 (TID 1108, ip-10-0-49-229.ec2.internal, executor 72, partition 78, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:31 INFO TaskSetManager: Starting task 79.0 in stage 6.0 (TID 1109, ip-10-0-49-229.ec2.internal, executor 72, partition 79, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:31 INFO TaskSetManager: Starting task 80.0 in stage 6.0 (TID 1110, ip-10-0-49-229.ec2.internal, executor 72, partition 80, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:31 INFO TaskSetManager: Starting task 81.0 in stage 6.0 (TID 1111, ip-10-0-49-229.ec2.internal, executor 72, partition 81, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:31 INFO ExecutorAllocationManager: New executor 72 has registered (new total is 15) | |
| 17/06/14 06:37:31 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:33580 with 2.8 GB RAM, BlockManagerId(72, ip-10-0-49-229.ec2.internal, 33580, None) | |
| 17/06/14 06:37:32 INFO TaskSetManager: Starting task 82.0 in stage 6.0 (TID 1112, ip-10-0-51-185.ec2.internal, executor 61, partition 82, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:32 INFO TaskSetManager: Finished task 24.1 in stage 6.0 (TID 1042) in 29958 ms on ip-10-0-51-185.ec2.internal (executor 61) (23/135) | |
| 17/06/14 06:37:32 INFO YarnAllocator: Driver requested a total number of 28 executor(s). | |
| 17/06/14 06:37:32 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 28 executors. | |
| 17/06/14 06:37:32 INFO TaskSetManager: Starting task 83.0 in stage 6.0 (TID 1113, ip-10-0-51-185.ec2.internal, executor 61, partition 83, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:32 INFO TaskSetManager: Finished task 52.1 in stage 6.0 (TID 1044) in 30040 ms on ip-10-0-51-185.ec2.internal (executor 61) (24/135) | |
| 17/06/14 06:37:32 INFO TaskSetManager: Starting task 84.0 in stage 6.0 (TID 1114, ip-10-0-51-185.ec2.internal, executor 61, partition 84, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:32 INFO TaskSetManager: Finished task 38.1 in stage 6.0 (TID 1045) in 30054 ms on ip-10-0-51-185.ec2.internal (executor 61) (25/135) | |
| 17/06/14 06:37:32 INFO TaskSetManager: Starting task 85.0 in stage 6.0 (TID 1115, ip-10-0-51-185.ec2.internal, executor 57, partition 85, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:32 INFO TaskSetManager: Finished task 59.0 in stage 6.0 (TID 1029) in 44318 ms on ip-10-0-51-185.ec2.internal (executor 57) (26/135) | |
| 17/06/14 06:37:32 INFO TaskSetManager: Starting task 86.0 in stage 6.0 (TID 1116, ip-10-0-50-96.ec2.internal, executor 70, partition 86, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:32 INFO TaskSetManager: Finished task 65.0 in stage 6.0 (TID 1087) in 23959 ms on ip-10-0-50-96.ec2.internal (executor 70) (27/135) | |
| 17/06/14 06:37:32 INFO YarnAllocator: Driver requested a total number of 27 executor(s). | |
| 17/06/14 06:37:32 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 27 executors. | |
| 17/06/14 06:37:32 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-229.ec2.internal:33580 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:32 INFO TaskSetManager: Starting task 87.0 in stage 6.0 (TID 1117, ip-10-0-50-96.ec2.internal, executor 65, partition 87, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:32 INFO TaskSetManager: Finished task 55.1 in stage 6.0 (TID 1067) in 26946 ms on ip-10-0-50-96.ec2.internal (executor 65) (28/135) | |
| 17/06/14 06:37:33 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.229:51294 | |
| 17/06/14 06:37:33 INFO TaskSetManager: Starting task 88.0 in stage 6.0 (TID 1118, ip-10-0-50-96.ec2.internal, executor 65, partition 88, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:33 INFO TaskSetManager: Finished task 30.1 in stage 6.0 (TID 1070) in 27528 ms on ip-10-0-50-96.ec2.internal (executor 65) (29/135) | |
| 17/06/14 06:37:33 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.229:51294 | |
| 17/06/14 06:37:37 INFO TaskSetManager: Starting task 89.0 in stage 6.0 (TID 1119, ip-10-0-51-185.ec2.internal, executor 69, partition 89, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:37 INFO TaskSetManager: Finished task 35.1 in stage 6.0 (TID 1081) in 28671 ms on ip-10-0-51-185.ec2.internal (executor 69) (30/135) | |
| 17/06/14 06:37:38 INFO BlockManagerInfo: Added broadcast_6_piece0 on disk on ip-10-0-51-185.ec2.internal:38110 (size: 30.1 KB) | |
| 17/06/14 06:37:38 INFO BlockManagerInfo: Removed rdd_22_3 on ip-10-0-51-185.ec2.internal:38110 in memory (size: 833.5 MB, free: 2.8 GB) | |
| 17/06/14 06:37:39 INFO BlockManagerInfo: Added rdd_22_71 in memory on ip-10-0-49-229.ec2.internal:38006 (size: 853.6 MB, free: 1973.5 MB) | |
| 17/06/14 06:37:40 INFO BlockManagerInfo: Added rdd_22_34 in memory on ip-10-0-49-157.ec2.internal:45574 (size: 836.7 MB, free: 1990.4 MB) | |
| 17/06/14 06:37:40 INFO BlockManagerInfo: Added rdd_22_44 in memory on ip-10-0-49-157.ec2.internal:45413 (size: 790.0 MB, free: 2037.1 MB) | |
| 17/06/14 06:37:40 INFO TaskSetManager: Starting task 90.0 in stage 6.0 (TID 1120, ip-10-0-49-229.ec2.internal, executor 59, partition 90, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:40 INFO TaskSetManager: Finished task 50.1 in stage 6.0 (TID 1037) in 39696 ms on ip-10-0-49-229.ec2.internal (executor 59) (31/135) | |
| 17/06/14 06:37:41 INFO YarnAllocator: Driver requested a total number of 26 executor(s). | |
| 17/06/14 06:37:41 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 26 executors. | |
| 17/06/14 06:37:41 INFO TaskSetManager: Starting task 91.0 in stage 6.0 (TID 1121, ip-10-0-49-229.ec2.internal, executor 59, partition 91, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:41 INFO TaskSetManager: Finished task 8.1 in stage 6.0 (TID 1036) in 40285 ms on ip-10-0-49-229.ec2.internal (executor 59) (32/135) | |
| 17/06/14 06:37:41 INFO TaskSetManager: Starting task 92.0 in stage 6.0 (TID 1122, ip-10-0-49-229.ec2.internal, executor 59, partition 92, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:41 INFO TaskSetManager: Finished task 71.0 in stage 6.0 (TID 1097) in 20691 ms on ip-10-0-49-229.ec2.internal (executor 59) (33/135) | |
| 17/06/14 06:37:41 INFO BlockManagerInfo: Added rdd_22_48 in memory on ip-10-0-49-157.ec2.internal:45413 (size: 830.8 MB, free: 1206.3 MB) | |
| 17/06/14 06:37:42 INFO TaskSetManager: Starting task 93.0 in stage 6.0 (TID 1123, ip-10-0-49-157.ec2.internal, executor 66, partition 93, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:42 INFO TaskSetManager: Finished task 5.1 in stage 6.0 (TID 1056) in 37152 ms on ip-10-0-49-157.ec2.internal (executor 66) (34/135) | |
| 17/06/14 06:37:42 INFO BlockManagerInfo: Added rdd_22_20 in memory on ip-10-0-49-157.ec2.internal:45574 (size: 865.6 MB, free: 1124.8 MB) | |
| 17/06/14 06:37:43 INFO BlockManagerInfo: Added rdd_22_32 in memory on ip-10-0-49-229.ec2.internal:45131 (size: 830.3 MB, free: 1996.8 MB) | |
| 17/06/14 06:37:44 INFO YarnAllocator: Launching container container_1497410974650_0004_01_038018 on host ip-10-0-49-229.ec2.internal | |
| 17/06/14 06:37:44 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 06:37:44 INFO YarnAllocator: Completed container container_1497410974650_0004_01_037909 on host: ip-10-0-49-229.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:37:44 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:44 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:44 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:37:44 ERROR YarnClusterScheduler: Lost executor 59 on ip-10-0-49-229.ec2.internal: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:44 WARN TaskSetManager: Lost task 90.0 in stage 6.0 (TID 1120, ip-10-0-49-229.ec2.internal, executor 59): ExecutorLostFailure (executor 59 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:44 WARN TaskSetManager: Lost task 72.0 in stage 6.0 (TID 1098, ip-10-0-49-229.ec2.internal, executor 59): ExecutorLostFailure (executor 59 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:44 WARN TaskSetManager: Lost task 92.0 in stage 6.0 (TID 1122, ip-10-0-49-229.ec2.internal, executor 59): ExecutorLostFailure (executor 59 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:44 WARN TaskSetManager: Lost task 91.0 in stage 6.0 (TID 1121, ip-10-0-49-229.ec2.internal, executor 59): ExecutorLostFailure (executor 59 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.6 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:44 INFO DAGScheduler: Executor lost: 59 (epoch 4) | |
| 17/06/14 06:37:44 INFO BlockManagerMasterEndpoint: Trying to remove executor 59 from BlockManagerMaster. | |
| 17/06/14 06:37:44 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(59, ip-10-0-49-229.ec2.internal, 38006, None) | |
| 17/06/14 06:37:44 INFO BlockManagerMaster: Removed 59 successfully in removeExecutor | |
| 17/06/14 06:37:44 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-49-229.ec2.internal:8041 | |
| 17/06/14 06:37:44 INFO ExecutorAllocationManager: Existing executor 59 has been removed (new total is 14) | |
| 17/06/14 06:37:44 WARN TransportChannelHandler: Exception in connection from /10.0.49.229:51194 | |
| java.io.IOException: Connection reset by peer | |
| at sun.nio.ch.FileDispatcherImpl.read0(Native Method) | |
| at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) | |
| at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) | |
| at sun.nio.ch.IOUtil.read(IOUtil.java:192) | |
| at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) | |
| at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221) | |
| at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899) | |
| at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 06:37:44 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 59. | |
| 17/06/14 06:37:44 ERROR YarnClusterScheduler: Lost an executor 59 (already removed): Pending loss reason. | |
| 17/06/14 06:37:44 INFO BlockManagerMaster: Removal of executor 59 requested | |
| 17/06/14 06:37:44 INFO YarnAllocator: Will request 1 executor container(s), each with 4 core(s) and 5632 MB memory (including 512 MB of overhead) | |
| 17/06/14 06:37:44 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 59 | |
| 17/06/14 06:37:44 INFO BlockManagerMasterEndpoint: Trying to remove executor 59 from BlockManagerMaster. | |
| 17/06/14 06:37:44 INFO YarnAllocator: Submitted 1 unlocalized container requests. | |
| 17/06/14 06:37:44 INFO TaskSetManager: Starting task 91.1 in stage 6.0 (TID 1124, ip-10-0-49-157.ec2.internal, executor 67, partition 91, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:44 INFO TaskSetManager: Finished task 44.1 in stage 6.0 (TID 1073) in 38644 ms on ip-10-0-49-157.ec2.internal (executor 67) (35/135) | |
| 17/06/14 06:37:44 INFO YarnAllocator: Driver requested a total number of 25 executor(s). | |
| 17/06/14 06:37:44 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 25 executors. | |
| 17/06/14 06:37:45 INFO TaskSetManager: Starting task 92.1 in stage 6.0 (TID 1125, ip-10-0-49-157.ec2.internal, executor 67, partition 92, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:45 INFO TaskSetManager: Finished task 48.1 in stage 6.0 (TID 1075) in 39087 ms on ip-10-0-49-157.ec2.internal (executor 67) (36/135) | |
| 17/06/14 06:37:45 INFO TaskSetManager: Starting task 72.1 in stage 6.0 (TID 1126, ip-10-0-49-157.ec2.internal, executor 68, partition 72, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:45 INFO TaskSetManager: Finished task 34.1 in stage 6.0 (TID 1076) in 38243 ms on ip-10-0-49-157.ec2.internal (executor 68) (37/135) | |
| 17/06/14 06:37:46 INFO TaskSetManager: Starting task 90.1 in stage 6.0 (TID 1127, ip-10-0-49-157.ec2.internal, executor 68, partition 90, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:46 INFO TaskSetManager: Finished task 20.1 in stage 6.0 (TID 1077) in 39229 ms on ip-10-0-49-157.ec2.internal (executor 68) (38/135) | |
| 17/06/14 06:37:46 INFO TaskSetManager: Starting task 94.0 in stage 6.0 (TID 1128, ip-10-0-49-229.ec2.internal, executor 63, partition 94, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:46 INFO TaskSetManager: Finished task 32.1 in stage 6.0 (TID 1053) in 42200 ms on ip-10-0-49-229.ec2.internal (executor 63) (39/135) | |
| 17/06/14 06:37:46 INFO YarnAllocator: Driver requested a total number of 24 executor(s). | |
| 17/06/14 06:37:46 INFO YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 24 executors. | |
| 17/06/14 06:37:46 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (10.0.49.229:51334) with ID 73 | |
| 17/06/14 06:37:46 INFO TaskSetManager: Starting task 95.0 in stage 6.0 (TID 1129, ip-10-0-49-229.ec2.internal, executor 73, partition 95, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:46 INFO TaskSetManager: Starting task 96.0 in stage 6.0 (TID 1130, ip-10-0-49-229.ec2.internal, executor 73, partition 96, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:46 INFO TaskSetManager: Starting task 97.0 in stage 6.0 (TID 1131, ip-10-0-49-229.ec2.internal, executor 73, partition 97, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:46 INFO TaskSetManager: Starting task 98.0 in stage 6.0 (TID 1132, ip-10-0-49-229.ec2.internal, executor 73, partition 98, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:46 INFO ExecutorAllocationManager: New executor 73 has registered (new total is 15) | |
| 17/06/14 06:37:46 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-0-49-229.ec2.internal:38888 with 2.8 GB RAM, BlockManagerId(73, ip-10-0-49-229.ec2.internal, 38888, None) | |
| 17/06/14 06:37:47 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-229.ec2.internal:38888 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:47 INFO BlockManagerInfo: Added broadcast_6_piece0 on disk on ip-10-0-49-157.ec2.internal:45413 (size: 30.1 KB) | |
| 17/06/14 06:37:47 INFO BlockManagerInfo: Removed rdd_22_44 on ip-10-0-49-157.ec2.internal:45413 in memory (size: 790.0 MB, free: 1996.3 MB) | |
| 17/06/14 06:37:47 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.229:51334 | |
| 17/06/14 06:37:47 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.229:51334 | |
| 17/06/14 06:37:48 INFO BlockManagerInfo: Added broadcast_6_piece0 on disk on ip-10-0-49-157.ec2.internal:45574 (size: 30.1 KB) | |
| 17/06/14 06:37:48 INFO BlockManagerInfo: Removed rdd_22_34 on ip-10-0-49-157.ec2.internal:45574 in memory (size: 836.7 MB, free: 1961.5 MB) | |
| 17/06/14 06:37:48 INFO BlockManagerInfo: Removed rdd_22_48 on ip-10-0-49-157.ec2.internal:45413 in memory (size: 830.8 MB, free: 2.8 GB) | |
| 17/06/14 06:37:48 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on ip-10-0-49-157.ec2.internal:42597 (size: 30.1 KB, free: 2.8 GB) | |
| 17/06/14 06:37:49 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 10.0.49.157:46710 | |
| 17/06/14 06:37:49 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 10.0.49.157:46710 | |
| 17/06/14 06:37:50 INFO BlockManagerInfo: Removed rdd_22_20 on ip-10-0-49-157.ec2.internal:45574 in memory (size: 865.6 MB, free: 2.8 GB) | |
| 17/06/14 06:37:52 INFO BlockManagerInfo: Added broadcast_6_piece0 on disk on ip-10-0-49-229.ec2.internal:45131 (size: 30.1 KB) | |
| 17/06/14 06:37:52 INFO BlockManagerInfo: Removed rdd_22_32 on ip-10-0-49-229.ec2.internal:45131 in memory (size: 830.3 MB, free: 2.8 GB) | |
| 17/06/14 06:37:53 INFO TaskSetManager: Starting task 99.0 in stage 6.0 (TID 1133, ip-10-0-50-96.ec2.internal, executor 71, partition 99, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:53 INFO TaskSetManager: Finished task 68.0 in stage 6.0 (TID 1094) in 32815 ms on ip-10-0-50-96.ec2.internal (executor 71) (40/135) | |
| 17/06/14 06:37:53 INFO TaskSetManager: Starting task 100.0 in stage 6.0 (TID 1134, ip-10-0-50-96.ec2.internal, executor 71, partition 100, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:53 INFO TaskSetManager: Finished task 66.0 in stage 6.0 (TID 1092) in 32827 ms on ip-10-0-50-96.ec2.internal (executor 71) (41/135) | |
| 17/06/14 06:37:54 INFO BlockManagerInfo: Added rdd_22_28 in memory on ip-10-0-51-185.ec2.internal:33361 (size: 850.4 MB, free: 1976.7 MB) | |
| 17/06/14 06:37:54 INFO BlockManagerInfo: Added rdd_22_0 in memory on ip-10-0-51-185.ec2.internal:33361 (size: 808.3 MB, free: 1168.4 MB) | |
| 17/06/14 06:37:54 INFO BlockManagerInfo: Added rdd_22_31 in memory on ip-10-0-50-96.ec2.internal:44645 (size: 848.2 MB, free: 1978.9 MB) | |
| 17/06/14 06:37:54 INFO TaskSetManager: Starting task 101.0 in stage 6.0 (TID 1135, ip-10-0-49-229.ec2.internal, executor 72, partition 101, PROCESS_LOCAL, 5837 bytes) | |
| 17/06/14 06:37:54 INFO TaskSetManager: Finished task 79.0 in stage 6.0 (TID 1109) in 22681 ms on ip-10-0-49-229.ec2.internal (executor 72) (42/135) | |
| 17/06/14 06:37:54 INFO YarnSchedulerBackend$YarnDriverEndpoint: Disabling executor 57. | |
| 17/06/14 06:37:54 INFO DAGScheduler: Executor lost: 57 (epoch 4) | |
| 17/06/14 06:37:54 INFO BlockManagerMasterEndpoint: Trying to remove executor 57 from BlockManagerMaster. | |
| 17/06/14 06:37:54 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(57, ip-10-0-51-185.ec2.internal, 38110, None) | |
| 17/06/14 06:37:54 INFO BlockManagerMaster: Removed 57 successfully in removeExecutor | |
| 17/06/14 06:37:54 INFO YarnAllocator: Launching container container_1497410974650_0004_01_038096 on host ip-10-0-51-185.ec2.internal | |
| 17/06/14 06:37:54 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. | |
| 17/06/14 06:37:54 INFO YarnAllocator: Completed container container_1497410974650_0004_01_037877 on host: ip-10-0-51-185.ec2.internal (state: COMPLETE, exit status: -104) | |
| 17/06/14 06:37:54 WARN YarnAllocator: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:54 ERROR YarnClusterScheduler: Lost executor 57 on ip-10-0-51-185.ec2.internal: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 85.0 in stage 6.0 (TID 1115, ip-10-0-51-185.ec2.internal, executor 57): ExecutorLostFailure (executor 57 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 9.1 in stage 6.0 (TID 1058, ip-10-0-51-185.ec2.internal, executor 57): ExecutorLostFailure (executor 57 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:54 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 51.3 in stage 6.0 (TID 1107, ip-10-0-51-185.ec2.internal, executor 57): ExecutorLostFailure (executor 57 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:54 ERROR TaskSetManager: Task 51 in stage 6.0 failed 4 times; aborting job | |
| 17/06/14 06:37:54 INFO ContainerManagementProtocolProxy: Opening proxy : ip-10-0-51-185.ec2.internal:8041 | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 16.1 in stage 6.0 (TID 1071, ip-10-0-51-185.ec2.internal, executor 57): ExecutorLostFailure (executor 57 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:54 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| 17/06/14 06:37:54 INFO BlockManagerMaster: Removal of executor 57 requested | |
| 17/06/14 06:37:54 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 57 | |
| 17/06/14 06:37:54 INFO BlockManagerMasterEndpoint: Trying to remove executor 57 from BlockManagerMaster. | |
| 17/06/14 06:37:54 INFO YarnClusterScheduler: Cancelling stage 6 | |
| 17/06/14 06:37:54 INFO ExecutorAllocationManager: Existing executor 57 has been removed (new total is 14) | |
| 17/06/14 06:37:54 INFO YarnClusterScheduler: Stage 6 was cancelled | |
| 17/06/14 06:37:54 INFO DAGScheduler: ResultStage 6 (saveAsTextFile at ShredJob.scala:412) failed in 72.869 s due to Job aborted due to stage failure: Task 51 in stage 6.0 failed 4 times, most recent failure: Lost task 51.3 in stage 6.0 (TID 1107, ip-10-0-51-185.ec2.internal, executor 57): ExecutorLostFailure (executor 57 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| Driver stacktrace: | |
| 17/06/14 06:37:54 INFO DAGScheduler: Job 1 failed: saveAsTextFile at ShredJob.scala:412, took 5781.569935 s | |
| 17/06/14 06:37:54 WARN ExecutorAllocationManager: No stages are running, but numRunningTasks != 0 | |
| 17/06/14 06:37:54 ERROR ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 51 in stage 6.0 failed 4 times, most recent failure: Lost task 51.3 in stage 6.0 (TID 1107, ip-10-0-51-185.ec2.internal, executor 57): ExecutorLostFailure (executor 57 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| Driver stacktrace: | |
| org.apache.spark.SparkException: Job aborted due to stage failure: Task 51 in stage 6.0 failed 4 times, most recent failure: Lost task 51.3 in stage 6.0 (TID 1107, ip-10-0-51-185.ec2.internal, executor 57): ExecutorLostFailure (executor 57 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| Driver stacktrace: | |
| at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1505) | |
| at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1493) | |
| at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1492) | |
| at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) | |
| at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) | |
| at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1492) | |
| at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:803) | |
| at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:803) | |
| at scala.Option.foreach(Option.scala:257) | |
| at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:803) | |
| at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1720) | |
| at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1675) | |
| at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1664) | |
| at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) | |
| at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:629) | |
| at org.apache.spark.SparkContext.runJob(SparkContext.scala:1918) | |
| at org.apache.spark.SparkContext.runJob(SparkContext.scala:1931) | |
| at org.apache.spark.SparkContext.runJob(SparkContext.scala:1951) | |
| at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1226) | |
| at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1168) | |
| at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1168) | |
| at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) | |
| at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) | |
| at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) | |
| at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1168) | |
| at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply$mcV$sp(PairRDDFunctions.scala:1071) | |
| at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1037) | |
| at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1037) | |
| at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) | |
| at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) | |
| at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) | |
| at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1037) | |
| at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply$mcV$sp(PairRDDFunctions.scala:963) | |
| at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:963) | |
| at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:963) | |
| at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) | |
| at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) | |
| at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) | |
| at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:962) | |
| at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply$mcV$sp(RDD.scala:1488) | |
| at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1467) | |
| at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1467) | |
| at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) | |
| at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) | |
| at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) | |
| at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1467) | |
| at com.snowplowanalytics.snowplow.storage.spark.ShredJob.run(ShredJob.scala:412) | |
| at com.snowplowanalytics.snowplow.storage.spark.ShredJob$.run(ShredJob.scala:78) | |
| at com.snowplowanalytics.snowplow.storage.spark.SparkJob$class.main(SparkJob.scala:32) | |
| at com.snowplowanalytics.snowplow.storage.spark.ShredJob$.main(ShredJob.scala:47) | |
| at com.snowplowanalytics.snowplow.storage.spark.ShredJob.main(ShredJob.scala) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637) | |
| 17/06/14 06:37:54 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 51 in stage 6.0 failed 4 times, most recent failure: Lost task 51.3 in stage 6.0 (TID 1107, ip-10-0-51-185.ec2.internal, executor 57): ExecutorLostFailure (executor 57 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 5.5 GB of 5.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. | |
| Driver stacktrace:) | |
| 17/06/14 06:37:54 WARN ExecutorAllocationManager: Attempted to mark unknown executor 57 idle | |
| 17/06/14 06:37:54 INFO SparkContext: Invoking stop() from shutdown hook | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 89.0 in stage 6.0 (TID 1119, ip-10-0-51-185.ec2.internal, executor 69): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 94.0 in stage 6.0 (TID 1128, ip-10-0-49-229.ec2.internal, executor 63): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 33.1 in stage 6.0 (TID 1054, ip-10-0-49-157.ec2.internal, executor 66): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 47.1 in stage 6.0 (TID 1057, ip-10-0-49-157.ec2.internal, executor 66): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 1.1 in stage 6.0 (TID 1047, ip-10-0-49-157.ec2.internal, executor 62): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 19.1 in stage 6.0 (TID 1055, ip-10-0-49-157.ec2.internal, executor 66): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 43.1 in stage 6.0 (TID 1048, ip-10-0-49-157.ec2.internal, executor 62): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 15.1 in stage 6.0 (TID 1046, ip-10-0-49-157.ec2.internal, executor 62): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 29.1 in stage 6.0 (TID 1049, ip-10-0-49-157.ec2.internal, executor 62): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 60.0 in stage 6.0 (TID 1082, ip-10-0-51-185.ec2.internal, executor 69): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 61.0 in stage 6.0 (TID 1083, ip-10-0-51-185.ec2.internal, executor 69): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 75.0 in stage 6.0 (TID 1101, ip-10-0-49-229.ec2.internal, executor 63): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 INFO SparkUI: Stopped Spark web UI at http://10.0.49.229:32886 | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 49.1 in stage 6.0 (TID 1080, ip-10-0-51-185.ec2.internal, executor 69): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 77.0 in stage 6.0 (TID 1103, ip-10-0-49-229.ec2.internal, executor 63): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 76.0 in stage 6.0 (TID 1102, ip-10-0-49-229.ec2.internal, executor 63): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerTaskEnd(6,0,ResultTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@626771,null) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerTaskEnd(6,0,ResultTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@28cecfd7,null) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerTaskEnd(6,0,ResultTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@53692df9,null) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(72, ip-10-0-49-229.ec2.internal, 33580, None),rdd_22_81,StorageLevel(1 replicas),0,0)) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(72, ip-10-0-49-229.ec2.internal, 33580, None),rdd_22_80,StorageLevel(1 replicas),0,0)) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 78.0 in stage 6.0 (TID 1108, ip-10-0-49-229.ec2.internal, executor 72): TaskCommitDenied (Driver denied task commit) for job: 6, partition: 78, attemptNumber: 0 | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerTaskEnd(6,0,ResultTask,TaskCommitDenied(6,78,0),org.apache.spark.scheduler.TaskInfo@1f2ca2ad,null) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 81.0 in stage 6.0 (TID 1111, ip-10-0-49-229.ec2.internal, executor 72): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerTaskEnd(6,0,ResultTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@c6617d6,null) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 80.0 in stage 6.0 (TID 1110, ip-10-0-49-229.ec2.internal, executor 72): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerTaskEnd(6,0,ResultTask,TaskKilled,org.apache.spark.scheduler.TaskInfo@5288d1fa,null) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(64, ip-10-0-50-96.ec2.internal, 44645, None),rdd_22_26,StorageLevel(1 replicas),0,0)) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(64, ip-10-0-50-96.ec2.internal, 44645, None),rdd_22_37,StorageLevel(1 replicas),0,0)) | |
| 17/06/14 06:37:54 INFO BlockManagerInfo: Added rdd_22_17 in memory on ip-10-0-50-96.ec2.internal:44645 (size: 846.8 MB, free: 1132.1 MB) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(64, ip-10-0-50-96.ec2.internal, 44645, None),rdd_22_17,StorageLevel(memory, deserialized, 1 replicas),887901912,0)) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(60, ip-10-0-51-185.ec2.internal, 33361, None),rdd_22_11,StorageLevel(1 replicas),0,0)) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(72, ip-10-0-49-229.ec2.internal, 33580, None),rdd_22_101,StorageLevel(1 replicas),0,0)) | |
| 17/06/14 06:37:54 INFO YarnAllocator: Driver requested a total number of 0 executor(s). | |
| 17/06/14 06:37:54 INFO YarnClusterSchedulerBackend: Shutting down all executors | |
| 17/06/14 06:37:54 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 97.0 in stage 6.0 (TID 1131, ip-10-0-49-229.ec2.internal, executor 73): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 98.0 in stage 6.0 (TID 1132, ip-10-0-49-229.ec2.internal, executor 73): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 WARN TaskSetManager: Lost task 95.0 in stage 6.0 (TID 1129, ip-10-0-49-229.ec2.internal, executor 73): TaskKilled (killed intentionally) | |
| 17/06/14 06:37:54 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices | |
| (serviceOption=None, | |
| services=List(), | |
| started=false) | |
| 17/06/14 06:37:54 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message. | |
| org.apache.spark.SparkException: Could not find CoarseGrainedScheduler. | |
| at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:154) | |
| at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:134) | |
| at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:570) | |
| at org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequestHandler.java:180) | |
| at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:109) | |
| at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:119) | |
| at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51) | |
| at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353) | |
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346) | |
| at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353) | |
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346) | |
| at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353) | |
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346) | |
| at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353) | |
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346) | |
| at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353) | |
| at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 06:37:54 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(61, ip-10-0-51-185.ec2.internal, 43408, None),rdd_22_10,StorageLevel(1 replicas),0,0)) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(61, ip-10-0-51-185.ec2.internal, 43408, None),rdd_22_82,StorageLevel(1 replicas),0,0)) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(61, ip-10-0-51-185.ec2.internal, 43408, None),rdd_22_83,StorageLevel(1 replicas),0,0)) | |
| 17/06/14 06:37:54 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message. | |
| org.apache.spark.SparkException: Could not find CoarseGrainedScheduler. | |
| at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:154) | |
| at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:134) | |
| at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:570) | |
| at org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequestHandler.java:180) | |
| at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:109) | |
| at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:119) | |
| at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51) | |
| at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353) | |
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346) | |
| at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353) | |
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346) | |
| at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353) | |
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346) | |
| at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353) | |
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346) | |
| at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367) | |
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353) | |
| at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) | |
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575) | |
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489) | |
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451) | |
| at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140) | |
| at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) | |
| at java.lang.Thread.run(Thread.java:745) | |
| 17/06/14 06:37:54 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(61, ip-10-0-51-185.ec2.internal, 43408, None),rdd_22_84,StorageLevel(1 replicas),0,0)) | |
| 17/06/14 06:37:54 INFO MemoryStore: MemoryStore cleared | |
| 17/06/14 06:37:54 INFO BlockManager: BlockManager stopped | |
| 17/06/14 06:37:54 INFO BlockManagerMaster: BlockManagerMaster stopped | |
| 17/06/14 06:37:54 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! | |
| 17/06/14 06:37:54 INFO SparkContext: Successfully stopped SparkContext | |
| 17/06/14 06:37:54 INFO ShutdownHookManager: Shutdown hook called | |
| 17/06/14 06:37:54 INFO ShutdownHookManager: Deleting directory /mnt/yarn/usercache/hadoop/appcache/application_1497410974650_0004/spark-40a69b25-5302-435b-8302-9b9367bdb8e5 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment