Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
2015-01-23 15:36:53,654 INFO [Reporter] yarn.YarnAllocationHandler (Logging.scala:logInfo(59)) - Container marked as failed: container_1421692415636_0052_01_000030. Exit status: 143. Diagnostics: Container [pid=35211,containerID=container_1421692415636_0052_01_000030] is running beyond physical memory limits. Current usage: 14.9 GB of 14.5 GB physical memory used; 41.3 GB of 72.5 GB virtual memory used. Killing container.
Dump of the process-tree for container_1421692415636_0052_01_000030 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 54101 36625 36625 35211 (python) 78 1 332730368 16834 python -m pyspark.daemon
|- 52140 36625 36625 35211 (python) 58 1 332730368 16837 python -m pyspark.daemon
|- 36625 35228 36625 35211 (python) 65 604 331685888 17694 python -m pyspark.daemon
|- 54612 36625 36625 35211 (python) 83 1 332730368 16868 python -m pyspark.daemon
|- 54104 36625 36625 35211 (python) 77 1 333344768 16988 python -m pyspark.daemon
|- 54107 36625 36625 35211 (python) 78 2 332730368 16834 python -m pyspark.daemon
|- 51006 36625 36625 35211 (python) 90 1 333139968 16988 python -m pyspark.daemon
|- 54493 36625 36625 35211 (python) 68 2 332730368 16837 python -m pyspark.daemon
|- 54415 36625 36625 35211 (python) 74 2 332730368 16834 python -m pyspark.daemon
|- 53477 36625 36625 35211 (python) 138 2 333500416 17051 python -m pyspark.daemon
|- 51952 36625 36625 35211 (python) 68 2 332533760 16837 python -m pyspark.daemon
|- 50990 36625 36625 35211 (python) 1769 114 527429632 64425 python -m pyspark.daemon
|- 52561 36625 36625 35211 (python) 78 2 332611584 16837 python -m pyspark.daemon
|- 54496 36625 36625 35211 (python) 66 2 332730368 16834 python -m pyspark.daemon
|- 51776 36625 36625 35211 (python) 63 1 332611584 16892 python -m pyspark.daemon
|- 51773 36625 36625 35211 (python) 68 0 332730368 16834 python -m pyspark.daemon
|- 53804 36625 36625 35211 (python) 166 2 333393920 17053 python -m pyspark.daemon
|- 51017 36625 36625 35211 (python) 99 2 333307904 17050 python -m pyspark.daemon
|- 35228 35211 35211 35211 (java) 299915 77999 11481403392 2150129 /usr/java/latest/bin/java -server -XX:OnOutOfMemoryError=kill %p -Xms8550m -Xmx8550m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/mnt/var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1421692415636_0052/container_1421692415636_0052_01_000030/tmp -Dspark.yarn.app.container.log.dir=/mnt/var/log/hadoop/userlogs/application_1421692415636_0052/container_1421692415636_0052_01_000030 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@ip-10-20-80-7.us-west-1.compute.internal:39747/user/CoarseGrainedScheduler 29 ip-10-20-80-12.us-west-1.compute.internal 2 application_1421692415636_0052
|- 53175 36625 36625 35211 (python) 142 2 333459456 17052 python -m pyspark.daemon
|- 54621 36625 36625 35211 (python) 76 2 332730368 16836 python -m pyspark.daemon
|- 52416 36625 36625 35211 (python) 82 2 332730368 16837 python -m pyspark.daemon
|- 54178 36625 36625 35211 (python) 86 2 332730368 16837 python -m pyspark.daemon
|- 53661 36625 36625 35211 (python) 176 3 333438976 17055 python -m pyspark.daemon
|- 52231 36625 36625 35211 (python) 61 2 332730368 16835 python -m pyspark.daemon
|- 53270 36625 36625 35211 (python) 136 4 333328384 17032 python -m pyspark.daemon
|- 54826 36625 36625 35211 (python) 76 3 332730368 16835 python -m pyspark.daemon
|- 54695 36625 36625 35211 (python) 428 24 451633152 45971 python -m pyspark.daemon
|- 52836 36625 36625 35211 (python) 151 1 333402112 17060 python -m pyspark.daemon
|- 54535 36625 36625 35211 (python) 74 3 332611584 16829 python -m pyspark.daemon
|- 51599 36625 36625 35211 (python) 76 0 332730368 16837 python -m pyspark.daemon
|- 35211 6662 35211 35211 (bash) 0 0 115630080 302 /bin/bash -c /usr/java/latest/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms8550m -Xmx8550m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/mnt/var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1421692415636_0052/container_1421692415636_0052_01_000030/tmp -Dspark.yarn.app.container.log.dir=/mnt/var/log/hadoop/userlogs/application_1421692415636_0052/container_1421692415636_0052_01_000030 org.apache.spark.executor.CoarseGrainedExecutorBackend akka.tcp://sparkDriver@ip-10-20-80-7.us-west-1.compute.internal:39747/user/CoarseGrainedScheduler 29 ip-10-20-80-12.us-west-1.compute.internal 2 application_1421692415636_0052 1> /mnt/var/log/hadoop/userlogs/application_1421692415636_0052/container_1421692415636_0052_01_000030/stdout 2> /mnt/var/log/hadoop/userlogs/application_1421692415636_0052/container_1421692415636_0052_01_000030/stderr
|- 52932 36625 36625 35211 (python) 155 2 333373440 17054 python -m pyspark.daemon
|- 54279 36625 36625 35211 (python) 76 2 332730368 16887 python -m pyspark.daemon
|- 51302 36625 36625 35211 (python) 78 1 332730368 16870 python -m pyspark.daemon
|- 53409 36625 36625 35211 (python) 135 2 333406208 17045 python -m pyspark.daemon
|- 54330 36625 36625 35211 (python) 72 2 332730368 16836 python -m pyspark.daemon
|- 54270 36625 36625 35211 (python) 82 0 332730368 16837 python -m pyspark.daemon
|- 54684 36625 36625 35211 (python) 66 2 332730368 16835 python -m pyspark.daemon
|- 52091 36625 36625 35211 (python) 16 0 331685888 16678 python -m pyspark.daemon
|- 53096 36625 36625 35211 (python) 138 3 333254656 17025 python -m pyspark.daemon
|- 53348 36625 36625 35211 (python) 140 3 333385728 17056 python -m pyspark.daemon
|- 51682 36625 36625 35211 (python) 54 1 332730368 16834 python -m pyspark.daemon
|- 54409 36625 36625 35211 (python) 74 1 332730368 16832 python -m pyspark.daemon
|- 54342 36625 36625 35211 (python) 72 2 332730368 16836 python -m pyspark.daemon
|- 53549 36625 36625 35211 (python) 159 1 333434880 17043 python -m pyspark.daemon
|- 54155 36625 36625 35211 (python) 80 1 332730368 16889 python -m pyspark.daemon
|- 54200 36625 36625 35211 (python) 73 2 332730368 16835 python -m pyspark.daemon
|- 51417 36625 36625 35211 (python) 62 0 332730368 16837 python -m pyspark.daemon
|- 52218 36625 36625 35211 (python) 68 2 332730368 16834 python -m pyspark.daemon
|- 51505 36625 36625 35211 (python) 46 0 332730368 16837 python -m pyspark.daemon
|- 54359 36625 36625 35211 (python) 75 4 332611584 16889 python -m pyspark.daemon
|- 54553 36625 36625 35211 (python) 74 3 332611584 16829 python -m pyspark.daemon
|- 52912 36625 36625 35211 (python) 142 1 333418496 17050 python -m pyspark.daemon
|- 52109 36625 36625 35211 (python) 59 5 332730368 16834 python -m pyspark.daemon
|- 52662 36625 36625 35211 (python) 158 2 333430784 17042 python -m pyspark.daemon
|- 54457 36625 36625 35211 (python) 73 0 332730368 16837 python -m pyspark.daemon
|- 53330 36625 36625 35211 (python) 143 1 333312000 17037 python -m pyspark.daemon
|- 54098 36625 36625 35211 (python) 98 2 333283328 17059 python -m pyspark.daemon
|- 51045 36625 36625 35211 (python) 79 0 332730368 16837 python -m pyspark.daemon
|- 51870 36625 36625 35211 (python) 64 1 332533760 16832 python -m pyspark.daemon
|- 54666 36625 36625 35211 (python) 74 2 332611584 16834 python -m pyspark.daemon
|- 52995 36625 36625 35211 (python) 132 2 333336576 17030 python -m pyspark.daemon
|- 53020 36625 36625 35211 (python) 132 4 333246464 17029 python -m pyspark.daemon
|- 53712 36625 36625 35211 (python) 166 5 333492224 17054 python -m pyspark.daemon
|- 53128 36625 36625 35211 (python) 136 3 333393920 17054 python -m pyspark.daemon
|- 52659 36625 36625 35211 (python) 1180 116 333492224 17103 python -m pyspark.daemon
|- 53638 36625 36625 35211 (python) 147 2 333402112 17025 python -m pyspark.daemon
|- 52706 36625 36625 35211 (python) 161 3 333410304 17096 python -m pyspark.daemon
|- 51525 36625 36625 35211 (python) 72 2 332730368 16833 python -m pyspark.daemon
|- 52300 36625 36625 35211 (python) 70 0 332730368 16834 python -m pyspark.daemon
|- 51864 36625 36625 35211 (python) 72 1 332611584 16832 python -m pyspark.daemon
|- 51983 36625 36625 35211 (python) 68 0 332730368 16837 python -m pyspark.daemon
|- 54567 36625 36625 35211 (python) 75 1 332730368 16834 python -m pyspark.daemon
|- 53425 36625 36625 35211 (python) 142 2 333410304 17061 python -m pyspark.daemon
|- 53795 36625 36625 35211 (python) 166 2 333479936 17054 python -m pyspark.daemon
|- 51669 36625 36625 35211 (python) 52 1 332730368 16835 python -m pyspark.daemon
|- 54758 36625 36625 35211 (python) 69 0 332730368 16834 python -m pyspark.daemon
|- 52375 36625 36625 35211 (python) 55 2 332644352 16837 python -m pyspark.daemon
|- 54131 36625 36625 35211 (python) 86 0 332730368 16837 python -m pyspark.daemon
|- 53842 36625 36625 35211 (python) 160 2 333484032 17064 python -m pyspark.daemon
|- 54261 36625 36625 35211 (python) 34 1 332533760 16773 python -m pyspark.daemon
|- 50977 36625 36625 35211 (python) 1273 120 523436032 63454 python -m pyspark.daemon
|- 53506 36625 36625 35211 (python) 147 4 333369344 17050 python -m pyspark.daemon
|- 52479 36625 36625 35211 (python) 87 2 332730368 16837 python -m pyspark.daemon
|- 53734 36625 36625 35211 (python) 195 1 333328384 17059 python -m pyspark.daemon
|- 51060 36625 36625 35211 (python) 63 2 332730368 16837 python -m pyspark.daemon
|- 51449 36625 36625 35211 (python) 84 0 332730368 16869 python -m pyspark.daemon
|- 52605 36625 36625 35211 (python) 80 0 332730368 16884 python -m pyspark.daemon
|- 52703 36625 36625 35211 (python) 1148 50 333520896 17106 python -m pyspark.daemon
|- 51346 36625 36625 35211 (python) 77 0 332730368 16834 python -m pyspark.daemon
|- 53196 36625 36625 35211 (python) 147 0 333373440 17043 python -m pyspark.daemon
|- 52326 36625 36625 35211 (python) 74 1 332730368 16837 python -m pyspark.daemon
|- 54905 36625 36625 35211 (python) 15 0 332271616 16709 python -m pyspark.daemon
|- 52801 36625 36625 35211 (python) 152 3 333488128 17048 python -m pyspark.daemon
|- 52051 36625 36625 35211 (python) 63 0 332730368 16831 python -m pyspark.daemon
|- 53585 36625 36625 35211 (python) 144 3 333393920 17035 python -m pyspark.daemon
|- 53054 36625 36625 35211 (python) 150 2 333434880 17068 python -m pyspark.daemon
|- 53253 36625 36625 35211 (python) 145 0 333459456 17076 python -m pyspark.daemon
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment