secret
Created

  • Download Gist
log1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
383453565 [kafka-processor-3] INFO kafka.network.Processor - Closing socket connection to /10.29.157.196.
383455128 [kafka-processor-2] INFO kafka.network.Processor - Closing socket connection to /10.29.173.42.
383463720 [kafka-processor-4] INFO kafka.network.Processor - Closing socket connection to /10.28.235.161.
383469529 [kafka-processor-6] FATAL kafka.log.Log - Halting due to unrecoverable I/O error while handling producer request
java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.FileDispatcher.write(FileDispatcher.java:57)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)
at sun.nio.ch.IOUtil.write(IOUtil.java:93)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:224)
at kafka.message.ByteBufferMessageSet.writeTo(ByteBufferMessageSet.scala:76)
at kafka.message.FileMessageSet.append(FileMessageSet.scala:159)
at kafka.log.Log.liftedTree1$1(Log.scala:229)
at kafka.log.Log.append(Log.scala:227)
at kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$handleProducerRequest(KafkaRequestHandlers.scala:69)
at kafka.server.KafkaRequestHandlers.handleProducerRequest(KafkaRequestHandlers.scala:53)
at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
at kafka.network.Processor.handle(SocketServer.scala:296)
at kafka.network.Processor.read(SocketServer.scala:319)
at kafka.network.Processor.run(SocketServer.scala:214)
at java.lang.Thread.run(Thread.java:679)
log2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107
0 [main] INFO kafka.server.KafkaServer - Starting Kafka server...
22 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-5'
49 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/lwe-core-logs-5/00000000000000000000.kafka in mutable mode, recovery true
404 [main] INFO kafka.message.FileMessageSet - recover high water mark:1916214
405 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
425 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-7'
426 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/solr-logs-7/00000000001610613165.kafka in mutable mode, recovery true
2932 [main] INFO kafka.message.FileMessageSet - recover high water mark:128489919
2932 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
2933 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-5'
2934 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/solr-logs-10258 [main] INFO kafka.message.FileMessageSet - recover high water mark:128625261
10258 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
10259 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-4'
10259 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/solr-logs-4/00000000001610613168.kafka in mutable mode, recovery true
12627 [main] INFO kafka.message.FileMessageSet - recover high water mark:129492302
12628 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
12628 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-4'
12629 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/lwe-core-logs-4/00000000000000000000.kafka in mutable mode, recovery true
12678 [main] INFO kafka.message.FileMessageSet - recover high water mark:1890901
12678 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
12679 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-1'
12679 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/lwe-core-logs-1/00000000000000000000.kafka in mutable mode, recovery true
12730 [main] INFO kafka.message.FileMessageSet - recover high water mark:1944354
12730 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
12730 [main] INFO kafka.log.LogManager - Loading log 'query-logs-1'
12731 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/query-logs-1/00000000000000000000.kafka in mutable mode, recovery true
12731 [main] INFO kafka.message.FileMessageSet - recover high water mark:75
12731 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
12731 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-0'
12732 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/solr-logs-0/00000000001610613136.kafka in mutable mode, recovery true
15108 [main] INFO kafka.message.FileMessageSet - recover high water mark:129667775
15108 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
15108 [main] INFO kafka.log.LogManager - Loading log 'query-logs-7'
15109 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/query-logs-7/00000000000000000000.kafka in mutable mode, recovery true
15109 [main] INFO kafka.message.FileMessageSet - recover high water mark:75
15109 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
15109 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-2'
15110 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/solr-logs-2/00000000001610612926.kafka in mutable mode, recovery true
17464 [main] INFO kafka.message.FileMessageSet - recover high water mark:128508920
17464 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
17465 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-6'
17465 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/lwe-core-logs-6/00000000000000000000.kafka in mutable mode, recovery true
17516 [main] INFO kafka.message.FileMessageSet - recover high water mark:1890717
17516 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
17517 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-3'
17517 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/lwe-core-logs-3/00000000000000000000.kafka in mutable mode, recovery true
17567 [main] INFO kafka.message.FileMessageSet - recover high water mark:1915506
17567 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
17568 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-1'
17568 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/solr-logs-1/00000000001610613178.kafka in mutable mode, recovery true
19977 [main] INFO kafka.message.FileMessageSet - recover high water mark:127901543
19977 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 153 bytes truncated.
 
19978 [main] INFO kafka.log.LogManager - starting log cleaner every 60000 ms
19986 [main] INFO kafka.server.KafkaZooKeeper - connecting to ZK: 10.29.194.104:2181,
19998 [ZkClient-EventThread-14-10.29.194.104:2181,] INFO org.I0Itec.zkclient.ZkEventThread - Starting ZkClient event thread.
20011 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
20011 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=ip-10-29-143-151.ec2.internal
20011 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.6.0_22
20011 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Sun Microsystems Inc.
20011 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
20011 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=:/opt/kafka-0.7.1/config/log4j.properties:/opt/kafka-0.7.1/libs/jopt-simple-3.2.jar:/opt/kafka-0.7.1/libs/kafka-0.7.1.jar:/opt/kafka-0.7.1/libs/log4j-1.2.15.jar:/opt/kafka-0.7.1/libs/scala-library.jar:/opt/kafka-0.7.1/libs/snappy-java-1.0.4.1.jar:/opt/kafka-0.7.1/libs/zkclient-0.1.jar:/opt/kafka-0.7.1/libs/zookeeper-3.3.4.jar
20011 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
20011 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
20011 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
20011 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
20011 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
20012 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=2.6.38-13-virtual
20012 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=hadoop
20012 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/hadoop
20012 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/etc/sv/kafka
20013 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=10.29.194.104:2181, sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@193936e1
20032 [main-SendThread()] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server /10.29.194.104:2181
20045 [main-SendThread(ip-10-29-194-104.ec2.internal:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to ip-10-29-194-104.ec2.internal/10.29.194.104:2181, initiating session
20053 [main-SendThread(ip-10-29-194-104.ec2.internal:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server ip-10-29-194-104.ec2.internal/10.29.194.104:2181, sessionid = 0x138b3d7a5941a86, negotiated timeout = 6000
20055 [main-EventThread] INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
20164 [kafka-acceptor] INFO kafka.network.Acceptor - Awaiting connections on port 9092
20167 [main] INFO kafka.utils.Mx4jLoader$ - Will not load MX4J, mx4j-tools.jar is not in the classpath
20167 [main] INFO kafka.server.KafkaZooKeeper - Registering broker /brokers/ids/0
20178 [main] INFO kafka.server.KafkaZooKeeper - Registering broker /brokers/ids/0 succeeded with id:0,creatorId:10.29.143.151-1343433921339,host:10.29.143.151,port:9092
20196 [main] INFO kafka.server.KafkaZooKeeper - Begin registering broker topic /brokers/topics/lwe-core-logs/0 with 8 partitions
20210 [main] INFO kafka.server.KafkaZooKeeper - End registering broker topic /brokers/topics/lwe-core-logs/0
20210 [main] INFO kafka.server.KafkaZooKeeper - Begin registering broker topic /brokers/topics/query-logs/0 with 8 partitions
20212 [main] INFO kafka.server.KafkaZooKeeper - End registering broker topic /brokers/topics/query-logs/0
20212 [main] INFO kafka.server.KafkaZooKeeper - Begin registering broker topic /brokers/topics/solr-logs/0 with 8 partitions
20216 [main] INFO kafka.server.KafkaZooKeeper - End registering broker topic /brokers/topics/solr-logs/0
20221 [main] INFO kafka.log.LogManager - Starting log flusher every 1000 ms with the following overrides Map()
20224 [main] INFO kafka.server.KafkaServer - Kafka server started.
20267 [kafka-processor-2] FATAL kafka.log.Log - Halting due to unrecoverable I/O error while handling producer request
java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.FileDispatcher.write(FileDispatcher.java:57)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)
at sun.nio.ch.IOUtil.write(IOUtil.java:93)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:224)
at kafka.message.ByteBufferMessageSet.writeTo(ByteBufferMessageSet.scala:76)
at kafka.message.FileMessageSet.append(FileMessageSet.scala:159)
at kafka.log.Log.liftedTree1$1(Log.scala:229)
at kafka.log.Log.append(Log.scala:227)
at kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$handleProducerRequest(KafkaRequestHandlers.scala:69)
at kafka.server.KafkaRequestHandlers.handleProducerRequest(KafkaRequestHandlers.scala:53)
at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
at kafka.network.Processor.handle(SocketServer.scala:296)
at kafka.network.Processor.read(SocketServer.scala:319)
at kafka.network.Processor.run(SocketServer.scala:214)
at java.lang.Thread.run(Thread.java:679)
log3
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143
0 [main] INFO kafka.server.KafkaServer - Starting Kafka server...
21 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-5'
45 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/lwe-core-logs-5/00000000000000000000.kafka in mutable mode, recovery true
379 [main] INFO kafka.message.FileMessageSet - recover high water mark:1916214
380 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
400 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-7'
401 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/solr-logs-7/00000000001610613165.kafka in mutable mode, recovery true
2861 [main] INFO kafka.message.FileMessageSet - recover high water mark:128490190
2861 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
2861 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-5'
2862 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/solr-logs-5/00000000001610613225.kafka in mutable mode, recovery true
5316 [main] INFO kafka.message.FileMessageSet - recover high water mark:129755119
5316 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
5317 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-3'
5317 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/solr-logs-3/00000000001610612968.kafka in mutable mode, recovery true
7715 [main] INFO kafka.message.FileMessageSet - recover high water mark:128392805
7715 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
7716 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-7'
7716 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/lwe-core-logs-7/00000000000000000000.kafka in mutable mode, recovery true
7767 [main] INFO kafka.message.FileMessageSet - recover high water mark:1917915
7768 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
7768 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-0'
7768 [main] INFO 7622 [main] INFO kafka.message.FileMessageSet - recover high water mark:128393005
7622 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
7622 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-7'
7623 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/lwe-core-logs-7/00000000000000000000.kafka in mutable mode, recovery true
7672 [main] INFO kafka.message.FileMessageSet - recover high water mark:1917915
7672 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
7672 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-0'
7673 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/lwe-core-logs-0/00000000000000000000.kafka in mutable mode, recovery true
7721 [main] INFO kafka.message.FileMessageSet - recover high water mark:1904008
7721 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
7721 [main] INFO kafka.log.LogManager - Loading log 'query-logs-2'
7722 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/query-logs-2/00000000000000000000.kafka in mutable mode, recovery true
7722 [main] INFO kafka.message.FileMessageSet - recover high water mark:72
7722 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
7722 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-2'
7723 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/lwe-core-logs-2/00000000000000000000.kafka in mutable mode, recovery true
7772 [main] INFO kafka.message.FileMessageSet - recover high water mark:1909263
7772 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
7773 [main] INFO kafka.log.LogManager - Loading log 'query-logs-0'
7773 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/query-logs-0/00000000000000000000.kafka in mutable mode, recovery true
7773 [main] INFO kafka.message.FileMessageSet - recover high water mark:219
7774 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
7774 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-6'
7774 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/solr-logs-6/00000000001610613047.kafka in mutable mode, recovery true
10073 [main] INFO kafka.message.FileMessageSet - recover high water mark:128626518
10074 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 170 bytes truncated.
10074 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-4'
10075 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/solr-logs-4/00000000001610613168.kafka in mutable mode, recovery true
12374 [main] INFO kafka.message.FileMessageSet - recover high water mark:129494756
12374 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
12375 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-4'
12375 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/lwe-core-logs-4/00000000000000000000.kafka in mutable mode, recovery true
12423 [main] INFO kafka.message.FileMessageSet - recover high water mark:1890901
12424 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
12424 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-1'
12424 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/lwe-core-logs-1/00000000000000000000.kafka in mutable mode, recovery true
12474 [main] INFO kafka.message.FileMessageSet - recover high water mark:1944354
12474 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
12474 [main] INFO kafka.log.LogManager - Loading log 'query-logs-1'
12475 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/query-logs-1/00000000000000000000.kafka in mutable mode, recovery true
12475 [main] INFO kafka.message.FileMessageSet - recover high water mark:75
12475 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
12475 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-0'
12476 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/solr-logs-0/00000000001610613136.kafka in mutable mode, recovery true
14784 [main] INFO kafka.message.FileMessageSet - recover high water mark:129677833
14784 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
14785 [main] INFO kafka.log.LogManager - Loading log 'query-logs-7'
14785 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/query-logs-7/00000000000000000000.kafka in mutable mode, recovery true
14785 [main] INFO kafka.message.FileMessageSet - recover high water mark:75
14785 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
14786 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-2'
14786 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/solr-logs-2/00000000001610612926.kafka in mutable mode, recovery true
17101 [main] INFO kafka.message.FileMessageSet - recover high water mark:128510009
17104 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
17104 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-6'
17105 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/lwe-core-logs-6/00000000000000000000.kafka in mutable mode, recovery true
17155 [main] INFO kafka.message.FileMessageSet - recover high water mark:1890717
17155 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
17155 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-3'
17155 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/lwe-core-logs-3/00000000000000000000.kafka in mutable mode, recovery true
17206 [main] INFO kafka.message.FileMessageSet - recover high water mark:1915506
17206 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
17206 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-1'
17207 [main] INFO kafka.log.Log - Loading the last segment /tmp/kafka-logs/solr-logs-1/00000000001610613178.kafka in mutable mode, recovery true
19546 [main] INFO kafka.message.FileMessageSet - recover high water mark:127904977
19546 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
19547 [main] INFO kafka.log.LogManager - starting log cleaner every 60000 ms
19555 [main] INFO kafka.server.KafkaZooKeeper - connecting to ZK: 10.29.194.104:2181,
19567 [ZkClient-EventThread-14-10.29.194.104:2181,] INFO org.I0Itec.zkclient.ZkEventThread - Starting ZkClient event thread.
19577 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
19577 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=ip-10-29-143-151.ec2.internal
19577 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.6.0_22
19577 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Sun Microsystems Inc.
19577 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
19577 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=:/opt/kafka-0.7.1/config/log4j.properties:/opt/kafka-0.7.1/libs/jopt-simple-3.2.jar:/opt/kafka-0.7.1/libs/kafka-0.7.1.jar:/opt/kafka-0.7.1/libs/log4j-1.2.15.jar:/opt/kafka-0.7.1/libs/scala-library.jar:/opt/kafka-0.7.1/libs/snappy-java-1.0.4.1.jar:/opt/kafka-0.7.1/libs/zkclient-0.1.jar:/opt/kafka-0.7.1/libs/zookeeper-3.3.4.jar
19577 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
19578 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
19578 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
19578 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
19578 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
19578 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=2.6.38-13-virtual
19578 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=hadoop
19578 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/hadoop
19578 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/etc/sv/kafka
19579 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=10.29.194.104:2181, sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@49d8c528
19596 [main-SendThread()] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server /10.29.194.104:2181
19604 [main-SendThread(ip-10-29-194-104.ec2.internal:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to ip-10-29-194-104.ec2.internal/10.29.194.104:2181, initiating session
19611 [main-SendThread(ip-10-29-194-104.ec2.internal:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server ip-10-29-194-104.ec2.internal/10.29.194.104:2181, sessionid = 0x138b3d7a5941a88, negotiated timeout = 6000
19613 [main-EventThread] INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
19715 [kafka-acceptor] INFO kafka.network.Acceptor - Awaiting connections on port 9092
19718 [main] INFO kafka.utils.Mx4jLoader$ - Will not load MX4J, mx4j-tools.jar is not in the classpath
19718 [main] INFO kafka.server.KafkaZooKeeper - Registering broker /brokers/ids/0
19732 [main] INFO kafka.server.KafkaZooKeeper - Registering broker /brokers/ids/0 succeeded with id:0,creatorId:10.29.143.151-1343433964064,host:10.29.143.151,port:9092
19747 [main] INFO kafka.server.KafkaZooKeeper - Begin registering broker topic /brokers/topics/lwe-core-logs/0 with 8 partitions
19751 [main] INFO kafka.server.KafkaZooKeeper - End registering broker topic /brokers/topics/lwe-core-logs/0
19752 [main] INFO kafka.server.KafkaZooKeeper - Begin registering broker topic /brokers/topics/query-logs/0 with 8 partitions
19755 [main] INFO kafka.server.KafkaZooKeeper - End registering broker topic /brokers/topics/query-logs/0
19755 [main] INFO kafka.server.KafkaZooKeeper - Begin registering broker topic /brokers/topics/solr-logs/0 with 8 partitions
19758 [main] INFO kafka.server.KafkaZooKeeper - End registering broker topic /brokers/topics/solr-logs/0
19762 [main] INFO kafka.log.LogManager - Starting log flusher every 1000 ms with the following overrides Map()
19763 [main] INFO kafka.server.KafkaServer - Kafka server started.
19967 [kafka-processor-0] FATAL kafka.log.Log - Halting due to unrecoverable I/O error while handling producer request
java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.FileDispatcher.write(FileDispatcher.java:57)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)
at sun.nio.ch.IOUtil.write(IOUtil.java:93)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:224)
at kafka.message.ByteBufferMessageSet.writeTo(ByteBufferMessageSet.scala:76)
at kafka.message.FileMessageSet.append(FileMessageSet.scala:159)
at kafka.log.Log.liftedTree1$1(Log.scala:229)
at kafka.log.Log.append(Log.scala:227)
at kafka.server.KafkaRequestHandlers.kafka$server$KafkaRequestHandlers$$handleProducerRequest(KafkaRequestHandlers.scala:69)
at kafka.server.KafkaRequestHandlers.handleProducerRequest(KafkaRequestHandlers.scala:53)
at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
at kafka.server.KafkaRequestHandlers$$anonfun$handlerFor$1.apply(KafkaRequestHandlers.scala:38)
at kafka.network.Processor.handle(SocketServer.scala:296)
at kafka.network.Processor.read(SocketServer.scala:319)
at kafka.network.Processor.run(SocketServer.scala:214)
at java.lang.Thread.run(Thread.java:679)
0 [main] INFO kafka.server.KafkaServer - Starting Kafka server...
log4
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117
0 [main] INFO kafka.server.KafkaServer - Starting Kafka server...
22 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-0'
46 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/lwe-core-logs-0/00000000000000000000.kafka in mutable mode, recovery true
413 [main] INFO kafka.message.FileMessageSet - recover high water mark:1904588
414 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
489 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-1'
490 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/lwe-core-logs-1/00000000000000000000.kafka in mutable mode, recovery true
569 [main] INFO kafka.message.FileMessageSet - recover high water mark:1945152
569 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
570 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-2'
570 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/lwe-core-logs-2/00000000000000000000.kafka in mutable mode, recovery true
641 [main] INFO kafka.message.FileMessageSet - recover high water mark:1910029
641 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
641 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-3'
642 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/lwe-core-logs-3/00000000000000000000.kafka in mutable mode, recovery true
1233 [main] INFO kafka.message.FileMessageSet - recover high water mark:1916396
1234 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
1234 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-4'
1235 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/lwe-core-logs-4/00000000000000000000.kafka in mutable mode, recovery true
1296 [main] INFO kafka.message.FileMessageSet - recover high water mark:1891577
1296 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
1297 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-5'
1297 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/lwe-core-logs-5/00000000000000000000.kafka in mutable mode, recovery true
1359 [main] INFO kafka.message.FileMessageSet - recover high water mark:1916848
1359 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
1359 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-6'
1360 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/lwe-core-logs-6/00000000000000000000.kafka in mutable mode, recovery true
1422 [main] INFO kafka.message.FileMessageSet - recover high water mark:1891367
1422 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
1423 [main] INFO kafka.log.LogManager - Loading log 'lwe-core-logs-7'
1423 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/lwe-core-logs-7/00000000000000000000.kafka in mutable mode, recovery true
1485 [main] INFO kafka.message.FileMessageSet - recover high water mark:1918675
1485 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
1486 [main] INFO kafka.log.LogManager - Loading log 'query-logs-0'
1486 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/query-logs-0/00000000000000000000.kafka in mutable mode, recovery true
1492 [main] INFO kafka.message.FileMessageSet - recover high water mark:219
1492 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
1493 [main] INFO kafka.log.LogManager - Loading log 'query-logs-1'
1493 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/query-logs-1/00000000000000000000.kafka in mutable mode, recovery true
 
1494 [main] INFO kafka.message.FileMessageSet - recover high water mark:75
1494 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
1494 [main] INFO kafka.log.LogManager - Loading log 'query-logs-2'
1495 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/query-logs-2/00000000000000000000.kafka in mutable mode, recovery true
1495 [main] INFO kafka.message.FileMessageSet - recover high water mark:72
1495 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
1496 [main] INFO kafka.log.LogManager - Loading log 'query-logs-7'
1496 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/query-logs-7/00000000000000000000.kafka in mutable mode, recovery true
1497 [main] INFO kafka.message.FileMessageSet - recover high water mark:75
1497 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 0 seconds. 0 bytes truncated.
1497 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-0'
1498 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/solr-logs-0/00000000001610613136.kafka in mutable mode, recovery true
4324 [main] INFO kafka.message.FileMessageSet - recover high water mark:130760624
4324 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
4324 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-1'
4325 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/solr-logs-1/00000000001610613178.kafka in mutable mode, recovery true
7042 [main] INFO kafka.message.FileMessageSet - recover high water mark:128978806
7042 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
7043 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-2'
7044 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/solr-logs-2/00000000001610612926.kafka in mutable mode, recovery true
9826 [main] INFO kafka.message.FileMessageSet - recover high water mark:129580927
9827 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
9827 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-3'
9828 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/solr-logs-3/00000000001610612968.kafka in mutable mode, recovery true
12447 [main] INFO kafka.message.FileMessageSet - recover high water mark:129449849
12448 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
12448 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-4'
12449 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/solr-logs-4/00000000001610613168.kafka in mutable mode, recovery true
15354 [main] INFO kafka.message.FileMessageSet - recover high water mark:130568075
15354 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
15354 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-5'
15355 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/solr-logs-5/00000000001610613225.kafka in mutable mode, recovery true
19473 [main] INFO kafka.message.FileMessageSet - recover high water mark:130805676
19473 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 4 seconds. 0 bytes truncated.
19473 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-6'
19474 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/solr-logs-6/00000000001610613047.kafka in mutable mode, recovery true
22016 [main] INFO kafka.message.FileMessageSet - recover high water mark:129687452
22016 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
22016 [main] INFO kafka.log.LogManager - Loading log 'solr-logs-7'
22017 [main] INFO kafka.log.Log - Loading the last segment /data/kafka-logs/solr-logs-7/00000000001610613165.kafka in mutable mode, recovery true
24572 [main] INFO kafka.message.FileMessageSet - recover high water mark:129564614
24572 [main] INFO kafka.message.FileMessageSet - Recovery succeeded in 2 seconds. 0 bytes truncated.
24573 [main] INFO kafka.log.LogManager - starting log cleaner every 60000 ms
24617 [main] INFO kafka.server.KafkaZooKeeper - connecting to ZK: 10.29.194.104:2181,
24636 [ZkClient-EventThread-14-10.29.194.104:2181,] INFO org.I0Itec.zkclient.ZkEventThread - Starting ZkClient event thread.
24648 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT
24648 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=ip-10-29-143-151.ec2.internal
24648 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.6.0_22
24648 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Sun Microsystems Inc.
24648 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-6-openjdk/jre
24648 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=:/opt/kafka-0.7.1/config/log4j.properties:/opt/kafka-0.7.1/libs/jopt-simple-3.2.jar:/opt/kafka-0.7.1/libs/kafka-0.7.1.jar:/opt/kafka-0.7.1/libs/log4j-1.2.15.jar:/opt/kafka-0.7.1/libs/scala-library.jar:/opt/kafka-0.7.1/libs/snappy-java-1.0.4.1.jar:/opt/kafka-0.7.1/libs/zkclient-0.1.jar:/opt/kafka-0.7.1/libs/zookeeper-3.3.4.jar
24648 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
24648 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
24649 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
24649 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
24649 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
24649 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=2.6.38-13-virtual
24649 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=hadoop
24649 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/hadoop
24649 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/etc/sv/kafka
24650 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=10.29.194.104:2181, sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@62402af3
24746 [main-SendThread()] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server /10.29.194.104:2181
24771 [main-SendThread(ip-10-29-194-104.ec2.internal:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to ip-10-29-194-104.ec2.internal/10.29.194.104:2181, initiating session
24779 [main-SendThread(ip-10-29-194-104.ec2.internal:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server ip-10-29-194-104.ec2.internal/10.29.194.104:2181, sessionid = 0x138b3d7a59443af, negotiated timeout = 6000
24782 [main-EventThread] INFO org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
24969 [kafka-acceptor] INFO kafka.network.Acceptor - Awaiting connections on port 9092
24972 [main] INFO kafka.utils.Mx4jLoader$ - Will not load MX4J, mx4j-tools.jar is not in the classpath
24973 [main] INFO kafka.server.KafkaZooKeeper - Registering broker /brokers/ids/0
24989 [main] INFO kafka.server.KafkaZooKeeper - Registering broker /brokers/ids/0 succeeded with id:0,creatorId:10.29.143.151-1343656857979,host:10.29.143.151,port:9092
25043 [main] INFO kafka.server.KafkaZooKeeper - Begin registering broker topic /brokers/topics/lwe-core-logs/0 with 8 partitions
25050 [main] INFO kafka.server.KafkaZooKeeper - End registering broker topic /brokers/topics/lwe-core-logs/0
25051 [main] INFO kafka.server.KafkaZooKeeper - Begin registering broker topic /brokers/topics/query-logs/0 with 8 partitions
25056 [main] INFO kafka.server.KafkaZooKeeper - End registering broker topic /brokers/topics/query-logs/0
25056 [main] INFO kafka.server.KafkaZooKeeper - Begin registering broker topic /brokers/topics/solr-logs/0 with 8 partitions
25060 [main] INFO kafka.server.KafkaZooKeeper - End registering broker topic /brokers/topics/solr-logs/0
25078 [main] INFO kafka.log.LogManager - Starting log flusher every 1000 ms with the following overrides Map()
25082 [main] INFO kafka.server.KafkaServer - Kafka server started.

The first file is the initial encounter with the exception, the next two files are just some logs showing the recovery and subsequent failure (disk is still out of space). We have a supervisor that was restarting Kafka, but it backed off after a few tries. The last file shows the startup after I fixed the disk space issue.

Please sign in to comment on this gist.

Something went wrong with that request. Please try again.