rdhabalia

ZK-latency log at broker:
[pulsar-zk-session-watcher-10-1] WARN  c.y.p.z.ZooKeeperSessionWatcher      - zoo keeper disconnected, waiting to reconnect, time remaining = 25 seconds

ZK nc stat
(1) before test
zk_version      3.4.6--1, built on 08/05/2015 23:17 GMT
zk_avg_latency  11
zk_max_latency  3720
zk_min_latency  0
zk_packets_received     30146197

rdhabalia

2017-04-01 01:35:52,314 - ERROR - [main:TestListener@41] - ------------ Test Failed - com.yahoo.pulsar.client.impl.BrokerClientIntegrationTest / testCloseConnectionOnBrokerRejectedRequest -- attrs: []
java.lang.AssertionError: expected [false] but found [true]
	at org.testng.Assert.fail(Assert.java:94)
	at org.testng.Assert.failNotEquals(Assert.java:494)
	at org.testng.Assert.assertFalse(Assert.java:63)
	at org.testng.Assert.assertFalse(Assert.java:73)
	at com.yahoo.pulsar.client.impl.BrokerClientIntegrationTest.testCloseConnectionOnBrokerRejectedRequest(BrokerClientIntegrationTest.java:526)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

rdhabalia

    
Deadlock- jstack of pulsar (when create topic thread gets blocked):

Thread 22396: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.StampedLock.acquireWrite(boolean, long) @bci=619, line=1119 (Compiled frame)
 - java.util.concurrent.locks.StampedLock.writeLock() @bci=43, line=354 (Compiled frame)
 - com.yahoo.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.remove(java.lang.Object, java.lang.Object, int) @bci=1, line=302 (Compiled frame)
 - com.yahoo.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.access$200(com.yahoo.pulsar.common.util.collections.ConcurrentOpenHashMap$Section, java.lang.Object, java.lang.Object, int) @bci=4, line=178 (Compiled frame)
 - com.yahoo.pulsar.common.util.collections.ConcurrentOpenHashMap.remove(java.lang.Object, java.lang.Object) @bci=24, line=140 (Interpreted frame)

rdhabalia

07:23:49.246 [main-SendThread(<ip:host>)] WARN  org.apache.zookeeper.ClientCnxn      - Exception while seeking for r/w server <zk-host-address>
java.net.UnknownHostException: <zk-host-address>
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) ~[na:1.8.0_112]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_112]
        at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_112]
        at java.net.Socket.connect(Socket.java:538) ~[na:1.8.0_112]
        at java.net.Socket.<init>(Socket.java:434) ~[na:1.8.0_112]
        at java.net.Socket.<init>(Socket.java:211) ~[na:1.8.0_112]
        at org.apache.zookeeper.ClientCnxn$SendThread.pingRwServer(ClientCnxn.java:1141) [zookeeper-3.4.6.jar:3.4.6-1569965]
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1076) [zookeeper-3.4.6.jar:3.4.6-1569965]

rdhabalia

2017-04-23 17:51:19
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode):

"Attach Listener" #219 daemon prio=9 os_prio=0 tid=0x00007f78dc007000 nid=0x7f71 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"logback-1" #218 daemon prio=5 os_prio=0 tid=0x00007f7874020000 nid=0x7151 waiting on condition [0x00007f7849979000]

rdhabalia

2017-04-23 17:56:53
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode):

"Attach Listener" #219 daemon prio=9 os_prio=0 tid=0x00007f78dc007000 nid=0x7f71 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"logback-1" #218 daemon prio=5 os_prio=0 tid=0x00007f7874020000 nid=0x7151 waiting on condition [0x00007f7849979000]

rdhabalia

/**
 * Copyright 2016 Yahoo Inc.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software

rdhabalia

2017-07-14 23:35:34
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode):

"Attach Listener" #22031 daemon prio=9 os_prio=31 tid=0x00007fbe5b892800 nid=0x4ae67 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"ForkJoinPool.commonPool-worker-0" #22030 daemon prio=5 os_prio=31 tid=0x00007fbe57b8b000 nid=0x51a63 waiting on condition [0x0000700024f31000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000740c2e6d0> (a java.util.concurrent.ForkJoinPool)

rdhabalia

2017-07-18 00:49:37
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.102-b14 mixed mode):

"Attach Listener" #25 daemon prio=9 os_prio=0 tid=0x00007f3948001000 nid=0x4292 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"pulsar-websocket-1-3" #24 prio=5 os_prio=0 tid=0x00007f3900059800 nid=0x413f waiting on condition [0x00007f396901f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d65ab4e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

rdhabalia

Why Custom Rate-limiter?

A. Usecase: Per second rate-limiting not satisfied by Guava-RateLimiter
(1) Guava RateLimiter: For X permits: it releases X/1000 permits every msec. therefore, for permits=2/sec => 
it release 1st permit on first 500msec and 2nd permit on next 500ms. therfore, if 2 request comes with in 500msec duration then 
2nd request fails to acquire permit though we have configured 2 permits/second.
https://github.com/google/guava/blob/v15.0/guava/src/com/google/common/util/concurrent/RateLimiter.java#L102
(2) Custon RateLimiter: it releases X permits every second. so, in above usecase: if 2 requests comes at the same time then both will 
acquire the permit.