Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Geo Replication Demo without a global zookeeper

This doc demonstrates how to do geo-replication across multiple pulsar clusters without a global configuration store (zookeeper).

This demo is using docker-compose to start 3 pulsar clusters. Each pulsar cluster has 1 zk, 1 bk, and 1 broker. The docker compose file can be found at https://gist.github.com/sijie/63737459112471a82957ae20bd78adb5.

The information of all the three clusters is listed in the following table:

zk configuration store broker
beijing zk-beijing zk-beijing broker-beijing
shanghai zk-shanghai zk-shanghai broker-shanghai
guangzhou zk-guangzhou zk-guangzhou broker-guangzhou

0. prepare the cluster

Prepare the docker image

This demon is using pulsar's test image apachepulsar/pulsar-test-latest-version. You can build the docker image locally or pull the image from docker hub.

  1. build it from source code:
mvn install -DskipTests -Pdocker   
  1. or use docker pull
docker pull apachepulsar/pulsar-test-latest-version 

start all nodes:

Download the docker compose file from https://gist.github.com/sijie/63737459112471a82957ae20bd78adb5.

In the directory that contains the downloaded docker compose file, run following command to start 3 clusters.

docker-compose up

Since it will start 9 docker containers locally, please make sure you allocate enough memory for your docker daemon.

1. Start additional containers as client for each cluster.

Start a client container for cluster beijing

Open a new terminal:

a. start a docker instance

docker run --name client-beijing  -it --rm  --network cluster_pulsar apachepulsar/pulsar-test-latest-version:latest /bin/bash

b. in this docker instance run this command to replace the broker address in client.conf

$ sed -i "s/localhost/broker-beijing/g" conf/client.conf

Start a client container for cluster shanghai

Open a new terminal:

a. start a docker instance

docker run --name client-shanghai  -it --rm  --network cluster_pulsar apachepulsar/pulsar-test-latest-version:latest /bin/bash

b. in this docker instance run this command to replace the broker address in client.conf

$ sed -i "s/localhost/broker-shanghai/g" conf/client.conf

Start a client container for cluster guangzhou

Open a new terminal:

a. start a docker instance

docker run --name client-guangzhou  -it --rm  --network cluster_pulsar apachepulsar/pulsar-test-latest-version:latest /bin/bash

b. in this docker instance run this command to replace the broker address in client.conf

$ sed -i "s/localhost/broker-guangzhou/g" conf/client.conf

2. Create clusters

NOTE: this is the extra step required for settinig up geo-replication when there is no global configuration store.

Since there is no global configuration store for all three clusters, they don't know each other at this moment. You need to create clusters on each cluster to tell a cluster how it can access the other two clusters.

a. Create shanghai and guangzhou clusters on beijing cluster

On client-beijing container, run following commands to create cluster shanghai and guangzhou.

bin/pulsar-admin clusters create --url http://broker-shanghai:8080 --broker-url pulsar://broker-shanghai:6650 shanghai
bin/pulsar-admin clusters create --url http://broker-guangzhou:8080 --broker-url pulsar://broker-guangzhou:6650 guangzhou

These commands basically tells the brokers in beijing cluster how they can access cluster shanghai and guangzhou.

b. Create beijing and guangzhou clusters on shanghai cluster

On client-shanghai container, run following commands to create cluster beijing and guangzhou.

bin/pulsar-admin clusters create --url http://broker-beijing:8080 --broker-url pulsar://broker-beijing:6650 beijing
bin/pulsar-admin clusters create --url http://broker-guangzhou:8080 --broker-url pulsar://broker-guangzhou:6650 guangzhou

c. Create beijing and shanghai clusters on guangzhou cluster

On client-guangzhou container, run following commands to create cluster beijing and shanghai.

bin/pulsar-admin clusters create --url http://broker-beijing:8080 --broker-url pulsar://broker-beijing:6650 beijing
bin/pulsar-admin clusters create --url http://broker-shanghai:8080 --broker-url pulsar://broker-shanghai:6650 shanghai

After executing the above commands, each cluster knows how to connect to the other two clusters now. Let's move to create tenants and namespaces.

2. Create tenants and namespace

Since we don't setup a global configuration store for three clusters, these three clusters don't share tenants and namespace policies. So you have to create the tenants and namespaces on all three clusters.

on the client containers (client-beijing / client-shanghai / client-guangzhou) run following commands to create a tenant and a namespace.

bin/pulsar-admin tenants create my-tenant  --allowed-clusters beijing,shanghai,guangzhou
bin/pulsar-admin namespaces create my-tenant/my-namespace --clusters beijing,shanghai,guangzhou

3. verify messages send and receive.

a. in both client-shanghai and client-guangzhou, create a subscription for the test topic, they will wait receive messages from client-beijing

in client-shanghai

bin/pulsar-client consume -s "sub-shanghai" my-tenant/my-namespace/t1 -n 0

in client-guangzhou

bin/pulsar-client consume -s "sub-guangzhou" my-tenant/my-namespace/t1 -n 0

b. in client-beijing, produce message to the topic

bin/pulsar-client produce  my-tenant/my-namespace/t1  --messages "hello-from-beijing-1" -n 10

You will see both client-shanghai and client-guangzhou will receive all the messages produced by client-beijing.

@nnivruth
Copy link

nnivruth commented Oct 21, 2019

@sijie stuck at 3a. with the below error.

02:12:00.040 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Error while consuming messages
02:12:00.041 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
org.apache.pulsar.client.api.PulsarClientException$BrokerPersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:271) ~[org.apache.pulsar-pulsar-client-api-2.4.0.jar:2.4.0]
at org.apache.pulsar.client.impl.ConsumerBuilderImpl.subscribe(ConsumerBuilderImpl.java:90) ~[org.apache.pulsar-pulsar-client-original-2.4.0.jar:2.4.0]
at org.apache.pulsar.client.cli.CmdConsume.consume(CmdConsume.java:162) [org.apache.pulsar-pulsar-client-tools-2.4.0.jar:2.4.0]
at org.apache.pulsar.client.cli.CmdConsume.run(CmdConsume.java:151) [org.apache.pulsar-pulsar-client-tools-2.4.0.jar:2.4.0]
at org.apache.pulsar.client.cli.PulsarClientTool.run(PulsarClientTool.java:133) [org.apache.pulsar-pulsar-client-tools-2.4.0.jar:2.4.0]
at org.apache.pulsar.client.cli.PulsarClientTool.main(PulsarClientTool.java:165) [org.apache.pulsar-pulsar-client-tools-2.4.0.jar:2.4.0]
02:12:00.047 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 0 messages successfully consumed

@dockerzhang
Copy link

dockerzhang commented Dec 26, 2019

@sijie If pulsar clusters are configured using TLS, Only https://xxx:8443/ and pulsar+ssl://xxx:6651/ are available, do you have any advice for geo-replication? I tried to create cluster with --url-secure/--broker-url-secure, but it was not work for me.

@one70six
Copy link

one70six commented Jan 22, 2020

@sijie, same issue is happening on my test clusters while trying to stand up geo-replication between 2 clusters using TLS authentication and authorization. Even though you set the --url-secure/--broker-url-secure params, it still tries to use the http 8080 port, etc.

@one70six
Copy link

one70six commented Jan 22, 2020

SOLVED

I figured it out, at least for v2.3.2!!!
There are a few things that need to be enabled in order to make the TLS broker to broker traffic work correctly:

  1. In the broker.conf, you must set the following parameters to TRUE. Even though it states that one it is Deprecated, it is still being used. Basically you need the replicationTlsEnabled and brokerClientTlsEnabled parameters set to TRUE

broker.conf

# Deprecated - Enable TLS when talking with other clusters to replicate messages
replicationTlsEnabled=true

# Authentication settings of the broker itself. Used when the broker connects to other brokers,
# either in same or other clusters
brokerClientTlsEnabled=true
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
brokerClientAuthenticationParameters=tlsCertFile:/opt/pulsar/certificates/broker.cert.pem,tlsKeyFile:/opt/pulsar/certificates/broker.key-pk8.pem
brokerClientTrustCertsFilePath=/opt/pulsar/certificates/ca.cert.pem
  1. If you are using a standalone cluster, then the standalone.conf also needs the same parameters from #1 above enabled and set, except for some reason the replicationTlsEnabled parameter can be left alone:

standalone.conf

#replicationTlsEnabled=true   <--- Not Needed, commented out here
brokerClientTlsEnabled=true
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationTls
brokerClientAuthenticationParameters=tlsCertFile:/opt/pulsar/certificates/broker.cert.pem,tlsKeyFile:/opt/pulsar/certificates/broker.key-pk8.pem
brokerClientTrustCertsFilePath=/opt/pulsar/certificates/ca.cert.pem

Works great after that!

@sijie
Copy link
Author

sijie commented Jan 23, 2020

@one70six thank you! I will update this gist.

@one70six
Copy link

one70six commented Jan 24, 2020

No problem @sijie, anytime!

@dockerzhang
Copy link

dockerzhang commented Jan 5, 2021

The flowing configuration has tested on v2.6.2 if enable TLS for geo-replication

Create TLS certificates

reference to ransport Encryption using TLS

Add configuration for broker.conf or standalone.conf

tlsEnabled=true
replicationTlsEnabled=true
brokerClientTlsEnabled=true
advertisedAddress=
brokerServicePortTls=6651
webServicePortTls=8443
brokerClientAuthenticationParameters=tlsCertFile:/data/test/pulsar/my-ca/broker.cert.pem,tlsKeyFile:/data/test/pulsar/my-ca/broker.key-pk8.pem
brokerClientTrustCertsFilePath=/data/test/pulsar/my-ca/certs/ca.cert.pem
tlsCertificateFilePath=/data/test/pulsar/my-ca/broker.cert.pem
tlsKeyFilePath=/data/test/pulsar/my-ca/broker.key-pk8.pem
tlsTrustCertsFilePath=/data/test/pulsar/my-ca/certs/ca.cert.pem

Others

if 9127 is merged, we only need set the --url-secure/--broker-url-secure params.

@sijie
Copy link
Author

sijie commented Jan 8, 2021

@dockerzhang much appreciated!

@marcoaureliox
Copy link

marcoaureliox commented Feb 9, 2021

Hi, nice demo for geo-replication!
For my use case, I need to connect 2 clusters and use JWT authentication, but when I try to test I get the error "Unable to authenticate: Failed to authentication token: JWT signature does not match locally computed signature. JWT validity cannot be asserted and should not be trusted. "
Where can I configure the token for geo-replication?

Thank you.

@flippedme
Copy link

flippedme commented Jul 19, 2021

@sijie stuck at 3a. with the below error.

02:12:00.040 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Error while consuming messages
02:12:00.041 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
org.apache.pulsar.client.api.PulsarClientException$BrokerPersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:271) ~[org.apache.pulsar-pulsar-client-api-2.4.0.jar:2.4.0]
at org.apache.pulsar.client.impl.ConsumerBuilderImpl.subscribe(ConsumerBuilderImpl.java:90) ~[org.apache.pulsar-pulsar-client-original-2.4.0.jar:2.4.0]
at org.apache.pulsar.client.cli.CmdConsume.consume(CmdConsume.java:162) [org.apache.pulsar-pulsar-client-tools-2.4.0.jar:2.4.0]
at org.apache.pulsar.client.cli.CmdConsume.run(CmdConsume.java:151) [org.apache.pulsar-pulsar-client-tools-2.4.0.jar:2.4.0]
at org.apache.pulsar.client.cli.PulsarClientTool.run(PulsarClientTool.java:133) [org.apache.pulsar-pulsar-client-tools-2.4.0.jar:2.4.0]
at org.apache.pulsar.client.cli.PulsarClientTool.main(PulsarClientTool.java:165) [org.apache.pulsar-pulsar-client-tools-2.4.0.jar:2.4.0]
02:12:00.047 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 0 messages successfully consumed

I got the same exception, no one care enough to solve this problem?

@Honey1991
Copy link

Honey1991 commented Feb 16, 2022

@sijie I am trying to create the cluster of Beijing, Shanghai
I can ping from client ( Beijing-->Shanghai and Shanghai-->Beijing ) which means these containers are on network.

I also verified zk, bk, broker for Beijing, Shanghai total 6 containers + 2 additional client containers are in network

When i try to execute below on client-beijing
bin/pulsar-admin clusters create --url http://broker-shanghai:8080 --broker-url pulsar://broker-shanghai:6650 shanghai

it gives me below error

null

Reason: javax.ws.rs.ProcessingException: Connection refused: broker-beijing/x.x.x.x:8080

Do i need to expose any port . Please suggest I am stuck

I have exactly followed this document

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment