jiazhai

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0

jiazhai

KOP support Kafka listeners config of type "PLAINTEXT" and "SSL". You could set config like listeners=PLAINTEXT://localhost:9092,SSL://localhost:9093. Please reference Kafka SSL document for how to config SSL keys. Here is some steps that you need to be able to connect KOP through SSL.

1. create SSL related Keys.

Here is an example of a bash script to create related CA and jks files.

jiazhai

Steps for KoP authentication and authorization Based on JWT.

Please reference to Pulsar Security Overview to understand the Pulsar authen and author. Currently KoP support Pulsar Token auth

Before start, preparing a KoP tar ball and a Pulsar tar ball(mainly use puslar-admin tools in this tar ball).

Some background before tutorial

In KoP, we currently support SASL_PLAINTEXT, in which a username/password pair needs provide. We reused the Token provider in Pulsar to implement a KoP SASL authentication, the authorization still use Pulsar authorization.

jiazhai

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0

jiazhai

This doc demonstrates how to do geo-replication across multiple pulsar clusters without a global configuration store (zookeeper).

This demo is using docker-compose to start 3 pulsar clusters. Each pulsar cluster has 1 zk, 1 bk, and 1 broker. The docker compose file can be found at https://gist.github.com/jiazhai/72aef9bfbfdd3fe421084c2973247a64

The information of all the three clusters is listed in the following table:

	zk	configuration store	broker

jiazhai

This is based on https://gist.github.com/jiazhai/1cd48ab6c8f6c3012c895df9746799b5

0. prepare a docker image

1， build it from source code:

mvn install -DskipTests -Pdocker   

2， or use docker pull

jiazhai

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0

jiazhai

jiazhai

目标：

在现有的Share的订阅模式之上，添加一种新的订阅模式Key_Failover。这种订阅模式，可以保证同一个订阅中的多个consumer可以共享订阅同一个partiton；同时又可以保证partiton中的每一个Key对应的消息只被发送到一个consumer。

现状和要解决的问题

现有的订阅模式中，share模式可以对同一个partition中的消息进行共享的消费，可以高效的清除partition的backlog。 但是在很多场景中，顺序性也是很常见的需求。在share模式中，我们不能保证同一个Key对应的消息会被发送到固定的某一个consumer中。每个consumer对消息的顺序性得不到保证。

我们想要在新的订阅模式Key_Failover中，保证同一个订阅中的多个consumer可以共享订阅同一个partiton；同时又可以保证partiton中的每一个Key对应的消息只被发送到一个consumer。

解决方法

jiazhai

tenant: "public"
namespace: "default"
name: "debezium-postgres-source"
topicName: "debezium-postgres-topic"
archive: "connectors/pulsar-io-debezium-postgres-2.4.0-SNAPSHOT.nar"

parallelism: 1

configs:
  ## config for pg, docker image: debezium/example-postgress:0.8