Skip to content

Instantly share code, notes, and snippets.

@yangl

yangl/Pulsa集群模式最佳实践.md

Last active December 16, 2022 08:57
Show Gist options
  • Save yangl/a00d117ea1d80ad9f02b639b3a29059e to your computer and use it in GitHub Desktop.
Save yangl/a00d117ea1d80ad9f02b639b3a29059e to your computer and use it in GitHub Desktop.
Download ZIP
Pulsar二进制预编译包建议使用StreamNative分发的二进制包 https://github.com/streamnative/pulsar/releases
Raw
Pulsa集群模式最佳实践.md

zk集群划分：

  • metadataStore 单独集群；
  • bookie & configurationMetadataStore 共享集群；

bookie跨机房大集群共享存储、broker层面独立小集群

注：bookie与metadataStore使用不同的zk集群时，pulsar目前的默认机架感知策略实现是有bug的，该bug会导致机架感知策略不对&数据恢复时报错等。（该bug可关注 https://github.com/apache/pulsar/pull/18672#issuecomment-1347634103）

ZK集群启动

初始化metadata

  1. bookie初始化
export BOOKIE_CONF=/app/conf/bookkeeper.conf && ./bookkeeper shell metaformat

org.apache.bookkeeper.bookie.BookieShell
--conf conf/bookkeeper1.conf metaformat
  1. broker初始化

注:existing-bk-metadata-service-uri记得加上最后的/ledgers

org.apache.pulsar.PulsarClusterMetadataSetup

export PULSAR_BROKER_CONF=/app/conf/broker.conf && ./bin/pulsar initialize-cluster-metadata \
  --cluster sf-pulsar-1 \
  --metadata-store zk:10.207.128.13:2181,10.207.128.14:2181,10.207.128.15:2181,10.207.128.16:2181,10.207.128.17:2181/sf-pulsar-1 \
  --configuration-metadata-store zk:10.207.128.13:2181,10.207.128.14:2181,10.207.128.15:2181,10.207.128.16:2181,10.207.128.17:2181/sf-pulsar-1 \
  --existing-bk-metadata-service-uri "zk+hierarchical://10.206.128.154:2181;10.206.128.155:2181;10.206.128.156:2181;10.206.128.157:2181;10.206.128.158:2181/bk1/ledgers" \
  --web-service-url http://10.206.64.12:18080 \
  --broker-service-url pulsar://10.206.64.12:16650
  1. 事务初始化
org.apache.pulsar.PulsarTransactionCoordinatorMetadataSetup

export PULSAR_BROKER_CONF=/app/conf/broker.conf && ./pulsar initialize-transaction-coordinator-metadata --cluster sf-pulsar-1 --configuration-store 10.206.128.212:2181,10.206.128.213:2181,10.206.128.214:2181,10.206.128.215:2181,10.206.128.216:2181/pconfig

### BookieServer 启动三个


```shell

org.apache.bookkeeper.proto.BookieServer

-Dlog4j2.configurationFile=D:\opensource\pulsar\conf\log4j2.xml

--conf conf/bookkeeper1.conf

bookkeeper1.conf bookkeeper2.conf bookkeeper3.conf 配置主要配置如下参数

journalDirectories=/app/data/bookkeeper/journal
ledgerDirectories=/app/data/bookkeeper/ledgers

metadataServiceUri=zk+hierarchical://10.206.128.154:2181;10.206.128.155:2181;10.206.128.156:2181;10.206.128.157:2181;10.206.128.158:2181/bk1

useHostNameAsBookieID=true
autoRecoveryDaemonEnabled=false

httpServerEnabled=true
httpServerPort=8000
httpServerClass=org.apache.bookkeeper.http.vertx.VertxHttpServer

enableStatistics=true
statsProviderClass=org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider
prometheusStatsHttpPort=8000

PulsarBrokerStarter 启动两个

org.apache.pulsar.PulsarBrokerStarter

-Dlog4j2.configurationFile=D:\opensource\pulsar\conf\log4j2.xml

--broker-conf conf/broker1.conf

broker1.conf broker2.conf 配置主要配置如下参数

metadataStoreUrl=zk:10.206.128.154:2181,10.206.128.155:2181,10.206.128.156:2181,10.206.128.157:2181,10.206.128.158:2181/sf-pulsar-1
configurationMetadataStoreUrl=zk:10.206.128.154:2181,10.206.128.155:2181,10.206.128.156:2181,10.206.128.157:2181,10.206.128.158:2181/pconf
brokerServicePort=16650
webServicePort=18080
clusterName=sf-pulsar-1
ttlDurationDefaultInSeconds=86400
allowAutoTopicCreation=false
allowAutoTopicCreationType=partitioned
allowAutoSubscriptionCreation=false
brokerDeleteInactiveTopicsEnabled=false
brokerDeduplicationEnabled=true
defaultNumberOfNamespaceBundles=16
maxTopicsPerNamespace=1024
enableNonPersistentTopics=false
authenticationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken
authorizationEnabled=true
authorizationAllowWildcardsMatching=true
superUserRoles=sfadmin
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
brokerClientAuthenticationParameters=token:eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJzZmFkbWluIn0.H1Rbd9PM_APogpyK26G8qGvNF4zxJWA0Wy7XmvJqKPibP8xVxhz1Y264_pkKyAZ4CKrkTXx3975Hyi9aeXLo1Vj71j7PbIyg4QqR0dKRT66PmklLbqMOxlzmZiY92pE6_wAMKwexBkvBEwCxmKcWzNs3G6G9AVRPq82BWXJsEcXt1X0LIsAT4-drpU_JqB_ZiuiUu3tolVH00sj5B3rQJzzaS4b6whBJVVZxY1PfspKunrEYYLzMJOw2O7FUiNvv2_wk-IVXwFrmq9UaDklNSp8kKx0F4pzx1mEPWHFGILH4vOrPbZYhn9F0EeTdU7_13WVyaC156VWulOdjTQ5Bfw
tokenPublicKey=file:///app/conf/sf-pulsar-1-public.key
bookkeeperMetadataServiceUri=zk+hierarchical://10.206.128.154:2181;10.206.128.155:2181;10.206.128.156:2181;10.206.128.157:2181;10.206.128.158:2181/bk1
bookkeeperClientRackawarePolicyEnabled=true
bookkeeperClientRegionawarePolicyEnabled=false
bookkeeperClientEnforceMinNumRacksPerWriteQuorum=true
bookkeeperClientMinNumRacksPerWriteQuorum=2

managedLedgerDefaultEnsembleSize=3
managedLedgerDefaultWriteQuorum=3
managedLedgerDefaultAckQuorum=2

managedLedgerMaxEnsembleSize=9
managedLedgerMaxWriteQuorum=6
managedLedgerMaxAckQuorum=4

defaultRetentionTimeInMinutes=4320
defaultRetentionSizeInMB=-1
transactionCoordinatorEnabled=true

loadBalancerOverrideBrokerNicSpeedGbps=1

单独启动autorecovery

export PULSAR_BOOKKEEPER_CONF='/app/conf/bookkeeper-autorecovery.conf'
sh pulsar/bin/pulsar-daemon start autorecovery
###### bookkeeper client url config #####
metadataServiceUri=zk+hierarchical://10.206.128.154:2181;10.206.128.155:2181;10.206.128.156:2181;10.206.128.157:2181;10.206.128.158:2181/bk1

###### bookkeeper client rack config #####
ensemblePlacementPolicy=org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy
reppDnsResolverClass=org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping
minNumRacksPerWriteQuorum=2
enforceMinNumRacksPerWriteQuorum=true

##### BookieRackAffinityMapping config #####
metadataStoreUrl=zk:10.206.128.154:2181,10.206.128.155:2181,10.206.128.156:2181,10.206.128.157:2181,10.206.128.158:2181/sf-pulsar-1
zkTimeout=3000

BK机架内置原生脚本感知

https://gist.github.com/yangl/63e02ff7daaac3e6a4d06c3327bea65e

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment