Project: Apache DolphinScheduler: Support etcd as registry
Related Issue: apache/dolphinscheduler#8975
Code Repository: https://github.com/wjf222/dolphinscheduler
Mentors: Wenjun Ruan,Caishunfeng
The registry of DolphinScheduler is mainly used to store metadata information of Master and Worker nodes. Currently dolphinscheduler-registry-api has defined the standard for implementing the plugin.
Currently DolphinScheduler has implemented the Zookeepeer plugin as a registry, Zookeeper uses the client-side framework Curator to implement the following interfaces, interface functionality is divided into several main parts: CRUD, Subscribe (Watch), Auth, Sessiontime Lock, Active close,ConnectionStateListen.
Etcd is a distributed key-value storage system with excellent mechanisms for read and write performance, consistency and availability, making it ideal for the configuration center role. This can help user who only familiar with etcd to use dolphinscheduler.
This module is the etcd registry plugin module, this plugin will use etcd as the registry center.
If you want to set the registry center as etcd, you need to set the registry properties in master/worker/api/appplication.yml
registry:
type: etcd
endpoints: "http://etcd0:2379, http://etcd1:2379, http://etcd2:2379"
# The options below have default values
namespace: dolphinscheduler
connectionTimeout: 9s
# The unit is milliseconds
retryDelay: 60ms
retryMaxDelay: 300ms
retryMaxDuration: 1500ms
# The following options are set according to personal needs
user: ""
password: ""
authority: ""
loadBalancerPolicy: ""After do this config, you can start your DolphinScheduler cluster, your cluster will use etcd as registry center to store server metadata.
Gsoc allowed me to work on a large project for the first time and implement a feature. Community members have actively reviewed my code, which has greatly improved the code I wrote compared to the original version, and gained a deeper understanding of non-code issues such as licenses, specifications, and ease of use. Special thanks to Mr. Wenjun for his help.
Thank you
Regards
Wenjun Ruan,Caishunfeng