Skip to content

Instantly share code, notes, and snippets.

@bergerx
Created July 15, 2015 16:35
Show Gist options
  • Save bergerx/f3538e023862504072ad to your computer and use it in GitHub Desktop.
Save bergerx/f3538e023862504072ad to your computer and use it in GitHub Desktop.
auto heal for mysql replication
TODO: not complete, not even close
# These would be needed to implement this
# * zk - for a zookeeper connection
# * marathon - this services entry in marathon, for getting information about service, peer instances and peers' health check
# init() and join() is application specific functions.
# init() is for initializing the app cluster and will also be the
# very first node in the cluster.
# join() will be called for adding the new instance (maybe replicas) to existing cluster,
# this will be called in case of any failures on other instances or
# make_me_master() will be called to convert a slave into a master when a master fails,
# this could make this node the new master
# TODO: this should also trigger a config change in other slaves
# TODO: each slave should know their uniq-id's
# This script is designed to replace docker entrypoint
def start(zk, marathon):
while 1:
if zk.lock(marathon.appname, marathon.my_id, timeout=30sec): # get a zookeeper lock, TODO: this should return true if i was the lock holder in the past
# this lock will be used one time in the services life,
# just to prevent race condition if we initially spawn
# several instances at the same time.
init() # will block until this instance dies
zk.relase()
else:
# go on only if we have a master
if master = marathon.get_master():
if wait_for_master_to_become_healthy(master, timeout=10):
fail hard
join(master) # if needed, we can also get and use other healthy pairs from marathon
if lock(marathon.appname, marathon.my_id):
make_me_master()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment