auto heal for mysql replication

gistfile1.txt

TODO: not complete, not even close

# These would be needed to implement this
# * zk - for a zookeeper connection
# * marathon - this services entry in marathon, for getting information about service, peer instances and peers' health check

# init() and join() is application specific functions.
# init() is for initializing the app cluster and will also be the
#        very first node in the cluster.
# join() will be called for adding the new instance (maybe replicas) to existing cluster,
#        this will be called in case of any failures on other instances or 
# make_me_master() will be called to convert a slave into a master when a master fails,
#                  this could make this node the new master
#                  TODO: this should also trigger a config change in other slaves

# TODO: each slave should know their uniq-id's

# This script is designed to replace docker entrypoint
def start(zk, marathon):
  while 1:
    if zk.lock(marathon.appname, marathon.my_id, timeout=30sec): # get a zookeeper lock, TODO: this should return true if i was the lock holder in the past
      # this lock will be used one time in the services life,
      # just to prevent race condition if we initially spawn
      # several instances at the same time.
      init() # will block until this instance dies
      zk.relase()
    else:
      # go on only if we have a master
      if master = marathon.get_master():
        if wait_for_master_to_become_healthy(master, timeout=10):
           fail hard
        join(master) # if needed, we can also get and use other healthy pairs from marathon
        if lock(marathon.appname, marathon.my_id):
          make_me_master()