Skip to content

Instantly share code, notes, and snippets.

@scottsb
Last active October 25, 2021 20:28
Show Gist options
  • Save scottsb/cc4a10b57314f63febe2 to your computer and use it in GitHub Desktop.
Save scottsb/cc4a10b57314f63febe2 to your computer and use it in GitHub Desktop.
Guide to Resetting a csync2 Cluster

Guide to Resetting a csync2 Cluster

Introduction

These are possible steps to reset a csync2 cluster that has been seriously fubared. This is an apocalyptic approach and should only be used when more surgical fixes (like correcting an individual conflict) aren't workable.

Use Cases

This will solve errors like:

    ERROR from peer 10.0.0.1: File is also marked dirty here!

And, depending on the cause, it may fix errors like:

    Database backend is exceedingly busy => Terminating (requesting retry).
    ERROR from peer 10.0.0.1: Connection closed.

Important Notes

References to lsyncd assume you're triggering csync2 via lsyncd (as with puppet-clustersync) but can be safely ignored if that doesn't apply to you.

As written these commands do not specify a csync2 config file. If you are using csync2 with a named config, each csync2 command should include -C [config-name].

Instructions

Phase 1

  1. On all servers, forcibly disable all the automatic syncing:

     service lsyncd stop
     killall csync2
    
  2. Choose one server that's closest to the correct state and manually adjust files as needed to make it the authoritative master.

  3. On all servers, update csync2 database to match filesystem but without marking anything as needing to be synced out to other servers:

     csync2 -cIr /
    

Phase 2

Run all commands in this phase from the authoritative server selected in the last phase.

  1. Find all differences between authoritative server and remotes and mark for sync:

     csync2 -TUXI
    
  2. Reset database to force current server to be winner on any conflicts:

     csync2 -fr /
    
  3. Run a sync to all other servers:

     csync2 -xr /
    

Phase 3

  1. On all servers, confirm that all files are in sync (if successful will list no files):

     csync2 -T
    
  2. On all servers, restart automatic sync service:

     service lsyncd start
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment