hostname-hash-bucketing-restart.py

import random

# Number of VMs that will be restarted.
N=16
# Bucket the nodes randomly by hashing their hostnames and putting them in hourly buckets.
BUCKETS = 7*24
# Number of simulations we will make.
SIMULATIONS=100000

collisions = 0
for i in range(SIMULATIONS):
    buckets = set()
    for j in range(N):
        bucket = random.randint(0, BUCKETS)
        if bucket in buckets:
            collisions += 1
            break
        buckets.add(bucket)

print 1.0 * collisions / SIMULATIONS

random-sleep-restart.py

import random

# Random number of minutes we sleep before a security upgrade.
RANDSLEEP=7*24*60
# Number of minutes a restart takes.
RESTART=10
# Number of VMs that will be restarted.
N=16
# Number of simulations we will make.
SIMULATIONS=100000

collisions = 0
for i in range(SIMULATIONS):
    # Generate the restart times for all VMs.
    samples = [RANDSLEEP * random.random() for i in range(N)]
    samples.sort()
    for j in range(N-1):
        if samples[j+1]-samples[j] < RESTART:
            # We found two nodes that would be down within the RESTART
            # interval. That is, both would be down.
            collisions +=1
            break

print 1.0 * collisions / SIMULATIONS

$ python random-sleep-restart.py
0.21198
$ python hostname-hash-bucketing-restart.py
0.52245

First strategy has a 21% chance of VMs restarting at the same time. Second strategy has a 50% change of VM restarts colliding.