sweeper description

The 3 modules use the sweeper:
AAE tree rebuild
Object expiry
Tombstone reaper

The sweeper adds support for modules to register fold jobs they would like to have scheduled.
This mechanism lets us perform a number of processing tasks on the IOPs/bandwidth needed to sweep the data.
The sweeper keeps track of which partition that needs to be scheduled and ask all partispants if they would like to join.

Object expiry:
New metadata for TTL <<"X-Riak-TTL”>> riakc_object.erl clear_ttl/1 get_ttl/1 set_ttl/2
Also possible to set TTL on buckets with bucket properties.
Riak delete expired keys that are accessed and return not_found.
The sweeper is used to delete expired keys that are not accessed.

Tombstone reaper:

Backround:
"On the downstream clusters if realtime replication does not deliver the delete 
(the realtime queue overloads, overflows, exits before allobjects transfers or crashes)
and the object tombstone is removed before before a fullsync, the object continues to exist in the remote cluster(s). 
This is consuming more and more disk space over time for them and needs to be resolved."
One solution to this is to use delayed tombstone reaps.
But this use timers which increase memory use and can create congestion in the erlang vm timer handling.

With the sweeper we can use keep all tombstone for a longer period to make sure they are replicated.
The sweeper can then reap tombstone when they are older then the configured grace period.
The grace period needs to be configured to make sure all objects gets replicated. Default 1w.

AAE tree rebuild
Before the AAE tree would run a fold once per week when it had expired.
Now with the sweeper it will schedule a sweep once per week instead.
Since the sweeper have support for sweep windows it’s also possible to move these rebuild to the clusters quiet period.

config:

## Limit how many sweeps can happen concurrently.
sweeper.concurrency_limit = 1

## Same as bitcask.merge.policy
riak_kv.sweep.policy = always 
## riak_kv.sweep.policy = window
## riak_kv.sweep.window.start = 3
## riak_kv.sweep.window.end = 17

tombstone.grace_period = off  %% Configure how long tombstones are kept until the sweeper are allowed to reap them.
                              %%  Grace period should be long enough for all tombstone to be replicated.
tombstone.sweep_interval = 1d %% How often should the reap sweep be runned
## Tombstone reap sweeps inactive with grace period off
## The sweep reaper works independently of the delete mode.
## To get the expected behavior with tombstone only reaped by the swepper
## the user user should use delete_mode=keep to keep the tombstone until the sweeper run.
## Should we force delete_mode=keep when tombstone.grace_period enabled?

obj_ttl.sweep_interval = off
# obj ttl sweeps inactive with sweep_interval off

# AAE tree rebuild sweep use anti_entropy.tree.expiry as sweep interval

Description how object get expired and deleted by sweeper.
https://docs.google.com/presentation/d/1Co71htx0AtXzXOsGCCx5ZzFiw5cSl-tkzDvC6-Ovw-I/pub?start=false&loop=false&delayms=3000&slide=id.gfb3564a29_0_305