MSPileup partial pileup feature - candidate workflow

mspileup.md

End-user interaction with MSPileup and behavior of the server

This gist provides a candidate workflow for the partial pileup feature in MSPileup. For that purpose, note that:

1. MSPileup: is the RESTful web-service
2. MSPileupTasks: is the asynchronous daemon actually executing MSPileup tasks, including the partial pileup placement.

After many discussions in this PR: dmwm/WMCore#11807 and over Zoom as well, a candidate workflow has been identified and it provides the ability to: i) keep history of pileup containers and their fractions (who made the change and when as well) ii) identify when fraction changes and a new placement needs to be performed iii) identify whether the fraction increases or decreases

A description of the workflow follows:

1. End user creates a new pileup document with the mandatory fields:

{'pileupName': 'blah1', 'pileupType': 'premix', 'active': true, 'expectedRSEs': ['rse_A'], 'containerFraction': 1.0}

2. MSPileup persists this information in the database as follows (fields not-relevant are skipped for simplicity purposes):

{'pileupName': 'blah1', 'customName': '', 'containerFraction': 1.0, 'transition': [
{'customDID': 'blah1', 'DN': user1, 'containerFraction': 1.0 (initial fraction), 'updateTime': create timestamp}]}

3. MSPileupTasks read the pileup document and based on the containerFraction and the last transition record, it decides whether there is any change and/or if it's an increase or decrease fraction. NO, there are no changes to the fraction. --> it will follow the standard pileup data placement.

4. Next, the end user updates the pileup configuration with something like: {'pileupName': 'blah1', 'containerFraction': 0.5}

5. MSPileup persists information in the database as follows:

{'pileupName': 'blah1', 'customName': '', 'containerFraction': 0.5, 'transition': [
{'customDID': 'blah1', 'DN': user1, 'containerFraction': 1.0 (initial fraction), 'updateTime': create timestamp},
{'customDID': 'blah1-V1', 'DN': user1, 'containerFraction': 1.0 (previous fraction), 'updateTime': end user timestamp}]}

6. MSPileupTasks read the pileup document and based on the containerFraction and the last transition record, it decides whether there is any change and/or if it's an increase or decrease fraction. YES, there is a change AND it is a decrease.

7. then MSPileupTasks creates the new custom container, attachments and rules and if everything is successful, it updates the last transition record as (it updates top level customName; and customDID + previousFraction from the last transition record):

{'pileupName': 'blah1', 'customName': 'blah1-V1', 'containerFraction': 0.5, 'transition': [
{'customDID': 'blah1', 'DN': user1, 'containerFraction': 1.0 (initial fraction), 'updateTime': create timestamp},
{'customDID': 'blah1-V1', 'DN': user1, 'containerFraction': 0.5 (current fraction), 'updateTime': end user timestamp}]}

8. Next, the end user decides to update it once again with something like: {'pileupName': 'blah1', 'containerFraction': 0.75}

9. MSPileup persists information in the database as follows:

{'pileupName': 'blah1', 'customName': 'blah1-V1', 'containerFraction': 0.75, 'transition': [
{'customDID': 'blah1', 'DN': user1, 'containerFraction': 1.0 (initial fraction), 'updateTime': create timestamp},
{'customDID': 'blah1-V1', 'DN': user1, 'containerFraction': 0.5 (current fraction), 'updateTime': end user timestamp},
{'customDID': 'blah1-V2', 'DN': user1, 'containerFraction': 0.5 (previous fraction), 'updateTime': end user timestamp}]}

10. MSPileupTasks read the pileup document and based on the containerFraction and the last transition record, it decides whether there is any change and/or if it's an increase or decrease fraction. YES, there is a change AND it is an increase.

11. then MSPileupTasks creates the new custom container, attachments and rules and if everything is successful, it updates the last transition record as (it updates top level customName; and customDID + previousFraction from the last transition record):

{'pileupName': 'blah1', 'customName': 'blah1-V2', 'containerFraction': 0.75, 'transition': [
{'customDID': 'blah1', 'DN': user1, 'containerFraction': 1.0 (initial fraction), 'updateTime': create timestamp},
{'customDID': 'blah1-V1', 'DN': user1, 'containerFraction': 0.5 (previous fraction), 'updateTime': end user timestamp},
{'customDID': 'blah1-V2', 'DN': user1, 'containerFraction': 0.75 (current fraction), 'updateTime': end user timestamp}]}

and so on. In summary:

• MSPileup task creates a new transition record (with a snapshot of the current containerFraction)
• MSPileupTasks updates the latest transition record (with the new container fraction and custom pileup name)

TODO: we have also considered adding a state field to the transition record, such that we can easily see whether that fraction update has already been processed by the system or not. Discussions about the granularity of this field need to happen as well (is boolean good enough? or we want to use something more meanigful for debugging?)

Logic for decreasing container fraction

1. Imagine that the original pileup has 4 blocks:

orig_pileup = [block1, block2, block3, block4]

2. then Person_A makes a call for partial pileup with containerFraction=0.5, hence decreasing the pileup availability.
3. MSPileupTask server creates a container like (note that block assignment could be undeterminstic):

list_total_blocks = all_block_names_in(orig_pileup)
portion = ceil(list_total_blocks * containerFraction)  # equals to 2 in this case
# next we create a container, which here we call custom_pileup_1
Next we need to identify which blocks will have to be attached to custom_pileup_1 (based on portion)
custom_pileup_1 = list_total_blocks[:portion]  # now we attach these blocks to custom_pileup_1
# example final custom container
custom_pileup_1 = [block1, block2] 

4. then Person_B makes a call for partial pileup with containerFraction=0.25, hence decreasing again the pileup availability.
5. MSPileupTask server creates a second custom container, example:

list_total_blocks = all_block_names_in(orig_pileup)
list_custom_blocks = all_block_names_in(custom_pileup_1)
portion = ceil(list_total_blocks * containerFraction)  # equals to 1 in this case
# next we create a container, which here we call custom_pileup_2
Next we need to identify which blocks will have to be attached to custom_pileup_2 (based on portion)
custom_pileup_2 = list_custom_blocks[:portion]  # now we attach these blocks to custom_pileup_2
# example final custom container
custom_pileup_2 = [block1] 

Logic for increasing container fraction

1. Imagine that we have a custom pileup already in the system with fraction of 25% (containerFraction=0.25)

custom_pileup_x = [block1]
# based on an original pileup with 4 blocks, e.g.:
# orig_pileup = [block1, block2, block3, block4]

2. then Person_A makes a call for partial pileup with containerFraction=0.5, hence increasing the pileup availability.
3. MSPileupTask server creates a container like (note that block assignment could be undeterministic):

list_total_blocks = all_block_names_in(orig_pileup)
list_custom_blocks = all_block_names_in(custom_pileup_x)
portion = ceil(list_total_blocks * containerFraction)  # equals to 2 in this case
# next we create a container, which here we call custom_pileup_y
Next we need to identify which blocks will have to be attached to custom_pileup_y (based on portion)
custom_pileup_y = list_custom_blocks  # now we attach these blocks to custom_pileup_y
while len(custom_pileup_y) == portion:
    for block in list_total_blocks:
        if block not in custom_pileup_y:
            add this block to custom_pileup_y
            break
# example final custom container
custom_pileup_y = [block1, block2] 

4. then Person_B makes a call for partial pileup with containerFraction=0.75, hence increasing again the pileup availability.
5. MSPileupTask server creates a second custom container, example:

list_total_blocks = all_block_names_in(orig_pileup)
list_custom_blocks = all_block_names_in(custom_pileup_y)
portion = ceil(list_total_blocks * containerFraction)  # equals to 3 in this case
# next we create a container, which here we call custom_pileup_z
Next we need to identify which blocks will have to be attached to custom_pileup_z (based on portion)
custom_pileup_z = list_custom_blocks  # now we attach these blocks to custom_pileup_z
while len(custom_pileup_z) == portion:
    for block in list_total_blocks:
        if block not in custom_pileup_z:
            add this block to custom_pileup_z
            break
# example final custom container
custom_pileup_z = [block1, block2, block3] 

The increase/decrease logic needs to keep in mind that we always want to minimize data (block) transfers.