Skip to content

Instantly share code, notes, and snippets.

@slaporte

slaporte/work-allocation.md

Last active October 14, 2017 22:55
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save slaporte/40f1cfb7e7eebd73b8ca0a0f21455865 to your computer and use it in GitHub Desktop.
Save slaporte/40f1cfb7e7eebd73b8ca0a0f21455865 to your computer and use it in GitHub Desktop.
Download ZIP
Montage work allocation
Raw
work-allocation.md

Custom work allocation in Montage

Input

  • List of new jurors, with both username and work allocation
  • List of old jurors to discard votes: the default approach will be to preserve votes
  • Quorum: if the new quorum is less than the existing quorum, then we also need a discard strategy. We can either discard randomly, or keep best votes

Work allocation

Think of the work allocation as the percentage of photos that the juror will evaluate. Nobody can see more than 100%. Work allocations may be between 0.0 to 1.0 per juror, and the total of all juror's work allocations must add up to the quorum.

For example:

Entries = 200
Quorum = 2
Voting tasks = 400

New jurors:
(Mahmoud, 1.0) = 200 tasks
(Stephen, 0.5) = 100 tasks
(Pawel, 0.5) = 100 tasks

If we adjust work allocation in a round (without discarding votes), then we should count the jury's submitted votes as existing work, or minimum new work allocation. We should think of the work allocation as a description of the total work in the round, not just the open votes that are yet to be submitted. I think this makes it more intuitive for coordinators and jurors: they would prefer that juror workloads do not change unless it's explicitly adjusted.

For example: Mahmoud has finished 50% of his votes (100 votes submitted), then his existing work is 0.5 at a minimum. If we lower his work allocation from 1.0 to 0.5, then he will get no further tasks and we will need to adjust Stephen and Pawel's work allocation accordingly to add up to quorum.

Question: if we decrease someone's work allocation below their existing work, should it discard votes?

Process

  • Remove jurors
  • If applicable, discard votes from removed jurors (default strategy is preserve)
  • If applicable, requalify entries based on removed jurors and disqualify entries based on new jurors
  • Adjust quorum
  • If quorum has decreased and there are existing votes, discard tasks according to the discard strategy (either random or keep best; I think keep best should be default)
  • Calculate existing work, to confirm the new allocation is valid.
  • Add new jurors
  • Calculat the new work allocation: the difference between the existing work allocation and the submitted work allocation.
  • Assign open votes based on new work allocation

We should keep a separate function for assigning votes based on the work allocation, in case the coordinator adds new entries to the round.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment