Decentralized Judging Idea

gistfile1.txt

Potential judging setup


Note: This may eventually require a 20 USDC deposit from each competitor for each contest (so people have something to lose). I think we’re moving in this direction eventually anyways. 

Contest ends
3-day “public” judging period begins
Judging repo with all issues is open-sourced
For each contest, 5% of contest cost will become a judging pot (For $75k contest this looks like: $50k contest pot, $20k senior pay, $3.5k judging pot, $1.5k to Sherlock)
Anyone can comment on any issue and “judge it.” Example format:

“Severity: High
Duplicate: #12”


Severity params: High, Medium, non-issue
Duplicate params: no, #X (X represents the “lowest issue number” that’s a duplicate of this issue)


First comment (per issue) gets the credit. If the first issue is incorrect, the second, third, etc. comments are checked. 

Scoring works like this:

Correct High: 5x weight in the judging pot; Incorrect High (false positive or false negative): -10x weight PENALTY
Correct Medium: 3x; Incorrect Medium (false positive or false negative): -6x
Correct non-issue: 1x; Incorrect non-issue (false positive or false negative): -2x
Correct duplicate: 1x; Incorrect duplicate (false positive or false negative): -2x
Duplicate is only correct if it references the lowest issue # that is a duplicate of the issue


Public judging ends after 3 days.

Sherlock judge comes in to check each issue. Comments on each issue with format:

“Severity: High
Duplicate: #12

Best duplicate: Yes”

The earliest comment that matches the judge’s comment is rewarded, any earlier comments that do not match the judge’s are penalized. 
A script will 1) create issue labels to match the judge’s comment 2) close non-“best” duplicate issues 3) Make a new comment for each “duplicate” that re-classifies it as a duplicate of the best issue (i.e. Duplicate of #16)

Then the protocol team can review the labeled issues that are still open. 

Later in the process, another script will award/deduct points from the public judging issues based on whether the issue’s final state matches a public judge’s comment. 

This can be sibyl attacked, I (script) can create 30 new accounts, make a single comment "Duplicate of #1", "Duplicate of #2" and if the 25th account get's rewarded I make a profit as the other 29 get a penalty but they having nothing to lose. So as the top comment suggests we need a 20 USDC deposit (something to lose) but building this functionality in the frontend will take significant development effort.

Let's assume the sibyl attack was solved, I don't think it's the most efficient model as the deduplication suggestions are sorted by "least recently suggested", sorting them by "highest quality" (which option B) is aiming for) will make it more efficient, as the first suggestion is more likely to be valid.

(edit) It's actually possible to sort the deduplication suggestions by "highest quality" with this model as well, assuming we keep a reputation score in the back. The judge doesn't have to look at the least recent suggestion first, he can go over the suggestions in any order.

So than it boils down to the incentives for the judges, speed = everything in this model, which incentivizes a race to the bottom where people leave suggestions when they are 90% certain but can be first (85% certain but can be first will be a better strategy), instead of 99% certain after looking at the issues for a couple hours extra. Because the judging period is a fixed period either way (3 days), I'm convinced that making that the only time constraint will result in higher quality suggestions overall. Option B) is aiming for this.

Ok, noted