This document serves as my personal QA judging guideline in relation to C4 contests.
- QA: Quality Assurance
- L: Low-Risk
- NC: Non-Critical
- I: Informational
- SC: Supreme Court
QA submission scores are based on the following formula, slightly reminiscent of C4's:
QA Score = L * 10 + NC * 3 + I
L = Low-Risk Findings
NC = Non-Critical Findings
I = Informational / Style Findings
At first, all QA reports are manually reviewed and their valid submissions are noted. If a QA report lacks sufficient quality, it will be ignored from the scoring formula.
For each submission graded, a dedicated response is provided to the relevant issue that details the submission's score as well as which segments of the QA report were considered as valid.
The best report is selected based on the top score, provided that there is no conflict (i.e. two reports having the same score).
Grade A reports are selected based on the top Math.min(5% of submissions, 5) reports with a minimum score of 20 score-wise provided that there is no conflict.
Grade B reports are selected based on a minimum score of 10.
If any conflict arises, it is manually resolved by a review of the overall quality of the submission.
Likewise, the above serves merely as a guideline and the ultimate grade is manually assessed/confirmed.
Manual review is a mandatory step as the comparative quality of reports needs to be assessed rather than their point-based quality.
Any finding in a QA report that is deemed valid, has not been identified in the bot report, and does not pertain to a gas optimization is considered informational. These involve (non-exhaustively):
- Potential Code Refactors
- Introductions of Events
- Yielding of Variables in Mutable Calls
- Typographic / Misleading Name Adjustments
The burden of proof in these types of submissions is greatly minimized and the bare requirement for a valid submission is to highlight what code is wrong and how to fix it, without necessarily rationalizing their impact correctly.
Rationalization does play a role in whether a report is graded of insufficient quality, however, no penalization is applied if f.e. a QA of NC was submitted as L, an M risk was submitted as NC, and so on.
Any medium-risk (or higher) vulnerability that was submitted as part of a QA report will be credited an L rating.
At my discretion, if a Warden has sufficiently argued in the QA report that the submission may be upgrade-able and they have focused adequately on the issue, I may reward them a reduced percentage out of the report's pool.
Any finding that falls in the following categories is entirely ignored when scoring QAs:
- Centralization Criticisms (Suitable for Analysis per relevant SC verdict)
- Gas Optimizations (Suitable for Gas Reports)
- Invalidated Issues (Searchable in the Findings Repository)
- Incorrect Statements
- Generic / Insufficiently Precise Submissions
Each contest judged is attached to a single instance of this document based on the revision hashes. As such, changes to this document do not retroactively apply and are merely future enhancements. I am more than happy to discuss my decisions above with any Warden, Judge, or interested party alike so feel free to reach out to my Discord at 0xsomeone.