Relevance was always based on subject matter experts.
We measure based on the relevance of the query, despite the fact that other documents may be relevant. It needs to pertain to the query -- that's what we care about.
relevant | non-relevant | |
---|---|---|
retrieved | A & B | ~A & B |
not retrieved | A & ~B | ~A & ~B |
Essentially getting a measure of non-relevant retrieved items, out of the all non-relevant. Is a good indicator of things that aren't relevant.
Total of Non Relevant and Retrived / Non Relevant
| Ā & B |
----------
| Ā |