- RFC from mods
Post follows.
TL;DR: Charcoal is the organization behind SmokeDetector. Since January 2017, we've been casting up to three flags automatically on posts that our systems are confident are spam, on behalf of users who have opted in to that system. We'd like to increase that to four or five automatic flags to reduce the time that spam is alive on the network.
Charcoal is a user-run organization that is primarily responsible for the spam-detecting bot SmokeDetector. Over the past four years, SmokeDetector has helped us find spam on the Stack Exchange network to flag and destroy manually. In January 2017, with the blessing of Stack Exchange, we started running an "autoflagging" project, wherein our systems automatically cast up to three flags on a post if they're confident that it's spam, under the names of willing users. If you missed that happening entirely, we wrote a meta post on Meta Stack Exchange - or there's a slightly more concise explanation on our website.
Great! We currently have 215 users who have opted in to the autoflagging system (you can sign up too, if you're interested). Our system has autoflagged around ~30,000 posts (29,592 to be exact), of which the vast majority (29,526) were confirmed to be spam - that's 99.7% accurate.
We'd like to expand our autoflagging system. Currently, we cast up to three flags on posts we're confident are spam (on most sites, see below); we'd like to increase that number to four or five flags.
Just so we're up-front about this: this is an experiment. Ultimately, we're trying to do these things:
- Reduce the time that spam remains on the sites before being deleted
- Lower the number of humans who have to see or interact with spam manually
Increasing the number of flags we cast on spam automatically should accomplish both of these things:
- Automatic flags are near-instant; manual flags can take minutes to be cast - that means that increasing the ratio of automatic to manual flags results in a shorter time before six flags accumulate and the spam is deleted.
- Automatic flags are not cast by humans. Fewer humans, therefore, are forced to see or interact with the spam.
The data we have backs this up. In terms of time to deletion, we saw a significant drop in the time it took to delete spam when we began our autoflagging project. Take a look at this graph from the meta post on the subject for an excellent visual representation of that. Before we started autoflagging, spam spent an average of 400 hours per day alive across the network; with autoflagging in place, the average is around 40 hours per day, reduced by a factor of 10.
If this change goes ahead, these things are likely to happen:
- It will only take one or two manual flags from users to spam-nuke an autoflagged post, instead of the current three. Posts that are not autoflagged will, of course, still require six flags to nuke.
- There may be an increase in posts spam-nuked entirely by Charcoal members, who may not be active on the site.
- You will see a reduction in the time spam spends on the site before being deleted.
- Fewer humans will have to see each spam post.
The last two of those things are indisputably good things. The first two, however, are more controversial, and are the reason we want to have a discussion here on meta before we make this happen. What follows are the major concerns we've seen, and what we can do about them or why we don't think they're an issue - we'd like to hear your thoughts.
Spam flags are a powerful feature that need some care in applying correctly. This is a concern that came up when we originally built the autoflagging system, so we already have safeguards built in.
- We only autoflag a post if we're more than 99.5% sure it's spam. (Technically, the precise condition certainty varies by conditions set by the users who have opted in to autoflagging, but it's always above 99.5% - more detail on that on our website).
- If the system breaks down or bugs out and starts flagging things it shouldn't, all Charcoal members and all network moderators have access to a command that immediately halts all flagging activity and requires intervention from a system administrator to re-enable. Outside of testing, no one has had to use the kill switch thus far.
- We never unilaterally nuke a post. There are currently three manual flags required in addition to the automatic flags to nuke a post; this increase proposal still retains at least one manual flag.
We also make sure that everything has human oversight at all times. While only three humans currently have to manually flag the post, there are always more users than that reviewing the system's decisions and classifications; if a post is flagged that shouldn't have been, we are alerted and can alert the relevant moderators to clear the spam flags. Again, this is very rare: over the past year, we've flagged a mere 66 posts that shouldn't have been, compared to 29,592 spam posts (that's 99.7% accurate overall). We allow users to set their own flagging conditions, provided they don't go below our baseline 99.50% certainty. We recommend, however, that users set a certainty of 100.00% - those who set their conditions below that are likely to see more false positives flagged using their account.
Finally, as this proposal decreases the required manual involvement to nuke a post, to compensate for that lower human-involvement barrier, we will correspondingly increase the required accuracy before casting the extra automatic flags. For example, we currently require 99.5% accuracy before casting autoflags; while we haven't decided on concrete numbers yet, it would be possible to require 99.9% accuracy for four autoflags, and 99.99% accuracy for five autoflags. (For reference, humans are accurate 95.4% of the time, or 87.3% on Stack Overflow - those are stats that jmac (a former Community Manager) looked up for us last year when we started autoflagging).
As I said many times, this proposal reduces the required manual involvement to nuke a post. Since Charcoal members also cast manual flags on top of the automatic flags cast by the system, that's also likely to increase the number of posts that are nuked entirely by Charcoal members who are not active on this site. Some posts already have six flags cast on them by Charcoal (both autoflagging and manual flags), but the proportion of posts that applies to is likely to increase. We've dug out some statistics about this: these two graphs show the number of flags cast on posts by Charcoal systems. The following is a graph for the most highly-spammed sites only, but we also have a graph for all other sites.
We don't think this is an issue in terms of subject matter expertise: the spam we see on the Stack Exchange network is broadly the same wherever you go - you don't need any subject matter expertise or activity on a particular site to be able to tell what's spam and what's not. We do, however, recognize that it's possible that a site's community may want to handle its own spam; if that's the case, we're happy to turn the autoflagging system off on that site or retain it at its current levels.
At the moment, two sites - English Language & Usage and The Workplace - require only three flags to automatically delete a post. Our current system only casts a single autoflag on spam posts on those sites. Right now, our current plan is to keep those sites at just one autoflag, though we're open to discussing to have it increased to two flags. As I said just before, it's possible to set site-specific overrides; we currently have those set on those two sites.
We'd like to hear your thoughts. We appreciate that quite a lot of the stuff we do at Charcoal is fairly invisible to the sites, so we want to be as open as possible. If you'd like data or specific reports, let us know and we'll try to add them in - we already have a lot of reporting around autoflagging, so it may already exist. If there's other stuff we can do to explain or to help you make an informed decision about whether you want this, drop an answer or a comment on this post. Charcoal members will be hanging around this post to respond to your concerns.
1 There are some flags from Charcoal members that don't appear in these statistics, for various reasons, but the majority are there.