Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?

TL;DR: We did it, so...yes.


What is this?

Charcoal is the organization behind the SmokeDetector bot and other nice things. This bot scans new posts across the entire network for spam and reports it to various chatrooms where people can act on it. If a post has been created or edited, anywhere on the network, we've probably seen it. The bot utilizes our knowledge of how spammers work and what they have previously posted to come up with common patterns and rules to detect spam in the new and updated posts. You've likely seen the SmokeDetector bot if you visit chatrooms such as Tavern on the Meta, Charcoal HQ, SO Close Vote Reviewers and others across the network. Over time, the bot has become very accurate.

Now we are leveraging the years of data and accuracy to automatically cast spam flags. With approximately 57,000 posts to draw from and over 45,000 true positives, we have a vast trove of data to utilize.

What are we doing?

For over 3 years, SmokeDetector has reported potential spam so that users can flag the posts as appropriate. Users have provided feedback to inform the bot on whether the detection was correct or not (referred to as "feedback"). This feedback is stored in our web dashboard, metasmoke (code). Over time, we've used this feedback to evaluate our patterns ("reasons") and improve our accuracy. Several of our reasons are over 99.9% accurate.

Early last year, and after getting a baseline accuracy from jmac (thank you!), we realized we could use the system to automatically cast spam flags. On Stack Overflow the current accuracy of users flagging spam posts is 85.7%. Across the rest of the network users are 95.4% accurate. We determined we can beat those numbers and eliminate spam from Stack Overflow and the rest of the network even faster.

Without going into too much detail (if you really want it, it's available on our website), we leverage the accuracy of each existing reason to come up with a weight indicating how certain the system is that a post is spam. If this value exceeds a specific threshold, the system will cast up to three spam flags on the post. We cast multiple flags utilizing a number of different users' accounts and the Stack Exchange API. Via metasmoke, users are given the opportunity to enable their accounts to be used to flag spam. When a post is eligible for flagging because it exceeded the threshold set by each individual user, accounts are randomly selected from the pool of enabled users to cast a single flag each, up to a maximum of three per post so that we never unilaterally nuke something.

What are our safety checks?

We designed the entire system with accuracy and sanity checks in mind. Our design collaborations are available for your browsing pleasure (RFC 1, RFC 2, RFC 3). The major things that make this system safe and sane are:

  • We give users a choice as to how accurate they want to be with their automatic flags. Before casting any flags, we check that the preferences the user has set result in a spam detection accuracy of over 99.5% over a sample of at least 1000 posts. Remember, on SO the current accuracy is 85.7% and network wide it is 95.4%.
  • We do not unilaterally spam nuke a post, regardless of how sure we are it is spam. This means that a human must be involved to finish off a post, even on the few sites with lower spam thresholds.
  • We’ve designed the system to be tolerant of faults - if there’s a malfunction anywhere in the system, any user with access to SmokeDetector can immediately halt all automatic flagging - this includes all network moderators. If this happens, it needs a system administrator to step in to re-enable flags.
  • We've discussed this with a community manager and have their blessing on the project.

Results

We have been casting an average of 60-70 automatic flags per day for over two months, for a total of just over 4000 flags network wide. These flags were cast by 22 different users. In that time, we've had four false positives. We would like to be able to automatically cancel these particular cases. This isn't possible though, so we've created a feature request to retract flags via the API. In the mean time, the flags are either manually retracted by the user or declined by a moderator.

Weights and Accuracy

The above graph plots the weight of the reasons against its overall volume of reports and accuracy. As minimum weight increases, accuracy (yellow line and rightmost Y-axis) and total reports (blue line) on the left-hand scale increase.

Automatic Flags per day

This shows the number of posts we've automatically flagged per day over the last month. The jump on February 15th, is due to increasing the number of automatic flags from 1 per post to 3 per post. You can see a live version of this graph on metasmoke's autoflagging page.

Spam Hours

Spam arrives on Stack Overflow in waves. It is easy to see the time of day that many spam reports come in. The hours, above, are UTC time. The busiest spam times of day are the 8 hour block between 4am and Noon. We have affectionately named this "spam hour" in the chat room.

Average Time to Deletion

Our goal is to delete spam quickly and accurately. The graph shows the time it takes for a reported spam post to be removed from the network. This section has three trend lines that show these averages. The first, red, section is when we were simply reporting the posts to chatrooms and all flags had to come from users. You can see we were are pretty constant in the time it takes to remove spam during this period. It took, on average, just over five minutes to get a post removed.

The green trend line is when we were issuing a single automatic flag. At implementation, we eliminated a full minute from time to deletion and after a month we'd eliminated two full minutes compared to no automatic flags.

The last section, the orange, is when we implemented three automatic flags to most sites. This was rolled out last week, but it's already had a dramatic improvement on the time to deletion. We are seeing between 1 and 2 minutes to time to deletion.

As mentioned above, spam arrives in waves. The dashed and dotted lines on the graph show the average deletion time during these two different time periods. The dashed lines show deletion time during 4am and Noon UTC, the dotted lines show the rest of the 24 hour period. An interesting thing this graph shows is that time to deletion during spam hour was higher when we didn't cast any automatic flags. It was removed faster outside of spam hour. That reversed when we started issuing a single auto-flag. The spam hour time to deletion is slightly lower than the average. Comparing the two time periods though, time to deletion during non-spam hour at the end of the non-flagging time period and the end of the single flag period are roughly the same.

We'll update these in a few weeks too, to better show the trend we are seeing with three automatic flags.

Discussion

We are confident in SmokeDetector and the three years of history it has. We've had many talented developers assist us over the years and many more users have provided feedback to improve our detection rules. Let us know what you want us to elaborate on, features you're wondering about or would like to see added, or things we might have missed in the process or the tooling. Take a look at the feature we'd really like Stack Exchange to consider so that we can further improve this system (and some of the other community built systems). We'll have Charcoal members hanging around and answering your questions. Alternatively, feel free to drop into Charcoal HQ and have a chat.

@angussidney

This comment has been minimized.

Copy link

angussidney commented Feb 15, 2017

Suggestions:

  • Change the link to metasmoke (under 'What are we doing') to http://metasmoke.erwaysoftware.com/ (since we use it more often, it is more stable, etc)
  • Link to http://charcoal-se.org/people.html in the second last sentence when you reference to 'Charcoal members'
  • We did have a signup link in the middle of the post somewhere, should we or shouldn't we put it in here too?
  • maximium -> maximum, last sentence under what we are doing (thanks Glorfindel)

Things we need to consider before posting:

  • Where are we going to post? (strawpoll maybe?)
  • When are we going to post? (what timezone?)
  • Who is going to post it? (I vote Undo)
  • Who is going to be our representitives to reply to comments etc? (MS admins of course, maybe a few others, and everyone else can help out newcomers to chat) (we don't want too many people doing this, or otherwise it will sound like we're all barging in_
@M-A-Ramezani

This comment has been minimized.

Copy link

M-A-Ramezani commented Feb 15, 2017

Since we're unsure of the feedback people would give (Although I personally think it will be overwhelmingly positive), I think we shouldn't put a sign-up link in the end so as not to look like we're advertising Smokey, but rather telling the world about it. @angussidney

@Glorfindel83

This comment has been minimized.

Copy link

Glorfindel83 commented Feb 15, 2017

Replace/extend the link of the FP (which has now been deleted) with its Metasmoke link and/or a screenshot (suggested by @mithrandir)

@FelixSFD

This comment has been minimized.

Copy link

FelixSFD commented Feb 15, 2017

No handdrawn circles? 👎

😃

@rschrieken

This comment has been minimized.

Copy link

rschrieken commented Feb 15, 2017

Are you going to tag it Announcement? On a more serious note: What is the question we ask? Approval? Or is it just informing, Do we expect something from readers? If they answer, what would we like to see there?

@AWegnerGitHub

This comment has been minimized.

Copy link
Owner Author

AWegnerGitHub commented Feb 15, 2017

@angussidney I provided a couple answers to these questions in chat: http://chat.stackexchange.com/transcript/message/35452044#35452044 and fixed the issues you pointed out.

@M-A-Ramezani There is a sign up link in the post already. It's behind the word "enabled".

@Glorfindel83 I've added a link to a screenshot

@FelixSFD I aim to please. Have several freehand circles: http://i.imgur.com/M9u5FQF.png

@rschrieken I've edited the "Discussion" section a bit after bringing it up in chat to solicit more input.

@ArtOfCode-

This comment has been minimized.

Copy link

ArtOfCode- commented Feb 15, 2017

I was using metasmoke.charcoal-se.org because last time Undo was posting links on meta (for the election primary tracker), he used a charcoal-se.org address to avoid exposing his domain. I really don't mind which we use, so it's a question of whether @Undo1 still cares about that.

@Undo1

This comment has been minimized.

Copy link

Undo1 commented Feb 15, 2017

@ArtOfCode- Probably not (heck, I'm using my real name on GitHub.), but it doesn't really matter - worst case, if we lose charcoal-se.org, we just edit the post. Your call. charcoal-se.org might look better.

@rschrieken I don't really know what the rules around that tag are. It'd probably up to an employee to add it if they really wanted to, and I doubt they would.

@Aralun

This comment has been minimized.

Copy link

Aralun commented Feb 16, 2017

Small typos and corrections:

with common patterns and rules to detect in the new and updated posts → to detect spam in the new and updated
a weight to indicate → a weight indicating
which were off topic anyway → off-topic

@AWegnerGitHub

This comment has been minimized.

Copy link
Owner Author

AWegnerGitHub commented Feb 16, 2017

@Aralun Thanks. Those have been fixed

@angussidney

This comment has been minimized.

Copy link

angussidney commented Feb 18, 2017

Things that need to be updated before we post:

  • stats for number of posts and number of TPs
  • total number of autoflags (almost 4000 now)
  • graphs for number of posts per day, deletion times, etc
@angussidney

This comment has been minimized.

Copy link

angussidney commented Feb 19, 2017

Final suggestion: link to Pops' message where we say we have the blessing of a CM

@Glorfindel83

This comment has been minimized.

Copy link

Glorfindel83 commented Feb 19, 2017

@AWegnerGitHub

This comment has been minimized.

Copy link
Owner Author

AWegnerGitHub commented Feb 20, 2017

Things that need to be updated before we post:

  • [DONE] stats for number of posts and number of TPs
  • [DONE] total number of autoflags (almost 4000 now)
  • graphs for number of posts per day, deletion times, etc
@magisch

This comment has been minimized.

Copy link

magisch commented Feb 20, 2017

In that time, we've had four.

Should be changed to

In that time, we've had four false positives.

@angussidney

This comment has been minimized.

Copy link

angussidney commented Feb 20, 2017

You missed the number of autoflags stat at the top of the results section, should be 4000 not 3000

@Glorfindel83

This comment has been minimized.

Copy link

Glorfindel83 commented Feb 20, 2017

Since this is not possible, the flags are either declined by a moderator or manually retracted by the user. I'd reverse the order - put the least harmful option first: Since this is not possible, the flags are either manually retracted by the user or declined by a moderator.

@angussidney

This comment has been minimized.

Copy link

angussidney commented Feb 20, 2017

Also, it may be worth adding a call to action after that sentence - 'If you'd like to see us automatically revert false positive flags, please go vote on that meta proposal!'

@j-f1

This comment has been minimized.

Copy link

j-f1 commented Feb 20, 2017

👍 on @angussidney’s proposal.

@magisch

This comment has been minimized.

Copy link

magisch commented Feb 20, 2017

@angussidney I'd phrase it differently "Since this is currently not possible (we have asked(link) for this(link) previously), the flags are either...."

@Glorfindel83

This comment has been minimized.

Copy link

Glorfindel83 commented Feb 20, 2017

@angussidney If Undo posts this, that'd be him asking for upvotes on his own feature request. I don't want to go into that territory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.