Timvde/#add-on developer logs regarding AMO reviews Secret

## #add-on developer logs regarding AMO reviews
Oct 26 11:07:24 <Timvde>	Given the (mostly negative) feedback on automatically reviewed add-ons, what's the plan for moving forward? Will add-ons that haven't been reviewed by a person be marked as such on AMO? What about automatic updates?
Oct 26 11:52:00 <TheOne>	Timvde: we don't plan on doing that
Oct 26 11:53:45 <Timvde>	TheOne: Why not? How will you guarantee the safety of non-manually reviewed add-ons?
Oct 26 11:54:46 <TheOne>	because it will discourage users from installing add-ons which have not been reviewed yet, which puts those developers at disadvantage
Oct 26 11:54:46 <Timvde>	I read articles about malware on Chrome's add-on store all the time, I don't want this to happen to Firefox too :/
Oct 26 11:55:35 <Timvde>	TheOne: And now you're putting users at a disadvantage, because they cannot know whether an add-on ships malware or not...
Oct 26 11:56:22 <TheOne>	human review does not prevent malware
Oct 26 11:56:37 <TheOne>	and as we did in the past, we are very quick with taking down listing that are malicious
Oct 26 11:56:40 <Timvde>	Not completely, but *a lot* better than automated review
Oct 26 11:57:01 <Timvde>	TheOne: if an add-on update has already been shipped to thousands of people, it's too late
Oct 26 11:57:07 <TheOne>	upfront review is just not sustainable, given the load of submissions we get
Oct 26 11:57:25 <TheOne>	Timvde: we have and will blocklist malware immediately. No change of process there
Oct 26 11:57:52 <TheOne>	if you come across malware, please let us know right away
Oct 26 11:58:54 <Timvde>	TheOne: I understand. And I suppose that automatic review is an acceptable solution for many users, given the number of Chrome users (although I mostly think that they don't realise the problem or think about it).
Oct 26 11:59:33 <Timvde>	But for people who *do* care, having an option to distinguish between automatically and manually reviewed add-ons, and only enable automatic updates after add-ons have been reviewed, is a must.
Oct 26 11:59:53 <Timvde>	It's by far the biggest advantage AMO has over the Chrome store
Oct 26 12:00:04 <TheOne>	Timvde: and to be fair, the malware submission rate in the past has been very close to 0%. Of course that might change now that add-ons are approved automatically, but we are prepared for that
Oct 26 12:00:18 <Timvde>	How are you prepared for that?
Oct 26 12:00:56 <Timvde>	If you can give other guarantees that malware won't get a chance to flourish in AMO like it does in the Chrome store, I'm happy too.
Oct 26 12:01:15 <Timvde>	But at this moment, I'm really worried.
Oct 26 12:01:38 <TheOne>	unrevieweds add-ons are not unsafe by default and we don't want to give that impression to the user
Oct 26 12:03:13 <TheOne>	no one can guarantee that there won't be a single malicious add-on on AMO, and no one did, even before post-review. However, will have been and continue to do our best to prevent it
Oct 26 12:03:15 <Timvde>	TheOne: They are definitely *untrusted* by default.
Oct 26 12:03:31 <TheOne>	Timvde: that's an entirely different thing
Oct 26 12:03:49 <TheOne>	reviewed add-ons don't ensure trust
Oct 26 12:04:33 <Timvde>	They don't ensure full 100% trust, no. But it's a scale. My trust in non-reviewed add-ons is unexisting, my trust in reviewed add-ons is pretty high.
Oct 26 12:04:52 <TheOne>	that is not what review is about
Oct 26 12:06:16 <Timvde>	What is review about, then?
Oct 26 12:07:13 <Timvde>	For me (and many others), it's about Mozilla confirming that they have read through the source code, and as far as they can see, the add-on does what it states to do, there are no security issues and there is no hidden extra functionality (like a coin miner or user tracking)
Oct 26 12:13:39 <Timvde>	TheOne: It's not about guaranteeing that there won't ever be a single malicious add-on on AMO, it's about reducing the risk and making the number of malicious add-ons be close to 0%, and not having news articles all the time about "add-on x started to ship malware".
Oct 26 12:14:41 <Timvde>	I understand the need of faster throughput, but at least some way to distinguish between automatically and manually reviewed add-ons is necessary. In the worst case, I can fall back on manually updating my add-ons.
Oct 26 12:15:09 <TheOne>	Ok, how would you solve it?
Oct 26 12:18:33 <Timvde>	Mark add-ons as automatically reviewed/human reviewed in some way on AMO, and give an option (even a hidding about:config switch would be fine for me) to only auto-update add-ons after the human review has happened.
[...]
Oct 26 23:25:04 <TheOne>	Timvde: I think I explained why we can't expose that information
[...]
Oct 27 22:17:05 <Timvde>	TheOne: I It's still an improvement over not releasing at all (old system), and I'd be fine with having it somewhere in the small letters. But users who *do* care, should be able to access the information.
Oct 27 22:17:27 <Timvde>	Manual reviews are in general quoted as a large advantage of Firefox over Chrome
Oct 27 22:18:18 <Timvde>	By not exposing review information, you are putting users second
[...]
Oct 28 09:30:46 <TheOne>	Timvde: no. We have not exposed review information before post-review, so nothin changes here. And developers disagree that manual reviews is an advantage in Firefox. Developers are part of the community too. If we can’t attract them, users won’t even get great add-ons
Oct 28 09:32:45 <Timvde>	TheOne: Add-ons before post-review were just not available, and I'm not sure what the yellow "Experimental" button exactly meant, but at least it notified users that they should be careful for some reason. I'm sure that some developers will disagree that manual reviews are better, but given the vast superiority of Firefox add-ons in the past, that doesn't sound like a major problem.
Oct 28 09:33:36 <TheOne>	oh, it was a huge problem and that’s why we had to change process
Oct 28 09:34:55 <Timvde>	TheOne: I'm not asking to remove the automatic review, by the way. I understand the disadvantages of manual reviews. Just give us *some* way to disambiguate.
Oct 28 09:35:15 <Timvde>	It doesn't even have to be a big warning label for me
Oct 28 09:35:53 <Timvde>	But given the history of Chrome add-ons, I just *can't* put any trust in automatically reviewed add-ons.
Oct 28 09:37:07 <TheOne>	 I already explained why we can’t do that. It puts most developers in disadvantage, as low risk add-ons (which might not need human review) will appear as not as safe as others
Oct 28 09:38:49 <Timvde>	If you don't put a big warning label, but just some small notice in the sidebar, most people won't even know
Oct 28 09:39:17 <Timvde>	And are you now claiming that for "low-risk add-ons" human reviews might even be skipped completely?
Oct 28 09:39:25 <Timvde>	What's a "low-risk" add-on?
Oct 28 09:40:31 <TheOne>	For example, I have a very simple add-on that allows you to open multiple URL’s at once, it has almost no permissions and the code is only a few lines. It will likely never get reviewed because there is pretty much no risk here. But I wouldn’t want that information show up on my listing page, since users will hesitate to install it
Oct 28 09:41:44 <Timvde>	How will you identify whether it is necessary to review? You can't without actually reviewing, right?
Oct 28 09:41:52 <TheOne>	Saigon, unreviewed doesn’t man unsafe. Which is what you imply if you expose that information
Oct 28 09:41:55 <Timvde>	A review for such an add-on will only take a few minutes anyway
Oct 28 09:42:06 <TheOne>	*Again
Oct 28 09:42:20 <Timvde>	But unreviewed means untrusted.
Oct 28 09:42:47 <TheOne>	That is right, it will only take a few minutes, but those minutes are better spent on a more complex add-on
Oct 28 09:42:55 <Timvde>	There is indeed a difference between both, but I also want to know about untrusted add-ons.
Oct 28 09:43:39 <TheOne>	Ok, we‘re going in circles here (Like I said, review is not about trust). I’m off doing something else
Oct 28 09:44:08 <Timvde>	TheOne: You haven't actually explained what reviews are about, according to you.
Oct 28 09:45:20 <Timvde>	I explained my point of view, but if you aren't giving yours, of course we can't have a proper discussion
	Oct 26 11:07:24 <Timvde> Given the (mostly negative) feedback on automatically reviewed add-ons, what's the plan for moving forward? Will add-ons that haven't been reviewed by a person be marked as such on AMO? What about automatic updates?
	Oct 26 11:52:00 <TheOne> Timvde: we don't plan on doing that
	Oct 26 11:53:45 <Timvde> TheOne: Why not? How will you guarantee the safety of non-manually reviewed add-ons?
	Oct 26 11:54:46 <TheOne> because it will discourage users from installing add-ons which have not been reviewed yet, which puts those developers at disadvantage
	Oct 26 11:54:46 <Timvde> I read articles about malware on Chrome's add-on store all the time, I don't want this to happen to Firefox too :/
	Oct 26 11:55:35 <Timvde> TheOne: And now you're putting users at a disadvantage, because they cannot know whether an add-on ships malware or not...
	Oct 26 11:56:22 <TheOne> human review does not prevent malware
	Oct 26 11:56:37 <TheOne> and as we did in the past, we are very quick with taking down listing that are malicious
	Oct 26 11:56:40 <Timvde> Not completely, but a lot better than automated review
	Oct 26 11:57:01 <Timvde> TheOne: if an add-on update has already been shipped to thousands of people, it's too late
	Oct 26 11:57:07 <TheOne> upfront review is just not sustainable, given the load of submissions we get
	Oct 26 11:57:25 <TheOne> Timvde: we have and will blocklist malware immediately. No change of process there
	Oct 26 11:57:52 <TheOne> if you come across malware, please let us know right away
	Oct 26 11:58:54 <Timvde> TheOne: I understand. And I suppose that automatic review is an acceptable solution for many users, given the number of Chrome users (although I mostly think that they don't realise the problem or think about it).
	Oct 26 11:59:33 <Timvde> But for people who do care, having an option to distinguish between automatically and manually reviewed add-ons, and only enable automatic updates after add-ons have been reviewed, is a must.
	Oct 26 11:59:53 <Timvde> It's by far the biggest advantage AMO has over the Chrome store
	Oct 26 12:00:04 <TheOne> Timvde: and to be fair, the malware submission rate in the past has been very close to 0%. Of course that might change now that add-ons are approved automatically, but we are prepared for that
	Oct 26 12:00:18 <Timvde> How are you prepared for that?
	Oct 26 12:00:56 <Timvde> If you can give other guarantees that malware won't get a chance to flourish in AMO like it does in the Chrome store, I'm happy too.
	Oct 26 12:01:15 <Timvde> But at this moment, I'm really worried.
	Oct 26 12:01:38 <TheOne> unrevieweds add-ons are not unsafe by default and we don't want to give that impression to the user
	Oct 26 12:03:13 <TheOne> no one can guarantee that there won't be a single malicious add-on on AMO, and no one did, even before post-review. However, will have been and continue to do our best to prevent it
	Oct 26 12:03:15 <Timvde> TheOne: They are definitely untrusted by default.
	Oct 26 12:03:31 <TheOne> Timvde: that's an entirely different thing
	Oct 26 12:03:49 <TheOne> reviewed add-ons don't ensure trust
	Oct 26 12:04:33 <Timvde> They don't ensure full 100% trust, no. But it's a scale. My trust in non-reviewed add-ons is unexisting, my trust in reviewed add-ons is pretty high.
	Oct 26 12:04:52 <TheOne> that is not what review is about
	Oct 26 12:06:16 <Timvde> What is review about, then?
	Oct 26 12:07:13 <Timvde> For me (and many others), it's about Mozilla confirming that they have read through the source code, and as far as they can see, the add-on does what it states to do, there are no security issues and there is no hidden extra functionality (like a coin miner or user tracking)
	Oct 26 12:13:39 <Timvde> TheOne: It's not about guaranteeing that there won't ever be a single malicious add-on on AMO, it's about reducing the risk and making the number of malicious add-ons be close to 0%, and not having news articles all the time about "add-on x started to ship malware".
	Oct 26 12:14:41 <Timvde> I understand the need of faster throughput, but at least some way to distinguish between automatically and manually reviewed add-ons is necessary. In the worst case, I can fall back on manually updating my add-ons.
	Oct 26 12:15:09 <TheOne> Ok, how would you solve it?
	Oct 26 12:18:33 <Timvde> Mark add-ons as automatically reviewed/human reviewed in some way on AMO, and give an option (even a hidding about:config switch would be fine for me) to only auto-update add-ons after the human review has happened.
	[...]
	Oct 26 23:25:04 <TheOne> Timvde: I think I explained why we can't expose that information
	[...]
	Oct 27 22:17:05 <Timvde> TheOne: I It's still an improvement over not releasing at all (old system), and I'd be fine with having it somewhere in the small letters. But users who do care, should be able to access the information.
	Oct 27 22:17:27 <Timvde> Manual reviews are in general quoted as a large advantage of Firefox over Chrome
	Oct 27 22:18:18 <Timvde> By not exposing review information, you are putting users second
	[...]
	Oct 28 09:30:46 <TheOne> Timvde: no. We have not exposed review information before post-review, so nothin changes here. And developers disagree that manual reviews is an advantage in Firefox. Developers are part of the community too. If we can’t attract them, users won’t even get great add-ons
	Oct 28 09:32:45 <Timvde> TheOne: Add-ons before post-review were just not available, and I'm not sure what the yellow "Experimental" button exactly meant, but at least it notified users that they should be careful for some reason. I'm sure that some developers will disagree that manual reviews are better, but given the vast superiority of Firefox add-ons in the past, that doesn't sound like a major problem.
	Oct 28 09:33:36 <TheOne> oh, it was a huge problem and that’s why we had to change process
	Oct 28 09:34:55 <Timvde> TheOne: I'm not asking to remove the automatic review, by the way. I understand the disadvantages of manual reviews. Just give us some way to disambiguate.
	Oct 28 09:35:15 <Timvde> It doesn't even have to be a big warning label for me
	Oct 28 09:35:53 <Timvde> But given the history of Chrome add-ons, I just can't put any trust in automatically reviewed add-ons.
	Oct 28 09:37:07 <TheOne> I already explained why we can’t do that. It puts most developers in disadvantage, as low risk add-ons (which might not need human review) will appear as not as safe as others
	Oct 28 09:38:49 <Timvde> If you don't put a big warning label, but just some small notice in the sidebar, most people won't even know
	Oct 28 09:39:17 <Timvde> And are you now claiming that for "low-risk add-ons" human reviews might even be skipped completely?
	Oct 28 09:39:25 <Timvde> What's a "low-risk" add-on?
	Oct 28 09:40:31 <TheOne> For example, I have a very simple add-on that allows you to open multiple URL’s at once, it has almost no permissions and the code is only a few lines. It will likely never get reviewed because there is pretty much no risk here. But I wouldn’t want that information show up on my listing page, since users will hesitate to install it
	Oct 28 09:41:44 <Timvde> How will you identify whether it is necessary to review? You can't without actually reviewing, right?
	Oct 28 09:41:52 <TheOne> Saigon, unreviewed doesn’t man unsafe. Which is what you imply if you expose that information
	Oct 28 09:41:55 <Timvde> A review for such an add-on will only take a few minutes anyway
	Oct 28 09:42:06 <TheOne> *Again
	Oct 28 09:42:20 <Timvde> But unreviewed means untrusted.
	Oct 28 09:42:47 <TheOne> That is right, it will only take a few minutes, but those minutes are better spent on a more complex add-on
	Oct 28 09:42:55 <Timvde> There is indeed a difference between both, but I also want to know about untrusted add-ons.
	Oct 28 09:43:39 <TheOne> Ok, we‘re going in circles here (Like I said, review is not about trust). I’m off doing something else
	Oct 28 09:44:08 <Timvde> TheOne: You haven't actually explained what reviews are about, according to you.
	Oct 28 09:45:20 <Timvde> I explained my point of view, but if you aren't giving yours, of course we can't have a proper discussion