CFR-testing.md

CFR Testing instructions

General Setup

These instructions apply to testing all variants including control.

Start by creating a new profile.

In about:config, you should ensure the following preferences are set:

• browser.search.region to US
• devtools.chrome.enabled to true

Simulating shield cohorts

In order to simulate a shield cohort, you will need to update the value of a single pref:

browser.newtabpage.activity-stream.asrouter.messageProviders

The values for this pref are as follows:

Control

[]

Cohort 1 (One per day)

[{"id":"cfr", "cohort": "one_per_day", "frequency": {"custom": [{"period": "daily", "cap": 1}]}, "type":"local","localProvider":"CFRMessageProvider","enabled":true}]

Cohort 2 (Three per day)

[{"id":"cfr", "cohort": "three_per_day", "frequency": {"custom": [{"period": "daily", "cap": 1}]}, "type":"local","localProvider":"CFRMessageProvider","enabled":true}]

Expected Behaviour for each cohort

The expected general behaviour for each cohort is as follows:

Control

No add-on recommendations should show up under any circumstances, even when the targeting conditions are met.

Cohort 1 (One per day)

An add-on recommendation should show up when users have visited targeted sites a sufficient number of times for the following 5 add-ons: Facebook container, Google translate, Enhancer for Youtube, Wikipedia Context Menu Search, Reddit enhancement suite

Only one recommendation should ever be shown per add-on, and only one recommendation of any kind should be shown per day.

Cohort 2 (Three per day cap)

Like the previous cohort, an add-on recommendation should show up when users have visisted targeted sites a sufficient number of times for the previously mentioned 5 add-ons.

An individual recommendation may be shown a maximum of three times per add-on, but only one recommendation of any kind should be shown per day.

Appearance of a recommendation

When a recommendation appears, it will show up in the right-hand corner of the awesomebar:

How to trigger targeting conditions for a recommendation

Assuming you have completed the General Setup section above and and set the experiment preference as described in Simulating Shield Cohorts, follow these steps to trigger the appropriate targeting conditions. Replace facebook.com in these steps with whatever targeting URL you are testing.

Increase frecency of facebook.com

• Do the following steps 5 times:
  • open a new tab
  • copy and paste https://facebook.com into the awesome bar OR click on a facebook top site on the new tab page
    • IMPORTANT: just typing in the URL or clicking on a regular link will NOT accumulate enough frecency in 5 visits.
  • load the site

Verify frecency via the browser console

• Open the browser toolbox console and run the following code to see a list of organic top sites and their frecencies:

NewTabUtils.activityStreamProvider.getTopFrecentSites({numItems: 25}).then(r => console.log(r.map(s=>`${s.url} [${s.frecency}]`).join('\n')))

• You should see facebook.com listed as having a frecency of 10000. If it is not at least then thousand, visit https://facebook.com some more and re-run the code above to check.
• Restart the browser (this is needed to reset the frecency cache in Activity Stream)

Trigger the recommendation

• Visit https://facebook.com one more time

You should now see the recommendation.