Skip to content

Instantly share code, notes, and snippets.

@escapewindow

escapewindow/check.sh Secret

Forked from nthomas-mozilla/funnel.md
Last active May 5, 2017 20:07
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save escapewindow/5dba93a02bb346f5ac3677477106a46f to your computer and use it in GitHub Desktop.
Save escapewindow/5dba93a02bb346f5ac3677477106a46f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
check.sh
#!/bin/bash
# Check the various bouncer links: look for 302 and locations for correctness
for i in 110 111 112 113; do
echo ${i}
curl -sIL "https://download.mozilla.org/?product=firefox-stub-f$i&os=win&lang=en-US" | egrep 'Location|^HTTP'
echo
curl -sIL "https://download.mozilla.org/?product=firefox-latest-f$i&os=win&lang=en-US" | egrep 'Location|^HTTP'
echo
echo
done
Raw
funnel.md

Introduction

This Funnelcake test is to measure potential improvements in initial user experience - a streamlined stub installer, improved auto-migration of Chrome bookmarks etc, "tour notifications" (via addon, probably https://github.com/mozilla/onboard). The four builds are

  • funnelcake 110 - control - current stub
  • funnelcake 111 - streamlined stub, auto-migration, tour variation A
  • funnelcake 112 - streamlined stub, auto-migration, tour variation B
  • funnelcake 113 - streamlined stub, no auto-migration, tour variation A

Source: https://bugzilla.mozilla.org/show_bug.cgi?id=1348127#c1

Before you start

  • You need write access to https://github.com/mozilla-partners/funnelcake . You can get this from mkaply or nthomas.
  • You need ssh access to cltbld@partner-repack1. Someone with access can add your ssh pubkey to ~cltlbd/.ssh/authorized_keys.

Quick links

  • Master bug https://bugzilla.mozilla.org/show_bug.cgi?id=1348127
  • Co-ordinator - Nicole Yee (:nicoleyee)
  • Also leading - Peter Dolanjski (:pdol)
  • Addon-dev - Schalk Neethling (:espressive)
  • Stub streamlining - Adam Gashlin [:agashlin] (bug 1353576)
  • Automigration - Gijs Kruitbosch (this code has been landed and stable for a while)
  • SoftVision - Justin Williams jwilliams@softvision.com (Las Vegas, not Romania)
  • Releng - releaseduty (jlorenzo, aki, backup from jlund)

No IRC channel known.

Timeline & State

3/31: Working prototype of Tour Notification add-on
4/3 - 4/7: Development Window for Internal QA (QA done by onboarding team)
4/7: Development Deadline
4/10: Submit XPI to AMO (assigned to ?), add test to Optimizely (jcrawford)
4/10- 4/12: RelEng build funnelcakes with self-signed XPI
4/13 - 4/19: Softvision QA of funnelcake builds
4/19 - 4/20: Replace self-signed XPI with signed XPI downloaded from AMO
4/20: Release funnelcakes (live at 1%)
4/23: "Go-Live" for funnelcakes (increasing to 10%)

Current state @ 4/12 - behind schedule. The addon hasn't been provided, internal QA in dev team happening.

Calendar events are being used to track some of these dates.

Delivery mechanism

A user visits www.mozilla.org to download firefox

If the user matches test requirements (US, windows UA, etc) then a system called Optimizely modifies the download button to a custom stub installer, e.g. from product=firefox-stub to

https://download.mozilla.org/?product=firefox-stub-f111&os=win&lang=en-US

This is a redirect to

https://download-installer.cdn.mozilla.net/pub/firefox/releases/53.0-funnelcake111/win32/en-US/Firefox%20Setup%20Stub%2053.0.exe

The stub installer pulls in full funnelcake from

https://download.mozilla.org/?product=firefox-latest-f111&os=win&lang=en-US

which would redirect to

http://download.cdn.mozilla.net/pub/firefox/releases/53.0-funnelcake111/win32/en-US/Firefox%20Setup%2053.0.exe

Releng Work

[X] Generate stub installers

  • The stub installer requires a compilation, and we'd like a record of what was done, so we create a branch in mozilla-release and run a win32 en-US build to get the stub
  • Adam has patches ready at https://bugzilla.mozilla.org/show_bug.cgi?id=1353576#c16
  • created FUNNELCAKE_110_111_112_113_BRANCH and pushed patches, details in comment #17, treeherder
  • copied stubs to candidates, eg on partner-repack1
s3cmd -c ~/.s3cfg-mozilla cp \
  "s3://net-mozaws-prod-delivery-firefox/pub/firefox/tinderbox-builds/mozilla-release-win32/1491998108/firefox-53.0.en-US.win32.installer-stub.exe" \
  "s3://net-mozaws-prod-delivery-firefox/pub/firefox/candidates/53.0-candidates/build5/partner-repacks/funnelcake110/v1/win32/en-US/Firefox Setup Stub 53.0.exe"

[X] Generate full installer configs

    # needs to be unpacked for Firefox to load
    cd <funnelcake_repo>/desktop/funnelcake111/distribution/
    mkdir extensions
    cd extensions
    wget -O foo.xpi <attachment_url>
    # we can also rename the xpi to @onboard-v1.xpi instead of unzipping - still need to check
    # what's in install.rdf to verify the name
    unzip foo.xpi  && rm foo.xpi
    # look up <em:id> in install.rdf. May be "@onboard-v1". Won't load if dir and install.rdf don't match
    mkdir <em:id>
    mv * <em:id>
    # This ignored the .jpmignore file; I moved it into <em:id> as well.

    # copy to 112 and 113
    cd <funnelcake_repo>/desktop/
    rsync -a --delete funnelcake111/distribution/extensions/ funnelcake112/distribution/extensions/
    rsync -a --delete funnelcake111/distribution/extensions/ funnelcake113/distribution/extensions/
    
    # commit and push

[X] Generate full installers

for FUNNEL in 110 111 112 113; do
    ./partner-repacks.py -v ${VERSION} \
        -n ${BUILDNUMBER} \
        -d ../partners/funnelcake/desktop \
        -p funnelcake${FUNNEL} \
        2>&1 | tee `date +logs/%Y%m%d-%H%M%S.log`
done
  • upload will happen automatically
  • ignore the information about bouncer there
  • announce the urls on the bug, eg

https://archive.mozilla.org/pub/firefox/candidates/53.0-candidates/build5/partner-repacks/funnelcake110/v1/win32/en-US/Firefox%20Setup%2053.0.exe

  • Bonus points for giving a sha256 hash with each, you can grab those easily with
find repacked_builds/signed/partner-repacks/funnelcake{110..113}/v1 -name '*.exe' -print0 | xargs -0 openssl dgst -sha256

[X] Set up bouncer for testing

Revising the builds

For fixes in the funnelcake config or add-on changes

  • speculative iteration can be discouraged. Changes to distribution.ini can be tested by editing the file on disk (after install), and the add-on devs know how to test before submitting a new xpi to us or AMO
  • when change is needed, also modify each repack.cfg to bump the output_dir definition from v1 to v2 (or higher as necessary), so that the builds are uploaded to a new directory. Reusing urls leads to confusion due to the long cache time on archive.m.o, ie it serves the older content
  • When updating the add-on I suggest cleaning out all the existing files, eg
cd <funnelcake_repo>/desktop/funnelcake111/distribution/extensions/@onboard-v1/
rm -rf * .[^.]*
wget -O foo.xpi <attachment_url>
unzip foo.xpi  && rm foo.xpi
# look up <em:id> in install.rdf and verify it still matches directory name

# copy to 112 and 113
cd <funnelcake_repo>/desktop/
rsync -a --delete funnelcake111/distribution/extensions/ funnelcake112/distribution/extensions/
rsync -a --delete funnelcake111/distribution/extensions/ funnelcake113/distribution/extensions/
  • the changes need to land on master of the funnelcake repo, via a branch there or a fork & PR
  • regenerate the full installers (see "Generate full installers" above)
  • best to copy the stub installers forward, eg
s3cmd -c ~/.s3cfg-mozilla cp \
  "s3://net-mozaws-prod-delivery-firefox/pub/firefox/candidates/53.0-candidates/build5/partner-repacks/funnelcake110/v1/win32/en-US/Firefox Setup Stub 53.0.exe" \
  "s3://net-mozaws-prod-delivery-firefox/pub/firefox/candidates/53.0-candidates/build5/partner-repacks/funnelcake110/v2/win32/en-US/Firefox Setup Stub 53.0.exe"
  • if you make a mistake with copying you can contact oremj on IRC, or file a bug with Cloud Services
  • update the 8 bouncer locations for the new paths (ie the vN change), check they work (eg with curl -IL, and checking HTTP code and 302 location for correctness)
  • announce new urls on the bug

For a new RC

  • we can copy forward the stubs, unless we had to make a NSIS change in the main build. eg, but watch out for vN or buildN differing by the time you get to here
s3cmd -c ~/.s3cfg-mozilla cp \
  "s3://net-mozaws-prod-delivery-firefox/pub/firefox/candidates/53.0-candidates/build5/partner-repacks/funnelcake110/v1/win32/en-US/Firefox Setup Stub 53.0.exe" \
  "s3://net-mozaws-prod-delivery-firefox/pub/firefox/candidates/53.0-candidates/build6/partner-repacks/funnelcake110/v1/win32/en-US/Firefox Setup Stub 53.0.exe"
  • regenerate the full installers using the new BUILDNUMBER (see Generate full installers above)
  • update stub and full locations in bouncer for testing (buildN will definitely have changed)
  • announce new urls on the bug

[x] Regenerate full installers with final addon

  • once the addon is finalised and AMO-signed, download it from amo, and land in the funnelcake repo, with a vN bump for output_dir again. Same as 'For fixes in the funnelcake config or add-on changes' above
  • copy the stubs forward, and regenerate the funnelcake installers, bouncer updates, and urls announced in the bug
  • it's not clear what QA will happen at this point, maybe addon devs confirm ok

[ ] Ready to go live

  • copy the files from the candidates directory to releases, eg
# define VERSION, BUILDNUMBER, CONFIG_VERSION (aka N in vN)
for FUNNEL in 110 111 112 113; do
    s3cmd -c ~/.s3cfg-mozilla sync \
      "s3://net-mozaws-prod-delivery-firefox/pub/firefox/candidates/${VERSION}-candidates/build${BUILDNUMBER}/partner-repacks/funnelcake${FUNNEL}/v${CONFIG_VERSION}/" \
      "s3://net-mozaws-prod-delivery-firefox/pub/firefox/releases/${VERSION}-funnelcake${FUNNEL}/"
done
  • swap bouncer locations to use
firefox-stub-f${FUNNEL}    /firefox/releases/${VERSION}-funnelcake${FUNNEL}/win32/en-US/Firefox Setup Stub ${VERSION}.exe
firefox-latest-f${FUNNEL}  /firefox/releases/${VERSION}-funnelcake${FUNNEL}/win32/en-US/Firefox Setup ${VERSION}.exe
  • verify bouncer links working (eg by curl -IL and checking HTTP code and 302 locations for correctness)
  • someone checks the end-to-end again
  • optimizely enabled at 1% for a few days, then will be increased to 10% to ship for real

Funnelcake still live when we do a chemspill/dot release (e.g. 53.0.1)

Historically it's taken 1-2 weeks to get a large enough cohort of users, and a point release is fairly likely in that timeframe. Refreshing is fairly similar to work above

  • follow 'For a new RC' section, except
    • when copying the stub forward, adjust the version in the filename as well as the candidates directory, reset buildN to build1
    • in the past we haven't tested the builds again
  • ship by repeating 'Ready to go live' above. The safest approach is to pause distribution in Optimizely first, if you can raise Chris More (cmore) or his designate. Bouncer will keep happily serving though, so you can do it hot if you are feeling bold

Smoketesting

If you've (re)generated builds and want to do a quick check they work, then you can do that with a Windows VM. The pre-flight check is

  • uninstall existing Firefox, at least the one in the standard location
  • remove existing Firefox profiles by using Windows Explorer to go to %APPDATA%\Mozilla, and remove/rename the Firefox directory

When running the stub you can look for things like

  • signing/branding: stub is signed by Mozilla Corporation, Firefox logo is used
  • correct stub style is used: existing for 110, streamlined for 111-113
  • after install the firstrun page has correct f=NNN query parameter, and content is as expected (not sure if the addon will modify that this time)
  • data is migrated in from Chrome
  • the addon appears in about:addons. It may be hard to do more than this because of it waiting 24 hours before starting notifications
Raw
stubs.sh
#!/bin/bash
# copy stub installers forward
# define VERSION, BUILDNUMBER, OLD_CONFIG_VERSION, CONFIG_VERSION (aka N in vN)
VERSION=53.0
BUILDNUMBER=6
OLD_CONFIG_VERSION=9
CONFIG_VERSION=11
# 110 has a normal named stub installer; it's the control
s3cmd -c ~/.s3cfg-mozilla cp \
"s3://net-mozaws-prod-delivery-firefox/pub/firefox/candidates/${VERSION}-candidates/build$BUILDNUMBER/partner-repacks/funnelcake110/v$OLD_CONFIG_VERSION/win32/en-US/Firefox Setup Stub ${VERSION}.exe" \
"s3://net-mozaws-prod-delivery-firefox/pub/firefox/candidates/${VERSION}-candidates/build$BUILDNUMBER/partner-repacks/funnelcake110/v$CONFIG_VERSION/win32/en-US/Firefox Setup Stub ${VERSION}.exe"
# 111 through 113 have the stub installer renamed to "Firefox Installer" as part of the funnelcake test.
for FUNNEL in 111 112 113; do
s3cmd -c ~/.s3cfg-mozilla cp \
"s3://net-mozaws-prod-delivery-firefox/pub/firefox/candidates/${VERSION}-candidates/build$BUILDNUMBER/partner-repacks/funnelcake$FUNNEL/v$OLD_CONFIG_VERSION/win32/en-US/Firefox Installer.exe" \
"s3://net-mozaws-prod-delivery-firefox/pub/firefox/candidates/${VERSION}-candidates/build$BUILDNUMBER/partner-repacks/funnelcake$FUNNEL/v$CONFIG_VERSION/win32/en-US/Firefox Installer.exe"
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment