Created
October 16, 2018 12:21
-
-
Save MihaiTabara/94009cde294cde03b02b14328078bda1 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| comparing with https://hg.mozilla.org/releases/mozilla-esr60 | |
| searching for changes | |
| changeset: 502889:e563b7789525 | |
| parent: 502881:25228484906d | |
| user: Simon Fraser <sfraser@mozilla.com> | |
| date: Wed Jun 27 07:34:19 2018 +0000 | |
| summary: Bug 1469803 Add bouncer check to run on cron. r=mtabara a=release | |
| diff --git a/.cron.yml b/.cron.yml | |
| --- a/.cron.yml | |
| +++ b/.cron.yml | |
| @@ -97,16 +97,37 @@ jobs: | |
| target-tasks-method: searchfox_index | |
| run-on-projects: | |
| - mozilla-central | |
| when: | |
| by-project: | |
| mozilla-central: [{hour: 10, minute: 30}] | |
| # No default | |
| + - name: bouncer-check | |
| + job: | |
| + type: decision-task | |
| + treeherder-symbol: Rel | |
| + target-tasks-method: bouncer_check | |
| + run-on-projects: | |
| + - mozilla-beta | |
| + - mozilla-release | |
| + when: | |
| + by-project: | |
| + # No default branch | |
| + mozilla-beta: | |
| + - {hour: 7, minute: 0} | |
| + - {hour: 19, minute: 0} | |
| + mozilla-release: | |
| + - {hour: 7, minute: 0} | |
| + - {hour: 19, minute: 0} | |
| + mozilla-esr60: | |
| + - {hour: 7, minute: 0} | |
| + - {hour: 19, minute: 0} | |
| + | |
| - name: periodic-update | |
| job: | |
| type: decision-task | |
| treeherder-symbol: Nfile | |
| target-tasks-method: file_update | |
| run-on-projects: | |
| - mozilla-central | |
| - mozilla-beta | |
| diff --git a/taskcluster/ci/release-bouncer-check/kind.yml b/taskcluster/ci/bouncer-check/kind.yml | |
| copy from taskcluster/ci/release-bouncer-check/kind.yml | |
| copy to taskcluster/ci/bouncer-check/kind.yml | |
| --- a/taskcluster/ci/release-bouncer-check/kind.yml | |
| +++ b/taskcluster/ci/bouncer-check/kind.yml | |
| @@ -1,19 +1,15 @@ | |
| # This Source Code Form is subject to the terms of the Mozilla Public | |
| # License, v. 2.0. If a copy of the MPL was not distributed with this | |
| # file, You can obtain one at http://mozilla.org/MPL/2.0/. | |
| loader: taskgraph.loader.transform:loader | |
| -kind-dependencies: | |
| - - release-beetmover-push-to-release | |
| - | |
| transforms: | |
| - - taskgraph.transforms.release_deps:transforms | |
| - taskgraph.transforms.bouncer_check:transforms | |
| - taskgraph.transforms.job:transforms | |
| - taskgraph.transforms.task:transforms | |
| job-defaults: | |
| name: bouncer-check | |
| description: bouncer check | |
| run-on-projects: [] # to make sure this never runs as part of CI | |
| @@ -24,48 +20,56 @@ job-defaults: | |
| docker-image: {in-tree: "update-verify"} | |
| run: | |
| using: run-task | |
| sparse-profile: mozharness | |
| attributes: | |
| build_platform: linux64 | |
| build_type: opt | |
| treeherder: | |
| - symbol: Rel(BncChk) | |
| + symbol: Rel(ckbouncer) | |
| kind: test | |
| tier: 1 | |
| jobs: | |
| firefox: | |
| shipping-product: firefox | |
| index: | |
| product: firefox | |
| job-name: firefox-bouncer-check | |
| run: | |
| config: | |
| by-project: | |
| mozilla-beta: | |
| - releases/bouncer_firefox_beta.py | |
| mozilla-release: | |
| - releases/bouncer_firefox_release.py | |
| - mozilla-esr60: | |
| - - releases/bouncer_firefox_esr.py | |
| - jamun: | |
| - - releases/dev_bouncer_firefox_esr.py | |
| default: | |
| - releases/dev_bouncer_firefox_beta.py | |
| + product-field: | |
| + by-project: | |
| + mozilla-beta: LATEST_FIREFOX_RELEASED_DEVEL_VERSION | |
| + mozilla-release: LATEST_FIREFOX_VERSION | |
| + mozilla-esr60: FIREFOX_ESR | |
| + default: LATEST_FIREFOX_DEVEL_VERSION | |
| + products-url: https://product-details.mozilla.org/1.0/firefox_versions.json | |
| treeherder: | |
| platform: firefox-release/opt | |
| devedition: | |
| shipping-product: devedition | |
| index: | |
| product: devedition | |
| job-name: devedition-bouncer-check | |
| run: | |
| config: | |
| by-project: | |
| mozilla-beta: | |
| - releases/bouncer_firefox_devedition.py | |
| default: | |
| - releases/dev_bouncer_firefox_devedition.py | |
| + product-field: | |
| + by-project: | |
| + mozilla-beta: LATEST_FIREFOX_RELEASED_DEVEL_VERSION | |
| + default: LATEST_FIREFOX_DEVEL_VERSION | |
| + products-url: https://product-details.mozilla.org/1.0/firefox_versions.json | |
| treeherder: | |
| platform: devedition-release/opt | |
| diff --git a/taskcluster/ci/release-bouncer-check/kind.yml b/taskcluster/ci/release-bouncer-check/kind.yml | |
| --- a/taskcluster/ci/release-bouncer-check/kind.yml | |
| +++ b/taskcluster/ci/release-bouncer-check/kind.yml | |
| @@ -9,18 +9,18 @@ kind-dependencies: | |
| transforms: | |
| - taskgraph.transforms.release_deps:transforms | |
| - taskgraph.transforms.bouncer_check:transforms | |
| - taskgraph.transforms.job:transforms | |
| - taskgraph.transforms.task:transforms | |
| job-defaults: | |
| - name: bouncer-check | |
| - description: bouncer check | |
| + name: release-bouncer-check | |
| + description: release bouncer check | |
| run-on-projects: [] # to make sure this never runs as part of CI | |
| shipping-phase: push | |
| worker-type: aws-provisioner-v1/gecko-{level}-b-linux | |
| worker: | |
| max-run-time: 1200 | |
| docker-image: {in-tree: "update-verify"} | |
| run: | |
| using: run-task | |
| @@ -33,17 +33,17 @@ job-defaults: | |
| kind: test | |
| tier: 1 | |
| jobs: | |
| firefox: | |
| shipping-product: firefox | |
| index: | |
| product: firefox | |
| - job-name: firefox-bouncer-check | |
| + job-name: firefox-release-bouncer-check | |
| run: | |
| config: | |
| by-project: | |
| mozilla-beta: | |
| - releases/bouncer_firefox_beta.py | |
| mozilla-release: | |
| - releases/bouncer_firefox_release.py | |
| mozilla-esr60: | |
| @@ -54,17 +54,17 @@ jobs: | |
| - releases/dev_bouncer_firefox_beta.py | |
| treeherder: | |
| platform: firefox-release/opt | |
| devedition: | |
| shipping-product: devedition | |
| index: | |
| product: devedition | |
| - job-name: devedition-bouncer-check | |
| + job-name: devedition-release-bouncer-check | |
| run: | |
| config: | |
| by-project: | |
| mozilla-beta: | |
| - releases/bouncer_firefox_devedition.py | |
| default: | |
| - releases/dev_bouncer_firefox_devedition.py | |
| treeherder: | |
| diff --git a/taskcluster/docs/kinds.rst b/taskcluster/docs/kinds.rst | |
| --- a/taskcluster/docs/kinds.rst | |
| +++ b/taskcluster/docs/kinds.rst | |
| @@ -295,19 +295,23 @@ Submits bouncer updates for releases. | |
| release-mark-as-shipped | |
| ----------------------- | |
| Marks releases as shipped in Ship-It v1 | |
| release-bouncer-aliases | |
| ----------------------- | |
| Update Bouncer's (download.mozilla.org) "latest" aliases. | |
| +bouncer-check | |
| +------------- | |
| +Checks Bouncer (download.mozilla.org) uptake. | |
| + | |
| release-bouncer-check | |
| --------------------- | |
| -Checks Bouncer (download.mozilla.org) uptake. | |
| +Checks Bouncer (download.mozilla.org) uptake as part of the release tasks. | |
| release-generate-checksums | |
| -------------------------- | |
| Generate the per-release checksums along with the summaries | |
| release-generate-checksums-signing | |
| ---------------------------------- | |
| Sign the pre-release checksums produced by the above task | |
| diff --git a/taskcluster/taskgraph/target_tasks.py b/taskcluster/taskgraph/target_tasks.py | |
| --- a/taskcluster/taskgraph/target_tasks.py | |
| +++ b/taskcluster/taskgraph/target_tasks.py | |
| @@ -554,8 +554,18 @@ def target_tasks_searchfox(full_task_gra | |
| @_target_task('file_update') | |
| def target_tasks_file_update(full_task_graph, parameters, graph_config): | |
| """Select the set of tasks required to perform nightly in-tree file updates | |
| """ | |
| def filter(task): | |
| # For now any task in the repo-update kind is ok | |
| return task.kind in ['repo-update'] | |
| return [l for l, t in full_task_graph.tasks.iteritems() if filter(t)] | |
| + | |
| + | |
| +@_target_task('bouncer_check') | |
| +def target_tasks_bouncer_check(full_task_graph, parameters, graph_config): | |
| + """Select the set of tasks required to perform bouncer version verification. | |
| + """ | |
| + def filter(task): | |
| + # For now any task in the repo-update kind is ok | |
| + return task.kind in ['bouncer-check'] | |
| + return [l for l, t in full_task_graph.tasks.iteritems() if filter(t)] | |
| diff --git a/taskcluster/taskgraph/transforms/bouncer_check.py b/taskcluster/taskgraph/transforms/bouncer_check.py | |
| --- a/taskcluster/taskgraph/transforms/bouncer_check.py | |
| +++ b/taskcluster/taskgraph/transforms/bouncer_check.py | |
| @@ -16,25 +16,22 @@ from taskgraph.util.schema import ( | |
| import logging | |
| logger = logging.getLogger(__name__) | |
| transforms = TransformSequence() | |
| @transforms.add | |
| def add_command(config, jobs): | |
| - release_config = get_release_config(config) | |
| - version = release_config["version"] | |
| for job in jobs: | |
| job = copy.deepcopy(job) # don't overwrite dict values here | |
| command = [ | |
| "cd", "/builds/worker/checkouts/gecko", "&&", | |
| "./mach", "python", | |
| "testing/mozharness/scripts/release/bouncer_check.py", | |
| - "--version={}".format(version), | |
| ] | |
| job["run"]["command"] = command | |
| yield job | |
| @transforms.add | |
| def add_previous_versions(config, jobs): | |
| release_config = get_release_config(config) | |
| @@ -52,27 +49,42 @@ def add_previous_versions(config, jobs): | |
| yield job | |
| @transforms.add | |
| def handle_keyed_by(config, jobs): | |
| """Resolve fields that can be keyed by project, etc.""" | |
| fields = [ | |
| "run.config", | |
| + "run.product-field", | |
| "run.extra-config", | |
| ] | |
| + | |
| + release_config = get_release_config(config) | |
| + version = release_config["version"] | |
| + | |
| for job in jobs: | |
| job = copy.deepcopy(job) # don't overwrite dict values here | |
| for field in fields: | |
| resolve_keyed_by(item=job, field=field, item_name=job['name'], | |
| project=config.params['project']) | |
| for cfg in job["run"]["config"]: | |
| job["run"]["command"].extend(["--config", cfg]) | |
| + if config.kind == "bouncer-check": | |
| + job["run"]["command"].extend([ | |
| + "--product-field={}".format(job["run"]["product-field"]), | |
| + "--products-url={}".format(job["run"]["products-url"]), | |
| + ]) | |
| + del job["run"]["product-field"] | |
| + del job["run"]["products-url"] | |
| + elif config.kind == "release-bouncer-check": | |
| + job["run"]["command"].append("--version={}".format(version)) | |
| + | |
| del job["run"]["config"] | |
| if 'extra-config' in job['run']: | |
| env = job['worker'].setdefault('env', {}) | |
| env['EXTRA_MOZHARNESS_CONFIG'] = json.dumps(job['run']['extra-config']) | |
| del job["run"]["extra-config"] | |
| yield job | |
| diff --git a/testing/mozharness/scripts/release/bouncer_check.py b/testing/mozharness/scripts/release/bouncer_check.py | |
| --- a/testing/mozharness/scripts/release/bouncer_check.py | |
| +++ b/testing/mozharness/scripts/release/bouncer_check.py | |
| @@ -22,16 +22,26 @@ BOUNCER_URL_PATTERN = "{bouncer_prefix}? | |
| class BouncerCheck(BaseScript, VirtualenvMixin): | |
| config_options = [ | |
| [["--version"], { | |
| "dest": "version", | |
| "help": "Version of release, eg: 39.0b5", | |
| }], | |
| + [["--product-field"], { | |
| + "dest": "product_field", | |
| + "help": "Version field of release from product details, eg: LATEST_FIREFOX_VERSION", | |
| + }], | |
| + [["--products-url"], { | |
| + "dest": "products_url", | |
| + "help": "The URL of the current Firefox product versions", | |
| + "type": str, | |
| + "default": "https://product-details.mozilla.org/1.0/firefox_versions.json", | |
| + }], | |
| [["--previous-version"], { | |
| "dest": "prev_versions", | |
| "action": "extend", | |
| "help": "Previous version(s)", | |
| }], | |
| [["--locale"], { | |
| "dest": "locales", | |
| # Intentionally limited for several reasons: | |
| @@ -71,28 +81,60 @@ class BouncerCheck(BaseScript, Virtualen | |
| ], | |
| default_actions=[ | |
| "create-virtualenv", | |
| "activate-virtualenv", | |
| "check-bouncer", | |
| ], | |
| ) | |
| + def _pre_config_lock(self, rw_config): | |
| + super(BouncerCheck, self)._pre_config_lock(rw_config) | |
| + | |
| + if "product_field" not in self.config: | |
| + return | |
| + | |
| + firefox_versions = self.load_json_url(self.config["products_url"]) | |
| + | |
| + if self.config['product_field'] not in firefox_versions: | |
| + self.fatal('Unknown Firefox label: {}'.format(self.config['product_field'])) | |
| + self.config["version"] = firefox_versions[self.config["product_field"]] | |
| + self.log("Set Firefox version {}".format(self.config["version"])) | |
| + | |
| def check_url(self, session, url): | |
| from redo import retry | |
| + try: | |
| + from urllib.parse import urlparse | |
| + except ImportError: | |
| + # Python 2 | |
| + from urlparse import urlparse | |
| + | |
| + mozilla_locations = [ | |
| + 'download-installer.cdn.mozilla.net', | |
| + 'download.cdn.mozilla.net', | |
| + 'download.mozilla.org', | |
| + 'archive.mozilla.org', | |
| + ] | |
| def do_check_url(): | |
| self.log("Checking {}".format(url)) | |
| r = session.head(url, verify=True, timeout=10, allow_redirects=True) | |
| try: | |
| r.raise_for_status() | |
| except Exception: | |
| self.warning("FAIL: {}, status: {}".format(url, r.status_code)) | |
| raise | |
| + final_url = urlparse(r.url) | |
| + if final_url.scheme != 'https': | |
| + self.warning('FAIL: URL scheme is not https: {}'.format(r.url)) | |
| + | |
| + if final_url.netloc not in mozilla_locations: | |
| + self.warning('FAIL: host not in allowed locations: {}'.format(r.url)) | |
| + | |
| retry(do_check_url, sleeptime=3, max_sleeptime=10, attempts=3) | |
| def get_urls(self): | |
| for product in self.config["products"].values(): | |
| if not product["check_uptake"]: | |
| continue | |
| product_name = product["product-name"] % {"version": self.config["version"]} | |
| for path in product["paths"].values(): | |
| changeset: 502890:bb1676d26e9c | |
| tag: tip | |
| user: Mihai Tabara <mtabara@mozilla.com> | |
| date: Tue Oct 16 13:07:30 2018 +0100 | |
| summary: Bug 1492639 - fixes to bouncer check. r=sfraser a=release | |
| diff --git a/.cron.yml b/.cron.yml | |
| --- a/.cron.yml | |
| +++ b/.cron.yml | |
| @@ -105,16 +105,17 @@ jobs: | |
| - name: bouncer-check | |
| job: | |
| type: decision-task | |
| treeherder-symbol: Rel | |
| target-tasks-method: bouncer_check | |
| run-on-projects: | |
| - mozilla-beta | |
| - mozilla-release | |
| + - mozilla-esr60 | |
| when: | |
| by-project: | |
| # No default branch | |
| mozilla-beta: | |
| - {hour: 7, minute: 0} | |
| - {hour: 19, minute: 0} | |
| mozilla-release: | |
| - {hour: 7, minute: 0} | |
| diff --git a/taskcluster/ci/bouncer-check/kind.yml b/taskcluster/ci/bouncer-check/kind.yml | |
| --- a/taskcluster/ci/bouncer-check/kind.yml | |
| +++ b/taskcluster/ci/bouncer-check/kind.yml | |
| @@ -37,16 +37,18 @@ jobs: | |
| job-name: firefox-bouncer-check | |
| run: | |
| config: | |
| by-project: | |
| mozilla-beta: | |
| - releases/bouncer_firefox_beta.py | |
| mozilla-release: | |
| - releases/bouncer_firefox_release.py | |
| + mozilla-esr60: | |
| + - releases/bouncer_firefox_esr.py | |
| default: | |
| - releases/dev_bouncer_firefox_beta.py | |
| product-field: | |
| by-project: | |
| mozilla-beta: LATEST_FIREFOX_RELEASED_DEVEL_VERSION | |
| mozilla-release: LATEST_FIREFOX_VERSION | |
| mozilla-esr60: FIREFOX_ESR | |
| default: LATEST_FIREFOX_DEVEL_VERSION | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment