Skip to content

Instantly share code, notes, and snippets.

@wlach

wlach/gist:01e8727f2aba66af613400156a3825ec

Created January 16, 2020 20:30
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save wlach/01e8727f2aba66af613400156a3825ec to your computer and use it in GitHub Desktop.
Save wlach/01e8727f2aba66af613400156a3825ec to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
1
2
3
*** Reading local file: /app/logs/missioncontrol/missioncontrol_etl/2020-01-15T22:40:35/3.log
[2020-01-15 23:14:06,576] {models.py:1359} INFO - Dependencies all met for <TaskInstance: missioncontrol.missioncontrol_etl 2020-01-15T22:40:35.339783+00:00 [queued]>
[2020-01-15 23:14:06,579] {models.py:1359} INFO - Dependencies all met for <TaskInstance: missioncontrol.missioncontrol_etl 2020-01-15T22:40:35.339783+00:00 [queued]>
[2020-01-15 23:14:06,580] {models.py:1571} INFO -
--------------------------------------------------------------------------------
Starting attempt 3 of 3
--------------------------------------------------------------------------------
[2020-01-15 23:14:06,589] {models.py:1593} INFO - Executing <Task(GKEPodOperator): missioncontrol_etl> on 2020-01-15T22:40:35.339783+00:00
[2020-01-15 23:14:06,590] {base_task_runner.py:118} INFO - Running: ['bash', '-c', u'airflow run missioncontrol missioncontrol_etl 2020-01-15T22:40:35.339783+00:00 --job_id 188 --raw -sd DAGS_FOLDER/mission_control.py --cfg_path /tmp/tmpegugvK']
[2020-01-15 23:14:07,402] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl [2020-01-15 23:14:07,401] {settings.py:174} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=3600, pid=2041
[2020-01-15 23:14:08,028] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl [2020-01-15 23:14:08,027] {default_celery.py:90} WARNING - You have configured a result_backend of redis://redis:6379/0, it is highly recommended to use an alternative result_backend (i.e. a database).
[2020-01-15 23:14:08,169] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl /usr/local/lib/python2.7/site-packages/airflow/utils/helpers.py:356: DeprecationWarning: Importing 'BaseSensorOperator' directly from 'airflow.operators' has been deprecated. Please import from 'airflow.operators.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
[2020-01-15 23:14:08,170] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl DeprecationWarning)
[2020-01-15 23:14:08,423] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl [2020-01-15 23:14:08,423] {__init__.py:51} INFO - Using executor CeleryExecutor
[2020-01-15 23:14:08,586] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl [2020-01-15 23:14:08,585] {models.py:273} INFO - Filling up the DagBag from /app/dags/mission_control.py
[2020-01-15 23:14:09,026] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl [2020-01-15 23:14:09,026] {cli.py:520} INFO - Running <TaskInstance: missioncontrol.missioncontrol_etl 2020-01-15T22:40:35.339783+00:00 [running]> on host cb4a5a73da07
[2020-01-15 23:14:09,416] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl Fetching cluster endpoint and auth data.
[2020-01-15 23:14:09,852] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl kubeconfig entry generated for bq-load-gke-1.
[2020-01-15 23:14:10,041] {logging_mixin.py:95} WARNING - /usr/local/lib/python2.7/site-packages/airflow/contrib/kubernetes/kubernetes_request_factory/pod_request_factory.py:44: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
req = yaml.load(self._yaml)
[2020-01-15 23:14:10,536] {logging_mixin.py:95} INFO - [2020-01-15 23:14:10,536] {pod_launcher.py:121} INFO - Event: missioncontrol-etl-bc5f9e44 had an event of type Pending
[2020-01-15 23:14:11,622] {logging_mixin.py:95} INFO - [2020-01-15 23:14:11,621] {pod_launcher.py:121} INFO - Event: missioncontrol-etl-bc5f9e44 had an event of type Pending
[2020-01-15 23:14:12,669] {logging_mixin.py:95} INFO - [2020-01-15 23:14:12,668] {pod_launcher.py:121} INFO - Event: missioncontrol-etl-bc5f9e44 had an event of type Running
[2020-01-15 23:14:12,751] {logging_mixin.py:95} INFO - [2020-01-15 23:14:12,751] {pod_launcher.py:104} INFO - + echo 'Checking credentials...'
[2020-01-15 23:14:12,753] {logging_mixin.py:95} INFO - [2020-01-15 23:14:12,753] {pod_launcher.py:104} INFO - + gsutil ls gs://missioncontrol-v2
[2020-01-15 23:14:12,754] {logging_mixin.py:95} INFO - [2020-01-15 23:14:12,754] {pod_launcher.py:104} INFO - No JSON credentials provided, using default scopes.
[2020-01-15 23:14:12,755] {logging_mixin.py:95} INFO - [2020-01-15 23:14:12,755] {pod_launcher.py:104} INFO - Checking credentials...
[2020-01-15 23:14:12,756] {logging_mixin.py:95} INFO - [2020-01-15 23:14:12,756] {pod_launcher.py:104} INFO - gs://missioncontrol-v2/docker-compose.yml
[2020-01-15 23:14:12,757] {logging_mixin.py:95} INFO - [2020-01-15 23:14:12,757] {pod_launcher.py:104} INFO - gs://missioncontrol-v2//
[2020-01-15 23:14:12,758] {logging_mixin.py:95} INFO - [2020-01-15 23:14:12,758] {pod_launcher.py:104} INFO - gs://missioncontrol-v2/html/
[2020-01-15 23:14:12,758] {logging_mixin.py:95} INFO - [2020-01-15 23:14:12,758] {pod_launcher.py:104} INFO - + echo 'Running etl.R'
[2020-01-15 23:14:12,759] {logging_mixin.py:95} INFO - [2020-01-15 23:14:12,759] {pod_launcher.py:104} INFO - + Rscript etl.R
[2020-01-15 23:14:12,761] {logging_mixin.py:95} INFO - [2020-01-15 23:14:12,760] {pod_launcher.py:104} INFO - Running etl.R
[2020-01-15 23:14:14,353] {logging_mixin.py:95} INFO - [2020-01-15 23:14:14,353] {pod_launcher.py:104} INFO - Warning messages:
[2020-01-15 23:14:14,355] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:14:14,356] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:14:14,357] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:14:14,358] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:14:14,358] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:14:14,359] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:14:14,360] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:14:14,361] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 11: ordinal not in range(128)
[2020-01-15 23:14:14,361] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:14:14,362] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:14:14,363] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:14:14,364] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:14:14,364] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:14:14,364] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:14:14,365] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:14:14,366] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:14:14,366] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 11: ordinal not in range(128)
[2020-01-15 23:14:14,367] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:14:14,367] {logging_mixin.py:95} INFO - [2020-01-15 23:14:14,367] {pod_launcher.py:104} INFO - 2020-01-15 23:14:14 INFO::Starting Downloading Data
[2020-01-15 23:17:07,656] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,656] {pod_launcher.py:104} INFO - 2020-01-15 23:17:07 INFO::Not using cached queries
[2020-01-15 23:17:07,658] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,658] {pod_launcher.py:104} INFO - Starting data pull
[2020-01-15 23:17:07,659] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,659] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:07,660] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:07,661] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:07,662] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:07,663] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:07,664] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:07,664] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:07,665] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:07,666] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 96: ordinal not in range(128)
[2020-01-15 23:17:07,666] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:07,667] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:07,668] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:07,668] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:07,669] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:07,669] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:07,670] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:07,671] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:07,671] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 73: ordinal not in range(128)
[2020-01-15 23:17:07,672] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:07,672] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,672] {pod_launcher.py:104} INFO - pulling version=72.0.1
[2020-01-15 23:17:07,673] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,673] {pod_launcher.py:104} INFO - Dropping 17 rows that don't match `(date <= peak_date) or isLatest`:
[2020-01-15 23:17:07,674] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,673] {pod_launcher.py:104} INFO - date os c_version
[2020-01-15 23:17:07,674] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,674] {pod_launcher.py:104} INFO - 3 2020-01-10 Darwin 72.0
[2020-01-15 23:17:07,675] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,675] {pod_launcher.py:104} INFO - 4 2020-01-11 Darwin 72.0
[2020-01-15 23:17:07,675] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,675] {pod_launcher.py:104} INFO - 5 2020-01-12 Darwin 72.0
[2020-01-15 23:17:07,676] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,676] {pod_launcher.py:104} INFO - 6 2020-01-13 Darwin 72.0
[2020-01-15 23:17:07,677] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,677] {pod_launcher.py:104} INFO - 7 2020-01-14 Darwin 72.0
[2020-01-15 23:17:07,678] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,678] {pod_launcher.py:104} INFO - 10 2020-01-09 Linux 72.0
[2020-01-15 23:17:07,679] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,679] {pod_launcher.py:104} INFO - 11 2020-01-10 Linux 72.0
[2020-01-15 23:17:07,679] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,679] {pod_launcher.py:104} INFO - 12 2020-01-11 Linux 72.0
[2020-01-15 23:17:07,680] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,680] {pod_launcher.py:104} INFO - 13 2020-01-12 Linux 72.0
[2020-01-15 23:17:07,681] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,681] {pod_launcher.py:104} INFO - 14 2020-01-13 Linux 72.0
[2020-01-15 23:17:07,682] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,682] {pod_launcher.py:104} INFO - 15 2020-01-14 Linux 72.0
[2020-01-15 23:17:07,682] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,682] {pod_launcher.py:104} INFO - 18 2020-01-09 Windows_NT 72.0
[2020-01-15 23:17:07,683] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,683] {pod_launcher.py:104} INFO - 19 2020-01-10 Windows_NT 72.0
[2020-01-15 23:17:07,684] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,684] {pod_launcher.py:104} INFO - 20 2020-01-11 Windows_NT 72.0
[2020-01-15 23:17:07,685] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,685] {pod_launcher.py:104} INFO - 21 2020-01-12 Windows_NT 72.0
[2020-01-15 23:17:07,686] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,686] {pod_launcher.py:104} INFO - 22 2020-01-13 Windows_NT 72.0
[2020-01-15 23:17:07,686] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,686] {pod_launcher.py:104} INFO - 23 2020-01-14 Windows_NT 72.0
[2020-01-15 23:17:07,687] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,687] {pod_launcher.py:104} INFO - Done.
[2020-01-15 23:17:07,688] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,688] {pod_launcher.py:104} INFO - Beta metadata:
[2020-01-15 23:17:07,689] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,689] {pod_launcher.py:104} INFO - date version src maj_vers till
[2020-01-15 23:17:07,689] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,689] {pod_launcher.py:104} INFO - 96 2020-01-08 73.0b2 product-details 73 2020-01-08
[2020-01-15 23:17:07,690] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,690] {pod_launcher.py:104} INFO - 97 2020-01-08 72.0.1 buildhub 72 2020-01-10
[2020-01-15 23:17:07,690] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,690] {pod_launcher.py:104} INFO - 98 2020-01-10 73.0b3 product-details 73 2020-01-13
[2020-01-15 23:17:07,691] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,691] {pod_launcher.py:104} INFO - 99 2020-01-13 73.0b4 product-details 73 2020-01-15
[2020-01-15 23:17:07,692] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,692] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:07,692] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,692] {pod_launcher.py:104} INFO - Pulling beta data...
[2020-01-15 23:17:07,693] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:07,694] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:07,694] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:07,695] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:07,695] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:07,696] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:07,696] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:07,697] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 73: ordinal not in range(128)
[2020-01-15 23:17:07,697] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:07,698] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,698] {pod_launcher.py:104} INFO - date os c_version
[2020-01-15 23:17:07,699] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,699] {pod_launcher.py:104} INFO - 6 2020-01-13 Darwin 73.0b3
[2020-01-15 23:17:07,699] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,699] {pod_launcher.py:104} INFO - 10 2020-01-13 Linux 73.0b3
[2020-01-15 23:17:07,700] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,700] {pod_launcher.py:104} INFO - 14 2020-01-13 Windows_NT 73.0b3
[2020-01-15 23:17:07,701] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,700] {pod_launcher.py:104} INFO - Done.
[2020-01-15 23:17:07,701] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,701] {pod_launcher.py:104} INFO - Nightly metadata:
[2020-01-15 23:17:07,702] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,702] {pod_launcher.py:104} INFO - build_id release_date till current_beta nightly_display_version
[2020-01-15 23:17:07,702] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,702] {pod_launcher.py:104} INFO - 0 20200112 2020-01-12 2020-01-15 73.0b3 74.0a1
[2020-01-15 23:17:07,703] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,703] {pod_launcher.py:104} INFO - 1 20200113 2020-01-13 2020-01-15 73.0b4 74.0a1
[2020-01-15 23:17:07,704] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,704] {pod_launcher.py:104} INFO - 2 20200114 2020-01-14 2020-01-15 73.0b4 74.0a1
[2020-01-15 23:17:07,704] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,704] {pod_launcher.py:104} INFO - 3 20200115 2020-01-15 2020-01-15 73.0b5 74.0a1
[2020-01-15 23:17:07,705] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,705] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:07,705] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,705] {pod_launcher.py:104} INFO - Pulling nightly data...
[2020-01-15 23:17:07,706] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:07,707] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:07,707] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:07,708] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:07,708] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:07,709] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:07,709] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:07,710] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 72: ordinal not in range(128)
[2020-01-15 23:17:07,710] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:07,711] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:07,711] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:07,712] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:07,712] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:07,713] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:07,713] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:07,714] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:07,714] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 72: ordinal not in range(128)
[2020-01-15 23:17:07,715] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:07,715] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:07,716] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:07,716] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:07,717] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:07,717] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:07,718] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:07,718] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:07,719] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 72: ordinal not in range(128)
[2020-01-15 23:17:07,719] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:07,720] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,720] {pod_launcher.py:104} INFO -
Downloading: 0rows [00:00, ?rows/s]
Downloading: 0rows [00:00, ?rows/s]
[2020-01-15 23:17:07,720] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:07,721] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:07,721] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:07,722] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:07,722] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:07,723] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:07,724] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:07,724] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 72: ordinal not in range(128)
[2020-01-15 23:17:07,725] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:07,725] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,725] {pod_launcher.py:104} INFO - pulling build_id=20200112
[2020-01-15 23:17:07,726] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,726] {pod_launcher.py:104} INFO - pulling build_id=20200113
[2020-01-15 23:17:07,727] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,727] {pod_launcher.py:104} INFO - pulling build_id=20200114
[2020-01-15 23:17:07,727] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,727] {pod_launcher.py:104} INFO - pulling build_id=20200115
[2020-01-15 23:17:07,728] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,728] {pod_launcher.py:104} INFO - Dropping 3 rows that don't match `date <= peak_date`:
[2020-01-15 23:17:07,728] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,728] {pod_launcher.py:104} INFO - date os c_version
[2020-01-15 23:17:07,729] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,729] {pod_launcher.py:104} INFO - 2 2020-01-14 Darwin 20200112
[2020-01-15 23:17:07,730] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,730] {pod_launcher.py:104} INFO - 5 2020-01-14 Linux 20200112
[2020-01-15 23:17:07,730] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,730] {pod_launcher.py:104} INFO - 8 2020-01-14 Windows_NT 20200112
[2020-01-15 23:17:07,731] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,731] {pod_launcher.py:104} INFO - Done.
[2020-01-15 23:17:07,732] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,732] {pod_launcher.py:104} INFO - Executing `
[2020-01-15 23:17:07,732] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,732] {pod_launcher.py:104} INFO - delete from `wlach-test-258214`.analysis.missioncontrol_v2_raw_data_test
[2020-01-15 23:17:07,733] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,733] {pod_launcher.py:104} INFO - where (channel='beta' and c_version='72.0.1' and date in ('2020-01-08', '2020-01-09', '2020-01-10'))
[2020-01-15 23:17:07,734] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,734] {pod_launcher.py:104} INFO - OR (channel='beta' and c_version='73.0b2' and date in ('2020-01-08', '2020-01-08', '2020-01-08'))
[2020-01-15 23:17:07,734] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,734] {pod_launcher.py:104} INFO - OR (channel='beta' and c_version='73.0b3' and date in ('2020-01-10', '2020-01-11', '2020-01-12', '2020-01-10', '2020-01-11', '2020-01-12', '2020-01-10', '2020-01-11', '2020-01-12'))
[2020-01-15 23:17:07,735] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,735] {pod_launcher.py:104} INFO - OR (channel='beta' and c_version='73.0b4' and date in ('2020-01-13', '2020-01-14', '2020-01-13', '2020-01-14', '2020-01-13', '2020-01-14'))
[2020-01-15 23:17:07,736] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,735] {pod_launcher.py:104} INFO - OR (channel='nightly' and c_version='20200112' and date in ('2020-01-12', '2020-01-13', '2020-01-12', '2020-01-13', '2020-01-12', '2020-01-13'))
[2020-01-15 23:17:07,736] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,736] {pod_launcher.py:104} INFO - OR (channel='nightly' and c_version='20200113' and date in ('2020-01-13', '2020-01-14', '2020-01-13', '2020-01-14', '2020-01-13', '2020-01-14'))
[2020-01-15 23:17:07,737] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,737] {pod_launcher.py:104} INFO - OR (channel='nightly' and c_version='20200114' and date in ('2020-01-14', '2020-01-14', '2020-01-14'))
[2020-01-15 23:17:07,737] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,737] {pod_launcher.py:104} INFO - OR (channel='release' and c_version='72.0' and date in ('2020-01-07', '2020-01-08', '2020-01-09', '2020-01-07', '2020-01-08', '2020-01-07', '2020-01-08'))
[2020-01-15 23:17:07,738] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,738] {pod_launcher.py:104} INFO - OR (channel='release' and c_version='72.0.1' and date in ('2020-01-08', '2020-01-09', '2020-01-10', '2020-01-11', '2020-01-12', '2020-01-13', '2020-01-14', '2020-01-08', '2020-01-09', '2020-01-10', '2020-01-11', '2020-01-12', '2020-01-13', '2020-01-14', '2020-01-08', '2020-01-09', '2020-01-10', '2020-01-11', '2020-01-12', '2020-01-13', '2020-01-14'))
[2020-01-15 23:17:07,739] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,739] {pod_launcher.py:104} INFO - `... Done.
[2020-01-15 23:17:07,739] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,739] {pod_launcher.py:104} INFO - CSV saved to /tmp/tmpgvd_3ccm
[2020-01-15 23:17:07,740] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,740] {pod_launcher.py:104} INFO - Success! Data uploaded to analysis.missioncontrol_v2_raw_data_test
[2020-01-15 23:17:07,741] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,740] {pod_launcher.py:104} INFO -
Upload complete.
[2020-01-15 23:17:07,741] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,741] {pod_launcher.py:104} INFO -
Waiting on bqjob_r462a3452acdec00a_0000016fab7ef706_1 ... (0s) Current status: RUNNING
Waiting on bqjob_r462a3452acdec00a_0000016fab7ef706_1 ... (1s) Current status: RUNNING
Waiting on bqjob_r462a3452acdec00a_0000016fab7ef706_1 ... (1s) Current status: DONE
[2020-01-15 23:17:07,742] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,742] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:07,742] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,742] {pod_launcher.py:104} INFO - n_rows dau_cversion_mm
[2020-01-15 23:17:07,743] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,743] {pod_launcher.py:104} INFO - 0 3530 1.383886
[2020-01-15 23:17:07,744] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,744] {pod_launcher.py:104} INFO - => `wlach-test-258214`.analysis.missioncontrol_v2_raw_data_test now has 3530 rows
[2020-01-15 23:17:07,744] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,744] {pod_launcher.py:104} INFO - 2020-01-15 23:17:07 INFO::Finished Downloading Data
[2020-01-15 23:17:07,745] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,745] {pod_launcher.py:104} INFO - + echo 'Running build.models.firefox.desktop.R'
[2020-01-15 23:17:07,745] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,745] {pod_launcher.py:104} INFO - + Rscript build.models.firefox.desktop.R --simple=0 --out=./all.the.data.intermediate.Rdata
[2020-01-15 23:17:07,746] {logging_mixin.py:95} INFO - [2020-01-15 23:17:07,746] {pod_launcher.py:104} INFO - Running build.models.firefox.desktop.R
[2020-01-15 23:17:09,938] {logging_mixin.py:95} INFO - [2020-01-15 23:17:09,938] {pod_launcher.py:104} INFO - Warning messages:
[2020-01-15 23:17:09,939] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:09,939] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:09,940] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:09,941] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:09,941] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:09,942] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:09,943] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:09,943] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 11: ordinal not in range(128)
[2020-01-15 23:17:09,944] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:09,945] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:09,945] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:09,946] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:09,947] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:09,947] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:09,948] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:09,948] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:09,949] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 11: ordinal not in range(128)
[2020-01-15 23:17:09,950] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:09,950] {logging_mixin.py:95} INFO - [2020-01-15 23:17:09,950] {pod_launcher.py:104} INFO - 2020-01-15 23:17:09 INFO::Using production models
[2020-01-15 23:17:09,960] {logging_mixin.py:95} INFO - [2020-01-15 23:17:09,960] {pod_launcher.py:104} INFO - 2020-01-15 23:17:09 INFO::Downloading raw data for channel release
[2020-01-15 23:17:09,962] {logging_mixin.py:95} INFO - [2020-01-15 23:17:09,962] {pod_launcher.py:104} INFO - 2020-01-15 23:17:09 INFO::Starting Gettting Model Data for channel release and nversions 3
[2020-01-15 23:17:12,130] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:12,131] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:12,132] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:12,133] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:12,134] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:12,134] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:12,135] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:12,136] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 97: ordinal not in range(128)
[2020-01-15 23:17:12,137] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:12,138] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:12,138] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:12,139] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:12,140] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:12,141] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:12,141] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:12,142] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:12,143] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 74: ordinal not in range(128)
[2020-01-15 23:17:12,143] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:12,144] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,144] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:12,145] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,145] {pod_launcher.py:104} INFO - select distinct major from `wlach-test-258214`.analysis.missioncontrol_v2_raw_data
[2020-01-15 23:17:12,146] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,146] {pod_launcher.py:104} INFO - where channel = 'release'
[2020-01-15 23:17:12,146] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,146] {pod_launcher.py:104} INFO - ORDER BY major desc
[2020-01-15 23:17:12,147] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,147] {pod_launcher.py:104} INFO - LIMIT 3
[2020-01-15 23:17:12,148] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,148] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:12,149] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,149] {pod_launcher.py:104} INFO - Previous majors for `release` were: [72, 71, 70]. Pulling now...
[2020-01-15 23:17:12,149] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,149] {pod_launcher.py:104} INFO - Running query:
[2020-01-15 23:17:12,150] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,150] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:12,151] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,151] {pod_launcher.py:104} INFO - select * from `wlach-test-258214`.analysis.missioncontrol_v2_raw_data
[2020-01-15 23:17:12,151] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,151] {pod_launcher.py:104} INFO - where channel = 'release'
[2020-01-15 23:17:12,152] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,152] {pod_launcher.py:104} INFO - and major in (72, 71, 70)
[2020-01-15 23:17:12,153] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,153] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:12,154] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,153] {pod_launcher.py:104} INFO - feather file saved to /tmp/RtmpSiv5KV/filee72f0f11e3
[2020-01-15 23:17:12,154] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,154] {pod_launcher.py:104} INFO - /tmp/RtmpSiv5KV/filee72f0f11e3
[2020-01-15 23:17:12,155] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,155] {pod_launcher.py:104} INFO - 2020-01-15 23:17:12 INFO::Finished Gettting Model Data for channel release and nversions 3
[2020-01-15 23:17:12,155] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,155] {pod_launcher.py:104} INFO - There were 13 warnings (use warnings() to see them)
[2020-01-15 23:17:12,385] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,384] {pod_launcher.py:104} INFO - 2020-01-15 23:17:12 INFO::Downloading raw data for channel beta
[2020-01-15 23:17:12,387] {logging_mixin.py:95} INFO - [2020-01-15 23:17:12,386] {pod_launcher.py:104} INFO - 2020-01-15 23:17:12 INFO::Starting Gettting Model Data for channel beta and nversions 3
[2020-01-15 23:17:14,544] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:14,544] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:14,546] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:14,546] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:14,547] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:14,547] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:14,548] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:14,548] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 97: ordinal not in range(128)
[2020-01-15 23:17:14,549] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:14,549] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:14,550] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:14,551] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:14,552] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:14,553] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:14,554] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:14,556] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:14,557] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 74: ordinal not in range(128)
[2020-01-15 23:17:14,558] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:14,559] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,559] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:14,561] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,560] {pod_launcher.py:104} INFO - select distinct major from `wlach-test-258214`.analysis.missioncontrol_v2_raw_data
[2020-01-15 23:17:14,562] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,562] {pod_launcher.py:104} INFO - where channel = 'beta'
[2020-01-15 23:17:14,564] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,563] {pod_launcher.py:104} INFO - ORDER BY major desc
[2020-01-15 23:17:14,565] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,565] {pod_launcher.py:104} INFO - LIMIT 3
[2020-01-15 23:17:14,566] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,566] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:14,567] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,567] {pod_launcher.py:104} INFO - Previous majors for `beta` were: [73, 72, 71]. Pulling now...
[2020-01-15 23:17:14,569] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,568] {pod_launcher.py:104} INFO - Running query:
[2020-01-15 23:17:14,569] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,569] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:14,571] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,570] {pod_launcher.py:104} INFO - select * from `wlach-test-258214`.analysis.missioncontrol_v2_raw_data
[2020-01-15 23:17:14,572] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,571] {pod_launcher.py:104} INFO - where channel = 'beta'
[2020-01-15 23:17:14,573] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,573] {pod_launcher.py:104} INFO - and major in (73, 72, 71)
[2020-01-15 23:17:14,574] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,574] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:14,576] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,576] {pod_launcher.py:104} INFO - feather file saved to /tmp/RtmpSiv5KV/filee72f29a5ee
[2020-01-15 23:17:14,577] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,577] {pod_launcher.py:104} INFO - /tmp/RtmpSiv5KV/filee72f29a5ee
[2020-01-15 23:17:14,579] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,579] {pod_launcher.py:104} INFO - 2020-01-15 23:17:14 INFO::Finished Gettting Model Data for channel beta and nversions 3
[2020-01-15 23:17:14,580] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,580] {pod_launcher.py:104} INFO - There were 13 warnings (use warnings() to see them)
[2020-01-15 23:17:14,581] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,581] {pod_launcher.py:104} INFO - 2020-01-15 23:17:14 INFO::Downloading raw data for channel nightly
[2020-01-15 23:17:14,582] {logging_mixin.py:95} INFO - [2020-01-15 23:17:14,582] {pod_launcher.py:104} INFO - 2020-01-15 23:17:14 INFO::Starting Gettting Model Data for channel nightly and nversions 3
[2020-01-15 23:17:16,611] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:16,612] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:16,613] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:16,613] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:16,614] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:16,615] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:16,615] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:16,616] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 97: ordinal not in range(128)
[2020-01-15 23:17:16,616] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:16,617] {logging_mixin.py:95} WARNING - Traceback (most recent call last):
[2020-01-15 23:17:16,617] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 868, in emit
[2020-01-15 23:17:16,617] {logging_mixin.py:95} WARNING - msg = self.format(record)
[2020-01-15 23:17:16,618] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 741, in format
[2020-01-15 23:17:16,618] {logging_mixin.py:95} WARNING - return fmt.format(record)
[2020-01-15 23:17:16,619] {logging_mixin.py:95} WARNING - File "/usr/local/lib/python2.7/logging/__init__.py", line 476, in format
[2020-01-15 23:17:16,619] {logging_mixin.py:95} WARNING - raise e
[2020-01-15 23:17:16,620] {logging_mixin.py:95} WARNING - UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 74: ordinal not in range(128)
[2020-01-15 23:17:16,620] {logging_mixin.py:95} WARNING - Logged from file pod_launcher.py, line 104
[2020-01-15 23:17:16,621] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,621] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:16,621] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,621] {pod_launcher.py:104} INFO - select distinct major from `wlach-test-258214`.analysis.missioncontrol_v2_raw_data
[2020-01-15 23:17:16,622] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,622] {pod_launcher.py:104} INFO - where channel = 'nightly'
[2020-01-15 23:17:16,623] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,623] {pod_launcher.py:104} INFO - ORDER BY major desc
[2020-01-15 23:17:16,623] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,623] {pod_launcher.py:104} INFO - LIMIT 3
[2020-01-15 23:17:16,624] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,624] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:16,625] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,624] {pod_launcher.py:104} INFO - Previous majors for `nightly` were: [74, 73, 72]. Pulling now...
[2020-01-15 23:17:16,625] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,625] {pod_launcher.py:104} INFO - Running query:
[2020-01-15 23:17:16,626] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,626] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:16,626] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,626] {pod_launcher.py:104} INFO - select * from `wlach-test-258214`.analysis.missioncontrol_v2_raw_data
[2020-01-15 23:17:16,627] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,627] {pod_launcher.py:104} INFO - where channel = 'nightly'
[2020-01-15 23:17:16,628] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,628] {pod_launcher.py:104} INFO - and major in (74, 73, 72)
[2020-01-15 23:17:16,628] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,628] {pod_launcher.py:104} INFO -
[2020-01-15 23:17:16,629] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,629] {pod_launcher.py:104} INFO - feather file saved to /tmp/RtmpSiv5KV/filee738108c39
[2020-01-15 23:17:16,629] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,629] {pod_launcher.py:104} INFO - /tmp/RtmpSiv5KV/filee738108c39
[2020-01-15 23:17:16,630] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,630] {pod_launcher.py:104} INFO - 2020-01-15 23:17:16 INFO::Finished Gettting Model Data for channel nightly and nversions 3
[2020-01-15 23:17:16,630] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,630] {pod_launcher.py:104} INFO - There were 13 warnings (use warnings() to see them)
[2020-01-15 23:17:16,631] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,631] {pod_launcher.py:104} INFO - 2020-01-15 23:17:16 INFO::Using following dates
[2020-01-15 23:17:16,632] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,632] {pod_launcher.py:104} INFO - os channel UsingDateTill
[2020-01-15 23:17:16,633] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,633] {pod_launcher.py:104} INFO - 1: Darwin release 2020-01-12
[2020-01-15 23:17:16,633] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,633] {pod_launcher.py:104} INFO - 2: Linux release 2020-01-12
[2020-01-15 23:17:16,634] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,634] {pod_launcher.py:104} INFO - 3: Windows_NT release 2020-01-12
[2020-01-15 23:17:16,635] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,635] {pod_launcher.py:104} INFO - os channel UsingDateTill
[2020-01-15 23:17:16,635] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,635] {pod_launcher.py:104} INFO - 1: Darwin beta 2020-01-12
[2020-01-15 23:17:16,636] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,635] {pod_launcher.py:104} INFO - 2: Linux beta 2020-01-12
[2020-01-15 23:17:16,636] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,636] {pod_launcher.py:104} INFO - 3: Windows_NT beta 2020-01-12
[2020-01-15 23:17:16,637] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,637] {pod_launcher.py:104} INFO - os channel UsingDateTill
[2020-01-15 23:17:16,637] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,637] {pod_launcher.py:104} INFO - 1: Darwin nightly 2020-01-12
[2020-01-15 23:17:16,638] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,638] {pod_launcher.py:104} INFO - 2: Linux nightly 2020-01-12
[2020-01-15 23:17:16,639] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,639] {pod_launcher.py:104} INFO - 3: Windows_NT nightly 2020-01-12
[2020-01-15 23:17:16,640] {logging_mixin.py:95} INFO - [2020-01-15 23:17:16,640] {pod_launcher.py:104} INFO - 2020-01-15 23:17:16 INFO::Started Release Models, simple.mode = 0
[2020-01-15 23:22:18,329] {models.py:1788} ERROR - ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1657, in _run_raw_task
result = task_copy.execute(context=context)
File "/app/dags/operators/gcp_container_operator.py", line 60, in execute
result = super(UpstreamGKEPodOperator, self).execute(context)
File "/usr/local/lib/python2.7/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 133, in execute
get_logs=self.get_logs)
File "/usr/local/lib/python2.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 90, in run_pod
return self._monitor_pod(pod, get_logs)
File "/usr/local/lib/python2.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 103, in _monitor_pod
for line in logs:
File "/usr/local/lib/python2.7/site-packages/urllib3/response.py", line 796, in __iter__
for chunk in self.stream(decode_content=True):
File "/usr/local/lib/python2.7/site-packages/urllib3/response.py", line 560, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/usr/local/lib/python2.7/site-packages/urllib3/response.py", line 781, in read_chunked
self._original_response.close()
File "/usr/local/lib/python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python2.7/site-packages/urllib3/response.py", line 443, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
[2020-01-15 23:22:18,338] {models.py:1817} INFO - All retries failed; marking task as FAILED
[2020-01-15 23:22:18,343] {logging_mixin.py:95} INFO - [2020-01-15 23:22:18,343] {log_email_backend.py:54} INFO -
Content-Type: multipart/mixed; boundary="===============1885000818552648692=="
MIME-Version: 1.0
Subject: Airflow alert: <TaskInstance: missioncontrol.missioncontrol_etl
2020-01-15T22:40:35.339783+00:00 [failed]>
From: telemetry-alerts@airflow.dev.mozaws.net
To: wlachance@mozilla.com, hwoo@mozilla.com
Date: Wed, 15 Jan 2020 23:22:18 +0000
--===============1885000818552648692==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Try 4 out of 3<br>Exception:<br>('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))<br>Log: <a href='http://localhost:8000/admin/airflow/log?dag_id=missioncontrol&task_id=missioncontrol_etl&execution_date=2020-01-15T22%3A40%3A35.339783%2B00%3A00'>Link</a><br>Host: cb4a5a73da07<br>Log file: /app/logs/missioncontrol/missioncontrol_etl/2020-01-15T22:40:35.339783+00:00.log<br>Mark success: <a href='http://localhost:8000/admin/airflow/success?task_id=missioncontrol_etl&dag_id=missioncontrol&execution_date=2020-01-15T22%3A40%3A35.339783%2B00%3A00&upstream=false&downstream=false'>Link</a><br>
--===============1885000818552648692==--
[2020-01-15 23:22:18,356] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl Traceback (most recent call last):
[2020-01-15 23:22:18,358] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/usr/local/bin/airflow", line 32, in <module>
[2020-01-15 23:22:18,359] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl args.func(args)
[2020-01-15 23:22:18,359] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/usr/local/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
[2020-01-15 23:22:18,361] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl return f(*args, **kwargs)
[2020-01-15 23:22:18,362] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 526, in run
[2020-01-15 23:22:18,363] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl _run(args, dag, ti)
[2020-01-15 23:22:18,364] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 445, in _run
[2020-01-15 23:22:18,364] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl pool=args.pool,
[2020-01-15 23:22:18,365] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 73, in wrapper
[2020-01-15 23:22:18,366] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl return func(*args, **kwargs)
[2020-01-15 23:22:18,366] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1657, in _run_raw_task
[2020-01-15 23:22:18,367] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl result = task_copy.execute(context=context)
[2020-01-15 23:22:18,367] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/app/dags/operators/gcp_container_operator.py", line 60, in execute
[2020-01-15 23:22:18,368] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl result = super(UpstreamGKEPodOperator, self).execute(context)
[2020-01-15 23:22:18,369] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/usr/local/lib/python2.7/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 133, in execute
[2020-01-15 23:22:18,369] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl get_logs=self.get_logs)
[2020-01-15 23:22:18,370] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/usr/local/lib/python2.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 90, in run_pod
[2020-01-15 23:22:18,370] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl return self._monitor_pod(pod, get_logs)
[2020-01-15 23:22:18,371] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/usr/local/lib/python2.7/site-packages/airflow/contrib/kubernetes/pod_launcher.py", line 103, in _monitor_pod
[2020-01-15 23:22:18,372] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl for line in logs:
[2020-01-15 23:22:18,372] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/usr/local/lib/python2.7/site-packages/urllib3/response.py", line 796, in __iter__
[2020-01-15 23:22:18,373] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl for chunk in self.stream(decode_content=True):
[2020-01-15 23:22:18,373] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/usr/local/lib/python2.7/site-packages/urllib3/response.py", line 560, in stream
[2020-01-15 23:22:18,374] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl for line in self.read_chunked(amt, decode_content=decode_content):
[2020-01-15 23:22:18,374] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/usr/local/lib/python2.7/site-packages/urllib3/response.py", line 781, in read_chunked
[2020-01-15 23:22:18,375] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl self._original_response.close()
[2020-01-15 23:22:18,376] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/usr/local/lib/python2.7/contextlib.py", line 35, in __exit__
[2020-01-15 23:22:18,376] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl self.gen.throw(type, value, traceback)
[2020-01-15 23:22:18,377] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl File "/usr/local/lib/python2.7/site-packages/urllib3/response.py", line 443, in _error_catcher
[2020-01-15 23:22:18,377] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl raise ProtocolError("Connection broken: %r" % e, e)
[2020-01-15 23:22:18,378] {base_task_runner.py:101} INFO - Job 188: Subtask missioncontrol_etl urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
[2020-01-15 23:22:22,052] {logging_mixin.py:95} INFO - [2020-01-15 23:22:22,051] {jobs.py:2527} INFO - Task exited with return code 1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment