Skip to content

Instantly share code, notes, and snippets.

@albinkjellin

albinkjellin/failed scan

Created August 24, 2021 12:52
Show Gist options
  • Save albinkjellin/c38335f71b4f404decd8e1e0350252c1 to your computer and use it in GitHub Desktop.
Save albinkjellin/c38335f71b4f404decd8e1e0350252c1 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
failed scan
| Starting new HTTPS connection (1): cloud.soda.io:443
| https://cloud.soda.io:443 "POST /api/command HTTP/1.1" 200 258
| Executing SQL query:
SELECT column_name, data_type, is_nullable FROM `shipping.INFORMATION_SCHEMA.COLUMNS` WHERE table_name = 'subscription';
| This service is instrumented using OpenTelemetry. OpenTelemetry could not be imported; please add opentelemetry-api and opentelemetry-instrumentation packages in order to get BigQuery Tracing data.
| Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
| Making request: POST https://oauth2.googleapis.com/token
| Starting new HTTPS connection (1): oauth2.googleapis.com:443
| https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
| Starting new HTTPS connection (1): bigquery.googleapis.com:443
| https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/curious-destiny-304908/jobs?prettyPrint=false HTTP/1.1" 200 None
| https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/curious-destiny-304908/queries/4593e702-9f52-45d4-8d0f-85c8bc6e7648?maxResults=0&location=US&prettyPrint=false HTTP/1.1" 200 None
| https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/curious-destiny-304908/jobs/4593e702-9f52-45d4-8d0f-85c8bc6e7648?location=US&prettyPrint=false HTTP/1.1" 200 None
| https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/curious-destiny-304908/queries/4593e702-9f52-45d4-8d0f-85c8bc6e7648?fields=jobReference%2CtotalRows%2CpageToken%2Crows&location=US&formatOptions.useInt64Timestamp=True&prettyPrint=false HTTP/1.1" 200 None
| SQL took 0:00:02.322194
| newtype (STRING)
| id (STRING)
| newfield2 (STRING)
| 3 columns:
| Query measurement: schema = [{'name': 'newtype', 'type': 'STRING', 'dataType': 'STRING', 'nullable': True, 'logicalType': 'text', 'semanticType': 'text'}, {'name': 'id', 'type': 'STRING', 'dataType': 'STRING', 'nullable': True, 'logicalType': 'text', 'semanticType': 'text'}, {'name': 'newfield2', 'type': 'STRING', 'dataType': 'STRING', 'nullable': True, 'logicalType': 'text', 'semanticType': 'text'}]
| Soda Cloud scan send measurements
| executing {'type': 'sodaSqlScanMeasurements', 'scanReference': '155a3e08-69fc-485f-9370-b1c45702fdbb', 'measurements': [{'metric': 'schema', 'value': [{'name': 'newtype', 'type': 'STRING', 'dataType': 'STRING', 'nullable': True, 'logicalType': 'text', 'semanticType': 'text'}, {'name': 'id', 'type': 'STRING', 'dataType': 'STRING', 'nullable': True, 'logicalType': 'text', 'semanticType': 'text'}, {'name': 'newfield2', 'type': 'STRING', 'dataType': 'STRING', 'nullable': True, 'logicalType': 'text', 'semanticType': 'text'}]}]}
| Starting new HTTPS connection (1): cloud.soda.io:443
| https://cloud.soda.io:443 "POST /api/command HTTP/1.1" 200 16
| Executing SQL query:
SELECT
COUNT(*),
COUNT(CASE WHEN NOT (newtype IS NULL) THEN 1 END),
COUNT(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN 1 END),
AVG(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN LENGTH(newtype) END),
MIN(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN LENGTH(newtype) END),
MAX(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN LENGTH(newtype) END),
COUNT(CASE WHEN NOT (id IS NULL) THEN 1 END),
COUNT(CASE WHEN NOT (id IS NULL) THEN 1 END),
AVG(CASE WHEN NOT (id IS NULL) THEN LENGTH(id) END),
MIN(CASE WHEN NOT (id IS NULL) THEN LENGTH(id) END),
MAX(CASE WHEN NOT (id IS NULL) THEN LENGTH(id) END),
COUNT(CASE WHEN NOT (newfield2 IS NULL) THEN 1 END),
COUNT(CASE WHEN NOT (newfield2 IS NULL) THEN 1 END),
AVG(CASE WHEN NOT (newfield2 IS NULL) THEN LENGTH(newfield2) END),
MIN(CASE WHEN NOT (newfield2 IS NULL) THEN LENGTH(newfield2) END),
MAX(CASE WHEN NOT (newfield2 IS NULL) THEN LENGTH(newfield2) END)
FROM `shipping.subscription`
| https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/curious-destiny-304908/jobs?prettyPrint=false HTTP/1.1" 200 None
--- Logging error ---
Traceback (most recent call last):
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 203, in _execute
self._query_job.result()
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/google/cloud/bigquery/job/query.py", line 1371, in result
do_get_result()
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func
return retry_target(
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/google/api_core/retry.py", line 190, in retry_target
return target()
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/google/cloud/bigquery/job/query.py", line 1361, in do_get_result
super(QueryJob, self).result(retry=retry, timeout=timeout)
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/google/cloud/bigquery/job/base.py", line 708, in result
return super(_AsyncJob, self).result(timeout=timeout, **kwargs)
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/google/api_core/future/polling.py", line 135, in result
raise self._exception
google.api_core.exceptions.BadRequest: 400 Unrecognized name: newfield2 at [13:24]
(job ID: 84d59a2d-32f9-4be6-bb68-57921eb7dda2)
-----Query Job SQL Follows-----
| . | . | . | . | . | . | . | . | . | . |
1:SELECT
2: COUNT(*),
3: COUNT(CASE WHEN NOT (newtype IS NULL) THEN 1 END),
4: COUNT(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN 1 END),
5: AVG(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN LENGTH(newtype) END),
6: MIN(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN LENGTH(newtype) END),
7: MAX(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN LENGTH(newtype) END),
8: COUNT(CASE WHEN NOT (id IS NULL) THEN 1 END),
9: COUNT(CASE WHEN NOT (id IS NULL) THEN 1 END),
10: AVG(CASE WHEN NOT (id IS NULL) THEN LENGTH(id) END),
11: MIN(CASE WHEN NOT (id IS NULL) THEN LENGTH(id) END),
12: MAX(CASE WHEN NOT (id IS NULL) THEN LENGTH(id) END),
13: COUNT(CASE WHEN NOT (newfield2 IS NULL) THEN 1 END),
14: COUNT(CASE WHEN NOT (newfield2 IS NULL) THEN 1 END),
15: AVG(CASE WHEN NOT (newfield2 IS NULL) THEN LENGTH(newfield2) END),
16: MIN(CASE WHEN NOT (newfield2 IS NULL) THEN LENGTH(newfield2) END),
17: MAX(CASE WHEN NOT (newfield2 IS NULL) THEN LENGTH(newfield2) END)
18:FROM `shipping.subscription`
| . | . | . | . | . | . | . | . | . | . |
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/sodasql/scan/scan.py", line 297, in _query_aggregations
query_result_tuple = self.warehouse.sql_fetchone(sql)
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/sodasql/scan/warehouse.py", line 27, in sql_fetchone
return sql_fetchone(self.connection, sql)
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/sodasql/scan/db.py", line 20, in sql_fetchone
return sql_fetchone_description(connection, sql)[0]
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/sodasql/scan/db.py", line 33, in sql_fetchone_description
cursor.execute(sql)
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/_helpers.py", line 489, in with_closed_check
return method(self, *args, **kwargs)
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 166, in execute
self._execute(
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 205, in _execute
raise exceptions.DatabaseError(exc)
google.cloud.bigquery.dbapi.exceptions.DatabaseError: 400 Unrecognized name: newfield2 at [13:24]
(job ID: 84d59a2d-32f9-4be6-bb68-57921eb7dda2)
-----Query Job SQL Follows-----
| . | . | . | . | . | . | . | . | . | . |
1:SELECT
2: COUNT(*),
3: COUNT(CASE WHEN NOT (newtype IS NULL) THEN 1 END),
4: COUNT(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN 1 END),
5: AVG(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN LENGTH(newtype) END),
6: MIN(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN LENGTH(newtype) END),
7: MAX(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN LENGTH(newtype) END),
8: COUNT(CASE WHEN NOT (id IS NULL) THEN 1 END),
9: COUNT(CASE WHEN NOT (id IS NULL) THEN 1 END),
10: AVG(CASE WHEN NOT (id IS NULL) THEN LENGTH(id) END),
11: MIN(CASE WHEN NOT (id IS NULL) THEN LENGTH(id) END),
12: MAX(CASE WHEN NOT (id IS NULL) THEN LENGTH(id) END),
13: COUNT(CASE WHEN NOT (newfield2 IS NULL) THEN 1 END),
14: COUNT(CASE WHEN NOT (newfield2 IS NULL) THEN 1 END),
15: AVG(CASE WHEN NOT (newfield2 IS NULL) THEN LENGTH(newfield2) END),
16: MIN(CASE WHEN NOT (newfield2 IS NULL) THEN LENGTH(newfield2) END),
17: MAX(CASE WHEN NOT (newfield2 IS NULL) THEN LENGTH(newfield2) END)
18:FROM `shipping.subscription`
| . | . | . | . | . | . | . | . | . | . |
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/homebrew/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/logging/__init__.py", line 1083, in emit
msg = self.format(record)
File "/opt/homebrew/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/logging/__init__.py", line 927, in format
return fmt.format(record)
File "/opt/homebrew/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/logging/__init__.py", line 663, in format
record.message = record.getMessage()
File "/opt/homebrew/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/logging/__init__.py", line 367, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "/Users/albin/dev/py/env/2.1.0b14/bin/soda", line 8, in <module>
sys.exit(main())
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/sodasql/cli/cli.py", line 368, in scan
scan_result: ScanResult = scan.execute()
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/sodasql/scan/scan.py", line 87, in execute
self._query_aggregations()
File "/Users/albin/dev/py/env/2.1.0b14/lib/python3.9/site-packages/sodasql/scan/scan.py", line 339, in _query_aggregations
logging.debug(f'Exception during aggregation query', e)
Message: 'Exception during aggregation query'
Arguments: (DatabaseError(BadRequest('Unrecognized name: newfield2 at [13:24]')),)
| Executing SQL query:
WITH group_by_value AS (
SELECT
newtype AS value,
COUNT(*) AS frequency
FROM `shipping.subscription`
WHERE NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event'))
GROUP BY newtype
)
SELECT value
FROM group_by_value
ORDER BY value ASC
LIMIT 5
| https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/curious-destiny-304908/jobs?prettyPrint=false HTTP/1.1" 200 None
| https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/curious-destiny-304908/queries/197c56d7-8666-4e1c-b27b-da1cc78432ee?maxResults=0&location=US&prettyPrint=false HTTP/1.1" 200 None
| https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/curious-destiny-304908/jobs/197c56d7-8666-4e1c-b27b-da1cc78432ee?location=US&prettyPrint=false HTTP/1.1" 200 None
| https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/curious-destiny-304908/queries/197c56d7-8666-4e1c-b27b-da1cc78432ee?fields=jobReference%2CtotalRows%2CpageToken%2Crows&location=US&formatOptions.useInt64Timestamp=True&prettyPrint=false HTTP/1.1" 200 None
| SQL took 0:00:01.423343
| Query measurement: mins(newtype) = []
| Executing SQL query:
WITH group_by_value AS (
SELECT
newtype AS value,
COUNT(*) AS frequency
FROM `shipping.subscription`
WHERE NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event'))
GROUP BY newtype
)
SELECT value
FROM group_by_value
ORDER BY value DESC
LIMIT 5
| https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/curious-destiny-304908/jobs?prettyPrint=false HTTP/1.1" 200 None
| https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/curious-destiny-304908/queries/8a31c561-5076-48ec-898f-85e52db4f290?maxResults=0&location=US&prettyPrint=false HTTP/1.1" 200 None
| https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/curious-destiny-304908/jobs/8a31c561-5076-48ec-898f-85e52db4f290?location=US&prettyPrint=false HTTP/1.1" 200 None
| https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/curious-destiny-304908/queries/8a31c561-5076-48ec-898f-85e52db4f290?fields=jobReference%2CtotalRows%2CpageToken%2Crows&location=US&formatOptions.useInt64Timestamp=True&prettyPrint=false HTTP/1.1" 200 None
| SQL took 0:00:01.548976
| Query measurement: maxs(newtype) = []
| Executing SQL query:
WITH group_by_value AS (
SELECT
newtype AS value,
COUNT(*) AS frequency
FROM `shipping.subscription`
WHERE NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event'))
GROUP BY newtype
)
SELECT value, frequency
FROM group_by_value
ORDER BY frequency DESC
LIMIT 5
| https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/curious-destiny-304908/jobs?prettyPrint=false HTTP/1.1" 200 None
| https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/curious-destiny-304908/queries/f970e7c7-0e0e-46ca-9a50-59fb9eefc492?maxResults=0&location=US&prettyPrint=false HTTP/1.1" 200 None
| https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/curious-destiny-304908/jobs/f970e7c7-0e0e-46ca-9a50-59fb9eefc492?location=US&prettyPrint=false HTTP/1.1" 200 None
| https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/curious-destiny-304908/queries/f970e7c7-0e0e-46ca-9a50-59fb9eefc492?fields=jobReference%2CtotalRows%2CpageToken%2Crows&location=US&formatOptions.useInt64Timestamp=True&prettyPrint=false HTTP/1.1" 200 None
| SQL took 0:00:01.300698
| Query measurement: frequent_values(newtype) = []
| Soda Cloud scan send measurements
| executing {'type': 'sodaSqlScanMeasurements', 'scanReference': '155a3e08-69fc-485f-9370-b1c45702fdbb', 'measurements': [{'metric': 'mins', 'value': [], 'columnName': 'newtype'}, {'metric': 'maxs', 'value': [], 'columnName': 'newtype'}, {'metric': 'frequent_values', 'value': [], 'columnName': 'newtype'}]}
| Starting new HTTPS connection (1): cloud.soda.io:443
| https://cloud.soda.io:443 "POST /api/command HTTP/1.1" 200 16
| Fetching custom metrics with scanReference: 155a3e08-69fc-485f-9370-b1c45702fdbb
| Starting new HTTPS connection (1): cloud.soda.io:443
| https://cloud.soda.io:443 "POST /api/query HTTP/1.1" 200 2
| Test error for "row_count > 0": name 'row_count' is not defined
| Soda Cloud scan send test results
| executing {'type': 'sodaSqlScanTestResults', 'scanReference': '155a3e08-69fc-485f-9370-b1c45702fdbb', 'testResults': [{'id': '{"expression":"row_count > 0"}', 'title': 'test(row_count > 0)', 'description': 'test(row_count > 0)', 'expression': 'row_count > 0', 'error': "name 'row_count' is not defined"}]}
| Starting new HTTPS connection (1): cloud.soda.io:443
| https://cloud.soda.io:443 "POST /api/command HTTP/1.1" 200 16
| Test error for "values_percentage > 50": name 'values_percentage' is not defined
| Soda Cloud scan send test results
| executing {'type': 'sodaSqlScanTestResults', 'scanReference': '155a3e08-69fc-485f-9370-b1c45702fdbb', 'testResults': [{'id': '{"column":"newtype","expression":"values_percentage > 50"}', 'title': 'column(newtype) test(values_percentage > 50)', 'description': 'column(newtype) test(values_percentage > 50)', 'expression': 'values_percentage > 50', 'columnName': 'newtype', 'error': "name 'values_percentage' is not defined"}]}
| Starting new HTTPS connection (1): cloud.soda.io:443
| https://cloud.soda.io:443 "POST /api/command HTTP/1.1" 200 16
| Executed 4 queries in 0:00:09.007487
| Soda Cloud scan end with errors
| executing {'type': 'sodaSqlScanEnd', 'scanReference': '155a3e08-69fc-485f-9370-b1c45702fdbb', 'errors': [{'type': 'error', 'message': 'Exception during aggregation query', 'exception': "400 Unrecognized name: newfield2 at [13:24]\n\n(job ID: 84d59a2d-32f9-4be6-bb68-57921eb7dda2)\n\n -----Query Job SQL Follows----- \n\n | . | . | . | . | . | . | . | . | . | . |\n 1:SELECT \n 2: COUNT(*),\n 3: COUNT(CASE WHEN NOT (newtype IS NULL) THEN 1 END),\n 4: COUNT(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN 1 END),\n 5: AVG(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN LENGTH(newtype) END),\n 6: MIN(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN LENGTH(newtype) END),\n 7: MAX(CASE WHEN NOT (newtype IS NULL) AND (newtype IN ('regular','gift','event')) THEN LENGTH(newtype) END),\n 8: COUNT(CASE WHEN NOT (id IS NULL) THEN 1 END),\n 9: COUNT(CASE WHEN NOT (id IS NULL) THEN 1 END),\n 10: AVG(CASE WHEN NOT (id IS NULL) THEN LENGTH(id) END),\n 11: MIN(CASE WHEN NOT (id IS NULL) THEN LENGTH(id) END),\n 12: MAX(CASE WHEN NOT (id IS NULL) THEN LENGTH(id) END),\n 13: COUNT(CASE WHEN NOT (newfield2 IS NULL) THEN 1 END),\n 14: COUNT(CASE WHEN NOT (newfield2 IS NULL) THEN 1 END),\n 15: AVG(CASE WHEN NOT (newfield2 IS NULL) THEN LENGTH(newfield2) END),\n 16: MIN(CASE WHEN NOT (newfield2 IS NULL) THEN LENGTH(newfield2) END),\n 17: MAX(CASE WHEN NOT (newfield2 IS NULL) THEN LENGTH(newfield2) END) \n 18:FROM `shipping.subscription`\n | . | . | . | . | . | . | . | . | . | . |"}, {'type': 'test_execution_error', 'message': 'Test "row_count > 0" failed', 'exception': "name 'row_count' is not defined"}, {'type': 'test_execution_error', 'message': 'Test "values_percentage > 50" failed', 'exception': "name 'values_percentage' is not defined"}]}
| Starting new HTTPS connection (1): cloud.soda.io:443
| https://cloud.soda.io:443 "POST /api/command HTTP/1.1" 200 2
| Scan summary ------
| 4 measurements computed
| 2 tests executed
| 2 of 2 tests failed:
| Test test(row_count > 0) failed with measurements null
| Test column(newtype) test(values_percentage > 50) failed with measurements null
| Errors occurred!
| [error] Exception during aggregation query
| [test_execution_error] Test "row_count > 0" failed
| [test_execution_error] Test "values_percentage > 50" failed
| Exiting with code 1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment