This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
23:12:06.184 | INFO | prefect.engine - Created flow run 'perfect-swan' for flow 'Motive Pipeline' | |
-- Motive ETL: Begin -- | |
Querying the Motive API.. | |
23:12:06.439 | INFO | Flow run 'perfect-swan' - Created task run 'get_all_vehicle_info-0' for task 'get_all_vehicle_info' | |
23:12:06.440 | INFO | Flow run 'perfect-swan' - Executing 'get_all_vehicle_info-0' immediately... | |
23:12:07.484 | INFO | Task run 'get_all_vehicle_info-0' - Finished in state Completed() | |
23:12:07.518 | INFO | Flow run 'perfect-swan' - Created task run 'save_to_json-0' for task 'save_to_json' | |
23:12:07.518 | INFO | Flow run 'perfect-swan' - Executing 'save_to_json-0' immediately... | |
Data saved to MOTIVE_GETALLVEHICLEINFO_RAW_RESP__2023-03-30_23-12-06-404057.json | |
23:12:07.606 | INFO | Task run 'save_to_json-0' - Finished in state Completed() | |
23:12:07.637 | INFO | Flow run 'perfect-swan' - Created task run 'flatten_vehicle_info-0' for task 'flatten_vehicle_info' | |
23:12:07.638 | INFO | Flow run 'perfect-swan' - Executing 'flatten_vehicle_info-0' immediately... | |
23:12:07.745 | INFO | Task run 'flatten_vehicle_info-0' - Finished in state Completed() | |
23:12:07.781 | INFO | Flow run 'perfect-swan' - Created task run 'convert_ordered_dict_list_to_tuple_list-0' for task 'convert_ordered_dict_list_to_tuple_list' | |
23:12:07.781 | INFO | Flow run 'perfect-swan' - Executing 'convert_ordered_dict_list_to_tuple_list-0' immediately... | |
23:12:07.861 | INFO | Task run 'convert_ordered_dict_list_to_tuple_list-0' - Finished in state Completed() | |
23:12:07.892 | INFO | Flow run 'perfect-swan' - Created task run 'save_to_csv-0' for task 'save_to_csv' | |
23:12:07.893 | INFO | Flow run 'perfect-swan' - Executing 'save_to_csv-0' immediately... | |
Wrote 13 rows to MOTIVE_IMPORTS__2023-03-30_23-12-06-404057.csv | |
23:12:07.970 | INFO | Task run 'save_to_csv-0' - Finished in state Completed() | |
23:12:07.996 | INFO | Flow run 'perfect-swan' - Created task run 'create_az_blob_client-0' for task 'create_az_blob_client' | |
23:12:07.996 | INFO | Flow run 'perfect-swan' - Executing 'create_az_blob_client-0' immediately... | |
Connecting to Azure Blob Storage: "addsbi" | |
23:12:08.076 | INFO | Task run 'create_az_blob_client-0' - Finished in state Completed() | |
23:12:08.131 | INFO | Flow run 'perfect-swan' - Created task run 'get_az_container-0' for task 'get_az_container' | |
23:12:08.134 | INFO | Flow run 'perfect-swan' - Executing 'get_az_container-0' immediately... | |
23:12:19.055 | INFO | Task run 'get_az_container-0' - Finished in state Completed() | |
23:12:19.103 | INFO | Flow run 'perfect-swan' - Created task run 'create_azure_sql_connection-0' for task 'create_azure_sql_connection' | |
23:12:19.104 | INFO | Flow run 'perfect-swan' - Executing 'create_azure_sql_connection-0' immediately... | |
Connecting to Azure SQL DB. | |
23:12:19.579 | INFO | Task run 'create_azure_sql_connection-0' - Finished in state Completed() | |
23:12:19.617 | INFO | Flow run 'perfect-swan' - Created task run 'write_to_az_blob-0' for task 'write_to_az_blob' | |
23:12:19.619 | INFO | Flow run 'perfect-swan' - Executing 'write_to_az_blob-0' immediately... | |
Uploading "MOTIVE_IMPORTS__2023-03-30_23-12-06-404057.csv" to "MOTIVE_Raw/2023/3" in Azure Blob Storage. | |
23:12:19.888 | INFO | Task run 'write_to_az_blob-0' - Finished in state Completed() | |
23:12:19.916 | INFO | Flow run 'perfect-swan' - Created task run 'write_to_az_blob-1' for task 'write_to_az_blob' | |
23:12:19.917 | INFO | Flow run 'perfect-swan' - Executing 'write_to_az_blob-1' immediately... | |
Uploading "MOTIVE_GETALLVEHICLEINFO_RAW_RESP__2023-03-30_23-12-06-404057.json" to "Motive_Full_JSON/2023/3" in Azure Blob Storage. | |
23:12:20.117 | INFO | Task run 'write_to_az_blob-1' - Finished in state Completed() | |
23:12:20.121 | ERROR | Flow run 'perfect-swan' - Crash detected! Execution was interrupted by an unexpected exception: Traceback (most recent call last): | |
File "/usr/lib/python3.9/contextlib.py", line 193, in __aexit__ | |
await self.gen.athrow(typ, value, traceback) | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/task_runners.py", line 166, in start | |
yield self | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/engine.py", line 393, in begin_flow_run | |
terminal_or_paused_state = await orchestrate_flow_run( | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/engine.py", line 673, in orchestrate_flow_run | |
result = await flow_call.aresult() | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 181, in aresult | |
return await asyncio.wrap_future(self.future) | |
asyncio.exceptions.CancelledError | |
During handling of the above exception, another exception occurred: | |
pyodbc.ProgrammingError: No results. Previous SQL was not a query. | |
^CTraceback (most recent call last): | |
File "<string>", line 1, in <module> | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/flows.py", line 468, in __call__ | |
return enter_flow_run_engine_from_flow_call( | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/engine.py", line 182, in enter_flow_run_engine_from_flow_call | |
retval = from_sync.wait_for_call_in_loop_thread( | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/_internal/concurrency/api.py", line 137, in wait_for_call_in_loop_thread | |
return call.result() | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 173, in result | |
return self.future.result(timeout=timeout) | |
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result | |
return self.__get_result() | |
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result | |
raise self._exception | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 218, in _run_async | |
result = await coro | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/client/utilities.py", line 40, in with_injected_client | |
return await fn(*args, **kwargs) | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/engine.py", line 252, in create_then_begin_flow_run | |
state = await begin_flow_run( | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/engine.py", line 393, in begin_flow_run | |
terminal_or_paused_state = await orchestrate_flow_run( | |
File "/usr/lib/python3.9/contextlib.py", line 651, in __aexit__ | |
raise exc_details[1] | |
File "/usr/lib/python3.9/contextlib.py", line 193, in __aexit__ | |
await self.gen.athrow(typ, value, traceback) | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/engine.py", line 1719, in report_flow_run_crashes | |
raise exc from None | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/engine.py", line 1692, in report_flow_run_crashes | |
yield | |
File "/usr/lib/python3.9/contextlib.py", line 634, in __aexit__ | |
cb_suppress = await cb(*exc_details) | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 662, in __aexit__ | |
raise exceptions[0] | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/engine.py", line 1196, in create_task_run_then_submit | |
task_run = await create_task_run( | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/engine.py", line 1231, in create_task_run | |
task_inputs = {k: await collect_task_run_inputs(v) for k, v in parameters.items()} | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/engine.py", line 1231, in <dictcomp> | |
task_inputs = {k: await collect_task_run_inputs(v) for k, v in parameters.items()} | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/engine.py", line 1093, in collect_task_run_inputs | |
visit_collection( | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/utilities/collections.py", line 326, in visit_collection | |
items = [visit_nested(o) for o in expr] | |
File "/home/zmassia/.cache/pypoetry/virtualenvs/adds-bi-AfT2lUg4-py3.9/lib/python3.9/site-packages/prefect/utilities/collections.py", line 326, in <listcomp> | |
items = [visit_nested(o) for o in expr] | |
pyodbc.ProgrammingError: No results. Previous SQL was not a query. | |
zmassia@adds-bi-db:~/dev/adds-bi$ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import pyodbc | |
from prefect import task, flow | |
@task | |
def connect_to_db(): | |
dsn = 'driver=ODBC Driver 18 for SQL Server;uid=XXXXXX;pwd=XXXXXX!;database=XXXXXX' | |
srv = 'tcp:XXXXXX.database.windows.net,1433' | |
return pyodbc.connect( | |
f'{dsn};server={srv};encrypt=yes;trustservercertificate=yes;trustedconnection=yes;', | |
autocommit=False | |
) | |
@task | |
def query_db(cur): | |
results = cur.execute('SELECT * FROM dbo.TABLE').fetchall() | |
return results | |
@flow | |
def main(): | |
db = connect_to_db() | |
xs = query_db(db.cursor()) | |
print(xs) | |
if __name__ == '__main__': | |
main() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
23:28:49.964 | INFO | prefect.engine - Created flow run 'elastic-catfish' for flow 'main' | |
23:28:50.114 | INFO | Flow run 'elastic-catfish' - Created task run 'connect_to_db-0' for task 'connect_to_db' | |
23:28:50.115 | INFO | Flow run 'elastic-catfish' - Executing 'connect_to_db-0' immediately... | |
23:28:50.643 | INFO | Task run 'connect_to_db-0' - Finished in state Completed() | |
23:28:50.648 | ERROR | Flow run 'elastic-catfish' - Crash detected! Execution was interrupted by an unexpected exception: pyodbc.ProgrammingError: No results. Previous SQL was not a query. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment