Skip to content

Instantly share code, notes, and snippets.

@BagelOrb

BagelOrb/debug-internal.log Secret

Last active July 4, 2023 09:26
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save BagelOrb/916e2efc1a527cf397fd453a18cf7d2c to your computer and use it in GitHub Desktop.
Save BagelOrb/916e2efc1a527cf397fd453a18cf7d2c to your computer and use it in GitHub Desktop.
Download ZIP
debug-internal.log
Raw
debug-internal.log
This file has been truncated, but you can view the full file.
2023-07-04 11:05:41,717 INFO StreamThr :2443566 [internal.py:wandb_internal():86] W&B internal server running at pid: 2443566, started at: 2023-07-04 11:05:41.717287
2023-07-04 11:05:41,718 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status
2023-07-04 11:05:41,722 INFO WriterThread:2443566 [datastore.py:open_for_write():85] open: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/run-w53a3qn5.wandb
2023-07-04 11:05:41,722 DEBUG SenderThread:2443566 [sender.py:send():369] send: header
2023-07-04 11:05:41,724 DEBUG SenderThread:2443566 [sender.py:send():369] send: run
2023-07-04 11:05:42,065 INFO SenderThread:2443566 [dir_watcher.py:__init__():211] watching files in: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files
2023-07-04 11:05:42,065 INFO SenderThread:2443566 [sender.py:_start_run_threads():1100] run started: w53a3qn5 with start time 1688461541.71774
2023-07-04 11:05:42,065 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:42,065 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:42,067 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: check_version
2023-07-04 11:05:42,067 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: check_version
2023-07-04 11:05:42,207 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: run_start
2023-07-04 11:05:42,211 DEBUG HandlerThread:2443566 [system_info.py:__init__():31] System info init
2023-07-04 11:05:42,211 DEBUG HandlerThread:2443566 [system_info.py:__init__():46] System info init done
2023-07-04 11:05:42,211 INFO HandlerThread:2443566 [system_monitor.py:start():181] Starting system monitor
2023-07-04 11:05:42,211 INFO SystemMonitor:2443566 [system_monitor.py:_start():145] Starting system asset monitoring threads
2023-07-04 11:05:42,212 INFO HandlerThread:2443566 [system_monitor.py:probe():201] Collecting system info
2023-07-04 11:05:42,212 INFO SystemMonitor:2443566 [interfaces.py:start():190] Started cpu monitoring
2023-07-04 11:05:42,213 INFO SystemMonitor:2443566 [interfaces.py:start():190] Started disk monitoring
2023-07-04 11:05:42,213 INFO SystemMonitor:2443566 [interfaces.py:start():190] Started gpu monitoring
2023-07-04 11:05:42,213 INFO SystemMonitor:2443566 [interfaces.py:start():190] Started memory monitoring
2023-07-04 11:05:42,213 INFO SystemMonitor:2443566 [interfaces.py:start():190] Started network monitoring
2023-07-04 11:05:42,261 DEBUG HandlerThread:2443566 [system_info.py:probe():195] Probing system
2023-07-04 11:05:42,262 DEBUG HandlerThread:2443566 [system_info.py:_probe_git():180] Probing git
2023-07-04 11:05:42,264 DEBUG HandlerThread:2443566 [system_info.py:_probe_git():188] Probing git done
2023-07-04 11:05:42,264 DEBUG HandlerThread:2443566 [system_info.py:probe():240] Probing system done
2023-07-04 11:05:42,264 DEBUG HandlerThread:2443566 [system_monitor.py:probe():210] {'os': 'Linux-5.15.0-75-generic-x86_64-with-glibc2.31', 'python': '3.10.11', 'heartbeatAt': '2023-07-04T09:05:42.261161', 'startedAt': '2023-07-04T09:05:41.713813', 'docker': None, 'cuda': None, 'args': None, 'state': 'running', 'program': '<python with no main file>', 'git': {'remote': 'git@bitbucket.org:robin-radar-systems/deeplearning.git', 'commit': '623d273eff79b19ecfd78e59872bdfbccd5c573b'}, 'email': 'tim.kuipers@robinradar.com', 'root': '/home/tim.kuipers/dev/deeplearning', 'host': 'WS-Tim-kuipers', 'username': 'tim.kuipers', 'executable': '/home/tim.kuipers/mambaforge-pypy3/envs/torchenv/bin/python3.10', 'cpu_count': 24, 'cpu_count_logical': 32, 'cpu_freq': {'current': 3155.8483125000002, 'min': 800.0, 'max': 4937.5}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 5500.0}, {'current': 3000.0, 'min': 800.0, 'max': 5500.0}, {'current': 3000.0, 'min': 800.0, 'max': 5500.0}, {'current': 3000.0, 'min': 800.0, 'max': 5500.0}, {'current': 3000.0, 'min': 800.0, 'max': 5500.0}, {'current': 3000.0, 'min': 800.0, 'max': 5500.0}, {'current': 5499.975, 'min': 800.0, 'max': 5500.0}, {'current': 3000.0, 'min': 800.0, 'max': 5500.0}, {'current': 3000.0, 'min': 800.0, 'max': 5800.0}, {'current': 3000.0, 'min': 800.0, 'max': 5800.0}, {'current': 5607.009, 'min': 800.0, 'max': 5800.0}, {'current': 3000.0, 'min': 800.0, 'max': 5800.0}, {'current': 3000.0, 'min': 800.0, 'max': 5500.0}, {'current': 3000.0, 'min': 800.0, 'max': 5500.0}, {'current': 3000.0, 'min': 800.0, 'max': 5500.0}, {'current': 3000.0, 'min': 800.0, 'max': 5500.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}, {'current': 1351.494, 'min': 800.0, 'max': 4300.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}, {'current': 3000.0, 'min': 800.0, 'max': 4300.0}], 'disk': {'total': 3663.5948944091797, 'used': 108.34808731079102}, 'gpu': 'NVIDIA GeForce RTX 4090', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4090', 'memory_total': 25757220864}], 'memory': {'total': 62.56899642944336}}
2023-07-04 11:05:42,264 INFO HandlerThread:2443566 [system_monitor.py:probe():211] Finished collecting system info
2023-07-04 11:05:42,264 INFO HandlerThread:2443566 [system_monitor.py:probe():214] Publishing system info
2023-07-04 11:05:42,264 DEBUG HandlerThread:2443566 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment
2023-07-04 11:05:42,264 DEBUG HandlerThread:2443566 [system_info.py:_save_pip():67] Saving pip packages done
2023-07-04 11:05:42,264 DEBUG HandlerThread:2443566 [system_info.py:_save_conda():74] Saving list of conda packages installed into the current environment
2023-07-04 11:05:43,065 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/requirements.txt
2023-07-04 11:05:43,065 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:05:43,066 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/conda-environment.yaml
2023-07-04 11:05:43,982 DEBUG HandlerThread:2443566 [system_info.py:_save_conda():86] Saving conda packages done
2023-07-04 11:05:43,982 INFO HandlerThread:2443566 [system_monitor.py:probe():216] Finished publishing system info
2023-07-04 11:05:43,983 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:05:43,984 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-metadata.json with policy now
2023-07-04 11:05:43,989 DEBUG SenderThread:2443566 [sender.py:send():369] send: telemetry
2023-07-04 11:05:43,989 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: stop_status
2023-07-04 11:05:43,989 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: stop_status
2023-07-04 11:05:44,066 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/conda-environment.yaml
2023-07-04 11:05:44,066 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-metadata.json
2023-07-04 11:05:44,066 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/output.log
2023-07-04 11:05:44,453 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/p61qn2mt-wandb-metadata.json
2023-07-04 11:05:45,321 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,321 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:45,321 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:45,321 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,322 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,322 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:45,322 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:45,322 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:45,323 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:45,323 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:45,323 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:45,323 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:45,323 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:05:45,486 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:45,486 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:05:45,756 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,757 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,757 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,757 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,769 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,769 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,769 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,917 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:45,921 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,921 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,921 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,922 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,933 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,934 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:45,934 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,066 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:05:46,066 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/output.log
2023-07-04 11:05:46,066 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_3_4c6c1ab939cda8ea4a72.table.json
2023-07-04 11:05:46,066 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_10_83a53314926c797ba154.table.json
2023-07-04 11:05:46,067 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media
2023-07-04 11:05:46,067 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:05:46,075 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:46,078 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,079 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,079 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,079 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,091 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,092 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,092 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,240 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:46,245 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,245 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,245 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,245 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,257 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,258 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,258 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,396 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:46,401 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,401 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,401 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,402 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,413 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,414 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,414 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,513 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/k700nyek-media/table/batch confusion matrix_table_3_4c6c1ab939cda8ea4a72.table.json
2023-07-04 11:05:46,550 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/y67h9ctm-media/table/batch confusion matrix_table_10_83a53314926c797ba154.table.json
2023-07-04 11:05:46,564 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:46,568 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,568 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,568 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,569 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,580 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,581 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,581 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,616 INFO wandb-upload_2:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpbspchntf
2023-07-04 11:05:46,728 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:46,732 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,732 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,732 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,732 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,744 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,745 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,745 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,895 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:46,898 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,899 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,899 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,899 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,911 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,912 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:46,912 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,063 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:47,066 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,067 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,067 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,067 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,067 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_31_60d88c3a48deec0f756e.table.json
2023-07-04 11:05:47,067 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_52_e740c4cc03ea03bcbd2c.table.json
2023-07-04 11:05:47,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_45_543f8336e4d5ebce0e61.table.json
2023-07-04 11:05:47,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_38_ac8c4a0dfc8df0779199.table.json
2023-07-04 11:05:47,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_17_a7d0d7e6a80ecc2ddc2f.table.json
2023-07-04 11:05:47,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_59_83a53314926c797ba154.table.json
2023-07-04 11:05:47,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_24_8fbc5f39b29508567cad.table.json
2023-07-04 11:05:47,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:05:47,079 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,079 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,079 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,229 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:47,234 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,234 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,234 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,235 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,245 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,246 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,246 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,384 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:47,388 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,389 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,389 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,389 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,400 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,401 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,401 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,434 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk1OTg4', 'digest': '0bc79c2ee78b0b89325bf981d45bdc7d', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': None}, 'version': 'latest'}
2023-07-04 11:05:47,435 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:05:47,435 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:05:47,435 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_3_4c6c1ab939cda8ea4a72.table.json with policy now
2023-07-04 11:05:47,435 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:47,435 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:47,435 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:47,435 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:47,435 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:47,435 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:47,436 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:47,436 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:47,436 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:47,436 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:47,436 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:47,436 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:47,436 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:47,436 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:47,436 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:47,436 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:47,436 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:47,436 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:47,436 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:47,436 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:47,436 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:47,436 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:05:47,436 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:05:47,555 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:47,595 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,596 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,596 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,596 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,596 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,596 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,596 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,727 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:47,732 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,733 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,733 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,733 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,745 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,746 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,746 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,852 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/9nw4z09u-media/table/batch confusion matrix_table_31_60d88c3a48deec0f756e.table.json
2023-07-04 11:05:47,875 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/36jbz6be-media/table/batch confusion matrix_table_52_e740c4cc03ea03bcbd2c.table.json
2023-07-04 11:05:47,879 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/plux6ljc-media/table/batch confusion matrix_table_59_83a53314926c797ba154.table.json
2023-07-04 11:05:47,879 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/he54ljb7-media/table/batch confusion matrix_table_17_a7d0d7e6a80ecc2ddc2f.table.json
2023-07-04 11:05:47,889 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/a92q21sk-media/table/batch confusion matrix_table_45_543f8336e4d5ebce0e61.table.json
2023-07-04 11:05:47,895 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/m4p86ffv-media/table/batch confusion matrix_table_38_ac8c4a0dfc8df0779199.table.json
2023-07-04 11:05:47,910 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:47,913 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,914 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,914 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,914 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,921 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/4rqfofvd-media/table/batch confusion matrix_table_24_8fbc5f39b29508567cad.table.json
2023-07-04 11:05:47,926 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,927 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,927 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:47,957 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/g0uuhvan-media/table/batch confusion matrix_table_3_4c6c1ab939cda8ea4a72.table.json
2023-07-04 11:05:48,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:05:48,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_94_5e8599edc26a28b0bf56.table.json
2023-07-04 11:05:48,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_66_d4d30958e51757b86519.table.json
2023-07-04 11:05:48,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_87_542681021078313e717c.table.json
2023-07-04 11:05:48,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_80_e6cdd73690272b208c21.table.json
2023-07-04 11:05:48,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_73_a7873205fa75b58e62cb.table.json
2023-07-04 11:05:48,069 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:05:48,089 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:48,092 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,093 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,093 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,093 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,105 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,105 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,105 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,264 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:48,268 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,268 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,268 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,269 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,280 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,281 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,281 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,421 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:48,425 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,425 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,425 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,425 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,437 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,438 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,438 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,480 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/bmsceugi-media/table/batch confusion matrix_table_94_5e8599edc26a28b0bf56.table.json
2023-07-04 11:05:48,485 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ylcl2287-media/table/batch confusion matrix_table_73_a7873205fa75b58e62cb.table.json
2023-07-04 11:05:48,505 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/qoungw8w-media/table/batch confusion matrix_table_80_e6cdd73690272b208c21.table.json
2023-07-04 11:05:48,537 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/59k3f9ok-media/table/batch confusion matrix_table_87_542681021078313e717c.table.json
2023-07-04 11:05:48,540 INFO wandb-upload_0:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmptzktl02w
2023-07-04 11:05:48,544 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/vmg9s4sb-media/table/batch confusion matrix_table_66_d4d30958e51757b86519.table.json
2023-07-04 11:05:48,595 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:48,599 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,599 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,600 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,600 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,612 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,612 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,612 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,769 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:48,772 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,773 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,773 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,773 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,785 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,785 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,785 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,947 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:48,951 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,951 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,951 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,951 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,963 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,964 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:48,964 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_115_30f70383ce856197d41e.table.json
2023-07-04 11:05:49,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_122_eeeae82c9a9a95bb6adc.table.json
2023-07-04 11:05:49,068 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_101_5dcc195a388d9733922f.table.json
2023-07-04 11:05:49,069 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_108_542681021078313e717c.table.json
2023-07-04 11:05:49,069 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_129_de884991beb9c4b33208.table.json
2023-07-04 11:05:49,069 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_136_1e727c5a957bfc8e9c63.table.json
2023-07-04 11:05:49,069 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:05:49,110 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:49,117 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,117 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,117 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,117 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,130 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,130 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,130 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,275 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:49,279 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,279 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,279 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,279 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,291 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,292 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,292 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,339 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MDAz', 'digest': 'cff6c7d6e76288f9670860e1252bd336', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk1OTg4', 'versionIndex': 0}}, 'version': 'latest'}
2023-07-04 11:05:49,339 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:05:49,339 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_10_83a53314926c797ba154.table.json with policy now
2023-07-04 11:05:49,340 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:49,340 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:49,340 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:49,340 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:49,340 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:49,342 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:49,343 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:49,343 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:49,344 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:49,345 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:49,345 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:49,346 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:49,346 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:49,346 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:49,346 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:49,346 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:49,346 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:49,346 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:49,346 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:49,346 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:49,346 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:49,346 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:05:49,346 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:05:49,440 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:49,443 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,444 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,444 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,444 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,467 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,468 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,468 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,626 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:49,629 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,630 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,630 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,631 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:49,737 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/xm5ulx8e-media/table/batch confusion matrix_table_108_542681021078313e717c.table.json
2023-07-04 11:05:49,749 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/e27b55oa-media/table/batch confusion matrix_table_129_de884991beb9c4b33208.table.json
2023-07-04 11:05:49,750 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/aogonli1-media/table/batch confusion matrix_table_101_5dcc195a388d9733922f.table.json
2023-07-04 11:05:49,751 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/zkb8p53e-media/table/batch confusion matrix_table_136_1e727c5a957bfc8e9c63.table.json
2023-07-04 11:05:49,784 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/7xfzwq8u-media/table/batch confusion matrix_table_115_30f70383ce856197d41e.table.json
2023-07-04 11:05:49,794 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/7nyqrh93-media/table/batch confusion matrix_table_10_83a53314926c797ba154.table.json
2023-07-04 11:05:49,798 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/2tu55obi-media/table/batch confusion matrix_table_122_eeeae82c9a9a95bb6adc.table.json
2023-07-04 11:05:50,069 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:05:50,069 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_143_1cbad7cefce9713fd067.table.json
2023-07-04 11:05:50,069 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_164_46db23fd30a0e1b36a53.table.json
2023-07-04 11:05:50,069 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_150_542681021078313e717c.table.json
2023-07-04 11:05:50,069 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_157_da63951ea7d6deb32ee6.table.json
2023-07-04 11:05:50,069 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:05:50,393 INFO wandb-upload_0:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp7694wtrg
2023-07-04 11:05:50,475 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/h2jbbu28-media/table/batch confusion matrix_table_164_46db23fd30a0e1b36a53.table.json
2023-07-04 11:05:50,485 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/cr158ark-media/table/batch confusion matrix_table_150_542681021078313e717c.table.json
2023-07-04 11:05:50,494 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/dfn8itc7-media/table/batch confusion matrix_table_157_da63951ea7d6deb32ee6.table.json
2023-07-04 11:05:50,762 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/xb14ta8l-media/table/batch confusion matrix_table_143_1cbad7cefce9713fd067.table.json
2023-07-04 11:05:51,112 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MDE5', 'digest': '53f642263f9a570b14e8e3fcbc284320', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MDAz', 'versionIndex': 1}}, 'version': 'latest'}
2023-07-04 11:05:51,112 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:05:51,112 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_17_a7d0d7e6a80ecc2ddc2f.table.json with policy now
2023-07-04 11:05:51,113 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:51,113 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:51,113 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:51,114 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:51,114 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:51,115 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:51,115 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:51,115 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:51,115 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:51,115 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:51,116 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:51,116 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:51,116 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:51,116 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:51,116 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:51,116 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:51,116 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:51,116 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:51,116 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:51,116 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:51,116 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:51,116 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:05:51,116 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:05:51,552 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/kg4csfs2-media/table/batch confusion matrix_table_17_a7d0d7e6a80ecc2ddc2f.table.json
2023-07-04 11:05:52,070 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:05:52,132 INFO wandb-upload_5:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpxed_khiv
2023-07-04 11:05:52,623 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:52,627 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:52,957 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MDMy', 'digest': 'fa30b10c61603a43b7c0fe65726ad705', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MDE5', 'versionIndex': 2}}, 'version': 'latest'}
2023-07-04 11:05:52,957 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:05:52,957 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:05:52,957 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_24_8fbc5f39b29508567cad.table.json with policy now
2023-07-04 11:05:52,957 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:52,957 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:52,958 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:52,959 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:52,959 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:52,959 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:52,959 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:52,959 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:52,959 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:52,959 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:52,959 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:52,961 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:52,961 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:52,961 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:52,961 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:52,961 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:52,961 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:52,961 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:52,961 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:52,961 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:52,961 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:52,961 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:05:52,961 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:05:53,070 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:05:53,070 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_168_ffd1a5189d6cc2274a7c.table.json
2023-07-04 11:05:53,070 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:05:53,417 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/sfbr7zj1-media/table/batch confusion matrix_table_24_8fbc5f39b29508567cad.table.json
2023-07-04 11:05:53,478 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/cw9nfjsb-media/table/test confusion matrix_table_168_ffd1a5189d6cc2274a7c.table.json
2023-07-04 11:05:53,927 INFO wandb-upload_7:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpxbh_1gv9
2023-07-04 11:05:53,942 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:53,943 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:53,943 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,095 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:54,099 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,100 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,100 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,100 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,113 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,113 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,113 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,266 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:54,270 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,270 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,270 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,270 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,282 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,282 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,283 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,424 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:54,427 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,428 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,428 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,428 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,440 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,440 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,441 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,603 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:54,608 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,608 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,609 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,609 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,621 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,621 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,622 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,682 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MDQ5', 'digest': 'fe5835a7d3655f2a2cc48c1f999d14d5', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MDMy', 'versionIndex': 3}}, 'version': 'latest'}
2023-07-04 11:05:54,683 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:05:54,683 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_31_60d88c3a48deec0f756e.table.json with policy now
2023-07-04 11:05:54,683 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:54,683 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:54,683 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:54,683 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:54,683 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:54,683 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:54,683 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:54,683 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:54,684 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:54,684 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:54,684 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:54,684 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:54,684 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:54,684 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:54,685 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:54,685 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:54,685 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:54,685 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:54,685 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:54,685 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:54,685 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:54,685 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:05:54,685 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:05:54,775 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:54,778 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,779 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,779 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,779 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,791 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,791 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,792 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,966 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:54,970 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,970 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,971 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,971 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,982 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,983 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:54,983 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,071 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:05:55,071 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_186_4c6c1ab939cda8ea4a72.table.json
2023-07-04 11:05:55,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_179_542681021078313e717c.table.json
2023-07-04 11:05:55,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_200_1824ba9126a6c7452162.table.json
2023-07-04 11:05:55,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_193_f53c8a8d77ec5f5a4bce.table.json
2023-07-04 11:05:55,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_207_a7873205fa75b58e62cb.table.json
2023-07-04 11:05:55,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_172_e6cdd73690272b208c21.table.json
2023-07-04 11:05:55,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:05:55,115 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/0r21icos-media/table/batch confusion matrix_table_31_60d88c3a48deec0f756e.table.json
2023-07-04 11:05:55,129 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:55,134 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,134 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,135 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,135 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,148 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,148 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,148 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,306 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:55,311 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,311 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,311 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,311 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,323 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,324 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,324 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,483 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/93r3w15s-media/table/batch confusion matrix_table_207_a7873205fa75b58e62cb.table.json
2023-07-04 11:05:55,488 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/fi7hi3lw-media/table/batch confusion matrix_table_200_1824ba9126a6c7452162.table.json
2023-07-04 11:05:55,498 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:55,502 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,502 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,502 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,502 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,504 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/vvcmh7gt-media/table/batch confusion matrix_table_186_4c6c1ab939cda8ea4a72.table.json
2023-07-04 11:05:55,509 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/j1f0a53q-media/table/batch confusion matrix_table_172_e6cdd73690272b208c21.table.json
2023-07-04 11:05:55,511 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/3vtwk3gn-media/table/batch confusion matrix_table_193_f53c8a8d77ec5f5a4bce.table.json
2023-07-04 11:05:55,514 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,514 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,515 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,519 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/5p2i196u-media/table/batch confusion matrix_table_179_542681021078313e717c.table.json
2023-07-04 11:05:55,659 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:55,662 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,663 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,663 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,663 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,675 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,676 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,676 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,784 INFO wandb-upload_7:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpxo34gm3e
2023-07-04 11:05:55,840 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:55,843 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,843 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,844 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,844 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,856 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,857 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,857 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:55,998 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:56,002 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,002 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,003 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,003 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,014 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,015 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,015 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_242_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:05:56,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_249_ae8fd1dddcf14dc27777.table.json
2023-07-04 11:05:56,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_228_fe4098d4c25d56edcebd.table.json
2023-07-04 11:05:56,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_235_4c6c1ab939cda8ea4a72.table.json
2023-07-04 11:05:56,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_221_1824ba9126a6c7452162.table.json
2023-07-04 11:05:56,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_214_542681021078313e717c.table.json
2023-07-04 11:05:56,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:05:56,165 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:56,169 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,169 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,170 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,170 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,182 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,182 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,328 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:56,333 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,333 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,333 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,334 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,346 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,346 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,346 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,489 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MDY4', 'digest': '6a3a259389f4b35016a328b157f91a91', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MDQ5', 'versionIndex': 4}}, 'version': 'latest'}
2023-07-04 11:05:56,490 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:05:56,490 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_38_ac8c4a0dfc8df0779199.table.json with policy now
2023-07-04 11:05:56,490 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:56,490 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:56,491 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:56,491 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:56,491 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:56,491 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:56,491 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:56,491 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:56,491 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:56,492 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,493 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:56,493 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:56,493 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:56,493 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:56,493 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:56,493 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:56,493 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:56,494 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:56,494 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:56,494 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:56,494 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:56,494 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:56,494 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:56,494 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:05:56,494 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:05:56,539 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,540 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,540 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,540 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,540 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,541 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,662 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:56,665 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,665 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,666 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,666 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,678 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,679 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,679 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,821 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:56,825 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,825 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,826 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,826 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,837 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,838 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,838 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,890 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/0htyxu3n-media/table/batch confusion matrix_table_242_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:05:56,892 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/76ejl7tw-media/table/batch confusion matrix_table_249_ae8fd1dddcf14dc27777.table.json
2023-07-04 11:05:56,901 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/4pw68qyf-media/table/batch confusion matrix_table_235_4c6c1ab939cda8ea4a72.table.json
2023-07-04 11:05:56,916 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/9rg611yv-media/table/batch confusion matrix_table_221_1824ba9126a6c7452162.table.json
2023-07-04 11:05:56,928 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/cs42lysz-media/table/batch confusion matrix_table_214_542681021078313e717c.table.json
2023-07-04 11:05:56,930 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/48vuamvy-media/table/batch confusion matrix_table_38_ac8c4a0dfc8df0779199.table.json
2023-07-04 11:05:56,930 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/bhhconi4-media/table/batch confusion matrix_table_228_fe4098d4c25d56edcebd.table.json
2023-07-04 11:05:56,993 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:56,997 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,997 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,997 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:56,998 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,010 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,010 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,010 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:05:57,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_284_83a53314926c797ba154.table.json
2023-07-04 11:05:57,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_256_e6cdd73690272b208c21.table.json
2023-07-04 11:05:57,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_277_d4d30958e51757b86519.table.json
2023-07-04 11:05:57,072 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_291_a115e530add99a97b366.table.json
2023-07-04 11:05:57,073 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_270_e6cdd73690272b208c21.table.json
2023-07-04 11:05:57,073 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_263_e04570dcb762a9eb0c10.table.json
2023-07-04 11:05:57,073 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:05:57,167 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:57,171 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,171 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,171 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,172 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,184 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,342 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:57,347 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,348 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,348 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,348 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,361 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,361 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,361 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,489 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ab0rpg80-media/table/batch confusion matrix_table_263_e04570dcb762a9eb0c10.table.json
2023-07-04 11:05:57,501 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/uhguqs2b-media/table/batch confusion matrix_table_291_a115e530add99a97b366.table.json
2023-07-04 11:05:57,517 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/9pgbafma-media/table/batch confusion matrix_table_284_83a53314926c797ba154.table.json
2023-07-04 11:05:57,519 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:57,522 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,523 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,523 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,524 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,528 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/jq7eee1i-media/table/batch confusion matrix_table_277_d4d30958e51757b86519.table.json
2023-07-04 11:05:57,529 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/tijc5k0e-media/table/batch confusion matrix_table_256_e6cdd73690272b208c21.table.json
2023-07-04 11:05:57,535 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,535 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,536 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,569 INFO wandb-upload_4:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmppoolj420
2023-07-04 11:05:57,597 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/5zaw5u3k-media/table/batch confusion matrix_table_270_e6cdd73690272b208c21.table.json
2023-07-04 11:05:57,695 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:57,699 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,700 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,700 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,700 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,712 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,712 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,713 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,858 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:57,863 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,863 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,864 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,864 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,876 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,876 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:57,877 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:58,024 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:05:58,028 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:58,028 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:58,028 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:58,028 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:05:58,073 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_312_1824ba9126a6c7452162.table.json
2023-07-04 11:05:58,073 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_319_f4f42398d019499f73ea.table.json
2023-07-04 11:05:58,073 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_333_42b451464002c81d89cb.table.json
2023-07-04 11:05:58,073 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_298_1824ba9126a6c7452162.table.json
2023-07-04 11:05:58,073 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_326_e04570dcb762a9eb0c10.table.json
2023-07-04 11:05:58,073 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_305_5dcc195a388d9733922f.table.json
2023-07-04 11:05:58,074 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:05:58,303 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MDg3', 'digest': '918eccfe5e0fb2bea447e85979c621a6', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MDY4', 'versionIndex': 5}}, 'version': 'latest'}
2023-07-04 11:05:58,304 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:05:58,304 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_45_543f8336e4d5ebce0e61.table.json with policy now
2023-07-04 11:05:58,304 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:05:58,304 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:58,305 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:58,305 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:58,305 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:58,305 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:58,305 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:58,305 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:58,305 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:58,306 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:58,306 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:58,307 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:58,307 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:58,307 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:58,307 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:58,307 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:58,307 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:58,307 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:58,307 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:58,307 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:05:58,307 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:05:58,307 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:05:58,307 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:05:58,307 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:05:58,705 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/x4ire9t0-media/table/batch confusion matrix_table_326_e04570dcb762a9eb0c10.table.json
2023-07-04 11:05:58,721 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/3gob7u23-media/table/batch confusion matrix_table_319_f4f42398d019499f73ea.table.json
2023-07-04 11:05:58,757 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ht6958nu-media/table/batch confusion matrix_table_312_1824ba9126a6c7452162.table.json
2023-07-04 11:05:58,758 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/cerag2xm-media/table/batch confusion matrix_table_305_5dcc195a388d9733922f.table.json
2023-07-04 11:05:58,760 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/igi4zvnw-media/table/batch confusion matrix_table_298_1824ba9126a6c7452162.table.json
2023-07-04 11:05:58,776 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/xhp6qk12-media/table/batch confusion matrix_table_333_42b451464002c81d89cb.table.json
2023-07-04 11:05:58,784 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/o3mkstp3-media/table/batch confusion matrix_table_45_543f8336e4d5ebce0e61.table.json
2023-07-04 11:05:59,008 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: stop_status
2023-07-04 11:05:59,073 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:05:59,389 INFO wandb-upload_4:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp7g5rlvff
2023-07-04 11:06:00,176 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MTAz', 'digest': 'abf3a1dbd763e7670e03af36edb82e12', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MDg3', 'versionIndex': 6}}, 'version': 'latest'}
2023-07-04 11:06:00,177 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:00,177 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_52_e740c4cc03ea03bcbd2c.table.json with policy now
2023-07-04 11:06:00,177 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:00,177 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:00,178 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:00,178 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:00,178 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:00,178 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:00,178 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:00,178 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:00,179 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:00,179 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:00,179 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:00,179 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:00,179 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:00,179 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:00,179 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:00,179 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:00,180 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:00,180 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:00,180 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:00,180 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:00,181 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:00,181 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:00,181 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:00,686 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/b3uzggge-media/table/batch confusion matrix_table_52_e740c4cc03ea03bcbd2c.table.json
2023-07-04 11:06:01,074 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:01,171 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:01,175 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:01,257 INFO wandb-upload_3:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmprl_r7q0j
2023-07-04 11:06:02,003 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MTE5', 'digest': 'cff6c7d6e76288f9670860e1252bd336', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MTAz', 'versionIndex': 7}}, 'version': 'latest'}
2023-07-04 11:06:02,003 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:02,003 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_59_83a53314926c797ba154.table.json with policy now
2023-07-04 11:06:02,003 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:02,003 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:02,004 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:02,004 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:02,004 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:02,004 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:02,004 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:02,004 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:02,005 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:02,005 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:02,005 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:02,006 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:02,006 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:02,007 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:02,007 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:02,007 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:02,007 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:02,007 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:02,007 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:02,007 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:02,007 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:02,008 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:02,008 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:02,074 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:02,074 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_337_103b671e2f0c5c11418a.table.json
2023-07-04 11:06:02,074 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:02,438 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,439 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,439 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,456 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/aw7fcu2f-media/table/batch confusion matrix_table_59_83a53314926c797ba154.table.json
2023-07-04 11:06:02,498 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/0j4ere0z-media/table/test confusion matrix_table_337_103b671e2f0c5c11418a.table.json
2023-07-04 11:06:02,585 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:02,591 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,591 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,591 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,591 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,604 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,604 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,604 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,767 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:02,771 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,772 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,772 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,772 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,784 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,784 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,785 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,936 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:02,940 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,940 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,940 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,940 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,952 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,953 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:02,953 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,038 INFO wandb-upload_2:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmprlih_paq
2023-07-04 11:06:03,075 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_341_1824ba9126a6c7452162.table.json
2023-07-04 11:06:03,075 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_348_1824ba9126a6c7452162.table.json
2023-07-04 11:06:03,075 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_355_f804732f172d0e6ed3d9.table.json
2023-07-04 11:06:03,075 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:03,117 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:03,121 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,121 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,121 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,122 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,133 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,134 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,134 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,274 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:03,278 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,278 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,278 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,278 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,290 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,291 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,291 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,449 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:03,453 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,453 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,453 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,453 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,465 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,466 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,466 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,620 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:03,624 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,624 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,624 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,625 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,637 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,637 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,637 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,802 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:03,807 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,807 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,808 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,808 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,820 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,820 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,821 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,889 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MTM2', 'digest': 'e233af727ab1eeacb0b94ee496beb333', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MTE5', 'versionIndex': 8}}, 'version': 'latest'}
2023-07-04 11:06:03,889 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:03,889 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_66_d4d30958e51757b86519.table.json with policy now
2023-07-04 11:06:03,889 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:06:03,889 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:03,890 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:03,890 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:03,890 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:03,890 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:03,890 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:03,890 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:03,890 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:03,890 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:03,890 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:03,891 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:03,891 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:03,891 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:03,891 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:03,892 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:03,892 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:03,892 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:03,892 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:03,892 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:03,892 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:03,892 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:03,893 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:03,893 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:03,975 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:03,979 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,979 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,980 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,980 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,992 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,992 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:03,993 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,073 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:06:04,075 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:04,075 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_369_67f7f9c78b661281633a.table.json
2023-07-04 11:06:04,075 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_383_1824ba9126a6c7452162.table.json
2023-07-04 11:06:04,075 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_362_eeeae82c9a9a95bb6adc.table.json
2023-07-04 11:06:04,076 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_390_5fdac077eca64fafc222.table.json
2023-07-04 11:06:04,076 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_376_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:04,076 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_397_542681021078313e717c.table.json
2023-07-04 11:06:04,076 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:04,154 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:04,158 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,158 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,159 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,159 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,170 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,171 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,171 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,278 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/xvqzcsen-media/table/batch confusion matrix_table_355_f804732f172d0e6ed3d9.table.json
2023-07-04 11:06:04,305 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/g6t5bu37-media/table/batch confusion matrix_table_341_1824ba9126a6c7452162.table.json
2023-07-04 11:06:04,320 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:04,324 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,324 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,325 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,325 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,337 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,337 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,338 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,348 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/qd1lk0ur-media/table/batch confusion matrix_table_348_1824ba9126a6c7452162.table.json
2023-07-04 11:06:04,358 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ru84zjn4-media/table/batch confusion matrix_table_66_d4d30958e51757b86519.table.json
2023-07-04 11:06:04,472 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/49gxceue-media/table/batch confusion matrix_table_362_eeeae82c9a9a95bb6adc.table.json
2023-07-04 11:06:04,482 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/1tmzn538-media/table/batch confusion matrix_table_383_1824ba9126a6c7452162.table.json
2023-07-04 11:06:04,491 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/0g13psr8-media/table/batch confusion matrix_table_397_542681021078313e717c.table.json
2023-07-04 11:06:04,498 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:04,501 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,502 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,502 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,502 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,506 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/9ryx1eoa-media/table/batch confusion matrix_table_376_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:04,514 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,515 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,515 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,522 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/j1upxe4i-media/table/batch confusion matrix_table_369_67f7f9c78b661281633a.table.json
2023-07-04 11:06:04,571 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/xe457drs-media/table/batch confusion matrix_table_390_5fdac077eca64fafc222.table.json
2023-07-04 11:06:04,658 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:04,662 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,662 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,662 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,663 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,674 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,675 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,675 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,820 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:04,825 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,826 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,826 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,826 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,838 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,839 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,839 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:04,898 INFO wandb-upload_1:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp4cshcbm0
2023-07-04 11:06:04,995 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:05,000 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,000 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,000 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,001 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,013 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,013 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,013 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,076 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_425_542681021078313e717c.table.json
2023-07-04 11:06:05,076 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_432_ae8fd1dddcf14dc27777.table.json
2023-07-04 11:06:05,076 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_411_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:05,076 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_404_c2c89839631ee54a153b.table.json
2023-07-04 11:06:05,076 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_439_1824ba9126a6c7452162.table.json
2023-07-04 11:06:05,077 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_418_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:05,077 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:05,157 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:05,161 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,162 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,162 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,163 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,174 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,174 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,175 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,338 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:05,342 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,343 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,343 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,343 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,355 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,355 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,356 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,521 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:05,526 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,526 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,526 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,526 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,538 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,539 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,539 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,656 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MTUw', 'digest': 'a470c23319bd4452f5032ac4f80dc921', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MTM2', 'versionIndex': 9}}, 'version': 'latest'}
2023-07-04 11:06:05,656 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:05,656 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_73_a7873205fa75b58e62cb.table.json with policy now
2023-07-04 11:06:05,657 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:05,657 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:05,658 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:05,658 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:05,658 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:05,661 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:05,661 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:05,661 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:05,662 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:05,662 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:05,663 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:05,663 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:05,664 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:05,664 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:05,664 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:05,664 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:05,664 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:05,664 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:05,664 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:05,664 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:05,664 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:05,666 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:05,666 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:05,684 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:05,687 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,688 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,688 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,688 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,700 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,700 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,701 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,864 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:05,867 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,868 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,868 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,868 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,880 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,881 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:05,881 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,026 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:06,030 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,030 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,031 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,031 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,043 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,043 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,044 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,068 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/0nwqy3xo-media/table/batch confusion matrix_table_439_1824ba9126a6c7452162.table.json
2023-07-04 11:06:06,070 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/nvo36qm5-media/table/batch confusion matrix_table_425_542681021078313e717c.table.json
2023-07-04 11:06:06,076 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:06,076 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_481_d4d30958e51757b86519.table.json
2023-07-04 11:06:06,076 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_467_8fbc5f39b29508567cad.table.json
2023-07-04 11:06:06,077 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_446_19d420f9c457760a0e7a.table.json
2023-07-04 11:06:06,077 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_460_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:06,077 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_453_e6cdd73690272b208c21.table.json
2023-07-04 11:06:06,077 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_474_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:06,077 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:06,082 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/s7yvjdw8-media/table/batch confusion matrix_table_411_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:06,083 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/acjvaeia-media/table/batch confusion matrix_table_418_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:06,101 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/hg26lqje-media/table/batch confusion matrix_table_432_ae8fd1dddcf14dc27777.table.json
2023-07-04 11:06:06,103 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/rtbq1wrn-media/table/batch confusion matrix_table_404_c2c89839631ee54a153b.table.json
2023-07-04 11:06:06,133 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/vbe90v6n-media/table/batch confusion matrix_table_73_a7873205fa75b58e62cb.table.json
2023-07-04 11:06:06,199 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:06,203 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,204 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,204 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,204 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,216 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,217 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,218 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,370 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:06,375 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,376 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,376 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,376 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,388 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,389 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,390 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,474 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/bakd8go7-media/table/batch confusion matrix_table_460_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:06,485 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/5r6pkspx-media/table/batch confusion matrix_table_467_8fbc5f39b29508567cad.table.json
2023-07-04 11:06:06,500 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/2eli8a73-media/table/batch confusion matrix_table_474_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:06,514 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/t1lb959b-media/table/batch confusion matrix_table_481_d4d30958e51757b86519.table.json
2023-07-04 11:06:06,530 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/lpm9x5xs-media/table/batch confusion matrix_table_446_19d420f9c457760a0e7a.table.json
2023-07-04 11:06:06,534 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:06,538 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,538 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,538 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,539 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:06,550 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/qpxpndl4-media/table/batch confusion matrix_table_453_e6cdd73690272b208c21.table.json
2023-07-04 11:06:07,077 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_488_542681021078313e717c.table.json
2023-07-04 11:06:07,077 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_495_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:07,077 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_502_6f3ddb785c9b58b4d785.table.json
2023-07-04 11:06:07,077 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:07,174 INFO wandb-upload_9:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpbacesuj8
2023-07-04 11:06:07,497 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/3paym3rf-media/table/batch confusion matrix_table_502_6f3ddb785c9b58b4d785.table.json
2023-07-04 11:06:07,498 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/xxtpxd03-media/table/batch confusion matrix_table_495_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:07,513 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ylvesi66-media/table/batch confusion matrix_table_488_542681021078313e717c.table.json
2023-07-04 11:06:08,083 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MTc0', 'digest': '4a8245ff2145856179e818c0a39ff309', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MTUw', 'versionIndex': 10}}, 'version': 'latest'}
2023-07-04 11:06:08,083 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:08,083 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_80_e6cdd73690272b208c21.table.json with policy now
2023-07-04 11:06:08,083 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:08,083 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:08,083 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:08,083 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:08,083 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:08,084 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:08,084 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:08,084 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:08,085 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:08,085 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:08,085 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:08,085 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:08,085 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:08,085 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:08,085 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:08,085 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:08,085 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:08,085 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:08,085 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:08,085 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:08,086 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:08,086 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:08,086 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:08,531 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ui3j5d7o-media/table/batch confusion matrix_table_80_e6cdd73690272b208c21.table.json
2023-07-04 11:06:09,078 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:09,087 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:06:09,152 INFO wandb-upload_3:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp2ron6gr5
2023-07-04 11:06:09,715 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:09,719 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:09,982 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MjA1', 'digest': 'fa6ef188b0ff928e08d33dac84eac5ef', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MTc0', 'versionIndex': 11}}, 'version': 'latest'}
2023-07-04 11:06:09,982 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:09,982 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_87_542681021078313e717c.table.json with policy now
2023-07-04 11:06:09,982 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:09,982 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:09,982 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:06:09,983 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:09,983 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:09,983 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:09,983 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:09,983 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:09,983 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:09,983 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:09,984 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:09,984 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:09,984 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:09,984 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:09,984 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:09,984 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:09,985 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:09,985 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:09,985 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:09,986 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:09,986 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:09,986 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:09,986 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:09,986 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:10,078 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:10,078 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_506_fe711bb6556a1d9a19ed.table.json
2023-07-04 11:06:10,079 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:10,474 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/c2cgx5yy-media/table/batch confusion matrix_table_87_542681021078313e717c.table.json
2023-07-04 11:06:10,520 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/v2wx2wol-media/table/test confusion matrix_table_506_fe711bb6556a1d9a19ed.table.json
2023-07-04 11:06:11,002 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,003 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,004 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,007 INFO wandb-upload_4:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpobkc9qeo
2023-07-04 11:06:11,165 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:11,169 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,170 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,170 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,170 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,348 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:11,353 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,353 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,353 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,353 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,365 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,366 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,366 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,527 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:11,530 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,530 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,531 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,531 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,543 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,543 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,544 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,737 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:11,741 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,741 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,742 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,742 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,753 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,754 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,754 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,918 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MjIz', 'digest': '579eb8003bda3a67515d003caa4ea028', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MjA1', 'versionIndex': 12}}, 'version': 'latest'}
2023-07-04 11:06:11,918 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:11,918 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_94_5e8599edc26a28b0bf56.table.json with policy now
2023-07-04 11:06:11,918 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:11,918 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:11,918 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:11,919 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:11,919 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:11,919 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:11,919 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:11,919 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:11,919 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:11,919 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:11,919 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:11,919 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:11,919 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:11,919 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:11,921 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:11,921 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:11,921 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:11,921 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:11,921 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:11,921 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:11,921 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:11,921 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:11,921 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:11,982 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:11,986 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,986 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,986 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,987 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,998 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,999 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:11,999 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,080 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:12,080 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_531_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:12,080 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_517_542681021078313e717c.table.json
2023-07-04 11:06:12,080 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_510_1824ba9126a6c7452162.table.json
2023-07-04 11:06:12,080 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_538_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:12,080 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_524_83a53314926c797ba154.table.json
2023-07-04 11:06:12,080 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:12,159 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:12,163 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,163 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,164 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,164 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,176 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,177 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,177 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,352 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:12,356 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,356 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,356 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,356 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,368 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,368 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,369 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,420 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/gi9e4w1u-media/table/batch confusion matrix_table_94_5e8599edc26a28b0bf56.table.json
2023-07-04 11:06:12,510 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/jjz0a5kc-media/table/batch confusion matrix_table_510_1824ba9126a6c7452162.table.json
2023-07-04 11:06:12,511 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/h0cbk7vj-media/table/batch confusion matrix_table_517_542681021078313e717c.table.json
2023-07-04 11:06:12,513 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/7tjdphlw-media/table/batch confusion matrix_table_538_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:12,514 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:12,520 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,520 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,520 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,521 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,521 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/z8yswsks-media/table/batch confusion matrix_table_524_83a53314926c797ba154.table.json
2023-07-04 11:06:12,532 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,532 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,532 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,553 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/p736qllg-media/table/batch confusion matrix_table_531_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:12,679 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:12,682 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,682 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,682 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,683 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,695 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,696 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,696 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,862 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:12,865 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,866 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,866 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,866 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,878 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,879 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:12,879 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,020 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:13,024 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,024 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,024 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,024 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,037 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,037 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,037 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,080 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_580_1824ba9126a6c7452162.table.json
2023-07-04 11:06:13,080 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_566_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:13,080 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_545_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:13,081 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_559_1824ba9126a6c7452162.table.json
2023-07-04 11:06:13,081 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_552_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:13,081 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_573_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:13,081 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:13,085 INFO wandb-upload_7:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmprjp9q4e_
2023-07-04 11:06:13,189 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:13,193 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,193 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,193 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,193 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,205 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,206 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,206 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,381 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:13,384 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,384 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,385 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,385 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,396 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,397 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,397 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,491 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/upw59en9-media/table/batch confusion matrix_table_545_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:13,528 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/zasnpf20-media/table/batch confusion matrix_table_580_1824ba9126a6c7452162.table.json
2023-07-04 11:06:13,529 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/u6uycedn-media/table/batch confusion matrix_table_566_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:13,611 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:13,615 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,615 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,615 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,615 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,627 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,628 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,628 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,832 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:13,837 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,837 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,837 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,837 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,849 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,850 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,850 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:13,974 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MjQ2', 'digest': 'c191bebe3fdfeda6da9d99c78921a0a6', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MjIz', 'versionIndex': 13}}, 'version': 'latest'}
2023-07-04 11:06:13,974 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:13,974 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_101_5dcc195a388d9733922f.table.json with policy now
2023-07-04 11:06:13,974 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:13,974 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:13,975 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:13,975 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:13,975 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:13,975 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:13,975 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:13,975 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:13,975 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:13,975 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:13,976 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:13,976 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:13,976 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:13,976 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:13,977 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:13,977 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:13,977 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:13,977 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:13,977 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:13,977 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:13,977 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:13,978 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:13,978 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:14,028 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:14,031 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,031 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,031 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,031 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,044 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,044 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,044 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,081 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:14,081 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_587_542681021078313e717c.table.json
2023-07-04 11:06:14,081 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_594_83a53314926c797ba154.table.json
2023-07-04 11:06:14,081 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_601_ae8fd1dddcf14dc27777.table.json
2023-07-04 11:06:14,081 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_608_e6cdd73690272b208c21.table.json
2023-07-04 11:06:14,082 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_615_38f4fb86d5b08d411896.table.json
2023-07-04 11:06:14,082 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:14,111 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:06:14,196 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:14,199 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,199 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,200 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,200 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,212 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,212 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,212 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,372 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/6hu412d6-media/table/batch confusion matrix_table_552_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:14,374 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:14,378 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,378 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,378 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,379 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,390 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,391 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,391 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,422 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/uushnew6-media/table/batch confusion matrix_table_559_1824ba9126a6c7452162.table.json
2023-07-04 11:06:14,439 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/27d5ombn-media/table/batch confusion matrix_table_573_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:14,452 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/yfbdll48-media/table/batch confusion matrix_table_101_5dcc195a388d9733922f.table.json
2023-07-04 11:06:14,492 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/99dnhqmg-media/table/batch confusion matrix_table_608_e6cdd73690272b208c21.table.json
2023-07-04 11:06:14,498 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/83h0nf5t-media/table/batch confusion matrix_table_587_542681021078313e717c.table.json
2023-07-04 11:06:14,516 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/57treqgo-media/table/batch confusion matrix_table_594_83a53314926c797ba154.table.json
2023-07-04 11:06:14,522 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/9kl9ms0u-media/table/batch confusion matrix_table_615_38f4fb86d5b08d411896.table.json
2023-07-04 11:06:14,546 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:14,547 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/lb7xp2xr-media/table/batch confusion matrix_table_601_ae8fd1dddcf14dc27777.table.json
2023-07-04 11:06:14,549 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,550 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,550 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,550 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,562 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,563 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,563 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,729 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:14,732 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,733 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,733 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,733 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,745 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,745 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,746 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,895 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:14,900 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,900 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,901 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,901 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,913 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,913 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:14,913 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,071 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:15,076 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,076 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,076 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,076 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,081 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_650_1824ba9126a6c7452162.table.json
2023-07-04 11:06:15,082 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_636_542681021078313e717c.table.json
2023-07-04 11:06:15,082 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_657_e6cdd73690272b208c21.table.json
2023-07-04 11:06:15,082 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_629_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:15,082 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_622_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:15,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_643_67f7f9c78b661281633a.table.json
2023-07-04 11:06:15,086 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:15,088 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,089 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,089 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,101 INFO wandb-upload_8:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp18i25hoa
2023-07-04 11:06:15,261 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:15,268 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,268 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,268 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,269 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,276 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,276 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,276 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,462 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:15,466 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,466 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,466 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,466 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:15,508 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/5bb1gnel-media/table/batch confusion matrix_table_657_e6cdd73690272b208c21.table.json
2023-07-04 11:06:15,515 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ipul8g4c-media/table/batch confusion matrix_table_650_1824ba9126a6c7452162.table.json
2023-07-04 11:06:15,522 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/8g7pka3f-media/table/batch confusion matrix_table_643_67f7f9c78b661281633a.table.json
2023-07-04 11:06:15,537 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/2z6s47dt-media/table/batch confusion matrix_table_629_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:15,537 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/tfajzugu-media/table/batch confusion matrix_table_636_542681021078313e717c.table.json
2023-07-04 11:06:15,555 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/nhf1u1zw-media/table/batch confusion matrix_table_622_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:15,926 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2Mjc1', 'digest': 'fa6ef188b0ff928e08d33dac84eac5ef', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MjQ2', 'versionIndex': 14}}, 'version': 'latest'}
2023-07-04 11:06:15,927 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:06:16,082 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_671_3b09597a4c50af185128.table.json
2023-07-04 11:06:16,082 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_664_67f7f9c78b661281633a.table.json
2023-07-04 11:06:16,082 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:16,097 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:16,097 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_108_542681021078313e717c.table.json with policy now
2023-07-04 11:06:16,097 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:16,097 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:16,098 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:16,098 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:16,098 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:16,098 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:16,098 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:16,098 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:16,098 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:16,098 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:16,098 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:16,098 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:16,099 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:16,099 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:16,099 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:16,099 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:16,099 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:16,099 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:16,099 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:16,099 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:16,100 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:16,100 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:16,100 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:16,493 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/uoa07qrr-media/table/batch confusion matrix_table_671_3b09597a4c50af185128.table.json
2023-07-04 11:06:16,497 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/yvymhd07-media/table/batch confusion matrix_table_664_67f7f9c78b661281633a.table.json
2023-07-04 11:06:16,567 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/sz0jm9br-media/table/batch confusion matrix_table_108_542681021078313e717c.table.json
2023-07-04 11:06:17,142 INFO wandb-upload_3:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp91zno8w5
2023-07-04 11:06:17,912 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2Mjkx', 'digest': '3a2257deedb2d5a6db7cfa9e3a492b60', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2Mjc1', 'versionIndex': 15}}, 'version': 'latest'}
2023-07-04 11:06:17,912 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:17,912 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_115_30f70383ce856197d41e.table.json with policy now
2023-07-04 11:06:17,912 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:17,913 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:17,913 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:17,913 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:17,913 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:17,913 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:17,913 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:17,913 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:17,913 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:17,913 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:17,914 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:17,914 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:17,914 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:17,914 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:17,914 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:17,914 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:17,915 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:17,915 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:17,915 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:17,915 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:17,915 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:17,915 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:17,915 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:18,083 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:18,083 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/config.yaml
2023-07-04 11:06:18,409 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/x6ykiz2b-media/table/batch confusion matrix_table_115_30f70383ce856197d41e.table.json
2023-07-04 11:06:18,680 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:18,684 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:19,025 INFO wandb-upload_0:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp8cisc5hq
2023-07-04 11:06:19,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:06:19,824 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MzE1', 'digest': 'bc0225fb0fab0db61303384269115142', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2Mjkx', 'versionIndex': 16}}, 'version': 'latest'}
2023-07-04 11:06:19,825 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:19,825 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_122_eeeae82c9a9a95bb6adc.table.json with policy now
2023-07-04 11:06:19,825 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:19,825 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:19,826 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:19,826 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:19,826 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:19,826 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:19,826 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:19,826 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:19,826 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:19,827 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:19,828 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:19,828 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:19,828 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:19,828 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:19,828 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:19,828 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:19,828 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:19,828 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:19,828 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:19,828 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:19,829 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:19,829 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:19,829 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:19,953 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:19,954 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:19,954 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,083 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:20,083 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_675_a5d259012c0dcca70a08.table.json
2023-07-04 11:06:20,083 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:20,116 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:20,119 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,120 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,120 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,120 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,133 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,134 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,134 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,284 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:20,288 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,288 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,288 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,288 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,300 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,300 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,301 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,329 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/l01zer8s-media/table/batch confusion matrix_table_122_eeeae82c9a9a95bb6adc.table.json
2023-07-04 11:06:20,449 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:20,454 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,454 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,454 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,454 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,466 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,467 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,467 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,533 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/8ofopvqn-media/table/test confusion matrix_table_675_a5d259012c0dcca70a08.table.json
2023-07-04 11:06:20,612 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:20,617 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,618 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,618 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,619 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,630 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,631 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,631 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,797 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:20,802 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,802 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,802 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,803 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,814 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,815 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,815 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,849 INFO wandb-upload_7:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpdr6kpbt0
2023-07-04 11:06:20,958 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:20,962 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,962 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,962 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,962 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,974 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,975 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:20,975 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,133 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:21,136 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,137 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,137 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,137 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,149 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,149 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,149 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,306 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:21,310 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,310 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,310 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,310 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,322 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,322 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,323 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,533 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:21,537 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,538 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,538 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,538 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,550 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,550 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,550 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,694 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:21,698 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,699 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,699 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,699 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,711 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,712 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,712 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,776 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MzMw', 'digest': 'c37f48dc0e297004b1f3c03425b230d4', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MzE1', 'versionIndex': 17}}, 'version': 'latest'}
2023-07-04 11:06:21,776 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:21,776 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:06:21,776 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_129_de884991beb9c4b33208.table.json with policy now
2023-07-04 11:06:21,777 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:21,777 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:21,777 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:21,777 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:21,777 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:21,777 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:21,777 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:21,777 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:21,779 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:21,779 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:21,779 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:21,779 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:21,779 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:21,779 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:21,779 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:21,779 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:21,779 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:21,779 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:21,779 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:21,779 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:21,779 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:21,779 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:21,780 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:21,859 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:21,863 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,863 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,863 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,864 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,875 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,876 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:21,876 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,044 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:22,048 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,049 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,049 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,049 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,061 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,062 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,062 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,084 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:22,084 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_714_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:22,084 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_728_542681021078313e717c.table.json
2023-07-04 11:06:22,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_749_1824ba9126a6c7452162.table.json
2023-07-04 11:06:22,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_735_e6cdd73690272b208c21.table.json
2023-07-04 11:06:22,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_679_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:22,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_756_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:22,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_693_e6cdd73690272b208c21.table.json
2023-07-04 11:06:22,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_721_da75045c8d94682c0dec.table.json
2023-07-04 11:06:22,087 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_700_542681021078313e717c.table.json
2023-07-04 11:06:22,087 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_742_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:22,088 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_686_4c6c1ab939cda8ea4a72.table.json
2023-07-04 11:06:22,088 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_707_e6cdd73690272b208c21.table.json
2023-07-04 11:06:22,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:22,225 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:22,227 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,228 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,229 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,229 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,239 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,240 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,240 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,265 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/hbrailw8-media/table/batch confusion matrix_table_129_de884991beb9c4b33208.table.json
2023-07-04 11:06:22,387 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:22,391 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,391 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,391 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,391 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,403 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,403 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,404 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,497 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/o3chgklc-media/table/batch confusion matrix_table_693_e6cdd73690272b208c21.table.json
2023-07-04 11:06:22,497 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/62aqzc39-media/table/batch confusion matrix_table_679_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:22,498 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/oyd2fknz-media/table/batch confusion matrix_table_728_542681021078313e717c.table.json
2023-07-04 11:06:22,499 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/z18qq1bb-media/table/batch confusion matrix_table_714_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:22,511 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/d8hjlw1r-media/table/batch confusion matrix_table_756_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:22,513 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/43dcsukm-media/table/batch confusion matrix_table_700_542681021078313e717c.table.json
2023-07-04 11:06:22,529 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/kvishqeh-media/table/batch confusion matrix_table_735_e6cdd73690272b208c21.table.json
2023-07-04 11:06:22,536 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/hywf1v5o-media/table/batch confusion matrix_table_749_1824ba9126a6c7452162.table.json
2023-07-04 11:06:22,542 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/rwmi295l-media/table/batch confusion matrix_table_742_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:22,548 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/wu59i3km-media/table/batch confusion matrix_table_721_da75045c8d94682c0dec.table.json
2023-07-04 11:06:22,554 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ubdhntgt-media/table/batch confusion matrix_table_686_4c6c1ab939cda8ea4a72.table.json
2023-07-04 11:06:22,557 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/m0b01zfl-media/table/batch confusion matrix_table_707_e6cdd73690272b208c21.table.json
2023-07-04 11:06:22,557 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:22,561 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,561 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,562 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,562 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,574 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,574 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,574 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,725 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:22,728 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,728 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,728 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,728 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,740 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,741 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,741 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,858 INFO wandb-upload_9:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpwerp_z7s
2023-07-04 11:06:22,904 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:22,908 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,908 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,908 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,908 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,920 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,920 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:22,921 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,080 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:23,084 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,084 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,084 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,084 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,096 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,097 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,097 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,250 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:23,254 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,255 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,255 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,255 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,266 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,267 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,267 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,425 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:23,429 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,429 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,429 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,429 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,442 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,442 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,442 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,604 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:23,607 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,607 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,607 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,608 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,620 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,620 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,620 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,650 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2MzUy', 'digest': '7b0e2746c8f50a97b82b1cf3469eb77f', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MzMw', 'versionIndex': 18}}, 'version': 'latest'}
2023-07-04 11:06:23,650 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:23,650 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_136_1e727c5a957bfc8e9c63.table.json with policy now
2023-07-04 11:06:23,651 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:23,651 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:23,651 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:23,651 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:23,651 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:23,651 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:23,651 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:23,651 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:23,652 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:23,652 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:23,652 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:23,652 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:23,652 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:23,652 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:23,652 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:23,652 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:23,652 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:23,653 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:23,653 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:23,653 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:23,653 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:23,654 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:23,654 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:23,776 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:23,823 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,824 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,824 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,824 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,824 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,825 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,825 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,958 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:23,961 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,962 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,962 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,962 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,972 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,973 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:23,973 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:24,084 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:24,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_763_67f7f9c78b661281633a.table.json
2023-07-04 11:06:24,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_805_e1213d070eb371177398.table.json
2023-07-04 11:06:24,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_798_1824ba9126a6c7452162.table.json
2023-07-04 11:06:24,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_833_1824ba9126a6c7452162.table.json
2023-07-04 11:06:24,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_826_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:24,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_812_542681021078313e717c.table.json
2023-07-04 11:06:24,086 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_770_542681021078313e717c.table.json
2023-07-04 11:06:24,086 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_791_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:24,086 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_819_1824ba9126a6c7452162.table.json
2023-07-04 11:06:24,088 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_784_67f7f9c78b661281633a.table.json
2023-07-04 11:06:24,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_777_1824ba9126a6c7452162.table.json
2023-07-04 11:06:24,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:24,104 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/9hsrfyee-media/table/batch confusion matrix_table_136_1e727c5a957bfc8e9c63.table.json
2023-07-04 11:06:24,134 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:24,138 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:24,138 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:24,138 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:24,138 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:24,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:06:24,504 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/j18vmenm-media/table/batch confusion matrix_table_763_67f7f9c78b661281633a.table.json
2023-07-04 11:06:24,504 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/bjn4tid0-media/table/batch confusion matrix_table_798_1824ba9126a6c7452162.table.json
2023-07-04 11:06:24,513 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/48j4bc8y-media/table/batch confusion matrix_table_833_1824ba9126a6c7452162.table.json
2023-07-04 11:06:24,514 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/jfxvwzmj-media/table/batch confusion matrix_table_805_e1213d070eb371177398.table.json
2023-07-04 11:06:24,514 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/00xdtm4n-media/table/batch confusion matrix_table_791_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:24,515 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/fnlosyu8-media/table/batch confusion matrix_table_770_542681021078313e717c.table.json
2023-07-04 11:06:24,515 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/b3shg7qz-media/table/batch confusion matrix_table_812_542681021078313e717c.table.json
2023-07-04 11:06:24,560 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/bgb1cdsj-media/table/batch confusion matrix_table_826_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:24,588 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/8zp942z2-media/table/batch confusion matrix_table_819_1824ba9126a6c7452162.table.json
2023-07-04 11:06:24,589 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/8nj7rws3-media/table/batch confusion matrix_table_777_1824ba9126a6c7452162.table.json
2023-07-04 11:06:24,591 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/1pyag2r4-media/table/batch confusion matrix_table_784_67f7f9c78b661281633a.table.json
2023-07-04 11:06:24,761 INFO wandb-upload_10:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmphyrfjh_k
2023-07-04 11:06:25,552 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2Mzcw', 'digest': 'ba34d8181b13fe2bc25a0a8ede8fa308', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2MzUy', 'versionIndex': 19}}, 'version': 'latest'}
2023-07-04 11:06:25,552 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:25,552 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_143_1cbad7cefce9713fd067.table.json with policy now
2023-07-04 11:06:25,553 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:25,553 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:25,553 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:25,554 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:25,554 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:25,554 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:25,554 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:25,554 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:25,554 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:25,554 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:25,554 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:25,554 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:25,554 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:25,554 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:25,555 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:25,556 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:25,556 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:25,556 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:25,556 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:25,556 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:25,556 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:25,556 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:25,556 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:26,024 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/z7b7wecd-media/table/batch confusion matrix_table_143_1cbad7cefce9713fd067.table.json
2023-07-04 11:06:26,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:26,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_840_17ac89d9b34a12ddbdd5.table.json
2023-07-04 11:06:26,085 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:26,554 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/wrcpwrdv-media/table/batch confusion matrix_table_840_17ac89d9b34a12ddbdd5.table.json
2023-07-04 11:06:26,618 INFO wandb-upload_5:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp6y8nnlgm
2023-07-04 11:06:27,369 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:27,372 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:27,414 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2Mzg3', 'digest': 'fa6ef188b0ff928e08d33dac84eac5ef', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2Mzcw', 'versionIndex': 20}}, 'version': 'latest'}
2023-07-04 11:06:27,414 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:27,414 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_150_542681021078313e717c.table.json with policy now
2023-07-04 11:06:27,414 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:06:27,415 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:27,415 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:27,415 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:27,415 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:27,415 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:27,415 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:27,415 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:27,415 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:27,415 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:27,415 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:27,416 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:27,416 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:27,416 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:27,416 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:27,416 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:27,416 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:27,416 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:27,417 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:27,417 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:27,417 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:27,418 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:27,418 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:27,418 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:27,855 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/o39aawxi-media/table/batch confusion matrix_table_150_542681021078313e717c.table.json
2023-07-04 11:06:28,086 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:28,086 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_844_779bca52e4f34e45df60.table.json
2023-07-04 11:06:28,086 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:28,485 INFO wandb-upload_2:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp9nnao3ce
2023-07-04 11:06:28,500 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/23xfmgz9-media/table/test confusion matrix_table_844_779bca52e4f34e45df60.table.json
2023-07-04 11:06:28,681 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:28,682 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:28,682 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:28,841 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:28,845 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:28,845 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:28,845 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:28,845 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:28,858 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:28,858 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:28,859 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,002 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:29,007 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,008 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,008 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,008 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,020 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,021 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,021 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,179 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:29,182 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,195 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,196 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,196 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,202 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:06:29,260 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NDA4', 'digest': '53124b271e7052d7751d95dcce2ad15d', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2Mzg3', 'versionIndex': 21}}, 'version': 'latest'}
2023-07-04 11:06:29,260 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:29,261 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_157_da63951ea7d6deb32ee6.table.json with policy now
2023-07-04 11:06:29,261 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:29,261 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:29,262 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:29,262 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:29,262 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:29,262 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:29,262 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:29,262 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:29,263 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:29,263 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:29,263 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:29,263 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:29,263 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:29,263 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:29,263 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:29,264 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:29,264 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:29,264 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:29,264 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:29,264 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:29,264 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:29,264 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:29,264 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:29,340 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:29,343 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,344 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,344 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,344 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,356 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,356 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,357 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,502 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:29,506 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,507 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,507 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,507 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,518 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,519 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,519 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,697 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/exvabvov-media/table/batch confusion matrix_table_157_da63951ea7d6deb32ee6.table.json
2023-07-04 11:06:29,702 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:29,706 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,706 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,706 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,706 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,717 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,718 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,718 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,861 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:29,864 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,864 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,865 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,865 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,877 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,877 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:29,878 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,030 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:30,034 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,034 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,034 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,034 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,046 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,047 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,047 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,087 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:30,087 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_897_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:30,087 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_848_da75045c8d94682c0dec.table.json
2023-07-04 11:06:30,087 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_883_e6cdd73690272b208c21.table.json
2023-07-04 11:06:30,087 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_876_67f7f9c78b661281633a.table.json
2023-07-04 11:06:30,088 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_862_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:30,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_855_e6cdd73690272b208c21.table.json
2023-07-04 11:06:30,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_890_1824ba9126a6c7452162.table.json
2023-07-04 11:06:30,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_869_67f7f9c78b661281633a.table.json
2023-07-04 11:06:30,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:30,218 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:30,223 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,223 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,223 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,223 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,235 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,236 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,236 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,378 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:30,381 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,381 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,382 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,382 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,394 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,395 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,395 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,400 INFO wandb-upload_3:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp3o6fnsqd
2023-07-04 11:06:30,497 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/5dc2syb9-media/table/batch confusion matrix_table_848_da75045c8d94682c0dec.table.json
2023-07-04 11:06:30,507 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/h9jlktsi-media/table/batch confusion matrix_table_883_e6cdd73690272b208c21.table.json
2023-07-04 11:06:30,508 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/56498rdk-media/table/batch confusion matrix_table_890_1824ba9126a6c7452162.table.json
2023-07-04 11:06:30,522 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/dfmcvpru-media/table/batch confusion matrix_table_897_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:30,535 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ktaa48ey-media/table/batch confusion matrix_table_876_67f7f9c78b661281633a.table.json
2023-07-04 11:06:30,547 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/nkmxibxw-media/table/batch confusion matrix_table_862_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:30,550 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/dcgttsj3-media/table/batch confusion matrix_table_855_e6cdd73690272b208c21.table.json
2023-07-04 11:06:30,550 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:30,554 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,554 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,555 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,555 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,559 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/rcj8eid2-media/table/batch confusion matrix_table_869_67f7f9c78b661281633a.table.json
2023-07-04 11:06:30,567 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,567 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,567 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,736 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:30,741 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,741 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,741 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,741 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,753 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,754 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,754 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,900 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:30,903 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,903 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,903 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,904 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,916 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,916 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:30,916 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,081 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:31,084 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,085 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,085 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,085 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,097 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,098 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,098 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,216 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NDIy', 'digest': '8c1d7e0a870295b90155221bc95a02ae', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NDA4', 'versionIndex': 22}}, 'version': 'latest'}
2023-07-04 11:06:31,216 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:31,216 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_164_46db23fd30a0e1b36a53.table.json with policy now
2023-07-04 11:06:31,216 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:31,216 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:31,216 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:31,217 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:31,217 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:31,217 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:31,217 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:31,217 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:31,217 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:31,217 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:31,217 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:31,217 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:31,218 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:31,218 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:31,251 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:31,254 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,255 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,255 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,255 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,267 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,268 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,268 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,426 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:31,429 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,430 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,430 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,430 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,442 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,443 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,443 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,586 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:31,590 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,590 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,590 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,590 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,602 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,603 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,603 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,694 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/hgrskl3z-media/table/batch confusion matrix_table_164_46db23fd30a0e1b36a53.table.json
2023-07-04 11:06:31,751 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:31,754 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,755 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,755 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,755 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,767 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,767 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,767 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:31,926 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:32,021 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,021 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,021 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,022 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,033 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,034 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,034 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,088 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:32,088 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/output.log
2023-07-04 11:06:32,088 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_974_1824ba9126a6c7452162.table.json
2023-07-04 11:06:32,088 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_904_1824ba9126a6c7452162.table.json
2023-07-04 11:06:32,088 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_960_67f7f9c78b661281633a.table.json
2023-07-04 11:06:32,088 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_946_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:32,088 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_925_1824ba9126a6c7452162.table.json
2023-07-04 11:06:32,088 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_939_83a53314926c797ba154.table.json
2023-07-04 11:06:32,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_967_67f7f9c78b661281633a.table.json
2023-07-04 11:06:32,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_918_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:32,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_932_542681021078313e717c.table.json
2023-07-04 11:06:32,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_911_1824ba9126a6c7452162.table.json
2023-07-04 11:06:32,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_953_da75045c8d94682c0dec.table.json
2023-07-04 11:06:32,091 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:32,177 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:32,180 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,180 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,181 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,181 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,193 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,193 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,193 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,327 INFO wandb-upload_12:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp_sz61vrv
2023-07-04 11:06:32,352 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:32,356 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,356 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,356 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,356 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,368 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,368 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,369 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,494 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/iwi69lym-media/table/batch confusion matrix_table_960_67f7f9c78b661281633a.table.json
2023-07-04 11:06:32,496 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/t687i66o-media/table/batch confusion matrix_table_925_1824ba9126a6c7452162.table.json
2023-07-04 11:06:32,500 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/pwbldtwg-media/table/batch confusion matrix_table_904_1824ba9126a6c7452162.table.json
2023-07-04 11:06:32,505 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/968ukj4m-media/table/batch confusion matrix_table_911_1824ba9126a6c7452162.table.json
2023-07-04 11:06:32,510 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/fd0wtzl1-media/table/batch confusion matrix_table_967_67f7f9c78b661281633a.table.json
2023-07-04 11:06:32,512 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/bfrlrh48-media/table/batch confusion matrix_table_939_83a53314926c797ba154.table.json
2023-07-04 11:06:32,513 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/p5dtf4ct-media/table/batch confusion matrix_table_932_542681021078313e717c.table.json
2023-07-04 11:06:32,513 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:32,517 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,517 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,517 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,518 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,530 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,530 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,530 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,535 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/pupx77vn-media/table/batch confusion matrix_table_974_1824ba9126a6c7452162.table.json
2023-07-04 11:06:32,535 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/rny9cgon-media/table/batch confusion matrix_table_953_da75045c8d94682c0dec.table.json
2023-07-04 11:06:32,536 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/yrglaf6q-media/table/batch confusion matrix_table_918_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:32,656 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/pu1b1ju2-media/table/batch confusion matrix_table_946_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:32,688 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:32,692 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,692 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,692 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,692 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,702 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,703 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,703 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,847 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:32,850 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,851 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,851 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:32,851 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:33,149 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-testconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NDM0', 'digest': '9732232e9f4c115fe50df33f2a347f41', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMTE1', 'latestArtifact': None}, 'version': 'latest'}
2023-07-04 11:06:33,149 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:33,149 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/test confusion matrix_table_168_ffd1a5189d6cc2274a7c.table.json with policy now
2023-07-04 11:06:33,149 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:06:33,150 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:33,150 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:33,150 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:33,150 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:33,150 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:33,151 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:33,151 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:33,151 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:33,151 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:33,151 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:33,151 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:33,151 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:33,151 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:33,151 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:33,617 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/dkofegt0-media/table/test confusion matrix_table_168_ffd1a5189d6cc2274a7c.table.json
2023-07-04 11:06:34,088 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:34,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1009_42b451464002c81d89cb.table.json
2023-07-04 11:06:34,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1002_e6cdd73690272b208c21.table.json
2023-07-04 11:06:34,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_988_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:34,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_981_1824ba9126a6c7452162.table.json
2023-07-04 11:06:34,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_995_91631cdbd9a35f428cc7.table.json
2023-07-04 11:06:34,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:34,180 INFO wandb-upload_6:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpq2dupw2z
2023-07-04 11:06:34,225 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:06:34,488 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/o8bjwmwn-media/table/batch confusion matrix_table_1009_42b451464002c81d89cb.table.json
2023-07-04 11:06:34,493 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/vxgroqen-media/table/batch confusion matrix_table_1002_e6cdd73690272b208c21.table.json
2023-07-04 11:06:34,503 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/nxn5btdu-media/table/batch confusion matrix_table_981_1824ba9126a6c7452162.table.json
2023-07-04 11:06:34,509 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/5xer46j5-media/table/batch confusion matrix_table_988_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:34,513 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/lcs78zgb-media/table/batch confusion matrix_table_995_91631cdbd9a35f428cc7.table.json
2023-07-04 11:06:34,868 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NDQ5', 'digest': '4a8245ff2145856179e818c0a39ff309', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NDIy', 'versionIndex': 23}}, 'version': 'latest'}
2023-07-04 11:06:34,868 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:34,868 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_172_e6cdd73690272b208c21.table.json with policy now
2023-07-04 11:06:34,869 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:34,869 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:34,869 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:34,869 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:34,869 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:34,869 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:34,870 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:34,870 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:34,871 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:34,871 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:34,871 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:34,871 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:34,871 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:34,871 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:34,871 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:34,871 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:34,871 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:34,872 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:34,872 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:34,872 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:34,872 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:34,872 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:34,872 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:35,302 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/v7hd2hr4-media/table/batch confusion matrix_table_172_e6cdd73690272b208c21.table.json
2023-07-04 11:06:35,887 INFO wandb-upload_11:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpf9k_ekmk
2023-07-04 11:06:35,956 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:35,960 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:36,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:36,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_1013_50025681b919183fa018.table.json
2023-07-04 11:06:36,089 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:36,643 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NDY2', 'digest': 'fa6ef188b0ff928e08d33dac84eac5ef', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NDQ5', 'versionIndex': 24}}, 'version': 'latest'}
2023-07-04 11:06:36,643 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:36,643 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_179_542681021078313e717c.table.json with policy now
2023-07-04 11:06:36,644 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:36,644 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:36,645 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:36,645 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:36,645 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:36,645 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:36,645 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:36,645 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:36,646 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:36,646 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:36,646 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:36,646 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:36,646 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:36,646 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:36,646 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:36,646 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:36,646 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:36,647 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:36,647 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:36,647 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:36,647 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:36,647 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:36,647 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:37,075 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/21nqf4vy-media/table/test confusion matrix_table_1013_50025681b919183fa018.table.json
2023-07-04 11:06:37,139 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/eaaotqg6-media/table/batch confusion matrix_table_179_542681021078313e717c.table.json
2023-07-04 11:06:37,235 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,236 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,237 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,398 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:37,402 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,403 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,403 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,403 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,415 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,416 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,416 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,565 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:37,569 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,569 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,570 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,570 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,581 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,582 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,582 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,698 INFO wandb-upload_4:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmptkgvel0r
2023-07-04 11:06:37,741 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:37,745 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,745 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,745 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,746 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,757 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,757 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,758 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,919 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:37,923 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,923 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,923 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,923 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,935 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,936 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:37,936 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,081 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:38,085 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,085 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,085 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,086 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,090 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:38,090 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1045_e6cdd73690272b208c21.table.json
2023-07-04 11:06:38,090 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1017_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:38,090 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1024_542681021078313e717c.table.json
2023-07-04 11:06:38,090 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1031_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:38,091 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1038_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:38,091 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:38,097 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,098 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,098 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,253 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:38,257 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,257 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,257 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,257 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,269 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,269 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,270 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,408 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NDg0', 'digest': '0bc79c2ee78b0b89325bf981d45bdc7d', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NDY2', 'versionIndex': 25}}, 'version': 'latest'}
2023-07-04 11:06:38,409 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:38,409 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:06:38,409 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_186_4c6c1ab939cda8ea4a72.table.json with policy now
2023-07-04 11:06:38,409 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:38,409 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:38,409 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:38,409 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:38,410 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:38,410 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:38,410 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:38,410 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:38,410 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:38,410 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:38,410 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:38,411 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:38,411 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:38,412 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:38,412 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:38,412 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:38,412 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:38,412 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:38,412 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:38,412 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:38,412 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:38,412 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:38,412 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:38,428 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:38,432 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,432 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,432 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,432 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,445 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,445 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,445 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,606 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:38,611 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,611 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,611 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,611 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,623 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,624 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,624 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,772 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:38,775 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,776 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,776 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,776 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,788 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,788 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,788 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,804 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/68i1rpsn-media/table/batch confusion matrix_table_1031_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:38,809 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/2npz4fge-media/table/batch confusion matrix_table_1024_542681021078313e717c.table.json
2023-07-04 11:06:38,815 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/zwjd3nf9-media/table/batch confusion matrix_table_1017_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:38,833 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ph8mlb38-media/table/batch confusion matrix_table_1045_e6cdd73690272b208c21.table.json
2023-07-04 11:06:38,833 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/qopb4g6z-media/table/batch confusion matrix_table_1038_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:38,868 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/c4dzzw1w-media/table/batch confusion matrix_table_186_4c6c1ab939cda8ea4a72.table.json
2023-07-04 11:06:38,931 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:38,934 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,934 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,935 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,935 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,947 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,947 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:38,948 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,088 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:39,091 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,092 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,092 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,092 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,104 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,105 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,105 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,229 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:06:39,267 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:39,271 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,271 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,271 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,271 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,283 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,284 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,284 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,438 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:39,444 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,444 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,444 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,444 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,456 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,456 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,456 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,462 INFO wandb-upload_6:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp468ym3qw
2023-07-04 11:06:39,603 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:39,606 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,607 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,607 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,607 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,619 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,619 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,619 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,776 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:39,780 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,780 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,780 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,781 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,793 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,793 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,793 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,954 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:39,958 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,958 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,958 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,958 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,970 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,971 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:39,971 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,091 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:40,091 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1122_da63951ea7d6deb32ee6.table.json
2023-07-04 11:06:40,091 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1101_1824ba9126a6c7452162.table.json
2023-07-04 11:06:40,091 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1108_1824ba9126a6c7452162.table.json
2023-07-04 11:06:40,091 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1080_1824ba9126a6c7452162.table.json
2023-07-04 11:06:40,092 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1059_1824ba9126a6c7452162.table.json
2023-07-04 11:06:40,092 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1087_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:40,092 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1115_da75045c8d94682c0dec.table.json
2023-07-04 11:06:40,092 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1094_1824ba9126a6c7452162.table.json
2023-07-04 11:06:40,092 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1073_67f7f9c78b661281633a.table.json
2023-07-04 11:06:40,092 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1066_da75045c8d94682c0dec.table.json
2023-07-04 11:06:40,093 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1052_e6cdd73690272b208c21.table.json
2023-07-04 11:06:40,093 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:40,133 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:40,137 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,138 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,138 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,138 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,150 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,150 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,150 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,280 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NTE1', 'digest': '6217e01046411282754cba679cea36dc', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NDg0', 'versionIndex': 26}}, 'version': 'latest'}
2023-07-04 11:06:40,280 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:40,281 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_193_f53c8a8d77ec5f5a4bce.table.json with policy now
2023-07-04 11:06:40,285 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:40,285 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:40,286 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:40,286 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:40,286 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:40,286 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:40,286 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:40,287 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:40,287 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:40,287 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:40,287 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:40,288 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:40,288 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:40,288 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:40,288 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:40,288 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:40,288 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:40,288 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:40,288 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:40,289 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:40,289 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:40,289 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:40,289 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:40,302 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:40,306 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,306 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,306 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,306 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,318 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,319 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,319 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,485 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:40,488 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,488 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,489 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,489 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,500 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,501 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,501 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,664 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:40,667 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,668 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,668 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,668 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,681 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,681 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/6j1ut3nn-media/table/batch confusion matrix_table_1087_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:40,681 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,681 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,695 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/w0sjn4mo-media/table/batch confusion matrix_table_1108_1824ba9126a6c7452162.table.json
2023-07-04 11:06:40,696 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/8168dcro-media/table/batch confusion matrix_table_1101_1824ba9126a6c7452162.table.json
2023-07-04 11:06:40,698 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/acz7ennm-media/table/batch confusion matrix_table_1122_da63951ea7d6deb32ee6.table.json
2023-07-04 11:06:40,700 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/vvqg3ova-media/table/batch confusion matrix_table_1080_1824ba9126a6c7452162.table.json
2023-07-04 11:06:40,701 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/4v2fhgq7-media/table/batch confusion matrix_table_1059_1824ba9126a6c7452162.table.json
2023-07-04 11:06:40,726 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/yugp8tw0-media/table/batch confusion matrix_table_1115_da75045c8d94682c0dec.table.json
2023-07-04 11:06:40,731 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/am7kj5nb-media/table/batch confusion matrix_table_1066_da75045c8d94682c0dec.table.json
2023-07-04 11:06:40,732 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/1pcklrm8-media/table/batch confusion matrix_table_1094_1824ba9126a6c7452162.table.json
2023-07-04 11:06:40,748 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/vk3zo7ly-media/table/batch confusion matrix_table_1052_e6cdd73690272b208c21.table.json
2023-07-04 11:06:40,750 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/7to7o13u-media/table/batch confusion matrix_table_1073_67f7f9c78b661281633a.table.json
2023-07-04 11:06:40,768 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/apkxokym-media/table/batch confusion matrix_table_193_f53c8a8d77ec5f5a4bce.table.json
2023-07-04 11:06:40,827 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:40,831 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,831 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,831 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,831 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,843 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,844 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:40,844 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,005 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:41,010 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,010 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,011 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,011 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,022 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,023 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,023 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,177 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:41,181 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,181 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,181 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,181 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,192 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,192 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,193 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,361 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:41,365 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,365 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,365 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,366 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:41,399 INFO wandb-upload_7:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp7lnnqec6
2023-07-04 11:06:42,092 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:42,092 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1171_d4d30958e51757b86519.table.json
2023-07-04 11:06:42,092 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1150_542681021078313e717c.table.json
2023-07-04 11:06:42,092 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1164_91631cdbd9a35f428cc7.table.json
2023-07-04 11:06:42,092 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1157_790e3abd7e78db96a7a4.table.json
2023-07-04 11:06:42,092 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1129_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:42,092 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1136_67f7f9c78b661281633a.table.json
2023-07-04 11:06:42,093 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1178_42b451464002c81d89cb.table.json
2023-07-04 11:06:42,093 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1143_e6cdd73690272b208c21.table.json
2023-07-04 11:06:42,093 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:42,178 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NTM0', 'digest': 'bbfea6e1c3ac9785e93ad1a602378a3a', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NTE1', 'versionIndex': 27}}, 'version': 'latest'}
2023-07-04 11:06:42,178 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:42,178 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_200_1824ba9126a6c7452162.table.json with policy now
2023-07-04 11:06:42,183 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:42,183 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:42,184 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:42,184 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:42,184 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:42,184 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:42,184 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:42,184 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:42,184 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:42,184 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:42,184 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:42,185 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:42,185 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:42,185 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:42,185 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:42,185 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:42,185 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:42,186 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:42,186 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:42,186 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:42,186 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:42,186 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:42,186 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:42,213 DEBUG SystemMonitor:2443566 [system_monitor.py:_start():159] Starting system metrics aggregation loop
2023-07-04 11:06:42,586 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/uog8d6xf-media/table/batch confusion matrix_table_1178_42b451464002c81d89cb.table.json
2023-07-04 11:06:42,586 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/1aeg3ung-media/table/batch confusion matrix_table_1129_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:42,591 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/3p3xewxs-media/table/batch confusion matrix_table_1171_d4d30958e51757b86519.table.json
2023-07-04 11:06:42,601 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/4f1mmngn-media/table/batch confusion matrix_table_1164_91631cdbd9a35f428cc7.table.json
2023-07-04 11:06:42,602 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/4v9v4izr-media/table/batch confusion matrix_table_1143_e6cdd73690272b208c21.table.json
2023-07-04 11:06:42,603 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/3rgbczm4-media/table/batch confusion matrix_table_1157_790e3abd7e78db96a7a4.table.json
2023-07-04 11:06:42,623 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/w730p3ri-media/table/batch confusion matrix_table_1150_542681021078313e717c.table.json
2023-07-04 11:06:42,646 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/03enbrim-media/table/batch confusion matrix_table_1136_67f7f9c78b661281633a.table.json
2023-07-04 11:06:42,671 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/7x1zb7gh-media/table/batch confusion matrix_table_200_1824ba9126a6c7452162.table.json
2023-07-04 11:06:43,194 INFO wandb-upload_10:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp9jyyw1xh
2023-07-04 11:06:43,971 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NTUz', 'digest': 'a470c23319bd4452f5032ac4f80dc921', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NTM0', 'versionIndex': 28}}, 'version': 'latest'}
2023-07-04 11:06:43,972 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:43,972 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_207_a7873205fa75b58e62cb.table.json with policy now
2023-07-04 11:06:43,972 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:06:43,972 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:43,972 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:43,973 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:43,973 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:43,973 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:43,973 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:43,973 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:43,973 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:43,974 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:43,975 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:43,975 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:43,975 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:43,975 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:43,975 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:43,975 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:43,975 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:43,975 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:43,975 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:43,975 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:43,975 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:43,975 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:43,975 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:43,975 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:44,093 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:44,264 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:06:44,418 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/kbvxerqv-media/table/batch confusion matrix_table_207_a7873205fa75b58e62cb.table.json
2023-07-04 11:06:44,424 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:44,427 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:44,977 INFO wandb-upload_12:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpqwxr81j3
2023-07-04 11:06:45,705 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:45,706 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:45,706 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:45,717 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NTcz', 'digest': 'fa6ef188b0ff928e08d33dac84eac5ef', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NTUz', 'versionIndex': 29}}, 'version': 'latest'}
2023-07-04 11:06:45,717 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:45,718 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_214_542681021078313e717c.table.json with policy now
2023-07-04 11:06:45,718 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:45,718 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:45,718 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:45,718 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:45,718 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:45,719 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:45,719 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:45,719 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:45,719 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:45,719 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:45,719 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:45,719 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:45,720 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:45,720 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:45,721 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:45,721 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:45,721 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:45,721 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:45,721 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:45,721 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:45,721 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:45,722 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:45,722 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:45,864 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:45,907 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:45,908 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:45,908 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:45,908 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:45,909 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:45,909 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:45,909 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,044 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:46,047 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,047 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,048 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,048 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,060 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,060 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,061 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,093 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:46,094 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1193_1824ba9126a6c7452162.table.json
2023-07-04 11:06:46,094 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_1182_779bca52e4f34e45df60.table.json
2023-07-04 11:06:46,094 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1186_542681021078313e717c.table.json
2023-07-04 11:06:46,094 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:46,196 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/12kmgskc-media/table/batch confusion matrix_table_214_542681021078313e717c.table.json
2023-07-04 11:06:46,203 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:46,207 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,207 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,207 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,208 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,220 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,220 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,220 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,374 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:46,378 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,378 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,378 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,379 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,390 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,391 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,391 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,494 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/evy8876v-media/table/batch confusion matrix_table_1193_1824ba9126a6c7452162.table.json
2023-07-04 11:06:46,530 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/6t1ym77h-media/table/test confusion matrix_table_1182_779bca52e4f34e45df60.table.json
2023-07-04 11:06:46,548 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:46,552 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,553 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,553 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,553 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,555 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/dbqch7g9-media/table/batch confusion matrix_table_1186_542681021078313e717c.table.json
2023-07-04 11:06:46,565 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,565 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,566 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,710 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:46,713 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,713 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,713 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,714 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,726 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,726 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,726 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,781 INFO wandb-upload_10:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpql3scjyy
2023-07-04 11:06:46,873 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:46,876 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,877 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,877 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,877 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,888 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,889 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:46,889 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,049 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:47,053 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,053 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,053 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,054 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,065 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,066 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,066 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,221 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:47,224 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,225 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,225 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,225 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,237 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,237 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,237 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,404 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:47,409 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,409 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,410 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,410 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,422 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,422 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,422 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,550 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NTg1', 'digest': 'bbfea6e1c3ac9785e93ad1a602378a3a', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NTcz', 'versionIndex': 30}}, 'version': 'latest'}
2023-07-04 11:06:47,550 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:47,550 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_221_1824ba9126a6c7452162.table.json with policy now
2023-07-04 11:06:47,551 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:47,551 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:47,551 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:47,551 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:47,551 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:47,553 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:47,553 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:47,553 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:47,553 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:47,553 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:47,553 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:47,553 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:47,554 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:47,554 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:47,554 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:47,554 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:47,554 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:47,554 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:47,554 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:47,554 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:47,555 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:47,555 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:47,555 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:47,568 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:47,571 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,571 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,572 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,572 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,584 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,584 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,584 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,745 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:47,749 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,749 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,750 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,750 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,762 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,763 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,763 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,914 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:47,918 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,918 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,918 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,919 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,931 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,932 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:47,932 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,009 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/52pnpxgf-media/table/batch confusion matrix_table_221_1824ba9126a6c7452162.table.json
2023-07-04 11:06:48,094 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:48,095 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1235_e6cdd73690272b208c21.table.json
2023-07-04 11:06:48,095 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1242_1824ba9126a6c7452162.table.json
2023-07-04 11:06:48,095 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1228_e6cdd73690272b208c21.table.json
2023-07-04 11:06:48,095 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1270_e6cdd73690272b208c21.table.json
2023-07-04 11:06:48,095 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1200_67f7f9c78b661281633a.table.json
2023-07-04 11:06:48,095 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1221_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:48,097 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1263_1824ba9126a6c7452162.table.json
2023-07-04 11:06:48,097 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:48,097 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1249_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:48,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1207_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:48,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1214_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:48,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1256_1824ba9126a6c7452162.table.json
2023-07-04 11:06:48,100 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,101 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,101 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,101 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,102 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:48,113 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,113 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,113 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,272 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:48,276 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,277 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,277 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,277 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,288 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,289 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,289 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,446 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:48,450 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,450 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,451 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,451 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,463 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,464 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,464 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,499 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/rowvrozp-media/table/batch confusion matrix_table_1221_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:48,500 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/j7jjqa8v-media/table/batch confusion matrix_table_1270_e6cdd73690272b208c21.table.json
2023-07-04 11:06:48,504 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/lu2waha0-media/table/batch confusion matrix_table_1214_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:48,505 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/0a364ck1-media/table/batch confusion matrix_table_1242_1824ba9126a6c7452162.table.json
2023-07-04 11:06:48,508 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/s203azbx-media/table/batch confusion matrix_table_1200_67f7f9c78b661281633a.table.json
2023-07-04 11:06:48,512 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/f7i74pb2-media/table/batch confusion matrix_table_1263_1824ba9126a6c7452162.table.json
2023-07-04 11:06:48,527 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/7g7di2ir-media/table/batch confusion matrix_table_1249_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:48,540 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/aus4lmnr-media/table/batch confusion matrix_table_1228_e6cdd73690272b208c21.table.json
2023-07-04 11:06:48,544 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/wp2zaoc6-media/table/batch confusion matrix_table_1207_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:48,548 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/polvd1yg-media/table/batch confusion matrix_table_1235_e6cdd73690272b208c21.table.json
2023-07-04 11:06:48,559 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/pv4p109r-media/table/batch confusion matrix_table_1256_1824ba9126a6c7452162.table.json
2023-07-04 11:06:48,624 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:48,628 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,628 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,628 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,629 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,641 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,641 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,641 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,695 INFO wandb-upload_1:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpxscqu73b
2023-07-04 11:06:48,801 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:48,805 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,805 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,805 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,805 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,817 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,817 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,818 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,962 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:48,967 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,967 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,967 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,967 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,979 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,980 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:48,980 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,124 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:49,129 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,129 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,130 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,130 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,141 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,142 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,142 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,292 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:06:49,300 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:49,303 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,304 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,304 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,304 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,316 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,317 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,317 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,462 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:49,466 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,466 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,467 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,467 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,479 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,480 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,481 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,492 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NTk0', 'digest': '04e74fc23e381ac87c5da107b120c37a', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NTg1', 'versionIndex': 31}}, 'version': 'latest'}
2023-07-04 11:06:49,494 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:06:49,640 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:49,665 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:49,665 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_228_fe4098d4c25d56edcebd.table.json with policy now
2023-07-04 11:06:49,665 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:49,665 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:49,665 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:49,665 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:49,666 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:49,666 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:49,666 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:49,666 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:49,666 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:49,666 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:49,666 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:49,666 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:49,666 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:49,666 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:49,666 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:49,666 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:49,666 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:49,666 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:49,666 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:49,667 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:49,667 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:49,667 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:49,667 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:49,687 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,688 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,688 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,688 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,689 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,689 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,689 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,806 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:49,810 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,811 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,811 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:49,811 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:50,095 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:50,096 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/config.yaml
2023-07-04 11:06:50,096 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1347_04e8ade01248395789aa.table.json
2023-07-04 11:06:50,096 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1277_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:50,096 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1326_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:50,096 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1340_1824ba9126a6c7452162.table.json
2023-07-04 11:06:50,096 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1305_1824ba9126a6c7452162.table.json
2023-07-04 11:06:50,096 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1312_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:50,096 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1333_1824ba9126a6c7452162.table.json
2023-07-04 11:06:50,097 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1284_e6cdd73690272b208c21.table.json
2023-07-04 11:06:50,097 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1298_67f7f9c78b661281633a.table.json
2023-07-04 11:06:50,098 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1291_1824ba9126a6c7452162.table.json
2023-07-04 11:06:50,098 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1319_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:50,098 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:50,124 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/fgp7sl7i-media/table/batch confusion matrix_table_228_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:50,494 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/b8uog7v1-media/table/batch confusion matrix_table_1284_e6cdd73690272b208c21.table.json
2023-07-04 11:06:50,505 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/fs0n4qsc-media/table/batch confusion matrix_table_1277_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:50,520 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/uessh9ie-media/table/batch confusion matrix_table_1347_04e8ade01248395789aa.table.json
2023-07-04 11:06:50,522 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/nim7418v-media/table/batch confusion matrix_table_1340_1824ba9126a6c7452162.table.json
2023-07-04 11:06:50,523 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/7cts2485-media/table/batch confusion matrix_table_1298_67f7f9c78b661281633a.table.json
2023-07-04 11:06:50,525 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/8ztwjo5j-media/table/batch confusion matrix_table_1305_1824ba9126a6c7452162.table.json
2023-07-04 11:06:50,525 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/zsh105cn-media/table/batch confusion matrix_table_1326_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:50,575 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/mf1cftnk-media/table/batch confusion matrix_table_1333_1824ba9126a6c7452162.table.json
2023-07-04 11:06:50,575 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/b1u0l7tx-media/table/batch confusion matrix_table_1291_1824ba9126a6c7452162.table.json
2023-07-04 11:06:50,577 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/kw6o8hn4-media/table/batch confusion matrix_table_1312_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:50,578 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ztq3hx3a-media/table/batch confusion matrix_table_1319_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:50,692 INFO wandb-upload_3:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpa624pv1j
2023-07-04 11:06:51,460 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NjE2', 'digest': '0bc79c2ee78b0b89325bf981d45bdc7d', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NTk0', 'versionIndex': 32}}, 'version': 'latest'}
2023-07-04 11:06:51,460 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:51,460 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_235_4c6c1ab939cda8ea4a72.table.json with policy now
2023-07-04 11:06:51,460 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:51,460 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:51,461 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:51,462 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:51,462 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:51,462 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:51,462 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:51,462 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:51,462 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:51,462 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:51,462 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:51,463 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:51,463 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:51,463 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:51,463 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:51,463 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:51,463 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:51,464 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:51,464 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:51,464 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:51,464 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:51,464 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:51,464 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:51,899 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/z9qlr6db-media/table/batch confusion matrix_table_235_4c6c1ab939cda8ea4a72.table.json
2023-07-04 11:06:52,096 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:52,471 INFO wandb-upload_5:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp_7p0r42t
2023-07-04 11:06:52,856 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:52,861 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:53,308 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NjI5', 'digest': 'b2fab797a007dd198cfd78e1c08bac1e', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NjE2', 'versionIndex': 33}}, 'version': 'latest'}
2023-07-04 11:06:53,308 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:53,308 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_242_ac4ad9c89ab9067ca814.table.json with policy now
2023-07-04 11:06:53,308 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:53,308 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:53,309 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:53,309 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:53,309 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:53,309 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:53,309 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:53,310 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:53,310 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:53,310 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:53,310 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:53,310 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:53,310 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:53,310 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:53,311 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:53,311 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:53,311 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:53,311 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:53,311 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:53,311 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:53,311 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:53,311 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:53,311 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:53,764 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ax9o55y5-media/table/batch confusion matrix_table_242_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:06:54,163 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,164 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,164 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,315 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:54,319 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,319 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,320 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,320 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,329 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:06:54,332 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,332 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,333 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,342 INFO wandb-upload_9:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpibi89cwv
2023-07-04 11:06:54,492 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:54,497 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,498 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,498 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,498 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,510 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,510 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,511 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,676 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:54,680 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,680 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,680 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,680 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,693 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,693 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,693 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,853 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:54,857 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,857 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,857 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,858 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,869 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,870 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:54,870 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,016 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:55,020 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,020 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,020 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,021 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,032 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,033 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,033 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,098 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:55,098 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1369_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:55,098 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1355_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:55,098 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1376_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:55,098 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1383_e6cdd73690272b208c21.table.json
2023-07-04 11:06:55,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_1351_779bca52e4f34e45df60.table.json
2023-07-04 11:06:55,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1362_e6cdd73690272b208c21.table.json
2023-07-04 11:06:55,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:55,182 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NjM5', 'digest': 'bf791e3651b8dfa4ebaf7a7be8f85367', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NjI5', 'versionIndex': 34}}, 'version': 'latest'}
2023-07-04 11:06:55,182 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:55,182 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:06:55,182 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_249_ae8fd1dddcf14dc27777.table.json with policy now
2023-07-04 11:06:55,182 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:55,183 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:55,183 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:55,183 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:55,183 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:55,183 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:55,184 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:55,184 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:55,184 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:55,184 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:55,184 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:55,184 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:55,184 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:55,184 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:55,184 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:55,184 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:55,184 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:55,184 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:55,185 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:55,185 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:55,185 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:55,185 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:55,185 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:55,185 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:55,223 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,224 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,224 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,224 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,224 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,224 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,224 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,344 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:55,348 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,348 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,348 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,349 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,360 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,361 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,361 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,511 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:55,514 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,515 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,515 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,515 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,527 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,528 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,528 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,568 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/rfc6xgt0-media/table/batch confusion matrix_table_1369_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:55,569 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/o6wam0ca-media/table/test confusion matrix_table_1351_779bca52e4f34e45df60.table.json
2023-07-04 11:06:55,574 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/94f95ws8-media/table/batch confusion matrix_table_1383_e6cdd73690272b208c21.table.json
2023-07-04 11:06:55,574 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/7aztf6vu-media/table/batch confusion matrix_table_1355_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:55,575 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/45cy4tp9-media/table/batch confusion matrix_table_1362_e6cdd73690272b208c21.table.json
2023-07-04 11:06:55,615 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/2zwzzszi-media/table/batch confusion matrix_table_1376_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:55,624 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/kx05fpv2-media/table/batch confusion matrix_table_249_ae8fd1dddcf14dc27777.table.json
2023-07-04 11:06:55,671 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:55,676 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,676 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,676 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,676 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,688 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,689 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,689 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,859 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:55,862 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,863 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,863 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,863 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,875 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,876 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:55,876 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,036 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:56,039 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,039 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,039 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,040 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,051 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,052 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,052 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,218 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:56,218 INFO wandb-upload_12:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp_f_hx4b8
2023-07-04 11:06:56,222 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,222 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,222 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,223 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,234 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,234 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,235 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,391 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:56,395 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,395 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,395 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,395 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,407 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,407 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,407 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,553 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:56,556 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,557 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,557 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,558 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,569 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,570 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,570 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,716 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:56,719 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,720 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,720 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,720 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,732 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,733 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,733 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,889 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:56,893 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,893 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,894 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,894 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,905 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,905 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:56,906 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,012 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NjY2', 'digest': '4a8245ff2145856179e818c0a39ff309', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NjM5', 'versionIndex': 35}}, 'version': 'latest'}
2023-07-04 11:06:57,012 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:57,012 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_256_e6cdd73690272b208c21.table.json with policy now
2023-07-04 11:06:57,012 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:57,012 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:57,012 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:57,012 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:57,012 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:57,013 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:57,013 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:57,013 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:57,013 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:57,013 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:57,013 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:57,013 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:57,013 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:57,013 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:57,013 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:57,013 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:57,013 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:57,013 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:57,014 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:57,014 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:57,014 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:57,014 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:57,015 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:57,071 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:57,074 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,074 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,075 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,075 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,086 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,087 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,087 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,233 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:57,236 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,236 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,237 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,237 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,249 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,249 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,250 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,405 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:57,409 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,409 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,410 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,410 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,421 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,422 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,422 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,459 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/qck1diye-media/table/batch confusion matrix_table_256_e6cdd73690272b208c21.table.json
2023-07-04 11:06:57,572 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:57,575 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,576 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,576 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,576 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,587 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,588 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,588 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,734 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:57,737 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,738 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,738 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,738 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,750 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,750 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,750 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,898 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:57,901 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,901 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,902 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,902 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,915 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,915 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:57,915 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:58,032 INFO wandb-upload_4:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp9aljmer8
2023-07-04 11:06:58,080 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:58,083 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:58,084 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:58,084 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:58,084 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:58,094 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:58,094 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:58,095 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:58,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:06:58,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1481_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:58,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1460_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:58,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1467_1824ba9126a6c7452162.table.json
2023-07-04 11:06:58,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1397_e6cdd73690272b208c21.table.json
2023-07-04 11:06:58,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1404_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:58,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1432_542681021078313e717c.table.json
2023-07-04 11:06:58,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1502_1824ba9126a6c7452162.table.json
2023-07-04 11:06:58,099 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1425_542681021078313e717c.table.json
2023-07-04 11:06:58,100 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1418_88cf2ee27a4f19ec3b33.table.json
2023-07-04 11:06:58,100 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1439_1824ba9126a6c7452162.table.json
2023-07-04 11:06:58,100 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1509_542681021078313e717c.table.json
2023-07-04 11:06:58,100 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1495_da75045c8d94682c0dec.table.json
2023-07-04 11:06:58,100 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1488_5fa4ee3184f5f47c930f.table.json
2023-07-04 11:06:58,100 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1390_67f7f9c78b661281633a.table.json
2023-07-04 11:06:58,100 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1474_e6cdd73690272b208c21.table.json
2023-07-04 11:06:58,100 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1411_da75045c8d94682c0dec.table.json
2023-07-04 11:06:58,100 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1453_1824ba9126a6c7452162.table.json
2023-07-04 11:06:58,100 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1446_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:58,100 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:06:58,240 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:06:58,244 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:58,244 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:58,244 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:58,244 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:06:58,789 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2Njkz', 'digest': 'bfa65c69bf202040df9d3d570a90cc60', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NjY2', 'versionIndex': 36}}, 'version': 'latest'}
2023-07-04 11:06:58,789 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:06:58,789 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_263_e04570dcb762a9eb0c10.table.json with policy now
2023-07-04 11:06:58,798 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:58,798 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:58,799 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:58,799 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:58,800 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:58,802 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:58,802 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:58,802 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:58,802 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:58,803 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:58,803 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:58,803 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:58,803 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:58,803 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:58,803 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:58,803 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:58,803 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:58,803 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:58,803 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:06:58,803 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:06:58,804 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:06:58,804 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:06:58,804 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:06:59,184 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/rdks5ect-media/table/batch confusion matrix_table_1418_88cf2ee27a4f19ec3b33.table.json
2023-07-04 11:06:59,188 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/tthpxvfl-media/table/batch confusion matrix_table_1481_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:59,199 INFO wandb-upload_17:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/r583cakz-media/table/batch confusion matrix_table_1425_542681021078313e717c.table.json
2023-07-04 11:06:59,204 INFO wandb-upload_16:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/n6tsfawz-media/table/batch confusion matrix_table_1432_542681021078313e717c.table.json
2023-07-04 11:06:59,206 INFO wandb-upload_15:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/e65tn9cf-media/table/batch confusion matrix_table_1404_fe4098d4c25d56edcebd.table.json
2023-07-04 11:06:59,207 INFO wandb-upload_14:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/5zcfww6x-media/table/batch confusion matrix_table_1467_1824ba9126a6c7452162.table.json
2023-07-04 11:06:59,212 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/siroh7vx-media/table/batch confusion matrix_table_1439_1824ba9126a6c7452162.table.json
2023-07-04 11:06:59,219 INFO wandb-upload_13:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/cd54l352-media/table/batch confusion matrix_table_1460_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:59,239 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/i3yeuly1-media/table/batch confusion matrix_table_1397_e6cdd73690272b208c21.table.json
2023-07-04 11:06:59,252 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/hc46m7sn-media/table/batch confusion matrix_table_1488_5fa4ee3184f5f47c930f.table.json
2023-07-04 11:06:59,294 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/yfdu5i26-media/table/batch confusion matrix_table_1390_67f7f9c78b661281633a.table.json
2023-07-04 11:06:59,296 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/edidvmhu-media/table/batch confusion matrix_table_1453_1824ba9126a6c7452162.table.json
2023-07-04 11:06:59,297 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/2chz7xvp-media/table/batch confusion matrix_table_1502_1824ba9126a6c7452162.table.json
2023-07-04 11:06:59,298 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/brnk3os6-media/table/batch confusion matrix_table_1411_da75045c8d94682c0dec.table.json
2023-07-04 11:06:59,299 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/uyiqr2dx-media/table/batch confusion matrix_table_1446_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:59,299 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/p8nh1d4n-media/table/batch confusion matrix_table_1495_da75045c8d94682c0dec.table.json
2023-07-04 11:06:59,299 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/9lisuqy7-media/table/batch confusion matrix_table_1509_542681021078313e717c.table.json
2023-07-04 11:06:59,344 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:06:59,348 INFO wandb-upload_18:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/rs19z9r7-media/table/batch confusion matrix_table_263_e04570dcb762a9eb0c10.table.json
2023-07-04 11:06:59,349 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/814tt8w3-media/table/batch confusion matrix_table_1474_e6cdd73690272b208c21.table.json
2023-07-04 11:06:59,899 INFO wandb-upload_8:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp9ja5icds
2023-07-04 11:07:00,706 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NzA4', 'digest': '4a8245ff2145856179e818c0a39ff309', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2Njkz', 'versionIndex': 37}}, 'version': 'latest'}
2023-07-04 11:07:00,706 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:00,706 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_270_e6cdd73690272b208c21.table.json with policy now
2023-07-04 11:07:00,706 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:00,706 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:07:00,706 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:00,707 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:00,707 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:00,707 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:00,708 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:00,708 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:00,708 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:00,708 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:00,708 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:00,708 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:00,708 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:00,709 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:00,709 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:00,710 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:00,710 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:00,710 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:00,710 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:00,710 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:00,710 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:00,710 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:00,710 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:00,710 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:01,100 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:01,100 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1516_42b451464002c81d89cb.table.json
2023-07-04 11:07:01,100 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:01,168 INFO wandb-upload_17:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/2jm62e6n-media/table/batch confusion matrix_table_270_e6cdd73690272b208c21.table.json
2023-07-04 11:07:01,467 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:01,472 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:01,496 INFO wandb-upload_15:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ennffxtt-media/table/batch confusion matrix_table_1516_42b451464002c81d89cb.table.json
2023-07-04 11:07:01,758 INFO wandb-upload_11:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpmjee371e
2023-07-04 11:07:02,614 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NzMx', 'digest': 'e233af727ab1eeacb0b94ee496beb333', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NzA4', 'versionIndex': 38}}, 'version': 'latest'}
2023-07-04 11:07:02,614 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:02,614 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_277_d4d30958e51757b86519.table.json with policy now
2023-07-04 11:07:02,614 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:02,615 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:02,615 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:02,615 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:02,615 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:02,615 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:02,615 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:02,615 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:02,616 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:02,616 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:02,616 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:02,617 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:02,617 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:02,617 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:02,617 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:02,617 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:02,617 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:02,617 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:02,617 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:02,617 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:02,617 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:02,617 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:02,617 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:02,726 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:02,727 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:02,727 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:02,870 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:02,874 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:02,874 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:02,874 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:02,874 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:02,887 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:02,887 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:02,887 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,032 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:03,035 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,036 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,036 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,037 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,048 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,049 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,049 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,059 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/qqw29nt5-media/table/batch confusion matrix_table_277_d4d30958e51757b86519.table.json
2023-07-04 11:07:03,194 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:03,197 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,198 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,198 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,200 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,210 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,211 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,211 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,370 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:03,373 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,374 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,374 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,374 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,386 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,387 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,387 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,547 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:03,551 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,551 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,551 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,551 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,564 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,564 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,564 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,662 INFO wandb-upload_12:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpvwgqskpb
2023-07-04 11:07:03,710 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:03,714 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,715 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,715 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,715 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,727 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,728 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,728 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,873 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:03,877 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,877 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,877 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,878 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,889 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,890 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:03,890 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,030 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:04,033 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,034 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,034 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,034 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,046 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,047 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,047 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,101 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:04,101 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1545_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:04,101 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1524_542681021078313e717c.table.json
2023-07-04 11:07:04,101 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1559_1824ba9126a6c7452162.table.json
2023-07-04 11:07:04,101 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1566_542681021078313e717c.table.json
2023-07-04 11:07:04,101 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1538_e6cdd73690272b208c21.table.json
2023-07-04 11:07:04,101 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1531_e6cdd73690272b208c21.table.json
2023-07-04 11:07:04,101 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_1520_779bca52e4f34e45df60.table.json
2023-07-04 11:07:04,101 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1573_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:04,101 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1552_83a53314926c797ba154.table.json
2023-07-04 11:07:04,102 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:04,189 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:04,193 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,193 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,194 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,194 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,206 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,207 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,207 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,364 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:04,367 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:07:04,368 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,368 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,368 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,369 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,380 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,381 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,381 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,488 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2NzU5', 'digest': 'cff6c7d6e76288f9670860e1252bd336', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NzMx', 'versionIndex': 39}}, 'version': 'latest'}
2023-07-04 11:07:04,489 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:04,489 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_284_83a53314926c797ba154.table.json with policy now
2023-07-04 11:07:04,489 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:04,489 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:04,491 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:04,491 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:04,492 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:04,493 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:04,493 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:04,493 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:04,496 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:04,498 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:04,498 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:04,499 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:04,499 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:04,499 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:04,499 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:04,499 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:04,499 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:04,499 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:04,499 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:04,499 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:04,499 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:04,500 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:04,500 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:04,537 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:04,540 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,540 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,541 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,541 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,553 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,553 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,553 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,713 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:04,717 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,717 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,717 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,717 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,729 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,730 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,730 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,887 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/lljblw0o-media/table/batch confusion matrix_table_1524_542681021078313e717c.table.json
2023-07-04 11:07:04,890 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:04,893 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,893 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,893 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,894 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,894 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/t98zy1m3-media/table/batch confusion matrix_table_1538_e6cdd73690272b208c21.table.json
2023-07-04 11:07:04,897 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/hwzsv8u4-media/table/batch confusion matrix_table_1566_542681021078313e717c.table.json
2023-07-04 11:07:04,900 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ptd28d20-media/table/batch confusion matrix_table_1552_83a53314926c797ba154.table.json
2023-07-04 11:07:04,904 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/km710g86-media/table/batch confusion matrix_table_1545_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:04,906 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,906 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,906 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:04,912 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/01hinhmq-media/table/batch confusion matrix_table_1531_e6cdd73690272b208c21.table.json
2023-07-04 11:07:04,923 INFO wandb-upload_18:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/46kim7lg-media/table/test confusion matrix_table_1520_779bca52e4f34e45df60.table.json
2023-07-04 11:07:04,924 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/usf2rhjn-media/table/batch confusion matrix_table_1573_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:04,934 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/t82zi9an-media/table/batch confusion matrix_table_1559_1824ba9126a6c7452162.table.json
2023-07-04 11:07:05,023 INFO wandb-upload_14:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/awfcth1m-media/table/batch confusion matrix_table_284_83a53314926c797ba154.table.json
2023-07-04 11:07:05,054 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:05,057 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,058 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,058 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,058 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,070 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,070 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,071 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,217 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:05,220 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,220 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,221 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,221 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,232 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,233 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,233 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,397 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:05,401 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,402 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,402 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,402 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,414 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,415 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,415 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,560 INFO wandb-upload_13:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp3duoftk6
2023-07-04 11:07:05,571 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:05,575 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,575 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,575 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,576 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,587 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,588 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,588 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,761 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:05,765 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,765 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,766 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,766 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,778 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,778 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,779 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,927 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:05,932 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,932 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,933 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,933 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,945 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,946 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:05,946 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,102 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:06,105 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,105 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,106 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,106 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,117 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,118 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,118 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,262 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:06,265 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,266 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,266 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,266 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,279 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,279 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,279 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,326 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2Nzc3', 'digest': 'ffa5b164478c4a2f0b7c6a5778dc8a7d', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NzU5', 'versionIndex': 40}}, 'version': 'latest'}
2023-07-04 11:07:06,326 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:06,326 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_291_a115e530add99a97b366.table.json with policy now
2023-07-04 11:07:06,326 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:06,326 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:06,327 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:07:06,327 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:06,327 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:06,327 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:06,327 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:06,328 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:06,328 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:06,329 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:06,329 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:06,329 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:06,329 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:06,329 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:06,330 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:06,330 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:06,330 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:06,330 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:06,330 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:06,330 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:06,330 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:06,330 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:06,330 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:06,330 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:06,423 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:06,426 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,427 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,427 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,427 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,439 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,439 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,440 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,606 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:06,610 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,611 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,611 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,611 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,622 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,622 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,622 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,761 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/epbox9rw-media/table/batch confusion matrix_table_291_a115e530add99a97b366.table.json
2023-07-04 11:07:06,769 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:06,772 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,773 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,773 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:06,773 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:07,102 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:07,102 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1580_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:07,102 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1622_1824ba9126a6c7452162.table.json
2023-07-04 11:07:07,102 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1587_e6cdd73690272b208c21.table.json
2023-07-04 11:07:07,102 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1615_1824ba9126a6c7452162.table.json
2023-07-04 11:07:07,102 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1664_1824ba9126a6c7452162.table.json
2023-07-04 11:07:07,102 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1629_1824ba9126a6c7452162.table.json
2023-07-04 11:07:07,102 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1657_67f7f9c78b661281633a.table.json
2023-07-04 11:07:07,102 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1671_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:07,103 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1678_e6cdd73690272b208c21.table.json
2023-07-04 11:07:07,103 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1685_17ac89d9b34a12ddbdd5.table.json
2023-07-04 11:07:07,103 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1594_1824ba9126a6c7452162.table.json
2023-07-04 11:07:07,103 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1650_542681021078313e717c.table.json
2023-07-04 11:07:07,103 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1601_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:07,103 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1643_1824ba9126a6c7452162.table.json
2023-07-04 11:07:07,103 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1608_1824ba9126a6c7452162.table.json
2023-07-04 11:07:07,103 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1636_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:07,103 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:07,358 INFO wandb-upload_3:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpjylpdslx
2023-07-04 11:07:07,496 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/q1m4vgbe-media/table/batch confusion matrix_table_1629_1824ba9126a6c7452162.table.json
2023-07-04 11:07:07,498 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/j0te9rpg-media/table/batch confusion matrix_table_1664_1824ba9126a6c7452162.table.json
2023-07-04 11:07:07,500 INFO wandb-upload_15:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/7kqrkmov-media/table/batch confusion matrix_table_1657_67f7f9c78b661281633a.table.json
2023-07-04 11:07:07,504 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/5g9kfkc7-media/table/batch confusion matrix_table_1678_e6cdd73690272b208c21.table.json
2023-07-04 11:07:07,510 INFO wandb-upload_13:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/wf3j46lm-media/table/batch confusion matrix_table_1671_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:07,511 INFO wandb-upload_17:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ju9wsf9o-media/table/batch confusion matrix_table_1622_1824ba9126a6c7452162.table.json
2023-07-04 11:07:07,528 INFO wandb-upload_16:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/jznmzxl6-media/table/batch confusion matrix_table_1685_17ac89d9b34a12ddbdd5.table.json
2023-07-04 11:07:07,538 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/dktsi8hc-media/table/batch confusion matrix_table_1580_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:07,539 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/il8hv5l6-media/table/batch confusion matrix_table_1587_e6cdd73690272b208c21.table.json
2023-07-04 11:07:07,549 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/myxjra03-media/table/batch confusion matrix_table_1615_1824ba9126a6c7452162.table.json
2023-07-04 11:07:07,557 INFO wandb-upload_18:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/fg19rw4c-media/table/batch confusion matrix_table_1650_542681021078313e717c.table.json
2023-07-04 11:07:07,564 INFO wandb-upload_14:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/sk98lhkj-media/table/batch confusion matrix_table_1601_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:07,568 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/vfhot6od-media/table/batch confusion matrix_table_1636_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:07,570 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/50rukfrq-media/table/batch confusion matrix_table_1643_1824ba9126a6c7452162.table.json
2023-07-04 11:07:07,571 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/tgmlc8yx-media/table/batch confusion matrix_table_1594_1824ba9126a6c7452162.table.json
2023-07-04 11:07:07,602 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/vkw3cjw8-media/table/batch confusion matrix_table_1608_1824ba9126a6c7452162.table.json
2023-07-04 11:07:08,191 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2Nzkw', 'digest': 'bbfea6e1c3ac9785e93ad1a602378a3a', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2Nzc3', 'versionIndex': 41}}, 'version': 'latest'}
2023-07-04 11:07:08,191 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:08,191 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_298_1824ba9126a6c7452162.table.json with policy now
2023-07-04 11:07:08,191 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:08,192 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:08,192 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:08,192 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:08,192 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:08,193 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:08,193 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:08,193 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:08,193 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:08,193 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:08,193 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:08,194 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:08,194 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:08,194 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:08,195 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:08,195 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:08,195 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:08,195 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:08,195 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:08,195 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:08,195 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:08,195 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:08,195 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:09,223 INFO wandb-upload_3:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp_56np37p
2023-07-04 11:07:09,225 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/4yrvytni-media/table/batch confusion matrix_table_298_1824ba9126a6c7452162.table.json
2023-07-04 11:07:09,399 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:07:09,897 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:09,904 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:10,027 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2ODEw', 'digest': 'c191bebe3fdfeda6da9d99c78921a0a6', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2Nzkw', 'versionIndex': 42}}, 'version': 'latest'}
2023-07-04 11:07:10,028 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:10,028 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_305_5dcc195a388d9733922f.table.json with policy now
2023-07-04 11:07:10,028 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:10,028 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:10,028 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:10,029 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:10,029 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:10,029 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:10,029 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:10,029 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:10,029 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:10,029 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:10,029 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:10,029 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:10,029 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:10,030 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:10,030 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:10,030 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:10,030 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:10,030 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:10,030 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:10,030 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:10,030 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:10,030 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:10,030 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:10,103 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:10,103 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_1689_305e3095c153af2320a3.table.json
2023-07-04 11:07:10,103 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:10,484 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/am2ynpv3-media/table/batch confusion matrix_table_305_5dcc195a388d9733922f.table.json
2023-07-04 11:07:10,576 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/25161ztk-media/table/test confusion matrix_table_1689_305e3095c153af2320a3.table.json
2023-07-04 11:07:11,124 INFO wandb-upload_17:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp0l7wjp_k
2023-07-04 11:07:11,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,185 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,185 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,354 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:11,358 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,358 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,358 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,358 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,371 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,371 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,371 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,529 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:11,532 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,532 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,533 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,533 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,545 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,545 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,545 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,715 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:11,718 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,719 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,719 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,719 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,731 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,731 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,732 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,891 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:11,894 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,894 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,895 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,895 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,907 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,908 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,908 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:11,909 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2ODI3', 'digest': 'bbfea6e1c3ac9785e93ad1a602378a3a', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2ODEw', 'versionIndex': 43}}, 'version': 'latest'}
2023-07-04 11:07:11,909 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:11,909 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_312_1824ba9126a6c7452162.table.json with policy now
2023-07-04 11:07:11,909 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:07:11,909 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:11,909 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:11,910 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:11,910 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:11,910 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:11,911 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:11,911 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:11,911 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:11,911 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:11,911 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:11,911 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:11,911 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:11,911 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:11,911 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:11,911 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:11,911 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:11,911 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:11,912 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:11,912 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:11,912 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:11,912 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:11,912 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:11,912 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:12,053 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:12,095 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,096 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,096 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,096 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,096 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,096 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,096 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,218 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:12,222 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,222 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,223 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,223 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,234 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,235 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,235 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,392 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:12,396 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,396 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,397 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,397 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,404 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/c7l1zi3u-media/table/batch confusion matrix_table_312_1824ba9126a6c7452162.table.json
2023-07-04 11:07:12,409 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,410 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,410 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,554 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:12,558 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,558 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,558 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,558 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,571 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,571 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,571 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,765 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:12,769 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,769 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,769 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,770 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,782 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,782 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,783 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,931 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:12,935 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,935 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,935 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,936 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,947 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,948 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:12,948 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,001 INFO wandb-upload_8:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpqg_fw956
2023-07-04 11:07:13,104 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:13,104 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1742_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:13,104 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1707_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:13,105 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1756_1824ba9126a6c7452162.table.json
2023-07-04 11:07:13,105 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1714_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:13,105 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1749_542681021078313e717c.table.json
2023-07-04 11:07:13,105 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1693_542681021078313e717c.table.json
2023-07-04 11:07:13,105 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1721_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:13,105 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1700_e6cdd73690272b208c21.table.json
2023-07-04 11:07:13,105 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1735_e6cdd73690272b208c21.table.json
2023-07-04 11:07:13,105 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1728_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:13,105 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:13,108 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:13,112 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,112 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,112 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,112 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,124 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,125 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,125 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,281 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:13,285 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,285 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,286 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,286 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,298 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,298 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,298 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,448 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:13,453 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,454 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,454 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,454 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,466 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,467 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,467 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,616 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:13,620 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,620 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,620 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,620 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,633 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,633 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,633 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,791 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2ODQy', 'digest': 'caae1d0b723602673990e5477f4f94ba', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2ODI3', 'versionIndex': 44}}, 'version': 'latest'}
2023-07-04 11:07:13,791 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:13,791 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_319_f4f42398d019499f73ea.table.json with policy now
2023-07-04 11:07:13,791 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:13,791 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:13,792 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:13,793 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:13,793 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:13,796 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:13,796 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:13,798 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:13,798 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,798 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:13,799 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:13,799 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:13,799 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:13,799 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:13,799 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:13,799 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:13,799 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:13,800 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:13,800 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:13,800 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:13,800 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:13,800 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:13,800 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:13,800 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:13,800 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:13,839 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,839 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,840 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,840 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,840 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,840 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,953 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:13,957 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,957 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,958 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,958 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,969 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,970 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:13,970 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,127 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:14,131 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,131 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,131 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,131 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,144 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,144 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,145 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,181 INFO wandb-upload_14:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/f758r3hv-media/table/batch confusion matrix_table_1742_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:14,186 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/b1n3kjx1-media/table/batch confusion matrix_table_1693_542681021078313e717c.table.json
2023-07-04 11:07:14,191 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/1ddowhfk-media/table/batch confusion matrix_table_1721_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:14,196 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/8gzycxuy-media/table/batch confusion matrix_table_1707_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:14,199 INFO wandb-upload_15:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/1vyah997-media/table/batch confusion matrix_table_1728_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:14,209 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/zva91byo-media/table/batch confusion matrix_table_1714_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:14,210 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/d6z9o30x-media/table/batch confusion matrix_table_1735_e6cdd73690272b208c21.table.json
2023-07-04 11:07:14,221 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/mjhvvcqi-media/table/batch confusion matrix_table_1749_542681021078313e717c.table.json
2023-07-04 11:07:14,242 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/hi8gplzq-media/table/batch confusion matrix_table_1756_1824ba9126a6c7452162.table.json
2023-07-04 11:07:14,255 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/3vkn4d4e-media/table/batch confusion matrix_table_1700_e6cdd73690272b208c21.table.json
2023-07-04 11:07:14,284 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/3fymnkn4-media/table/batch confusion matrix_table_319_f4f42398d019499f73ea.table.json
2023-07-04 11:07:14,284 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:14,287 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,288 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,288 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,288 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,300 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,300 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,300 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,410 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:07:14,462 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:14,467 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,467 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,468 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,468 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,481 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,481 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,482 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,642 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:14,645 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,646 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,646 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,647 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,658 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,659 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,659 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,801 INFO wandb-upload_11:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpf6vflxoz
2023-07-04 11:07:14,822 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:14,825 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,826 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,826 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,826 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,838 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,838 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,838 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,986 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:14,990 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,990 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,990 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:14,991 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,002 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,002 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,002 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,164 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:15,168 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,169 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,169 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,169 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,180 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,180 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,180 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,323 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:15,326 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,326 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,326 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,326 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:15,643 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2ODY3', 'digest': 'bfa65c69bf202040df9d3d570a90cc60', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2ODQy', 'versionIndex': 45}}, 'version': 'latest'}
2023-07-04 11:07:15,643 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:15,643 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_326_e04570dcb762a9eb0c10.table.json with policy now
2023-07-04 11:07:15,644 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:15,644 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:15,644 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:15,644 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:15,644 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:15,645 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:15,645 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:15,645 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:15,645 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:15,645 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:15,645 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:15,645 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:15,645 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:15,645 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:15,646 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:15,646 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:15,646 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:15,646 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:15,646 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:15,646 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:15,647 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:15,647 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:15,648 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:16,105 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1840_e6cdd73690272b208c21.table.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1854_04e8ade01248395789aa.table.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1819_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1784_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1826_1824ba9126a6c7452162.table.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1763_1824ba9126a6c7452162.table.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1847_67f7f9c78b661281633a.table.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1791_1824ba9126a6c7452162.table.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1812_542681021078313e717c.table.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1833_67f7f9c78b661281633a.table.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1798_e6cdd73690272b208c21.table.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1805_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1770_790e3abd7e78db96a7a4.table.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1777_da75045c8d94682c0dec.table.json
2023-07-04 11:07:16,106 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:16,118 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/4eyzxour-media/table/batch confusion matrix_table_326_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:16,508 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/1fsi92ls-media/table/batch confusion matrix_table_1847_67f7f9c78b661281633a.table.json
2023-07-04 11:07:16,508 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/g7lvxsnq-media/table/batch confusion matrix_table_1784_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:16,513 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/np72w7aq-media/table/batch confusion matrix_table_1763_1824ba9126a6c7452162.table.json
2023-07-04 11:07:16,520 INFO wandb-upload_17:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/mgcoqeez-media/table/batch confusion matrix_table_1791_1824ba9126a6c7452162.table.json
2023-07-04 11:07:16,522 INFO wandb-upload_14:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/seqeqd6x-media/table/batch confusion matrix_table_1819_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:16,526 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/3szvwuxy-media/table/batch confusion matrix_table_1798_e6cdd73690272b208c21.table.json
2023-07-04 11:07:16,551 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/iii5vv17-media/table/batch confusion matrix_table_1777_da75045c8d94682c0dec.table.json
2023-07-04 11:07:16,556 INFO wandb-upload_13:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/kiulyb5j-media/table/batch confusion matrix_table_1854_04e8ade01248395789aa.table.json
2023-07-04 11:07:16,556 INFO wandb-upload_18:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/607f2vjq-media/table/batch confusion matrix_table_1840_e6cdd73690272b208c21.table.json
2023-07-04 11:07:16,560 INFO wandb-upload_15:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/l2mby0qx-media/table/batch confusion matrix_table_1826_1824ba9126a6c7452162.table.json
2023-07-04 11:07:16,568 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/09u54o6k-media/table/batch confusion matrix_table_1833_67f7f9c78b661281633a.table.json
2023-07-04 11:07:16,569 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/nehz4mi7-media/table/batch confusion matrix_table_1805_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:16,575 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/k9s599e3-media/table/batch confusion matrix_table_1812_542681021078313e717c.table.json
2023-07-04 11:07:16,581 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/duf3o65e-media/table/batch confusion matrix_table_1770_790e3abd7e78db96a7a4.table.json
2023-07-04 11:07:16,724 INFO wandb-upload_7:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpzyziz3ls
2023-07-04 11:07:17,507 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2ODgz', 'digest': 'a35b352c36cd76a2f238a76914c98534', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2ODY3', 'versionIndex': 46}}, 'version': 'latest'}
2023-07-04 11:07:17,508 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:17,508 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_333_42b451464002c81d89cb.table.json with policy now
2023-07-04 11:07:17,508 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:07:17,508 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:17,508 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:17,509 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:17,509 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:17,509 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:17,509 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:17,509 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:17,509 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:17,510 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:17,510 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:17,510 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:17,510 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:17,510 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: stop_status
2023-07-04 11:07:17,624 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:17,624 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:17,963 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/6zwhme29-media/table/batch confusion matrix_table_333_42b451464002c81d89cb.table.json
2023-07-04 11:07:18,475 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:18,479 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:18,843 INFO wandb-upload_16:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpk2bnm9zd
2023-07-04 11:07:19,107 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:19,107 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/output.log
2023-07-04 11:07:19,107 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_1858_9e0e577472816ad5a224.table.json
2023-07-04 11:07:19,107 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:19,431 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:07:19,651 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-testconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2ODk4', 'digest': 'bc4d9d4f4ab66cc42947f4c6849ca5f4', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMTE1', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2NDM0', 'versionIndex': 0}}, 'version': 'latest'}
2023-07-04 11:07:19,651 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:19,651 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/test confusion matrix_table_337_103b671e2f0c5c11418a.table.json with policy now
2023-07-04 11:07:19,651 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:19,651 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:19,651 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:19,651 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:19,652 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:19,652 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:19,652 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:19,652 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:19,652 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:19,652 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:19,652 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:19,652 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:19,652 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:19,652 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:19,750 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:19,751 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:19,751 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:19,915 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:19,919 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:19,919 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:19,920 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:19,920 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:19,932 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:19,932 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:19,932 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,075 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:20,078 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,079 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,079 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,079 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,091 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/zmsaxtsm-media/table/test confusion matrix_table_1858_9e0e577472816ad5a224.table.json
2023-07-04 11:07:20,091 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,092 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,092 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,118 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/15bkudmi-media/table/test confusion matrix_table_337_103b671e2f0c5c11418a.table.json
2023-07-04 11:07:20,256 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:20,260 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,261 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,261 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,261 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,273 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,273 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,273 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,434 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:20,438 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,439 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,439 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,439 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,451 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,451 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,451 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,609 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:20,613 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,613 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,613 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,614 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,626 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,626 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,626 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,708 INFO wandb-upload_14:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmprk6l7lgt
2023-07-04 11:07:20,785 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:20,789 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,790 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,790 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,790 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,802 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,802 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,802 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,958 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:20,961 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,961 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,961 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,962 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,973 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,974 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:20,974 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,134 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:21,137 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,138 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,138 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,138 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,149 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,150 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,150 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,297 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:21,300 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,300 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,301 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,301 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,313 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,313 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,313 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,462 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:21,466 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,467 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,467 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,467 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,479 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,480 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,480 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,498 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2OTI3', 'digest': 'bbfea6e1c3ac9785e93ad1a602378a3a', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2ODgz', 'versionIndex': 47}}, 'version': 'latest'}
2023-07-04 11:07:21,498 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:21,498 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_341_1824ba9126a6c7452162.table.json with policy now
2023-07-04 11:07:21,498 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:21,498 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:21,499 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:21,499 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:21,499 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:21,499 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:21,500 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:21,500 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:21,501 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:21,501 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:21,501 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:21,501 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:21,501 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:21,501 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:21,501 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:21,501 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:21,501 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:21,502 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:21,502 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:21,502 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:21,502 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:21,502 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:21,502 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:21,638 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:21,679 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,680 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,680 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,680 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,680 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,680 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,680 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,796 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2OTI3', 'digest': 'bbfea6e1c3ac9785e93ad1a602378a3a', 'state': 'COMMITTED', 'aliases': [{'artifactCollectionName': 'run-w53a3qn5-batchconfusionmatrix_table', 'alias': 'latest'}, {'artifactCollectionName': 'run-w53a3qn5-batchconfusionmatrix_table', 'alias': 'v48'}], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2OTI3', 'versionIndex': 48}}, 'version': 'v48'}
2023-07-04 11:07:21,796 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:21,796 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_348_1824ba9126a6c7452162.table.json with policy now
2023-07-04 11:07:21,796 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:21,796 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:21,797 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:21,797 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:21,797 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:21,797 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:21,797 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:21,797 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:21,797 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:21,797 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:21,797 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:21,797 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:21,798 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:21,798 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:21,798 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:21,798 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:21,798 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:21,798 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:21,798 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:21,798 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:21,798 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:21,798 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:21,798 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:21,817 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:21,820 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,820 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,821 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,821 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,833 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,833 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,833 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,931 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/k3zmv91y-media/table/batch confusion matrix_table_341_1824ba9126a6c7452162.table.json
2023-07-04 11:07:21,975 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:21,979 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,980 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,980 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,980 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,992 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,993 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:21,993 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,108 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:22,108 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/output.log
2023-07-04 11:07:22,108 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1939_1824ba9126a6c7452162.table.json
2023-07-04 11:07:22,108 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1883_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:22,108 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1897_67f7f9c78b661281633a.table.json
2023-07-04 11:07:22,109 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1918_542681021078313e717c.table.json
2023-07-04 11:07:22,109 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1946_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:22,109 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1869_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:22,109 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1932_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:22,109 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1862_1824ba9126a6c7452162.table.json
2023-07-04 11:07:22,109 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1911_da75045c8d94682c0dec.table.json
2023-07-04 11:07:22,109 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1876_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:22,109 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1904_e6cdd73690272b208c21.table.json
2023-07-04 11:07:22,110 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1925_67f7f9c78b661281633a.table.json
2023-07-04 11:07:22,110 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1890_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:22,111 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:22,151 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:22,155 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,155 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,155 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,155 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,167 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,167 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,168 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,242 INFO wandb-upload_18:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/po103mxs-media/table/batch confusion matrix_table_348_1824ba9126a6c7452162.table.json
2023-07-04 11:07:22,327 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:22,330 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,330 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,331 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,331 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,343 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,344 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,344 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,506 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/37jdax1z-media/table/batch confusion matrix_table_1939_1824ba9126a6c7452162.table.json
2023-07-04 11:07:22,511 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:22,514 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,514 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,515 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,515 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,523 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/f2dkaldo-media/table/batch confusion matrix_table_1883_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:22,525 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/u4cc6sdk-media/table/batch confusion matrix_table_1918_542681021078313e717c.table.json
2023-07-04 11:07:22,526 INFO wandb-upload_14:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/pwyzp68p-media/table/batch confusion matrix_table_1862_1824ba9126a6c7452162.table.json
2023-07-04 11:07:22,527 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,528 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,528 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,530 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/csr5e10t-media/table/batch confusion matrix_table_1932_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:22,546 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/6xsuqefi-media/table/batch confusion matrix_table_1925_67f7f9c78b661281633a.table.json
2023-07-04 11:07:22,547 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/jhyr4f3z-media/table/batch confusion matrix_table_1897_67f7f9c78b661281633a.table.json
2023-07-04 11:07:22,549 INFO wandb-upload_17:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/yczx0zbt-media/table/batch confusion matrix_table_1869_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:22,558 INFO wandb-upload_15:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/iqwp3w19-media/table/batch confusion matrix_table_1911_da75045c8d94682c0dec.table.json
2023-07-04 11:07:22,561 INFO wandb-upload_16:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/wuz7qi8r-media/table/batch confusion matrix_table_1946_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:22,566 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/nyxcsdj3-media/table/batch confusion matrix_table_1876_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:22,566 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/wx2mpzj9-media/table/batch confusion matrix_table_1904_e6cdd73690272b208c21.table.json
2023-07-04 11:07:22,586 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/if2p3vp0-media/table/batch confusion matrix_table_1890_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:22,682 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:22,686 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,686 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,687 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,687 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,698 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,699 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,699 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,847 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:22,851 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,851 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,851 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,852 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,854 INFO wandb-upload_12:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp57pgyvft
2023-07-04 11:07:22,863 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,864 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:22,864 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,009 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:23,012 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,013 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,013 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,014 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,026 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,026 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,026 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,170 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:23,175 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,175 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,175 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,175 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,187 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,187 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,188 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,335 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:23,338 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,339 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,339 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,339 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,351 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,351 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,352 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,509 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:23,512 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,513 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,513 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,513 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,525 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,526 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,526 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,696 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:23,699 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,699 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,699 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,700 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,711 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,711 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,711 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,875 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2OTM5', 'digest': '796a02317e60240fc91e6d8151207462', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2OTI3', 'versionIndex': 48}}, 'version': 'latest'}
2023-07-04 11:07:23,877 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:07:23,908 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:23,912 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,912 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,912 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:23,913 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:24,079 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:24,079 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_355_f804732f172d0e6ed3d9.table.json with policy now
2023-07-04 11:07:24,079 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:24,080 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:24,080 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:24,080 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:24,080 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:24,080 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:24,080 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:24,080 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:24,080 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:24,080 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:24,080 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:24,081 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:24,081 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:24,081 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:24,081 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:24,081 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:24,081 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:24,081 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:24,081 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:24,081 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:24,081 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:24,081 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:24,081 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:24,432 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:07:24,541 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/6cu8oi3z-media/table/batch confusion matrix_table_355_f804732f172d0e6ed3d9.table.json
2023-07-04 11:07:25,109 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:25,109 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/config.yaml
2023-07-04 11:07:25,109 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1960_da75045c8d94682c0dec.table.json
2023-07-04 11:07:25,110 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1953_67f7f9c78b661281633a.table.json
2023-07-04 11:07:25,110 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1967_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:25,110 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1995_67f7f9c78b661281633a.table.json
2023-07-04 11:07:25,110 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1974_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:25,110 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2023_04e8ade01248395789aa.table.json
2023-07-04 11:07:25,110 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2009_67f7f9c78b661281633a.table.json
2023-07-04 11:07:25,110 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2002_e6cdd73690272b208c21.table.json
2023-07-04 11:07:25,111 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1988_542681021078313e717c.table.json
2023-07-04 11:07:25,111 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_1981_e6cdd73690272b208c21.table.json
2023-07-04 11:07:25,111 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2016_1824ba9126a6c7452162.table.json
2023-07-04 11:07:25,112 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:25,199 INFO wandb-upload_13:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpl3qtb6l8
2023-07-04 11:07:25,505 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/0ucr7w41-media/table/batch confusion matrix_table_1953_67f7f9c78b661281633a.table.json
2023-07-04 11:07:25,524 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/qk8brftq-media/table/batch confusion matrix_table_1995_67f7f9c78b661281633a.table.json
2023-07-04 11:07:25,525 INFO wandb-upload_17:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/hggnhskd-media/table/batch confusion matrix_table_1974_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:25,528 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/3xjlsi1i-media/table/batch confusion matrix_table_1981_e6cdd73690272b208c21.table.json
2023-07-04 11:07:25,532 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/9jy7ulul-media/table/batch confusion matrix_table_2002_e6cdd73690272b208c21.table.json
2023-07-04 11:07:25,540 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/xtwlgsol-media/table/batch confusion matrix_table_1988_542681021078313e717c.table.json
2023-07-04 11:07:25,542 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/oi5u0pzp-media/table/batch confusion matrix_table_2009_67f7f9c78b661281633a.table.json
2023-07-04 11:07:25,543 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/2kyxt7nw-media/table/batch confusion matrix_table_1960_da75045c8d94682c0dec.table.json
2023-07-04 11:07:25,565 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/afdwpx2e-media/table/batch confusion matrix_table_2016_1824ba9126a6c7452162.table.json
2023-07-04 11:07:25,584 INFO wandb-upload_14:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ec8at857-media/table/batch confusion matrix_table_1967_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:25,609 INFO wandb-upload_16:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/tcko48iv-media/table/batch confusion matrix_table_2023_04e8ade01248395789aa.table.json
2023-07-04 11:07:26,976 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2OTYz', 'digest': 'bc0225fb0fab0db61303384269115142', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2OTM5', 'versionIndex': 49}}, 'version': 'latest'}
2023-07-04 11:07:26,976 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:26,976 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_362_eeeae82c9a9a95bb6adc.table.json with policy now
2023-07-04 11:07:26,976 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:26,976 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:26,977 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:26,977 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:26,977 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:26,978 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:26,978 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:26,978 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:26,978 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:26,978 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:26,978 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:26,979 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:26,980 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:26,980 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:26,980 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:26,980 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:26,980 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:26,980 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:26,980 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:26,980 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:26,981 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:26,981 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:26,981 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:27,019 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:27,023 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:27,442 INFO wandb-upload_15:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/a23okhvw-media/table/batch confusion matrix_table_362_eeeae82c9a9a95bb6adc.table.json
2023-07-04 11:07:27,999 INFO wandb-upload_5:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpci2on1z8
2023-07-04 11:07:28,111 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:28,111 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_2027_ef4d5a0e9ae0977a714a.table.json
2023-07-04 11:07:28,111 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:28,300 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,300 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,301 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,468 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:28,472 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,473 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,474 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,474 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,485 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,485 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,485 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,648 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:28,651 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,651 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,652 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,652 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,664 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,665 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,665 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,823 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:28,827 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk2OTk5', 'digest': 'e49119cf9301187de896e09069857dbe', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2OTYz', 'versionIndex': 50}}, 'version': 'latest'}
2023-07-04 11:07:28,827 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:28,827 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_369_67f7f9c78b661281633a.table.json with policy now
2023-07-04 11:07:28,827 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:28,827 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:28,828 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,828 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,828 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:28,829 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,829 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:28,829 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,829 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:28,830 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:28,830 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:28,830 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:28,830 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:28,830 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:28,831 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:28,832 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:28,832 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:28,832 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:28,832 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:28,832 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:28,833 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:28,833 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:28,833 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:28,833 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:28,833 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:28,833 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:28,833 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:28,840 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,883 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,884 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,990 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:28,993 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,993 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,993 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:28,994 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,005 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,005 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,006 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,166 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:29,169 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,170 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,170 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,170 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,182 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,266 INFO wandb-upload_18:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/r8obbr32-media/table/test confusion matrix_table_2027_ef4d5a0e9ae0977a714a.table.json
2023-07-04 11:07:29,270 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/u3zlyks2-media/table/batch confusion matrix_table_369_67f7f9c78b661281633a.table.json
2023-07-04 11:07:29,327 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:29,331 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,331 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,331 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,332 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,344 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,344 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,344 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,453 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:07:29,505 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:29,508 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,509 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,509 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,509 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,520 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,521 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,521 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,664 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:29,667 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,668 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,668 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,668 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,681 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,681 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,681 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,829 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:29,832 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,832 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,833 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,833 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,845 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,846 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,846 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,992 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:29,995 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,996 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,996 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:29,996 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,008 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,008 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,009 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,166 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:30,170 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,170 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,170 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,171 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,182 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,183 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,335 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:30,338 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,339 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,339 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,340 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,350 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,351 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,351 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,505 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:30,509 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,509 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,510 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,510 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,514 INFO wandb-upload_11:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp57dmambu
2023-07-04 11:07:30,521 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,522 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,522 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,680 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:30,683 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,684 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,684 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,685 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,696 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,697 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,697 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,865 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:30,869 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,869 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,869 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,869 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,881 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,882 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:30,882 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,032 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:31,035 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,036 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,036 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,036 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,048 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,049 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,049 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,199 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:31,203 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,203 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,203 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,204 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,215 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,215 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,215 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,263 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MDM0', 'digest': 'bfa65c69bf202040df9d3d570a90cc60', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2OTk5', 'versionIndex': 51}}, 'version': 'latest'}
2023-07-04 11:07:31,263 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:31,264 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_376_e04570dcb762a9eb0c10.table.json with policy now
2023-07-04 11:07:31,264 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:31,264 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:31,264 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:07:31,265 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:31,265 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:31,265 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:31,265 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:31,266 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:31,266 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:31,266 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:31,266 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:31,266 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:31,267 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:31,267 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:31,268 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:31,268 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:31,268 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:31,268 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:31,268 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:31,268 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:31,268 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:31,269 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:31,269 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:31,269 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:31,360 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:31,364 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,365 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,365 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,365 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,377 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,378 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,378 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,542 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:31,545 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,546 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,546 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,546 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,557 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,558 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,558 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,705 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:31,709 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,710 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,710 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,710 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,717 INFO wandb-upload_17:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ua4pt9qn-media/table/batch confusion matrix_table_376_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:31,721 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,722 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,722 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,879 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:31,883 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,883 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,883 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,883 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,895 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,896 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:31,896 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,061 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:32,064 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,065 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,065 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,065 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,077 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,079 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,079 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,112 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:32,113 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2101_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:32,113 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2094_1824ba9126a6c7452162.table.json
2023-07-04 11:07:32,113 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2045_e6cdd73690272b208c21.table.json
2023-07-04 11:07:32,114 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2129_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:32,114 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2080_1824ba9126a6c7452162.table.json
2023-07-04 11:07:32,115 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2178_790e3abd7e78db96a7a4.table.json
2023-07-04 11:07:32,115 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2031_67f7f9c78b661281633a.table.json
2023-07-04 11:07:32,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2157_e6cdd73690272b208c21.table.json
2023-07-04 11:07:32,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2038_542681021078313e717c.table.json
2023-07-04 11:07:32,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2171_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:32,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2073_1824ba9126a6c7452162.table.json
2023-07-04 11:07:32,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2136_542681021078313e717c.table.json
2023-07-04 11:07:32,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2122_1824ba9126a6c7452162.table.json
2023-07-04 11:07:32,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2052_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:32,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2108_542681021078313e717c.table.json
2023-07-04 11:07:32,118 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2150_1824ba9126a6c7452162.table.json
2023-07-04 11:07:32,118 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2087_1824ba9126a6c7452162.table.json
2023-07-04 11:07:32,118 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2143_1824ba9126a6c7452162.table.json
2023-07-04 11:07:32,118 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2164_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:32,118 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2059_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:32,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2066_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:32,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2115_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:32,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:32,245 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:32,249 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,250 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,250 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,251 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,261 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,262 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,262 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,310 INFO wandb-upload_12:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpmbtsiytf
2023-07-04 11:07:32,428 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:32,432 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,432 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,433 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,433 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:32,537 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/j4aj6mpb-media/table/batch confusion matrix_table_2171_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:32,546 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/md3pqfo2-media/table/batch confusion matrix_table_2080_1824ba9126a6c7452162.table.json
2023-07-04 11:07:32,546 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/c6iuptrr-media/table/batch confusion matrix_table_2094_1824ba9126a6c7452162.table.json
2023-07-04 11:07:32,547 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ron4yoyr-media/table/batch confusion matrix_table_2101_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:32,547 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/b6l6sn2b-media/table/batch confusion matrix_table_2031_67f7f9c78b661281633a.table.json
2023-07-04 11:07:32,559 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/x4pr6pz8-media/table/batch confusion matrix_table_2157_e6cdd73690272b208c21.table.json
2023-07-04 11:07:32,560 INFO wandb-upload_15:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/d2wl1fid-media/table/batch confusion matrix_table_2129_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:32,566 INFO wandb-upload_13:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ypri7275-media/table/batch confusion matrix_table_2178_790e3abd7e78db96a7a4.table.json
2023-07-04 11:07:32,580 INFO wandb-upload_16:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/i9tulqoj-media/table/batch confusion matrix_table_2045_e6cdd73690272b208c21.table.json
2023-07-04 11:07:32,587 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/4kkg0rm7-media/table/batch confusion matrix_table_2150_1824ba9126a6c7452162.table.json
2023-07-04 11:07:32,589 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/cvfnzb5o-media/table/batch confusion matrix_table_2073_1824ba9126a6c7452162.table.json
2023-07-04 11:07:32,591 INFO wandb-upload_17:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/aua4ymg7-media/table/batch confusion matrix_table_2038_542681021078313e717c.table.json
2023-07-04 11:07:32,593 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/55pn3055-media/table/batch confusion matrix_table_2052_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:32,610 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/4cwvno6l-media/table/batch confusion matrix_table_2122_1824ba9126a6c7452162.table.json
2023-07-04 11:07:32,641 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/orthcjm9-media/table/batch confusion matrix_table_2108_542681021078313e717c.table.json
2023-07-04 11:07:32,645 INFO wandb-upload_22:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/nd87a8ba-media/table/batch confusion matrix_table_2115_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:32,656 INFO wandb-upload_18:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/y6vziixt-media/table/batch confusion matrix_table_2087_1824ba9126a6c7452162.table.json
2023-07-04 11:07:32,662 INFO wandb-upload_21:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/99yn1lrk-media/table/batch confusion matrix_table_2066_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:32,689 INFO wandb-upload_14:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ojvhd9rq-media/table/batch confusion matrix_table_2143_1824ba9126a6c7452162.table.json
2023-07-04 11:07:32,697 INFO wandb-upload_19:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/jteddg1p-media/table/batch confusion matrix_table_2164_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:32,702 INFO wandb-upload_20:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/acpku7o5-media/table/batch confusion matrix_table_2059_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:32,709 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/4e6okat9-media/table/batch confusion matrix_table_2136_542681021078313e717c.table.json
2023-07-04 11:07:33,135 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MDU4', 'digest': 'bbfea6e1c3ac9785e93ad1a602378a3a', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MDM0', 'versionIndex': 52}}, 'version': 'latest'}
2023-07-04 11:07:33,135 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:33,135 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_383_1824ba9126a6c7452162.table.json with policy now
2023-07-04 11:07:33,135 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:33,135 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:33,136 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:33,136 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:33,136 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:33,136 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:33,136 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:33,136 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:33,137 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:33,137 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:33,137 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:33,137 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:33,137 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:33,137 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:33,137 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:33,137 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:33,137 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:33,137 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:33,137 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:33,137 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:33,138 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:33,138 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:33,138 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:33,583 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/5aax3hz5-media/table/batch confusion matrix_table_383_1824ba9126a6c7452162.table.json
2023-07-04 11:07:34,154 INFO wandb-upload_5:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpgze5n1u_
2023-07-04 11:07:34,477 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:07:34,970 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MDc5', 'digest': 'bfe2ff5d86e2590643a01ca6c5d56925', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MDU4', 'versionIndex': 53}}, 'version': 'latest'}
2023-07-04 11:07:34,970 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:34,970 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_390_5fdac077eca64fafc222.table.json with policy now
2023-07-04 11:07:34,971 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:34,971 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:34,971 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:34,971 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:34,971 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:34,971 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:34,972 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:34,972 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:34,973 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:34,973 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:34,973 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:34,973 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:34,973 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:34,973 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:34,973 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:34,973 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:34,974 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:34,974 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:34,974 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:34,974 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:34,974 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:34,974 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:34,974 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:35,427 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/zetmpjl4-media/table/batch confusion matrix_table_390_5fdac077eca64fafc222.table.json
2023-07-04 11:07:35,518 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:35,522 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:36,011 INFO wandb-upload_11:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpxv0y_9_u
2023-07-04 11:07:36,114 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:36,114 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_2196_50025681b919183fa018.table.json
2023-07-04 11:07:36,114 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2192_17ac89d9b34a12ddbdd5.table.json
2023-07-04 11:07:36,114 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2185_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:36,114 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:36,798 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MDk3', 'digest': 'fa6ef188b0ff928e08d33dac84eac5ef', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MDc5', 'versionIndex': 54}}, 'version': 'latest'}
2023-07-04 11:07:36,798 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:36,798 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_397_542681021078313e717c.table.json with policy now
2023-07-04 11:07:36,799 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:07:36,799 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:36,799 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:36,799 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:36,799 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:36,800 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:36,800 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:36,800 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:36,800 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:36,800 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:36,800 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:36,800 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:36,800 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:36,800 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:36,800 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:36,800 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:36,800 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:36,800 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:36,801 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:36,801 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:36,801 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:36,801 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:36,801 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:36,801 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:36,803 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:36,803 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:36,803 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:36,967 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:36,971 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:36,971 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:36,971 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:36,971 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:36,983 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:36,984 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:36,984 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,129 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:37,133 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,133 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,133 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,134 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,146 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,146 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,146 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,196 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/p03kczc2-media/table/batch confusion matrix_table_2185_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:37,210 INFO wandb-upload_16:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/inflw6gv-media/table/batch confusion matrix_table_2192_17ac89d9b34a12ddbdd5.table.json
2023-07-04 11:07:37,243 INFO wandb-upload_17:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/i7kwnvq9-media/table/batch confusion matrix_table_397_542681021078313e717c.table.json
2023-07-04 11:07:37,258 INFO wandb-upload_13:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/1h5hwvsu-media/table/test confusion matrix_table_2196_50025681b919183fa018.table.json
2023-07-04 11:07:37,288 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:37,292 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,293 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,293 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,293 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,305 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,305 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,305 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,455 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:37,458 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,459 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,459 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,459 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,471 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,472 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,472 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,638 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:37,642 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,642 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,642 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,642 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,654 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,655 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,655 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,819 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:37,823 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,823 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,823 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,823 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,836 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,836 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,836 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,881 INFO wandb-upload_9:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp9tdiiy4p
2023-07-04 11:07:37,994 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:37,998 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,998 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,999 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:37,999 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,011 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,011 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,012 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,178 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:38,181 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,182 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,182 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,182 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,194 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,195 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,195 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,360 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:38,364 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,364 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,364 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,365 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,377 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,377 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,377 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,525 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:38,529 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,530 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,530 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,530 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,542 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,543 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,543 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,626 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MTEy', 'digest': '6823c1729bb0e58fb6c52cdcd16e1cb0', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MDk3', 'versionIndex': 55}}, 'version': 'latest'}
2023-07-04 11:07:38,626 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:38,626 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_404_c2c89839631ee54a153b.table.json with policy now
2023-07-04 11:07:38,627 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:38,627 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:38,627 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:38,627 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:38,627 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:38,627 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:38,627 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:38,627 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:38,627 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:38,627 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:38,627 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:38,628 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:38,628 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:38,628 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:38,629 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:38,629 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:38,629 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:38,629 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:38,629 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:38,629 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:38,629 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:38,629 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:38,629 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:38,705 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:38,708 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,708 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,709 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,709 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,720 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,721 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,721 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,871 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:38,874 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,874 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,875 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,875 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,887 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,887 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:38,888 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,053 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:39,057 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,057 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,057 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,058 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,069 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,069 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,070 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,105 INFO wandb-upload_22:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/q01s0tuy-media/table/batch confusion matrix_table_404_c2c89839631ee54a153b.table.json
2023-07-04 11:07:39,215 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:39,218 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,218 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,219 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,219 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,231 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,231 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,231 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,388 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:39,391 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,392 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,392 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,392 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,404 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,405 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,405 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,507 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:07:39,566 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:39,570 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,570 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,571 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,571 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,582 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,583 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,583 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,659 INFO wandb-upload_21:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpemizckko
2023-07-04 11:07:39,743 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:39,747 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,747 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,747 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,748 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,760 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,760 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,761 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,918 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:39,922 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,922 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,922 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,923 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,934 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,935 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:39,935 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,094 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:40,098 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,098 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,098 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,099 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,110 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,111 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,111 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,115 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:40,115 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2256_67f7f9c78b661281633a.table.json
2023-07-04 11:07:40,115 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2312_e6cdd73690272b208c21.table.json
2023-07-04 11:07:40,115 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2235_e6cdd73690272b208c21.table.json
2023-07-04 11:07:40,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2221_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2214_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2298_67f7f9c78b661281633a.table.json
2023-07-04 11:07:40,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2200_e6cdd73690272b208c21.table.json
2023-07-04 11:07:40,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2242_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:40,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2284_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:40,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2291_e6cdd73690272b208c21.table.json
2023-07-04 11:07:40,116 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2326_67f7f9c78b661281633a.table.json
2023-07-04 11:07:40,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2277_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2263_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2249_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2270_542681021078313e717c.table.json
2023-07-04 11:07:40,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2319_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2305_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2207_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2228_e6cdd73690272b208c21.table.json
2023-07-04 11:07:40,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:40,261 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:40,266 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,266 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,266 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,267 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,278 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,279 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,279 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,421 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:40,424 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,425 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,425 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,425 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,437 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,438 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,438 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,450 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MTI0', 'digest': 'b2fab797a007dd198cfd78e1c08bac1e', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MTEy', 'versionIndex': 56}}, 'version': 'latest'}
2023-07-04 11:07:40,450 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:40,451 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_411_ac4ad9c89ab9067ca814.table.json with policy now
2023-07-04 11:07:40,451 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:40,452 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:40,453 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:40,454 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:40,454 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:40,456 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:40,457 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:40,457 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:40,460 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:40,464 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:40,464 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:40,465 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:40,465 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:40,465 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:40,465 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:40,465 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:40,465 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:40,466 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:40,466 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:40,466 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:40,466 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:40,466 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:40,467 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:40,589 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:40,628 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,628 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,630 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,630 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,630 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,630 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,632 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,756 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:40,760 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,760 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,760 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,760 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,771 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,771 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,772 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,850 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/bncaht1q-media/table/batch confusion matrix_table_2284_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:40,853 INFO wandb-upload_20:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/vwf1xqfl-media/table/batch confusion matrix_table_2312_e6cdd73690272b208c21.table.json
2023-07-04 11:07:40,853 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/04t0n024-media/table/batch confusion matrix_table_2200_e6cdd73690272b208c21.table.json
2023-07-04 11:07:40,862 INFO wandb-upload_19:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/0zv1n3v6-media/table/batch confusion matrix_table_2256_67f7f9c78b661281633a.table.json
2023-07-04 11:07:40,863 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/tbf8axsj-media/table/batch confusion matrix_table_2214_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,875 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/7x2qnye0-media/table/batch confusion matrix_table_2235_e6cdd73690272b208c21.table.json
2023-07-04 11:07:40,903 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/4tgy7rnm-media/table/batch confusion matrix_table_2221_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,903 INFO wandb-upload_15:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/gs3levvq-media/table/batch confusion matrix_table_2291_e6cdd73690272b208c21.table.json
2023-07-04 11:07:40,905 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/78kcp6pe-media/table/batch confusion matrix_table_2298_67f7f9c78b661281633a.table.json
2023-07-04 11:07:40,911 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/cnmfs71i-media/table/batch confusion matrix_table_2263_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,914 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/c5vkzmgh-media/table/batch confusion matrix_table_2207_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,923 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/t2agtfxf-media/table/batch confusion matrix_table_2242_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:40,929 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/5aiqft8u-media/table/batch confusion matrix_table_2228_e6cdd73690272b208c21.table.json
2023-07-04 11:07:40,930 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/mnewr60p-media/table/batch confusion matrix_table_2249_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,932 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/q8excjo9-media/table/batch confusion matrix_table_2277_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,934 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/d7bdmgxk-media/table/batch confusion matrix_table_2326_67f7f9c78b661281633a.table.json
2023-07-04 11:07:40,935 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:40,939 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,940 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,940 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,940 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:40,956 INFO wandb-upload_16:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/gupwhrce-media/table/batch confusion matrix_table_2270_542681021078313e717c.table.json
2023-07-04 11:07:40,966 INFO wandb-upload_17:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/7bv3rrwc-media/table/batch confusion matrix_table_2319_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,971 INFO wandb-upload_13:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/7656btrd-media/table/batch confusion matrix_table_2305_1824ba9126a6c7452162.table.json
2023-07-04 11:07:40,981 INFO wandb-upload_22:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/aqj78j33-media/table/batch confusion matrix_table_411_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:41,607 INFO wandb-upload_21:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpqgv0t_5m
2023-07-04 11:07:42,355 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MTQx', 'digest': '04e74fc23e381ac87c5da107b120c37a', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MTI0', 'versionIndex': 57}}, 'version': 'latest'}
2023-07-04 11:07:42,355 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:42,356 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_418_fe4098d4c25d56edcebd.table.json with policy now
2023-07-04 11:07:42,356 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:42,356 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:42,356 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:42,356 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:07:42,356 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:42,357 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:42,358 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:42,358 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:42,358 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:42,359 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:42,359 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:42,360 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:42,360 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:42,360 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:42,360 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:42,360 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:42,360 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:42,360 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:42,360 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:42,360 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:42,360 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:42,360 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:42,360 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:42,361 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:42,840 INFO wandb-upload_14:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ot65qwbj-media/table/batch confusion matrix_table_418_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:43,407 INFO wandb-upload_20:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpz1m_grk4
2023-07-04 11:07:44,047 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:44,051 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:44,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:44,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2354_e6cdd73690272b208c21.table.json
2023-07-04 11:07:44,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2340_1824ba9126a6c7452162.table.json
2023-07-04 11:07:44,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2361_e1039e7ebf7dc1fdf6c1.table.json
2023-07-04 11:07:44,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2347_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:44,117 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_2365_f0313d9f5962cb4fa2b9.table.json
2023-07-04 11:07:44,118 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2333_542681021078313e717c.table.json
2023-07-04 11:07:44,118 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:44,170 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MTU5', 'digest': 'fa6ef188b0ff928e08d33dac84eac5ef', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MTQx', 'versionIndex': 58}}, 'version': 'latest'}
2023-07-04 11:07:44,171 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:44,171 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_425_542681021078313e717c.table.json with policy now
2023-07-04 11:07:44,173 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:44,173 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:44,176 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:44,176 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:44,176 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:44,176 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:44,176 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:44,176 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:44,177 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:44,177 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:44,177 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:44,177 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:44,177 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:44,177 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:44,177 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:44,177 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:44,177 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:44,177 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:44,177 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:44,177 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:44,177 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:44,178 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:44,178 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:44,543 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:07:44,559 INFO wandb-upload_15:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/lmqnhcks-media/table/test confusion matrix_table_2365_f0313d9f5962cb4fa2b9.table.json
2023-07-04 11:07:44,581 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/5fz3g5g3-media/table/batch confusion matrix_table_2340_1824ba9126a6c7452162.table.json
2023-07-04 11:07:44,583 INFO wandb-upload_19:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/f7sxz1un-media/table/batch confusion matrix_table_2354_e6cdd73690272b208c21.table.json
2023-07-04 11:07:44,588 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/a9oecdl0-media/table/batch confusion matrix_table_2361_e1039e7ebf7dc1fdf6c1.table.json
2023-07-04 11:07:44,592 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/dvahkjwo-media/table/batch confusion matrix_table_2347_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:44,610 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/480iga2u-media/table/batch confusion matrix_table_425_542681021078313e717c.table.json
2023-07-04 11:07:44,616 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/v8dao5vb-media/table/batch confusion matrix_table_2333_542681021078313e717c.table.json
2023-07-04 11:07:45,214 INFO wandb-upload_9:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmprleeqc19
2023-07-04 11:07:45,305 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,306 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,306 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,472 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:45,476 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,477 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,477 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,477 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,489 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,489 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,490 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,651 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:45,657 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,657 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,657 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,658 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,670 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,670 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,671 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,813 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:45,817 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,817 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,818 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,818 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,830 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,830 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,830 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,955 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MTc2', 'digest': 'bf791e3651b8dfa4ebaf7a7be8f85367', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MTU5', 'versionIndex': 59}}, 'version': 'latest'}
2023-07-04 11:07:45,955 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:45,955 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_432_ae8fd1dddcf14dc27777.table.json with policy now
2023-07-04 11:07:45,955 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:45,955 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:45,956 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:45,956 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:45,956 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:45,957 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:45,957 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:45,957 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:45,958 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:45,958 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:45,958 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:45,958 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:45,958 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:45,958 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:45,958 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:45,959 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:45,959 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:45,959 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:45,959 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:45,959 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:45,959 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:45,959 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:45,959 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:45,977 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:45,980 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,980 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,981 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,981 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,993 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,993 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:45,994 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,135 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:46,139 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,139 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,140 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,140 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,151 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,152 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,152 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,308 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:46,311 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,312 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,312 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,312 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,323 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,324 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,324 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,396 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/w7dkb5kh-media/table/batch confusion matrix_table_432_ae8fd1dddcf14dc27777.table.json
2023-07-04 11:07:46,492 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:46,496 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,497 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,497 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,497 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,509 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,510 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,510 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,661 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:46,665 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,665 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,665 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,666 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,677 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,677 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,678 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,843 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:46,847 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,847 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,847 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,847 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,859 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,860 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,860 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:46,987 INFO wandb-upload_16:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpzhrf21mt
2023-07-04 11:07:47,004 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:47,008 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,008 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,009 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,009 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,020 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,021 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,021 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,182 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:47,185 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,185 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,186 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,186 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,197 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,198 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,198 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,343 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:47,346 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,347 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,347 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,347 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,359 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,359 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,360 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,525 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:47,529 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,529 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,530 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,530 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,541 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,542 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,542 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,685 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:47,690 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,690 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,690 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,690 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,702 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,702 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,703 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,820 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MTkz', 'digest': 'bbfea6e1c3ac9785e93ad1a602378a3a', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MTc2', 'versionIndex': 60}}, 'version': 'latest'}
2023-07-04 11:07:47,820 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:47,820 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:07:47,820 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_439_1824ba9126a6c7452162.table.json with policy now
2023-07-04 11:07:47,821 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:47,821 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:47,821 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:47,821 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:47,821 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:47,821 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:47,821 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:47,821 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:47,821 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:47,822 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:47,822 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:47,822 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:47,822 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:47,822 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:47,822 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:47,822 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:47,822 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:47,822 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:47,822 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:47,822 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:47,823 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:47,823 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:47,823 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:47,898 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:47,901 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,902 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,902 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,902 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,913 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,914 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:47,914 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,064 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:48,067 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,068 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,068 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,068 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,079 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,080 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,080 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,118 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:48,118 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2474_542681021078313e717c.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2446_91631cdbd9a35f428cc7.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2390_67f7f9c78b661281633a.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2460_da75045c8d94682c0dec.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2369_67f7f9c78b661281633a.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2467_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2418_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2383_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2397_e6cdd73690272b208c21.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2411_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2453_542681021078313e717c.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2376_1824ba9126a6c7452162.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2439_1824ba9126a6c7452162.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2425_542681021078313e717c.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2432_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2404_542681021078313e717c.table.json
2023-07-04 11:07:48,119 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:48,226 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:48,232 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,233 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,233 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,233 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,245 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,245 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,245 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,295 INFO wandb-upload_13:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/falglehj-media/table/batch confusion matrix_table_439_1824ba9126a6c7452162.table.json
2023-07-04 11:07:48,399 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:48,403 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,403 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,403 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,403 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,416 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,416 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,416 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,521 INFO wandb-upload_15:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/xshzbsi8-media/table/batch confusion matrix_table_2369_67f7f9c78b661281633a.table.json
2023-07-04 11:07:48,526 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/3dy1e318-media/table/batch confusion matrix_table_2446_91631cdbd9a35f428cc7.table.json
2023-07-04 11:07:48,533 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/q0ji77en-media/table/batch confusion matrix_table_2418_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:48,534 INFO wandb-upload_18:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/xfpm5spk-media/table/batch confusion matrix_table_2474_542681021078313e717c.table.json
2023-07-04 11:07:48,536 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/z9vj8ctv-media/table/batch confusion matrix_table_2460_da75045c8d94682c0dec.table.json
2023-07-04 11:07:48,543 INFO wandb-upload_19:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/t6mdhx8h-media/table/batch confusion matrix_table_2467_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:48,545 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/f9ufluez-media/table/batch confusion matrix_table_2397_e6cdd73690272b208c21.table.json
2023-07-04 11:07:48,545 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/5glxhe0p-media/table/batch confusion matrix_table_2390_67f7f9c78b661281633a.table.json
2023-07-04 11:07:48,550 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/i5qpsmm1-media/table/batch confusion matrix_table_2383_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:48,552 INFO wandb-upload_14:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/zkfx9mx0-media/table/batch confusion matrix_table_2439_1824ba9126a6c7452162.table.json
2023-07-04 11:07:48,557 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/unjp5zlv-media/table/batch confusion matrix_table_2411_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:48,557 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:48,560 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,560 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,560 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,560 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,572 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,572 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,572 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,581 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/q557b9xa-media/table/batch confusion matrix_table_2404_542681021078313e717c.table.json
2023-07-04 11:07:48,588 INFO wandb-upload_22:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/oewzdl4z-media/table/batch confusion matrix_table_2376_1824ba9126a6c7452162.table.json
2023-07-04 11:07:48,601 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ltpq7n64-media/table/batch confusion matrix_table_2432_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:48,602 INFO wandb-upload_17:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/6g90ru9o-media/table/batch confusion matrix_table_2453_542681021078313e717c.table.json
2023-07-04 11:07:48,613 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/l3vr7c6t-media/table/batch confusion matrix_table_2425_542681021078313e717c.table.json
2023-07-04 11:07:48,732 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:48,735 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,736 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,736 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,736 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,748 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,748 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,748 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,898 INFO wandb-upload_16:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp1qklv1lb
2023-07-04 11:07:48,907 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:48,912 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,912 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,912 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,912 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,924 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,925 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:48,925 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,068 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:49,072 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,072 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,072 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,072 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,084 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,084 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,085 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,230 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:49,234 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,235 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,235 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,235 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,245 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,246 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,246 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,390 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:49,394 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,394 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,395 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,395 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:49,544 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:07:49,700 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MjA2', 'digest': 'de61fa0d70b6214a2949d00ab7208a23', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MTkz', 'versionIndex': 61}}, 'version': 'latest'}
2023-07-04 11:07:49,700 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:49,700 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_446_19d420f9c457760a0e7a.table.json with policy now
2023-07-04 11:07:49,701 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:49,701 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:49,701 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:49,702 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:49,702 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:49,703 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:49,703 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:49,703 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:49,703 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:49,703 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:49,703 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:49,704 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:49,704 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:49,704 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:49,704 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:49,704 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:49,704 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:49,704 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:49,704 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:49,704 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:49,705 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:49,705 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:49,705 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:50,164 INFO wandb-upload_20:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/j22tk09m-media/table/batch confusion matrix_table_446_19d420f9c457760a0e7a.table.json
2023-07-04 11:07:50,724 INFO wandb-upload_4:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmp3910s3bm
2023-07-04 11:07:51,587 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MjE1', 'digest': '4a8245ff2145856179e818c0a39ff309', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MjA2', 'versionIndex': 62}}, 'version': 'latest'}
2023-07-04 11:07:51,587 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:51,587 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_453_e6cdd73690272b208c21.table.json with policy now
2023-07-04 11:07:51,587 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:51,587 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:51,587 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:51,587 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:51,587 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:51,588 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:51,588 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:51,588 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:51,588 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:51,588 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:51,588 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:51,588 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:51,588 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:51,588 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:51,589 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:51,590 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:51,590 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:51,590 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:51,590 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:51,590 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:51,590 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:51,590 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:51,590 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:52,079 INFO wandb-upload_13:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/s78e3vch-media/table/batch confusion matrix_table_453_e6cdd73690272b208c21.table.json
2023-07-04 11:07:52,120 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:52,120 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2495_e6cdd73690272b208c21.table.json
2023-07-04 11:07:52,120 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2509_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:52,121 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2530_42b451464002c81d89cb.table.json
2023-07-04 11:07:52,121 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2502_1824ba9126a6c7452162.table.json
2023-07-04 11:07:52,121 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2516_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:52,121 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2523_d2c8f983b9de909a4cda.table.json
2023-07-04 11:07:52,121 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2481_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:52,121 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2488_542681021078313e717c.table.json
2023-07-04 11:07:52,121 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:52,530 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/m4nvpih9-media/table/batch confusion matrix_table_2481_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:52,532 INFO wandb-upload_14:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/gyu92bjw-media/table/batch confusion matrix_table_2523_d2c8f983b9de909a4cda.table.json
2023-07-04 11:07:52,546 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/fhd91k4y-media/table/batch confusion matrix_table_2488_542681021078313e717c.table.json
2023-07-04 11:07:52,548 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/8st9wv3j-media/table/batch confusion matrix_table_2516_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:52,553 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/9u1bhu5r-media/table/batch confusion matrix_table_2509_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:52,562 INFO wandb-upload_18:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ukeo0p5m-media/table/batch confusion matrix_table_2530_42b451464002c81d89cb.table.json
2023-07-04 11:07:52,572 INFO wandb-upload_15:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/lu5kr50n-media/table/batch confusion matrix_table_2495_e6cdd73690272b208c21.table.json
2023-07-04 11:07:52,598 INFO wandb-upload_19:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/6qhq7v0i-media/table/batch confusion matrix_table_2502_1824ba9126a6c7452162.table.json
2023-07-04 11:07:52,617 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:52,620 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:52,744 INFO wandb-upload_17:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpnhdvsn4e
2023-07-04 11:07:53,565 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MjI4', 'digest': 'bfa65c69bf202040df9d3d570a90cc60', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MjE1', 'versionIndex': 63}}, 'version': 'latest'}
2023-07-04 11:07:53,565 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:53,565 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_460_e04570dcb762a9eb0c10.table.json with policy now
2023-07-04 11:07:53,565 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:07:53,565 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:53,566 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:53,566 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:53,566 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:53,566 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:53,566 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:53,567 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:53,567 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:53,567 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:53,568 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:53,568 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:53,568 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:53,568 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:53,568 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:53,568 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:53,568 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:53,568 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:53,568 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:53,568 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:53,568 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:53,568 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:53,568 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:53,568 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:53,903 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:53,904 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:53,904 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,003 INFO wandb-upload_16:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/g8sfdr77-media/table/batch confusion matrix_table_460_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:54,060 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:54,065 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,066 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,066 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,066 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,078 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,079 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,079 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,242 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:54,246 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,246 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,247 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,247 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,258 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,259 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,259 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,417 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:54,420 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,421 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,421 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,421 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,433 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,433 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,433 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,586 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:07:54,591 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:54,595 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,596 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,596 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,596 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,608 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,609 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,609 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,683 INFO wandb-upload_6:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpzkzy663v
2023-07-04 11:07:54,757 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:54,762 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,762 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,762 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,762 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,774 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,774 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,775 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,935 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:54,939 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,940 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,940 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,940 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,952 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,952 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:54,952 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,096 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:55,101 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,101 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,101 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,101 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,113 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,114 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,114 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,259 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:55,262 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,263 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,263 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,263 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,275 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,275 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,276 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,431 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:55,434 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,434 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,434 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,435 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,446 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,447 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,447 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,497 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MjY2', 'digest': 'fa30b10c61603a43b7c0fe65726ad705', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MjI4', 'versionIndex': 64}}, 'version': 'latest'}
2023-07-04 11:07:55,497 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:55,497 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_467_8fbc5f39b29508567cad.table.json with policy now
2023-07-04 11:07:55,498 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:55,498 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:55,498 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:55,498 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:55,498 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:55,499 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:55,499 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:55,499 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:55,499 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:55,499 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:55,499 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:55,499 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:55,499 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:55,499 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:55,499 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:55,500 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:55,500 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:55,500 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:55,500 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:55,500 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:55,501 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:55,501 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:55,501 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:55,603 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:55,607 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,607 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,607 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,608 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,619 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,620 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,620 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,779 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:55,784 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,785 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,785 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,785 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,797 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,797 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,797 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,938 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/w9rby4pt-media/table/batch confusion matrix_table_467_8fbc5f39b29508567cad.table.json
2023-07-04 11:07:55,956 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:55,960 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,961 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,961 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,961 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,972 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,973 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:55,973 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,122 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:07:56,122 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2587_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:56,122 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_2534_ef4d5a0e9ae0977a714a.table.json
2023-07-04 11:07:56,122 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2566_67f7f9c78b661281633a.table.json
2023-07-04 11:07:56,122 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2552_67f7f9c78b661281633a.table.json
2023-07-04 11:07:56,122 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2573_542681021078313e717c.table.json
2023-07-04 11:07:56,122 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2545_67f7f9c78b661281633a.table.json
2023-07-04 11:07:56,122 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2559_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:56,123 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2601_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:56,123 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2580_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:56,123 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2608_1824ba9126a6c7452162.table.json
2023-07-04 11:07:56,123 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2538_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:56,123 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2615_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:56,123 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2594_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:56,123 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:07:56,133 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:56,136 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,137 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,137 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,138 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,149 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,149 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,149 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,308 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:56,313 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,313 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,314 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,314 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,325 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,326 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,326 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,497 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:56,501 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,501 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,502 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,502 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,513 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,514 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,514 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,523 INFO wandb-upload_18:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/oxz3f89k-media/table/batch confusion matrix_table_2601_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:56,525 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/jdnpbw3m-media/table/batch confusion matrix_table_2545_67f7f9c78b661281633a.table.json
2023-07-04 11:07:56,527 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/53ow50i0-media/table/batch confusion matrix_table_2566_67f7f9c78b661281633a.table.json
2023-07-04 11:07:56,531 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/i6mzl4j2-media/table/batch confusion matrix_table_2573_542681021078313e717c.table.json
2023-07-04 11:07:56,532 INFO wandb-upload_22:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/8tgr380p-media/table/batch confusion matrix_table_2552_67f7f9c78b661281633a.table.json
2023-07-04 11:07:56,534 INFO wandb-upload_19:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/v8cgpjdc-media/table/batch confusion matrix_table_2580_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:07:56,554 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/9iay7eic-media/table/test confusion matrix_table_2534_ef4d5a0e9ae0977a714a.table.json
2023-07-04 11:07:56,555 INFO wandb-upload_20:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/owqfvhli-media/table/batch confusion matrix_table_2538_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:56,568 INFO wandb-upload_13:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmplxwujllq
2023-07-04 11:07:56,578 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/o353emwx-media/table/batch confusion matrix_table_2559_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:56,578 INFO wandb-upload_21:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/x5zcvnti-media/table/batch confusion matrix_table_2615_e04570dcb762a9eb0c10.table.json
2023-07-04 11:07:56,579 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/notrkt9p-media/table/batch confusion matrix_table_2594_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:56,585 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/t01hlvbm-media/table/batch confusion matrix_table_2608_1824ba9126a6c7452162.table.json
2023-07-04 11:07:56,609 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/iiyq9ud0-media/table/batch confusion matrix_table_2587_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:56,663 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:56,667 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,667 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,667 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,667 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,679 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,680 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,680 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,834 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:56,838 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,838 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,838 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,838 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,850 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,851 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:56,851 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,013 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:57,016 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,017 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,017 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,017 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,029 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,030 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,030 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,201 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:57,205 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,205 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,206 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,206 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,217 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,218 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,218 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,382 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:57,386 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,386 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,386 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,387 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,399 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,399 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,399 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,429 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3Mjkz', 'digest': '04e74fc23e381ac87c5da107b120c37a', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MjY2', 'versionIndex': 65}}, 'version': 'latest'}
2023-07-04 11:07:57,430 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:57,430 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_474_fe4098d4c25d56edcebd.table.json with policy now
2023-07-04 11:07:57,430 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:57,430 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:57,430 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:57,430 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:57,430 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:57,431 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:57,431 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:57,431 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:57,431 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:57,431 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:57,431 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:57,431 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:57,431 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:57,431 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:57,431 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:57,431 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:57,431 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:57,431 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:57,432 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:57,432 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:57,432 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:57,432 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:57,432 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:57,561 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:57,603 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,604 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,604 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,604 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,604 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,604 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,604 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,728 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:57,732 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,732 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,732 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,732 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,744 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,744 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,744 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,859 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/2ijltslr-media/table/batch confusion matrix_table_474_fe4098d4c25d56edcebd.table.json
2023-07-04 11:07:57,893 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:57,899 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,899 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,899 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,900 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,911 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,912 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:57,912 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:58,068 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:07:58,072 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:58,072 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:58,072 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:58,073 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:07:58,450 INFO wandb-upload_5:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpqq6pezl4
2023-07-04 11:07:59,280 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MzA2', 'digest': 'e233af727ab1eeacb0b94ee496beb333', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3Mjkz', 'versionIndex': 66}}, 'version': 'latest'}
2023-07-04 11:07:59,282 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:07:59,464 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:07:59,464 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_481_d4d30958e51757b86519.table.json with policy now
2023-07-04 11:07:59,465 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:59,465 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:59,465 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:59,465 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:59,465 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:59,465 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:59,465 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:59,465 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:59,466 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:59,467 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:59,467 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:59,467 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:59,467 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:59,467 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:59,467 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:59,467 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:59,467 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:59,467 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:59,467 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:07:59,467 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:07:59,467 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:07:59,468 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:07:59,468 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:07:59,619 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:07:59,946 INFO wandb-upload_17:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/k0875szo-media/table/batch confusion matrix_table_481_d4d30958e51757b86519.table.json
2023-07-04 11:08:00,124 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:08:00,124 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/config.yaml
2023-07-04 11:08:00,124 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2685_67f7f9c78b661281633a.table.json
2023-07-04 11:08:00,124 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2629_542681021078313e717c.table.json
2023-07-04 11:08:00,124 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2678_1824ba9126a6c7452162.table.json
2023-07-04 11:08:00,124 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2650_91631cdbd9a35f428cc7.table.json
2023-07-04 11:08:00,124 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2664_1824ba9126a6c7452162.table.json
2023-07-04 11:08:00,124 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2643_e6cdd73690272b208c21.table.json
2023-07-04 11:08:00,124 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2636_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:00,124 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2657_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:00,124 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2692_1824ba9126a6c7452162.table.json
2023-07-04 11:08:00,125 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2622_1824ba9126a6c7452162.table.json
2023-07-04 11:08:00,125 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2699_42b451464002c81d89cb.table.json
2023-07-04 11:08:00,125 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2671_e6cdd73690272b208c21.table.json
2023-07-04 11:08:00,125 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:08:00,515 INFO wandb-upload_6:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpl3tfrmof
2023-07-04 11:08:00,522 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/iue8ci4a-media/table/batch confusion matrix_table_2678_1824ba9126a6c7452162.table.json
2023-07-04 11:08:00,523 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/7hq3rwct-media/table/batch confusion matrix_table_2657_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:00,530 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/6mo5527m-media/table/batch confusion matrix_table_2629_542681021078313e717c.table.json
2023-07-04 11:08:00,539 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/go1x1s2k-media/table/batch confusion matrix_table_2685_67f7f9c78b661281633a.table.json
2023-07-04 11:08:00,539 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/pmt83r9d-media/table/batch confusion matrix_table_2692_1824ba9126a6c7452162.table.json
2023-07-04 11:08:00,549 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/rbjfzaii-media/table/batch confusion matrix_table_2636_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:00,551 INFO wandb-upload_20:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/z5ccge9v-media/table/batch confusion matrix_table_2664_1824ba9126a6c7452162.table.json
2023-07-04 11:08:00,566 INFO wandb-upload_17:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/p5pmm9vz-media/table/batch confusion matrix_table_2699_42b451464002c81d89cb.table.json
2023-07-04 11:08:00,572 INFO wandb-upload_19:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/alft76dn-media/table/batch confusion matrix_table_2650_91631cdbd9a35f428cc7.table.json
2023-07-04 11:08:00,576 INFO wandb-upload_18:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/gyom8okj-media/table/batch confusion matrix_table_2671_e6cdd73690272b208c21.table.json
2023-07-04 11:08:00,589 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/rhdfzsud-media/table/batch confusion matrix_table_2643_e6cdd73690272b208c21.table.json
2023-07-04 11:08:00,590 INFO wandb-upload_5:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/6ijxoaoo-media/table/batch confusion matrix_table_2622_1824ba9126a6c7452162.table.json
2023-07-04 11:08:01,307 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:01,310 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:01,331 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MzU2', 'digest': 'fa6ef188b0ff928e08d33dac84eac5ef', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MzA2', 'versionIndex': 67}}, 'version': 'latest'}
2023-07-04 11:08:01,331 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:08:01,332 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_488_542681021078313e717c.table.json with policy now
2023-07-04 11:08:01,332 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:01,332 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:01,332 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:01,332 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:01,332 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:01,332 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:01,333 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:01,333 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:01,333 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:01,333 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:01,333 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:01,333 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:01,333 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:01,333 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:01,333 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:01,333 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:01,333 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:01,333 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:01,333 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:01,333 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:01,334 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:01,334 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:08:01,334 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:08:01,772 INFO wandb-upload_22:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/cdapz75l-media/table/batch confusion matrix_table_488_542681021078313e717c.table.json
2023-07-04 11:08:02,330 INFO wandb-upload_13:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpapd5v41k
2023-07-04 11:08:02,562 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,603 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,603 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,712 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:02,717 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,718 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,718 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,718 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,730 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,730 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,731 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,879 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:02,882 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,883 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,883 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,883 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,895 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,895 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:02,895 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,040 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:03,043 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,044 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,044 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,044 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,057 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,057 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,057 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,077 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3MzY5', 'digest': 'b2fab797a007dd198cfd78e1c08bac1e', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MzU2', 'versionIndex': 68}}, 'version': 'latest'}
2023-07-04 11:08:03,077 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:08:03,077 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_495_ac4ad9c89ab9067ca814.table.json with policy now
2023-07-04 11:08:03,077 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:03,077 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:03,077 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:03,077 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:03,077 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:03,077 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:03,078 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:03,078 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:03,078 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:03,078 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:03,078 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:03,079 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:03,079 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:03,079 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:03,079 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:03,079 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:03,079 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:03,079 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:03,079 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:03,079 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:03,079 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:03,079 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:08:03,079 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:08:03,202 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:03,243 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,244 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,244 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,244 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,244 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,244 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,244 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,363 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:03,366 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,366 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,367 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,367 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,378 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,379 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,379 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,526 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:03,530 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,531 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,531 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,531 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,542 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,543 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,543 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,547 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/bda3ki1x-media/table/batch confusion matrix_table_495_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:08:03,704 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:03,707 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,708 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,708 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,708 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,719 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,720 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,720 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,884 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:03,888 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,888 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,889 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,889 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,900 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,901 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:03,901 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,059 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:04,062 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,063 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,063 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,063 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,075 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,075 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,075 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,107 INFO wandb-upload_15:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpmn5f3bny
2023-07-04 11:08:04,126 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:08:04,126 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2749_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:04,126 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/test confusion matrix_table_2703_8c8b86e9dcf7c097c97e.table.json
2023-07-04 11:08:04,126 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2714_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:04,126 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2756_67f7f9c78b661281633a.table.json
2023-07-04 11:08:04,126 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2721_da75045c8d94682c0dec.table.json
2023-07-04 11:08:04,127 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2728_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:04,127 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2742_da75045c8d94682c0dec.table.json
2023-07-04 11:08:04,127 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2763_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:04,127 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2735_fe4098d4c25d56edcebd.table.json
2023-07-04 11:08:04,127 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2707_1824ba9126a6c7452162.table.json
2023-07-04 11:08:04,127 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:08:04,239 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:04,243 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,243 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,243 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,243 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,255 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,255 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,256 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,421 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:04,425 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,426 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,426 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,426 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,438 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,438 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,438 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,586 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:04,589 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,590 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,590 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,590 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,602 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,602 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,603 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,635 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:08:04,754 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:04,757 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,758 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,758 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,758 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,769 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,770 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,770 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,880 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3Mzky', 'digest': '0bddc9f582d0416f78077d326ec4d49d', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3MzY5', 'versionIndex': 69}}, 'version': 'latest'}
2023-07-04 11:08:04,880 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:08:04,880 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_502_6f3ddb785c9b58b4d785.table.json with policy now
2023-07-04 11:08:04,881 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:04,881 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:08:04,882 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:04,883 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:04,883 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:04,884 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:04,886 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:04,889 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:04,889 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:04,890 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:04,890 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:04,890 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:04,890 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:04,890 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:08:04,890 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:08:04,927 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:04,930 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,930 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,930 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,931 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,942 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,943 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:04,943 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,092 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:05,095 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,096 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,096 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,096 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,108 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,108 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,109 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,253 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:05,257 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,257 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,257 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,258 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,269 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,270 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,270 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,272 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/dsyowq8j-media/table/batch confusion matrix_table_2749_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:05,283 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/pr0687gg-media/table/test confusion matrix_table_2703_8c8b86e9dcf7c097c97e.table.json
2023-07-04 11:08:05,283 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/36ndcbpa-media/table/batch confusion matrix_table_2714_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:05,289 INFO wandb-upload_20:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/me5slq8p-media/table/batch confusion matrix_table_2707_1824ba9126a6c7452162.table.json
2023-07-04 11:08:05,290 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/h5pcrljw-media/table/batch confusion matrix_table_2763_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:05,290 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/uum9n9i1-media/table/batch confusion matrix_table_2756_67f7f9c78b661281633a.table.json
2023-07-04 11:08:05,308 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/miyls21c-media/table/batch confusion matrix_table_2735_fe4098d4c25d56edcebd.table.json
2023-07-04 11:08:05,310 INFO wandb-upload_2:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/45856587-media/table/batch confusion matrix_table_2721_da75045c8d94682c0dec.table.json
2023-07-04 11:08:05,343 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/rkqaayzo-media/table/batch confusion matrix_table_2742_da75045c8d94682c0dec.table.json
2023-07-04 11:08:05,349 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/xr0lnr3y-media/table/batch confusion matrix_table_2728_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:05,368 INFO wandb-upload_19:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/rgfz3pis-media/table/batch confusion matrix_table_502_6f3ddb785c9b58b4d785.table.json
2023-07-04 11:08:05,438 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:05,442 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,442 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,442 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,442 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,454 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,454 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,455 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,597 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:05,602 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,602 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,602 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,602 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,615 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,615 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,615 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,786 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:05,789 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,790 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,790 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,790 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,802 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,802 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,803 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,954 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:05,957 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,958 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,958 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,958 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,969 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,970 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:05,970 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,138 INFO wandb-upload_10:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpeew54fsp
2023-07-04 11:08:06,140 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:06,145 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,145 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,145 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,145 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,157 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,157 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,158 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,318 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:06,321 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,322 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,322 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,322 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,334 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,334 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,334 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,499 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:06,503 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,503 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,503 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,504 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,514 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,515 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,515 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,681 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:06,685 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,685 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,685 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,686 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:06,894 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-testconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3NDA3', 'digest': '1bbab8b967350e5f1eddc17025f19dd6', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMTE1', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk2ODk4', 'versionIndex': 1}}, 'version': 'latest'}
2023-07-04 11:08:06,894 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:08:06,894 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/test confusion matrix_table_506_fe711bb6556a1d9a19ed.table.json with policy now
2023-07-04 11:08:06,894 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:06,895 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:06,895 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:06,895 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:06,895 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:06,895 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:06,896 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:06,896 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:06,897 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:06,897 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:06,897 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:06,897 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:06,897 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:08:06,897 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:08:07,386 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/0h5obo91-media/table/test confusion matrix_table_506_fe711bb6556a1d9a19ed.table.json
2023-07-04 11:08:07,956 INFO wandb-upload_21:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpu1z1sazm
2023-07-04 11:08:08,817 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3NDI5', 'digest': 'bbfea6e1c3ac9785e93ad1a602378a3a', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3Mzky', 'versionIndex': 70}}, 'version': 'latest'}
2023-07-04 11:08:08,817 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:08:08,817 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_510_1824ba9126a6c7452162.table.json with policy now
2023-07-04 11:08:08,817 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:08,817 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:08,818 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:08,818 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:08,818 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:08,818 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:08,818 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:08,818 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:08,820 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:08,820 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:08,820 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:08,820 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:08,820 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:08,820 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:08,820 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:08,820 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:08,820 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:08,820 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:08,820 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:08,820 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:08,820 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:08,821 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:08:08,821 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:08:09,127 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/wandb-summary.json
2023-07-04 11:08:09,128 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/output.log
2023-07-04 11:08:09,128 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2805_1824ba9126a6c7452162.table.json
2023-07-04 11:08:09,128 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2826_67f7f9c78b661281633a.table.json
2023-07-04 11:08:09,128 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2840_67f7f9c78b661281633a.table.json
2023-07-04 11:08:09,128 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2777_1824ba9126a6c7452162.table.json
2023-07-04 11:08:09,128 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2868_04e8ade01248395789aa.table.json
2023-07-04 11:08:09,128 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2798_5fa4ee3184f5f47c930f.table.json
2023-07-04 11:08:09,128 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2812_e6cdd73690272b208c21.table.json
2023-07-04 11:08:09,128 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2861_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:09,129 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2770_542681021078313e717c.table.json
2023-07-04 11:08:09,130 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2833_542681021078313e717c.table.json
2023-07-04 11:08:09,130 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2791_1824ba9126a6c7452162.table.json
2023-07-04 11:08:09,130 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2854_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:09,130 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2847_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:08:09,130 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2819_790e3abd7e78db96a7a4.table.json
2023-07-04 11:08:09,130 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_created():272] file/dir created: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table/batch confusion matrix_table_2784_8a5579a03df2e76bcab5.table.json
2023-07-04 11:08:09,130 INFO Thread-12 :2443566 [dir_watcher.py:_on_file_modified():289] file/dir modified: /home/tim.kuipers/dev/deeplearning/sandbox/tims_iris_drone/wandb/run-20230704_110541-w53a3qn5/files/media/table
2023-07-04 11:08:09,295 INFO wandb-upload_14:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/oad269z7-media/table/batch confusion matrix_table_510_1824ba9126a6c7452162.table.json
2023-07-04 11:08:09,526 INFO wandb-upload_19:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/z1r5c8h7-media/table/batch confusion matrix_table_2861_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:09,527 INFO wandb-upload_8:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/l89jxii1-media/table/batch confusion matrix_table_2868_04e8ade01248395789aa.table.json
2023-07-04 11:08:09,543 INFO wandb-upload_17:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ncxxu6w8-media/table/batch confusion matrix_table_2826_67f7f9c78b661281633a.table.json
2023-07-04 11:08:09,545 INFO wandb-upload_7:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/pgj505ez-media/table/batch confusion matrix_table_2812_e6cdd73690272b208c21.table.json
2023-07-04 11:08:09,547 INFO wandb-upload_10:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/4tlr12xg-media/table/batch confusion matrix_table_2770_542681021078313e717c.table.json
2023-07-04 11:08:09,548 INFO wandb-upload_16:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/h4knaufp-media/table/batch confusion matrix_table_2805_1824ba9126a6c7452162.table.json
2023-07-04 11:08:09,558 INFO wandb-upload_6:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/kytl2z7y-media/table/batch confusion matrix_table_2777_1824ba9126a6c7452162.table.json
2023-07-04 11:08:09,559 INFO wandb-upload_4:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ier0yxym-media/table/batch confusion matrix_table_2840_67f7f9c78b661281633a.table.json
2023-07-04 11:08:09,565 INFO wandb-upload_3:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/446eo21n-media/table/batch confusion matrix_table_2798_5fa4ee3184f5f47c930f.table.json
2023-07-04 11:08:09,579 INFO wandb-upload_21:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/3sxx3lek-media/table/batch confusion matrix_table_2791_1824ba9126a6c7452162.table.json
2023-07-04 11:08:09,586 INFO wandb-upload_15:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/34ldiniz-media/table/batch confusion matrix_table_2854_e04570dcb762a9eb0c10.table.json
2023-07-04 11:08:09,591 INFO wandb-upload_11:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/ayxt4lqf-media/table/batch confusion matrix_table_2847_ac4ad9c89ab9067ca814.table.json
2023-07-04 11:08:09,594 INFO wandb-upload_1:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/jnrgexy9-media/table/batch confusion matrix_table_2833_542681021078313e717c.table.json
2023-07-04 11:08:09,595 INFO wandb-upload_9:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/9h65yirt-media/table/batch confusion matrix_table_2819_790e3abd7e78db96a7a4.table.json
2023-07-04 11:08:09,600 INFO wandb-upload_0:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/98jkec5d-media/table/batch confusion matrix_table_2784_8a5579a03df2e76bcab5.table.json
2023-07-04 11:08:09,655 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: keepalive
2023-07-04 11:08:09,846 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:09,850 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:09,893 INFO wandb-upload_22:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpmjo76vuq
2023-07-04 11:08:10,570 INFO SenderThread:2443566 [sender.py:send_request_log_artifact():1414] logged artifact run-w53a3qn5-batchconfusionmatrix_table - {'id': 'QXJ0aWZhY3Q6NTA1MTk3NDQz', 'digest': 'fa6ef188b0ff928e08d33dac84eac5ef', 'state': 'PENDING', 'aliases': [], 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjgwNjEwMDA5', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6NTA1MTk3NDI5', 'versionIndex': 71}}, 'version': 'latest'}
2023-07-04 11:08:10,570 DEBUG SenderThread:2443566 [sender.py:send():369] send: files
2023-07-04 11:08:10,570 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file media/table/batch confusion matrix_table_517_542681021078313e717c.table.json with policy now
2023-07-04 11:08:10,570 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:10,571 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:10,571 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: status_report
2023-07-04 11:08:10,571 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:10,571 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:10,571 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:10,573 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:10,574 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:10,574 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:10,574 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:10,574 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:10,574 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:10,574 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:10,574 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:10,574 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:10,574 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:10,575 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:10,575 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:10,575 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:10,575 DEBUG SenderThread:2443566 [sender.py:send():369] send: history
2023-07-04 11:08:10,575 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: summary_record
2023-07-04 11:08:10,575 INFO SenderThread:2443566 [sender.py:_save_file():1354] saving file wandb-summary.json with policy end
2023-07-04 11:08:10,575 DEBUG SenderThread:2443566 [sender.py:send():369] send: config
2023-07-04 11:08:10,575 DEBUG SenderThread:2443566 [sender.py:send_request():396] send_request: log_artifact
2023-07-04 11:08:11,028 INFO wandb-upload_12:2443566 [upload_job.py:push():133] Uploaded file /tmp/tmpkgtfr5dawandb/h5si2xs2-media/table/batch confusion matrix_table_517_542681021078313e717c.table.json
2023-07-04 11:08:11,147 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,147 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,148 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,306 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:11,311 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,311 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,311 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,311 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,323 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,324 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,324 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,506 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: log_artifact
2023-07-04 11:08:11,510 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,510 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,511 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,511 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,522 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,522 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,522 DEBUG HandlerThread:2443566 [handler.py:handle_request():144] handle_request: partial_history
2023-07-04 11:08:11,626 INFO wandb-upload_20:2443566 [upload_job.py:push():91] Uploaded file /home/tim.kuipers/.local/share/wandb/artifacts/staging/tmpc8n075
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment