Skip to content

Instantly share code, notes, and snippets.

@kuanb

kuanb/consolidated_worker_sched_logs_2.txt

Last active July 5, 2017 23:05
Show Gist options
  • Save kuanb/a71f001002c86f0aaa2915c2dcd4c70a to your computer and use it in GitHub Desktop.
Save kuanb/a71f001002c86f0aaa2915c2dcd4c70a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
consolidated_worker_sched_logs_2.txt
Note: Looks like workers completely crashed, were restarted (by the Docker container within which they were running) and then got out of sync (?).
...
Jul 05 15:14:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Register tcp://10.0.0.132:38621
Jul 05 15:14:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Register tcp://10.0.0.132:39862
Jul 05 15:14:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Starting worker compute stream, tcp://10.0.0.132:42596
Jul 05 15:14:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Starting worker compute stream, tcp://10.0.0.132:38847
Jul 05 15:14:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Starting worker compute stream, tcp://10.0.0.132:38621
Jul 05 15:14:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Starting worker compute stream, tcp://10.0.0.132:39862
Jul 05 15:14:10 ip-10-0-0-216 dhclient: DHCPREQUEST of 10.0.0.216 on ens3 to 10.0.0.1 port 67 (xid=0x62124586)
Jul 05 15:14:10 ip-10-0-0-216 dhclient: DHCPACK of 10.0.0.216 from 10.0.0.1
Jul 05 15:14:10 ip-10-0-0-216 dhclient: bound to 10.0.0.216 -- renewal in 1437 seconds.
Jul 05 15:15:26 ip-10-0-0-116 dhclient: DHCPREQUEST of 10.0.0.116 on ens3 to 10.0.0.1 port 67 (xid=0x6b1ef41)
Jul 05 15:15:26 ip-10-0-0-116 dhclient: DHCPACK of 10.0.0.116 from 10.0.0.1
Jul 05 15:15:26 ip-10-0-0-116 dhclient: bound to 10.0.0.116 -- renewal in 1547 seconds.
Jul 05 15:15:57 ip-10-0-0-194 dhclient: DHCPREQUEST of 10.0.0.194 on ens3 to 10.0.0.1 port 67 (xid=0x3e5314bd)
Jul 05 15:15:57 ip-10-0-0-194 dhclient: DHCPACK of 10.0.0.194 from 10.0.0.1
Jul 05 15:15:57 ip-10-0-0-194 dhclient: bound to 10.0.0.194 -- renewal in 1390 seconds.
Jul 05 15:16:26 ip-10-0-0-35 /usr/lib/snapd/snapd: snapmgr.go:496: DEBUG: Next refresh scheduled for 2017-07-05 22:16:26.718125165 +0000 UTC.
Jul 05 15:16:27 ip-10-0-0-35 /usr/lib/snapd/snapd: snapmgr.go:422: No snaps to auto-refresh found
Jul 05 15:16:27 ip-10-0-0-35 snapd: 2017/07/05 22:16:27.405638 snapmgr.go:422: No snaps to auto-refresh found
Jul 05 15:16:56 ip-10-0-0-132 /usr/lib/snapd/snapd: snapmgr.go:496: DEBUG: Next refresh scheduled for 2017-07-05 22:16:56.800676019 +0000 UTC.
Jul 05 15:16:57 ip-10-0-0-132 /usr/lib/snapd/snapd: snapmgr.go:422: No snaps to auto-refresh found
Jul 05 15:16:57 ip-10-0-0-132 snapd: 2017/07/05 22:16:57.470774 snapmgr.go:422: No snaps to auto-refresh found
Jul 05 15:17:01 ip-10-0-0-248 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-248 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-248 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-157 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-157 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-157 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-164 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-164 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-164 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-194 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-194 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-89 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-89 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-194 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-89 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-57 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-57 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-57 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-116 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-116 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-116 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-130 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-130 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-130 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-29 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-29 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-29 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-60 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-60 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-60 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-147 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-147 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-147 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-216 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-216 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-216 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-35 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-35 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-35 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-95 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-95 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-95 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-252 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-252 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-252 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-106 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-106 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-106 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-132 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-132 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-251 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-251 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-251 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:17:01 ip-10-0-0-98 CRON: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 05 15:17:01 ip-10-0-0-98 CRON: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 05 15:17:01 ip-10-0-0-98 CRON: pam_unix(cron:session): session closed for user root
Jul 05 15:19:29 ip-10-0-0-157 dhclient: DHCPREQUEST of 10.0.0.157 on ens3 to 10.0.0.1 port 67 (xid=0x3d5886cb)
Jul 05 15:19:29 ip-10-0-0-157 dhclient: DHCPACK of 10.0.0.157 from 10.0.0.1
Jul 05 15:19:29 ip-10-0-0-157 dhclient: bound to 10.0.0.157 -- renewal in 1537 seconds.
Jul 05 15:20:45 ip-10-0-0-106 dhclient: DHCPREQUEST of 10.0.0.106 on ens3 to 10.0.0.1 port 67 (xid=0x6161a5df)
Jul 05 15:20:45 ip-10-0-0-106 dhclient: DHCPACK of 10.0.0.106 from 10.0.0.1
Jul 05 15:20:45 ip-10-0-0-106 dhclient: bound to 10.0.0.106 -- renewal in 1550 seconds.
Jul 05 15:21:25 ip-10-0-0-35 /usr/lib/snapd/snapd: snapmgr.go:496: DEBUG: Next refresh scheduled for 2017-07-06 01:34:14.230918497 +0000 UTC.
Jul 05 15:21:55 ip-10-0-0-132 /usr/lib/snapd/snapd: snapmgr.go:496: DEBUG: Next refresh scheduled for 2017-07-06 00:23:34.684044678 +0000 UTC.
Jul 05 15:22:04 ip-10-0-0-130 dhclient: DHCPREQUEST of 10.0.0.130 on ens3 to 10.0.0.1 port 67 (xid=0x18d71104)
Jul 05 15:22:04 ip-10-0-0-130 dhclient: DHCPACK of 10.0.0.130 from 10.0.0.1
Jul 05 15:22:04 ip-10-0-0-130 dhclient: bound to 10.0.0.130 -- renewal in 1618 seconds.
Jul 05 15:23:27 ip-10-0-0-29 dhclient: DHCPREQUEST of 10.0.0.29 on ens3 to 10.0.0.1 port 67 (xid=0x6c02b633)
Jul 05 15:23:27 ip-10-0-0-29 dhclient: DHCPACK of 10.0.0.29 from 10.0.0.1
Jul 05 15:23:27 ip-10-0-0-29 dhclient: bound to 10.0.0.29 -- renewal in 1713 seconds.
Jul 05 15:24:54 ip-10-0-0-60 dhclient: DHCPREQUEST of 10.0.0.60 on ens3 to 10.0.0.1 port 67 (xid=0x60e02115)
Jul 05 15:24:54 ip-10-0-0-60 dhclient: DHCPACK of 10.0.0.60 from 10.0.0.1
Jul 05 15:24:54 ip-10-0-0-60 dhclient: bound to 10.0.0.60 -- renewal in 1764 seconds.
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Remove worker tcp://10.0.0.132:42596
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Removed worker tcp://10.0.0.132:42596
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Remove worker tcp://10.0.0.132:39862
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Removed worker tcp://10.0.0.132:39862
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Remove worker tcp://10.0.0.132:38621
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Removed worker tcp://10.0.0.132:38621
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Remove worker tcp://10.0.0.132:38847
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - INFO - Removed worker tcp://10.0.0.132:38847
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-132 adoring_sinoussi: distributed.worker - INFO - Close compute stream
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: distributed.scheduler - ERROR - Workers don't have promised keys. This should never occur
Jul 05 15:26:06 ip-10-0-0-164 trusting_swanson: NoneType
Jul 05 15:26:06 ip-10-0-0-132 adoring_sinoussi: distributed.worker - INFO - Stopping worker at tcp://10.0.0.132:39862
Jul 05 15:26:06 ip-10-0-0-132 adoring_sinoussi: distributed.worker - INFO - Close compute stream
Jul 05 15:26:06 ip-10-0-0-132 adoring_sinoussi: distributed.worker - INFO - Stopping worker at tcp://10.0.0.132:38621
Jul 05 15:26:06 ip-10-0-0-132 adoring_sinoussi: distributed.worker - INFO - Stopping worker at tcp://10.0.0.132:38847
Jul 05 15:26:06 ip-10-0-0-132 adoring_sinoussi: distributed.worker - INFO - Close compute stream
Jul 05 15:26:06 ip-10-0-0-132 adoring_sinoussi: distributed.worker - INFO - Close compute stream
Jul 05 15:26:06 ip-10-0-0-132 adoring_sinoussi: distributed.worker - INFO - Comm closed
Jul 05 15:26:06 ip-10-0-0-132 adoring_sinoussi: distributed.nanny - INFO - Closing Nanny at 'tcp://10.0.0.132:34143'
Jul 05 15:26:06 ip-10-0-0-132 adoring_sinoussi: distributed.nanny - INFO - Closing Nanny at 'tcp://10.0.0.132:43542'
Jul 05 15:26:06 ip-10-0-0-132 adoring_sinoussi: distributed.nanny - INFO - Closing Nanny at 'tcp://10.0.0.132:38938'
Jul 05 15:26:06 ip-10-0-0-132 adoring_sinoussi: distributed.nanny - INFO - Closing Nanny at 'tcp://10.0.0.132:34920'
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Jul 05 15:26:12 ip-10-0-0-132 adoring_sinoussi: distributed.dask_worker - INFO - End worker
Raw
local_traceback.txt
$ time python test_dask.py 10000
Partitions: 2000
Beginning computation...
distributed.utils - ERROR - Could not find dependent ('assign-8e22812327e097546d36918e3ff74c83', 1584). Check worker logs
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/distributed/utils.py", line 223, in f
result[0] = yield make_coro()
File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/usr/local/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1063, in run
yielded = self.gen.throw(*exc_info)
File "/usr/local/lib/python3.5/site-packages/distributed/client.py", line 1156, in _gather
traceback)
File "/usr/local/lib/python3.5/site-packages/six-1.10.0-py3.5.egg/six.py", line 686, in reraise
raise value
ValueError: Could not find dependent ('assign-8e22812327e097546d36918e3ff74c83', 1584). Check worker logs
Traceback (most recent call last):
File "test_dask.py", line 63, in <module>
computed = distances.compute()
File "/usr/local/lib/python3.5/site-packages/dask/base.py", line 97, in compute
(result,) = compute(self, traverse=False, **kwargs)
File "/usr/local/lib/python3.5/site-packages/dask/base.py", line 204, in compute
results = get(dsk, keys, **kwargs)
File "/usr/local/lib/python3.5/site-packages/distributed/client.py", line 1764, in get
results = self.gather(packed)
File "/usr/local/lib/python3.5/site-packages/distributed/client.py", line 1263, in gather
direct=direct)
File "/usr/local/lib/python3.5/site-packages/distributed/client.py", line 489, in sync
return sync(self.loop, func, *args, **kwargs)
File "/usr/local/lib/python3.5/site-packages/distributed/utils.py", line 234, in sync
six.reraise(*error[0])
File "/usr/local/lib/python3.5/site-packages/six-1.10.0-py3.5.egg/six.py", line 686, in reraise
raise value
File "/usr/local/lib/python3.5/site-packages/distributed/utils.py", line 223, in f
result[0] = yield make_coro()
File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
value = future.result()
File "/usr/local/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1063, in run
yielded = self.gen.throw(*exc_info)
File "/usr/local/lib/python3.5/site-packages/distributed/client.py", line 1156, in _gather
traceback)
File "/usr/local/lib/python3.5/site-packages/six-1.10.0-py3.5.egg/six.py", line 686, in reraise
raise value
ValueError: Could not find dependent ('assign-8e22812327e097546d36918e3ff74c83', 1584). Check worker logs
Raw
local_traceback_2.txt
time python test_dask.py 10000
Partitions: 2000
Beginning computation...
distributed.client - WARNING - Couldn't gather keys {"('drop-908546b6c2fc92a9770686e5990eeea9', 359)": ['tcp://10.0.0.132:39862'], "('drop-908546b6c2fc92a9770686e5990eeea9', 787)": ['tcp://10.0.0.132:38621'], "('drop-908546b6c2fc92a9770686e5990eeea9', 49)": ['tcp://10.0.0.132:38847'], "('drop-908546b6c2fc92a9770686e5990eeea9', 951)": ['tcp://10.0.0.132:42596'], "('drop-908546b6c2fc92a9770686e5990eeea9', 477)": ['tcp://10.0.0.132:39862'], "('drop-908546b6c2fc92a9770686e5990eeea9', 705)": ['tcp://10.0.0.132:38847'], "('drop-908546b6c2fc92a9770686e5990eeea9', 603)": ['tcp://10.0.0.132:38621'], "('drop-908546b6c2fc92a9770686e5990eeea9', 500)": ['tcp://10.0.0.132:39862'], "('drop-908546b6c2fc92a9770686e5990eeea9', 944)": ['tcp://10.0.0.132:38621'], "('drop-908546b6c2fc92a9770686e5990eeea9', 1986)": ['tcp://10.0.0.132:38621'], "('drop-908546b6c2fc92a9770686e5990eeea9', 1997)": ['tcp://10.0.0.132:38621'], "('drop-908546b6c2fc92a9770686e5990eeea9', 685)": ['tcp://10.0.0.132:38847'], "('drop-908546b6c2fc92a9770686e5990eeea9', 56)": ['tcp://10.0.0.132:39862'], "('drop-908546b6c2fc92a9770686e5990eeea9', 1741)": ['tcp://10.0.0.132:38847'], "('drop-908546b6c2fc92a9770686e5990eeea9', 837)": ['tcp://10.0.0.132:38847'], "('drop-908546b6c2fc92a9770686e5990eeea9', 83)": ['tcp://10.0.0.132:38621'], "('drop-908546b6c2fc92a9770686e5990eeea9', 270)": ['tcp://10.0.0.132:39862'], "('drop-908546b6c2fc92a9770686e5990eeea9', 766)": ['tcp://10.0.0.132:42596'], "('drop-908546b6c2fc92a9770686e5990eeea9', 860)": ['tcp://10.0.0.132:42596'], "('drop-908546b6c2fc92a9770686e5990eeea9', 889)": ['tcp://10.0.0.132:38847'], "('drop-908546b6c2fc92a9770686e5990eeea9', 746)": ['tcp://10.0.0.132:39862']}
Computation completed.
id_from id_to distance
0 14856 14856 0.000000
1 14856 8716 17777.332006
2 14856 661 35648.613342
3 14856 709 40575.208147
4 14856 717 33844.863950
5 14856 733 33766.449677
6 14856 1107 34949.987615
7 14856 15349 945.888256
8 14856 14750 65.493771
9 14856 27507 32522.199075
real 28m4.636s
user 2m21.250s
sys 1m23.560s
Raw
nonresponsive_worker.txt
am_closed_error
Jul 05 16:04:58 ip-10-0-0-106 agitated_knuth: raise CommClosedError("%s: %s" % (exc.__class__.__name__, exc))
Jul 05 16:04:58 ip-10-0-0-106 agitated_knuth: distributed.comm.core.CommClosedError: ConnectionResetError: [Errno 104] Connection reset by peer
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: distributed.worker - ERROR - Worker stream died during communication: tcp://10.0.0.132:43562
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: Traceback (most recent call last):
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/distributed/comm/tcp.py", line 160, in read
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: frame = yield stream.read_bytes(length)
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: value = future.result()
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: raise_exc_info(self._exc_info)
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "<string>", line 4, in raise_exc_info
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: tornado.iostream.StreamClosedError: Stream is closed
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: During handling of the above exception, another exception occurred:
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: Traceback (most recent call last):
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/distributed/worker.py", line 1589, in gather_dep
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: who=self.address)
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: value = future.result()
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: raise_exc_info(self._exc_info)
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "<string>", line 4, in raise_exc_info
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1063, in run
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: yielded = self.gen.throw(*exc_info)
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/distributed/core.py", line 475, in send_recv_from_rpc
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: result = yield send_recv(comm=comm, op=key, **kwargs)
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: value = future.result()
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: raise_exc_info(self._exc_info)
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "<string>", line 4, in raise_exc_info
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1063, in run
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: yielded = self.gen.throw(*exc_info)
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/distributed/core.py", line 310, in send_recv
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: response = yield comm.read()
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: value = future.result()
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: raise_exc_info(self._exc_info)
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "<string>", line 4, in raise_exc_info
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1063, in run
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: yielded = self.gen.throw(*exc_info)
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/distributed/comm/tcp.py", line 166, in read
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: convert_stream_closed_error(e)
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: File "/usr/local/lib/python3.5/site-packages/distributed/comm/tcp.py", line 104, in convert_stream_closed_error
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: raise CommClosedError("%s: %s" % (exc.__class__.__name__, exc))
Jul 05 16:05:00 ip-10-0-0-35 lucid_hopper: distributed.comm.core.CommClosedError: ConnectionResetError: [Errno 104] Connection reset by peer
Raw
worker_logs.txt
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: distributed.worker - ERROR - Worker stream died during communication: tcp://10.0.0.148:40972
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: Traceback (most recent call last):
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/distributed/comm/tcp.py", line 160, in read
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: frame = yield stream.read_bytes(length)
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: value = future.result()
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: raise_exc_info(self._exc_info)
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "<string>", line 4, in raise_exc_info
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: tornado.iostream.StreamClosedError: Stream is closed
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: During handling of the above exception, another exception occurred:
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: Traceback (most recent call last):
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/distributed/worker.py", line 1589, in gather_dep
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: who=self.address)
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: value = future.result()
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: raise_exc_info(self._exc_info)
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "<string>", line 4, in raise_exc_info
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1063, in run
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: yielded = self.gen.throw(*exc_info)
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/distributed/core.py", line 475, in send_recv_from_rpc
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: result = yield send_recv(comm=comm, op=key, **kwargs)
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: value = future.result()
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: raise_exc_info(self._exc_info)
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "<string>", line 4, in raise_exc_info
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1063, in run
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: yielded = self.gen.throw(*exc_info)
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/distributed/core.py", line 310, in send_recv
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: response = yield comm.read()
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: value = future.result()
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: raise_exc_info(self._exc_info)
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "<string>", line 4, in raise_exc_info
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/tornado/gen.py", line 1063, in run
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: yielded = self.gen.throw(*exc_info)
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/distributed/comm/tcp.py", line 166, in read
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: convert_stream_closed_error(e)
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: File "/usr/local/lib/python3.5/site-packages/distributed/comm/tcp.py", line 104, in convert_stream_closed_error
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: raise CommClosedError("%s: %s" % (exc.__class__.__name__, exc))
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: distributed.comm.core.CommClosedError: ConnectionResetError: [Errno 104] Connection reset by peer
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: distributed.worker - INFO - Can't find dependencies for key ('apply-e2822a499b43efe00d82019c048e976d', 53)
Jul 05 12:00:37 ip-10-0-0-140 youthful_gates: distributed.worker - INFO - Dependent not found: ('assign-8e22812327e097546d36918e3ff74c83', 53) 2 . Asking scheduler
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment