On the zone:
[root@ds (us-west-agc:bnbntl) ~]# mdata-put set:state success
plat_recv timeout
receive timeout, resetting protocol...
plat_recv timeout
receive timeout, resetting protocol...
plat_recv timeout
receive timeout, resetting protocol...
plat_recv timeout
receive timeout, resetting protocol...
From the metadata
service log on the CN global:
[2021-03-04T22:07:43.251Z] DEBUG: metadata/484186 on cn01: createServersOnExistingZones for 19 zones
[2021-03-04T22:07:43.251Z] INFO: metadata/484186 on cn01: created zone metadata sockets on 0 / 19 zones took 0.000272372s (272.37us)
[2021-03-04T22:08:20.371Z] ERROR: metadata/484186 on cn01: could not set metadata (key "set:state") (brand=joyent, zonename=33bbd39b-752c-425e-8d5f-a600773fe966)
MultiError: first of 1 error: first of 1 error: vminfod watchForEvent "VM.js writeAllMetadata (33bbd39b-752c-425e-8d5f-a600773fe966: customer_metadata,internal_metadata,tags)" timeout exceeded
at /usr/vm/node_modules/vasync/lib/vasync.js:99:14
at /usr/vm/node_modules/VM.js:16887:29
at /usr/vm/node_modules/VM.js:17072:13
at /usr/node/0.10/node_modules/async.js:240:13
at /usr/node/0.10/node_modules/async.js:144:21
at /usr/node/0.10/node_modules/async.js:237:17
at /usr/node/0.10/node_modules/async.js:600:34
at /usr/vm/node_modules/VM.js:17014:21
at /usr/vm/node_modules/VM.js:3690:9
at next (/usr/vm/node_modules/vasync/lib/vasync.js:843:6)
Caused by: MultiError: first of 1 error: vminfod watchForEvent "VM.js writeAllMetadata (33bbd39b-752c-425e-8d5f-a600773fe966: customer_metadata,internal_metadata,tags)" timeout exceeded
at /usr/vm/node_modules/vasync/lib/vasync.js:99:14
at done (/usr/vm/node_modules/vminfod/client.js:697:9)
at vminfodEventStreamWatchForEventTimeout (/usr/vm/node_modules/vminfod/client.js:635:13)
at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
Caused by: Error: vminfod watchForEvent "VM.js writeAllMetadata (33bbd39b-752c-425e-8d5f-a600773fe966: customer_metadata,internal_metadata,tags)" timeout exceeded
at vminfodEventStreamWatchForEventTimeout (/usr/vm/node_modules/vminfod/client.js:633:23)
at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
[2021-03-04T22:08:20.371Z] ERROR: metadata/484186 on cn01: first of 1 error: first of 1 error: vminfod watchForEvent "VM.js writeAllMetadata (33bbd39b-752c-425e-8d5f-a600773fe966: customer_metadata,internal_metadata,tags)" timeout exceeded (brand=joyent, zonename=33bbd39b-752c-425e-8d5f-a600773fe966)
[2021-03-04T22:08:20.371Z] ERROR: metadata/484186 on cn01: Socket for 33bbd39b-752c-425e-8d5f-a600773fe966 closed before we could write anything. (brand=joyent, zonename=33bbd39b-752c-425e-8d5f-a600773fe966)
[2021-03-04T22:08:20.371Z] INFO: metadata/484186 on cn01: handled PUT set:state (brand=joyent, zonename=33bbd39b-752c-425e-8d5f-a600773fe966, response=FAILURE)
MultiError: first of 1 error: first of 1 error: vminfod watchForEvent "VM.js writeAllMetadata (33bbd39b-752c-425e-8d5f-a600773fe966: customer_metadata,internal_metadata,tags)" timeout exceeded
at /usr/vm/node_modules/vasync/lib/vasync.js:99:14
at /usr/vm/node_modules/VM.js:16887:29
at /usr/vm/node_modules/VM.js:17072:13
at /usr/node/0.10/node_modules/async.js:240:13
at /usr/node/0.10/node_modules/async.js:144:21
at /usr/node/0.10/node_modules/async.js:237:17
at /usr/node/0.10/node_modules/async.js:600:34
at /usr/vm/node_modules/VM.js:17014:21
at /usr/vm/node_modules/VM.js:3690:9
at next (/usr/vm/node_modules/vasync/lib/vasync.js:843:6)
Caused by: MultiError: first of 1 error: vminfod watchForEvent "VM.js writeAllMetadata (33bbd39b-752c-425e-8d5f-a600773fe966: customer_metadata,internal_metadata,tags)" timeout exceeded
at /usr/vm/node_modules/vasync/lib/vasync.js:99:14
at done (/usr/vm/node_modules/vminfod/client.js:697:9)
at vminfodEventStreamWatchForEventTimeout (/usr/vm/node_modules/vminfod/client.js:635:13)
at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
Caused by: Error: vminfod watchForEvent "VM.js writeAllMetadata (33bbd39b-752c-425e-8d5f-a600773fe966: customer_metadata,internal_metadata,tags)" timeout exceeded
at vminfodEventStreamWatchForEventTimeout (/usr/vm/node_modules/vminfod/client.js:633:23)
at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
--
query: {
"cmd": "PUT",
"arg": "set:state",
"req_v": 2,
"elapsed": 60099.073655
}
[2021-03-04T22:11:43.261Z] DEBUG: metadata/484186 on cn01: createServersOnExistingZones for 19 zones
[2021-03-04T22:11:43.261Z] INFO: metadata/484186 on cn01: created zone metadata sockets on 0 / 19 zones took 0.000291621s (291.62us)
[2021-03-04T22:11:50.499Z] ERROR: metadata/484186 on cn01: Socket for 33bbd39b-752c-425e-8d5f-a600773fe966 closed before we could write anything. (brand=joyent, zonename=33bbd39b-752c-425e-8d5f-a600773fe966)
If I cancel the
mdata-put
command while it's hanging I can see that the value does actually get written tometadata.json
in the global but then running amdata-get
command does not return the value that was written:Then after a little while it does return the value: