Triton metadata put error

triton_metadata_put_error.md

On the zone:

[root@ds (us-west-agc:bnbntl) ~]# mdata-put set:state success
plat_recv timeout
receive timeout, resetting protocol...
plat_recv timeout
receive timeout, resetting protocol...
plat_recv timeout
receive timeout, resetting protocol...
plat_recv timeout
receive timeout, resetting protocol...

From the metadata service log on the CN global:

[2021-03-04T22:07:43.251Z] DEBUG: metadata/484186 on cn01: createServersOnExistingZones for 19 zones
[2021-03-04T22:07:43.251Z]  INFO: metadata/484186 on cn01: created zone metadata sockets on 0 / 19 zones took 0.000272372s (272.37us)
[2021-03-04T22:08:20.371Z] ERROR: metadata/484186 on cn01: could not set metadata (key "set:state") (brand=joyent, zonename=33bbd39b-752c-425e-8d5f-a600773fe966)
    MultiError: first of 1 error: first of 1 error: vminfod watchForEvent "VM.js writeAllMetadata (33bbd39b-752c-425e-8d5f-a600773fe966: customer_metadata,internal_metadata,tags)" timeout exceeded
        at /usr/vm/node_modules/vasync/lib/vasync.js:99:14
        at /usr/vm/node_modules/VM.js:16887:29
        at /usr/vm/node_modules/VM.js:17072:13
        at /usr/node/0.10/node_modules/async.js:240:13
        at /usr/node/0.10/node_modules/async.js:144:21
        at /usr/node/0.10/node_modules/async.js:237:17
        at /usr/node/0.10/node_modules/async.js:600:34
        at /usr/vm/node_modules/VM.js:17014:21
        at /usr/vm/node_modules/VM.js:3690:9
        at next (/usr/vm/node_modules/vasync/lib/vasync.js:843:6)
    Caused by: MultiError: first of 1 error: vminfod watchForEvent "VM.js writeAllMetadata (33bbd39b-752c-425e-8d5f-a600773fe966: customer_metadata,internal_metadata,tags)" timeout exceeded
        at /usr/vm/node_modules/vasync/lib/vasync.js:99:14
        at done (/usr/vm/node_modules/vminfod/client.js:697:9)
        at vminfodEventStreamWatchForEventTimeout (/usr/vm/node_modules/vminfod/client.js:635:13)
        at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
    Caused by: Error: vminfod watchForEvent "VM.js writeAllMetadata (33bbd39b-752c-425e-8d5f-a600773fe966: customer_metadata,internal_metadata,tags)" timeout exceeded
        at vminfodEventStreamWatchForEventTimeout (/usr/vm/node_modules/vminfod/client.js:633:23)
        at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
[2021-03-04T22:08:20.371Z] ERROR: metadata/484186 on cn01: first of 1 error: first of 1 error: vminfod watchForEvent "VM.js writeAllMetadata (33bbd39b-752c-425e-8d5f-a600773fe966: customer_metadata,internal_metadata,tags)" timeout exceeded (brand=joyent, zonename=33bbd39b-752c-425e-8d5f-a600773fe966)
[2021-03-04T22:08:20.371Z] ERROR: metadata/484186 on cn01: Socket for 33bbd39b-752c-425e-8d5f-a600773fe966 closed before we could write anything. (brand=joyent, zonename=33bbd39b-752c-425e-8d5f-a600773fe966)
[2021-03-04T22:08:20.371Z]  INFO: metadata/484186 on cn01: handled PUT set:state (brand=joyent, zonename=33bbd39b-752c-425e-8d5f-a600773fe966, response=FAILURE)
    MultiError: first of 1 error: first of 1 error: vminfod watchForEvent "VM.js writeAllMetadata (33bbd39b-752c-425e-8d5f-a600773fe966: customer_metadata,internal_metadata,tags)" timeout exceeded
        at /usr/vm/node_modules/vasync/lib/vasync.js:99:14
        at /usr/vm/node_modules/VM.js:16887:29
        at /usr/vm/node_modules/VM.js:17072:13
        at /usr/node/0.10/node_modules/async.js:240:13
        at /usr/node/0.10/node_modules/async.js:144:21
        at /usr/node/0.10/node_modules/async.js:237:17
        at /usr/node/0.10/node_modules/async.js:600:34
        at /usr/vm/node_modules/VM.js:17014:21
        at /usr/vm/node_modules/VM.js:3690:9
        at next (/usr/vm/node_modules/vasync/lib/vasync.js:843:6)
    Caused by: MultiError: first of 1 error: vminfod watchForEvent "VM.js writeAllMetadata (33bbd39b-752c-425e-8d5f-a600773fe966: customer_metadata,internal_metadata,tags)" timeout exceeded
        at /usr/vm/node_modules/vasync/lib/vasync.js:99:14
        at done (/usr/vm/node_modules/vminfod/client.js:697:9)
        at vminfodEventStreamWatchForEventTimeout (/usr/vm/node_modules/vminfod/client.js:635:13)
        at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
    Caused by: Error: vminfod watchForEvent "VM.js writeAllMetadata (33bbd39b-752c-425e-8d5f-a600773fe966: customer_metadata,internal_metadata,tags)" timeout exceeded
        at vminfodEventStreamWatchForEventTimeout (/usr/vm/node_modules/vminfod/client.js:633:23)
        at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
    --
    query: {
      "cmd": "PUT",
      "arg": "set:state",
      "req_v": 2,
      "elapsed": 60099.073655
    }
[2021-03-04T22:11:43.261Z] DEBUG: metadata/484186 on cn01: createServersOnExistingZones for 19 zones
[2021-03-04T22:11:43.261Z]  INFO: metadata/484186 on cn01: created zone metadata sockets on 0 / 19 zones took 0.000291621s (291.62us)
[2021-03-04T22:11:50.499Z] ERROR: metadata/484186 on cn01: Socket for 33bbd39b-752c-425e-8d5f-a600773fe966 closed before we could write anything. (brand=joyent, zonename=33bbd39b-752c-425e-8d5f-a600773fe966)

If I cancel the mdata-put command while it's hanging I can see that the value does actually get written to metadata.json in the global but then running a mdata-get command does not return the value that was written:

[root@cn01 (us-west-agc) /zones/33bbd39b-752c-425e-8d5f-a600773fe966/config]# grep foo metadata.json 
    "foo": "bar"

[root@ds (us-west-agc:bnbntl) ~]# mdata-get foo
No metadata for 'foo'

Then after a little while it does return the value:

[root@ds (us-west-agc:bnbntl) ~]# mdata-get foo
bar