Triton Docker Copy Issues

triton_docker_copy.md

Chads-MacBook-Pro:certs chad$ triton-docker --tls cp cert.crt fw:/etc/nginx/

It hangs for a long time and I see this in the docker service log:

[2021-12-20T15:45:21.837Z]  INFO: docker/55916 on 63174fb7-7a4f-4bb3-8db2-f0909a50c5df:  (req_id=3f9e19fb-e7d3-41cd-ba13-b08eb94b3e01)
    evt: {
      "ph": "b",
      "name": "containerstatarchive",
      "req_seq": 1640015121827001
    }
[2021-12-20T15:45:21.975Z] DEBUG: docker/55916 on 63174fb7-7a4f-4bb3-8db2-f0909a50c5df: Found 1 VMs (req_id=3f9e19fb-e7d3-41cd-ba13-b08eb94b3e01, account=c2b6c7a5-1643-e0fd-bb3a-8fbf77fc8d61, login=chad)
[2021-12-20T15:45:21.977Z] DEBUG: docker/55916 on 63174fb7-7a4f-4bb3-8db2-f0909a50c5df: req (req_id=3f9e19fb-e7d3-41cd-ba13-b08eb94b3e01, account=c2b6c7a5-1643-e0fd-bb3a-8fbf77fc8d61, login=chad, req.remoteAddress=::ffff:192.168.0.30, req.remotePort=53708)
    HEAD /v1.24/containers/fw/archive?path=%2Fetc%2Fnginx%2F HTTP/1.1
    host: 192.168.0.202:2376
    user-agent: Docker-Client/1.12.6 (darwin)
[2021-12-20T15:45:22.177Z] DEBUG: docker/55916 on 63174fb7-7a4f-4bb3-8db2-f0909a50c5df: createVmWaitTicket: waitlist ticket created successfully (req_id=3f9e19fb-e7d3-41cd-ba13-b08eb94b3e01, account=c2b6c7a5-1643-e0fd-bb3a-8fbf77fc8d61, login=chad, uuid=6a2883f1-7086-c9ac-a896-c3b416b0f05e)
[2021-12-20T15:45:22.184Z] DEBUG: docker/55916 on 63174fb7-7a4f-4bb3-8db2-f0909a50c5df: archiveStream: acquiring waitlist ticket (req_id=3f9e19fb-e7d3-41cd-ba13-b08eb94b3e01, account=c2b6c7a5-1643-e0fd-bb3a-8fbf77fc8d61, login=chad, mode=stat, path=/etc/nginx/)

Then it finishes but does not succeed:

Chads-MacBook-Pro:certs chad$ triton-docker --tls cp cert.crt fw:/etc/nginx/
no such directory

And I see this error in the docker service log:

[2021-12-20T15:46:51.868Z]  INFO: docker/ufds/55916 on 63174fb7-7a4f-4bb3-8db2-f0909a50c5df: LDAP client gone idle
[2021-12-20T15:46:51.872Z]  INFO: docker/ufds/55916 on 63174fb7-7a4f-4bb3-8db2-f0909a50c5df: LDAP client disconnected
[2021-12-20T15:49:21.335Z] DEBUG: docker/55916 on 63174fb7-7a4f-4bb3-8db2-f0909a50c5df: waitOnVMWaitTicket: waitlist ticket acquired (req_id=3f9e19fb-e7d3-41cd-ba13-b08eb94b3e01, account=c2b6c7a5-1643-e0fd-bb3a-8fbf77fc8d61, login=chad, ticket={})
[2021-12-20T15:49:21.336Z] DEBUG: docker/55916 on 63174fb7-7a4f-4bb3-8db2-f0909a50c5df: archiveStream: calling dockerCopy (req_id=3f9e19fb-e7d3-41cd-ba13-b08eb94b3e01, account=c2b6c7a5-1643-e0fd-bb3a-8fbf77fc8d61, login=chad, mode=stat, path=/etc/nginx/)
Uncaught Error: connect EADDRNOTAVAIL 10.88.88.7 - Local (0.0.0.0:0)

FROM
Socket.emit (events.js:160:7)
emitErrorNT (net.js:1290:8)
_combinedTickCallback (internal/process/next_tick.js:80:11)
process._tickDomainCallback (internal/process/next_tick.js:128:9)
[ Dec 20 15:49:21 Stopping because all processes in service exited. ]
[ Dec 20 15:49:21 Executing stop method (:kill). ]
[ Dec 20 15:49:21 Executing start method ("/opt/smartdc/docker/smf/method/docker start"). ]
+ . /lib/svc/share/smf_include.sh
++ SMF_EXIT_OK=0
++ SMF_EXIT_NODAEMON=94
++ SMF_EXIT_ERR_FATAL=95
++ SMF_EXIT_ERR_CONFIG=96
++ SMF_EXIT_MON_DEGRADE=97
++ SMF_EXIT_MON_OFFLINE=98
++ SMF_EXIT_ERR_NOSMF=99
++ SMF_EXIT_ERR_PERM=100
+ PATH=/usr/sbin:/usr/bin
+ export PATH
+ case "$1" in
+ exit 0
+ /usr/bin/ctrun -l child -o noorphan /opt/smartdc/docker/build/node/bin/node --abort_on_uncaught_exception /opt/smartdc/docker/lib/docker.js
[ Dec 20 15:49:21 Method "start" exited with status 0. ]
[2021-12-20T15:49:22.362Z]  INFO: docker/57712 on 63174fb7-7a4f-4bb3-8db2-f0909a50c5df: Loading config from "/opt/smartdc/docker/etc/config.json"
[2021-12-20T15:49:22.385Z] DEBUG: docker/57712 on 63174fb7-7a4f-4bb3-8db2-f0909a50c5df: Creating moray client (host=moray.neptune.chadschmelter.com, noCache=true, port=2020, reconnect=true)
    dns: {
      "resolvers": [
        "binder.neptune.chadschmelter.com"
      ]
    }

Other times I get this instead which fails quicker:

Chads-MacBook-Pro:certs chad$ triton-docker --tls cp cert.crt fw:/etc/nginx/
An error occurred trying to connect: Put https://192.168.0.202:2376/v1.24/containers/fw/archive?noOverwriteDirNonDir=true&path=%2Fetc%2Fnginx%2F: EOF

Here is what I found in the cn-agent log when I encounter the An error occurred trying to connect error:

[2021-12-20T20:11:41.488Z]  WARN: cn-agent/71190 on headnode: Desired uuid was 5c2e28ab-b9e2-d8e5-18fc-94c691a3baae
[2021-12-20T20:11:41.489Z]  INFO: cn-agent/71190 on headnode: docker_copy task params (req_id=c7d9bece-f10e-44d7-a489-9537bc34d8bb)
    task: {
      "task": "docker_copy",
      "params": {
        "uuid": "df3e8376-e13d-e289-a813-ff1a073d12b9",
        "path": "/etc/nginx/",
        "mode": "stat"
      }
    }
[2021-12-20T20:11:41.505Z]  INFO: cn-agent/71190 on headnode: Child logging to /var/log/cn-agent/logs/1640031101490-83589-docker_copy.log
[2021-12-20T20:11:41.505Z]  INFO: cn-agent/83589 on headnode: Executing task module: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/tasks/docker_copy
[2021-12-20T20:11:41.637Z]  INFO: cn-agent/83589 on headnode: Received "ready" event.
[2021-12-20T20:11:41.637Z]  INFO: cn-agent/83589 on headnode: Sending "start" event with payload to child.
[2021-12-20T20:11:41.640Z]  INFO: cn-agent/83589 on headnode: Received a task event from child task process: progress
[2021-12-20T20:11:41.640Z]  INFO: cn-agent/83589 on headnode: Received a task event from child task process: start
[2021-12-20T20:11:41.802Z]  INFO: docker-stdio/83590 on headnode: getDockerContainerPathStat (req_id=c7d9bece-f10e-44d7-a489-9537bc34d8bb, path=/etc/nginx/, norm=/etc/nginx/)
    abspath: /zones/df3e8376-e13d-e289-a813-ff1a073d12b9/root/etc/nginx/
[2021-12-20T20:11:41.805Z]  INFO: cn-agent/83589 on headnode: Received a task event from child task process: progress
[2021-12-20T20:11:41.805Z]  INFO: cn-agent/83589 on headnode: Received a task event from child task process: finish
[2021-12-20T20:11:41.809Z]  INFO: cn-agent/71190 on headnode: handled: 200 (req_id=c7d9bece-f10e-44d7-a489-9537bc34d8bb, route=posttasks, audit=true, remoteAddress=10.88.88.22, remotePort=51054, latency=326, _audit=true, req.query={}, req.version=*)
    POST /tasks HTTP/1.1
    x-server-uuid: 5c2e28ab-b9e2-d8e5-18fc-94c691a3baae
    x-request-id: c7d9bece-f10e-44d7-a489-9537bc34d8bb
    accept: application/json
    content-type: application/json
    user-agent: restify/4.3.0 (x64-sunos; v8/5.1.281.111; OpenSSL/1.0.2r) node/6.17.0
    date: Mon, 20 Dec 2021 20:11:41 GMT
    content-length: 114
    content-md5: jx+tAa+LlcwR3yOo1bVkgQ==
    host: 10.88.88.7:5309
    connection: keep-alive
    --
    HTTP/1.1 200 OK
    content-type: application/json
    content-length: 142
    date: Mon, 20 Dec 2021 20:11:41 GMT
    server: Compute Node Agent
    x-request-id: c7d9bece-f10e-44d7-a489-9537bc34d8bb
    x-response-time: 323
    x-server-name: headnode
    --
    req.timers: {
      "bunyan": 592,
      "parseAccept": 771,
      "parseAuthorization": 101,
      "parseQueryString": 585,
      "readBody": 1197,
      "parseBody": 221,
      "addHeaders": 65,
      "handler-0": 320216
    }
[2021-12-20T20:11:41.824Z]  INFO: cn-agent/83589 on headnode: Child terminated cleanly.

I also see this higher up but it looks like it gets logged right after starting the service up:

(node) v8::ObjectTemplate::Set() with non-primitive values is deprecated
(node) and will stop working in the next major release.

==== JS stack trace =========================================

Security context: 0xb0a2b18d <JS Object>#0#
    1: .node [module.js:604] [pc=ae39b6a3] (this=0xa64c544d <an Object with map 93623ae5>#1#,module=0xbdab58c9 <a Module with map 93624509>#2#,filename=0xbdab4d5d <String[75]: /opt/smartdc/agents/lib/node_modules/cn-agent/
node_modules/kstat/kstat.node>)
    2: load [module.js:494] [pc=ae38a5ba] (this=0xbdab58c9 <a Module with map 93624509>#2#,filename=0xbdab4d5d <String[75]: /opt/smartdc/agents/lib/node_modules/cn-agent/node_modules/kstat/kstat.node>)
    3: tryModuleLoad(aka tryModuleLoad) [module.js:453] [pc=ae38a23a] (this=0xb0a08121 <undefined>,module=0xbdab58c9 <a Module with map 93624509>#2#,filename=0xbdab4d5d <String[75]: /opt/smartdc/agents/lib/node_modules/cn-
agent/node_modules/kstat/kstat.node>)
    4: _load [module.js:445] [pc=ae380d99] (this=0xb0ad4035 <JS Function Module (SharedFunctionInfo b0aa8f11)>#3#,request=0xadcff865 <String[5]: kstat>,parent=0xbdaa0ebd <a Module with map 93624509>#4#,isMain=0xb0a08159 <f
alse>)
    5: require [module.js:~501] [pc=ae3cfb07] (this=0xbdaa0ebd <a Module with map 93624509>#4#,path=0xadcff865 <String[5]: kstat>)
    6: require(aka require) [internal/module.js:20] [pc=ae390b80] (this=0xb0a08121 <undefined>,path=0xadcff865 <String[5]: kstat>)
    7: /* anonymous */ [/opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/index.js:20] [pc=94e1c664] (this=0xbdaa0ee5 <an Object with map 93609415>#5#,exports=0xbdaa0ee5 <an Object with map 93609415>#5#,re
quire=0xbdaaee11 <JS Function require (SharedFunctionInfo b0ac8141)>#6#,module=0xbdaa0ebd <a Module with map 93624509>#4#,__filename=0xbdaa006d <String[75]: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smarto
s/index.js>,__dirname=0xbdaaeddd <String[66]: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos>)
    8: _compile [module.js:577] [pc=ae390195] (this=0xbdaa0ebd <a Module with map 93624509>#4#,content=0xbdaa173d <Very long string[26579]>#7#,filename=0xbdaa006d <String[75]: /opt/smartdc/agents/lib/node_modules/cn-agent/
lib/backends/smartos/index.js>)
    9: .js [module.js:586] [pc=ae38b68e] (this=0xa64c544d <an Object with map 93623ae5>#1#,module=0xbdaa0ebd <a Module with map 93624509>#4#,filename=0xbdaa006d <String[75]: /opt/smartdc/agents/lib/node_modules/cn-agent/li
b/backends/smartos/index.js>)
   10: load [module.js:494] [pc=ae38a5ba] (this=0xbdaa0ebd <a Module with map 93624509>#4#,filename=0xbdaa006d <String[75]: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/index.js>)
   11: tryModuleLoad(aka tryModuleLoad) [module.js:453] [pc=ae38a23a] (this=0xb0a08121 <undefined>,module=0xbdaa0ebd <a Module with map 93624509>#4#,filename=0xbdaa006d <String[75]: /opt/smartdc/agents/lib/node_modules/cn-
agent/lib/backends/smartos/index.js>)
   12: _load [module.js:445] [pc=ae380d99] (this=0xb0ad4035 <JS Function Module (SharedFunctionInfo b0aa8f11)>#3#,request=0xbda9e7cd <String[23]: ../lib/backends/smartos>,parent=0xa64c28b9 <a Module with map 93624509>#8#,i
sMain=0xb0a08159 <false>)
   13: require [module.js:~501] [pc=ae3cfb07] (this=0xa64c28b9 <a Module with map 93624509>#8#,path=0xbda9e7cd <String[23]: ../lib/backends/smartos>)
   14: require(aka require) [internal/module.js:20] [pc=ae390b80] (this=0xb0a08121 <undefined>,path=0xbda9e7cd <String[23]: ../lib/backends/smartos>)
   15: loadBackend [/opt/smartdc/agents/lib/node_modules/cn-agent/bin/cn-agent.js:52] [pc=94e1b917] (this=0xb0ad4e65 <JS Global Object>#9#,opts=0xbda9e675 <an Object with map 9365b611>#10#)
   16: main [/opt/smartdc/agents/lib/node_modules/cn-agent/bin/cn-agent.js:74] [pc=94e0bde5] (this=0xb0ad4e65 <JS Global Object>#9#)
   17: /* anonymous */ [/opt/smartdc/agents/lib/node_modules/cn-agent/bin/cn-agent.js:30] [pc=ae3907e3] (this=0xa64c2a19 <an Object with map 93609415>#11#,exports=0xa64c2a19 <an Object with map 93609415>#11#,require=0xa64c
2901 <JS Function require (SharedFunctionInfo b0ac8141)>#12#,module=0xa64c28b9 <a Module with map 93624509>#8#,__filename=0xa64c29cd <String[61]: /opt/smartdc/agents/lib/node_modules/cn-agent/bin/cn-agent.js>,__dirname=0xa
64c29b9 <String[49]: /opt/smartdc/agents/lib/node_modules/cn-agent/bin>)
   18: _compile [module.js:577] [pc=ae390195] (this=0xa64c28b9 <a Module with map 93624509>#8#,content=0xa64c3f69 <Very long string[5334]>#13#,filename=0xa64c29cd <String[61]: /opt/smartdc/agents/lib/node_modules/cn-agent/
bin/cn-agent.js>)
   19: .js [module.js:586] [pc=ae38b68e] (this=0xa64c544d <an Object with map 93623ae5>#1#,module=0xa64c28b9 <a Module with map 93624509>#8#,filename=0xa64c29cd <String[61]: /opt/smartdc/agents/lib/node_modules/cn-agent/bi
n/cn-agent.js>)
   20: load [module.js:494] [pc=ae38a5ba] (this=0xa64c28b9 <a Module with map 93624509>#8#,filename=0xa64c29cd <String[61]: /opt/smartdc/agents/lib/node_modules/cn-agent/bin/cn-agent.js>)
   21: tryModuleLoad(aka tryModuleLoad) [module.js:453] [pc=ae38a23a] (this=0xb0a08121 <undefined>,module=0xa64c28b9 <a Module with map 93624509>#8#,filename=0xa64c29cd <String[61]: /opt/smartdc/agents/lib/node_modules/cn-
agent/bin/cn-agent.js>)
   22: _load [module.js:445] [pc=ae380d99] (this=0xb0ad4035 <JS Function Module (SharedFunctionInfo b0aa8f11)>#3#,request=0xb0ac0591 <String[58]: /opt/smartdc/agents/lib/node_modules/cn-agent/bin/cn-agent>,parent=0xb0a0810
9 <null>,isMain=0xb0a08141 <true>)
   23: /* anonymous */(aka /* anonymous */) [module.js:611] [pc=ae3809e2] (this=0xb0a08121 <undefined>)
   24: run(aka run) [bootstrap_node.js:394] [pc=ae3808d8] (this=0xb0a08121 <undefined>,entryFunction=0xb0abf79d <JS Function Module.runMain (SharedFunctionInfo b0aa97f9)>#14#)
   25: startup(aka startup) [bootstrap_node.js:160] [pc=ae355550] (this=0xb0a08121 <undefined>)
   26: /* anonymous */(aka /* anonymous */) [bootstrap_node.js:507] [pc=ae353f95] (this=0xb0a08109 <null>,process=0xb0ad41d9 <a process with map 93615151>#15#)
=====================

This is from the cnapi log:

[2021-12-20T20:45:42.305Z]  INFO: cnapi/31769 on 1e71a2d5-6945-44fc-858b-1a90c2411ff9: posting task to http://10.88.88.7:5309/tasks (req_id=900d40a5-ef0b-458c-b04f-c46cc7b891d5)
[2021-12-20T20:45:42.307Z]  INFO: cnapi/31769 on 1e71a2d5-6945-44fc-858b-1a90c2411ff9: done posting task to client
[2021-12-20T20:45:42.694Z]  INFO: cnapi/31769 on 1e71a2d5-6945-44fc-858b-1a90c2411ff9:  (req_id=900d40a5-ef0b-458c-b04f-c46cc7b891d5)
    evt: {
      "ph": "e",
      "name": "vmdockercopy",
      "req_seq": 1640033142204008
    }
[2021-12-20T20:45:42.694Z]  INFO: cnapi/31769 on 1e71a2d5-6945-44fc-858b-1a90c2411ff9: handled: 200 (req_id=900d40a5-ef0b-458c-b04f-c46cc7b891d5, route=vmdockercopy, audit=true, remoteAddress=::ffff:10.88.88.45, remotePort
=61328, latency=490, _audit=true, req.query={}, req.version=*)
    POST /servers/5c2e28ab-b9e2-d8e5-18fc-94c691a3baae/vms/df3e8376-e13d-e289-a813-ff1a073d12b9/docker-copy HTTP/1.1
    x-request-id: 900d40a5-ef0b-458c-b04f-c46cc7b891d5
    accept: application/json
    content-type: application/json
    user-agent: restify/1.6.0 (x64-sunos; v8/5.1.281.111; OpenSSL/1.0.2r) node/6.17.1
    date: Mon, 20 Dec 2021 20:45:42 GMT
    content-length: 36
    content-md5: tJuTEW60u0ekVUovnG5aGw==
    host: cnapi.neptune.chadschmelter.com
    connection: keep-alive
    --
    HTTP/1.1 200 OK
    content-type: application/json
    content-length: 142
    date: Mon, 20 Dec 2021 20:45:42 GMT
    server: cnapi/1.26.4
    x-request-id: 900d40a5-ef0b-458c-b04f-c46cc7b891d5
    x-response-time: 490
    x-server-name: 1e71a2d5-6945-44fc-858b-1a90c2411ff9
    --
    req.timers: {
      "bunyan": 19,
      "setTracing": 87,
      "parseAccept": 61,
      "parseQueryString": 13,
      "readBody": 91,
      "parseBody": 30,
      "restifyValidator": 8,
      "setDefaultTimeouts": 11,
      "setHeaders": 6,
      "handlerEnsureStash": 3,
      "handlerEnsureConnectionTimeout": 7,
      "handlerEnsureConnectionMoray": 3,
      "handlerEnsureServerPrepopulated": 19972,
      "handlerEnsureVmPrepopulated": 202,
      "handlerEnsureServerRunning": 61,
      "handlerVmDockerCopy": 469072
    }

Found this in /var/log/cn-agent/logs/1640121712068-79262-docker_copy.log:

[2021-12-21T21:21:52.208Z]  INFO: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/tasks/docker_copy/79262 on headnode:  (req_id=82037ac6-a391-4455-a568-58269cb62913)
    evt: {
      "ph": "b",
      "name": "docker_copy"
    }
[2021-12-21T21:21:52.210Z] DEBUG: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/tasks/docker_copy/79262 on headnode: Child ready to start, sending ready event to parent (req_id=82037ac6-a391-4455-a568-58269cb62913)
[2021-12-21T21:21:52.212Z] DEBUG: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/tasks/docker_copy/79262 on headnode: Child received hydracp message from parent: (req_id=82037ac6-a391-4455-a568-58269cb62913)
[2021-12-21T21:21:52.212Z] DEBUG: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/tasks/docker_copy/79262 on headnode:  (req_id=82037ac6-a391-4455-a568-58269cb62913, action=start, req.req_id=82037ac6-a391-4455-a568-58269cb62913, req.req_host=10.88.88.7:5309, req.task=docker_copy)
    undefined undefined HTTP/1.1
    --
    taskspath: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/tasks
    --
    req.serverAddress: {
      "address": "10.88.88.7",
      "family": "IPv4",
      "port": 5309
    }
    --
    req.params: {
      "uuid": "301d1fdb-fe11-4d72-ce10-f2319fe546f9",
      "path": "/etc/nginx/",
      "mode": "stat"
    }
    --
    req.sysinfo: {
      "Live Image": "20211104T023339Z",
      "System Type": "SunOS",
      "Boot Time": "1640118926",
      "Datacenter Name": "neptune",
      "SDC Version": "7.0",
      "Manufacturer": "Intel(R) Client Systems",
      "Product": "NUC8i7BEH",
      "Serial Number": "G6BE838005D2",
      "SKU Number": "BOXNUC8i7BEH",
      "HW Version": "J72992-303",
      "HW Family": "Intel NUC",
      "Setup": "true",
      "VM Capable": true,
      "Bhyve Capable": true,
      "Bhyve Max Vcpus": 32,
      "HVM API": true,
      "CPU Type": "Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz",
      "CPU Virtualization": "vmx",
      "CPU Physical Cores": 1,
      "Admin NIC Tag": "admin",
      "Admin IP": "10.88.88.7",
      "UUID": "5c2e28ab-b9e2-d8e5-18fc-94c691a3baae",
      "Hostname": "headnode",
      "CPU Total Cores": 8,
      "CPU Socket Count": 1,
      "CPU Core Count": 4,
      "CPU Online Count": 8,
      "CPU Count": 8,
      "MiB of Memory": "32633",
      "Zpool": "zones",
      "Zpool Encrypted": false,
      "Zpool Disks": "c2t00253853015161C8d0",
      "Zpool Profile": "striped",
      "Zpool Creation": 1587597230,
      "Zpool Size in GiB": 899,
      "Zpool Recovery": {},
      "Disks": {
        "c2t00253853015161C8d0": {
          "Size in GB": 1000
        }
      },
      "Boot Parameters": {
        "module_name_0": "environment",
        "console": "text",
        "boot_args": "",
        "bootargs": "",
        "bootfile": "/os/20211104T023339Z/platform/i86pc/kernel/amd64/unix",
        "efi_version": "2.70",
        "headnode": "true",
        "ipxe": "false",
        "ipxe_archive": "/boot/default.ipxe",
        "ipxe_bootfile": "/boot/ipxe.lkrn",
        "ipxe_efi": "/boot/ipxe.efi",
        "os_console": "text",
        "platform_version": "20211104T023339Z",
        "prev_archive": "/os/20210520T001536Z/platform/i86pc/amd64/boot_archive",
        "prev_hash": "/os/20210520T001536Z/platform/i86pc/amd64/boot_archive.hash",
        "prev_platform": "/os/20210520T001536Z/platform/i86pc/kernel/amd64/unix",
        "prev_version": "20210520T001536Z",
        "screen__cols": "98",
        "screen__rows": "36",
        "screen_font": "8x16",
        "screen_height": "600",
        "screen_width": "800",
        "smt_enabled": "true",
        "efi_systype": "64",
        "plat_max_ncpus": "8",
        "boot_max_ncpus": "8"
      },
      "Psrinfo": {
        "smt_enabled": "true"
      },
      "SDC Agents": [
        {
          "name": "cn-agent",
          "version": "2.14.5"
        },
        {
          "name": "agents_core",
          "version": "2.2.2"
        },
        {
          "name": "marlin",
          "version": "0.0.3"
        },
        {
          "name": "firewaller",
          "version": "1.6.1"
        },
        {
          "name": "vm-agent",
          "version": "1.9.0"
        },
        {
          "name": "cmon-agent",
          "version": "1.17.1"
        },
        {
          "name": "hagfish-watcher",
          "version": "1.0.1-release-20211104-20211104T022859Z-gc0fd24e"
        },
        {
          "name": "smartlogin",
          "version": "0.2.2-release-20211104-20211104T030717Z-g3bb7593"
        },
        {
          "name": "net-agent",
          "version": "2.3.3"
        },
        {
          "name": "amon-agent",
          "version": "2.0.0"
        },
        {
          "name": "config-agent",
          "version": "1.8.4"
        },
        {
          "name": "amon-relay",
          "version": "2.0.0"
        }
      ],
      "Network Interfaces": {
        "e1000g0": {
          "MAC Address": "94:c6:91:a3:ba:ae",
          "ip4addr": "10.88.88.7",
          "Link Status": "up",
          "NIC Names": [
            "external",
            "sdc_underlay",
            "admin"
          ]
        }
      },
      "Virtual Network Interfaces": {
        "external0": {
          "MAC Address": "02:08:20:e5:66:28",
          "ip4addr": "192.168.0.160",
          "Link Status": "up",
          "Host Interface": "e1000g0",
          "VLAN": "0"
        },
        "sdc_underlay0": {
          "MAC Address": "90:b8:d0:91:f4:46",
          "ip4addr": "10.77.77.207",
          "Link Status": "up",
          "Host Interface": "e1000g0",
          "Overlay Nic Tags": [
            "sdc_overlay"
          ],
          "VLAN": "0"
        }
      },
      "Link Aggregations": {}
    }
[2021-12-21T21:21:52.212Z]  INFO: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/tasks/docker_copy/79262 on headnode: Instantiating /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/tasks/docker_copy (req_id=82037ac6-a391-4455-a568-58269cb62913)
[2021-12-21T21:21:52.213Z]  INFO: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/tasks/docker_copy/79262 on headnode: Received event (progress) from task instance (req_id=82037ac6-a391-4455-a568-58269cb62913)
    event: {
      "value": 0
    }
[2021-12-21T21:21:52.213Z]  INFO: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/tasks/docker_copy/79262 on headnode: Received event (start) from task instance (req_id=82037ac6-a391-4455-a568-58269cb62913, event={})
[2021-12-21T21:21:52.369Z]  INFO: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/tasks/docker_copy/79262 on headnode: Received event (progress) from task instance (req_id=82037ac6-a391-4455-a568-58269cb62913)
    event: {
      "value": 100
    }
[2021-12-21T21:21:52.369Z]  INFO: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/tasks/docker_copy/79262 on headnode: Received event (finish) from task instance (req_id=82037ac6-a391-4455-a568-58269cb62913)
    event: {
      "host": "10.88.88.7",
      "containerPathStat": {
        "name": "etc/nginx/",
        "size": 12,
        "mtime": "2021-05-04T14:37:57.173Z",
        "linkTarget": "",
        "mode": 2147483648
      }
    }
[2021-12-21T21:21:52.371Z]  INFO: /opt/smartdc/agents/lib/node_modules/cn-agent/lib/backends/smartos/tasks/docker_copy/79262 on headnode:  (req_id=82037ac6-a391-4455-a568-58269cb62913)
    evt: {
      "ph": "e",
      "name": "docker_copy"
    }

I also just noticed that a core file gets created in /root/core on the docker zone every time this happens:

[root@63174fb7-7a4f-4bb3-8db2-f0909a50c5df (neptune:docker0) ~]# pstack core
core 'core' of 71216:	/opt/smartdc/docker/build/node/bin/node --abort_on_uncaught_exception 
--------------------- thread# 1 / lwp# 1 ---------------------
 00000000016cd669 _ZN2v84base2OS5AbortEv () + 9
 00000000014e5c26 _ZN2v88internal13Runtime_ThrowEiPPNS0_6ObjectEPNS0_7IsolateE () + 116
 000039e12ec060c7 ???????? ()
 000039e12ec8b203 ???????? ()
 000039e12ec1c7d5 ???????? ()
 000039e12ed1a33f ???????? ()
 000039e12ed3dddc ???????? ()
 000039e12ec1ec1a ???????? ()
 000039e12ec1948f ???????? ()
 0000000001273a2d _ZN2v88internal9Execution4CallEPNS0_7IsolateENS0_6HandleINS0_6ObjectEEES6_iPS6_ () + dd
 0000000000f85857 _ZN2v88Function4CallENS_5LocalINS_7ContextEEENS1_INS_5ValueEEEiPS5_ () + 147
 0000000000f90621 _ZN2v88Function4CallENS_5LocalINS_5ValueEEEiPS3_ () + 41
 0000000000a99b79 _ZN4node9AsyncWrap12MakeCallbackEN2v85LocalINS1_8FunctionEEEiPNS2_INS1_5ValueEEE () + 339
 0000000000b10286 _ZN4node10StreamBase8EmitDataElN2v85LocalINS1_6ObjectEEES4_ () + b6
 0000000000b12606 _ZN4node10StreamWrap10OnReadImplElPK8uv_buf_t14uv_handle_typePv () + f6
 0000000000b12a98 _ZN4node10StreamWrap6OnReadEP11uv_stream_slPK8uv_buf_t () + 78
 0000000000d10396 uv__read () + 206
 0000000000d109e8 uv__stream_io () + 208
 0000000000d16dc5 uv__io_poll () + 3d5
 0000000000d04b6f uv_run () + 13f
 0000000000ab7430 _ZN4node5StartEiPPc () + 530
 0000000000a97eb3 _start_crt () + 83
 0000000000a97e18 _start () + 18
--------------------- thread# 2 / lwp# 2 ---------------------
 fffffc7fef0408f7 __lwp_park () + 17
 fffffc7fef033040 sema_wait (24523a0) + 10
 fffffc7fef026e22 sem_wait (24523a0) + 22
 0000000000d13054 uv_sem_wait () + 24
 0000000000ab7ec2 _ZN4node21DebugSignalThreadMainEPv () + 12
 fffffc7fef04059c _thrp_setup (fffffc7fef1e0240) + 6c
 fffffc7fef0408b0 _lwp_start ()
--------------------- thread# 3 / lwp# 3 ---------------------
 fffffc7fef0408f7 __lwp_park () + 17
 fffffc7fef033040 sema_wait (24b0e28) + 10
 fffffc7fef026e22 sem_wait (24b0e28) + 22
 00000000016cc928 _ZN2v84base9Semaphore4WaitEv () + 18
 0000000000b566f9 _ZN2v88platform9TaskQueue7GetNextEv () + 29
 0000000000b5684c _ZN2v88platform12WorkerThread3RunEv () + 2c
 00000000016cd479 _ZN2v84baseL11ThreadEntryEPv () + 39
 fffffc7fef04059c _thrp_setup (fffffc7fef1e0a40) + 6c
 fffffc7fef0408b0 _lwp_start ()
--------------------- thread# 4 / lwp# 4 ---------------------
 fffffc7fef0408f7 __lwp_park () + 17
 fffffc7fef033040 sema_wait (24b0e28) + 10
 fffffc7fef026e22 sem_wait (24b0e28) + 22
 00000000016cc928 _ZN2v84base9Semaphore4WaitEv () + 18
 0000000000b566f9 _ZN2v88platform9TaskQueue7GetNextEv () + 29
 0000000000b5684c _ZN2v88platform12WorkerThread3RunEv () + 2c
 00000000016cd479 _ZN2v84baseL11ThreadEntryEPv () + 39
 fffffc7fef04059c _thrp_setup (fffffc7fef1e1240) + 6c
 fffffc7fef0408b0 _lwp_start ()
--------------------- thread# 5 / lwp# 5 ---------------------
 fffffc7fef0408f7 __lwp_park () + 17
 fffffc7fef033040 sema_wait (24b0e28) + 10
 fffffc7fef026e22 sem_wait (24b0e28) + 22
 00000000016cc928 _ZN2v84base9Semaphore4WaitEv () + 18
 0000000000b566f9 _ZN2v88platform9TaskQueue7GetNextEv () + 29
 0000000000b5684c _ZN2v88platform12WorkerThread3RunEv () + 2c
 00000000016cd479 _ZN2v84baseL11ThreadEntryEPv () + 39
 fffffc7fef04059c _thrp_setup (fffffc7fef1e1a40) + 6c
 fffffc7fef0408b0 _lwp_start ()
--------------------- thread# 6 / lwp# 6 ---------------------
 fffffc7fef0408f7 __lwp_park () + 17
 fffffc7fef033040 sema_wait (24b0e28) + 10
 fffffc7fef026e22 sem_wait (24b0e28) + 22
 00000000016cc928 _ZN2v84base9Semaphore4WaitEv () + 18
 0000000000b566f9 _ZN2v88platform9TaskQueue7GetNextEv () + 29
 0000000000b5684c _ZN2v88platform12WorkerThread3RunEv () + 2c
 00000000016cd479 _ZN2v84baseL11ThreadEntryEPv () + 39
 fffffc7fef04059c _thrp_setup (fffffc7fef1e2240) + 6c
 fffffc7fef0408b0 _lwp_start ()
--------------------- thread# 7 / lwp# 7 ---------------------
 fffffc7fef0408f7 __lwp_park () + 17
 fffffc7fef039d8b cond_wait_queue (2453f30, 2453f10, 0) + 5b
 fffffc7fef03a1dd cond_wait_common (2453f30, 2453f10, 0) + 1dd
 fffffc7fef03a3ac __cond_wait (2453f30, 2453f10) + 6c
 fffffc7fef03a434 cond_wait (2453f30, 2453f10) + 24
 fffffc7fef03a46b pthread_cond_wait (2453f30, 2453f10) + b
 0000000000d131c9 uv_cond_wait () + 9
 0000000000d01fd8 worker () + 48
 fffffc7fef04059c _thrp_setup (fffffc7fef1e2a40) + 6c
 fffffc7fef0408b0 _lwp_start ()
--------------------- thread# 8 / lwp# 8 ---------------------
 fffffc7fef0408f7 __lwp_park () + 17
 fffffc7fef039d8b cond_wait_queue (2453f30, 2453f10, 0) + 5b
 fffffc7fef03a1dd cond_wait_common (2453f30, 2453f10, 0) + 1dd
 fffffc7fef03a3ac __cond_wait (2453f30, 2453f10) + 6c
 fffffc7fef03a434 cond_wait (2453f30, 2453f10) + 24
 fffffc7fef03a46b pthread_cond_wait (2453f30, 2453f10) + b
 0000000000d131c9 uv_cond_wait () + 9
 0000000000d01fd8 worker () + 48
 fffffc7fef04059c _thrp_setup (fffffc7fef1e3240) + 6c
 fffffc7fef0408b0 _lwp_start ()
--------------------- thread# 9 / lwp# 9 ---------------------
 fffffc7fef0408f7 __lwp_park () + 17
 fffffc7fef039d8b cond_wait_queue (2453f30, 2453f10, 0) + 5b
 fffffc7fef03a1dd cond_wait_common (2453f30, 2453f10, 0) + 1dd
 fffffc7fef03a3ac __cond_wait (2453f30, 2453f10) + 6c
 fffffc7fef03a434 cond_wait (2453f30, 2453f10) + 24
 fffffc7fef03a46b pthread_cond_wait (2453f30, 2453f10) + b
 0000000000d131c9 uv_cond_wait () + 9
 0000000000d01fd8 worker () + 48
 fffffc7fef04059c _thrp_setup (fffffc7fef1e3a40) + 6c
 fffffc7fef0408b0 _lwp_start ()
-------------------- thread# 10 / lwp# 10 --------------------
 fffffc7fef0408f7 __lwp_park () + 17
 fffffc7fef039d8b cond_wait_queue (2453f30, 2453f10, 0) + 5b
 fffffc7fef03a1dd cond_wait_common (2453f30, 2453f10, 0) + 1dd
 fffffc7fef03a3ac __cond_wait (2453f30, 2453f10) + 6c
 fffffc7fef03a434 cond_wait (2453f30, 2453f10) + 24
 fffffc7fef03a46b pthread_cond_wait (2453f30, 2453f10) + b
 0000000000d131c9 uv_cond_wait () + 9
 0000000000d01fd8 worker () + 48
 fffffc7fef04059c _thrp_setup (fffffc7fef1e4240) + 6c
 fffffc7fef0408b0 _lwp_start ()
-------------------- thread# 11 / lwp# 11 --------------------
 fffffc7fef0408f7 __lwp_park () + 17
 fffffc7fef039d8b cond_wait_queue (fffffc7fef186880, fffffc7fef1868a0, fffffc7fed922e60) + 5b
 fffffc7fef03a1dd cond_wait_common (fffffc7fef186880, fffffc7fef1868a0, fffffc7fed922e60) + 1dd
 fffffc7fef03a4e5 __cond_timedwait (fffffc7fef186880, fffffc7fef1868a0, fffffc7fed922f50) + 65
 fffffc7fef03a5a1 cond_timedwait (fffffc7fef186880, fffffc7fef1868a0, fffffc7fed922f50) + 31
 fffffc7fef139d96 umem_update_thread (0) + 1c6
 fffffc7fef04059c _thrp_setup (fffffc7fef1e4a40) + 6c
 fffffc7fef0408b0 _lwp_start ()

Here is ::jsstack from mdb on the core dump file that was generated:

> ::jsstack
native: v8::base::OS::Abort+9
native: v8::internal::Runtime_Throw+0x116
        (1 internal frame elided)
js:     emit
        (1 internal frame elided)
js:     emitErrorNT
js:     _combinedTickCallback
js:     _tickDomainCallback
        (1 internal frame elided)
        (1 internal frame elided)
native: v8::internal::Execution::Call+0xdd
native: v8::Function::Call+0x147
native: v8::Function::Call+0x41
native: node::AsyncWrap::MakeCallback+0x339
native: node::StreamBase::EmitData+0xb6
native: node::StreamWrap::OnReadImpl+0xf6
native: node::StreamWrap::OnRead+0x78
native: uv__read+0x206
native: uv__stream_io+0x208
native: uv__io_poll+0x3d5
native: uv_run+0x13f
native: node::Start+0x530
native: _start_crt+0x83
native: _start+0x18