Skip to content

Instantly share code, notes, and snippets.

View blackwood821's full-sized avatar

blackwood821

1 follower · 0 following
View GitHub Profile
@blackwood821
blackwood821 / docker_triton_functions.sh
Created July 14, 2021 14:09
function is_docker
{
if [ "${container}" = "docker" ]; then
return 0
else
return 1
fi
}
function is_triton
@blackwood821
blackwood821 / docker-compose.yml
Created July 13, 2021 17:45
version: '3'
services:
nginx:
image: nginx
container_name: nginx
restart: always
ports:
- "80:80"
- "443:443"
volumes:
@blackwood821
blackwood821 / cmon_plugin_error.md
Created July 2, 2021 16:38
CMON Plugin Error 
[2021-07-02T16:34:10.283Z]  WARN: cmon-agent/6223 on cn01: collectors-common/plugin/jvm/11db239b-3c76-445a-b5f2-0321c8b0b78c metrics unavailable, treating as empty
NotAvailableError: jvm: failed to execute: exec "/opt/custom/cmon/vm-plugins/jvm.prom 11db239b-3c76-445a-b5f2-0321c8b0b78c": unexpectedly terminated by signal SIGKILL
    at _afterExecution (/opt/smartdc/agents/lib/node_modules/cmon-agent/lib/instrumenter/collectors-common/plugin.js:221:34)
    at _onForkWaited (/opt/smartdc/agents/lib/node_modules/cmon-agent/lib/plugin-executor.js:60:9)
    at onChildExited (/opt/smartdc/agents/lib/node_modules/cmon-agent/node_modules/forkexec/lib/forkexec.js:83:2)
    at /opt/smartdc/agents/lib/node_modules/cmon-agent/node_modules/forkexec/lib/forkexec.js:53:4
    at ChildProcess.exithandler (child_process.js:207:5)
    at emitTwo (events.js:87:13)
    at ChildProcess.emit (events.js:172:7)
@blackwood821
blackwood821 / VM_reprovision_debug.md
Last active June 16, 2021 19:18
Triton Reprovision Issues

Every once in a while I will encounter the following error on a VM reprovision:

Command failed: imgadm get: error (ImageNotInstalled): image "372d550b-b224-46ec-950b-0c6a2421cad4" 
was not found on zpool "undefined"

I added logging to the reprovision() function in VM.js which logs vmobj right after it's populated from this line:

vmobj = vminfodEventGetVM(uuid, ready_ev);
@blackwood821
blackwood821 / smartos_lx_emulation_issue.md
Last active June 9, 2021 16:02
Smartos LX Emulation Issue

Oracle 12 forms can’t run on SmartOS LX zone

Oracle Forms 12 requires the following Linux kernel parameters to be set but the SmartOS LX emulation does not map these kernel parameters to Illumos kernel parameters.

sysctl -w net.core.rmem_max=2096304
sysctl -w net.core.wmem_max=2096304

We were able to set the equivalent underlying Illumos kernel parameters with the command below but the issue is that the LX emulation lies to the Linux system call and returns some arbitrary value even though we have changed the kernel param in Illumos under the hood.

@blackwood821
blackwood821 / migrate_cn_to_another_dc.md
Last active May 19, 2021 22:58

While the CN is still running in the source Triton DC, follow the steps in https://docs.joyent.com/private-cloud/troubleshooting/reset but skip the factory reset step because that is what tells SmartOS to destroy the zpool. Once the CN has been forgotten from the source Triton DC then power it down, cable it up to the target DC and when it iPXE boots the first time Triton will see it as already setup so it won’t give you the option to set it up. Replace the DC name and necessary IP addresses in the following files on the CN:

/opt/smartdc/agents/etc/vm-agent.config.json
/opt/smartdc/agents/etc/net-agent.config.json
/opt/smartdc/agents/etc/cn-agent.config.json
/opt/smartdc/registrar/etc/config.json
/opt/smartdc/config/node.config
/opt/smartdc/agents/etc/smartlogin.cfg
@blackwood821
blackwood821 / triton_reprovision_bug.md
Last active May 17, 2021 18:13 
[root@cn02 (us-west-agc) ~]# zfs get mounted,mountpoint zones/cores/a2d5015e-e073-4758-b37f-44aa2aabddbf
NAME                                              PROPERTY    VALUE                                              SOURCE
zones/cores/a2d5015e-e073-4758-b37f-44aa2aabddbf  mounted     yes                                                -
zones/cores/a2d5015e-e073-4758-b37f-44aa2aabddbf  mountpoint  /zones/a2d5015e-e073-4758-b37f-44aa2aabddbf/cores  local

[root@cn02 (us-west-agc) ~]# zfs umount /zones/a2d5015e-e073-4758-b37f-44aa2aabddbf/cores
cannot unmount '/zones/a2d5015e-e073-4758-b37f-44aa2aabddbf/cores': not a mountpoint
@blackwood821
blackwood821 / sdc-oneachnode_error.md
Created May 14, 2021 15:10 
[root@headnode (us-west-rno) ~]# sdc-server list
HOSTNAME             UUID                                 VERSION    SETUP    STATUS      RAM  ADMIN_IP       
cn01                 44454c4c-3300-1050-8051-b5c04f393033     7.0     true   unknown  1048193  10.10.65.50    
headnode             44454c4c-4c00-1039-8053-b9c04f4a4832     7.0     true   running   262049  10.10.65.10    
[root@headnode (us-west-rno) ~]# sdc-oneachnode -n headnode uptime
{"name":"sdc-oneachnode","hostname":"headnode","pid":215433,"component":"URConnection","level":50,"err":{"message":"connect timeout","name":"Error","stack":"Error: connect timeout\n    at null.<anonymous> (/zones/690741eb-28bf-4920-9eac-bcbc728cdf4f/root/opt/smartdc/sdc/node_modules/urclient/lib/mq.js:178:13)\n    at Timer.listOnTimeout [as ontimeout] (timers.js:121:15)"},"msg":"amqp error","time":"2021-05-14T15:10:11.012Z","v":0}

VError: amqp error: connect timeout
    at URConnection._disconnect (/zones/690741eb-28bf-4920-9eac-bcbc728cdf4f/root/opt/smartdc/sdc/n
@blackwood821
blackwood821 / triton_cn_enc_recovery.md
Created May 14, 2021 13:43 
[root@e4-43-4b-9c-b0-a9 ~]# tail -20 $(svcs -L filesystem/smartdc) 
+ is_encr=aes-256-gcm
+ [[ aes-256-gcm != \o\f\f ]]
+ unlock_pool zones
+ local pool=zones
++ zfs get -Hpo value keystatus zones
+ local keystatus=unavailable
+ [[ unavailable == \a\v\a\i\l\a\b\l\e ]]
+ kbmadm unlock zones
@blackwood821
blackwood821 / recovery_config.md
Last active April 14, 2021 20:00
Triton KBMAPI recovery config 
[root@9676e574-47e5-4987-b116-e712e5719e41 (us-west-agc:kbmapi0) ~]# kbmctl recovery ls
UUID                                  STAGED_TOKENS  ACTIVATED_TOKENS  STATE
76f89dae-0399-5456-bf64-8d1574033076  0              2                 active
00da4496-a19c-5aa0-acaa-599f51f5f20f  0              0                 staged