Skip to content

Instantly share code, notes, and snippets.

@mgerdts

mgerdts/0-coal-setup.md

Last active November 1, 2018 21:07
Show Gist options
  • Save mgerdts/c6c5c82bc4e997531685c0352e17a73b to your computer and use it in GitHub Desktop.
Save mgerdts/c6c5c82bc4e997531685c0352e17a73b to your computer and use it in GitHub Desktop.
Download ZIP
coal and docker and a particular version
Raw
0-coal-setup.md 
mget /Joyent_Dev/public/SmartDataCenter/release-20170316-20170315T223833Z/headnode/coal-release-20170316-20170315T223833Z-gc64b017-4gb.tgz | tar xvzf -

Follow first boot instructions at

https://github.com/joyent/triton/blob/master/docs/developer-guide/coal-setup.md

Then using sdc-docker/README.md as a guide,

Copy keys

scp ~/.ssh/id_rsa.pub coal:/var/tmp
ssh coal

Create user

[root@headnode (dc1) ~]# sdc-useradm create -A login=mgerdts email=mike.gerdts@joyent.com userpassword=foo12bar
User cfff7abd-07ea-42fc-b76c-343083c745e0 (login "mgerdts") created
[root@headnode (dc1) ~]# sdc-useradm add-key mgerdts /var/tmp/id_rsa.pub
Key "6f:7b:b7:f6:f4:1a:a1:28:70:fb:33:03:ec:04:fb:b5" added to user "mgerdts"

Perform post-setup tasks in coal

sdcadm post-setup common-external-nics && sleep 10  # imgapi needs external
sdcadm post-setup dev-headnode-prov
sdcadm post-setup dev-sample-data  # sample packages for docker containers
sdcadm post-setup cloudapi
sdcadm post-setup docker
sdcadm experimental update dockerlogger

Downgrade to the right version of docker and dockerlogger.

The previous step installed the latest version of docker and dockerlogger. We must downgrade. Notice experimental when updating dockerlogger.

[root@headnode (dc1) ~]# updates-imgadm list -C release name=docker | grep 201804
19e16030-3ded-11e8-9bfe-b7fa6dd1650e  docker  release-20180412-20180412T005303Z-g3b2a1e2  I      smartos  2018-04-12T01:01:16Z
f6064e06-48fe-11e8-84a3-5f9feeaa1ad2  docker  release-20180426-20180426T025818Z-g3269740  I      smartos  2018-04-26T03:06:04Z
[root@headnode (dc1) ~]# sdcadm up -C release docker@release-20180426-20180426T025818Z-g3269740
Using channel release

This update will make the following changes:
    download 1 image (234 MiB):
        image f6064e06-48fe-11e8-84a3-5f9feeaa1ad2
            (docker@release-20180426-20180426T025818Z-g3269740)
    update "docker" service to image f6064e06-48fe-11e8-84a3-5f9feeaa1ad2
        (docker@release-20180426-20180426T025818Z-g3269740):
        instance "861339d5-f409-452c-b835-1da6fd8178a7" (docker0) on server 564d2f7d-0c7f-f8fc-7343-81a1b99a6f2f

Would you like to continue? [y/N] y
...

[root@headnode (dc1) ~]# updates-imgadm list -C release name=dockerlogger | grep 201804
62bab6b0-7c81-43d0-bf8c-fec609bf301a  dockerlogger  1.0.0-release-20180412-20180412T013342Z-g2926f3c  -      other  2018-04-12T01:34:18Z
ed0a198c-3429-4458-b339-602aed988b3c  dockerlogger  1.0.0-release-20180426-20180426T035113Z-g2926f3c  -      other  2018-04-26T03:51:47Z
[root@headnode (dc1) ~]# sdcadm up -C release dockerlogger@1.0.0-release-20180426-20180426T035113Z-g2926f3c
Using channel release
Up-to-date.
[root@headnode (dc1) ~]# sdcadm insts | grep docker
861339d5-f409-452c-b835-1da6fd8178a7  docker           headnode  release-20180426-20180426T025818Z-g3269740        docker0
33079158-c1e7-442d-9a59-c8cef164e20a  dockerlogger     headnode  1.0.0-master-20180730T121722Z-g0b220f6            -
[root@headnode (dc1) ~]# sdcadm experimental update -C release dockerlogger@1.0.0-release-20180426-20180426T035113Z-g2926f3c
Using channel release

This update will make the following changes:
    download 1 image (7 MiB):
        image ed0a198c-3429-4458-b339-602aed988b3c
            (dockerlogger@1.0.0-release-20180426-20180426T035113Z-g2926f3c)
    update "dockerlogger" service to image ed0a198c-3429-4458-b339-602aed988b3c
        dockerlogger@1.0.0-release-20180426-20180426T035113Z-g2926f3c
    in 1 servers

Would you like to continue? [y/N] y

Upgrade imgapi and cn-agent

[root@headnode (dc1) ~]# sdcadm up -C release imgapi@release-20180426-20180426T031305Z-g806153b
Using channel release

This update will make the following changes:
    download 1 image (254 MiB):
        image 7a230786-4901-11e8-b82e-232810ba1b4e
            (imgapi@release-20180426-20180426T031305Z-g806153b)
    update "imgapi" service to image 7a230786-4901-11e8-b82e-232810ba1b4e
        (imgapi@release-20180426-20180426T031305Z-g806153b)

Would you like to continue? [y/N] y

[root@headnode (dc1) ~]# sdcadm experimental update -C release cn-agent@4aafdf5d-396f-4903-97e8-a34536da8d1d
Using channel release

This update will make the following changes:
    download 1 image (16 MiB):
        image 4aafdf5d-396f-4903-97e8-a34536da8d1d
            (cn-agent@2.1.0)
    update "cn-agent" service to image 4aafdf5d-396f-4903-97e8-a34536da8d1d
        (cn-agent@2.1.0)
    on 1 servers

Would you like to continue? [y/N] y

Generate client cert

Back in the host:

$ git remote -v
origin	git@github.com:joyent/sdc-docker.git (fetch)
origin	git@github.com:joyent/sdc-docker.git (push)
$ ./tools/sdc-docker-setup.sh coal mgerdts ~/.ssh/id_rsa
The authenticity of host '10.99.99.7 (10.99.99.7)' can't be established.
ECDSA key fingerprint is SHA256:XXK1/MebP7NrdHwkmNaeDtHg7rpARkeJ2xNmxH/esI0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.99.99.7' (ECDSA) to the list of known hosts.
Password:
Setting up Docker client for SDC using:
    CloudAPI:        https://10.88.88.5
    Account:         mgerdts
    Key:             /Users/mgerdts/.ssh/id_rsa

If you have a pass phrase on your key, the openssl command will
prompt you for your pass phrase now and again later.

Verifying CloudAPI access.
Enter pass phrase for /Users/mgerdts/.ssh/id_rsa:
CloudAPI access verified.

Generating client certificate from SSH private key.
Enter pass phrase for /Users/mgerdts/.ssh/id_rsa:
Wrote certificate files to /Users/mgerdts/.sdc/docker/mgerdts

Get Docker host endpoint from cloudapi.
Enter pass phrase for /Users/mgerdts/.ssh/id_rsa:
Docker service endpoint is: tcp://10.88.88.6:2376

* * *
Success. Set your environment as follows:

    export DOCKER_CERT_PATH=/Users/mgerdts/.sdc/docker/mgerdts
    export DOCKER_HOST=tcp://10.88.88.6:2376
    export DOCKER_CLIENT_TIMEOUT=300
    export COMPOSE_HTTP_TIMEOUT=300
    unset DOCKER_TLS_VERIFY
    alias docker="docker --tls"

Your Docker host is not a DNS name, but an IP. If you want to run docker
with TLS verification, you can configure to use a 'my.triton' DNS
name as follows (use this instead of the setup block above):

    sudo sed -e '$G; $s/$/10.88.88.6 my.triton/;' -i.bak /etc/hosts
    export DOCKER_CERT_PATH=/Users/mgerdts/.sdc/docker/mgerdts
    export DOCKER_HOST=tcp://my.triton:2376
    export DOCKER_TLS_VERIFY=1
    export DOCKER_CLIENT_TIMEOUT=300
    export COMPOSE_HTTP_TIMEOUT=300

Then you should be able to run 'docker info' and see your account
name 'SDCAccount: mgerdts' in the output.

Note: If you receive any docker compose warning about the
DOCKER_CLIENT_TIMEOUT environment variable being deprecated,
simply unset it and remove it from env.sh.

Run test

In one window:

$ ./runtest
=== iteration 1 at 0
Unable to find image 'bash:latest' locally
latest: Pulling from bash (req b3d6a0d7-74b1-4261-b185-fed17f29c891)
4fe2ade4980c: Pull complete
ec6d9ca5c66a: Pull complete
d8685fbd86ca: Pull complete
Digest: sha256:a5d748d88c425cce749f6e6a217468443f1afe9b489aad4d559812df932949da
Status: Downloaded newer image for bash:latest
ERRO[0026] error getting events from daemon: Error response from daemon: (NotImplemented) events is not implemented (8d619327-5b94-4e53-99d4-420543d5b9c1)
nameserver 8.8.8.8
nameserver 8.8.4.4
=== iteration 2 at 37
...

In another window:

$ ./cleantest
+ triton -p coal instance delete --wait evil_galileo
Delete instance evil_galileo (8a7dda3f-e704-46fc-fdc2-f4a3f82ae5ed, 5s)
+ triton -p coal instance delete --wait elated_ride
Delete instance elated_ride (ff219dd3-ae64-ecb4-825e-85793a0d87ee, 4s)
Waiting for the next batch
Raw
cleantest.sh
#! /bin/bash
while true; do
triton -p coal instance list -Ho name state=stopped docker=true | while read name; do
echo triton -p coal instance delete --wait $name
done | bash -x
echo "Waiting for the next batch"
sleep 60
done
Raw
runtest.sh
#! /bin/bash
. $HOME/.sdc/docker/mgerdts/env.sh
it=1
while true; do
echo === iteration $it at $SECONDS
docker --tls run bash -c 'if [[ -s /etc/resolv.conf ]]; then
cat /etc/resolv.conf;
else
while true; do
echo "####### resolv.conf is empty ######";
sleep 1;
done;
fi'
(( it = it + 1 ))
done
Raw
zonecfgwatch.d
#! /usr/sbin/dtrace -Cqs
/*
* Copyright 2018, Joyent, Inc.
*
* This script is used for detecting races in zonecfg, which suffers from
* "last writer wins" behavior.
*
* When "zonecfg -z <zone>" starts, it reads it uses open(2) to open the
* configuration file, reads it, then closes the file. If it needs to update
* the zone's configuration, it opens a temporary file for writing, writes
* to the file, closes it, then renames the temporary file to
* /etc/zones/<zone>.xml.
*
* When a potential race is detected (first rename while another process
* that read the file is still running), one or more lines with RACE is
* printed. If another rename is detected by a process that read before
* the race window started, that will result in CLOBBER being printed.
*/
#define ISGZ (curpsinfo->pr_zoneid == 0)
#define ISZONECFG (execname == "zonecfg")
#define ISZONECFGFILE(f) (dirname(stringof(f)) == "/etc/zones" && \
strlen(stringof(f)) == 51)
#define PROLOGUE() this->_t = walltimestamp; \
printf("%Y.%06d %s: ", this->_t, \
((this->_t % 1000000000) / 1000), \
probefunc)
#define EPILOGUE() printf("\n");
#define CURPID curpsinfo->pr_pid
#define CURPPID curpsinfo->pr_ppid
BEGIN
{
activecnt["dummy"] = 1;
loser[1] = 1;
starttime = timestamp;
printf("ready\n");
}
/*
* When zonecfg reads a configuration, we consider it active until the process
* exits. The predicate (roughly) ensures we only look at zonecfg opening
* /etc/zones/<uuid>.xml.
*
* Sadly, this is racy.
*/
syscall::open:entry
/ISGZ && ISZONECFG && ISZONECFGFILE(cleanpath(copyinstr(arg0)))/
{
PROLOGUE();
self->f = cleanpath(copyinstr(arg0));
self->slot = activecnt[self->f];
self->psargs = stringof(curpsinfo->pr_psargs);
self->readtime = timestamp;
activecnt[self->f]++;
activepid[self->f, self->slot] = CURPID;
activeppid[self->f, self->slot] = CURPPID;
activepsargs[self->f, self->slot] = self->psargs;
printf("zonecfg pid %d ppid %d opening %s slot %d (%s)",
CURPID, CURPPID, self->f, self->slot, self->psargs);
EPILOGUE();
}
syscall::rename:entry
/self->f == cleanpath(copyinstr(arg1)) && ISZONECFGFILE(copyinstr(arg1))/
{
PROLOGUE();
self->conflict = activecnt[self->f] == 1 ? 0 : 1;
printf("%srename of %s slot %d pid %d ppid %d (%s)",
loser[CURPID] ? "CLOBBER " : "",
self->f, self->slot, CURPID, CURPPID, self->psargs);
EPILOGUE();
}
#define PRINT_CONFLICT(n) loser[activepid[self->f, n]] = CURPID; \
printf(" CONFLICT %s slot %d pid %d ppid %d (%s)", \
self->f, n, activepid[self->f, n], \
activeppid[self->f, n], \
activepsargs[self->f, n]);
#define PROBE_CONFLICT(n) \
syscall::rename:entry \
/self->conflict && ISZONECFGFILE(copyinstr(arg1)) && self->slot != n && activepid[self->f, n] != 0/ \
{ \
PROLOGUE(); \
PRINT_CONFLICT(n); \
EPILOGUE(); \
}
PROBE_CONFLICT(0)
PROBE_CONFLICT(1)
exit
/self->f != 0/
{
PROLOGUE();
printf("pid %d", CURPID);
loser[CURPID] = 0;
activecnt[self->f]--;
active[self->f, self->slot] = 0;
activepid[self->f, self->slot] = 0;
activeppid[self->f, self->slot] = 0;
activepsargs[self->f, self->slot] = 0;
self->f = 0;
self->ps = 0;
self->conflict = 0;
EPILOGUE();
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment