Note: Use the excellent helm chart - ohaiwalt/cog-helm to get up quickly on k8s
Follow the guide below to understand the inner workings a bit better.
We will be creating 1 deployment for the cog server and 1 deployment for the relay server.
We chose to run Postgres outside the Kubernetes cluster and provide instructions how to expose all service ports on a single ELB.
Using an existing RDS instance, create fresh user and database on an existing RDS (Postgres)
$ psql -U $RDS_USERNAME -h $RDS_HOSTNAME -p $RDS_PORT -d postgres
postgres=> \l
postgres=> CREATE DATABASE cog;
postgres=> CREATE USER cog WITH PASSWORD 'mysupersecret';
postgres=> GRANT ALL PRIVILEGES ON DATABASE cog TO cog;
postgres=> \connect cog
cog=> CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
cog=> \q
Or provision a complete fresh RDS instance.
Update the secrets in 04-secrets
.
On OSX, generate uuid with uuidgen
but make sure to convert to lower case!
uuidgen | tr '[:upper:]' '[:lower:]'
generate token with openssl rand -hex 12
(or python -c 'import sys,os,binascii; sys.stdout.write(binascii.hexlify(os.urandom(16)).decode("utf-8"))'
)
Convert 04-secrets
to a kubernetes secret called cog
:
./06-env2secret.py -i 05-secrets -o 02-cog-secrets.yml -s cog
Deploy:
kubectl create -f 01-cog-service.yml
NOTE: Do NOT call your service "COG" - it will create an Environment variable COG_SERVICE_PORT
inside every container, causing port conflicts and configuration issues!
kubectl create -f 02-cog-secrets.yml
kubectl create -f 03-cog-deployment.yml
Note: COG_HOST
environment variable in Relay Deployment definition needs to be able to find the service as it will connect to MQTT message bus:
kubectl create -f 04-relay-deployment.yml
review relay logs:
kubectl logs `kubectl get po -l app=relay -o name | cut -d"/" -f2`
If you see this error:
client API version: 1.23, server API version: 1.22
review Docker Version (CoreOS stable is still on 1.10.3
, use $DOCKER_API_VERSION
to work around this)
If you see this error:
Dynamic configuration root dir not found.
or
msg="Got invocation request on /bot/commands/<relay-id>/ec2/instance-list"
msg="Creating environment 6bff5c76eaf64eb896bdc6736aff5f4e/ec2:0.1.0"
msg="Dynamic config not found. Checked: '/data/ec2/config.yaml' and '/data/ec2/config.yml'."
msg="Dynamic config not found. Checked: '/data/ec2/room_direct.yaml' and '/data/ec2/room_direct.yml'."
msg="Dynamic config not found. Checked: '/data/ec2/user_so0k.yaml' and '/data/ec2/user_so0k.yml'."
Note: When using Dynamic configurations controlled centrally on COG server, need to configure relay to allow RELAY_MANAGED_DYNAMIC_CONFIG
Add TCP Listeners to ELB
aws elb create-load-balancer-listeners --load-balancer-name k8s-elb --listeners "Protocol=TCP,LoadBalancerPort=4000,InstanceProtocol=TCP,InstancePort=34000"
aws elb create-load-balancer-listeners --load-balancer-name k8s-elb --listeners "Protocol=TCP,LoadBalancerPort=4001,InstanceProtocol=TCP,InstancePort=34001"
aws elb create-load-balancer-listeners --load-balancer-name k8s-elb --listeners "Protocol=TCP,LoadBalancerPort=4002,InstanceProtocol=TCP,InstancePort=34002"
Note: Use TLS termination on the ELB by setting the listener Protocol to HTTPS with a valid certificate (or use Let's encrypt).
Open ports on ELB Security Group (untested)
aws ec2 authorize-security-group-ingress --group-name sg-k8s-elb --protocol tcp --from-port 4000 --to-port 4002 --cidr "0.0.0.0/0"
Create CNAME for ELB in route53
cog.example.com -> elb.amazon-url
Get a shell
kubectl get po -l app=cog
kubectl exec -it cog-2663148853-e9bcb /bin/sh
Follow instructions from the cog book