pdxdevops-2015-11

pdxdevops-2015-11.txt

Announcement: CoffeeOps Group, Beaverton, meets Wed/Thu morning. ~10 people for first meetup

First Presentation: Kubernetes, Elson Rodriguez @elsonrodriguez, System Admin
Solves Outages/Deployment
If DevA, DevB, and DevC need to collocate services, dependency issues and ticket pong for Ops team waste time and resources
Today, we pin app to nodes. If node goes away, app goes away. App needs "more space" from the node.
An app is more than the code. It is a combination of
- code
- run time
- system dependencies
Containers, the combination of Namespaces and Cgroups, have been in the kernel since 2008
Already a part of Debian system services, the question is whether you want to take advantage of these features as well
Your current scheduler is your Ops dept, you should outsource it to Kubernetes

Terminology:
- node: is fundamental kubernetes object, a "machine", supported by Docker, kubelet process, and proxy
- pod: a collection of one or more containers. Lets containers run together as a unit in a single network namespace
- Replication Controller (rc): declares that the cluster should run a specific number a specific pod
 + Liveness probe for each pod
 + Mounts that might be required for your app (specified in rc's yaml). NFS example in presentation
- Services: a Kubernetes-aware software load balancer, presents collection of pods as a single service, targeted by labels applied to the pod.
 + specify port the service is exposed on
 + and the port on the pod to be load balanced

`kubectl get nodes` - Retrieves list of nodes, raw systems that are willing to run pods

Set of Selenium servers was used as demo, emulates a browser that is programatically controllable
`kubectl exec` can run commands, including interactive bash, on a container in a pod, first by default. A pod may be composed of multiple filesystem namespaces even though it shares a single network namespace.

Killing the java process internally fails health check, causing replication controller to kill pod and re-spawn somewhere else
Killing an entire node is recognized, and the replication controller schedules pods that were running on dead node on new node

"rolling-update" goes pod-by-pod, launches a new image, and kills the pods running old verison

helm is a separate project that acts as a package manager for Kubernetes apps.
`kubectl config use-context aws_kubernetes` switched over to a set of AWS kubernetes nodes (already configured)
Running
`helm install selenium`
https://github.com/helm/charts/tree/master/selenium
Created set of selenium servers, re-downloaded, and ran on the new AWS infrastructure, via Kubernetes

Q: How is kubeproxy (which powers service/software load balancer) performance?
A: Kubeproxy does add some overhead, new Kubernetes 1.1 uses iptables

Q: Multi-data-center Kubernetes?
A: You're better off keeping Kubernetes within the same datacenter. "Ubernetes" is a layer on top which helps tie multiple data centers together


---------------------------------------------------------------

Second Presentation: Provisioning (Asset Lifecycle) at Jive, Devon Peters, Tools Team

New Asset workflows
 - Intake
 - Provision
 - Production
Decommission workflows
 - Cleanup
 - Destroy

Collins - Asset Tracking/Source of Truth
xCAT - Provisioning/Hardware discovery
j-tech - Jive specific business logic

Tumblr named after Phil Collins, infrastructure automation. Features:
- CLI, Web interface, API & client libs
- Tracks servers, cabinets, racks, DC
- Permission system

<lots of unsettling pictures of Phil Collins>

Callbacks allow you to take actions on change (i.e. if it goes from new to unallocated, provision)

Assets
Tags - "All tags can be inspected and acted upon in callbacks"
 - Managed (hardcoded, facts)
 - Automated (programmatically applied)
 - Unmanaged (yours/custom)

Status - "Processes should drive status change, not people"
- Incomplete
- New
- Unallocated
- Provsioning
- Provsioned
- Allocated
- Canceled
- Decommissioned
- Maintenance
States - Additional info related to status
Type
Log - All changes audited and inspectable



xCAT - IBM
- Baremetal node discovery and provisioning
- Massively scalable (10k+ nodes)
- Helps you control the on-board ILOM (iDRAC)
- kickstart, monitoring, etc

Servers == nodes
nodes have properties and belong to nodegroups
`lsdef` and `chdef` to view and control nodes and nodegroups

Uses switchport layout for discovery
Discovery always uses DHCP for discovery, but once it is up, it can be assigned a static address
regexp to identify switchport extraction
Once discovered, PXE boots a genesis image, waiting for further instructions

Supports
- Kickstart with provided ISO image
- CoreOS boottarget, memory-based PXE'd initrd

To manually install,
`rinstall -O osimage=centos6.5-x86_64 dell002-dell022
`rcons` provides console output

Can use a nodegroup to PREVENT installation, such as adding all your nodes to a "provisioned" group that is untargeted in rinstall command, avoiding accidentally imaging a box

xCAT can also run post-install scripts after kickstart

Q: Cons?
A: xCAT is a beast, written by IBM, in Perl. Collins is rigid, do not defy the lifecycle model.

Q: Does collins instruct machine to talk to xCAT?
A: Let's talk about j-tech!

j-tech is callback for collins
Simple wrapper script specific in Collins yaml config format

Sets up DNS, IP allocation, and xCAT node attributes

Logs status back to Collins
Jive has ~100 profiles, roles, cpu, memory, xCAT membership, etc

Jive extended the collins_client gem, made aware of data centers, environments, etc. Can also export into rundeck
Collins data used in Puppet ENC, and made available via $::collins

thx Jestin Woods, Derek Olsen, and Tumblr. Also, IBM kinda.