-
DevOps is not a team or title, but a philosophy
-
DevOps culture, change behavior
-
Teams and Tools
- Kaizen habits
-
Tools
- IAC, CI, CD, Automation, Metrics
-
Matterhorn - cms
-
Sesnu - self registering
-
influxDB for metrics
-
Domain cookbook
-
recipe for each componenet (varnish, haproxy)
-
roles are wrappers on recipes
-
common functionality factored into full suppotring recipes
-
compnent recipes follow wrapper pattern for community cookbooks
-
Debian packages for code -> apt repo
-
Containers & micro-services
-
artifact repo for cookbooks
-
a place to discover other people's work
-
supermarket-omnibus wrapper
-
private s3 bucket
-
enterprise chef for auth
-
Chef
-
Docker
-
Apache Mesos / cluster manager
-
Marathon - process long running jobs
-
Chronos - cron
-
Consul - key value store / ZK
(This was the worst presentation)
github.com/bloomberg/chef-bach
- Bloomberg Clusters
- hadoop kafka
- environment specific
- networking/storage
- built regularly
- dedicated "bootstrap" server
- virtual machine
- dedicated chef-server
Lightweight Pre-req
-
low mem/storage
-
Rapid Deployment
- Vagrant to bring-up
- Vagrant ot reconfig
Services Offered c* obbler-cookbook
- kerberos kdc
To specify path inside the VM w/Vagrant
[[:vm,""]]
github.com/bloomberg/chef-back/tree/kerberos
- negative flagging - down
- positive flagging - is reconfiguring
- deadlock avoidance
Contention
- Jenkins wait_until_ready
freenode: #chef-bach
@jezhumble
"Agile: We now take orders standing up rather than sitting down"
- lead time for changes
- release frequency
- time to restore service
- change fail rate
app pef & infra monitoring tools to make business decisions daily
Westrum, "A tpology of organizationl cultures" http://bmj.co/1BRGh5q
how quickly can i restore service? normal change process for emergencies. if you can't use it, it's broken.
how quickly can i get a criticla fix to users?
how quickly can i validate if a feature is valuable?
Amazon May Deployment Stats MTBD 11.6 s 1079 deplyments in a single hour 10k mean hosts simulatneous deployment 30k max host simultanous receiving deploy
do less "Online Experimentation at Microsoft" http://standord.io/130uWeX
Impact Mapping, Gojko Adzic
Economic value optimization
Book: Release It! Design and Deploy Production ready Software
Steve Yegge's platform rant
glinden.blogspot.com : innovation culture 2006-04 early amazon shopping cart
Shankey kane - Your startup is broken, inside the toxic heart of tech culture
@clintoncwolfe
Perceptions may differ
http://www.devopsdictionary.com
Goat: a multifunctional individual who tends to ignore the silo boundaries of an organization
goatlike, caprine, Michael Ducy in 2013 slide stack Goat and the Silo
Pets hand maintained, huggable hosts, long lives
Bill Baker of Microsoft, popularized by Gavin McCance
Cattle nodes or servers that are easily replaced
Unicorn (1) a company where DevOps just works, (2) an impossible to satisfy job description
Canaries a set of nodes that get an update first, and watched closely
Yak shaving From a 1993 episode of Ren and Stimpy.
Chaos Monkey Software that kills running instances as part of resiliency testing. Part of the "Simian Army" toolkit.
ELK elasticsearch, logtash, kibana. Aims for the same functionality as Splunk
ZDT zero downtime. refering to any deployment strategy that results in no apparent outage
OODA Observe, Orient, Decide, Act. Cognitive loop used in incident response (et al).
CAMS Culture, automation, MEasurement, Sharing. DevOpsDays Mountain View
DSL Domain Specific Language. A syntax and command set designed so that users can write in their conceptual model
TLA Three letter acronym
CoffeeOps series of informal meetups at coffee shops to talk DevOps. Started by Jennifer Davis. coffeeops.org
HugOps empathy among engineers esp. after incident, disagreement, tragedy. Jordan Sissel, BRandon BUrotn, Adam Jacob.
ChatOps using chat to interface to your tooling
Brent character in the Phoenix Project. A human bottleneck.
Sparkly DevOps Princess
KiltOps uhhhh...
Monkeypatching
Joe Damato (packagecloud.io)
Slides: blog.packagecloud.io
tahdig - persian food, awesome
NOTE: GPG signatures are never checked in debian packages
1:1.2.8-1 epic version/package maintainer:upstream maintainer
yum (Yellowdog Updater, Modified)
apt (advanced package tool)
- debbuild
- rpmbuild
- git-buildpackage
- fpm
- omnibus (from chef)
- mock and pbuilder (more advanced)
- createrepo - yum
- reprepro - apt
pygpgme for yum
Codename: {distro}
Componenets: {names}
Architectures:
SignWith: {foo}
SignWith to sign repo metadata
gpg --list-keys
Not the same as debsigs/debsign sign the package
XML policy file for signature verification file
reprepro cookbook
apt-transport-https package
@muktaa
Codify your configuration Set of bash commands Example:
HelloScala Dockerfile dev.conf
FROM:ubuntu:14.04
RUN apt-get update
RUN apt-get install libfuse-dev
ADD dev.conf /etc/myapp-config/
Use cases:
- PaaS
- Microservices
- Lightweight Testing
The Challenge
- control the env vs syste mimage/runtime image
- Tradeoff between flexibilty and management
Config Management is the vein of DevOps Shell scripts to Chef
Immutable infrastructure
Chef-client being pull based from Chef Server
Using push jobs to run changes
Knife-ssh works like push. "Almost"
####Chef Push Jobs
Uses a Chef-specific message bus
Job is a set of commands to run on a node
- docker pull
- docker stop
- docker run
Doesn't mesh well with Chef config
####knife-ssh
Parallel ssh is slow feedback on deployment status is nice
####Chef Push Jobs (continued)
knife-jobs gem
knife-jobs cookbooks on each node
Findings: Chef Push jobs were too complex for use case. Didn't work all the time.
Application + Configuration = Image
Connfiguration in Docker: Credentials, Packages, Software, Database, Ports, Files.
Example:
- Mount S3 for the app instead of on the app host.
Be environment specific
Major Issues: Credential management.
Secure Credential Management
- Unsolved problem in Docker today / its not secure
- Credentials have to be hard-coded inside container
- Set them as environmental variable otherwise
What's inside:
- chef-client
- runit
- chef-init (runs on PID 1 of container)
- Bootstrap container without SSH connection
- Manage multiple services inside container
- Manage running state of your container
- Consistency across architectuers
- Mixed architecture applications
- Transitioning traditional architecture to containers
- Handling last mile configuration when container boots
- Getting the best of two worlds without complexity
gem install knife-container
- Running apps in containers is easy
- Debugging in containers is "very" hard
- You can very well run multiple services inside a docker container
- Docker networking is horrible
- Sequential progression (getting from no docker containers to iterating over docker containers) is difficult.