Philly DevOps Notes 2016-11-29

phillydevops20161129.md

Argument for why you shouldn't build private clouds for < 200K nodes: https://m.subbu.org/dont-build-private-clouds-9a54b3d30c8b#.hxdkjwtix

Main talk: Managing secrets in AWS w/ Hashicorp Vault

Why Vault?

• Platform independent
• Active development
• Great toolchain
• HA
• Authentication for users and systems
• Distributed master key never stored on disk
• In-memory dev mode for testing

Talking to Vault

• HTTPS REST JSON API
• vault command as a friendly CLI

Security

• Must be unsealed on start using Shamir's Secret Sharing
• Splits key into n shards
• k shards needed to derive original key (k < n)
• No one person can easily obtain the original key
• No need to revoke shards if people leave
• Key only stored in memory
• Restarting vault will lose key and seal the vault
• Access to secrets on limited lease basis which must be renewed

Backends

• aka "Plugins"
• Storage
• Secrets
• Authentication
• Auditing

Storage

• Many backends supported
• Consul, Zookeeper, etcd only support HA
• inmem is useful for dev mode / testing

Clustering

• Use storage backend w/ HA support
• 1 active n standby model
• Use Consul DNS or LB + healthcheck to find the active Vault
• Standby server will redirect to active by default

Secrets

• Static secrets
  • usernames
  • passwords
  • private keys
  • API tokens
• Dynamic secrets
  • AWS credentials
  • SSH credentials
  • X.509 certificates / CA

Tokens

• All authentication boils down to tokens
• Tokens are UUIDs returned by the Vault authentication backends
• Limited lifetime TTL
• Renewable
• Root Token generated at initialization is the topmost auth credential

Authentication backends

• AppRole
• AWS EC2 (authenticates instances to Vault)
• GitHub
• TLS
• LDAP
• Tokens
• Username & password

Roles

• Auth requests can contain a role
• Bind Vault users to policies
• Example use: can auto-authenticate EC2 instances to Vault

Policies

• ACLs applied to roles, users, groups
• Tokens can have multiple policies

Using secrets

• vault client
• REST API
• client libraries
• consul-template
  • use Jinja2 like: {{ with secret "secret/foo/bar/baz" }}

Learnings

• Don't use a self-signed CA
• For SSL, name your servers or use static IP addresses
• Clustering is easy. Do it.
• Think about your path hierarchy ahead of time
• Be prepared to RTFM
• Different backends behave differently
• Tokens do not live forever