Bridget Kromhout @ Pivotal
Great overview of a lot of standard devops practices, some of the sorrows that can result, and so on. Bridget gives a lot of talks - she's in an evangelist role at Pivotal.
She emphasized a lot of communication issues within orgs. Some recruiter cold emailed her and used as a selling point that their company had two OpenStack deploys. Why is that a good thing?!
"Good to be explicit and not assume defaults." - A great lesson for everyone's documentation ever!
A longer version she gave at CONFENGINE: https://www.youtube.com/watch?v=UjhIA6QTy5k
Zach Steindler @ Duo Security
Maslow's hierarchy of needs as DevOps. This was a great talk, he covered a lot of things.
Notably, he talked about getting hit rate and timing percentiles from Apache log data. Basically, he made the case for why you should care about this data - we already know why, but it's good to see some
Talked about the importance of dashboards, especially the Doom-esque high-level overviews that you look at every day.
David Barnett @ RightBrain Networks
How do I do log processing with EMR, Kinesis, S3, etc.? Similar exaple to Other Zach's talk (above) - identify badly behaving users via their access logs.
Zeppelin notebook - similar to iPython but for distributed data processing. In Python, using Spark. GeoIP data frames to have a look at where traffic is coming from. User-defined functions within Spark that can mutate columns in logging data.
He treated a lot of this talk as a hybrid live-canned demo, almost more like a pseudo-workshop than anything else.
Joining data together. A lot of this stuff is covered by Splunk pretty well, although you might have trouble doing it at scale given that it's access data.
When do you move to this scale? "If you can get it into a single instance, then a simple Python script is fine." Spark is (apparently) entirely in-memory - might need to move into Hadoop which reads off disk past that point. (Me: Is this right?)
Chris Short @ SolarWinds MSP
In the USAF for many years, even in the early aughts they were doing devops, just by any other name. Kanban, just on a whiteboard. Cross-training: "hey can I help with anything?" was pretty common.
Multiple people on a workflow: One person is a Bad Thing. The military thrives on redundancy. Rank mattered less than knowledge: training flowed from people who knew things to people who needed to learn things.
So much documentation: even documentation on documentation. He created WTF style docs: "What's the fix?"
Version control: they didn't have it back then :( "Good configs were nuggets of gold. If you had a good one, people wanted a copy." And, oh boy: "Disaster recovery was based off little more than hope." (Hoping your backup both worked and was the latest copy.)
Testing: "No budget for dev/test/stage." You might have spare equipment but it was very old. Find lowest impact users and deploy! Changes had to be bulletproof, and always had to have a rollback plan as a result.
Military failures are bad: Black Hawk Down, dealing with the Iraqi Army, etc. 1961 Goldsboro B-52 crash: four safeties on two nukes that fell out of a plane. Four safeties, three of which failed. One high-voltage switch away from creating "The Bay of North Carolina".
Hainan Island Crash: Plane crash, full of crypto info, they were frantically destroying data as they landed. He had to invalidate all this crypto and distribute new keys etc... they'd practiced before, only way it worked.
Plan Z: If all else fails, what will you do then?
Diversity: In any form, no one cared. Very very diverse. But if you don't speak out at all, folks don't know who has their back. So it's important not just to be an ally, but to be (if not vocal) at least known to be an ally.
good idea fairy: From Urban Dictionary: "Military term; An evil mythical creature that whispers advice and ideas into the ears of military leadership, causing hundreds unnecessary changes and countless wasted man-hours every year. The Good Idea Fairy should be shot on sight if she is seen in your area."
"Don't let the perfect be the enemy of the good." "If you pulled rank to get your idea accepted, you sucked."
Zach Musgrave @ Yelp
Oh hai.
Kevin Remde @ Microsoft
Basically a pitch for MSFT's PowerShell extensions, Azure, etc. Very Infra as Code oriented.
"Imperative and/or declarative: YOU DECIDE" - I do like that.
You can run the Azure stack in your own datacenter. Hmm, I did not know that.
Ry4an Brase @ Duo Security
(but this talk is not about Duo) At previous startup, he ended up as an exec. Full cessation of engineering output.
First response: FEAR. If you stop coding, how's your next interview gonna go? Fixed it all with money: 33% increase in pay. "If your company doesn't have a promotion path that doesn't involve a suit... run away."
Become a better "internal salesman" for the department. Competing with vendor salespeople. Ugh.
Eng managers shield their teams from BS above. Software execs create the BS from above.
Best takeaway: Better at "managing up". Help your manager frame things for the executives.
Tanya Atanasova @ RightBrain Networks
More women on exec board -> better company performance.
It's a huge problem.
Male CS students collaborate, don't include females.
Lack of role models.
What does make girls choose CS? Lots of social encouragement, for one.
Mentoring: Crucial. Best if from successful women in tech, but anyone can contribute.
Joshua Goldman @ SecureWorks
Birth of an Application: More focus on development than on support.
How do you handoff: various best practices.
"Release week" is really really bad.
Before adding CD to your company you need a good test enviornment, with data included, etc.