Istio Community meeting Circonus adapter overview

istio_bookinfo_latency.png

istio_community_meeting.md

Developing the Istio Mixer Circonus metrics adapter

Presented at the Istio community meeting 2017-12-14

Inception

Hey Fred, this Istio thing looks pretty cool - Theo

Ok, I'll go check it out - Fred

Should be pretty fast to implement with the circonus-gometrics library - Theo

Sounds good, yeah this does look cool. Probably couple weeks to code up - Fred

At the Istio presentation in August:

Send PRs! - @zackbutcher and @douglas-reid

...4 months later

PR accepted! - Fred

What is Circonus?

• Highly scalable, highly performant commercial monitoring solution
• Founded in 2010 by Theo Schlossnagle, background in distributed systems
• Born out the pain experienced with other monitoring solutions that didn't scale to high workloads
• Focused on data correctness and scalability via histogram based data storage
• Similar design principles to Google's Monarch; different from Prometheus which evolved from Borgmon

Why did we embark on this effort?

• High QPS workloads in Istio/Kubernetes environments is where Circonus excels over other monitoring solutions
• The circonus-gometrics library facilitated a well defined integration interface
• Istio looked cool!

Off we go, development against Istio version 0.1

• Used the statsd adapter as a guide
• Development done on an iMac with minikube, Docker w/ xhyve driver, homebrew for bazel
• Zack Butcher's Golang Development with Bazel helped out a lot with Bazel basics
• Started off with vim development with the vim-go plugin, moved to GogLand after tips from Istio devs
• Learned about fmt.sh and some of the other utilities the hard way, too focused on adapter code
• Submitted PR against 0.1 codebase shortly before the 0.2 handler framework was being implemented
• Good feedback from initial PR request, some comments shaped changes to circonus-gometrics test methods
• Lots of time spent on Bazel, observations inline with the discussion on Tuesday
• Being on the outside, it was somewhat challenging to determine what development style was desired based on existing code

Community testing day

• I participated in that, the guides were great in getting the BookInfo example set up
• Ended up submitting a couple of documentation PRs. Overall documentation is very good.
• Good to be able to interact with other members of the community as opposed to the 'outside in' experience developing alone

Adapter development redux, Istio version 0.2 adapter reworking

• Started from scratch by reading through the Adapter Developer's Guide; mostly a great document, but some abstract concepts take focus to fully grok
• Went through the Adapter Development Walkthrough docs, rebuilding the adapter by porting pieces of v0.1 code and reworking where needed
• Attempted writing a new set of tests with the testenv framework. Got it working, but was not able to introspect the handler to call test methods on the circonus-gometrics object. Ended up going with unit tests instead, calling HandleMetric directly in the tests. I like the testenv approach, and think it will be great with some additional utilities (or examples).
• Got the adapter working with the functional and unit tests in the guide - but we have to make sure it works under fire

Let's see it in action to make sure it works

• Tried to deploy the custom adapter on my iMac - whoops, looked like a WIP for Mac
• Deployed K8S on VMWare on Mac to load the custom adapter via @mandarjog's instructions
  • Huh, why didn't that work? Answer: Yo dawg, you put a VM in a VM. Try again
• Enter Google Kubernetes Engine, literally the day I needed it.
• Poof, you have a K8s cluster. Half an hour later, I had Istio up with BookInfo. A day later I had the Circonus adapter loaded (stuck on making my Docker image public in GCR for a few hours).
• A couple hours of bug fixes, then I threw some real requests at it. Here is a heatmap of the request latencies collected, along with median, 90th percentile, and 99th percentile overlays. Ok, I'm convinced it works now.

Github Istio repo reorganization

• Went to submit PR - oh, the repo has moved over here!
• Took a few hours to port the code over to the new repo
• Bazel bazel bazel...
• Used the now official custom adapter loading docs
• Ok let's test it all front to back again... ready to go!

PR #1737

• /me gulps and hopes this goes well
• Great constructive and honest feedback from Martin Taillefer - thanks Martin!
• Went back and hacked a log bridge in place to match circonus-gometrics golang log pkg
• Flailed with metrics flushing until I discovered I was passing my object to ScheduleDaemon() by value, not reference
• Figured out how to shutdown my metrics flusher goroutine properly via context.WithCancel()
• Side detour PR to fix CircleCI dependency cache busting for tests - thanks @rshriram
• /approved /lgtm 🍻
• Blog post writeup on Circonus.com blog

Things I learned

• I managed to recode against almost every API change in Mixer from 0.1 to current
• Mailing list responses were helpful for some 'oh duh' things I forgot to do
• GoLand really eats up a lot of CPU some of the time, but is great at identifying issues before compilation
• Bazel gets in a weird state about 30% of the time, and throws delays into your development work as a result
• PR reviewers were kind, yet called me out on stuff that didn't looks right (I really appreciate both things there)
• Is there an external slack channel somewhere? That could have been really helpful! (though maybe annoying to Istio devs)
• The end adapter came out really well codewise in my humble opinion. Was a lot of fun to write, maybe I'll do another!

Thank you!

• I'll be submitting a couple other documentation related PRs
• Feel free to ping me offline with questions, fredmoyer@gmail.com, @phredmoyer
• Happy to provide any additional feedback more in depth about the experience to team members
• Where can I get Istio schwag? 😃
• If you want to try out the adapter, you can grab a free Circonus account. Tweet me up with any questions or slack me at the Circonus Labs Slack. Adapter should be in core Istio with the 0.4 release (no custom loading needed then).