adleong

jvm:start_time{type="gauge"} 1.48668993E12
jvm:application_time_millis{type="gauge"} 144758.55
jvm:classes:total_loaded{type="gauge"} 20226.0
jvm:classes:current_loaded{type="gauge"} 20209.0
jvm:classes:total_unloaded{type="gauge"} 17.0
jvm:postGC:Par_Survivor_Space:max{type="gauge"} 2621440.0
jvm:postGC:Par_Survivor_Space:used{type="gauge"} 0.0
jvm:postGC:CMS_Old_Gen:max{type="gauge"} 1.79856998E9
jvm:postGC:CMS_Old_Gen:used{type="gauge"} 4.4330304E8
jvm:postGC:Par_Eden_Space:max{type="gauge"} 3.43670784E8

adleong

// Base Var
alex-boron@ val base = Var(1)
base: Var[Int] with Updatable[Int] with Extractable[Int] = Var(1)@848564270

// Derived Var using map (or flatMap, etc)
alex-boron@ val mapped = base.map { x => println("applying map"); x * 2 }
mapped: Var[Int] = com.twitter.util.Var$$anon$2@5d00e30e

// Observing the derived Var causes the map to be applied for each new observation
alex-boron@ val closable = mapped.changes.respond(println)

adleong

---
apiVersion: v1
kind: Service
metadata:
  name: webapp
spec:
  selector:
    app: webapp
  type: LoadBalancer
  ports:

adleong

Mahkani spice mix:
* 3 tbps curry powder
* 1 tsp garam masala
* 1 tsp coriander
* 1 tsp paprika (smoked, if possible)
* 2 cinamon sticks, snapped in half

Ingredients:
* Mahkani spice mix 
* Rice

adleong

# HELP request_total Total count of HTTP requests.
# TYPE request_total counter
request_total{authority="web-svc.emojivoto.svc.cluster.local",direction="inbound",tls="true",client_id="vote-bot.deployment.emojivoto.linkerd-managed.linkerd.svc.cluster.local"} 618981
request_total{authority="emoji-svc.emojivoto.svc.cluster.local:8080",direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="emoji",dst_namespace="emojivoto",dst_pod="emoji-dddd8f4b-rt8vj",dst_pod_template_hash="dddd8f4b",dst_service="emoji-svc",tls="true",server_id="emoji.deployment.emojivoto.linkerd-managed.linkerd.svc.cluster.local"} 620586
request_total{authority="voting-svc.emojivoto.svc.cluster.local:8080",direction="outbound",dst_control_plane_ns="linkerd",dst_deployment="voting",dst_namespace="emojivoto",dst_pod="voting-694d7cd777-n6mwj",dst_pod_template_hash="694d7cd777",dst_service="voting-svc",tls="true",server_id="voting.deployment.emojivoto.linkerd-managed.linkerd.svc.cluster.local"} 310292
request_total{authority="linkerd.i

adleong

Look at the contents of a data frame: copy data as raw (to paste buffer)

```
pbpaste | cut -c 11- | xxd -r -p | protoc --raw_decode
```

Drop the first 10 characters (first 5 bytes: the gRPC frame header), encode as binary, decode as raw protobuf.

You won't see field names, but you will get field numbers.
Try to match the field numbers and types to known protobuf message types using your brain.

adleong

---
apiVersion: v1
kind: Namespace
metadata:
  name: tracing
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: oc-agent-conf

adleong

/** Simple scala client */

import $ivy.`com.twitter::finagle-http:6.44.0`
import $ivy.`io.buoyant::finagle-h2:1.1.0`
import com.twitter.conversions.time._
import com.twitter.finagle.{Status => _, _}
import com.twitter.finagle.buoyant.h2._
import com.twitter.finagle.buoyant.H2
import com.twitter.io.Buf
import com.twitter.util._

adleong

Linkerd Stale Discovery Runbook

Identifying if you have stale endpoints

If you have a pod experiencing unexplained 503s, check the proxy logs from that pod. If you see connection errors to IP addresses which do not correspond to running pods, your Linkerd proxy likely has stale endpoints. The IP addresses with the connection errors likely correspond to pods which have been recently deleted.

adleong

An unmeshed client (10.8.2.27:45090) sends an HTTP request with curl to a meshed server (10.8.2.26:5000)
the client gracefully closes the connection before receiving a response and the client OS sees the socket in state FIN-WAIT-2
the server OS sees the corresponding inbound proxy socket (10.8.2.26:4143) in state CLOSE-WAIT
it stays this way for around 85 seconds before disappearing from the conntrack table