0-dnd-notes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f6b147a4-d75d-4121-9797-4a83286151b5",
   "metadata": {},
   "source": [
    "# Telemetry, Instrumentation, and Observability with OTel"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db527c1a-6e4d-443d-bf61-4b1e35dfe45f",
   "metadata": {},
   "source": [
    "## DnD Notes\n",
    "\n",
    "> OpenTelemetry is a set of APIs, SDKs, tooling and integrations that are designed for the creation and management of telemetry data such as traces, metrics, and logs. The project provides a vendor-agnostic implementation that can be configured to send telemetry data to the backend(s) of your choice. It supports a variety of popular open-source projects including Jaeger and Prometheus. [1](https://opentelemetry.io/docs/concepts/what-is-opentelemetry/)\n",
    "\n",
    "OpenTelemetry itself is just a set of standards which, when implemented, allow telemetry data to propagate through various systems and then be collected, exported, and displayed by observability tools\n",
    "\n",
    "### Telemetry\n",
    "\n",
    "Telemetry: automatic cellection of measurements (traces, metrics, logs) or other data in software running locally or remotely, and the automatic transmission of that data. Telemetry is made up of:\n",
    "\n",
    "- Traces\n",
    "    - Traces are made up of 'spans', spans contain RED metrics (**R**equest, **E**rror, **D**uration)\n",
    "    - Spans \"represent the work being done by **individual services or components involved in a request** as it flows through a system\", they represent a **single unit of work**\n",
    "    - Spans are defined by:\n",
    "        - operation name\n",
    "        - start time\n",
    "        - end time\n",
    "        - parent (optional)\n",
    "        - unique identifier\n",
    "        - context information*\n",
    "        - attributes (dict of primatives)\n",
    "        - events - timestamp in the span with a name and (optional) attributes\n",
    "    - A trace has a single 'root span' (for one the main logical operation triggered by the request) which can contain multiple 'child spans' (individual operations triggered as part of the request)\n",
    "    - To trace relationships and context between spans a Distributed Context* is used, this context includes identifiers and tags which are propagated from parent to child spans\n",
    "- Metrics\n",
    "    - Measurements captured at runtime, with a specific timestamp, comprised of things like:\n",
    "        1. counter - value that always increases, e.g. no. of requests, e.g. cumulative counters (total amount of data processed), cumulative histograms (average processing time, aggregated over ranges, 1-5 min, 5-10, 10-20, etc...)\n",
    "        2. measure - aggregated value over time, e.g. rates like average ram usage, processing requests per hour\n",
    "        3. observer - current set of values at a specific time, e.g. snapshot of current ram/cpu/network usage\n",
    "- Logs\n",
    "    - Timespamped text record, may or may not be associated with a span\n",
    "\n",
    "### Instrumentation\n",
    "\n",
    "Instrumentation is the process of implementing telemetry collection, it can be:\n",
    "\n",
    "- Manual instrumentation: telemetry is captured by manually defining traces, spans, metrics, and logs, within the code itself, explicitly calling OTel methods and classes\n",
    "- Automatic instrumentation: OTel has the concept of **instrumented libraries**, where telemetry collection is automatically enabled for specific libraries (e.g. python's logging library, FastAPI, etc...)\n",
    "\n",
    "### Observability\n",
    "\n",
    "OTel telemetry data is just some JSON with standardised fields, it's not very human readable\n",
    "\n",
    "Observability tools are what help you turn that telemetry into meaningful human-interpretable data by visualising traces, metrics, and logs, and by helping correlate and filter them\n",
    "\n",
    "One main benefit of using OTel is that all major observability tools have support this standard\n",
    "\n",
    "### Data Collection\n",
    "\n",
    "Once an application is instrumented and is outputting telemetry JSON, it needs to be captured and ingested. OT has a built in collector which recieve, process, and then export the telemetry data to multiple observability applications\n",
    "\n",
    "> The OpenTelemetry project facilitates the collection of telemetry data via the OpenTelemetry Collector. The OpenTelemetry Collector offers a vendor-agnostic implementation on how to receive, process, and export telemetry data. It removes the need to run, operate, and maintain multiple agents/collectors in order to support open-source observability data formats (e.g. Jaeger, Prometheus, etc.) sending to one or more open-source or commercial back-ends. In addition, the Collector gives end-users control of their data. The Collector is the default location instrumentation libraries export their telemetry data.\n",
    "\n",
    "Colectors are made up of three components:\n",
    "\n",
    "- <img width=\"32\" src=\"https://raw.github.com/open-telemetry/opentelemetry.io/main/iconography/32x32/Receivers.svg\"></img>\n",
    "`receivers`: How to get data into the Collector; these can be push or pull based\n",
    "- <img width=\"32\" src=\"https://raw.github.com/open-telemetry/opentelemetry.io/main/iconography/32x32/Processors.svg\"></img>\n",
    "`processors`: What to do with received data\n",
    "- <img width=\"32\" src=\"https://raw.github.com/open-telemetry/opentelemetry.io/main/iconography/32x32/Exporters.svg\"></img>\n",
    "`exporters`: Where to send received data; these can be push or pull based\n",
    "\n",
    "These components come together in **pipelines** which define the configuration and flow of telemetry data through all three components"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}

1-example-tracing.ipynb

2-example-jaeger.ipynb

3-example-flask.py

import flask
import requests

from opentelemetry import trace
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    ConsoleSpanExporter,
    SimpleSpanProcessor,
)

trace.set_tracer_provider(
    TracerProvider(
        resource=Resource.create({SERVICE_NAME: "flask-example"})
    )
)

jaeger_exporter = JaegerExporter(
    agent_host_name="localhost",
    agent_port=6831,
)

trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(jaeger_exporter)
)

app = flask.Flask(__name__)
FlaskInstrumentor().instrument_app(app)
RequestsInstrumentor().instrument()

@app.route("/")
def hello():
    tracer = trace.get_tracer(__name__)
    with tracer.start_as_current_span("example-request"):
        requests.get("http://www.example.com")
    return "hello"

app.run(debug=True, port=5000)

4-example-ipython.ipynb

5-future.ipynb