aiwantaozi

#!/usr/bin/env bash

# Script to (selectively) save/load multiple Docker images to/from a directory.
# Run ./save-load-docker-images.sh for help.

set -e

directory=$PWD
filter=""

aiwantaozi

Phase 1

Features

• support write audit log to file and ship to remote database - The audit log could be enabled on the global level, user could config audit log, remote database to ship audit log on the UI. Once user enable audit log and config the remote target, we will write audit log to file inside container and ship the log to the remote target. Audit log will include some metadata information in Phase 1.
  • local file - The audit log will generate inside the rancher server container.
  • remote database - we use fluentd to ship logs, user could config remote database endpoint in the UI, like mysql, elaticsearch, hadoop, kafka and so on.
• simple audit log list, query in UI - if user config outside mysql as the audit log storage, user could list and have simple audit log query. The global level audit log will show all cluster audit log, and the cluster level will only show the related cluster audit log.

Storage

why not store in etcd

• we may store several days audit l

aiwantaozi

what crd audit log look like?

It will include the impersonatedUser field, which is rancher user-id. this is for user create/update a new crd obj.

Metadata Level Sample

{
   "kind":"Event",
   "apiVersion":"audit.k8s.io/v1beta1",
   "metadata":{
      "creationTimestamp":"2018-05-04T08:48:27Z"
   },

aiwantaozi

Feature

• Could enable audit log when use RKE to deploy. may add a config here. https://rancher.com/docs/rancher/v2.x/en/installation/server-installation/ha-server-install/
• Could config global audit under security, the page will look like our other logging screen
• For each user cluster, if the cluster enable audit log, and config the global audit log target, will deploy fluentd to the rancher server cluster and get the deliver the log to global target.

aiwantaozi

CPU

CPULoad1

"cpu_load1"
represent the 1 min cpu load averages each core.
Scope: cluster, node
expr: 
    cluster: sum(node_load1) / count(node_cpu{mode="system"})
    node: sum(node_load1{instance=~"$node.*"}) / count(node_cpu{mode="system", instance=~"$node.*"})

CPULoad5

aiwantaozi

Cluster Metrics

cluster_memory_usage_percent

1 - sum(node_memory_MemAvailable_bytes{instance=~".*"}) by () / sum(node_memory_MemTotal_bytes{instance=~".*"}) by ()

cluster_network_transmit_packets_dropped_sum_rate

sum(node_network_transmit_drop_total{device!~"lo|veth.*|docker.*|flannel.*|cali.*|cbr.*",instance=~".*"}) by ()

cluster_cpu_load_5

sum(node_load1{instance=~".*"}) by () / count(node_cpu_seconds_total{mode="system",instance=~".*"}) by ()

cluster_cpu_user_seconds_sum_rate

sum(rate(node_cpu_seconds_total{mode!="user"}[5m])) by ()

aiwantaozi

Logging

Support feature

• Collect docker stdout/stderr logs

Merge Steps

Release rancher dapper 1809 and 1903 image

Now we use maiwj/dapper:windows-1903 and maiwj/dapper:windows-1809 to build

Transfer configmap-reload repo

Repo

Build configmap-reload windows image and update image in system charts pr

PR

aiwantaozi

Steps to integrate with aliyun log service

Kafka

Steps:

1. Go to the aliyun log server page and create a project, remember the region you select, different region have different access endpoint

Log Service Access Endpoint For Different Region

2. Continue to create logstore after project created

3. Base on this doc configure the rancher logging target

aiwantaozi

Features

Upgraded Fluentd gems

• Issue: rancher/rancher#22219
• Issue: rancher/rancher#20509
• Issue: rancher/rancher#20450

Added configurations for multi-line loggging

• Issue: rancher/rancher#22194
• Issue: rancher/rancher#21900

aiwantaozi

Logging trouble shooting steps

Dependence

Check docker log driver type

• Command: docker info
• Confirm logging driver is: json-file

Check docker root

• Command: docker info
• If your docker root is not the default /var/lib/docker, just update the docker root in Cluster Options page