Skip to content

Instantly share code, notes, and snippets.

@vigoo

vigoo/cx-operator-tutorial.md Secret

Last active February 2, 2021 12:36
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save vigoo/f4de0324e08898df38ba751a660cb1a8 to your computer and use it in GitHub Desktop.
Save vigoo/f4de0324e08898df38ba751a660cb1a8 to your computer and use it in GitHub Desktop.
Download ZIP
Coralogix Kubernetes Operator tutorial
Raw
cx-operator-tutorial.md

Coralogix Kubernetes Operator

What it is

The Coralogix Kubernetes Operator or (cx-operator) makes it possible to configure Coralogix through custom resources. These custom resources behave just like any other Kubernetes resource; their main benefit is that they can be packaged and deployed alongside services deployed in the cluster (through a Helm chart, for example), and their definitions can be managed by source code management tools. This also means that the process for modifying Coralogix rules and alerts can be the standard code review process applied to application changes. This is a massive win on all fronts: better auditing, better change management, easier rollbacks and more.

The cx-operator supports three types of custom resources:

  • Rule group sets (rulegroupsets.coralogix.com)
  • Alert sets (alertsets.coralogix.com)
  • FluentD Loggers (coralogixloggers.loggers.coralogix.com)

Installation

The Coralogix Kubernetes Operator can be installed to the Kubernets cluster with Helm.

The first step is to add cx-operator's repository:

helm repo add cx-operator https://coralogix.github.io/cx-operator/

To see if that worked, search for the cx-operator chart:

helm search repo cx-operator

It should find the latest published version of the Coralogix Kubernetes Operator:

NAME                   	CHART VERSION	APP VERSION	DESCRIPTION
cx-operator/cx-operator	0.3.5+ddf73a7	0.3.5      	Coralogix Kubernetes Operator

The operator communicates with the Coralogix API with the user's API key. This can be generated in the Account settings menu:

In Settings –> Account, Choose ‘Alerts API Access’ option and generate new Alerts & Rules API key:

** Note that only admin users have access to the API, So the option above will be visible only to admin users.

Once you have the key, it has to be stored in a Kubernetes Secret in your cluster:

kubectl create secret generic coralogix-operator-secrets --from-literal=RULES_API_TOKEN=00000000-0000-0000-0000-000000000000

Just replace the 00000000-0000-0000-0000-000000000000 with the API key from the settings page mentioned above. This will be used both for defining log processing rules and alerts.

With the secret in place all that remains is to install cx-operator into the cluster:

For accounts in Europe:

helm install --set config.coralogixApi.host=grpc-api.coralogix.com cx-operator cx-operator/cx-operator

For accounts in India:

helm install --set config.coralogixApi.host=grpc-api.app.coralogix.in cx-operator cx-operator/cx-operator

In case you also want to install a Prometheus ServiceMonitor for the operator, add the --set serviceMonitor.create=true option to the install command.

Log parsing rules

Log parsing rules can be defined using the rulegroupsets.coralogix.com custom resource. Please read the log parsing rules tutorial for more information about how these rules work.

To define one or more log processing rules with the Coralogix Kubernetes Operator, you have to create one (or more) rule group set resources.

The following example defines two rule groups:

apiVersion: "coralogix.com/v1"
kind: RuleGroupSet
metadata:
  name: test-rulegroupset-1
spec:
  rule-groups-sequence:
    - name: 'Operator Test Rules 1'
      matcher:
        applications:
          - app1
          - app2
      and-sequence:
        - or-group:
            - name: 'Delete prefix'
              enabled: true
              source-field: Text
              replace:
                rule: '.*{'
                dest-field: Text
                new-value: '{'
        - or-group:
            - name: 'Extract bytes and status'
              description: 'Extracting bytes and status from message field'
              enabled: true
              source-field: Text
              extract:
                rule: '.*'
            - name: 'Worker to category'
              enabled: true
              source-field: worker
              json-extract:
                dest-field: Category

    - name: 'Operator Test Rules 2'
      matcher: {}
      enabled: false
      and-sequence:
        - or-group:
            - name: 'Block 28000'
              description: 'Block 28000 pg error'
              enabled: true
              source-field: field1
              block:
                rule: 'sql_error_code\s*=\s*28000'
                keep-blocked-logs: false

The exact, up-to-date definition of this custom resource can be found here.

The top level rule-groups-sequence is a list of rule groups to create. Each of these has the following properties:

Field name Required? Description
name required The name of the rule group
description optional Textual description of the rule group
enabled optional Enabled state
hidden optional Controls the hidden state of the rule group
matcher required A set of log matching rules, see below
and-sequence required The list of rules belonging to the rule group

The matcher block can have the following items:

Field name Required? Description
applications optional List of application names to apply the rule group to
subsystems optional List of subsystem names to apply the rule group to
severities optional List of log severities (DEBUG, VERBOSE, INFO, WARNING, ERROR or CRITICAL to apply the rule group to

Each item of the and-sequence must be an or-group containing a list of rules. This hierarchy makes it possible to express complex rules in the form of "(Rule1 or Rule2) and Rule3 and (Rule4 or Rule5 or Rule6)".

Rules have the following properties:

Field name Required? Description
name required The name of the rule
description optional Textual description of the rule
source-field required The field of the log entry which is used by the rule
enabled optional The enabled state of the single rule

Additionally to these there must be exactly one of the following parameters:

  • extract with a field rule defines an extract rule that copies information from the source field to new fields.
  • json-extract with a field dest-field (one of CATEGORY, CLASSNAME, METHODNAME, THREADID or SEVERITY) defines a JSON extract rule
  • replace defines a replace rule with dest-field selecting the target field, rule containing the regular expression and new-value the value to set the destination field to.
  • parse defines a parse rule with dest-field selecting the target field and rule containing the regular expression.
  • allow defines an expression in rule that selects log entries to not be blocked. The optional keep-blocked-logs field enables storing the blocked entries on S3.
  • block is the opposite of allow, it defines an expression in rule that selects log entries to be blocked. The optional keep-blocked-logs field enables storing the blocked entries on S3.

To install the rule group set, apply it to the cluster:

kubectl apply -f test-rulegroupset-1.yaml

Then verify that the custom resource has been stored:

kubectl get rulegroupsets

Alerts

Alerts can be defined using the alertsets.coralogix.com. One AlertSet resource can contain any number of alerts. To learn more about the different alert types, read the following tutorials:

The following example defines a single alert:

apiVersion: "coralogix.com/v1"
kind: AlertSet
metadata:
  name: test-alertset-1
spec:
  alerts:
    - name: test-alert-2
      description: "Testing the alerts operator"
      isActive: false
      severity: WARNING
      filters:
        filterType: TEXT
        severities:
          - ERROR
          - CRITICAL
        metadata:
          applications:
            - production
          subsystems:
            - my-app
            - my-service
        text: "authentication failed"
      condition:
        type: MORE_THAN
        parameters:
          threshold: 120
          timeframe: 10Min
          groupBy: host
      notifications:
        emails:
          - security@mycompany.com
          - mgmt@mycompany.com
        integrations: []

The exact, up-to-date definition of this custom resource can be found here.

Each alert in the alert set can have the following fields:

Field name Required? Description
name required Name of the alert
description optional Textual description of the alert
isActive required Determines whether the alert is enabled or not
severity required One of INFO, WARNING or CRITICAL
expiration optional The expiration date for this alert
condition optional Specifies the condition of triggering the alert. Details below.
notifications optional Defines email and webhook notifications. Details below.
filters optional Filtering options, queries and ratio settings. Details below.
notifyEvery optional Supress time for the alert, in seconds
activeWhen optional Specifies the time frames the alert is active. Details below.
notificationPayloadFilters optional List of fields to be included in the alert message

Conditions

The optional condition block has one required field, type which can be:

  • IMMEDIATE
  • LESS_THAN
  • MORE_THAN
  • MORE_THAN_USUAL
  • NEW_VALUE

For all types except IMMEDIATE a second field is necessary, called properties, with the following contents:

Field name Required? Description
threshold optional The threshold parameter, defaults to 0
timeframe required The valid timeframes depend on the condition type. For NEW_VALUE alerts it can be 12H..3M, otherwise 5Min..24H
groupBy required for NEW_VALUE Specifies a field to aggregate into a histogram

Notifications

The optional notifications block specifies the recipients for the alert.

Field name Required? Description
emails optional A list of e-mail addresses
integrations optional A list of webhook integration names

See the Alert webhooks tutorial for more information on how to define these integrations.

Filters

The optional filters block contains several options to further customize the alert.

Field name Required? Description
filterType required One of TEXT, TEMPLATE or RATIO
severities optional A list of log severities to include. Each item must be one of DEBUG, VERBOSE, INFO, WARNING, ERROR or CRITICAL.
metadata optional A set of filters based on metadata. Details below.
alias optional An alias for the first query in case of ratio alerts. Default is "Query 1"
text optional The filter query
ratioAlerts required if filterType is RATIO Specifies the ratio alert. See below.

The metadata block can contain the following options, each one as a list of strings:

  • categories
  • applications
  • subsystems
  • computers
  • classes
  • methods
  • ipAddresses

For ratio alerts the ratioAlerts field contains further configuration possibilities. It is a list of ratio alert specifications, each having the following fields:

Field name Required? Description
alias optional Query alias, defaults to "Query N"
text optional The filter query string
severities optional A list of log severities to include. Each item must be one of DEBUG, VERBOSE, INFO, WARNING, ERROR or CRITICAL.
applications optional A list of application names to include.
subsystems optional A list of subsystem names to include

Active when

The optional activeWhen block has a single property, timeframes, which is a list of timeframe definitions. Each timeframe definition consists of:

Field name Required? Description
daysOfWeek optional A list of days the alert should be active. Each item must be one of MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY or SUNDAY.
range optional A time range with two fields, start and end, both containing a time value in the format of HH:MM:SS.

Alert sets installed through Kubernetes can be queried using the alertsets name:

kubectl get alertsets

FluentD

Coralogix supports integration with FluentD and the Coralogix Kubernetes Operator provides an easy way to set it up.

To install a FluentD daemon set in the Kubernetes cluster, create a CoralogixLogger custom resource:

apiVersion: "loggers.coralogix.com/v1"
kind: CoralogixLogger
metadata:
  name: test-logger
spec:
  cluster_name: TestCluster
  private_key: "01234567-0123-0123-0123-012345678901"

Once the resource is created, cx-operator will provision the following resources:

  • Service account
  • Cluster role
  • Cluster role bindings
  • Daemon set

The daemon set will run the registry.connect.redhat.com/coralogix/coralogix-fluentd:1.0.0 image with the CORALOGIX_PRIVATE_KEY and CORALOGIX_NAME environment variables set to the values defined in the resource.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment