Skip to content

Instantly share code, notes, and snippets.

@slim-bean
Created June 9, 2022 12:49
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save slim-bean/663521806c7b1aff98b386449f3487dc to your computer and use it in GitHub Desktop.
Loki Essential Config Settings 2022/06/09

Loki config reference

This is kept updated with best practices, it applies only to the single binary, non horizontal deployment

https://github.com/grafana/loki/blob/main/cmd/loki/loki-local-config.yaml

Command to diff configs

diff --color=always --side-by-side \
  <(docker run --rm -t -v "${PWD}":/config grafana/loki:2.5.0 -config.file=/config/loki-old.yaml \
    -print-config-stderr 2>&1 | sed '/Starting Loki/q' | tr -d '\r') \
  <(docker run --rm -t -v "${PWD}":/config grafana/loki:2.5.0 -config.file=/config/loki-new.yaml \
    -print-config-stderr 2>&1 | sed '/Starting Loki/q' | tr -d '\r') | less -R

Links to more info about zone awareness

https://grafana.com/docs/mimir/latest/operators-guide/configuring/configuring-zone-aware-replication/#configuring-ingester-time-series-replication
https://github.com/grafana/rollout-operator

Example config with all the settings discussed

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  http_server_read_timeout: 310s
  http_server_write_timeout: 310s
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600



common:
  path_prefix: /opt/loki
  storage:
    s3:
      s3: https://s3.us-east-2.amazonaws.com
      bucketnames: bucket123
      region: us-east-2
      access_key_id:
      secret_access_key:
  replication_factor: 3
  ring:
    kvstore:
      store: memberlist

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

memberlist:
  join_members:
    - loki:7946

querier:
  max_concurrent: 8 # default is 10
  query_timeout: 300s
  engine:
    timeout: 300s

ingester:
  chunk_encoding: snappy

ingester_client:
  grpc_client_config:
    max_recv_msg_size: 104857600

limits_config:
  ingestion_rate_mb: 20             # use to protect cluster form too much data
  ingestion_burst_size_mb: 30
  per_stream_rate_limit: 3          # use to protect individual ingesters from overload by a single stream
  per_stream_rate_limit_burst: 10
  max_global_streams_per_user: 5000 # try to keep a single tenant from generating more than 300k streams in a 24 hour period or label/series queries get very slow
  split_queries_by_interval: 30m    # default is 30m, decrease for very high volume data streams, increase for low volume data streams queried over longer time periods (weeks or months)
  max_query_parallelism: 32         # our ranges are typically 16-256, the higher you go the more queriers you need: max_query_parallelism <= [num queriers] * [max_concurrent]
  cardinality_limit: 500000         # default 100,000 increase if you get a 'cardinality limit exceeded for...' error

storage_config:
  disable_broad_index_queries: true # boltdb-shipper only
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment