getting started with fluentbit, fluent, dummy, aws

getting started with fluentbit.md

EC2 instance setup for fluent-bit installation

Step1: Single line install

A simple installation script is provided to be used for most Linux targets. This will always install the most recent version released.

curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

This is purely a convenience helper and should always be validated prior to use. The recommended secure deployment approach is to follow the instructions below.

Step2: Configure Yum

We provide fluent-bit through a Yum repository. In order to add the repository reference to your system, please add a new file called fluent-bit.repo in /etc/yum.repos.d/ with the following content: Amazon Linux 2

[fluent-bit]
name = Fluent Bit
baseurl = https://packages.fluentbit.io/amazonlinux/2/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
enabled=1

Amazon Linux 2023

[fluent-bit]
name = Fluent Bit
baseurl = https://packages.fluentbit.io/amazonlinux/2023/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
enabled=1

Step3: Install Fluent-Bit

Once your repository is configured, run the following command to install it:

sudo yum install fluent-bit

Now the following step is to instruct systemd to enable the service:

sudo systemctl start fluent-bit

If you do a status check, you should see a similar output like this:

systemctl status fluent-bit

● fluent-bit.service - Fluent Bit
   Loaded: loaded (/usr/lib/systemd/system/fluent-bit.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-07-07 02:08:01 BST; 9s ago
 Main PID: 3820 (fluent-bit)
   CGroup: /system.slice/fluent-bit.service
           └─3820 /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf
...

The default configuration of fluent-bit is collecting metrics of CPU usage and sending the records to the standard output, you can see the outgoing data in your /var/log/messages file.

step3: Write sample fluent-bit config

fluent-bit dummy plugin

The dummy input plugin, generates dummy events. It is useful for testing, debugging, benchmarking and getting started with Fluent Bit.

Configuration Parameters for dummy

The plugin supports the following configuration parameters:

Key	Description
Dummy	Dummy JSON record. Default: {"message":"dummy"}
Start_time_sec	Dummy base timestamp in seconds. Default: 0
Start_time_nsec	Dummy base timestamp in nanoseconds. Default: 0
Rate	Rate at which messages are generated expressed in how many times per second. Default: 1
Samples	If set, the events number will be limited. e.g. If Samples=3, the plugin only generates three events and stops.
Copies	Number of messages to generate each time they are generated. Defaults to 1.

Getting Started with fluent-bit dummy message producer code

You can run the plugin from the command line or through the configuration file: Command Line

fluent-bit -i dummy -o stdout

​
[2017/07/06 21:55:29] [ info] [engine] started
[0] dummy.0: [1499345730.015265366, {"message"=>"dummy"}]
[1] dummy.0: [1499345731.002371371, {"message"=>"dummy"}]
[2] dummy.0: [1499345732.000267932, {"message"=>"dummy"}]
[3] dummy.0: [1499345733.000757746, {"message"=>"dummy"}]

check all fluent-bit process

ps -aux|grep fluent-bit

Killing fluent-bit process

sudo kill -9 <process-id>

Configuration File

In your main configuration file append the following Input & Output sections:

[INPUT]
    Name   dummy
    Tag    dummy.log

[OUTPUT]
    Name   stdout
    Match  *

#  Run script
/opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fb-dummy.conf

Generating random logs in JSON

[INPUT]
    Name   dummy
    Tag    dummy.log

[OUTPUT]
    Name   stdout
    Match  dummy.log
    Format json_lines

#  Run script
/opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fb-dummy.conf

Generating random 10 logs per second in JSON

We will use Rate configuration parameter to 10 which will create 10 logs per second. Rate at which messages are generated expressed in how many times per second. Default: 1

[INPUT]
    Name   dummy
    Tag    dummy.log
    Rate   10 

[OUTPUT]
    Name   stdout
    Match  dummy.log
    Format json_lines

#  Run script
/opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fb-dummy.conf

Generating custom message with dummy in fluentbit

[INPUT]
    Name   dummy
    dummy  {"message":"ping", "id":"app1"}
    Tag    dummy.log
    Rate   1
    Copies 2

[OUTPUT]
    Name   stdout
    Match  dummy.log
    Format json_lines

#  Run script
/opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fb-dummy.conf

Generate 10 message per second send it to Amazon OpenSearch Ingestion Service (OSI Pipeline)

[INPUT]
  Name   dummy
  dummy  {"message":"ping", "id":"app1"}
  Tag    dummy.log
  Rate   10

[OUTPUT]
  Name   stdout
  Match  dummy.log
  Format json_lines

[OUTPUT]
  Name http
  Match dummy.log
  Host ingestion-pipeline-l4ntkea5exszl2c7vrahbbrgwe.us-east-1.osis.amazonaws.com
  Port 443
  URI /log-pipeline/test_ingestion_path
  Format json
  aws_auth true
  aws_region us-east-1
  aws_service osis
  Log_Level trace
  tls On