FluentBit: Local Setup For UDP Input Testing

FluentBit can be tricky to debug in remote environment. Anyway, it's always useful to have a local working setup to test the moving parts in isolation.

Below is a minimum ????????????-??????????????.?????? file that you can use for running the FluentBit container on a local machine:

services:
    fluent-bit:
        image: fluent/fluent-bit:2.2.2
        ports:
            - '5170:5170/udp'
        volumes:
            - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf        

Where:

• ?????????? element uses one of the available docker images from the Docker Hub server (https://hub.docker.com/r/fluent/fluent-bit/tags)
• ?????????? exposes a UDP port on the container
• ?????????????? mounts FluentBit configuration file ????????????-??????.???????? that is expected to exist in the same folder as the docker compose file

A configuration file, ????????????-??????.???????? , has configuration for the FluentBit service itself, plus two more sections, a UDP input plugin, and stdout output plugin. The stdout plugin is there for simplicity to monitor what is getting received by the UDP plugin. The content of ????????????-??????.???????? file:

[SERVICE]
    Flush                5
    Daemon           off
    Log_Level        debug

[INPUT]
    Name                 udp
    Tag                     test-udp-data
    Listen                 0.0.0.0
    Port                    5170
    Chunk_Size       32
    Buffer_Size        64
    Format                json

[OUTPUT]
    Name                stdout
    Match               test-udp-data        

Note, that the same port as specified in the UDP input plugin is exposed in a docker compose file.

Now you can run the docker compose:

docker compose up -d        

To create some UDP traffic, you can execute Netcat CLI command in a command shell:

nc -u 127.0.0.1 5170 
{"key1": 123, "key2": "val2"}
{"key3": 456, "key4": "val4"}
.....
CTRL + c        

To monitor whether FluentBit receives data and sends them to its console, check the logs in a docker:

docker compose logs -f        

In the output you should see the JSON objects that you sent from the console:

fluent-bit-1  | [0] udp.0: [[1731966021.915803791, {}], {"key1"=>123, "key2"=>"val2"}]
fluent-bit-1  | [2024/11/18 21:41:18] [debug] [input chunk] update output instances with new chunk size diff=51, records=1, input=udp.0
fluent-bit-1  | [0] udp.0: [[1731966078.540348554, {}], {"key3"=>456, "key4"=>"val4"}]
fluent-bit-1  | [2024/11/18 21:41:20] [debug] [task] created task=0xffffb8039640 id=0 OK
fluent-bit-1  | [2024/11/18 21:41:20] [debug] [out flush] cb_destroy coro_id=1
fluent-bit-1  | [2024/11/18 21:41:20] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
fluent-bit-1  | [2024/11/18 21:41:20] [debug] [task] destroy task=0xffffb8039640 (task_id=0)
fluent-bit-1  | [2024/11/18 21:42:25] [debug] [input chunk] update output instances with new chunk size diff=36, records=1, input=udp.0