Mastering Fluent Bit: Developer Guide to Service Section Configuration

This series is a general purpose getting started guide for those of us wanting to learn about the Cloud Native Computing Foundation (CNCF) project Fluent Bit. 

Each article in this series addresses a single topic by providing insights into what the topic is, why we are interested in exploring that topic, where to get started with the topic, and how to get hands-on with learning about the topic as it relates to the Fluent Bit project.

The idea is that each article can stand on its own, but that they also lead down a path that slowly increases our abilities to implement solutions with Fluent Bit telemetry pipelines.

Let's take a look at the topic of this article, using Fluent Bit tips and tricks for developers. In case you missed the previous article, check out using a Fluent Bit pipeline on a Kubernetes cluster to take control of all the logs being generated.

This article will be a hands-on tour of the things that help you as a developer testing out your Fluent Bit pipelines. We'll take a look at the services section of a developers telemetry pipeline configuration.

All examples in this article have been done on OSX and are assuming the reader is able to convert the actions shown here to their own local machines.

Where to get started

You should have explored the previous articles in this series to install and get started with Fluent Bit on your developer local machine. This can be with containers and it can be from source, either will work. Links at the end of this article will point you to a free hands-on workshop that lets you explore more of Fluent Bit in detail.

You can verify that you have a functioning installation by testing one of the two methods for using Fluent Bit, first the source installation, the second using a container image (with Podman in the following examples):

# For source installation.

$ fluent-bit -i dummy -o stdout

# For container installation.

$ podman run -ti ghcr.io/fluent/fluent-bit:4.0.4 -i dummy -o stdout

...

[0] dummy.0: [[1753105021.031338000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105022.033205000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105023.032600000, {}], {"message"=>"dummy"}]
[0] dummy.0: [[1753105024.033517000, {}], {"message"=>"dummy"}]

...

Let's look at a few tips and tricks to help you with your  local development testing.

Service section: hot reload

When first getting started with Fluent Bit the service section of the configuration does not seem to be that important. There is one thing that is really useful that is often missed by developers, and that's the ability to restart a source installation of Fluent Bit during inner development loop usage to quickly test configuration changes by configuring hot reloading. 

A simple example configuration file named fluent-bit.yaml can be found below:

service:

  flush: 1
  log_level: info
  hot_reload: on

pipeline:
  inputs:
    # This entry generates a test INFO log level message.
    - name:  dummy
      tag:   msg.info
      dummy: '{"message":"This is INFO message", "level":"INFO", "color": "yellow"}'

    # This entry generates a test ERROR log level message.
    - name:  dummy
      tag:   msg.error
      dummy: '{"message":"This is ERROR message", "level":"ERROR", "color": "red"}'

  outputs:
    - name: stdout
      match: '*'
      format: json_lines

Now let's run this as follows:

$ fluent-bit --config fluent-bit.yaml

...

{"date":1753107336.593176,"message":"This is INFO message","level":"INFO","color":"yellow"}
{"date":1753107336.593878,"message":"This is ERROR message","level":"ERROR","color":"red"}
{"date":1753107337.591496,"message":"This is INFO message","level":"INFO","color":"yellow"}
{"date":1753107337.591669,"message":"This is ERROR message","level":"ERROR","color":"red"}
{"date":1753107338.586933,"message":"This is INFO message","level":"INFO","color":"yellow"}
{"date":1753107338.587137,"message":"This is ERROR message","level":"ERROR","color":"red"}
...

This will alternate between the two messages and go on forever. In the process of testing and development, we decide to modify the configuration to filter messages with info status by removing the color key. This change means we need to restart the running Fluent Bit instance using the hot reload functionality.

Below the new configuration changes:

service:

  flush: 1
  log_level: info
  hot_reload: on

pipeline:
  inputs:
    # This entry generates a test INFO log level message.
    - name:  dummy
      tag:   msg.info
      dummy: '{"message":"This is INFO message", "level":"INFO", "color": "yellow"}'

    # This entry generates a test ERROR log level message.
    - name:  dummy
      tag:   msg.error
      dummy: '{"message":"This is ERROR message", "level":"ERROR", "color": "red"}'
  

  filters:

    - name: modify

      match: msg.info

      remove: color

  outputs:
    - name: stdout
      match: '*'
      format: json_lines

The first method to trigger the hot reloading without stopping our instance is done on Linux systems using SIGHUP as follows (note we are using PIDOF helper tool to identify our running Fluent Bit process id, but you can find this using PS or your favorite tool):

$ kill -s SIGHUP $(pidof fluent)

...

{"date":1753107336.593176,"message":"This is INFO message","level":"INFO","color":"yellow"}
{"date":1753107336.593878,"message":"This is ERROR message","level":"ERROR","color":"red"}
{"date":1753107337.591496,"message":"This is INFO message","level":"INFO","color":"yellow"}
{"date":1753107337.591669,"message":"This is ERROR message","level":"ERROR","color":"red"}
{"date":1753107338.586933,"message":"This is INFO message","level":"INFO","color":"yellow"}
{"date":1753107338.587137,"message":"This is ERROR message","level":"ERROR","color":"red"}
[2025/07/21 16:20:34] [engine] caught signal (SIGHUP)

... <<<< RESTARTING_FLUENT_BIT_LOG_MESSAGES 

{"date":1753107636.966377,"message":"An INFO message","level":"INFO"}
{"date":1753107636.966658,"message":"An ERROR message","level":"ERROR","color":"red"}
{"date":1753107637.96652,"message":"An INFO message","level":"INFO"}
{"date":1753107637.966711,"message":"An ERROR message","level":"ERROR","color":"red"}
{"date":1753107638.966835,"message":"An INFO message","level":"INFO"}
{"date":1753107638.966975,"message":"An ERROR message","level":"ERROR","color":"red"}
...

The log output catches the SIGHUP, restarts our Fluent Bit instance in place and applies the new configuration changes, resulting in the changed output as shown with color removed from all INFO level events.

Another way to access this hot reloading of your instance is to use the configurable HTTP server and leverage the provided endpoints:

• PUT /api/v2/reload
• POST /api/v2/reload

Add the following to the service section in the configuration file:

service:

  flush: 1
  log_level: info
  http_server: on
  http_listen: 0.0.0.0
  http_port: 2020
  hot_reload: on

Now start your instance (still using the same configuration as before, just with the new HTTP server added) and try the commands below to show hot reloading:

$ fluent-bit --config fluent-bit.yaml

...

{"date":1753344624.827537,"message":"An INFO message","level":"INFO"}
{"date":1753344624.827891,"message":"An ERROR message","level":"ERROR","color":"red"}
{"date":1753344625.829556,"message":"An INFO message","level":"INFO"}
{"date":1753344625.829864,"message":"An ERROR message","level":"ERROR","color":"red"}
{"date":1753344626.829288,"message":"An INFO message","level":"INFO"}
{"date":1753344626.829416,"message":"An ERROR message","level":"ERROR","color":"red"}
...

# In a different terminal window.
$ curl -X POST -d '{}' localhost:2020/api/v2/reload

{"reload":"done","status":0}

# Or in a different terminal window.
$ curl -X PUT -d '{}' localhost:2020/api/v2/reload

{"reload":"done","status":0}


# Meanwhile, back in the Fluent Bit terminal window...
#
...
{"date":1753344797.824589,"message":"An INFO message","level":"INFO"}
{"date":1753344797.824942,"message":"An ERROR message","level":"ERROR","color":"red"}
[2025/07/24 10:13:19] [engine] caught signal (SIGHUP)

... <<<< RESTARTING_FLUENT_BIT_LOG_MESSAGES 

{"date":1753344801.312548,"message":"An INFO message","level":"INFO"}
{"date":1753344801.31283,"message":"An ERROR message","level":"ERROR","color":"red"}
{"date":1753344802.309874,"message":"An INFO message","level":"INFO"}
{"date":1753344802.310077,"message":"An ERROR message","level":"ERROR","color":"red"}
...

If you are using a REST client, it might look something like this:

Note we can monitor the number of hot reloads that an instance has undertaken by using the following API call:

# The following API call returns the count of reloads done 

# on this instance, in the example below its been reloaded twice.

#

$ curl -X GET -d '{}' localhost:2020/api/v2/reload

{"hot_reload_count":2}

This wraps up a few handy tips and tricks for developers getting started with Fluent Bit in their solutions. The ability to set up and leverage hot reloading is a big help in speeding up your inner development loop experience.

More in the series

In this article you learned a few handy tricks for using Fluent Bit service section in the configuration to improve the inner developer loop experience. This article is based on this online free workshop.

There will be more in this series as you continue to learn how to configure, run, manage, and master the use of Fluent Bit in the wild. Next up, exploring some of the more interesting Fluent Bit input plugins for developers.