Are you ready for getting started with cloud native observability with telemetry pipelines?
This article is part of a series exploring a workshop guiding you through the open source project Fluent Bit, what it is, a basic installation, and setting up a first telemetry pipeline project. Learn how to manage your cloud native data from source to destination using the telemetry pipeline phases covering collection, aggregation, transformation, and forwarding from any source to any destination.
The previous article in this series helped with an installation of Fluent Bit on our local machine using the project source code. This time around we'll learn how to use Fluent Bit in a container on our local machine, including how to run the container while using local configuration files. You can find more details in the accompanying workshop lab.
Let's get started with Fluent Bit in a container.
Installing Fluent Bit using a container image is going to be demonstrated here using the open source project Podman. It's assumed you have already installed the Podman command line tooling previously. It should also be noted that the following code and command line examples are all based on using an OSX machine.
If you want to use other container tooling, such as Docker, most of the commands are the same with just a substitution of the tooling name (docker instead of podman).
Containerized Fluent Bit
While it's not that difficult to run Fluent Bit in a container, we'll also show you how to do it using local configuration files so that you can actually use it to build your first telemetry pipelines.
It's pretty straight forward to running Fluent Bit in a container, just start the container image as follows:
$ podman run --name fb -ti cr.fluentbit.io/fluent/fluent-bit:2.2.2
Let's take a look at what this command is actually doing. First you see a flag for giving the container a name we can reference (--name fb). Another is assigning the container a console for output and stay interactive (-ti) and finally, it's using the image version supported in this workshop (cr.fluentbit.io/fluent/fluent-bit:2.2.2).
You'll notice the container starts and takes over the console for its output, where Fluent Bit is measuring CPU usage and dumping it to the console (CTRL-C will stop the container):
Fluent Bit v2.2.2
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
____________________
< Fluent Bit v2.2.2 >
-------------------
\
\
\ __---__
_- /--______
__--( / \ )XXXXXXXXXXX\v.
.-XXX( O O )XXXXXXXXXXXXXXX-
/XXX( U ) XXXXXXX\
/XXXXX( )--_ XXXXXXXXXXX\
/XXXXX/ ( O ) XXXXXX \XXXXX\
XXXXX/ / XXXXXX \__ \XXXXX
XXXXXX__/ XXXXXX \__---->
---___ XXX__/ XXXXXX \__ /
\- --__/ ___/\ XXXXXX / ___--/=
\-\ ___/ XXXXXX '--- XXXXXX
\-\/XXX\ XXXXXX /XXXXX
\XXXXXXXXX \ /XXXXX/
\XXXXXX > _/XXXXX/
\XXXXX--__/ __-- XXXX/
-XXXXXXXX--------------- XXXXXX-
\XXXXXXXXXXXXXXXXXXXXXXXXXX/
""VXXXXXXXXXXXXXXXXXXV""
[2024/03/01 10:38:52] [ info] [fluent bit] version=2.2.2, commit=eeea396e88, pid=1
[2024/03/01 10:38:52] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/03/01 10:38:52] [ info] [cmetrics] version=0.6.6
[2024/03/01 10:38:52] [ info] [ctraces ] version=0.4.0
[2024/03/01 10:38:52] [ info] [input:cpu:cpu.0] initializing
[2024/03/01 10:38:52] [ info] [input:cpu:cpu.0] storage_strategy='memory' (memory only)
[2024/03/01 10:38:52] [ info] [sp] stream processor started
[2024/03/01 10:38:52] [ info] [output:stdout:stdout.0] worker #0 started
[0] cpu.local: [[1709289532.997338599, {}], {"cpu_p"=>0.250000, "user_p"=>0.000000, "system_p"=>0.250000, "cpu0.p_cpu"=>0.000000
[0] cpu.local: [[1709289533.996516160, {}], {"cpu_p"=>0.000000, "user_p"=>0.000000, "system_p"=>0.000000, "cpu0.p_cpu"=>0.000000Be sure to scroll to the right in the above window to see the full console output.
At any time should we encounter failures, during installation, testing, data population, or build results, don't worry! This can be rerun any time after you fix any problems reported. We might have to remove the Fluent Bit container depending on how far you get before something goes wrong. Just stop, remove, and restart it as follows:
$ podman container stop fb $ podman container rm fb $ podman run --name fb -ti cr.fluentbit.io/fluent/fluent-bit:2.2.2
There might come a moment where we are going to want to stop working with Fluent Bit and pause until a later time. To do this we can shutdown our container environment by stopping the running Fluent Bit container and then stopping the Podman virtual machine as follows:
$ podman container stop fb $ podman machine stop
Let's take a look at building our own container images with our specific telemetry pipeline configurations.
Building container images
Often you want to setup your own specific configuration and add that to the container image. This means you build your own container image and then run that with custom configurations copied into the container image.
For example, let's assume we have the following files in our current directory, all parts of a Fluent Bit telemetry pipeline configuration:
- workshop-fb.conf - the main configuration file importing all other split out configuration files.
- inputs.conf - file containing all input plugin configurations.
- outputs.conf - file container all output configurations.
To build a container image we need to provide a Buildfile, which defines what base container image to use and list where to copy our above configuration files into that base image as shown here:
FROM cr.fluentbit.io/fluent/fluent-bit:2.2.2
COPY ./workshop-fb.conf /fluent-bit/etc/fluent-bit.conf
COPY ./inputs.conf /fluent-bit/etc/inputs.conf
COPY ./outputs.conf /fluent-bit/etc/outputs.confOnce we have this file, we can now build our container image as follows:
$ podman build -t workshop-fb:v1 -f Buildfile
STEP 1/4: FROM cr.fluentbit.io/fluent/fluent-bit:2.2.2
STEP 2/4: COPY ./workshop-fb.conf /fluent-bit/etc/fluent-bit.conf
--> a379e7611210
STEP 3/4: COPY ./inputs.conf /fluent-bit/etc/inputs.conf
--> f39b10d3d6d0
STEP 4/4: COPY ./outputs.conf /fluent-bit/etc/outputs.conf
COMMIT workshop-fb:v1
--> b06df84452b6
Successfully tagged localhost/workshop-fb:v1
b06df84452b6eb7a040b75a1cc4088c0739a6a4e2a8bbc2007608529576ebebaThe build command uses the flag (-t workshop-fb:v1), which tags the container image with a name and version number. This helps us to run the image by name later in the next workshop lab. Furthermore it makes use of the (-f Buildfile) we just created to copy in our custom configuration files as shown in the build output.
This process now has us ready to start building our first pipelines, but what if you want to use other versions of Fluent Bit container images?
Other versions
You might be wondering how to run other versions of Fluent Bit in a container. For example, Fluent Bit v3.0.0 was just released, so this workshop will be updating to follow the releases. Feel free to give this a try by starting the container image as follows:
$ podman run --name fb -ti cr.fluentbit.io/fluent/fluent-bit:3.0.1This starts the container and takes over the console for its output, where Fluent Bit is measuring CPU usage and dumping it to the console (CTRL-C will stop the container):
Fluent Bit v3.0.1
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
___________.__ __ __________.__ __ ________
\_ _____/| | __ __ ____ _____/ |_ \______ \__|/ |_ ___ _\_____ \
| __) | | | | \_/ __ \ / \ __\ | | _/ \ __\ \ \/ / _(__ <
| \ | |_| | /\ ___/| | \ | | | \ || | \ / / \
\___ / |____/____/ \___ >___| /__| |______ /__||__| \_/ /______ /
\/ \/ \/ \/ \/
[2024/04/11 14:39:46] [ info] [fluent bit] version=3.0.1, commit=017b110ecf, pid=31593
[2024/04/11 14:39:46] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/04/11 14:39:46] [ info] [cmetrics] version=0.7.1
[2024/04/11 14:39:46] [ info] [ctraces ] version=0.4.0
[2024/03/29 15:29:26] [ info] [input:cpu:cpu.0] initializing
[2024/03/29 15:29:26] [ info] [input:cpu:cpu.0] storage_strategy='memory' (memory only)
[2024/03/29 15:29:26] [ info] [output:stdout:stdout.0] worker #0 started
[2024/03/29 15:29:26] [ info] [sp] stream processor started
[0] cpu.local: [[1711726167.221360412, {}], {"cpu_p"=>0.000000, "user_p"=>0.000000, "system_p"=>0.000000, "cpu0.p_cpu"=>1.000000, "cpu0.p_user"=>0.000000, "cpu0.p_system"=>1.000000, "cpu1.p_cpu"=>0.000000, "cpu1.p_user"=>0.000000, "cpu1.p_system"=>0.000000}]
[0] cpu.local: [[1711726168.216299447, {}], {"cpu_p"=>1.000000, "user_p"=>0.500000, "system_p"=>0.500000, "cpu0.p_cpu"=>0.000000, "cpu0.p_user"=>0.000000, "cpu0.p_system"=>0.000000, "cpu1.p_cpu"=>0.000000, "cpu1.p_user"=>0.000000, "cpu1.p_system"=>0.000000}]Be sure to scroll to the right in the above window to see the full console output. As you can see, this puts all available versions of Fluent Bit container images at your fingertips.
What's next?
This article helped us get Fluent Bit installed on our local machine using the available container builds. The series continues with the next step in this workshop, creating our first telemetry pipelines using either the source install or container images.
Stay tuned for more hands on material to help you with your cloud native observability journey.