What Is Fluent Bit?
| Editor’s note: This article is an excerpt from the Manning book, “Fluent Bit with Kubernetes:” Chapter 1, An Introduction to Fluent Bit. |
The first article in this series — “Fluent Bit, a Specialized Event Capture and Distribution Tool” — looked at why Fluent Bit is important. Now, let’s address some core concepts that influence almost every aspect of Fluent Bit. The most critical thing that we’ve encountered is the event. We should also consider what Fluent Bit does and doesn’t do to make events useful.
The other key concept in Fluent Bit is plugins. As we progress through the book, we’ll dig deeper into plugins, but at this stage, I’ll describe them as the building blocks of Fluent Bit’s functional capabilities.
Fluent Bit’s Core Elements
To interact with Fluent Bit’s events, whether they represent log events, traces, or metrics, we need to understand how each event is represented within Fluent Bit, which is the same way Fluentd does, with three mandatory elements.
As the figure below illustrates, Fluent Bit has three core elements with some additional elements that are opaque to us right now:
1. Metadata — Metadata is a list of key-value pairs with a mandatory key called Tag and related value. The Tag is a logical name associated with the events. We use the Tag to route events to the correct operation(s). (In the book, “Fluent Bit with Kubernetes,” we introduce strategies that allow us to manipulate a Tag and use intelligent naming conventions to help us.)
In Fluent Bit v1 and Fluentd, the metadata was only the Tag. To increase flexibility and allow Fluent Bit to carry other types of events (and traces), Fluent Bit v1.9 changed the metadata to hold additional key-value pairs about the nature of the record content, such as the type of event. (As you’ll see in the book, we can access the Tag value without referring to the fact that it’s part of the metadata, as Fluent Bit v1 and Fluentd have.)
In Fluent Bit v1 and Fluentd, the metadata was only the Tag. To increase flexibility and allow Fluent Bit to carry other types of events (and traces), Fluent Bit v1.9 changed the metadata to hold additional key-value pairs about the nature of the record content, such as the type of event. As we’ll see later in the book, we can access the Tag value without referring
2. Timestamp — Events without a timestamp are of limited value. Without the timestamp, we can’t determine whether an issue is current or new because we have no sense of when the event occurred. We can’t determine whether the event is a cause or an effect because we don’t know the order in which things occurred. –
3. Record — A record contains the event data (log, metric, or trace). The ability to access and manipulate the record within various plugins depends on the plugin type and the metadata describing the record. When the record contains a log, Fluent Bit, depending on the input and parsing, treats the record’s value as a list of key-value pairs or a single block of text. We can extract content and convert the payload to JSON, among other things. When we’re not processing an event, the record is held efficiently by serializing it using the MessagePack library.
The metadata can also denote that the record represents metrics or traces. In this case, the record takes on the following characteristics:
- Metrics — When we send and receive metrics, the data is in line with the Prometheus format (a non-JSON structure). But Fluent Bit gives us the means to retrieve and manipulate metrics data. Internally, metrics are handled by a library called CMetric, which other projects are starting to use.
- Traces — Traces are also handled as a special record payload and can be made into a record in which you can interact.
We’ll explore these aspects in greater detail as we explore these data sources. Although the movement of the content between the visible record and the opaque structure is not completely free today, handling this movement will become easier over time.
Log Event Structure
The figure below shows the data structure of Fluent Bit v1.9 and later, alongside the equivalent Fluent Bit v1 and Fluentd structure. Although the difference is subtle, it is noticeable when handling non-log events. It is worth noting that in the exceptional situation of caching log events in a file with a pre-1.9 version of Fluent Bit, trying to get a post-1.9 version of Fluent Bit to read those cached files will result in errors.
Log event structure for Fluent Bit v1.x and Fluentd (left) and Fluent Bit from v1.9 (right)
Logical Architecture
The figure below shows the Logical Fluent Bit architecture. This diagram helps orient you around the capabilities we’re exploring within Fluent Bit. The figure shows these logical components:
1. Input plugin (listener), input plugin (pulled) — Many representations of Fluent Bit don’t differentiate the types of input plugins. Although the contract between the plugin and the core of Fluent Bit (the pipeline processing) is unaltered, there are differences in the way the plugin is implemented that affect configuration and tuning considerations.
Network-centric inputs can be described as listeners; we connect to the network, and when data is received, we must process it. Large, sudden spikes here can cause backpressure; the source system invoking Fluent Bit can’t continue until we consume the event.
The pulled events, such as those that capture log events from a file as they’re written, require us to poll the file periodically to determine whether any new content has been added. The implementation of the input plugin can dictate the system’s throughput.
2. Custom input plugin — This capability can be characterized as a pulled or listener plugin. As we have support for network sources with HTTP — unless we have specialist encoding that is best handled by an input plugin rather than a decoder (a specialist feature available to parsers) — this feature is likely to adopt a pulled model. A custom plugin differentiates itself from other plugins because it is not part of the standard Fluent Bit release—any plugin built directly into the binary by a third party or through the extension options.
Parser — This provides the means to transform the received content into meaningful data, such as extracting the important values from the record or transforming it to JSON. A range of prebuilt parsers is available; many of these parsers are specializations of regular expressions. Parsers are typically used in conjunction with filters, but some input plugins can also use them.
Buffer — Depending on the plugin, buffering can be used in several places. Logically, it fits well here, as the primary objective of buffering is to allow us to flex to input and output performance differences that might occur, such as spikes in outputs from our sources or a slowdown in the consumption of our outputs. The buffer, therefore, prevents Fluent Bit from being a potential throughput constraint or point of data loss. If you’re sensitive about the risk of data loss, you can switch the buffer to use file storage, which can be read when the Fluent Bit process restarts. This approach does have a performance cost. The buffer has a storage interface layer that manages the data going into and out of the buffer and its physical implementation (file or memory); it also interacts with any relevant stream processors.
Filter, custom filter — Filters are the pipeline’s heavy lifters, providing the means to interact and manipulate events that have been received. Filters fetch and return the events that they process to the buffer. Normal filters are completely configuration-driven, but custom filters can be implemented in two ways:
- The typical approach is to invoke Lua scripts.
- We can implement more demanding or complex filters with C, Go and WebAssembly, following the approach used by custom input and output plugins.
Stream processing — Stream processing represents how we configure the new, advanced analytics capabilities. We can loop data from this analytical process back as an input so we can use the analytical values to enrich processing, such as creating time series data based on received events.
Output plugin, custom output plugin — As with the inputs, we’ve separated these types of plugins to draw attention to extensibility. The output plugin’s role is to retrieve events from a buffer and then store them or pass them to a third-party solution for onward processing (this may be data storage, but we may output to other Fluentd or Fluent Bit instances to delegate or aggregate work), depending on the plugin’s implementation.
We have defined the logical components more granularly than the official documentation does to help you understand their behavioral characteristics.
The official documentation focuses principally on input, filter and output, three of the four horizontal groups in figure below.
Logical Fluent Bit architecture, with the blocks representing logical features and the lines representing the possible flow of events. The standard Fluent Bit groupings are overlaid, but I’ve separated and illustrated the buffers slightly differently, as their positions are more logical than how they fit into the code base.
This article is part of a series. Read also:
- Fluent Bit, a Specialized Event Capture and Distribution Tool
- What’s Driving Fluent Bit Adoption?
- A Guide To Migrating From Fluentd To Fluent Bit
- What Are the Differences Between OTel, Fluent Bit and Fluentd?
