Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that user specifies, collects log events, and forwards them either to Elasticsearch or Logstash for indexing. Filebeat consists of two main components: inputs and harvesters. These components work together to tail files and send event data to the output that you specify.
Harvester
A harvester is responsible for reading the content of a single file. The harvester reads each file, line by line, and sends the content to the output. One harvester is started for each file. The harvester is responsible for opening and closing the file, which means that the file descriptor remains open while the harvester is running. If a file is removed or renamed while it’s being harvested, Filebeat continues to read the file. This has the side effect that the space on your disk is reserved until the harvester closes. By default, Filebeat keeps the file open until close_inactive value is reached.
==============================
Performance parameters
==============================
Closing a harvester has the following consequences:
- The file handler is closed, freeing up the underlying resources if the file was deleted while the harvester was still reading the file.
- The harvesting of the file will only be started again after scan_frequency has elapsed.
- If the file is moved or removed while the harvester is closed, harvesting of the file will not continue.
Logstash performance parameters
To control when a harvester is closed, use the close_* configuration options.
In file values/dev/observability/values.yml are following logstash performance parameters:
Input
An input is responsible for managing the harvesters and finding all sources to read from.
- logstash.env.ls_java_opts: "-Xms3g -Xmx3g"
Set heap memory for logstash process inside container.
Logstash statefulsets restart is required.
We use type log - the input finds all files on the drive that match the defined glob paths and starts a harvester for each file. Each input runs in its own Go routine.
- logstash.resources.requests.memory: 4Gi
logstash.resources.limits.memory: 4Gi
Set memory allocation (request/limit) for logstash pod.
Implementation
Overview
- logstash.resources.requests.cpu: 300m
logstash.resources.limits.cpu: 300m
Set CPU allocation (request/limit) for logstash pod.
This configuration deploys Filebeat on a Kubernetes cluster as a DaemonSet to collect and ship logs to Elasticsearch and Logstash. The setup includes custom scripts for log generation, secure connections with certificates, and monitoring.
- pipelines_yml_config.pipeline.workers: 1
Set number of workers for logstash pipeline.
Logstash statefulsets restart is required.
Components
Beat Configuration (filebeat.yaml):
- Kind: Beat
- Image: Uses Filebeat Docker image defined in values.yaml.
- Type: filebeat
- Elasticsearch Reference: Connects to the Elasticsearch instance named {{ .Release.Name }}-elasticsearch.
