# Simpl ELK
Git project dedicated for ELK on Simpl.
Requirements:
- service kube-state-metrics (it is not install by default) must be running on kubernetes cluster to fetch metric data about pod status
https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_state_and_event
## Administration Manual
<!-- TOC -->
* [Export dashboards to git repository](#export-dashboards-to-git-repository)
* [Import dashboards manually](#import-dashboards-manually)
* [FileBeat agent deployment and configuration](#filebeat-agent-deployment-and-configuration)
* [Description](#description)
* [Implementation](#implementation)
* [Integrating Filebeat as a Sidecar Pod for Cross Namespace Log Collection](#integrating-filebeat-as-a-sidecar-pod-for-cross-namespace-log-collection)
* [Infrastructure metrics](#infrastructure-metrics)
* [Usage of resources](#usage-of-resources)
* [Throughput](#throughput)
* [Response time and Availability](#response-time-and-availability)
* [Energy efficiency](#energy-efficiency)
* [Modify ILM policy](#modify-ilm-policy)
* [Apply changes on heartbeat-ilm](#apply-changes-on-heartbeat-ilm)
* [Apply changes on metricbeat-ilm](#apply-changes-on-metricbeat-ilm)
* [Apply changes on business-ilm and technical-ilm](#apply-changes-on-business-ilm-and-technical-ilm)
* [Performance parameters](#performance-parameters)
* [Logstash performance parameters](#logstash-performance-parameters)
* [Elasticsearch performance parameters](#elasticsearch-performance-parameters)
* [Kibana performance parameters](#kibana-performance-parameters)
* [Filebeat performance parameters](#filebeat-performance-parameters)
* [Monitoring API ](#monitoring-api)
* [Kibana](#kibana)
* [Elasticsearch](#elasticsearch)
<!-- TOC -->
### Export dashboards to git repository
Exporting dashboards:
- Login to Kibana
- Go to Stack Management → Saved Object
- Select Type: Dashboards
- Choose dashboards to export.
- Press "Export" to download ndjson file with dashboards to local PC.
- Rename downloaded file to dashboards.ndjson and upload to Git repository to path: eck-monitoring/charts/kibana/dashboards/
### Import dashboards manually
Loading dashboards:
- download [dashboards.ndjson](https://code.europa.eu/simpl/simpl-open/development/monitoring/eck-monitoring/-/tree/main/charts/kibana/dashboards?ref_type=heads) file from repo directory eck-monitoring/kibana/dashdoards/
- Choose file: dashboards.ndjson
- Click on download button:
- login to Kibana
- go to Stack Management -> Saved Object
- Choose Import:
- Set "Request action on conflict"
- Press "Import" and choose downloaded file.
- Press "Import"
- All dashboards should be imported.
Depend on changes fix conflicts.
### FileBeat agent deployment and configuration
#### Description:
Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that user specifies, collects log events, and forwards them either to Elasticsearch or Logstash for indexing. Filebeat consists of two main components: inputs and harvesters. These components work together to tail files and send event data to the output that you specify.
##### Harvester
A harvester is responsible for reading the content of a single file. The harvester reads each file, line by line, and sends the content to the output. One harvester is started for each file. The harvester is responsible for opening and closing the file, which means that the file descriptor remains open while the harvester is running. If a file is removed or renamed while it’s being harvested, Filebeat continues to read the file. This has the side effect that the space on your disk is reserved until the harvester closes. By default, Filebeat keeps the file open until close_inactive value is reached.
Closing a harvester has the following consequences:
- The file handler is closed, freeing up the underlying resources if the file was deleted while the harvester was still reading the file.
- The harvesting of the file will only be started again after scan_frequency has elapsed.
- If the file is moved or removed while the harvester is closed, harvesting of the file will not continue.
To control when a harvester is closed, use the close_* configuration options.
##### Input
An input is responsible for managing the harvesters and finding all sources to read from.
We use type log - the input finds all files on the drive that match the defined glob paths and starts a harvester for each file. Each input runs in its own Go routine.
#### Implementation
##### Overview
This configuration deploys Filebeat on a Kubernetes cluster as a DaemonSet to collect and ship logs to Elasticsearch and Logstash. The setup includes custom scripts for log generation, secure connections with certificates, and monitoring.
##### Components
Beat Configuration (filebeat.yaml):
- Kind: Beat
- Image: Uses Filebeat Docker image defined in values.yaml.
- Type: filebeat
- Elasticsearch Reference: Connects to the Elasticsearch instance named {{ .Release.Name }}-elasticsearch.
Pod Template:
Security Context: Runs as root user with file system group set to 1000.
Container Specs:
- Executes a custom script example.sh and starts Filebeat.
- Mounts various volumes for configuration, scripts, and certificates.
Environment Variables:
- Sets Elasticsearch and Logstash hosts, and monitoring credentials from secrets.
Volumes:
Config Volume: Loads filebeat.yml from a secret.
Example Script Volume: Loads example.sh from a ConfigMap.
Certificates Volumes: Loads necessary TLS certificates from secrets.
RBAC Configuration:
ServiceAccount: filebeat is created in the release namespace.
RoleBinding: Grants access to the Issuer for managing certificates.
Certificate Management:
Uses cert-manager to create TLS certificates for secure communication.
##### Supporting Resources
Secret (filebeat-config): Contains the Filebeat configuration file filebeat.yml encoded in Base64.
ConfigMap (filebeat-example-script): Provides example.sh for generating sample logs.
ServiceAccount: Ensures Filebeat has the necessary permissions to access Kubernetes resources.
RoleBinding: Links the service account with a role for accessing the Issuer.
##### values.yaml
Image Details: Defines the Filebeat image and tag.
Log Generation:
- totalMessages: Number of log messages to generate (can be infinite).
- messagesPerMinute: Rate of log generation.
TLS Configuration:
- Sets certificate duration and renewal parameters.
Filebeat Inputs and Outputs:
- Input: Configures Filebeat to read logs from example.log using multiline patterns.
- Output: Sends logs to Logstash with TLS enabled and configures Elasticsearch for monitoring.
##### Security
Uses secrets for sensitive data like TLS certificates and monitoring credentials.
Configures secure communication with Elasticsearch and Logstash using SSL/TLS.
#### Integrating Filebeat as a Sidecar Pod for Cross Namespace Log Collection
##### Overview
Integrating Filebeat as a sidecar container allows you to collect logs directly from applications running in different namespaces within your Kubernetes cluster. This setup ensures centralized log collection and monitoring, leveraging Filebeat's capabilities for log shipping to Elasticsearch or Logstash.
##### Deployment Steps
To integrate Filebeat as a sidecar pod for collecting logs from an application in a different namespace, follow these steps:
1. Modify Application Deployment YAML
Update your application's Deployment YAML (my-app.yaml) in the target namespace (app-namespace) to include Filebeat as a sidecar container.
```
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
namespace: app-namespace
spec:
replicas: 1
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-app
image: my-app-image:latest
volumeMounts:
- name: app-logs
mountPath: /var/log/myapp
- name: filebeat
image: {{ .Values.filebeat.image }}:{{ default .Values.elasticVersion .Values.filebeat.imageTag }}
command: ['sh', '-c', 'exec /usr/share/filebeat/filebeat -e -c /usr/share/filebeat/filebeat.yml']
volumeMounts:
- name: app-logs
mountPath: /var/log/myapp
- name: config
mountPath: /usr/share/filebeat/filebeat.yml
subPath: filebeat.yml
- name: filebeat-certs
mountPath: /usr/share/filebeat/certs
env:
- name: ELASTIC_ELASTICSEARCH_ES_HOSTS
value: 'https://elastic-elasticsearch-es-http.observability.svc:9200'
- name: LOGSTASH_HOSTS
value: 'logstash-{{ .Values.logstash.beats.pipelines_group_name }}-ls-api.observability.svc:5044'
- name: MONITORING_USER
valueFrom:
secretKeyRef:
name: user-monitoring-secret
key: username
- name: MONITORING_PASSWORD
valueFrom:
secretKeyRef:
name: user-monitoring-secret
key: password
volumes:
- name: app-logs
emptyDir: {}
- name: config
secret:
secretName: filebeat-config
- name: filebeat-certs
secret:
secretName: filebeat-certs-secret
```
Explanation:
Containers: Includes your application (my-app) container and Filebeat as a sidecar.
Volume Mounts: Ensures the application's log directory is mounted for Filebeat.
Environment Variables: Configures connections to Elasticsearch, Logstash, and monitoring credentials.
Volumes: Mounts Filebeat configuration (filebeat.yml) and TLS certificates from secrets.
2. Update Filebeat Configuration (Monitoring team task)
Ensure your Filebeat configuration (filebeat.yml) includes the necessary inputs and outputs for collecting and shipping logs from your application. Customize the configuration as per your logging requirements. To be aligned with monitoring team during Sprint 3.
3. Deploy and Verify
##### Security and Access Considerations
- RBAC Permissions: Ensure the ServiceAccount used by Filebeat (filebeat service account) has appropriate permissions (RoleBinding) to access resources in the app-namespace.
- Network Policies: Configure Kubernetes Network Policies to allow communication between namespaces if required.
- TLS Configuration: Secure communication with Elasticsearch or Logstash using TLS certificates stored in Kubernetes secrets.
### Infrastructure metrics
#### Usage of resources
Performance parameters like CPU, RAM, Disk will be collected by metricbeat.
All kubernetes components: nodes, pods will be monitor.
Elastic cluster will be monitor and stats can be available in Kibana "Stack Monitoring" section.
#### Throughput:
Throughput will be collected by metricbeat.
Following types of throughputs are available:
- network
- disk
Example graph for network throughput and other parameters:
Part of configuration:
```
modules:
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
process:
include_top_n:
by_cpu: 5
by_memory: 5
processes:
- .*
```
#### Response time and Availability
This parameter will be provided by heartbeat.
Example configuration:
```
heartbeat.monitors:
- type: tcp
schedule: '@every 5s'
hosts: ["elastic-elasticsearch-es-http.observability.svc:9200"]
- type: tcp
schedule: '@every 5s'
hosts: ["kibana.dev.simpl-europe.eu:443"]
- type: icmp
id: ping-myhost
name: My Host Ping
hosts: ["elastic-kibana-kb-http.observability.svc"]
schedule: '*/5 * * * * * *'
```
Available monitoring types:
- tcp
- icmp
- http
Response time parameter is available in monitor.duration.us field, example:
Availability parameter is in state.status field:
#### Energy efficiency
Energy Efficiency is calculated from field system.load.norm.5 field provided by metricbeat and maximum power used by CPU.
Pattern to calculate value:
'''
Ef = system_load * Pmax
'''
Where:
- Ef = Energy efficiency
- system_load = value from field system.load.norm.5
- Pmax = Maximum power used by CPU from vendor page
Parameter definition is "Load for the last 5 minutes divided by the number of cores."
Pmax has been assumed to 200 Watt.
### Modify ILM policy
Monitoring ELK stack are 5 ILM polices:
- business-ilm - responsible for index rotation with business ines logs
- technical-ilm - responsible for index rotation with technical logs
- metricbeat-ilm - responsible for index rotation with metrics collected from agents
- filebeat - responsible le for index rotation with ELK stack logs
- heartbeat-ilm - responsible for index rotation with services heartbeats
#### Apply changes on heartbeat-ilm:
Modify values/dev/observability/values.yml file and set new values:
```
heartbeat:
ilm:
hot:
max_age: 30d <- set max age for documents in hot phase
max_primary_shard_size: 50gb <- set max size for primary shard in hot phase
delete:
min_age: 365d <- set age after which index will be deleted
services:
heartbeat.monitors:
```
Restart heartbeat by command:
```kubectl rollout restart deployment heartbeat-beat-heartbeat```
#### Apply changes on metricbeat-ilm:
Modify values/dev/observability/values.yml file and set new values:
```
metricbeat:
ilm:
hot:
max_age: 30d <- set max age for documents in hot phase
max_primary_shard_size: 50gb <- set max size for primary shard in hot phase
delete:
min_age: 365d <- set age after which index will be deleted
resources:
requests:
memory: 500Mi
limits:
```
Restart metricbeat by command:
```kubectl rollout restart daemonset metricbeat-beat-metricbeat```
#### Apply changes on filebeat:
- Login to Kibana and go to: Stack Management -> Index Lifecycle Policies.
- Click on filebeat policy
- Modify "Hot phase" in advanced setting when disable "Use recommended defaults" and/or modify Delete phase if needed.
- Press "Save policy".
#### Apply changes on business-ilm and technical-ilm:
Modify values/dev/observability/values.yml file and set new values:
```
logstash:
ilm:
business:
hot:
max_age: 30d <- set max age for business documents in hot phase
max_primary_shard_size: 1gb <- set max size for business primary shard in hot phase
delete:
min_age: 30d <- set age after which business index will be deleted
technical:
hot:
max_age: 30d <- set max age for technical documents in hot phase
max_primary_shard_size: 1gb <- set max size for technical primary shard in hot phase
delete:
min_age: 30d <- set age after which technical index will be deleted
count_beats: 1
count_syslog: 0
```
Restart logstash statefulsets by command:
``` kubectl rollout restart sts logstash-beats-ls```
### Performance parameters
#### Logstash performance parameters
In file values/dev/observability/values.yml are following logstash performance parameters:
- logstash.env.ls_java_opts: "-Xms3g -Xmx3g"
Set heap memory for logstash process inside container.
Logstash statefulsets restart is required.
- logstash.resources.requests.memory: 4Gi
logstash.resources.limits.memory: 4Gi
Set memory allocation (request/limit) for logstash pod.
- logstash.resources.requests.cpu: 300m
logstash.resources.limits.cpu: 300m
Set CPU allocation (request/limit) for logstash pod.
- pipelines_yml_config.pipeline.workers: 1
Set number of workers for logstash pipeline.
Logstash statefulsets restart is required.
- pipelines_yml_config.pipeline.pipeline.batch.size: 125
Set batch_size for logstash pipeline.
Logstash statefulsets restart is required.
#### Elasticsearch performance parameters
- elasticsearch.diskSpace: 60Gi
Set disk size to store indices in elasticsearch pods
- elasticsearch.count: 3
Number of elasticsearch pods in stack
- elasticsearch.resources.requests.memory: 4Gi
elasticsearch.resources.limits.memory: 4Gi
Set memory allocation (request/limit) for elasticsearch pod.
- elasticsearch.resources.requests.cpu: 300m
elasticsearch.resources.limits.cpu: 300m
Set CPU allocation (request/limit) for elasticsearch pod.
#### Kibana performance parameters
- kibana.resources.requests.memory: 1Gi
kibana.resources.limits.memory: 1Gi
Set memory allocation (request/limit) for kibana pod.
- kibana.resources.requests.cpu: 300m
kibana.resources.limits.cpu: 300m
Set CPU allocation (request/limit) for kibana pod.
- kibana.count: 1
Number of kibana pods in stack
#### Filebeat performance parameters
- filebeat4agents.resources.requests.memory: 1Gi
filebeat4agents.resources.limits.memory: 1Gi
Set memory allocation (request/limit) for filebeat pod.
- filebeat4agents.resources.requests.cpu: 100m
filebeat4agents.resources.limits.cpu: 100m
Set CPU allocation (request/limit) for filebeat pod.
### Monitoring API
#### Kibana
API for Kibana is available at page: "https://kibana.<namespace>.<env>.simpl-europe.eu"
Example for stack observability at dev cluster: https://kibana.observability.dev.simpl-europe.eu/
Link to vendor manual page with API:
https://www.elastic.co/docs/api/doc/kibana/v8
Example API call to get all data views from Kibana:
```
https://kibana.observability.dev.simpl-europe.eu/api/data_views
```
Request:
```
$ curl -k -s -u elastic -X GET "https://kibana.observability.dev.simpl-europe.eu/api/data_views?v"
```
Response:
```
{"data_view":
[
{"id":"metricbeat-*",
"namespaces":["default"],
...
]
}
```
#### Elasticsearch
API for Elasticsearch is available at page: "https://elasticsearch.<namespace>.<env>.simpl-europe.eu"
Example for stack observability at dev cluster: https://elasticsearch.observability.dev.simpl-europe.eu/
Link to vendor manual page with API:
https://www.elastic.co/guide/en/elasticsearch/reference/current/rest-apis.html
Example API call to get cluster state:
```
https://elasticsearch.observability.dev.simpl-europe.eu/_cluster/health
```
Request:
```
$ curl -k -Ss -u elastic -X GET "https://elasticsearch.observability.dev.simpl-europe.eu/_cluster/health"
```
Response:
```
{
"cluster_name":"elastic-elasticsearch",
"status":"green",
"timed_out":false,
"number_of_nodes":3,
"number_of_data_nodes":3,
"active_primary_shards":40,
"active_shards":80,
"relocating_shards":0,
"initializing_shards":0,
"unassigned_shards":0,
"delayed_unassigned_shards":0,
"number_of_pending_tasks":0,
"number_of_in_flight_fetch":0,
"task_max_waiting_in_queue_millis":0,
"active_shards_percent_as_number":100.0
}
```