Simpl-Open Monitoring - ELK Administrators.md

# Simpl ELK
Git project dedicated for ELK on Simpl.

Requirements:
    -   service kube-state-metrics (it is not install by default) must be running on kubernetes cluster to fetch metric data about pod status
        https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_state_and_event


## Administration Manual

<!-- TOC -->

* [Export dashboards to git repository](#export-dashboards-to-git-repository)
* [Import dashboards manually](#import-dashboards-manually)
* [FileBeat agent deployment and configuration](#filebeat-agent-deployment-and-configuration)
  * [Description](#description)
  * [Implementation](#implementation)
  * [Integrating Filebeat as a Sidecar Pod for Cross Namespace Log Collection](#integrating-filebeat-as-a-sidecar-pod-for-cross-namespace-log-collection)
* [Infrastructure metrics](#infrastructure-metrics)
  * [Usage of resources](#usage-of-resources)
  * [Throughput](#throughput)
  * [Response time and Availability](#response-time-and-availability)
  * [Energy efficiency](#energy-efficiency)
* [Modify ILM policy](#modify-ilm-policy)
  * [Apply changes on heartbeat-ilm](#apply-changes-on-heartbeat-ilm)
  * [Apply changes on metricbeat-ilm](#apply-changes-on-metricbeat-ilm)
  * [Apply changes on business-ilm and technical-ilm](#apply-changes-on-business-ilm-and-technical-ilm)
* [Performance parameters](#performance-parameters)
  * [Logstash performance parameters](#logstash-performance-parameters)
  * [Elasticsearch performance parameters](#elasticsearch-performance-parameters)
  * [Kibana performance parameters](#kibana-performance-parameters)
  * [Filebeat performance parameters](#filebeat-performance-parameters)
* [Monitoring API ](#monitoring-api)
  * [Kibana](#kibana)
  * [Elasticsearch](#elasticsearch)



<!-- TOC -->

### Export dashboards to git repository

Exporting dashboards:
  - Login to Kibana
  - Go to Stack Management → Saved Object
  - Select Type: Dashboards
  - Choose dashboards to export.

![ExportDashoards](images/Export_dashboards_001.png)


  - Press "Export" to download ndjson file with dashboards to local PC.

![ExportDashoards](images/Export_dashboards_002.png)


  - Rename downloaded file to dashboards.ndjson and upload to Git repository to path: eck-monitoring/charts/kibana/dashboards/

![ExportDashoards](images/Export_dashboards_003.png)

### Import dashboards manually

Loading dashboards:

  - download [dashboards.ndjson](https://code.europa.eu/simpl/simpl-open/development/monitoring/eck-monitoring/-/tree/main/charts/kibana/dashboards?ref_type=heads) file from repo directory eck-monitoring/kibana/dashdoards/

![ImportDashoards](images/Import_dashboards_001.png)

  - Choose file: dashboards.ndjson
  - Click on download button:

![ImportDashoards](images/Import_dashboards_002.png)

  - login to Kibana
  - go to Stack Management -> Saved Object
  - Choose Import:

![ImportDashoards](images/Import_dashboards_003.png)

  - Set "Request action on conflict"
  - Press "Import" and choose downloaded file.

![ImportDashoards](images/Import_dashboards_004.png)  

  - Press "Import"

![ImportDashoards](images/Import_dashboards_005.png) 


  - All dashboards should be imported.

Depend on changes fix conflicts.

### FileBeat agent deployment and configuration

#### Description:
    Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that user specifies, collects log events, and forwards them either to Elasticsearch or Logstash for indexing. Filebeat consists of two main components: inputs and harvesters. These components work together to tail files and send event data to the output that you specify.

##### Harvester
    A harvester is responsible for reading the content of a single file. The harvester reads each file, line by line, and sends the content to the output. One harvester is started for each file. The harvester is responsible for opening and closing the file, which means that the file descriptor remains open while the harvester is running. If a file is removed or renamed while it’s being harvested, Filebeat continues to read the file. This has the side effect that the space on your disk is reserved until the harvester closes. By default, Filebeat keeps the file open until close_inactive value is reached.

    Closing a harvester has the following consequences:
        -   The file handler is closed, freeing up the underlying resources if the file was deleted while the harvester was still reading the file.
        -   The harvesting of the file will only be started again after scan_frequency has elapsed.
        -   If the file is moved or removed while the harvester is closed, harvesting of the file will not continue.

    To control when a harvester is closed, use the close_* configuration options.

##### Input
    An input is responsible for managing the harvesters and finding all sources to read from.

    We use type log - the input finds all files on the drive that match the defined glob paths and starts a harvester for each file. Each input runs in its own Go routine.

#### Implementation
##### Overview

    This configuration deploys Filebeat on a Kubernetes cluster as a DaemonSet to collect and ship logs to Elasticsearch and Logstash. The setup includes custom scripts for log generation, secure connections with certificates, and monitoring.

##### Components
        Beat Configuration (filebeat.yaml):
            -   Kind: Beat
            -   Image: Uses Filebeat Docker image defined in values.yaml.
            -   Type: filebeat
            -   Elasticsearch Reference: Connects to the Elasticsearch instance named {{ .Release.Name }}-elasticsearch.

        Pod Template:
            Security Context: Runs as root user with file system group set to 1000.
            Container Specs:
                -   Executes a custom script example.sh and starts Filebeat.
                -   Mounts various volumes for configuration, scripts, and certificates.
            Environment Variables:
                -   Sets Elasticsearch and Logstash hosts, and monitoring credentials from secrets.

        Volumes:
            Config Volume: Loads filebeat.yml from a secret.
            Example Script Volume: Loads example.sh from a ConfigMap.
            Certificates Volumes: Loads necessary TLS certificates from secrets.

        RBAC Configuration:
            ServiceAccount: filebeat is created in the release namespace.
            RoleBinding: Grants access to the Issuer for managing certificates.

        Certificate Management:
            Uses cert-manager to create TLS certificates for secure communication.

##### Supporting Resources

        Secret (filebeat-config): Contains the Filebeat configuration file filebeat.yml encoded in Base64.
        ConfigMap (filebeat-example-script): Provides example.sh for generating sample logs.
        ServiceAccount: Ensures Filebeat has the necessary permissions to access Kubernetes resources.
        RoleBinding: Links the service account with a role for accessing the Issuer.

##### values.yaml
        Image Details: Defines the Filebeat image and tag.
        Log Generation:
            -   totalMessages: Number of log messages to generate (can be infinite).
            -   messagesPerMinute: Rate of log generation.
        TLS Configuration:
            -   Sets certificate duration and renewal parameters.
        Filebeat Inputs and Outputs:
            -   Input: Configures Filebeat to read logs from example.log using multiline patterns.
            -   Output: Sends logs to Logstash with TLS enabled and configures Elasticsearch for monitoring.

##### Security
        Uses secrets for sensitive data like TLS certificates and monitoring credentials.
        Configures secure communication with Elasticsearch and Logstash using SSL/TLS.

#### Integrating Filebeat as a Sidecar Pod for Cross Namespace Log Collection
##### Overview

Integrating Filebeat as a sidecar container allows you to collect logs directly from applications running in different namespaces within your Kubernetes cluster. This setup ensures centralized log collection and monitoring, leveraging Filebeat's capabilities for log shipping to Elasticsearch or Logstash.

##### Deployment Steps

To integrate Filebeat as a sidecar pod for collecting logs from an application in a different namespace, follow these steps:

    1. Modify Application Deployment YAML

    Update your application's Deployment YAML (my-app.yaml) in the target namespace (app-namespace) to include Filebeat as a sidecar container.
```
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: my-app
      namespace: app-namespace
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: my-app
      template:
        metadata:
          labels:
            app: my-app
        spec:
          containers:
          - name: my-app
            image: my-app-image:latest
            volumeMounts:
            - name: app-logs
              mountPath: /var/log/myapp
          - name: filebeat
            image: {{ .Values.filebeat.image }}:{{ default .Values.elasticVersion  .Values.filebeat.imageTag }}
            command: ['sh', '-c', 'exec /usr/share/filebeat/filebeat -e -c /usr/share/filebeat/filebeat.yml']
            volumeMounts:
            - name: app-logs
              mountPath: /var/log/myapp
            - name: config
              mountPath: /usr/share/filebeat/filebeat.yml
              subPath: filebeat.yml
            - name: filebeat-certs
              mountPath: /usr/share/filebeat/certs
            env:
            - name: ELASTIC_ELASTICSEARCH_ES_HOSTS
              value: 'https://elastic-elasticsearch-es-http.observability.svc:9200'
            - name: LOGSTASH_HOSTS
              value: 'logstash-{{ .Values.logstash.beats.pipelines_group_name }}-ls-api.observability.svc:5044'
            - name: MONITORING_USER
              valueFrom:
                secretKeyRef:
                  name: user-monitoring-secret
                  key: username
            - name: MONITORING_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: user-monitoring-secret
                  key: password
          volumes:
          - name: app-logs
            emptyDir: {}
          - name: config
            secret:
              secretName: filebeat-config
          - name: filebeat-certs
            secret:
              secretName: filebeat-certs-secret
```
    Explanation:
        Containers: Includes your application (my-app) container and Filebeat as a sidecar.
        Volume Mounts: Ensures the application's log directory is mounted for Filebeat.
        Environment Variables: Configures connections to Elasticsearch, Logstash, and monitoring credentials.
        Volumes: Mounts Filebeat configuration (filebeat.yml) and TLS certificates from secrets.

    2. Update Filebeat Configuration (Monitoring team task)

    Ensure your Filebeat configuration (filebeat.yml) includes the necessary inputs and outputs for collecting and shipping logs from your application. Customize the configuration as per your logging requirements. To be aligned with monitoring team during Sprint 3.

    3. Deploy and Verify

##### Security and Access Considerations

    - RBAC Permissions: Ensure the ServiceAccount used by Filebeat (filebeat service account) has appropriate permissions (RoleBinding) to access resources in the app-namespace.
    - Network Policies: Configure Kubernetes Network Policies to allow communication between namespaces if required.
    - TLS Configuration: Secure communication with Elasticsearch or Logstash using TLS certificates stored in Kubernetes secrets.

### Infrastructure metrics

#### Usage of resources

Performance parameters like CPU, RAM, Disk will be collected by metricbeat.
All kubernetes components: nodes, pods will be monitor.
Elastic cluster will be monitor and stats can be available in Kibana "Stack Monitoring" section.

#### Throughput:

    Throughput will be collected by metricbeat.
    Following types of throughputs are available:
        -   network
        -   disk

Example graph for network throughput and other parameters:

![metricbeat_dashboard](images/metricbeat_dashboard_001.png) 

Part of configuration:

```
modules:
- module: system
  period: 10s
  metricsets:
  - cpu
  - load
  - memory
  - network
  - process
  - process_summary
  process:
    include_top_n:
      by_cpu: 5
      by_memory: 5
  processes:
  - .*
```

#### Response time and Availability
This parameter will be provided by heartbeat.
Example configuration:

```
heartbeat.monitors:
- type: tcp
  schedule: '@every 5s'
  hosts: ["elastic-elasticsearch-es-http.observability.svc:9200"]
- type: tcp
  schedule: '@every 5s'
  hosts: ["kibana.dev.simpl-europe.eu:443"]
- type: icmp
  id: ping-myhost
  name: My Host Ping
  hosts: ["elastic-kibana-kb-http.observability.svc"]
  schedule: '*/5 * * * * * *'
```


    Available monitoring types:
  - tcp
  - icmp
  - http

Response time parameter is available in monitor.duration.us field, example:

![metricbeat_dashboard](images/metricbeat_dashboard_002.png) 

Availability parameter is in state.status field:

![metricbeat_dashboard](images/metricbeat_dashboard_003.png) 

#### Energy efficiency

Energy Efficiency is calculated from field system.load.norm.5 field provided by metricbeat and maximum power used by CPU.
Pattern to calculate value:
'''
Ef = system_load * Pmax
'''

Where:
  - Ef = Energy efficiency
  - system_load = value from field  system.load.norm.5
  - Pmax = Maximum power used by CPU from vendor page


Parameter definition is "Load for the last 5 minutes divided by the number of cores."
Pmax has been assumed to 200 Watt.


### Modify ILM policy

Monitoring ELK stack are 5 ILM polices:

 - business-ilm - responsible  for index rotation with business  ines logs
 - technical-ilm - responsible  for index rotation with technical logs
 - metricbeat-ilm - responsible  for index rotation with metrics collected from agents
 - filebeat - responsible  le for index rotation with ELK stack logs 
 - heartbeat-ilm - responsible  for index rotation with services heartbeats 



#### Apply changes on heartbeat-ilm:

Modify values/dev/observability/values.yml file and set new values:
```
    heartbeat:
    ilm:
        hot:
        max_age: 30d                      <- set max age for documents in hot phase 
        max_primary_shard_size: 50gb      <- set max size for primary shard in hot phase 
        delete:
        min_age: 365d                     <- set age after which index will be deleted
    services:
        heartbeat.monitors:
```

    Restart heartbeat by command:
```kubectl rollout restart deployment heartbeat-beat-heartbeat```


#### Apply changes on metricbeat-ilm:

Modify values/dev/observability/values.yml file and set new values:


```    
metricbeat:
    ilm:
        hot:
        max_age: 30d                        <- set max age for documents in hot phase 
        max_primary_shard_size: 50gb        <- set max size for primary shard in hot phase 
        delete:
        min_age: 365d                       <- set age after which index will be deleted
    resources:
        requests:
        memory: 500Mi
        limits:
```


    Restart metricbeat by command:
```kubectl rollout restart daemonset metricbeat-beat-metricbeat```


#### Apply changes on filebeat:

  - Login to Kibana and go to: Stack Management -> Index Lifecycle Policies.
  - Click on filebeat policy
  - Modify "Hot phase" in advanced setting when disable "Use recommended defaults" and/or modify Delete phase if needed.
  - Press "Save policy".


#### Apply changes on business-ilm and technical-ilm:
   

Modify values/dev/observability/values.yml file and set new values:



```    
logstash:
    ilm:
        business:
        hot:
            max_age: 30d                    <- set max age for business documents in hot phase 
            max_primary_shard_size: 1gb     <- set max size for business primary shard in hot phase 
        delete:
            min_age: 30d                    <- set age after which business index will be deleted
        technical:
        hot:
            max_age: 30d                    <- set max age for technical documents in hot phase 
            max_primary_shard_size: 1gb     <- set max size for technical primary shard in hot phase
        delete:
            min_age: 30d                    <- set age after which technical index will be deleted
    count_beats: 1
    count_syslog: 0
```


    Restart logstash statefulsets by command:
```    kubectl rollout restart sts logstash-beats-ls```


### Performance parameters

#### Logstash performance parameters 

    In file values/dev/observability/values.yml are following logstash performance parameters:

    -   logstash.env.ls_java_opts: "-Xms3g -Xmx3g"
        Set heap memory for logstash process inside container.
        Logstash statefulsets restart is required. 

    -   logstash.resources.requests.memory: 4Gi
        logstash.resources.limits.memory: 4Gi
        Set memory allocation (request/limit) for logstash pod.

    -   logstash.resources.requests.cpu:  300m
        logstash.resources.limits.cpu: 300m
        Set CPU allocation (request/limit) for logstash pod.

    -   pipelines_yml_config.pipeline.workers: 1
        Set number of workers for logstash pipeline.
        Logstash statefulsets restart is required. 

    -   pipelines_yml_config.pipeline.pipeline.batch.size: 125
        Set batch_size for logstash pipeline.
        Logstash statefulsets restart is required. 

#### Elasticsearch performance parameters

    -   elasticsearch.diskSpace: 60Gi
        Set disk size to store indices in elasticsearch pods

    -   elasticsearch.count: 3
        Number of elasticsearch pods in stack

    -   elasticsearch.resources.requests.memory: 4Gi
        elasticsearch.resources.limits.memory: 4Gi
        Set memory allocation (request/limit) for elasticsearch pod.

    -   elasticsearch.resources.requests.cpu:  300m
        elasticsearch.resources.limits.cpu: 300m
        Set CPU allocation (request/limit) for elasticsearch pod.


#### Kibana performance parameters

    -   kibana.resources.requests.memory: 1Gi
        kibana.resources.limits.memory: 1Gi
        Set memory allocation (request/limit) for kibana pod.

    -   kibana.resources.requests.cpu:  300m
        kibana.resources.limits.cpu: 300m
        Set CPU allocation (request/limit) for kibana pod.

    -   kibana.count: 1
        Number of kibana pods in stack

#### Filebeat performance parameters

    -   filebeat4agents.resources.requests.memory: 1Gi
        filebeat4agents.resources.limits.memory: 1Gi
        Set memory allocation (request/limit) for filebeat pod.

    -   filebeat4agents.resources.requests.cpu:  100m
        filebeat4agents.resources.limits.cpu: 100m
        Set CPU allocation (request/limit) for filebeat pod.

### Monitoring API 

#### Kibana

    API for Kibana is available at page: "https://kibana.<namespace>.<env>.simpl-europe.eu"
    Example for stack observability at dev cluster:  https://kibana.observability.dev.simpl-europe.eu/


    Link to vendor manual page with API:
    https://www.elastic.co/docs/api/doc/kibana/v8


    Example API call to get all data views from Kibana:
```
https://kibana.observability.dev.simpl-europe.eu/api/data_views
```


Request:
```
$ curl -k -s -u elastic -X GET "https://kibana.observability.dev.simpl-europe.eu/api/data_views?v"
```


Response:
```
{"data_view":
[
{"id":"metricbeat-*",
"namespaces":["default"],
...
]
}
```

#### Elasticsearch

    API for Elasticsearch is available at page: "https://elasticsearch.<namespace>.<env>.simpl-europe.eu"
    Example for stack observability at dev cluster:  https://elasticsearch.observability.dev.simpl-europe.eu/


    Link to vendor manual page with API:
    https://www.elastic.co/guide/en/elasticsearch/reference/current/rest-apis.html


    Example API call to get cluster state:
```    
https://elasticsearch.observability.dev.simpl-europe.eu/_cluster/health
```

Request:
```
$ curl -k -Ss -u elastic -X GET "https://elasticsearch.observability.dev.simpl-europe.eu/_cluster/health"
```


    Response:
```
{
"cluster_name":"elastic-elasticsearch",
"status":"green",
"timed_out":false,
"number_of_nodes":3,
"number_of_data_nodes":3,
"active_primary_shards":40,
"active_shards":80,
"relocating_shards":0,
"initializing_shards":0,
"unassigned_shards":0,
"delayed_unassigned_shards":0,
"number_of_pending_tasks":0,
"number_of_in_flight_fetch":0,
"task_max_waiting_in_queue_millis":0,
"active_shards_percent_as_number":100.0
}
```