Code development platform for open source projects from the European Union institutions

Skip to content
Snippets Groups Projects
Commit 5bb65ef5 authored by Vara Bonthu's avatar Vara Bonthu
Browse files

Docs added for EMR on EKS module

parent b508c82f
No related branches found
No related tags found
No related merge requests found
Showing
with 140 additions and 66 deletions
# Agones
# Agones
[Agones](https://agones.dev/) is an open source platform for deploying, hosting, scaling, and orchestrating dedicated game servers for large scale multiplayer games on Kubernetes.
For complete project documentation, please visit the [Agones documentation site](https://agones.dev/site/docs/).
## Usage
## Usage
Agones can be deployed by enabling the add-on via the following.
```hcl
agones_enable = true
```
\ No newline at end of file
```
......@@ -6,10 +6,10 @@ Fluent Bit is an open source Log Processor and Forwarder which allows you to col
AWS provides a Fluent Bit image with plugins for both CloudWatch Logs and Kinesis Data Firehose. The [AWS for Fluent Bit](https://github.com/aws/aws-for-fluent-bit) image is available on the Amazon ECR Public Gallery. For more details, see [aws-for-fluent-bit](https://gallery.ecr.aws/aws-observability/aws-for-fluent-bit) on the Amazon ECR Public Gallery.
### Usage
### Usage
[aws-for-fluent-bit](kubernetes-addons/aws-for-fluent-bit/README.md) can be deployed by enabling the add-on via the following.
```hcl
aws_for_fluent_bit_enable = true
```
\ No newline at end of file
```
# AWS Load Balancer Controller
# AWS Load Balancer Controller
The [AWS Load Balancer Controller](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html) manages AWS Elastic Load Balancers for a Kubernetes cluster. The controller provisions the following resources:
* An AWS Application Load Balancer (ALB) when you create a Kubernetes Ingress.
* An AWS Network Load Balancer (NLB) when you create a Kubernetes Service of type LoadBalancer.
* An AWS Network Load Balancer (NLB) when you create a Kubernetes Service of type LoadBalancer.
For more information about AWS Load Balancer Controller please see the [official documentation](https://docs.aws.amazon.com/eks/latest/userguide/aws-load-balancer-controller.html).
......@@ -23,4 +23,4 @@ aws-load-balancer-controller 2/2 2
```
#### AWS Service annotations for LB Ingress Controller
Here is the link to get the AWS ELB [service annotations](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations/) for LB Ingress controller.
\ No newline at end of file
Here is the link to get the AWS ELB [service annotations](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations/) for LB Ingress controller.
......@@ -4,10 +4,10 @@
For complete documentation, please visit the [AWS Distro for OpenTelemetry documentation site](https://aws-otel.github.io/).
## Usage
## Usage
The AWS Distro for OpenTelemetry Collector can be deployed into an EKS cluster by enabling the add-on via the following.
```
aws_open_telemetry_enable = true
```
\ No newline at end of file
```
......@@ -10,4 +10,4 @@ cert-manger can be deployed by enabling the add-on via the following.
```hcl
cert_manager_enable = true
```
\ No newline at end of file
```
......@@ -4,13 +4,13 @@ Cluster Autoscaler is a tool that automatically adjusts the number of nodes in y
* Pods fail due to insufficient resources, or
* Pods are rescheduled onto other nodes due to being in nodes that are underutilized for an extended period of time.
The [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) add-on adds support for Cluster Autoscaler to an EKS cluster. It is typically installed as a **Deployment** in your cluster. It uses leader election to ensure high availability, but scaling is one done via one replica at a time.
## Usage
## Usage
[Cluster Autoscaler](kubernetes-addons/cluster-autoscaler/README.md) can be deployed by enabling the add-on via the following.
```hcl
cluster_autoscaler_enable = true
```
\ No newline at end of file
```
......@@ -2,10 +2,10 @@
[Fluent Bit for Fargate](https://aws.amazon.com/blogs/containers/fluent-bit-for-amazon-eks-on-aws-fargate-is-here/) configures Fluent Bit to forward Fargate Container logs to CloudWatch.
### Usage
### Usage
Fluent Bit for Fargate can be deployed by enabling the add-on via the following.
```hcl
fargate_fluent_bit_enable = true
```
\ No newline at end of file
```
......@@ -19,10 +19,10 @@ The framework currently provides support for the following add-ons.
| [OpenTelemetry](./aws-load-balancer-controller) | Deploys the OpenTelemetry Collector into an EKS cluster.
| [Prometheus](./prometheus) | Deploys Prometheus into an EKS cluster. |
| [Traefik](./traefik) | Deploys Traefik Proxy into an EKS cluster.
| [Windows VPC Controller](./windows-vpc-controllers) |
| [Windows VPC Controller](./windows-vpc-controllers) |
## Installation
## Installation
By default, the module is configured to fetch Helm Charts from Open Source repositories and Docker images from Docker Hub/Public ECR repositories. This requires outbound Internet connection from your EKS Cluster.
By default, the module is configured to fetch Helm Charts from Open Source repositories and Docker images from Docker Hub/Public ECR repositories. This requires outbound Internet connection from your EKS Cluster.
Alternatively you can download the Docker images for each add-on and push them to an AWS ECR repo and this can be accessed within an existing VPC using an ECR endpoint. For instructions on how to do so download existing images, and push them to ECR, see [ECR instructions](../docs/ecr-instructions.md). Each individual add-on directory contains a README.md file with info on the Helm repositories each add-on uses.
\ No newline at end of file
Alternatively you can download the Docker images for each add-on and push them to an AWS ECR repo and this can be accessed within an existing VPC using an ECR endpoint. For instructions on how to do so download existing images, and push them to ECR, see [ECR instructions](../docs/ecr-instructions.md). Each individual add-on directory contains a README.md file with info on the Helm repositories each add-on uses.
# Amazon EKS Add-ons
[Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html) provide installation and management of a curated set of add-ons for Amazon EKS clusters. All Amazon EKS add-ons include the latest security patches, bug fixes, and are validated by AWS to work with Amazon EKS. Amazon EKS add-ons allow you to consistently ensure that your Amazon EKS clusters are secure and stable and reduce the amount of work that you need to do in order to install, configure, and update add-ons.
[Amazon EKS add-ons](https://docs.aws.amazon.com/eks/latest/userguide/eks-add-ons.html) provide installation and management of a curated set of add-ons for Amazon EKS clusters. All Amazon EKS add-ons include the latest security patches, bug fixes, and are validated by AWS to work with Amazon EKS. Amazon EKS add-ons allow you to consistently ensure that your Amazon EKS clusters are secure and stable and reduce the amount of work that you need to do in order to install, configure, and update add-ons.
EKS currently provides support for the following managed add-ons.
......@@ -28,5 +28,5 @@ Our [Cluster Upgrade](../cluster-upgrades.md) guide demonstrates how you can lev
Additional information on updating a EKS cluster can be found in the [EKS documentation](https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html).
[Amazon VPC CNI]:(https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html)
[CoreDNS]:(https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html)
[kube-proxy]:(https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html)
\ No newline at end of file
[CoreDNS]:(https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html)
[kube-proxy]:(https://docs.aws.amazon.com/eks/latest/userguide/managing-kube-proxy.html)
# Metrics Server
# Metrics Server
Metrics Server is a scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines. It is not deployed by default in Amazon EKS clusters. The Metrics Server is commonly used by other Kubernetes add-ons, such as the Horizontal Pod Autoscaler, Vertical Autoscaling or the Kubernetes Dashboard.
> **Important**: Don't use Metrics Server when you need an accurate source of resource usage metrics or as a monitoring solution.
## Usage
## Usage
[Metrics Server](kubernetes-addons/metrics-server/README.md) can be deployed by enabling the add-on via the following.
......@@ -19,4 +19,4 @@ $ kubectl get deployments -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
metrics-server 1/1 1 1 20m
```
\ No newline at end of file
```
# Nginx
# Nginx
This add-on installs [Nginx Ingress Controller](https://kubernetes.github.io/ingress-nginx/deploy/) on Amazon EKS. The Nginx ingress controller uses [Nginx](https://www.nginx.org/) as a reverse proxy and load balancer.
This add-on installs [Nginx Ingress Controller](https://kubernetes.github.io/ingress-nginx/deploy/) on Amazon EKS. The Nginx ingress controller uses [Nginx](https://www.nginx.org/) as a reverse proxy and load balancer.
Other than handling Kubernetes ingress objects, this ingress controller can facilitate multi-tenancy and segregation of workload ingresses based on host name (host-based routing) and/or URL Path (path based routing).
Other than handling Kubernetes ingress objects, this ingress controller can facilitate multi-tenancy and segregation of workload ingresses based on host name (host-based routing) and/or URL Path (path based routing).
## Usage
......@@ -21,4 +21,3 @@ ssp-addon-nginx-ingress-78b8567p4q6 1/1 Running 0 4d10h
```
Note that the ingress controller is deployed in the `kube-system` namespace.
\ No newline at end of file
......@@ -4,10 +4,10 @@ OpenTelemetry offers a vendor-agnostic implementation on how to receive, process
For complete documentation, please visit the [OpenTelemetry Collector documentation site](https://opentelemetry.io/docs/).
## Usage
## Usage
The OpenTelemetry Collector can be deployed by enabling the add-on via the following.
```
open_telemetry_enable = true
```
\ No newline at end of file
```
......@@ -2,11 +2,11 @@
Prometheus is an open source monitoring and alerting service. Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.
This project provides support for installing a open source Prometheus server in your EKS cluster and for deploying a new Prometheus instance via [Amazon Managed Service for Prometheus](https://aws.amazon.com/prometheus/).
This project provides support for installing a open source Prometheus server in your EKS cluster and for deploying a new Prometheus instance via [Amazon Managed Service for Prometheus](https://aws.amazon.com/prometheus/).
## Usage
The following will deploy the Prometheus server into an EKS Cluster and provision a new Amazon Managed Service for Prometheus instance.
The following will deploy the Prometheus server into an EKS Cluster and provision a new Amazon Managed Service for Prometheus instance.
```hcl
# Creates the AMP workspace and all the relevent IAM Roles
......
# Traefik
# Traefik
Traefik is an open-source Edge Router that makes publishing your services a fun and easy experience. It receives requests on behalf of your system and finds out which components are responsible for handling them.
For complete project documentation, please visit the [Traefik documentation site](https://doc.traefik.io/traefik/).
## Usage
## Usage
[Traefik](kubernetes-addons/traefik-ingress/README.md) can be deployed by enabling the add-on via the following.
......@@ -22,12 +22,12 @@ $ kubectl port-forward svc/traefik -n kube-system 9000:9000
Now open the browser from your machine and enter the below URL to access Traefik Web UI.
```
```
http://127.0.0.1:9000/dashboard/
```
![alt text](https://github.com/aws-samples/aws-eks-accelerator-for-terraform/blob/a8ceac6c977a3ccbcb95ef7fb21fff0daf0b7081/images/traefik_web_ui.png "Traefik Dashboard")
#### AWS Service annotations for Traefik Ingress Controller
Here is the link to get the AWS ELB [service annotations](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations/) for Traefik Ingress controller
\ No newline at end of file
Here is the link to get the AWS ELB [service annotations](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/service/annotations/) for Traefik Ingress controller
# Windows VPC Controllers
# Windows VPC Controllers
## Pre-requisites
[cert-manager](https://cert-manager.io/) is currently needed to enable Windows support. The `cert-manager` [Helm chart](../cert-manager) will be automatically enabeld, if Windows support is enabled.
\ No newline at end of file
[cert-manager](https://cert-manager.io/) is currently needed to enable Windows support. The `cert-manager` [Helm chart](../cert-manager) will be automatically enabeld, if Windows support is enabled.
......@@ -2,9 +2,9 @@
#### Objective:
The purpose of this document is to provide an overview of the steps for upgrading the EKS Cluster from one version to another. Please note that EKS upgrade documentation gets published by AWS every year.
The purpose of this document is to provide an overview of the steps for upgrading the EKS Cluster from one version to another. Please note that EKS upgrade documentation gets published by AWS every year.
The current version of the upgrade documentation while writing this [README](https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html)
The current version of the upgrade documentation while writing this [README](https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html)
#### Pre-Requisites:
......@@ -22,13 +22,13 @@ This table shows the supported plugin versions for each EKS Kubernetes version
#### Steps to upgrade EKS cluster:
1. Change the version in Terraform to desired version under `base.tfvars`. See the example below
```hcl-terraform
kubernetes_version = "1.20"
```
2. Apply the changes to the cluster with Terraform. This step will upgrade the Control Plane and Data Plane to the newer version, and it will roughly take 35 mins to 1 hour
3. Once the Cluster is upgraded to desired version then please updated the following plugins as per the instructions
#### Steps to upgrade Add-ons:
......@@ -41,14 +41,14 @@ Just update the latest versions in `base.tfvars` file as shown below. EKS Addon
enable_kube_proxy_addon = true
kube_proxy_addon_version = "v1.20.4-eksbuild.2"
```
##### CoreDNS
##### CoreDNS
```hcl-terraform
enable_coredns_addon = true
coredns_addon_version = "v1.8.3-eksbuild.1"
```
##### VPC CNI
```hcl-terraform
......@@ -57,6 +57,6 @@ vpc_cni_addon_version = "v1.8.0-eksbuild.1"
```
Apply the changes to the cluster with Terraform.
## Important Note
Please note that you may need to update other Kubernetes Addons deployed through Helm Charts to match with new Kubernetes upgrade version
\ No newline at end of file
## Important Note
Please note that you may need to update other Kubernetes Addons deployed through Helm Charts to match with new Kubernetes upgrade version
# Docker upload to Elastic Container Registry
Download the docker image to your local Mac/Laptop
```
$ docker pull <image name>:<image tag>
```
Retrieve an authentication token and authenticate your Docker client to your registry. Use the AWS CLI:
```
```
$ aws ecr get-login-password --region <aws region> | docker login --username AWS --password-stdin <account id>.dkr.ecr.<aws region>.amazonaws.com
```
```
Create an ECR repo for your image.
Create an ECR repo for your image.
``` s
$ aws ecr create-repository --repository-name <image name> --image-scanning-configuration scanOnPush=true
$ aws ecr create-repository --repository-name <image name> --image-scanning-configuration scanOnPush=true
```
After the repo is created in ECR, tag your image so, you can push the image to this repository:
```
```
$ docker tag <image name>:<image tag> <account id>.dkr.ecr.<aws region.amazonaws.com/<image name>:<image tag>
```
Step6: Run the following command to push this image to your newly created AWS repository:
```
```
$ docker push <account id>.dkr.ecr.<aws region.amazonaws.com/<image name>:<image tag>
```
\ No newline at end of file
```
# EMR on EKS
EMR on EKS is a deployment option in EMR that allows you to automate the provisioning and management of open-source big data frameworks on EKS.
This module deploys the necessary resources to run EMR Spark Jobs on EKS Cluster.
- Create a new Namespace to run Spark workloads
- Create K8s Role and Role Binding to allow `emr-containers` user on a given namespace(`spark`)
- Create RBAC permissions and adding EMR on EKS service-linked role into aws-auth configmap
- Enables IAM Roles for Service Account (IRSA)
- Update trust relationship for job execution role
## Usage
[EMR on EKS](modules/emr-on-eks/README.md) can be deployed by enabling the module via the following.
```hcl
enable_emr_on_eks = true # Default is false
emr_on_eks_namespace = "spark" # Optional default value is spark
```
Once deployed, you can create Virtual EMR Cluster and execute Spark jobs. See the document below for more details.
https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up-registration.html
......@@ -2,9 +2,9 @@ site_name: Amazon EKS Accelerator for Terraform
repo_name: "aws-samples/aws-eks-accelerator-for-terraform"
repo_url: "https://github.com/aws-samples/aws-eks-accelerator-for-terraform"
docs_dir: "docs"
theme:
theme:
name: material
features:
features:
- tabs
nav:
- Overview: 'index.md'
......@@ -28,7 +28,7 @@ nav:
- Windows VPC Controllers: 'add-ons/windows-vpc-controllers.md'
markdown_extensions:
- def_list
- pymdownx.highlight
- pymdownx.highlight
- pymdownx.superfences
- pymdownx.inlinehilite
- pymdownx.tasklist:
......@@ -36,4 +36,4 @@ markdown_extensions:
- toc:
permalink: true
plugins:
- search
\ No newline at end of file
- search
# EMR on EKS Module
Amazon EMR on EKS provides a deployment option for Amazon EMR that allows you to run open-source big data frameworks on Amazon Elastic Kubernetes Service (Amazon EKS). With this deployment option, you can focus on running analytics workloads while Amazon EMR on EKS builds, configures, and manages containers for open-source applications.
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
## Requirements
No requirements.
## Providers
| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | n/a |
| <a name="provider_kubernetes"></a> [kubernetes](#provider\_kubernetes) | n/a |
| <a name="provider_null"></a> [null](#provider\_null) | n/a |
## Modules
No modules.
## Resources
| Name | Type |
|------|------|
| [aws_iam_policy.emr_on_eks_execution](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
| [aws_iam_role.emr_on_eks_execution](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
| [aws_iam_role_policy_attachment.emr_on_eks_execution](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [kubernetes_namespace.spark](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/namespace) | resource |
| [kubernetes_role.emr_containers](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/role) | resource |
| [kubernetes_role_binding.emr_containers](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/role_binding) | resource |
| [null_resource.update_trust_policy](https://registry.terraform.io/providers/hashicorp/null/latest/docs/resources/resource) | resource |
| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
## Inputs
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_eks_cluster_id"></a> [eks\_cluster\_id](#input\_eks\_cluster\_id) | n/a | `string` | n/a | yes |
| <a name="input_emr_on_eks_namespace"></a> [emr\_on\_eks\_namespace](#input\_emr\_on\_eks\_namespace) | n/a | `string` | `"spark"` | no |
| <a name="input_emr_on_eks_username"></a> [emr\_on\_eks\_username](#input\_emr\_on\_eks\_username) | n/a | `string` | `"emr-containers"` | no |
| <a name="input_environment"></a> [environment](#input\_environment) | n/a | `string` | n/a | yes |
| <a name="input_tenant"></a> [tenant](#input\_tenant) | n/a | `string` | n/a | yes |
| <a name="input_zone"></a> [zone](#input\_zone) | n/a | `string` | n/a | yes |
## Outputs
| Name | Description |
|------|-------------|
| <a name="output_emr_on_eks_role_arn"></a> [emr\_on\_eks\_role\_arn](#output\_emr\_on\_eks\_role\_arn) | IAM execution role ARN for EMR on EKS |
| <a name="output_emr_on_eks_role_id"></a> [emr\_on\_eks\_role\_id](#output\_emr\_on\_eks\_role\_id) | IAM execution role ID for EMR on EKS |
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment