2 posts tagged with "AWS"

To provide basic functionality over your Kubernetes cluster, we often need to configure Addons (usually in the form of Helm charts).

Understanding which permissions are required for the Helm charts is cruicial. Too many permissions, and you open yourself up to a breach if the service account is compromised. Too little, and you will run into runtime issues.

Documentation around required permissions for each addon are often limited, and we have to scour google to find the best permissions.

AWS EKS Blueprints​

EKS Blueprints is an official repository managed by AWS, to provision EKS clusters using AWS CDK.

The repository contains a lib/addon directory. Each addon defines required permissions for it to function. The policies are nested inside some functions, but they're simple enough to understand and copy.

For example -

• LB Controller Policy - Allows provisioning of load balancers.
• EBS CSI Driver Policy - Allows creating EBS volumes on your cluster.
• EFS CSI Driver Policy - Allows creating EFS volumes on your cluster.

We can further refine the search of all available policies by using GitHub search filters. The following filter will find all files containing polic in the lib/addons folder:

repo:aws-quickstart/cdk-eks-blueprints path:/^lib\/addons\// path:*polic*.ts

How to use​

If you're unsure about permissions for a specific addon, you can look at the EKS blueprints to help you figure them out. This is not a catch-all solution, but it may help you define fine grained permissions for your addons.

You can take these permissions, and copy them to your Terraform / Pulumi / AWS CDK code, and reference the blueprints.

Useful links​

• Addon is a helm chart that adds functionality to your cluster ()
• EKS is the managed Kubernetes solution on AWS
• IAM is the security engine for creating users, roles and policies.

This blog post is about high availbility block storage on AWS EKS.

The provided block storage, EBS, does not replicate across regions. This has major implications during AZ downtime. A We explore how Rook, an CNCF distibuted storage solution, can be used to provide a better infrastrcutre. We look at the pros and cons of managing a Rook cluster and some use-cases for it.

Why do I need distributed block storage?​

A common problem people run into is deploying a high-availbity service on top of AWS EKS.

A few years ago, we deployed a Prometheus instance on AWS. Everything ran smoothly, until the availability zone went down.

Suddenly, the pod was unschedulable. We found out the hard way that EBS storage is restricted to the AZ it was created in.

If you want to build a highly available system, your storage needs to be highly available as well.

Rook - Distributed Storage​

Rook allows us to create a highly available, distributed storage on EKS.

Simply put, Rook:

• Spins up a Ceph cluster on multiple AZs with provisioned capacity
• Lets you define a Storage Class
• PVCs (persistent volume claims) can then use this storage class for multi AZ

Rook has been a CNCF graduate since 2020, so it's extremely stable.

Setup​

There's a great video by Red Hat Developer on how to set up Rook on AWS.

I'd recommend installing Rook using the Rook Ceph helm chart to provision the Ceph cluster to better integrate with your GitOps environment.

Considerations​

• Rook comes with a signicant learning curve for understanding and using Ceph
• Ceph requires a lot of resources
  • Multi AZ requires multiple instance
  • Minimum recommended storage of 100GB per node (totalling 300GB for the cluster)

For the above reasons, to fully utilize and justify deploying Rook, you should have high storage requirements and justify the cost overhead of doing so.

Use Cases​

Prometheus High Availability​

Prometheus is a great candidate for Rook:

• Prometheus scans metrics perdiocially (usually every 30 seconds)
  • having it down for two minutes is not a deal breaker.
• Simplifies the HA setup for Prometheus
• Removes the need to send duplicate metrics to third parties which can get expensive.

Kafka​

Kafka relies on block storage which means the storage will not be available during AZ downtime.

While distributed by design, this has performance implications on your cluster. Here's a quick breakdown of how Kafka manages partitions:

• Topics are partitioned and saved to disk.
• Partitions are replicated across brokers using a replication factor.
• Each partition is assigned a partition leader that serves all reads and writes to the parition.

Using distributed storage will allow you to avoid the following Kafka shortcomings:

With distributed storage in place, in both scenarios the broker will reschedule on another AZ.

Notable Mentions​

• Redis (in non-cluster mode), can benefit from distributed storage.

Summary (TL;DR)​

Overall, distributed storage can be very useful, and Rook provides an easy setup for it.

Higher costs and Ceph maintenance should be considered and weighted against other disaster scenarios to understand whether it's worth it.

The larger your cluster and storage requirements, the more distributed storage becomes cost-efficient.

For non-production environments clusters, distributed storage makes little sense.

Useful Links​

• Kafka - Considerations for data storage
• Prometheus high availability