Implementation of Fluent Bit with AWS CloudWatch within the Logging in Kubernetes - Best Practice, Benefits, and Challenges

Introduction

Logging is important when monitoring and debugging applications in a fast-paced container orchestration world. With Kubernetes being the most popular container orchestration platform, a reliable logging solution is also needed to facilitate perfect observability. Fluent Bit as a lightweight log processor, together with AWS CloudWatch, goes a long way in helping make an effective, highly scalable logging solution for Kubernetes environments. It provides best practices, sample codes, and pros and cons of Fluent Bit and AWS CloudWatch for implementing Kubernetes logging.

Benefits of logging

• Enhanced Observability: Improved visibility into the application’s behavior and performance through centralized logging.
• Faster Debugging: Easier to identify and troubleshoot issues by aggregating logs from multiple sources.
• Compliance and Security: Ensures that log data is securely stored and accessible for compliance and auditing purposes.

Pros and Cons of Fluent Bit

Pros

• Centralized logging: Aggregated logs from all pods of Kubernetes at one point, which are easy to handle and analyze.
• Scalability: The combination of Fluent Bit with CloudWatch gives room for scalability purposes to enable high-volume logs to be smoothly filed.
• Cost efficiency: Fluent Bit is lightweight and consumes a low amount of resources compared to other log processors.
• It has a rich configuration, so Fluent Bit provides flexibility in log filtering, parsing, and routing.

Cons

• Initial Setup Complexity: The initial setup of Fluent Bit with CloudWatch can be complex and requires careful configuration.
• Cost: Depending on the log volume, the cost of storing logs in CloudWatch can be significant.
• Maintenance: Ongoing maintenance is required to ensure the logging setup remains efficient and cost-effective.

Set up Fluent Bit on EKS / AWS

I followed the link to set up Fluent Bit on EKS ( Elastic Kubernetes Service ) on AWS.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html        

Step 1: Create a namespace called `amazon-cloudwatch`

$ kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml        

Step 2: Create a ConfigMap named cluster-info with the cluster name and the Region to send logs to:

ClusterName=cluster-name
RegionName=cluster-region
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
kubectl create configmap fluent-bit-cluster-info \
--from-literal=cluster.name=${ClusterName} \
--from-literal=http.server=${FluentBitHttpServer} \
--from-literal=http.port=${FluentBitHttpPort} \
--from-literal=read.head=${FluentBitReadFromHead} \
--from-literal=read.tail=${FluentBitReadFromTail} \
--from-literal=logs.region=${RegionName} -n amazon-cloudwatch        

Note: An explanation of each term is in the doc .

Step 3: I choose the Linux machine's command from the doc:

$ kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml        

These are instructed by AWS Doc.

Role and Policy

Now I will share a very important part of Role and Policy which is required for permissions. We face a very common issue :

CreateLogStream API responded with error='AccessDeniedException'        

To resolve this issue we follow the steps:

Step A: Create a policy with the permissions

Name is for example: "FluentBitPolicy"

Step B: Create a Role with the type "Web Identity"

1. Select Web Identity:

In the IAM console, select the Web identity as the trusted entity type. This allows the role to be assumed by a Kubernetes service account.

2. Configure the Identity Provider:

Choose your OIDC provider, which should be your EKS cluster's OIDC provider URL. For Audience, use sts.amazonaws.com.

3. Trust Relationship:

Ensure the trust relationship allows the Kubernetes service account to assume the role.

To find the OIDC provider URL:

$ aws eks describe-cluster --name your-cluster-name --query "cluster.identity.oidc.issuer" --output text        

4. Attach the Policy we created earlier "FluentBitPolicy" from aws console

OR command if you want to do it by command

aws iam attach-role-policy --policy-arn arn:aws:iam::YOUR_ACCOUNT_ID:policy/FluentBitPolicy --role-name FluentBitRole        

5. Add Trust Policy to Role:

Replace YOUR_AWS_ACCOUNT_ID, REGION, and YOUR_EKS_CLUSTER_ID with your specific values.

6. Name the role for example: "FluentBitRole"

Optional 7. You can annotate the role with your service account if needed ( in other way: you can have YAML)

$ kubectl annotate serviceaccount fluent-bit \
  -n amazon-cloudwatch \
  eks.amazonaws.com/role-arn=arn:aws:iam::123456789012:role/your-iam-role-name        

These steps ensure your role policy setup for sending logs to cloudwatch.

Service Account Commands:

if you want to create a service account manually instead of YAML :

Configuration Step: You can optimize, and exclude logs from your cluster - you may customize your :

ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: amazon-cloudwatch
  labels:
    k8s-app: fluent-bit
.....        

and DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: amazon-cloudwatch
  labels:
    k8s-app: fluent-bit
    version: v1
    kubernetes.io/cluster-service: "true"
.....        

Verify Logs in CloudWatch

• Navigate to the CloudWatch console.
• Check the log group `/aws/containerinsights/<application-name>/application`.
• Verify the log streams prefixed with `<application-name>-<pod-name>`.

Conclusion

Fluent Bit and AWS CloudWatch integration offers robustness and scalability for a solution in the domain of Kubernetes logging. Best practices and resolutions of challenges help organizations achieve high-level observability, enhanced debugging, and security standard compliance in the application logs. This might need some initial effort and careful configuration, but it is all worth it, given the benefits that accrue in terms of centralized and efficient log management.

Resources:

1. Set up Fluent Bit as a DaemonSet to send logs to CloudWatch Logs
2. https://fluentbit.io/