Setting Up Grafana SMTP Alerting on Kubernetes for Enhanced Monitoring

In today's world of cloud-native applications and microservices, monitoring and alerting are critical to maintaining the health and performance of systems. Grafana is one of the most popular tools for monitoring, and when combined with Kubernetes, it offers a powerful solution for visualizing and alerting on your infrastructure’s metrics.

One of the essential features for a robust monitoring setup is alerting—especially via email. In this article, we’ll walk through how to configure Grafana on Kubernetes to send email alerts using SMTP, ensuring you're notified of any critical system events.

### Step 1: Create a Kubernetes Secret for SMTP Credentials

The first step in setting up Grafana SMTP alerting is to securely store the SMTP credentials using Kubernetes secrets. This way, we avoid hardcoding sensitive information like SMTP usernames and passwords directly in the manifest files.

To create the secret, use the following command:

kubectl create secret -n monitoring generic notifications-smtp  --from-literal=user=<smtp username>  --from-literal=password=<smtp password>  --from-literal=host=<smtp server:port>        

This command creates a secret called notifications-smtp in the monitoring namespace with your SMTP credentials.

### Step 2: Grafana Deployment with SMTP Configuration

Once the SMTP secret is created, it’s time to configure the Grafana deployment. The key part here is to reference the secret within the Grafana deployment manifest.

Below is the deployment configuration for Grafana (`deployment.yaml`), which integrates the SMTP configuration for alerting:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        env:
          - name: GF_SMTP_ENABLED
            value: "true"
          - name: GF_SMTP_FROM_ADDRESS
            value: "your_mail_from_address"
          - name: GF_SMTP_FROM_NAME
            value: "Grafana monitoring"
          - name: GF_SMTP_HOST
            valueFrom:
              secretKeyRef:
                name: notifications-smtp
                key: host
          - name: GF_SMTP_PASSWORD
            valueFrom:
              secretKeyRef:
                name: notifications-smtp
                key: password
          - name: GF_SMTP_USER
            valueFrom:
              secretKeyRef:
                name: notifications-smtp
                key: user
        ports:
        - name: grafana
          containerPort: 3000
        resources:
          limits:
            memory: "1Gi"
            cpu: "1000m"
          requests:
            memory: 500M
            cpu: "500m"
        volumeMounts:
          - mountPath: /var/lib/grafana
            name: grafana-storage
          - mountPath: /etc/grafana/provisioning/datasources
            name: grafana-datasources
            readOnly: false
      volumes:
        - name: grafana-storage
          emptyDir: {}
        - name: grafana-datasources
          configMap:
              defaultMode: 420
              name: grafana-datasources
        - name: grafana-config
          configMap:
            name: grafana-smtp-config        

In this configuration:

- We enable SMTP by setting GF_SMTP_ENABLED to "true".

- SMTP credentials (host, user, and password) are securely retrieved from the notifications-smtp secret.

- GF_SMTP_FROM_ADDRESS and GF_SMTP_FROM_NAME define the sender's email address and name for alert emails.

### Step 3: Configuring Grafana Data Sources

Now, let’s configure Grafana’s data sources. A common choice is Prometheus, which Grafana can use to scrape and visualize metrics. Below is a ConfigMap for configuring the Prometheus datasource:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  prometheus.yaml: |-
    {
        "apiVersion": 1,
        "datasources": [
            {
               "access": "proxy",
               "editable": true,
               "name": "prometheus",
               "orgId": 1,
               "type": "prometheus",
               "url": "https://prometheus-service.monitoring.svc:8080",
               "version": 1
            }
        ]
    }        

This configures Grafana to pull data from Prometheus running in the same Kubernetes namespace.

### Step 4: Exposing Grafana with a Kubernetes Service

To make Grafana accessible, we need to expose it via a Kubernetes Service. Below is an example service.yaml for exposing Grafana as a NodePort service:

apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '3000'
spec:
  selector:
    app: grafana
  type: NodePort
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 32000        

This service exposes Grafana on port 32000 for external access. It also includes annotations to allow Prometheus to scrape metrics from Grafana.

### Step 5: Deploying Grafana

Once you have all your files configured, it’s time to deploy Grafana and its associated resources:

kubectl apply -f grafana-datasource-config.yaml
kubectl apply -f service.yaml
kubectl apply -f deployment.yaml        

### Step 6: Accessing Grafana

Grafana should now be accessible at:

https://<node-ip>:32000        

You can log in using the default credentials (username: admin, password: admin).