Use AMP(Amazon Managed Prometheus) for AKS/GKE/anyother K8s with Roles-anywhere.

Are you running your workload in different clouds ? Lets say you're running your workload in AKS or GKE, but you don't to use GKE managed Prometheus or Azure managed Prometheus.

For this article we will be running our kubernetes Production cluster with AKS, Integrating Amazon Managed Prometheus with Azure Kubernetes Service (AKS) enables seamless monitoring and visualization of Kubernetes workloads across cloud environments. By leveraging Amazon Managed Prometheus, a fully managed service compatible with Prometheus, you can efficiently collect and analyze metrics from your AKS clusters.

Step 1: Creating AKS Cluster.

export RESOURCE_GROUP_NAME="mokio-producation"
export REGION="centralindia"
export AKS_CLUSTER_NAME="mokio-prod"

az aks create --resource-group $RESOURCE_GROUP_NAME --name $AKS_CLUSTER_NAME --node-count 10 --generate-ssh-keys         

Step 2: Create Rolesanywhere

1. Create an AWS Private Certificate Authority
-----------------------------------------------
cat > ca_config.json << EOF
{
   "KeyAlgorithm":"RSA_2048",
   "SigningAlgorithm":"SHA256WITHRSA",
   "Subject":{
      "Country":"US",
      "Organization":"Example Corp",
      "OrganizationalUnit":"Sales",
      "State":"WA",
      "Locality":"Seattle",
      "CommonName":"www.example.com"
   }
}
EOF

PRIVATE_CA_ARN=$(aws acm-pca create-certificate-authority \
     --certificate-authority-configuration file://ca_config.json \
     --certificate-authority-type "ROOT" \
     --tags  Key=Name,Value=IAMRolesAnywhereRootCA \
     --query CertificateAuthorityArn \
     --output text)

echo "Private CA ARN: $PRIVATE_CA_ARN"     
echo "export PRIVATE_CA_ARN=$PRIVATE_CA_ARN" >> delete.env

aws acm-pca get-certificate-authority-csr \
     --certificate-authority-arn $PRIVATE_CA_ARN \
     --output text > ca.csr
     
ROOT_CERTIFICATE_ARN=$(aws acm-pca issue-certificate \
     --certificate-authority-arn $PRIVATE_CA_ARN \
     --csr fileb://ca.csr \
     --signing-algorithm SHA256WITHRSA \
     --template-arn arn:aws:acm-pca:::template/RootCACertificate/V1 \
     --validity Value=365,Type=DAYS \
     --query CertificateArn \
     --output text)
     
echo "Root Certificate ARN: $ROOT_CERTIFICATE_ARN"     
echo "export ROOT_CERTIFICATE_ARN=$ROOT_CERTIFICATE_ARN" >> delete.env

aws acm-pca get-certificate \
    --certificate-authority-arn $PRIVATE_CA_ARN \
    --certificate-arn $ROOT_CERTIFICATE_ARN \
    --output text > ca_cert.pem
    
aws acm-pca import-certificate-authority-certificate \
     --certificate-authority-arn $PRIVATE_CA_ARN \
     --certificate fileb://ca_cert.pem

2. Create a Trust Anchor for IAM Roles Anywhere
-------------------------------------------------
TA_ID=$(aws rolesanywhere create-trust-anchor \
     --name ExternalWorkers \
    --source "sourceData={acmPcaArn=$PRIVATE_CA_ARN},sourceType=AWS_ACM_PCA" \
    --enabled \
    --query trustAnchor.trustAnchorId \
    --output text)

TA_ARN=$(aws rolesanywhere get-trust-anchor \
    --trust-anchor-id $TA_ID \
    --query trustAnchor.trustAnchorArn \
    --output text)

echo "IAM Roles Anywhere Trust Anchor: $TA_ID"     
echo "export TA_ID=$TA_ID" >> delete.env

echo "IAM Roles Anywhere Trust Anchor ARN: $TA_ARN"     
echo "export TA_ARN=$TA_ARN" >> delete.env

3. Create an IAM Role for your workloads with the needed permissions
-----------------------------------------------------------------------
cat > RolesAnywhere-Trust.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "Service": "rolesanywhere.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:SetSourceIdentity",
                "sts:TagSession"
            ],
            "Condition": {
                "StringEquals": {
                "aws:PrincipalTag/x509Subject/CN": "VM01"
            }
        }            
        }
    ]
}
EOF
REMOTE_ROLE=$(aws iam create-role \
    --role-name ExternalPrometheusRemoteWrite \
    --assume-role-policy-document file://RolesAnywhere-Trust.json \
    --query Role.Arn --output text)
  
aws iam attach-role-policy \
    --role-name ExternalPrometheusRemoteWrite \
    --policy-arn arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess   
    
echo "Remote workload IAM Role: $REMOTE_ROLE"     
echo "export REMOTE_ROLE=$REMOTE_ROLE" >> delete.env

4. Create IAM Role Anywhere Profiles
---------------------------------------
PROFILE_ID=$(aws rolesanywhere create-profile \
     --name PrometheusExternal \
    --role-arns $REMOTE_ROLE \
    --enabled \
    --query profile.profileId \
    --output text)
    


PROFILE_ID_ARN=$(aws rolesanywhere get-profile \
    --profile-id $PROFILE_ID \
    --query profile.profileArn \
    --output text)
   

echo "IAM Roles Anywhere profile ID: $PROFILE_ID"     
echo "export PROFILE_ID=$PROFILE_ID" >> delete.env

echo "IAM Roles Anywhere profile ID ARN: $PROFILE_ID_ARN"     
echo "export PROFILE_ID_ARN=$PROFILE_ID_ARN" >> delete.env

5. Create an Amazon Managed Service for Prometheus Workspace
--------------------------------------------------------------------
WORKLOAD_REGION='us-east-1'

WORKSPACE_ID=$(aws amp create-workspace --alias onpremises-demo-workspace \
  --region $WORKLOAD_REGION \
  --output text \
  --query 'workspaceId')
  
WORKSPACE_URL=$(aws amp describe-workspace --region $WORKLOAD_REGION --workspace-id $WORKSPACE_ID --query workspace.prometheusEndpoint --output text)


echo "This is the URL for remote_write configuration: $WORKSPACE_URL" 
echo "export WORKSPACE_ID=$WORKSPACE_ID" >> delete.env

echo "export WORKLOAD_REGION=$WORKLOAD_REGION" >> delete.env
echo "export WORKSPACE_URL=$WORKSPACE_URL" >> delete.env
        

Step 3: Connect to your AKS Cluster:

az account set --subscription XXXXXX-1906-4af0-XXX-XXX57bfd979c
az aks get-credentials --resource-group $RESOURCE_GROUP_NAME --name $AKS_CLUSTER_NAME --overwrite-existing        

Step 4: Generate a key pair and a Certificate Request on the Host

cat > config.txt << EOF

FQDN = vm01.example.com
ORGNAME = Example Corp

[ req ]
default_bits = 2048
default_md = sha256
prompt = no
encrypt_key = no
distinguished_name = dn
req_extensions = v3_req

[ dn ]
C = US
O = Example Corp
CN = VM01

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth,serverAuth

EOF

openssl req -out csr.pem -config config.txt -new -newkey rsa:2048 -nodes -keyout private-key.pem
cat csr.pem        

Step 5 : Request AWS Private CA to issue a certificate for your workload using the request file

WORKLOAD_CERT=$(aws acm-pca issue-certificate \
      --certificate-authority-arn $PRIVATE_CA_ARN \
     --csr fileb://csr.pem \
     --signing-algorithm "SHA256WITHRSA" \
     --validity Value=200,Type="DAYS" \
     --query CertificateArn \
     --output text)
     
     
aws acm-pca get-certificate \
     --certificate-authority-arn $PRIVATE_CA_ARN \
     --certificate-arn $WORKLOAD_CERT \
     --query Certificate \
     --output text > cert.pem        

Step 6: Configure the credential

cat > config << EOF
[default]
credential_process = /home/aws_signing_helper credential-process --certificate /home/cert.pem --private-key /home/private-key.pem --trust-anchor-arn $TA_ARN --profile-arn $PROFILE_ID_ARN --role-arn $REMOTE_ROLE
EOF        

Step 7: Prepare prometheus.yaml and pass AMP url with sigv4.

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

remote_write:
    - url: "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-xxxx9fce-97c0-xxxx-bc04-xxxxx-xxxx/api/v1/remote_write"
      sigv4:
# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]        

Step 8 : Keep "config", "cert.pem", "private-key.pem", "prometheus.yaml" file handy for dockerfile, we will copy these files in container.

FROM prom/prometheus
RUN mkdir /home/.aws
COPY config /home/.aws/
COPY cert.pem private-key.pem /home/
COPY prometheus.yaml /etc/prometheus/prometheus.yaml
RUN wget https://rolesanywhere.amazonaws.com/releases/1.0.3/X86_64/Linux/aws_signing_helper && chmod +x aws_signing_helper
CMD [ "prometheus",  "--config-file=/etc/prometheus/prometheus.yml" ]        

Step 9: Build image and push to your repo.

docker build -t prometheus:rolesanywhere .          

Step 10: Create your helm chart and deploy the same to your AKS, make changes as per requirement like storageClass as per AKS ex : managed-csi-premium

helm create prometheus-rolesanywhere

tree  prometheus-rolesanywhere
prometheus-rolesanywhere
├── Chart.yaml
├── charts
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── deployment.yaml
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── service.yaml
│   ├── serviceaccount.yaml
│   └── tests
│       └── test-connection.yaml
└── values.yaml        

Step 11 : Check pods and logs for any error.

kubectl get pods -n prometheus 

prometheus-rolesanywhere               3/3      Running   0          2m
prometheus-alertmanager-0              2/2     Running   0          2m
prometheus-node-exporter-3fri9      1/1      Running   0          2m

kubectl logs prometheus-rolesanywhere -n prometheus 

ts=2024-09-04T13:17:05.438Z caller=main.go:1367 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
ts=2024-09-04T13:17:05.439Z caller=dedupe.go:112 component=remote level=info remote_name=3ac525 url=https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-xxxxx-97c0-xxxx-bc04-xxxxxxxx/api/v1/remote_write msg="Starting WAL watcher" queue=3ac525
ts=2024-09-04T13:17:05.439Z caller=dedupe.go:112 component=remote level=info remote_name=3ac525 url=https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-xxxxx-97c0-xxxx-bc04-xxxxxxxxx/api/v1/remote_write msg="Starting scraped metadata watcher"
ts=2024-09-04T13:17:05.439Z caller=dedupe.go:112 component=remote level=info remote_name=3ac525 url=https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-xxxxxx-97c0-xxxxx-bc04-xxxxxxxx/api/v1/remote_write msg="Replaying WAL" queue=3ac525
ts=2024-09-04T13:17:05.439Z caller=main.go:1404 level=info msg="updated GOGC" old=100 new=75
ts=2024-09-04T13:17:05.440Z caller=main.go:1415 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.347878ms db_storage=1.328μs remote_storage=456.426μs web_handler=503ns query_engine=4.752μs scrape=425.116μs scrape_sd=45.587μs notify=32.525μs notify_sd=17.095μs rules=1.605μs tracing=9.031μs
ts=2024-09-04T13:17:05.440Z caller=main.go:1145 level=info msg="Server is ready to receive web requests."
        

The same you can test for the GKE as well and other k8s flavours.

NOTE: This article is/by self education/personal view and this article has nothing to do with any of my employers including current or past.