How Kubernetes authenticate internal access?

When you access the Kubernetes API server, you authenticate as a regular user. But what happens when Pods start making requests? How does Kubernetes verify the identity of these internal actors?

The answer is ServiceAccounts. In Kubernetes, both application Pods and system components authenticate themselves using the credentials associated with a specific ServiceAccount.

This post delves into how ServiceAccounts work, highlights the differences between the legacy strategy and the current approach, and guides you through creating, inspecting, and debugging these credentials.

Why is internal access authentication crucial?

You may wonder why pods inside the cluster need to connect to the Kubernetes API. There are some common scenarios:

• Controller pods need access to track logs and application status.
• CI/CD runners or auto scalers create, update, or delete other resources
• Providing access to Secrets or other sensitive information in the cluster.
• Integration with external services that trust the cluster authentication.

What is a Service Account in Kubernetes?

A ServiceAccount is a type of non-human account that, in Kubernetes, provides a distinct identity in a Kubernetes cluster.

A ServiceAccount provides an identity for processes that run in a Pod to authenticate to the cluster's API server.

• Identity and Credentials: ServiceAccounts use signed JSON Web Tokens (JWTs) to authenticate to the Kubernetes API server, and to any other system where a trust relationship exists. The client includes an Authorization: Bearer <token> header with the HTTP request.
• Creation: Kubernetes automatically creates a ServiceAccount object named default for every namespace. You can create additional ServiceAccounts for different workloads, each with a different set of permissions. To assign a ServiceAccount to a Pod, you set the spec.serviceAccountName field in the Pod specification
• Storage: By default, the Kubernetes control plane attaches a projected volume to Pods, and this volume includes a token file (/var/run/secrets/kubernetes.io/serviceaccount/token) for Kubernetes API access.
• Issuance and Validity: Depending on how the token was issued (either time-limited using a TokenRequest or using a legacy mechanism with a Secret). Short-lived tokens also support automatic rotation and can be scoped to specific audiences.

Legacy Service Account Tokens

Traditionally, Kubernetes created and stored tokens as Secrets in the cluster.

• Secret-based: The cluster’s control plane creates a Secret containing the corresponding service account token.
• Long-lived tokens: These tokens do not automatically rotate or expire, depending on Secret deletion. That increases the risk if a token is leaked or compromised.

Bound Service Account Tokens

To address the above concerns, Kubernetes introduced Bound service account tokens.

• API-based: API credentials are obtained directly using the TokenRequest API.
• Short-lived and automatically rotated: Tokens have a configurable time-to-live (TTL) and are refreshed automatically. A token expires either when the pod is deleted or after a defined lifespan (default: 1 hour).
• Fine-Grained Audiences: You can specify the audience for the token (e.g., the Kubernetes API server itself, or specific external services that trust the cluster CA).

Bound tokens became more stable and are the recommended approach in Kubernetes 1.22 and later.

Hands-On: Creating and Inspecting Tokens

In this section, let’s walk through the commands you can use to create, inspect, and debug Service Account tokens in a Kubernetes cluster.

1. Create a service account

Legacy service account - the K8s automatically create a Secret and add its reference to the service account.

apiVersion: v1
kind: ServiceAccount
metadata:  
  name: controller-svc-legacy
  namespace: controller-role
secrets:
- name: controller-svc-legacy-token-vj74z        

Bound service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: controller-svc
  namespace: controller-ns        

2. Create and assign a role to the service account

To grant permissions to a service account via RBAC (Role-Based Access Control), we create a role and assign it to the service account.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: controller-role-binding
  namespace: controller-ns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: controller-role
subjects:
- kind: ServiceAccount
  name: controller-svc
  namespace: controller-ns        

3. Verify and inspect the token

We read the token from the file in the pod and then use TokenReview to verify and inspect it. We can see the creation and last access time of the token.

kubectl exec -it controller-pod -- /bin/sh
# Inspect the token file
cat /var/run/secrets/kubernetes.io/serviceaccount/token
# Check filesystem timestamps (access, modify, change time)
stat /var/run/secrets/kubernetes.io/serviceaccount/token

Access: 2025-02-08 10:10:19.302900356 +0000
Modify: 2025-01-21 09:58:07.880324983 +0000
Change: 2025-01-21 09:58:07.880324983 +0000        

Create a TokenReview resource to ask the API server to validate a token. This is especially useful if you suspect issues with authentication or need to confirm which ServiceAccount identity the token represents.

# tokenreview.yaml
apiVersion: authentication.k8s.io/v1
kind: TokenReview
spec:
  token: <token from cat command>        

kubectl create -o yaml -f tokenreview.yaml        

The API server will respond with details about the token:

• authenticated: true confirming the token is valid.
• user.username reflecting the bound ServiceAccount (system:serviceaccount:controller-ns:controller-svc)
• pod-name linking it to the specific pod using the token.

apiVersion: authentication.k8s.io/v1
kind: TokenReview
metadata:
  creationTimestamp: null
spec:
  token: <>
status:
  audiences:
  - https://kubernetes.default.svc
  authenticated: true
  user:
    extra:
      authentication.kubernetes.io/credential-id:
      - JTI=d6ccd5f8-8b40-420a-a71e-ac8983d3tr0a
      authentication.kubernetes.io/node-name:
      - controller-node
      authentication.kubernetes.io/node-uid:
      - d1ab549f-4a55-4e64-902c-12cf85d2b2cd
      authentication.kubernetes.io/pod-name:
      - controller-5c8b949fdc-l2t5l
      authentication.kubernetes.io/pod-uid:
      - 23540e77-4f7a-45ec-8ecc-b85b4f40bfba
    groups:
    - system:serviceaccounts
    - system:serviceaccounts:controller-svc
    - system:authenticated
    uid: a9962ee2-5d6d-4086-a76b-af5c89c6d829
    username: system:serviceaccount:controller-ns:controller-svc        

Best Practices and Tips

• Use Bound Tokens When Possible: If you’re on Kubernetes 1.22 or newer, enable and use bound tokens for better security. They expire and automatically rotate, reducing long-term exposure of credentials.
• Rotate and Revoke Legacy Tokens: If you have older tokens stored in Secrets, consider rotating them out. At a minimum, revoke tokens if they’re no longer needed.
• Limit RBAC Permissions: Only grant the Role and ClusterRole permissions your workload needs. Resist the temptation to give your ServiceAccount cluster-admin rights.
• Use Namespaces for Isolation: Running different workloads in separate namespaces with distinct ServiceAccounts helps maintain security boundaries.

ServiceAccounts are foundational to Kubernetes security, allowing pods to authenticate to the API server. Bound tokens bring short-lived, auto-rotated credentials that address many of the security weaknesses found in legacy (long-lived) tokens.

References:

• https://kubernetes.io/docs/reference/access-authn-authz/authentication/
• https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/
• https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
• https://kubernetes.io/docs/concepts/security/service-accounts/