Rancher RKE Resolving Cattle Cluster Agent DNS Issue with Host Alias

Resolving Cattle Cluster Agent DNS Issue with Host Alias

This document describes how to troubleshoot and fix a DNS resolution issue encountered by the Cattle Cluster Agent in a Kubernetes cluster.

The Problem

The Cattle Cluster Agent was unable to connect to the Rancher server (rancher.ipa.prodevans.com) due to a DNS resolution problem. The cluster DNS was not configured to resolve the private hostname, and the default behavior (dnsPolicy: ClusterFirst) relied on the cluster DNS.

Troubleshooting Steps

1. Identifying the Issue: We examined the logs of the Cattle Cluster Agent pod (kubectl logs -f cattle-cluster-agent-<pod_name> -n cattle-system). The logs revealed an error message indicating that rancher.ipa.prodevans.com could not be resolved.

Verifying DNS Resolution: We confirmed that the cluster DNS server was not resolving the private hostname.

Solutions Explored

There were several potential solutions to address this issue:

1. Fix Cluster DNS: The ideal solution would be to fix the cluster DNS configuration to resolve external domains, including the private hostname. This might involve adjusting Coredns configuration or the configuration of the cluster's DNS provider.
2. Set dnsPolicy to Default: This approach instructs the pod to use the host's nameserver for resolution. While it might resolve the immediate issue, it's not ideal because the pod wouldn't rely on the cluster DNS for other lookups.
3. Manual DNS Configuration: We could modify the deployment to mount a custom DNS resolver configuration or a host file entry within the Cattle Cluster Agent pod. However, this approach has limitations in terms of security and maintainability.

Implemented Solution

We opted for a solution that directly addresses the issue for the Cattle Cluster Agent while minimizing changes:

1. Patch Deployment: We used kubectl patch to modify the deployment manifest and add a host alias for rancher.ipa.prodevans.com pointing to the correct IP address (192.168.1.24) within the pod spec.

kubectl -n cattle-system patch deployments cattle-cluster-agent --patch '{"spec": {"template": {"spec": {"hostAliases": [{"hostnames":["rancher.ipa.prodevans.com"],"ip": "192.168.1.24"}]}}}}}'        

1. Verify Resolution: We checked the logs again (kubectl logs -f cattle-cluster-agent-<new_pod_name> -n cattle-system) and confirmed that the pod successfully resolved rancher.ipa.prodevans.com using the provided IP address.

Conclusion

By patching the deployment with a host alias, we ensured the Cattle Cluster Agent could connect to the Rancher server and function properly. However, it's still recommended to investigate and fix the root cause of the DNS resolution issue within the cluster for a more robust and secure long-term solution.