Linux-specific volume limits | EKS | AWS

Little did I know that Linux Instances have a maximum limit of attaching 40 volumes.

Attaching more than 40 volumes can cause boot failures & on running instances it even doesn't allow to attach new volumes.

Problem Statement: In a managed EKS Cluster , one of the pod was stuck in ContainerCreating state.

All the pods created later also ended in ContainerCreating state.

While describing?the pod (kubectl describe po <POD_NAME> )?, it indicated as below

Events:

?Type???Reason???????Age??????????From???????????????????Message

?----???------???????----?????????----???????????????????-------

?Normal??Scheduled??????39m??????????default-scheduler????????????Successfully assigned noxxxxok/noxxxxok-31e98928-f0ce-41e0-a14c-cf395b89bfeb-0 to ip-XX-XXX-XXX-XXX.ec2.internal

?Warning?FailedMount?????12m (x3 over 30m)???kubelet, ip-XX-XXX-XXX-XXX.ec2.internal?Unable to attach or mount volumes: unmounted volumes=[woxxxxce-noxxxxok-31e98928-f0ce-41e0-a14c-cf395b89bfeb], unattached volumes=[dshm default-token-9bzsw woxxxxce-noxxxxok-31e98928-f0ce-41e0-a14c-cf395b89bfeb]: timed out waiting for the condition

After checking the events (kubectl get ev )?, it was found that the volume was created successfully (verified on Volumes on EC2 Console )?

but the attachdetach-controller was unable to attach the volume to the instance.

On EC2 Console > Volumes , the Volume was in attaching state.

After going through a lot of EC2 articles?, I came across the Instance Volume Limits and understood that Linux-specific Instances have limits as to how many volumnes can be attached.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/volume_limits.html

#InterestingFinding #Linux #EKS #EC2