XFS AWS EKS K8s no space left on device: unknown

xfs-aws-eks-k8s-no-space-left-on-device-unknown.md

If you're using AWS EKS and for some reason you get:

no space left on device: unknown

Check if:

• you're using XFS
• your disk usage is < 85% (At 85%, k8s triggers GC)
• you're pulling large 2GB docker images OR docker images with node_modules with hundreds of thousands of files

Then consider inode contiguous fragmentation issue:

https://support.microfocus.com/kb/doc.php?id=7014318

Userland (hacky) Solution:

Tune this to garbage collect more a: https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/

• imageGCHighThresholdPercent
• imageGCLowThresholdPercent

Kernel (proper) Solution:

• man mkfs.xfs
  • inode-options
    • sparse (enable? is this a performance hit?)
• ikeep vs nokeep: (Some other configuration about not unallocating inodes after file deletion.. some reusage thing? Maybe not..)

For diagnostics look at the before and after from a crictl rmi --prune run..:

xfs_db -r -c 'freesp -s' ${mount_name}
crictl rmi --prune
xfs_db -r -c 'freesp -s' ${mount_name}

And see how the contiguous inode histogram changes