Diagnosing and Fixing Pod Networking

Diagnosing and Fixing Pod Networking A workflow diagram generated by Archify. 01 / Diagnosis 02 / Remediation Pod fault seen · ContainerCreating · Diagnosis Pod fault seen ContainerCreating Step 1 inspect · CNI logs · Diagnosis Step 1 inspect CNI logs Step 2 test path · ping · nslookup · Diagnosis Step 2 test path ping · nslookup Cause class · IP · policy · DNS · Diagnosis Cause class IP · policy · DNS Resize IP pool · WARM_IP_TARGET · Remediation Resize IP pool WARM_IP_TARGET Restart config · aws-node · CoreDNS · Remediation Restart config aws-node · CoreDNS collect starts ok combine IP shortage applied policy · DNS Legend Diagnosis · Fix Cause classification Symptom

Step 1 — inspect state

  • • kubectl get pods -o wide shows which node each pod landed on and with what IP
  • • kubectl describe pod and describe node reveal events and allocatable IPs
  • • Read the CNI logs with kubectl logs -n kube-system -l k8s-app=aws-node

Step 2 — test the path

  • • Run ping, curl, and telnet via kubectl exec to narrow down how far traffic gets
  • • nslookup kubernetes.default isolates whether DNS resolution itself works
  • • kubectl get networkpolicy and describe networkpolicy expose blocking rules

Step 3 — remediate

  • • IP shortage: raise WARM_IP_TARGET and MINIMUM_IP_TARGET on aws-node
  • • Restart with kubectl rollout restart daemonset aws-node and deployment coredns
  • • Restarting a node is the last resort; drain first and uncordon afterwards