Why Is Kubelet Reporting Node NotReady

Why Is Kubelet Reporting Node NotReady
SHARE

A Kubernetes node suddenly changes from:

Ready
to:
NotReady

At first, everything looks normal. The node is still running. The server is reachable over SSH. Some containers may even still be serving traffic.

But Kubernetes reports:

kubectl get nodes
NAME       STATUS     ROLES    AGE
worker-01  NotReady   <none>   42d

Now workloads may stop being scheduled onto the node, pods can become unhealthy, and the cluster may start moving workloads elsewhere.

So what actually happened?

The answer is usually somewhere between the kubelet, operating system, container runtime, networking, and node resources.

In this article, we'll investigate seven common reasons a Kubernetes node becomes NotReady:

  1. DiskPressure
  2. PIDPressure
  3. MemoryPressure
  4. NetworkUnavailable
  5. CNI failure
  6. Container runtime failure
  7. Kubelet certificate expiration

First: What Does NotReady Actually Mean?

A node doesn't become NotReady simply because the machine is down. Kubernetes continuously receives node status and heartbeats from the kubelet.

A simplified flow looks like this:

Node
 |
 +-- kubelet
 |     |
 |     +-- Node status
 |     +-- Heartbeats
 |     +-- Container health
 |     +-- Resource conditions
 |
 +-- Container Runtime
 |
 +-- CNI
 |
 +-- Linux / OS

The control plane uses this information to determine whether the node is healthy enough to run workloads.

Check the node: kubectl get nodes

Then: kubectl describe node <node-name>

Pay particular attention to:

Conditions:
  Ready
  MemoryPressure
  DiskPressure
  PIDPressure
  NetworkUnavailable

These conditions are usually your first major clue.


1. DiskPressure

One of the most common causes of node problems is simply:

The node is running out of disk space or available inodes.

Check: df -h

Then check inode usage: df -i

You might discover:

Filesystem      Size  Used Avail Use%
/dev/sda1       100G   98G    2G  98%

 

Kubernetes and the container runtime need disk space for things such as:

  • container images
  • container writable layers
  • logs
  • emptyDir
  • kubelet data
  • container runtime metadata

A node under serious disk pressure can report: DiskPressure=True


Check the node condition

kubectl describe node <node-name>

Look for: DiskPressure True

Also inspect kubelet logs: journalctl -u kubelet --since "30 min ago"

Search for messages related to:

disk pressure
eviction
image garbage collection
filesystem
no space left

What to investigate

Find what's consuming disk: du -sh /var/log/*

Check container runtime storage and kubelet directories as well. Don't blindly delete files from runtime directories. Instead, identify the source of the growth. A common culprit is excessive container logging.


2. PIDPressure

Linux processes consume process IDs. A node running too many processes can eventually hit its PID limit.

Kubernetes represents this condition as: PIDPressure=True

Check the node: kubectl describe node <node-name>

Then inspect processes on the host: ps aux | wc -l

You can also inspect PID-related limits: cat /proc/sys/kernel/pid_max


What causes PID exhaustion?

Common causes include:

  • applications spawning excessive processes

  • broken workloads creating process storms

  • zombie processes

  • containers with unexpectedly high process counts

  • badly behaved agents or monitoring software

For example:

Application
    |
    +-- process
    +-- process
    +-- process
    +-- process
    +-- thousands more

Eventually the node can run out of available PIDs.


Troubleshooting

Look for processes consuming large numbers of PIDs. Inspect the busiest containers and workloads.

Then check kubelet logs:

journalctl -u kubelet --since "30 min ago"

If PID pressure is caused by a specific workload, fixing the application is more important than simply restarting kubelet.


3. MemoryPressure

Memory pressure occurs when the node doesn't have enough available memory.

Check: free -h

And: kubectl describe node <node-name>

You may see: MemoryPressure=True


Why does this happen?

Possible causes include:

  • containers exceeding memory limits

  • workloads without appropriate limits

  • memory leaks

  • too many pods on the node

  • system processes consuming memory

  • monitoring or security agents using unexpected amounts of RAM

Check memory-consuming processes: ps aux --sort=-%mem | head

For Kubernetes workloads: kubectl top node

and: kubectl top pods -A

if Metrics Server or another metrics provider is available.


Watch for OOM events

Check kernel logs: dmesg | grep -i oom

You may find evidence that the Linux kernel's out-of-memory mechanism has killed processes. Kubernetes may also start evicting pods when node memory crosses configured thresholds.

The important distinction is:

A NotReady node caused by memory pressure is usually a resource-capacity problem, not a kubelet bug.


4. NetworkUnavailable

A node can also report a networking-related condition.

Check: kubectl describe node <node-name>

Look for: NetworkUnavailable

This condition is particularly relevant during node startup or when the networking layer hasn't been successfully configured. The exact behavior depends on your Kubernetes networking setup and whether a cloud controller manager or CNI plugin manages the condition.


What can cause it?

Potential causes include:

  • CNI initialization failure

  • broken node networking

  • cloud networking problems

  • incorrect routes

  • missing interfaces

  • firewall/security-group changes

Start by checking the node's network interfaces: ip addr

Then routes: ip route

And basic connectivity: ping <gateway>

For Kubernetes networking, also inspect the CNI pods.


5. CNI Crash or Failure

Sometimes the node itself is healthy, but Kubernetes networking isn't. This can happen when the CNI plugin crashes or becomes unable to configure pod networking.

Depending on your cluster, the CNI might be something such as:

  • Cilium
  • Calico
  • Flannel
  • another networking implementation

Check networking pods: kubectl get pods -A

Look for CNI-related pods that are:

CrashLoopBackOff
Error
Pending
NotReady

Then inspect them: kubectl logs -n <namespace> <cni-pod>


Check the node

On the affected node, inspect: ip link

and: ip route

Also inspect CNI configuration: ls -la /etc/cni/net.d/

A broken CNI can lead to symptoms such as:

Node NotReady
Pod networking broken
New pods stuck
DNS failures
Service connectivity failures

Important distinction

Don't assume:

Node NotReady = kubelet failure.

A networking failure can make the node appear unhealthy even when the kubelet process itself is running. Always inspect the CNI layer separately.


6. Container Runtime Failure

Kubelet doesn't directly run containers. It communicates with a container runtime through the CRI.

A simplified architecture is:

Kubelet
   |
   | CRI
   v
Container Runtime
   |
   v
Containers

If the runtime becomes unavailable, kubelet may be unable to manage pods correctly.

Check the runtime service.

For containerd:

systemctl status containerd

Check its logs:

journalctl -u containerd --since "30 min ago"

You can also check whether the runtime is responding:

crictl info

if crictl is installed and configured.


Common runtime failures

Possible causes include:

  • runtime crash

  • corrupted runtime state

  • disk exhaustion

  • runtime upgrade problems

  • image filesystem issues

  • runtime socket problems

  • resource exhaustion

You may see kubelet errors such as:

failed to get runtime status
container runtime is down
failed to create pod sandbox

The key question becomes:

Is kubelet unhealthy, or is kubelet healthy but unable to communicate with the runtime?

That distinction saves a lot of debugging time.


7. Kubelet Certificate Expiration

This one can be particularly confusing. 

The node may look perfectly healthy at the operating-system level:

CPU: OK
Memory: OK
Disk: OK
Network: OK
Runtime: OK

But the kubelet may have a problem authenticating with the Kubernetes API server. Kubelet uses certificates for secure communication with the control plane. If the relevant certificate expires and renewal isn't working correctly, node heartbeats and API communication can fail.

The result may eventually look like: Node NotReady


Check kubelet logs

Start with: journalctl -u kubelet --since "1 hour ago"

Look for errors involving:

certificate
x509
expired
authentication
TLS

For example: x509: certificate has expired is a strong clue.


Why certificate problems are easy to miss

Certificate expiration often doesn't look like a resource problem.

You can have:

DiskPressure = False
MemoryPressure = False
PIDPressure = False

and still have: Ready = False

because the kubelet cannot successfully communicate with the API server. Always check kubelet logs when the usual node conditions look normal.


8. A Simple Production Troubleshooting Workflow

When a node suddenly becomes NotReady, don't start by rebooting it.

Follow the request path.

Step 1 — Check the node

kubectl get node <node-name>

Then:

kubectl describe node <node-name>

Check:

Ready
MemoryPressure
DiskPressure
PIDPressure
NetworkUnavailable

Step 2 — Check kubelet

SSH into the node: systemctl status kubelet

Then: journalctl -u kubelet --since "30 min ago"


Step 3 — Check resources

free -h
df -h
df -i

Look for:

Memory exhaustion
Disk exhaustion
Inode exhaustion

Step 4 — Check the runtime

For containerd: systemctl status containerd

Then: crictl info


Step 5 — Check networking

ip addr
ip route

Then check your CNI pods: kubectl get pods -A


Step 6 — Check certificates

Inspect kubelet logs for:

x509
certificate
expired
TLS
authentication

9. The Fast Diagnosis Matrix

SymptomFirst thing to check
DiskPressure=Truedf -h, df -i
MemoryPressure=Truefree -h, kubectl top node
PIDPressure=TrueProcess/PID count
NetworkUnavailable=TrueCNI and node networking
CNI pods crashingCNI logs/configuration
Runtime unavailablecontainerd / CRI
Certificate errorskubelet logs and certificates
Ready=False with normal resourceskubelet logs + API connectivity

10. The Production Incident Pattern

Imagine this happens at 2 AM.

Monitoring reports: Node worker-07: NotReady

You run: kubectl describe node worker-07

and see:

MemoryPressure=False
DiskPressure=False
PIDPressure=False
NetworkUnavailable=False
Ready=False

Don't immediately reboot the server. SSH into it.

Check: systemctl status kubelet

Then: journalctl -u kubelet --since "30 min ago"

You discover: x509: certificate has expired

Now the incident makes sense. The server wasn't overloaded. The container runtime wasn't necessarily broken. The kubelet simply couldn't establish valid authenticated communication with the API server.

That's why understanding the node condition + kubelet logs + underlying system is so important.


Final Takeaway

When Kubernetes reports: Node NotReady

the node isn't necessarily dead. The problem can come from several layers:

                    Node NotReady
                          |
       +------------------+------------------+
       |                  |                  |
       v                  v                  v
   Resources           Networking          Kubelet
       |                  |                  |
   DiskPressure       CNI failure       Runtime failure
   MemoryPressure     Network issue     Certificate
   PIDPressure

Start with: kubectl describe node <node-name>

Then move down the stack:

Node Conditions
      ↓
Kubelet
      ↓
Operating System
      ↓
Container Runtime
      ↓
CNI / Networking
      ↓
Certificates / API connectivity

The most important lesson is:

NotReady is a symptom, not a root cause.

Before restarting or replacing the node, find out why the kubelet stopped reporting the node as healthy.

A few minutes spent reading the node conditions and kubelet logs can turn a mysterious production incident into a very specific problem: disk, memory, PIDs, networking, CNI, runtime, or certificates.