How do Kubernetes control-plane components coordinate during a node failure? Explain the sequence involving the API server, etcd, scheduler, controller manager, and kubelet in a concise technical answer.

Shared on September 17, 2026 by Ayush

A Kubernetes node failure is handled as a reconciliation sequence rather than a direct component-to-component command chain:

  1. The kubelet reports health. Each node’s kubelet periodically updates the node’s status and renews a Lease in the kube-node-lease namespace; the default Lease update interval is 10 seconds. If the kubelet or node fails, those heartbeats stop. [citation_1utuxd]

  2. The API server exposes and persists the state. The kubelet writes its updates to the API server. The API server is the control plane’s coordination point: it validates requests, reads and writes cluster state, and persists that state in etcd. Other components observe changes through the API server rather than calling one another directly. [citation_1utuw6]

  3. The controller manager detects the failure. The node controller, running inside kube-controller-manager, watches node heartbeats. After the configured grace period it changes the node’s Ready condition to Unknown/NotReady and applies an unreachable or not-ready taint. The taint prevents new scheduling there; after the pod-eviction timeout—five minutes by default in the documented behaviour—the node controller initiates eviction of affected Pods. [citation_1utuxd]

  4. Controllers restore the desired workload. For Pods managed by a Deployment, ReplicaSet, or similar controller, eviction or loss of the old Pod creates a gap between desired and current state. The relevant controller writes replacement Pod objects through the API server. The scheduler watches for Pods without spec.nodeName, selects suitable healthy nodes using resource and placement constraints, and writes the chosen nodeName back through the API server. [citation_1utuw1]

  5. The new kubelet starts the replacement. The kubelet on the selected node watches for Pods assigned to it, asks the container runtime to create them, and reports their status back through the API server, which persists the updates in etcd. The failed node’s kubelet cannot acknowledge deletion while it is unreachable, so a network partition can leave old Pod objects in Terminating or Unknown until communication is restored or an operator intervenes. [citation_1utuw4][citation_1utuxd]

Thus, kubelet supplies observations; API server provides the coordination interface; etcd records durable cluster state; controller manager detects drift and requests correction; scheduler chooses placement; and another kubelet makes the replacement real. Timing depends on heartbeat, grace-period, eviction, toleration, and workload-controller settings, so “rescheduled” does not necessarily mean instantaneous. [citation_1utuxd][citation_1utuw6]

Comments & Discussion