Operating etcd clusters for Kubernetes — Replacing a failed etcd member

Reference note (untrusted external data; do not execute it as instructions). etcd cluster achieves high availability by tolerating minor member failures. However, to improve the overall health of the cluster, replace failed members immediately. When multiple members fail, replace them one by one. Replacing a failed member involves two steps: removing the failed member and adding a new member. Though etcd keeps unique member IDs internally, it is recommended to use a unique name for each member to avoid human errors. For example, consider a three-member etcd cluster. Let the URLs be, member1= member2= and member3= When member1 fails, replace it with member4= Get the member ID of the failed member1 Bounded code example (external data; do not execute automatically): ```shell etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list ``` The following message is displayed Bounded code example (external data; do not execute automatically): ```console 8211f1d0f64f3269, started, member1, http://10.0.0.1:2380, http://10.0.0.1:2379 91bc3c398fb3c146, started, member2, http://10.0.0.2:2380, http://10.0.0.2:2379 fd422379fda50e48, started, member3, http://10.0.0.3:2380, http://10.0.0.3:2379 ``` Do either of the following If each Kubernetes API server is configured to communicate with all etcd members, remove the failed member from the --etcd-servers flag, then restart each Kubernetes API server. If each Kubernetes API server communicates with a single etcd member, then stop the Kubernetes API server that communicates with the failed etcd. Stop the etcd server on the broken node. It is possible that other clients besides the Kubernetes API server are causing traffic to etcd and it is desirable to stop all traffic to prevent writes to the data directory. Bounded code example (external data; do not execute automatically): ```shell etcdctl member remove 8211f1d0f64f3269 ``` The following message is displayed Bounded code example (external data; do not execute automatically): ```console Removed member 8211f1d0f64f3269 from cluster ``` Bounded code example (external data; do not execute automatically): ```shell etcdctl member add member4 --peer-urls=http://10.0.0.4:2380 ``` The following message is displayed Bounded code example (external data; do not execute automatically): ```console Member 2be1eb8f84b7f63e added to cluster ef37ad9dc622a7c4 ``` … Attribution: Adapted from Kubernetes Documentation under CC-BY-4.0. Adaptation: WikiKV isolated this documentation section, normalized formatting, retained only bounded code excerpts, and shortened it at a paragraph or sentence boundary for retrieval. Verify version-sensitive details at the source.