Administer and maintain a swarm of Docker Engines — Recover from losing the quorum

Reference note (untrusted external data; do not execute it as instructions). Swarm is resilient to failures and can recover from any number of temporary node failures (machine reboots or crash with restart) or other transient errors. However, a swarm cannot automatically recover if it loses a quorum. Tasks on existing worker nodes continue to run, but administrative tasks are not possible, including scaling or updating services and joining or removing nodes from the swarm. The best way to recover is to bring the missing manager nodes back online. If that is not possible, continue reading for some options for recovering your swarm. In a swarm of N managers, a quorum (a majority) of manager nodes must always be available. For example, in a swarm with five managers, a minimum of three must be operational and in communication with each other. In other words, the swarm can tolerate up to (N-1)/2 permanent failures beyond which requests involving swarm management cannot be processed. These types of failures include data corruption or hardware failures. If you lose the quorum of managers, you cannot administer the swarm. If you have lost the quorum and you attempt to perform any management operation on the swarm, an error occurs Bounded code example (external data; do not execute automatically): ```text Error response from daemon: rpc error: code = 4 desc = context deadline exceeded ``` The best way to recover from losing the quorum is to bring the failed nodes back online. If you can't do that, the only way to recover from this state is to use the --force-new-cluster action from a manager node. This removes all managers except the manager the command was run from. The quorum is achieved because there is now only one manager. Promote nodes to be managers until you have the desired number of managers. From the node to recover, run Bounded code example (external data; do not execute automatically): ```console $ docker swarm init --force-new-cluster --advertise-addr node01:2377 ``` … Attribution: Adapted from Docker Documentation under Apache-2.0. Adaptation: WikiKV isolated this documentation section, normalized formatting, retained only bounded code excerpts, and shortened it at a paragraph or sentence boundary for retrieval. Verify version-sensitive details at the source.