Fix it now
The cluster stopped hearing from a node and removed it from membership. The published message says exactly that, and adds the two possibilities: the cluster service on that node may have stopped, or the node lost communication with the other active nodes. Rule out a genuine outage first, then look at the heartbeat path.
Get-ClusterNetwork | Format-Table Name, State, Role, Address
Get-Cluster | Format-List *subnet*
Get-ClusterLog -Destination C:\temp\ -TimeSpan 15 -UseLocalTime
- On the evicted node, check the System log around the timestamp for a restart, a bugcheck or a storage stall. If the node really went down, stop here and investigate that instead.
- Confirm the cluster service traffic is permitted between every node: UDP 3343 for heartbeats, TCP 3343 for node join operations, TCP 135 for RPC, and ICMP for validation. A rule that opens only UDP passes the heartbeat test and still blocks a node rejoining.
- Compare the adapter settings across nodes with
Get-NetAdapterAdvancedPropertyand make them consistent. Update driver and firmware together. - Read the threshold values the second command returns before assuming them. They differ between Windows Server releases and somebody may have already tuned them.
Raising the thresholds makes evictions less frequent by making the cluster slower to react to a node that has genuinely died. That delay is the price of the tolerance, and it is not free.
If the node stays in membership through a full backup and migration window, stop here. If it does not, the next section covers what actually drops these packets.
Why it happens
Cluster nodes exchange small heartbeat packets over every network enabled for cluster communication, using UDP port 3343 and a virtual adapter the cluster creates for itself. The packets go out on a fixed cadence, and a node is considered lost once a configured number of consecutive beats fail to arrive. That is the entire mechanism. It is deliberately unforgiving, because the alternative is leaving a genuinely dead node in membership and letting two nodes think they own the same storage.
Microsoft’s published message for the eviction is careful about this, and worth reading rather than skimming: the cluster service on the node may have stopped, or the node may have lost communication with the other active nodes. Those are different problems with different fixes, and the first thing to do is establish which one you have by looking at the evicted node’s own logs across the eviction. A node with a clean, continuous log did not go down.
Because the packets are tiny and frequent, anything introducing a few seconds of delay or loss produces an eviction. An offload feature misbehaving on a network adapter, a switch port renegotiating, a link saturated by live migration or a backup stream, a teaming configuration that does not match the switch, or a virtualised node briefly frozen by a host-level operation will all do it while the node itself stays up and healthy.
Microsoft’s documented causes for this are the same list: network congestion, packet loss, outdated adapter drivers or faulty adapters, antivirus blocking cluster communication, firewall misconfiguration, multi-site link problems and teaming configuration. It is a short list and it is worth working through it in order rather than reaching for the thresholds first.
Network adapter offload features are dropping the traffic
You have this one if One node or one adapter model is involved, and its advanced settings differ from the other nodes or from the vendor’s guidance.
- Update the adapter driver and the firmware together. Mismatched pairs cause exactly this and are common after a partial maintenance window.
- Compare advanced properties across nodes with
Get-NetAdapterAdvancedPropertyand make them consistent. - Change one setting at a time and leave the cluster running long enough to prove it, because these faults are intermittent by nature.
The heartbeat path is blocked or unstable
You have this one if Nodes cannot reach each other on the cluster network, or the eviction is accompanied by interface entries on more than one node.
- Confirm UDP 3343 is allowed in both directions between all nodes, including through any host-based firewall installed by an endpoint protection product. Check with
netsh advfirewall firewall show rule name="Failover Clusters (UDP-In)" verbose. - Open TCP 3343 as well. Microsoft’s port list includes it for node join operations, so a UDP-only rule lets heartbeats through and blocks a node rejoining.
- Confirm ICMP is permitted, which cluster validation needs.
- Ask the network team to confirm the switch ports for cluster interfaces come up immediately rather than waiting through spanning tree convergence.
- Check that any adapter teaming matches the switch configuration on both sides, since a mismatch drops traffic silently.
The heartbeat network is saturated by other traffic
You have this one if Evictions line up with backup windows, live migrations or storage replication rather than happening at random.
- Separate cluster and live migration traffic onto different networks, or apply bandwidth management so migration cannot consume the link.
- Restrict which networks live migration may use, so it cannot fall back onto the heartbeat path.
- Move backup traffic off the cluster network entirely.
- Re-test during the same window before declaring it fixed. This fault only appears under load.
The node froze rather than the network failing
You have this one if The evicted node shows a gap in its own logs across the eviction, or it is a virtual machine on a host that was busy at that moment.
- Look for storage timeouts and disk entries on the node immediately before the eviction.
- If the nodes are virtual machines, check whether host-level backup or checkpoint operations were running against them at that moment.
- Avoid taking host-level checkpoints of running cluster nodes. Back those guests up from inside instead.
- Check memory pressure and paging on the node, because a node paging heavily can stall long enough to be evicted.
The thresholds are tighter than the network you actually have
You have this one if A stretched or multi-site cluster evicts nodes across the site link while the local nodes are stable.
- Read the current values with
Get-Cluster | Format-List *subnet*before changing anything. Defaults differ between releases and yours may already be tuned. - Raise the cross-subnet values deliberately, in steps, and record what you changed and why.
- Fix the underlying latency or loss on the site link in parallel, because tolerance is not a cure.
Every increase is time the cluster will keep waiting before failing over from a node that really has died. Make that trade knowingly.
Full reference
What has to be open between nodes
| Traffic | Protocol and port | Why |
|---|---|---|
| Cluster service heartbeats | UDP 3343 | The mechanism this article is about |
| Cluster service node join | TCP 3343 | A node rejoining after an eviction |
| RPC | TCP 135 | Cluster management operations |
| SMB | TCP 445 | Cluster shared volumes and file traffic |
| Kerberos | UDP and TCP 464 | Authentication between nodes |
| ICMP4 and ICMP6 | n/a | Required for cluster validation |
A firewall rule that opens UDP 3343 and not TCP 3343 produces a cluster where heartbeats flow and an evicted node cannot rejoin. It is a common and confusing half-fix.
Heartbeat thresholds, and what they default to
| Setting | Windows Server 2019 | Windows Server 2016 | 2008 to 2012 R2 |
|---|---|---|---|
| SameSubnetDelay | 1 second | 1 second | 1 second |
| SameSubnetThreshold | 20 heartbeats | 10 heartbeats | 5 heartbeats |
| CrossSubnetDelay | 1 second | 1 second | 1 second |
| CrossSubnetThreshold | 20 heartbeats | 10 heartbeats | 5 heartbeats |
(Get-Cluster).SameSubnetThreshold = 40
(Get-Cluster).CrossSubnetThreshold = 40
Get-Cluster | Format-List *subnet*
These are properties on the cluster object rather than parameters of a Set- cmdlet, which surprises people who go looking for one. Read them before you change them: the defaults have moved between releases, so a cluster that looks tuned may simply be on a newer build, and one that looks default may have been tuned by somebody who has left.
Where to look first
| Pattern | Most likely area |
|---|---|
| One node evicts repeatedly, others never do | That node’s adapter, driver or switch port |
| Evictions cluster during backups or migrations | Saturation of a shared link |
| All nodes are virtual machines on the same host | Host-level operations freezing the guests |
| Evictions only across sites | Cross-subnet thresholds, or the link between sites |
| Evictions started after a maintenance window | A driver updated without its firmware, or a firewall rule changed |
Proving a fix rather than hoping
- Change one thing. These faults are intermittent, and two changes at once tell you nothing.
- Run cluster validation’s network tests and confirm they pass on every node.
- Leave the cluster running through a full backup and live migration window, because that is when the fault appears.
- Read the cluster log around any near miss. It records heartbeat state changes with timestamps, so you can see how many beats were missed before a decision was made. That is the only reliable way to tell a marginal network from a clean one.
- Only then consider whether the thresholds need changing, and record what you set and why.
The one thing not to do
Do not raise the thresholds to silence an alert while the underlying network fault is still there. You have not fixed anything; you have told the cluster to wait longer before reacting, which means a node that genuinely dies now takes longer to be replaced and its roles are down for longer. Tolerance is a legitimate setting for a network whose latency you cannot control, such as a stretched cluster across a site link. It is not a fix for a faulty adapter or a saturated switch port.
None of this costs anything in licensing terms. It is a network, driver or load problem on hardware you already run. New hardware may be justified if an adapter is genuinely faulty, but no software licence changes the outcome.
Every code this article covers
| Code | What it points at | Source |
|---|---|---|
Event ID 1135 |
Cluster node was removed from the active failover cluster membership. The service on that node may have stopped, or the node may have lost communication with the other active nodes | Microsoft Learn |
Event ID 1653 |
Not published by Microsoft. It appears where a node could not join the cluster; read it alongside the node join port requirements | not published by the vendor |
Event ID 1126 |
Not published by Microsoft. It appears around a cluster network interface other nodes cannot reach | not published by the vendor |
Event ID 1127 |
Not published by Microsoft. It appears around a failed cluster network interface; the entry names the interface | not published by the vendor |
Event ID 1592 |
Not published by Microsoft. It appears around lost communication with a specific node | not published by the vendor |
Event ID 1289 |
Not published by Microsoft. It appears where the cluster service could not use a network adapter it was given | not published by the vendor |
Confirm the fix worked
- Cluster validation’s network tests pass on every node.
- A full backup and live migration window passes with no node leaving membership.
Get-ClusterNetworkshows every cluster network up with the role you intended.- UDP 3343 and TCP 3343 are both permitted between every pair of nodes, in both directions.
- The threshold values are what you intended and are recorded somewhere other than the cluster itself.
Questions people ask about this
Does fixing this involve buying anything?
No, and it costs nothing in licensing terms. This is a network, driver or load problem on hardware you already run. New hardware may be justified if an adapter is genuinely faulty, but no software licence changes the outcome.
The node rejoined by itself. Do I still need to act?
Yes. A rejoin means the cluster recovered, not that the fault went away. Roles failed over and users were interrupted, and the next eviction will do the same.
Is UDP 3343 all I need to open?
No. Microsoft’s port list for the cluster service includes TCP 3343 for node join operations as well, plus TCP 135 for RPC and ICMP for validation. A UDP-only rule lets heartbeats through and blocks a node rejoining.
Should I put the heartbeat on its own network?
Cluster communication is allowed on every network you enable for it, and separating heavy traffic such as live migration and backup onto their own networks is what actually helps. A dedicated network that is never saturated is a good idea; an isolated one with no redundancy is not.
Can I see how close the cluster came to evicting a node?
Yes. The cluster log records heartbeat state changes with timestamps, so you can see how many beats were missed before the decision. Generate it with -UseLocalTime and it is the only reliable way to distinguish a marginal network from a clean one.
