Event ID 1177 and 1564: quorum is lost and the cluster service shuts down

Fix it now

Event ID 1177 is the cluster service shutting itself down because quorum was lost, and Event ID 1564 says the file share witness could not be arbitrated. Get the witness reachable again, or get the missing nodes back, and the cluster re-forms on its own.

Get-ClusterNode | Format-Table Name, State, NodeWeight, DynamicWeight
Get-ClusterQuorum
Test-NetConnection fs01 -Port 445
Test-Path '\\fs01\clusterwitness'

1. Read the two outputs together. Get-ClusterQuorum tells you what the witness is meant to be; the node table tells you how many votes are actually left.
2. If the witness is a file share, fix its access rather than its existence. Microsoft’s documented grant is Change and Read on the share, plus Modify, Read & execute, List folder contents and Read on the NTFS permissions, given to the cluster name object.
3. If the share has gone for good, repoint the witness: Set-ClusterQuorum -FileShareWitness \\fileserver\fsw
4. Start the cluster service on every node that is down, including ones you think are irrelevant, and let the cluster form normally once a majority exists.
5. Only if service must be restored and no majority can form, force quorum on the node with the most current data: Start-ClusterNode -ForceQuorum, then start the remaining nodes normally.

-ForceQuorum is the documented parameter name. -fq and -FixQuorum are aliases for the same switch, so older runbooks that use them are not wrong, just older.

If the cluster is up and the witness resource is online, you can stop here. The next section covers how the vote arithmetic actually works, so the same failure does not repeat.

Why it happens

A cluster keeps running while a majority of votes survives. Nodes hold votes, a witness may hold one more so the total stays odd, and the moment the majority is gone the remaining nodes stop the cluster service deliberately. Event ID 1177 is that decision being recorded: the cluster service is shutting down because quorum was lost, which Microsoft attributes to lost network connectivity between some or all nodes, or a failover of the witness.

The alternative is worse than downtime: two isolated groups of nodes, each convinced it owns the storage, both writing to it. A 1177 is the safety mechanism working, not the fault.

The witness is where most of these incidents start, and the events around 1177 tell you which part of it failed. Event ID 1564 is the file share witness failing to arbitrate for its share. Event ID 1562 is the same resource failing a periodic health check. Event ID 1558 is different in kind: the cluster detected a problem with the witness resource and is failing the witness over to another node to re-establish access to cluster configuration data, which is why a witness can look healthy on a node you were not looking at.

Full reference

The events that surround a 1177, and what each one is telling you

What a witness has to be

Microsoft publishes a specific set of requirements for a file share witness, and most of the witnesses that fail at the worst moment break one of them. The share must be dedicated to a single cluster and must not carry user or application data. It must support SMB 2 or later and have at least 5 MB free. It must not be a DFS path, and it must not sit on replicated storage, because either can leave two halves of the cluster looking at different copies. It should be physically separate from the cluster nodes or cluster sites it arbitrates for.

The permissions are narrower than they are usually granted. The cluster name object needs Change and Read on the SMB share, and Modify, Read & execute, List folder contents and Read on the NTFS security settings. Full Control is the habit, not the requirement.

Reading the vote arithmetic instead of counting servers

Get-ClusterNode with NodeWeight and DynamicWeight is the only honest answer to “how many nodes can I lose”. Dynamic quorum adjusts weights as nodes leave cleanly, and the witness vote is added or removed to keep the total odd, so the survivable loss changes as the cluster changes. The clearest published illustration of this is Event ID 5398, which prints votes required to start the cluster, votes available, and the number of nodes with votes when a cluster refuses to form.

Forcing quorum, and what it costs

Forcing quorum starts the cluster with a minority and tells it to treat that node’s copy of the cluster database as authoritative. Do it on one node only, choose the node with the most current data, and never do it on two nodes in separate network partitions – that is how you get two clusters writing to the same storage.

Once the cluster is up after a forced start, review the quorum configuration and the resource list before anything else. Anything changed after the copy you forced was current is gone, and the quickest way to lose a second afternoon is to assume the configuration you are looking at is the one you left.

When the witness looks fine and the cluster still stops

• Check name resolution from every node to the witness host, not just from the node you are logged on to.
• Check that the cluster name object still exists and is enabled in Active Directory. Stale-account cleanup tools disable cluster identities because they do not authenticate the way workstations do.
• Check whether the witness resource has moved. Event 1558 says the cluster will fail the witness over to another node, so the resource you are inspecting may not be the one that failed.
• On a stretched cluster, count votes per site. A site with more votes than the other decides every partition, which is usually not what the design intended.

Every code this article covers

Confirm the fix worked

1. Get-ClusterNode reports every node as Up, with the weights you expect.
2. Get-ClusterQuorum shows the witness type and location you intended.
3. The witness resource is online in Failover Cluster Manager, and stays online through a working day.
4. Test-NetConnection <witness-host> -Port 445 succeeds from every node, not just from one.
5. Drain and restart one node, and confirm the cluster survives that single loss without dropping below quorum.

Questions people ask about this