Event ID 1069 and Event ID 1146: cluster faults that take a whole DAG offline

Fix it now

Microsoft publishes Event ID 1069 as a cluster resource in a clustered service or application having failed. Underneath a database availability group that means the Windows failover cluster broke, not Exchange. Fix the cluster first; everything Exchange does with database copies sits on top of it.

Get-ClusterNode
Get-ClusterGroup
Get-ClusterResource
Get-ClusterQuorum
Get-ClusterLog -Destination C:\Temp -TimeSpan 60

1. Read the resource name and type from the 1069 entry. Microsoft’s own resolution starts there: check and correct problems with the application or service behind that resource.
2. Check the witness. Confirm the server is up, the share exists and every member can reach it by name.
3. Read the sequence in Event Viewer under Applications and Services Logs, then Microsoft, then Windows, then FailoverClustering.
4. If a resource cannot start within reasonable time limits, Microsoft names the Pending timeout property on the resource as the value to adjust, after diagnosing why it is slow.
5. Do not reseed databases or restart Exchange services while the cluster is broken. You will be reseeding into a group that cannot decide who owns anything.

Get-ClusterLog writes the detailed cluster log for the period you ask for. Sixty minutes around the incident is usually enough and is far easier to read than a full log.

If the cluster forms, quorum is intact and databases have mounted on their intended servers, you are done. If not, the next section covers quorum, the witness and the resource host process.

Why it happens

A database availability group is a Windows failover cluster with Exchange’s own manager layered on top. Exchange does not run its databases as traditional cluster resources, but it depends completely on the cluster for membership, for quorum and for the witness that breaks ties. When the cluster cannot establish quorum, nothing can decide which copy of a database should be active, so the safe outcome is that none of them are.

Event 1069 is published, and its wording is worth having exactly: cluster resource, in clustered service or application, failed. Microsoft’s symbolic name for it is RCM_RESOURCE_FAILURE, the source is Microsoft-Windows-FailoverClustering, and the documented cause is that a clustered resource failed to come online. The documented resolution is three things: correct problems with the application or service behind the resource, correct problems with cables or cluster-related devices, and adjust the resource’s properties – Pending timeout in particular – so there is enough time for the associated service to start.

The other two entries in this family are not published. One describes the process that hosts cluster resources being terminated and restarted; the other a node declining to start the cluster service because it cannot confirm its copy of the cluster configuration is current. Read what your own entries say. The second of those is a deliberate refusal rather than a fault, and clearing it means choosing a node you are confident about, which is a decision with consequences rather than a command to run casually.

Full reference

What is published about these entries

Forcing a cluster to start on a node you have not verified can bring the group up with an out-of-date configuration, and in the worst case allows a partitioned group to make conflicting decisions about which database copies are active. Establish which node holds the current configuration before you force anything, and never force two nodes on either side of a network partition.

Microsoft’s own resolution for a failed resource

1. Check and correct any problems with the application or service associated with the resource. If it cannot start within reasonable time limits, diagnose and resolve what is making it slow.
2. Check and correct any problems with cables or cluster-related devices.
3. Adjust the properties of the resource in the cluster configuration, especially the Pending timeout value, so it allows enough time for the associated application or service to start.
4. Verify by bringing the clustered service or application online through Failover Cluster Management and watching for further events.

Witness requirements worth checking every time

• The witness server cannot be a member of the DAG, and must be in the same Active Directory forest.
• One server can witness several DAGs, but each DAG needs its own witness directory.
• A non-Exchange witness server needs the Exchange Trusted Subsystem universal security group in its local Administrators group, granted before the DAG is created.
• Windows Firewall on the witness needs the File and Printer Sharing exception; the witness uses SMB port 445.
• Set-DatabaseAvailabilityGroup can reconfigure the witness server and directory where the witness lost its storage or somebody changed the share permissions.
• Do not host the witness on a member of the group, or on the same host or storage as the members.

Gathering evidence without guessing

Get-ClusterLog -Destination <path> -TimeSpan <minutes> writes the cluster’s own detailed log for the window you specify, on every node. That is the record of what the cluster decided and when, and it is considerably more informative than the event log summary. Collect it while the incident is fresh; the log rolls, and a cluster that has been flapping for a day will have overwritten the part you wanted.

Every code this article covers

Confirm the fix worked

1. Get-ClusterNode shows every member up and Get-ClusterQuorum reports the intended configuration.
2. The witness is reachable and being used, rather than merely configured.
3. Databases have mounted on their intended servers and every copy is healthy.
4. The failover clustering log records no further resource failures over the following day.

Questions people ask about this