Event ID 1069 and 1205: a clustered role keeps failing and will not stay online

Fix it now

One resource in the role failed and everything depending on it went offline with it. The resource named in the failure is where to start, and it is often the victim of a dependency below it rather than the fault itself. Read the dependency tree before you restart anything.

Get-ClusterResource | Where-Object State -ne 'Online' | Format-Table Name, State, OwnerGroup, ResourceType
Get-ClusterResource 'Cluster IP Address' | Get-ClusterResourceDependency
Get-ClusterLog -Destination C:\temp\ -TimeSpan 15 -UseLocalTime

1. Read the System log entries around the failure. The published wording names the resource, its type and the clustered role it belongs to, and the resource type tells you which subsystem to look at.
2. Work down the dependency tree rather than across it. Storage comes online first, then addresses, then the network name that depends on them, then the application that depends on all of it.
3. Fix the lowest failing dependency, then bring the whole role online from Failover Cluster Manager rather than starting resources individually.
4. If a numeric code appears in an entry, decode it with NET HELPMSG <code> rather than searching for the number.

Do not repeatedly force the role online, and do not remove a failing resource from the role to make the alarm stop. Both leave the cluster believing something is healthy when it is not.

If the role comes online and stays online through a move, stop here. If it fails again on the same node or bounces between nodes, the next section covers the health checks and the restart budget behind that behaviour.

Why it happens

A clustered role is a group of resources arranged in a dependency tree. The cluster brings them online from the bottom up: storage first, then addresses, then the network name that depends on them, then the application that depends on all of it. If any link fails, everything above it stays offline, so the resource that reports the failure can be several levels away from the actual problem. Microsoft’s published wording for the failure names the resource, its type and the role, which is exactly the three things you need to start.

Each resource is also health checked continuously, with a light check running frequently and a thorough one running less often. A resource that fails its check is restarted according to the role’s policy, which allows a set number of restarts in a set period. When that budget is exhausted the cluster either fails the role over to another node or stops trying and leaves it offline. That is why a role can look like it is being restarted constantly for an hour and then simply stop, with nothing new having happened.

The resource type in the failure is the fastest route to the right subsystem. An IP address resource points at addressing: a duplicate on the network, the wrong subnet, or an interface that is down. A network name resource points at Active Directory and DNS, because the name is backed by a computer object the cluster maintains. A physical disk points at storage. A generic service points at the application itself, and the cluster is usually reporting the truth about it.

Full reference

Resource type as a shortcut

Reading the tree rather than the alarm

Get-ClusterGroup | Format-Table Name, OwnerNode, State
Get-ClusterResource | Format-Table Name, State, OwnerGroup, ResourceType
Get-ClusterResource '<name>' | Get-ClusterResourceDependency

Start at the group, then list the resources in it, then walk the dependencies of the one that failed. The point of the third command is that it tells you what that resource is waiting for, which is where the real fault usually is. A network name that will not come online because its IP address resource is offline is not a directory problem, however much the name suggests otherwise.

The cluster log, used properly

Get-ClusterLog -Destination C:\temp\ -TimeSpan 15 -UseLocalTime

-UseLocalTime is worth remembering, because without it the timestamps are in UTC and you will spend the first ten minutes doing arithmetic. Generate the log after you know which resource to look for, not before: it is verbose, and it is only useful once the event log has told you what to search for. Where an entry carries a numeric code, NET HELPMSG <code> translates it, which is faster and more reliable than searching for the number.

Events you can trust, and events you cannot

Microsoft publishes the wording for the resource-failure event and for the role failing to come online, and those two are safe to match against what you are reading. Most of the other numbers in the cluster range are not published, including the ones for a duplicate address, an interface health check, and the several network-name variants. The one worth knowing about is the network name object update failure, which Microsoft documents as event 1207 rather than any of the neighbouring numbers people quote for it. Read the text of each entry and match on that, and treat the numbers as labels.

Pinning a role while you investigate

• The role’s properties let you prevent failback and restrict preferred owners, so you can hold it on one node while you work. Put the policy back afterwards.
• Moving the role to another node is a useful test rather than a fix: a role that runs elsewhere points at something node-specific, and the fault follows it back the next time the cluster moves it.
• Run cluster validation without the storage section while roles are online. It is safe there and it catches configuration differences between nodes that explain a great deal.
• Compare the failing node against a healthy one for the specific thing that failed: the service account, the registry settings the cluster does not replicate, the network adapters and their subnets.

None of this is a licensing problem and none of it costs anything. Failover Clustering is a feature of Windows Server you have already licensed, and a resource failure is a configuration, storage or application fault. No licence state stops a resource coming online.

Every code this article covers

Confirm the fix worked

1. Every resource in the role reports as online in Failover Cluster Manager.
2. The role moves to another node and back, coming online in both places.
3. No new resource failures are logged during those moves.
4. The failover and restart policy on the role is back to what you intended, if you changed it to investigate.
5. Cluster validation, run without the storage section, passes on every node.

Questions people ask about this