STONITH: The Definitive Guide to Node Fencing in Clustering
In the world of high availability and resilient infrastructures, STONITH stands as a cornerstone technique. Short for shooting the other node in the head, albeit historically phrased in a tongue‑in‑cheek manner, STONITH is non‑negotiable when it comes to preventing split‑brain scenarios in cluster environments. This comprehensive guide walks you through what STONITH is, why it matters, how it works, and how to implement and troubleshoot it effectively, with practical advice for real‑world deployments. Whether you’re architecting a new cluster or auditing an existing one, understanding STONITH is essential for reliable, safe, and maintainable systems.
What is STONITH?
STONITH is a fencing mechanism used in clustered computing to ensure that a misbehaving or unreachable node is decisively powered off or otherwise isolated from the cluster. The goal is to guarantee that only one instance of a resource or data set can be active at any time, thereby preventing data corruption and inconsistencies that arise when two parts of a cluster believe they hold the correct state simultaneously. In practice, STONITH acts as a last line of defence: if a node cannot be reliably contacted or is acting erratically, a fencing action is triggered to halt it.
The term STONITH is widely recognised in open‑source clustering stacks such as Pacemaker, Corosync, and related fencing agents. While some documentation uses the capitalised form STONITH, you will also encounter the more casual stonith in prose. Both refer to the same concept, though capitalised STONITH is considered the canonical acronym and is often preferred in technical discussions and configuration files.
Why STONITH matters in High Availability
A highly available cluster must tolerate failures without compromising data integrity. Without proper fencing, the cluster risks a split‑brain condition—where two or more nodes believe they are the active controller or primary holder of resources. This can lead to conflicting operations, duplicate writes, or divergent configurations. STONITH mitigates this risk by decisively fencing away the problematic node, ensuring that only one node can participate in quorum decisions and resource management at any given moment.
Key reasons for implementing STONITH include:
- Eliminating split‑brain by physically or logically isolating faulty nodes.
- Providing a clear boundary for resource managers to operate within, reducing race conditions.
- Enabling safe recovery and reintegration of previously fenced nodes after issues are resolved.
- Supporting compliance and auditability for critical workloads that demand strict operational guarantees.
Effective STONITH implementation aligns with broader high availability strategies, including proper quorum handling, resource fencing policies, and robust monitoring. It is not a replacement for good design but rather a vital component of a holistic HA strategy.
How STONITH works
The functioning of STONITH depends on a few core principles: fencing devices, communication reliability, and the orchestration by the cluster resource manager. In most environments, a fencing action is triggered when the cluster detects a node as failed or unresponsive, or when a resource fails to meet its expected state from that node. The fencing device then executes a preconfigured action to render the node unreachable or powered down, effectively removing it from the cluster’s operational set.
The basic concept
At its simplest, STONITH uses a fencing device to perform an automated action—such as power cycling a remote machine, disconnecting it from the network, or issuing a lockout on a storage device—so that the node cannot I/O‑compete with others. The cluster manager issues a fencing command, the device carries out the operation, and the cluster marks the node as fenced. Once fenced, the node cannot participate in quorum decisions or resource allocation until the fencing condition is cleared and the node re‑joins the cluster under controlled conditions.
Fencing vs power management
Fencing often relies on power management capabilities, whether through IPMI, iLO, DRAC, or other dedicated out‑of‑band management interfaces. These tools give administrators a safe, remote way to cut power or reset a node. A robust STONITH setup typically uses hardware or firmware‑level fencing rather than relying solely on software stubs. This reduces the risk of a stubborn software fault on the node preventing it from being fenced successfully.
Quorum, lockout, and state transition
STONITH interacts with quorum and state transitions in the cluster. When a node is fenced, it is effectively removed from the cluster’s decision‑making set. The cluster must still maintain quorum to continue operating, or it must gracefully degrade according to its configured policies. A well‑designed STONITH strategy prevents scenarios where two partitions can both claim authority, ensuring that the remaining, healthy partition can continue to provide services without risking data consistency.
Types of STONITH devices: hardware, software, and hybrid
Hardware fencing devices
Hardware fencing relies on dedicated out‑of‑band management interfaces such as IPMI, Redfish, iLO, or DRAC. These interfaces provide authoritative power control, sensor data, and remote management capabilities.
- IPMI (Intelligent Platform Management Interface): Common in many servers, offering remote power control and chassis management.
- Redfish: A modern alternative to IPMI with a RESTful API and improved security features.
- iLO/DRAC: Integrated Lights‑Out or Dell Remote Access Controllers provide robust, vendor‑specific fencing capabilities.
Advantages include independence from the host operating system, strong isolation from software faults, and rapid action. Drawbacks can include cost, configuration complexity, and reliance on out‑of‑band network availability.
Software fencing and fencing agents
Software fencing uses agents that communicate with the fencing resources, often leveraging the cluster management software’s built‑in capabilities. In Pacemaker, for example, fence agents encapsulate common fencing actions and translate cluster decisions into concrete operations on devices or systems.
- Fence agents for IPMI, LAN power distribution units (PDUs), or virtualization platforms.
- Agent configuration in the cluster manager, including timeout values and confirmation checks to avoid premature fencing.
- Software fencing is flexible and can cover virtual machines or containerized environments where hardware access is limited.
Software fencing is highly adaptable, but it relies on the host services or network paths remaining operational long enough to execute the fence, which is why hybrid designs are often preferred for critical setups.
Hybrid and multi‑path fencing
In demanding environments, administrators implement multiple fencing pathways to increase reliability. A hybrid approach might combine hardware fencing for physical hosts with software fencing for virtual machines and containers. Multi‑path fencing ensures that if one fencing path fails or is delayed, another path can complete the fencing operation to maintain cluster integrity.
STONITH in practice: Pacemaker, Corosync, and modern clusters
Across Linux‑based clusters, Pacemaker and Corosync are common combinations where STONITH plays a central role. Pacemaker acts as the cluster resource manager, orchestrating resources, constraints, and fencing. Corosync provides the messaging layer and quorum mechanisms. When a node misbehaves, Pacemaker requests a fence, and the configured fencing device executes the action to isolate the node. The outcome is a more predictable failover process and safer recovery for services.
How Pacemaker uses STONITH
Pacemaker requires STONITH to be configured as part of a robust HA setup. In practice, administrators define fencing devices in the cluster configuration, specify the fencing level, and set timeouts to handle slow responses. Pacemaker will attempt to verify fencing completion and will mark the node as fenced only after successful confirmation. The exact fencing action—power off, power cycle, or network isolation—depends on the device and policy.
Role of STONITH in cluster resource management
Beyond isolating faulty nodes, STONITH supports orderly cluster operations. For example, when a node loses connectivity but still holds resources, fencing prevents it from continuing to compete for those resources. This leads to cleaner failovers, faster restoration, and a lower risk of data corruption. Correctly implemented STONITH reduces manual intervention, enabling operators to focus on service delivery rather than remediation after an outage.
Configuration and best practices
Effective STONITH configuration requires careful planning and ongoing validation. Below are practical guidelines to help you design and maintain a reliable fencing strategy.
Plan before you deploy
Start with a documented fencing policy that covers:
- Which nodes or resources should be fenced under what conditions.
- Which fencing devices are available, including redundancy paths.
- Expected fencing timeouts and confirmation mechanisms.
- Recovery procedures after a node is fenced, including reintegration steps.
Enable STONITH in the cluster
In Pacemaker, STONITH must be enabled for the cluster to guarantee safety. Disable or enable options should be deliberate, with a clear rationale. A cluster without proper fencing is vulnerable to split‑brain and inconsistent states. Always test fencing in a controlled lab environment before rolling out to production.
Choose multiple fencing paths
Where feasible, implement more than one fencing path. For instance, combine IPMI power control with a PDU‑level lockout and a software fence for virtual machines. Multi‑path fencing reduces single points of failure and increases the likelihood that a fencing action can complete even if one path is temporarily unavailable.
Set sensible timeouts and verification
Configure fencing timeouts to balance speed with reliability. If a fence action takes too long, the cluster may time out and assume the node is still active, risking split‑brain. Include confirmation steps to verify that the node is truly fenced before moving resources elsewhere.
Test regularly and simulate failures
Regularly exercise your fencing configuration in a non‑production environment. Simulated failures help verify that STONITH triggers correctly, that actions complete, and that the cluster continues to operate safely during a failover. Include both partial and full network outages in tests to mirror real‑world scenarios.
Secure the fencing infrastructure
Fencing involves powerful capabilities. Ensure that access to fencing devices and their management interfaces is tightly controlled. Use role‑based access, strong authentication, and network segmentation to prevent tampering. Audit logging for fencing events is essential for post‑incident analysis.
Documentation and runbooks
Provide clear runbooks for operators detailing how to respond to fencing events, how to reintegrate fenced nodes, and how to handle false positives. Documentation helps maintain operational consistency and reduces risk during high‑pressure outages.
Common pitfalls and troubleshooting
Even well‑designed STONITH configurations can encounter challenges. Awareness of common pitfalls can save time and prevent disruptions.
False positives and unnecessary fencing
Unreliable monitoring, network flakiness, or misconfigured thresholds can trigger fencing prematurely. Verify monitoring paths, ensure accurate heartbeat signals, and fine‑tune the detection logic to distinguish between transient glitches and genuine failures.
Failed fencing actions
Sometimes, fencing actions fail due to misconfigured devices, network issues, or insufficient permissions. Maintain clear alerts, check device status, verify network reachability, and have a manual fallback plan if automatic fencing cannot complete.
Reintegration of fenced nodes
Past issues that led to fencing may recur if a node is reintegrated without addressing root causes. Establish a controlled reintegration process, validate that the node is healthy, and monitor for recurrence before returning it to normal operation.
Performance impact during fencing
In large clusters, frequent fencing operations can introduce latency in failover paths. Review your HA design to ensure that fencing actions do not unduly slow service recovery while still meeting safety guarantees.
Security considerations
STONITH and related fencing controls sit at a critical junction of security and reliability. Protecting these components is essential to prevent misuse or disruption of cluster operations.
- Secure management interfaces: Restrict access to IPMI, iLO, DRAC, and similar interfaces to trusted networks or VPNs.
- Strong authentication and role separation: Use unique accounts for operators, auditors, and automated processes with appropriate permissions.
- Auditability: Enable detailed event logging for all fencing actions and administrative changes.
- Network isolation: Place fencing channels on dedicated, secured networks to avoid interference from general traffic.
Case studies: real‑world STONITH in action
Understanding practical deployments helps translate theory into reliable practice. Below are anonymised, representative scenarios that illustrate how STONITH contributes to stability.
Case Study A: a business‑critical database cluster
A database cluster spanning two data centres relied on a hybrid STONITH strategy. Hardware fencing via IPMI provided rapid isolation of failing nodes, while software fencing ensured virtualized resources could be quarantined without physical intervention. The result was near‑instant failover with minimal data loss risk, and a clear process for reintegration after maintenance windows.
Case Study B: a virtualised environment with rapid provisioning
In a cloud‑native setup, fencing required coordination between hypervisor‑level controls and container orchestration. Pacemaker used a combination of fence agents for virtual machines and a power‑cycling policy for the host machines. This approach reduced failure windows and maintained service availability during unpredictable workloads.
The future of STONITH and evolving trends
As clusters become more dynamic and distributed, STONITH is evolving alongside changing architectures. Some of the notable trends include:
- Enhancements in fencing APIs and standardisation across vendors, making it easier to implement and manage consistently.
- Increased support for software‑defined fencing that complements hardware capabilities, particularly in virtualised and containerised environments.
- Improved security models for fencing operations, including better authentication, auditing, and anomaly detection to prevent misuse.
- Integration with automation and policy engines that enable adaptive fencing based on workload, time of day, or operational risk.
Despite these advances, the core principle remains unchanged: STONITH is about decisively isolating malfunctioning components to preserve the integrity and availability of the cluster. The best practices today remain relevant for tomorrow’s evolving landscapes.
Practical tips for building a resilient STONITH‑enabled cluster
- Document your fencing strategy in a central, accessible location and ensure team buy‑in from operators and engineers.
- Prefer hardware fencing where feasible for speed and reliability, complemented by software fencing for virtual resources.
- Test continuously: run regular drills that cover partial failures, complete outages, and reintegration scenarios.
- Maintain redundancy: ensure multiple fencing paths with independent power management and network channels.
- Monitor and alert: configure proactive alerts for fencing events, device health, and timeouts to enable rapid response.
- Protect fencing credentials: limit access, rotate credentials, and log every change to fencing configurations.
Conclusion
STONITH is a fundamental, if sometimes underappreciated, element of robust clustering. By providing a violence‑free, decisive method to isolate malfunctioning nodes, STONITH reduces the risk of split‑brain, protects data integrity, and supports clean, predictable failovers. A well‑designed fencing strategy—encompassing hardware and software fencing, thoughtful policies, and rigorous testing—translates into higher service availability, operational resilience, and peace of mind for teams responsible for critical systems. Embrace STONITH as a core pillar of your high‑availability architecture, and you’ll enjoy more reliable clusters, safer reintegration, and clearer incident handling when things go wrong.