High Availability: intelligent redundancy for resilient systems

·

High availability (HA) is the property of a system designed to remain operational despite the failure of its components, sustaining agreed service levels over extended periods. Its objective is not to eliminate failures — something impossible — but to prevent the failure of a single component from causing service downtime. This is achieved through intelligent redundancy: duplicating critical components and, above all, eliminating every single point of failure (SPOF).

Measuring availability: the "nines" and their limits

Availability is expressed as a percentage of operating time and translates into a concrete annual downtime window. It is worth keeping these figures in mind, because each additional nine multiplies cost significantly:

AvailabilityNameApproximate annual downtime
99%Two nines3 days and 15 hours
99.9%Three nines8 hours and 46 minutes
99.99%Four nines52 minutes
99.999%Five nines5 minutes and 15 seconds

Two metrics complement the percentage: MTBF (Mean Time Between Failures), which measures component reliability, and MTTR (Mean Time To Recovery), which measures recovery speed. Availability improves both by increasing MTBF and by reducing MTTR; in practice, automating recovery to lower MTTR is usually more cost-effective than pursuing infallible hardware.

Redundancy: active-active versus active-passive

There are two fundamental topologies. In active-passive, one node handles traffic while another waits on standby, ready to take over when the primary fails; it is simpler to operate but wastes capacity. In active-active, all nodes handle traffic simultaneously by sharing the load, which makes better use of resources and adds capacity, at the cost of requiring state synchronisation and conflict resolution. The choice depends on whether the application maintains state (stateful) or not (stateless): stateless applications scale in active-active mode naturally.

Load balancing

The load balancer is the component that distributes requests among available nodes and removes failing ones from rotation. It operates at different layers of the OSI model: a layer 4 (transport) balancer distributes by IP and port without inspecting content, while a layer 7 (application) balancer makes decisions based on the URL, cookies or HTTP headers, enabling advanced routing. Common algorithms include round-robin, least connections and IP hashing to maintain session affinity. A critical detail: the load balancer itself must not be a SPOF, so it is deployed in a redundant pair with a floating virtual IP.

Clustering and heartbeat protocols

A cluster groups several nodes that behave as a single logical system. Coordination relies on a heartbeat protocol: each node emits periodic signals and, when they stop being received from a peer, the cluster considers it failed and initiates failover. The greatest enemy of this mechanism is split-brain: if the network between nodes is partitioned but both remain alive, each may believe it is the sole survivor and assume the primary role, causing data corruption. This is prevented with quorum (a majority vote required to act, which demands an odd number of nodes or a witness node) and with fencing / STONITH techniques that power off the suspect node before it causes harm. Tools such as Pacemaker and Corosync on Linux, or Keepalived with VRRP, implement these patterns.

Beyond HA: geographic redundancy and data replication

High availability protects against component failures within a single site. Against a disaster affecting an entire data centre, geographic replication across zones and regions is used, with two objectives defined by the continuity plan: RTO (maximum acceptable time to restore service) and RPO (maximum tolerable data loss). Database replication can be synchronous (no data loss but with latency and coupling) or asynchronous (better performance at the cost of a larger RPO). It is important not to confuse HA with backup: redundancy also replicates logical errors and accidental deletions, so HA never replaces a verified backup.

High availability in the cloud and in containers

Cloud platforms reformulate HA with managed primitives. Providers organise their infrastructure into availability zones (independent data centres within a region) and geographically separated regions; distributing replicas across multiple zones protects against the failure of an entire centre. Auto-scaling groups automatically replace instances that fail a health check, reducing MTTR to minutes without human intervention. In the container world, an orchestrator such as Kubernetes implements the same principles declaratively: it defines a desired number of replicas for each service, monitors their health through liveness and readiness probes — the modern equivalent of heartbeats — and automatically reschedules failed containers on healthy nodes. Quorum logic reappears in the control plane: the cluster state store (etcd) requires a majority to avoid inconsistencies, so it is deployed on an odd number of nodes.

Application-level resilience patterns

Infrastructure redundancy is not enough if the application is not prepared to tolerate failures. The most widely adopted resilience patterns are the circuit breaker, which stops invoking a failing service to avoid exhausting resources waiting for responses that will never arrive; retries with exponential backoff, which retry a transient operation by spacing attempts to avoid saturating the system; well-tuned timeouts, which prevent a hung request from blocking a thread indefinitely; and bulkheads, which isolate resources so that the failure of one feature does not drag down the others. Designing the application as stateless, externalising state to shared stores, is what allows the load balancer to distribute traffic freely and any node to serve any request.

Regulatory framework and best practices

Service continuity is a normative requirement. The ISO 22301 standard defines the requirements for a business continuity management system, including the impact analysis that sets RTO and RPO values. ISO/IEC 27001 incorporates availability as one of the three pillars of information security, alongside confidentiality and integrity. At the operational level, Site Reliability Engineering (SRE) practices formalise SLOs, error budgets and chaos testing to validate that redundancy genuinely works.

Testing redundancy: chaos engineering

A redundant architecture that has never been tested under failure conditions is a hypothesis, not a guarantee. Chaos engineering formalises the practice of injecting controlled failures into production — shutting down a node, introducing network latency, exhausting resources — to verify that the switchover mechanisms respond as expected. The principle is counterintuitive but sound: it is better to trigger the failure in a planned way, at a controlled time and with the team prepared, than to discover at three in the morning that the failover does not work. These experiments always start with a hypothesis ("if this node goes down, the service will remain available with degradation below X%") and are executed with a limited blast radius and an immediate abort mechanism. Periodic switchover drills, known as game days, keep the team practised and validate that recovery documentation remains correct after every change.

The cost of availability: sizing without excess

Each additional level of availability multiplies the cost of infrastructure, operations and complexity. Moving from three to four nines may require doubling nodes, contracting cross-zone replication and reinforcing monitoring; reaching five nines demands active-active geographic redundancy and permanently on-call teams. The right decision is not technical but economic: the cost of downtime for the business (revenue loss, contractual penalties, reputational damage) is compared with the cost of preventing it. An e-commerce platform during a peak sale event justifies five nines; an internal monthly reporting system rarely needs more than three. Sizing availability per service — rather than applying a uniform level across the entire platform — is what allows investment where it truly matters and savings where the business can tolerate a brief interruption.

Common errors that undermine high availability

Serious incidents almost always betray the same oversights: leaving a hidden SPOF (a single network switch, a single power supply, an unredundant load balancer); never testing real failover, only to discover in the middle of a crisis that the replica does not start; ignoring the split-brain risk due to lack of quorum; confusing HA with backup; and designing for "five nines" without measuring whether the business needs them or can afford them. Another recurring failure is not monitoring the state of the passive node, which degrades silently until it is needed and fails to respond.

Frequently asked questions

What is the difference between high availability and disaster recovery? HA keeps the service running in the face of component failures within a single site, with automatic switchover in seconds. Disaster recovery (DR) restores the service after the loss of an entire site, with timescales defined by the RTO and RPO.

Do I always need five nines? No. Each additional nine considerably increases the cost of the architecture. The target level should be derived from the real impact of downtime on the business, not from a generic aspiration.

Does redundancy protect me from accidental data deletion? No. Redundancy replicates errors as well. For that you need versioned, tested backups that are independent of the production system.

What exactly is split-brain and how is it avoided? It is the situation in which a network partition causes two nodes to simultaneously believe they are the primary. It is avoided by requiring quorum (a majority) to act and by applying fencing to isolate the suspect node.

High availability is not bought; it is designed and tested. It starts by identifying and eliminating every single point of failure, chooses the redundancy topology that matches the application's state profile, coordinates nodes with heartbeats and quorum that prevent split-brain, and is validated through real switchover exercises. The number of nines must reflect the business impact, not a catalogue figure. At Summum Sistemas we design resilient architectures, define RTO and RPO targets together with the client and verify failover in a controlled manner, so that redundancy is demonstrated before an incident puts it to the test.