Over more than 20 years in enterprise IT, in environments with over 1,370 physical and virtual servers, the most important lesson has been this: high availability is not a single product but a management discipline in which many parts work together.
Define RPO and RTO first
Not every system is equally critical. Acceptable downtime and data loss can differ for email, PMS/POS, ERP, authentication or file services. RTO defines how quickly a service must be restored; RPO defines the acceptable window of data loss.
If these two targets are not written down, infrastructure investment easily becomes either more expensive than necessary or insufficient for the business. The technical architecture should be designed against these targets together with the business units, before any products are purchased.
Remove single points of failure layer by layer
Power and physical infrastructure
UPS, generator, dual power supplies, cooling, racks and environmental monitoring should be assessed together.
Network connectivity
Redundant switches, backup uplinks, independent internet lines and a sound loop-prevention design are required.
Servers and virtualization
Clustering, live migration, resource reserves and separate failure domains should be planned.
Storage
RAID alone is not enough; controller, connectivity, replication and capacity risks must also be monitored.
Identity and core services
DNS, DHCP, Active Directory and certificate services must not depend on a single server.
Application dependencies
Database, licensing, API and third-party service dependencies should be part of the architecture.
A backup you cannot restore is not a backup
A backup job marked as completed does not by itself provide assurance. For critical systems, restore tests, application consistency, retention periods and offsite or immutable copies should be verified regularly.
- The 3-2-1 approach with an offline or immutable copy
- Periodic restore tests for critical virtual machines
- Separation of duties and protection of the backup account
- Disaster recovery order documented according to business priority
Monitoring should do more than say a device is up
In addition to CPU, memory and disk usage, service response time, certificate expiry, backup results, replication lag, hardware health and user experience should be tracked. Alert thresholds must be meaningful; a flood of trivial alerts must not cause critical warnings to be missed.
Change management, patch planning, capacity trends and post-incident root cause analysis are also part of the operational side of high availability.
Test the architecture with controlled failure scenarios
When a link, host or storage path goes down, you need to see that the expected failover actually happens. Tests should be planned according to production risk, rollback steps written in advance and the results recorded.
Even the best architecture is not sustainable without documentation, clear ownership and regular drills. The goal is not just to bring systems back up, but to make clear who does what, and in which order, during an outage.
Questions to ask in a high availability review
- Are RPO and RTO targets for critical services written down?
- Is there any single point of failure in power, network, servers, storage or core services?
- Are regular, documented restore tests performed from backups?
- Are alerts, capacity and certificate expiry monitored centrally?
- Are the disaster recovery order and owners up to date?
Related guide: How to test a disaster recovery plan.