Resiliency is the ability of a system to keep providing useful service when something goes wrong.
Security protects systems from malicious activity, data theft, and improper behavior. Resiliency covers another part of the problem: traffic spikes, software bugs, hardware failures, deployment issues, network splits, and data center disruptions.
A resilient Java system is not created by adding one extra server. It starts with clear service expectations and continues through architecture, deployment, operations, and recovery planning.
The Problem
Many teams say that a system must be highly available without defining what that means.
That statement is too vague.
Does the login page need to be available? Does every feature need to work? Can the system show read-only data during an incident? Is a static courtesy page considered available? How much data loss is acceptable? How quickly must service return after failure?
Without answers, the team cannot design the right architecture.
Vague goal:
The system must always be up.
Better goal:
Payment creation must be available 99.9% per year.
During partial failure, payment history may be read-only.
No more than 5 minutes of payment status updates may be lost.
Resiliency starts with precise expectations.
Start with SLAs
A Service Level Agreement, or SLA, defines measurable service expectations.
The most common SLA is uptime. Uptime measures how available a system is during a defined period.
Example:
Uptime target: 99.9% per year
Scope: payment authorization API
Availability means: valid requests receive correct authorization responses
For complex applications, uptime must be scoped carefully. A system may technically respond while still being functionally unavailable. For example, it may show a static maintenance page, return stale data, or return incorrect results.
A meaningful uptime target should define:
Which feature is measured
What correct behavior means
Which time period is used
Which failures count as outage
Whether degraded mode counts as available
In microservices systems, different features may need different SLAs. A payment authorization service may require a stronger SLA than a reporting export.
Planned and Unplanned Downtime
Downtime usually falls into two broad groups.
Planned downtime happens because of predictable operations such as maintenance, deployments, upgrades, or infrastructure changes.
Unplanned downtime happens because of unexpected failures such as system crashes, hardware failures, bugs, traffic spikes, or broken dependencies.
Downtime
|
+--> Planned
| +--> Maintenance
| +--> Deployment
| +--> Upgrade
|
+--> Unplanned
+--> Crash
+--> Hardware failure
+--> Traffic spike
+--> Software bug
One way to reduce planned downtime is to release less often. That can work in traditional environments, but it creates a different problem: slower delivery and longer time to market.
Modern systems often reduce planned downtime through rolling releases or similar techniques. Instead of stopping the whole application, instances are updated gradually while the system continues serving traffic.
Rolling release:
Node A updated
Node B still serving
Node C still serving
Then:
Node A serving
Node B updated
Node C still serving
The goal is to keep useful service available while change is happening.
Reliability Metrics Beyond Uptime
Uptime is important, but it is not enough.
Several related metrics help explain how reliable a system really is.
| Metric | Meaning |
|---|---|
| Mean time between failures | Average time between outages |
| Mean time to recovery | Average time to restore service after downtime |
| Mean time to repair | Average time to permanently resolve the issue |
A system can meet an uptime SLA and still be unstable if failures happen often but are repaired quickly. A short mean time between failures signals fragility.
A high mean time to recovery may mean the team lacks automation, runbooks, monitoring, or training.
A high mean time to repair may indicate deeper architectural problems or weak troubleshooting tools.
Other useful service metrics include response time and error rate.
Useful SLA examples:
- 99.9% uptime for payment authorization
- 90% of profile lookups respond under target latency
- Error rate remains below the defined threshold per day
- Recovery completes within the target duration
Define the metric before choosing the technology.
Clustering
Clustering is one of the most common techniques for increasing resiliency.
A cluster is a set of components working concurrently in a mirrored or coordinated way. If one node fails, another node can keep serving requests.
Load balancer
|
+--> Node A
+--> Node B
+--> Node C
The cluster may replicate the state between nodes. It may also include redundant supporting infrastructure such as storage and networking.
A load balancer is often placed in front of the cluster. It routes requests to healthy nodes and stops sending traffic to failed nodes.
Client request
|
v
Load balancer
|
+--> healthy node
+--> healthy node
x--> failed node
Cluster nodes need to understand the state of the cluster. They commonly use heartbeat protocols, which exchange special messages over a network or filesystem. In Java ecosystems, JGroups is a known library used for heartbeat and leader election scenarios.
Clustering improves availability, but it has a cost.
The Cost of Clustering
Clusters are not free.
State replication can reduce performance. Setup can be complex. Network latency matters. Storage behavior matters. The number of nodes matters. Failure behavior must be tested.
Cluster tradeoff:
Higher availability
+
Better planned maintenance options
-
More operational complexity
-
State replication cost
-
Split-brain risk
A cluster also behaves like distributed storage when shared state is involved. That means consistency and partition behavior must be considered.
This is where split-brain scenarios become important.
Split-Brain, Quorum, and Witnesses
Split-brain happens when a cluster is divided into parts that cannot communicate with each other.
Before partition:
Node A <-> Node B <-> Node C
After partition:
Node A <-> Node B Node C
no communication
One side cannot know whether the other side is down or only unreachable. If both sides continue accepting writes, they may process conflicting operations.
To protect consistency, a cluster may stop operating or disable writes in one partition.
A quorum is the minimum number of nodes required for the cluster to operate. A common quorum rule is half of the cluster nodes plus one.
Three-node cluster:
quorum = 2
Partition:
Node A + Node B = majority, continues
Node C = minority, stops or denies writes
Clusters are often built with an odd number of nodes so one partition can form a majority.
Another option is a witness. A witness is a special node, often placed remotely, that helps decide which partition should continue when the cluster splits.
Node A + Node B Node C + Node D
\ /
\ /
Witness decides
These mechanisms exist because resiliency is not only about keeping processes alive. It is also about avoiding incorrect behavior during failure.
High Availability
High Availability, or HA, is related to clustering but works differently.
In a cluster, multiple nodes may serve requests at the same time. In an HA setup, one or a limited number of primary nodes usually serve requests, while failover nodes wait.
Normal HA operation:
Primary node -> serving traffic
Standby node -> ready to take over
When the primary fails, a standby node takes over.
There are two common standby models.
Hot Standby means the failover node is already running and more or less aligned with the primary. Recovery can be faster.
Cold Standby means the failover system is not fully running or lacks current data. Recovery can take longer.
Hot Standby:
higher cost, faster recovery
Cold Standby:
lower cost, slower recovery
HA is useful when full active-active clustering is too expensive, too complex, or unnecessary.
Disaster Recovery
Disaster Recovery, or DR, handles extreme failure scenarios.
A DR system is usually located in a remote geographical location and synchronized periodically. It is meant for situations where a whole data center is disrupted by events such as fire, earthquake, flooding, major infrastructure failure, or other serious disasters.
Primary data center
|
| periodic alignment
v
Remote DR site
DR is not the same as an everyday failover. It is designed for rare but severe events.
The more critical the system, the stronger the DR requirements usually become. In some industries, DR may be required by law or regulation.
Backup and Restore
Backup and restore are a core part of resiliency.
Backups protect against disasters, human error, software bugs, accidental deletion, corruption, and unexpected data loss.
A backup strategy is incomplete until restore is tested.
Backup strategy:
1. Backup data.
2. Backup configuration.
3. Encrypt backups where needed.
4. Store backups safely.
5. Periodically restore test data.
6. Verify completeness and usability.
A backup that cannot be restored is not a backup. It is only a file.
Encrypted backups need extra attention because restore depends on both the backup data and the ability to access the required keys.
RTO and RPO
Two important recovery metrics are Recovery Time Objective and Recovery Point Objective.
Recovery Time Objective, or RTO, is the time needed for a failover or recovery system to take over after the primary system fails.
Recovery Point Objective, or RPO, is the amount of data loss that is acceptable.
RTO answers:
How long can recovery take?
RPO answers:
How much data can we lose?
Examples:
Clustered payment service:
RTO: near zero or very low
RPO: near zero or very low
Daily reporting DR system:
RTO: several hours may be acceptable
RPO: up to 24 hours may be acceptable
RTO and RPO should be chosen by business impact, not by technical preference alone.
Physical Placement
Where the application runs matters.
Running nodes in different data centers provides strong resiliency, especially if the data centers are geographically far apart. The tradeoff is cost and latency. Connections between distant locations can be expensive and slower.
Cloud providers often describe separated data center locations as availability zones, grouped by geographical area.
Region
|
+--> Availability zone A
+--> Availability zone B
+--> Availability zone C
If separate data centers are too expensive, a system can still improve resiliency by using separate rooms inside one data center. Those rooms may have separate power, networking equipment, and cooling. This is cheaper and has better connectivity, but it does not protect against building-level disasters.
A lower level of separation is placing nodes in different racks. This can protect against rack-level failures but not room-level or data center-level failures.
Isolation strength:
Different data centers -> strongest, higher cost
Different rooms -> medium, lower cost
Different racks -> lower, cheaper
Same machine -> weakest
The placement strategy should match the SLA.
Functional Tiering
Not every feature needs the same level of resiliency.
A system can classify features by criticality and assign infrastructure accordingly.
Critical:
payment authorization
account access
security checks
Important:
payment history
customer notifications
Lower criticality:
analytics dashboard
monthly export
marketing recommendations
Critical functions may require resilient, expensive, geographically distributed systems. Less critical features may run on cheaper infrastructure and accept occasional downtime.
This is functional tiering.
It helps control cost while protecting what matters most.
Degraded Mode
A resilient system does not always need to be fully functional during failure.
Sometimes, partial service is better than a total outage.
Failure detected:
- Disable write operations
- Keep read-only access
- Serve cached reference data
- Queue requests for later processing
- Hide noncritical features
For example, if one backend dependency fails, the application may still show account details but disable new payment creation. If a reporting service fails, the core transaction path should remain available.
Degraded mode must be designed intentionally. It should not be an accident.
Common Mistakes
The first mistake is defining uptime without defining what available means.
The second mistake is assuming clustering solves every failure. Clustering introduces complexity and split-brain risk.
The third mistake is ignoring recovery metrics. Without RTO and RPO, the team cannot evaluate whether a design is good enough.
The fourth mistake is backing up data without testing the restore.
The fifth mistake is giving every feature the same resiliency target. That can make the system unnecessarily expensive.
The sixth mistake is placing redundant nodes too close together. Two nodes on the same physical machine do not protect against machine failure.
The seventh mistake is designing only for crashes and ignoring degraded behavior.
Checklist
- SLAs are defined for specific features.
- Availability has a precise meaning.
- Planned and unplanned downtime are considered separately.
- Mean time between failures is monitored.
- Mean time to recovery is monitored.
- Mean time to repair is monitored.
- Response time and error rate are considered where relevant.
- Clustering is used only where its complexity is justified.
- Split-brain behavior is understood.
- Quorum or witness strategy is defined where needed.
- HA standby model is clear.
- DR expectations are documented.
- Backups are restored in tests.
- RTO and RPO are defined by business impact.
- Physical placement matches resiliency needs.
- Critical and noncritical functions are tiered.
- Degraded mode is designed and tested.
Conclusion
Resiliency is not one technology. It is a set of decisions.
Start with SLAs. Define uptime carefully. Separate planned and unplanned downtime. Track recovery metrics. Use clustering where active redundancy is justified. Use HA and DR where failover or disaster recovery is the better fit. Back up data and test restore. Place systems according to the failure scenarios you need to survive.
A resilient Java system is not only able to stay online. It is able to fail in controlled, understood, and recoverable ways.