How Serverless Systems Handle Region Failure

If one cloud region fails, serverless apps can fail with it. I’d plan for five things up front: RTO, RPO, data copy, traffic switch, and testing.

Here’s the short version:

• Serverless does not remove regional risk.
• DNS failover moves traffic, not data.
• Active-active can cut downtime to near 0, but data writes and event handling get harder.
• Active-passive is simpler and can cost less, but recovery time depends on promotion, routing, and replication lag.
• Replication lag becomes your data-loss window.
• Idempotency, write control, and scheduler control stop duplicate work after failover.
• Health checks must test real readiness, not just whether an endpoint replies.
• Failback should be planned, not automatic.
• Testing is the proof. If you have not measured failover time and data loss, you do not yet know your recovery position.

In plain terms: I’d treat a serverless app as a regional system, not just a set of functions. That means mapping every dependency - databases, queues, buckets, secrets, schedulers, monitoring, and third-party services - then deciding how each one survives a regional outage.

A few points matter most:

• Set a maximum downtime target and a maximum data-loss target
• Pick active-active or active-passive based on those targets
• Define one write model for each data set
• Replicate state, events, and schedules
• Use DNS failover with low TTLs, but assume caches may still delay the switch
• Measure detection time, switch time, replay time, and user impact
• Rehearse failover and failback until the process is repeatable

Building Multi-Region (Active-Active) Serverless Apps

Quick comparison

The core idea is simple: code redeploy is not recovery. Recovery only works when data, events, access, quotas, and runbooks are already in place in the second region.

Choose a regional architecture: active-active or active-passive

::: @figure {Active-Active vs Active-Passive: Serverless Regional Failover Comparison} :::

Once you’ve set your RTO and RPO, the next choice is pretty direct: does the second region handle live traffic all the time, or only step in when the main region fails?

Before you decide, lock in one baseline that isn’t up for debate: both regions must have equivalent, deployable components ready before any incident happens. That includes code, configuration, access, secrets, data dependencies, observability, and infrastructure as code. If the second region is missing secrets, IAM policies, or service quotas, it is not ready for failover.

From there, the decision is simple in principle. The second region either serves live traffic, or it waits for failure.

When active-active gives you faster recovery

In an active-active setup, both regions serve production traffic at the same time. If one region goes down, the other keeps serving traffic without a separate promotion step. AWS characterises multi-region active-active as supporting near-zero RPO and potentially zero RTO.[2][1] If data is synchronised and traffic routing works as planned, recovery can be immediate.

That speed comes at a price: more moving parts.

Faster recovery only works if your data model can cope with concurrent writes and replayed events. You need to choose one write model:

• single-region ownership
• partitioned writes
• built-in conflict handling

Event consumers also need to be idempotent. During failover, retries and replayed messages are normal. Processing the same event twice must not create duplicate charges or duplicate records.

Scheduled functions need care as well. You’ll need either one elected regional scheduler or a distributed lock, so both regions don’t run the same job at the same time.

If that level of write coordination feels too expensive or too messy, active-passive is often the simpler route.

When active-passive keeps costs down

Active-passive sends production traffic through one region while the second region waits in reserve. That second region usually sits in one of three standby modes:

• Cold standby: code and infrastructure exist, but major resources still need to be created or configured during recovery.
• Warm standby: core APIs, functions, data stores, queues, and monitoring are deployed, though capacity may be lower.
• Hot standby: the second region is fully provisioned and kept up to date with replicated data, but it doesn’t usually serve user traffic.

The cost saving can be meaningful, but recovery stands or falls on automation. A passive region that hasn’t been tested is not a recovery plan. The only thing that matters is proven promotion time. Labels like warm or hot don’t mean much on their own.

Active-active vs active-passive: comparison table

Start with RTO and RPO, then test whether your data and event model can actually meet them. Active-active isn’t always the safer option. If your application can’t resolve concurrent writes or handle duplicate events with confidence, active-passive may give you a more dependable recovery result.

Next, define how data, queues, and scheduled jobs move under the model you choose.

Replicate data and route events across regions

Once you’ve chosen active-active or active-passive, the next step is simple to say and harder to get right: make sure state survives a regional outage.

DNS failover moves traffic. It does not move state. Your databases, queues and workers need to be in place in the recovery region before anything goes wrong. Replication keeps that state current. Backups help you restore it later.

Replicate application data with clear write rules

Start by listing every stateful part your application needs to serve a request: primary databases, object storage, configuration and secrets, schema migration state, and any durable event store. For each one, make an explicit replication decision.

The main choice is between synchronous and asynchronous replication.

With synchronous replication, a write is not acknowledged until it exists in another region. That can support near-zero RPO, but every write pays the price in cross-region latency.

With asynchronous replication, write latency is lower. But replication lag becomes your RPO. Track it, alert on it, and compare it with your target before an incident.

Alongside replication mode, define the write model for each data set. A single-writer model is usually the safest default for business records with strict ordering or complex rules. One region accepts writes, the other keeps a replica, and promotion happens only after the primary is confirmed unavailable or fenced off.

Multi-region writes make sense only when the data model has clear conflict semantics, such as append-only records or independent per-user ownership. Amazon DynamoDB Global Tables, for example, use last-writer-wins reconciliation for concurrent item updates, which means simultaneous writes to the same item can silently overwrite each other.[4][5] Azure Cosmos DB multi-region writes also need conflict-resolution rules for concurrent updates.[3] Use multi-writer only when those conflict rules line up with your business logic.

For object storage, turn on cross-region replication and versioning. Versioning gives you a way to recover an overwritten or deleted object without doing a full restore. Replicate metadata as well as the objects, and watch replication status, backlog age, failures and key errors. Your recovery procedure should also state which object version the application should use if replication had not finished at the moment of failover.

Make queues, topics and scheduled jobs recoverable

Event consumers in the recovery region need a durable copy of events. You can publish events to a replicated queue, mirror them with a provider feature, or write them first to a replicated event store. During recovery, resume from the last confirmed checkpoint. Don’t just look at whether a queue is empty or not. Check backlog age instead. Keep message IDs, correlation IDs, event type, schema version, creation time and ordering key on every message. Those fields are what make replay safe and auditable.

Expect duplicate delivery during failover. Every consumer needs to be idempotent. If the same event is processed twice, the result should be the same. Use stable event IDs, conditional writes and deduplication records. For ordered workflows, partition by entity key and process one partition at a time. Don’t assume global ordering across regions.

Some workloads are trickier. Both regions may receive the same event, but only one should act on it. In that case, use an active-region marker. Store one authoritative marker in a strongly consistent, or otherwise tightly protected, control store. Each worker checks that marker before processing and continues only if its region is active.

Pair this with a lease or fencing token that has expiry and renewal. If Region A fails and Region B takes over, Region A must not keep processing in the background. The marker on its own isn’t enough. Handlers still need idempotency keys and conditional state transitions, because a worker can start work just before losing its lease.

For scheduled jobs, deploy the schedule definition in both regions but gate execution with the active-region marker so only one region runs singleton work. Store the schedule, last run, next run and job parameters in replicated state. After promotion, decide what happens to missed runs: catch them up, skip them, or run them again. Use a deterministic job ID, such as the schedule name plus its intended execution time, and enforce it with a conditional write to stop duplicate effects from a replayed schedule.

Set up dead-letter queues in both regions, keep messages long enough for investigation, and include a controlled redrive process in your runbook that checks messages before replay.

Dependency mapping worksheet

Map every stateful and asynchronous dependency before an incident. Use one row for each database, bucket, queue, topic, event consumer, worker, scheduler, secrets store and external dependency.

Use the worksheet to show that every dependency has a replication path and a tested recovery action.

The worksheet should also record encryption keys, retention periods, ownership, dependencies outside the cloud provider, and the metric used to prove recovery. Any row without a tested replication method, failover action and validation method is incomplete.

With state replicated, the next step is to route users and monitor recovery.

Set up traffic failover and observability

With data replicated and events recoverable, the next job is making sure user traffic goes to the right region at the right moment - and knowing when that moment has actually arrived.

Configure DNS failover with realistic health checks

DNS failover changes the DNS answer when the primary region is judged unhealthy. In practice, you can use one hostname, such as api.example.co.uk, with routing records that send all traffic to the primary in an active-passive setup, or split traffic across both regions in active-active and remove the unhealthy region when needed.

A liveness check tells you the endpoint responds. A readiness check tells you the region can complete a real request. That distinction matters. A readiness endpoint should mark the region healthy only when it can handle a real request end to end.

To cut down false failovers, don’t act on a single failed probe. Route 53 marks an endpoint unhealthy after three failed probes.[8] That should sit alongside probes from multiple independent locations. One timeout on its own just isn’t enough proof to move traffic.

A low TTL helps, but it doesn’t solve the whole problem. AWS recommends 60 seconds or less for health-checked failover records,[6][7] yet recursive resolvers, operating system caches, clients and intermediary networks may all cache DNS answers on their own schedule. On top of that, long-lived connections might not look up DNS again until they reconnect. So TTL is only one part of the failover time budget, not a promise. A more grounded way to estimate switchover time is:

• detection
• DNS caching
• client reconnect delay

Those numbers need to be measured in your own setup, not guessed.

Routing only works if detection is quick enough to stay ahead of client caching and reconnect delays.

Monitor the right signals during a regional incident

Once routing can move, the next call is deciding when to move it.

A regional incident tends to show up in several places at once: rising HTTP 5xx error rates, climbing p95 and p99 latency, function throttling or concurrency saturation, dependency failures across databases, authentication and third-party calls, plus growing queue depth or event backlog age.

Replication lag and dead-letter queue growth can put RPO at risk even when user traffic still looks fine. That’s the trap. A region may keep serving reads while quietly falling behind on writes or asynchronous work - and that shortfall becomes your RPO issue the second failover is needed.

Send alarms to the right destination. High-confidence signals - such as sustained elevated errors, latency and throttling across multiple availability zones at the same time - should page the on-call engineer. That alert should link straight to the failover runbook, the current traffic split, health-check status and the decision rules for switching to the secondary region. Lower-confidence anomalies should land in an investigation queue instead of kicking off failover straight away.

Separate automated detection from automated failover. Start by automating detection and notification. Automate traffic movement only when the failure mode is well understood and the data model can handle it. If failover could lead to split-brain writes, duplicate event processing, stale reads, conflicting database updates or irreversible external side effects, require operator confirmation. If single-writer controls are already in place and the secondary region has been tested, automation makes sense. If not, an operator should decide before the secondary is allowed to write.

Create a regional incident dashboard with a common time basis. Label each signal by region, service, dependency and deployment version.

Run the failover, restore service and test the plan

Failover and failback runbook

Once detection and routing are set up, the runbook decides who does what, and when.

When incident signals fire, the first task is to confirm the fault is regional, not a single function, a bad deployment, or a short-lived dependency issue. Check provider status, regional health, and synthetic checks together. When the scope is clear, declare the incident, assign an incident commander, and stop deployments and configuration changes.

Before you reroute traffic, check the recovery region properly. That means quotas, concurrency limits, secrets, certificates, IAM permissions, and feature flags. Then check the data state: measure replication lag, identify the latest safe recovery point, and confirm which region has write authority.

If the recovery region passes those checks, cut over write authority first and traffic second. If replication still meets your RPO, promote writes in the recovery region and fence the failed writer so you don't end up with split-brain conflicts. Then switch traffic through DNS or global routing.

As traffic starts moving, bring back queue consumers, event subscriptions, and scheduled jobs in a controlled sequence. Replay eligible dead-letter messages with idempotency keys so the same action doesn't run twice. Then validate the user journeys that matter most, including:

• authentication
• writes
• reads
• payment or order flows

Only declare the region operational once those checks pass.

Failback should be treated as a planned migration, not an automatic snap-back. First, confirm the primary region has stayed stable for an agreed observation window. Then stop writes in the recovery region, sync accepted writes and events back, verify counts and checksums, run smoke tests, and shift traffic back in stages. Disable duplicate consumers and scheduled jobs in the recovery region only after the primary is confirmed as the authoritative region.

The runbook should also separate three recovery modes:

Test recovery against your RTO and RPO targets

Once recovery has been rehearsed, measure it against the RTO and RPO targets you set at the start.

A diagram doesn't prove you can recover. The only way to know is to run exercises that cover DNS failover, regional API loss, replication lag cases, queue replay, dead-letter redrive, secrets and permission checks, scheduled-job duplication, and controlled failback. Start in non-production environments or isolated traffic paths. Then move to limited production tests once the risk is understood.

After each exercise, record the actual numbers instead of marking it as pass or fail. The metrics that matter are detection time, decision time, failover time, data loss at cutover, replication lag, replay duration, and user-visible impact. Compare the restoration time with your RTO and document any gaps plainly. Then track corrective actions until they're done.

Conclusion: the minimum standard for serverless regional resilience

The minimum standard is straightforward: recovery must be repeatable, measured, and documented.

Regional resilience in a serverless system comes down to a small set of deliberate choices. Define RTO and RPO before you design anything. Choose active-active or active-passive with a clear view of the cost and complexity trade-off. Replicate data and events with clear write controls. Use DNS failover backed by strong observability.

But here's the part that tends to get skipped: none of that is enough unless the whole sequence, from failover to validation to failback, has been rehearsed and measured against actual targets.

Continuity depends on tested recovery, not diagrams.

FAQs