07/10/0206

The Primary Link Failed. The Backup Didn’t Take Over. Why?

The incident

The primary internet circuit drops, but users remain offline even though a backup link is installed. A spare circuit is not automatic resilience. The network must detect the failure, select the backup path, and have the routing and security configuration needed to carry traffic in both directions.

Why this symptom matters

Failover can fail at several stages. The interface may remain up while the provider path is broken; a health probe may continue to target a reachable device; a route may not be withdrawn; or the backup may lack NAT, DNS, or return routing. A single “link up” indicator does not prove that the full service is reachable.

What the evidence can tell you

Record the outage time, interface state, health-check results, active routes, provider alerts, and affected services. Determine whether only internet access failed or whether private services are also unreachable. Check if the backup circuit was ever tested with real user traffic rather than only a physical link check.

A practical troubleshooting approach

  1. Identify the actual failure point. Check whether the physical interface went down, the provider gateway stopped responding, or an upstream service became unreachable. A port can remain up while the provider path beyond it has failed. Compare local link status with probes to multiple stable destinations.
  2. Review failure detection. Inspect probe targets, thresholds, intervals, and failover timers. A health check aimed only at the directly connected gateway may miss a wider outage. Conversely, a probe that fails intermittently can cause unnecessary path changes.
  3. Confirm route selection. Check whether the primary route was removed or deprioritized and whether the backup route became active. Review tracking objects, metrics, priorities, and dynamic routing behavior. Confirm policy-based routing or static routes do not continue steering traffic to the failed link.
  4. Validate backup NAT and security policy. Make sure the backup interface has the correct source NAT, firewall rules, and DNS behavior. A route may switch successfully while outbound connections fail because the traffic retains an address that is not valid on the backup circuit.
  5. Check the return path. Verify that replies return through the backup link and are accepted by stateful devices. Provider source validation, asymmetric paths, or an incorrect default route can discard replies after outbound packets appear to leave.
  6. Run a controlled failover test. During an approved maintenance window, disconnect or simulate failure according to the procedure. Measure detection time, route change, service recovery, and failback behavior. Do not wait for a real outage to discover the backup configuration is incomplete.

Common mistakes to avoid

  • Testing failover by checking only whether the backup interface is physically up.
  • Using one probe target that may remain reachable even when the user-facing service is unavailable.
  • Failing to test NAT, DNS, and return traffic over the backup link.

How to verify the fix

After correcting the detection or path configuration, run a controlled test and confirm representative applications work over the backup circuit. Record how long failover takes and whether the primary path returns cleanly. Monitor route and session behavior during the test and verify that service stays stable afterward.

Plan for failback too

Resilience includes returning to the primary circuit. The network should avoid unstable switching if the primary link flaps, and the failback process should be tested for session impact. Record whether existing connections recover or need to be re-established.

Use more than one health signal

Where the design allows, health checks can evaluate multiple stable targets or a representative service. A single target may fail independently or remain available during a wider outage. Follow the approved design and avoid creating an overly sensitive probe.

Prevent the problem from returning

Schedule regular failover tests and document expected recovery times, probe targets, route behavior, and ownership. Monitor both circuit state and end-to-end service health. Keep NAT, firewall, and DNS configuration synchronized across primary and backup paths.

A deeper troubleshooting walkthrough

First confirm what “failed” means in the design. A physical interface can remain up while the provider path beyond it is unusable, and a single probe target can fail even though the circuit is healthy. Review the failover mechanism: interface state tracking, route tracking, dynamic routing, SD-WAN policy, or a provider-managed service. Compare the configured failure signal with the actual outage. If the system watches only carrier state, it may never detect a remote black hole.

Follow the decision chain from detection to forwarding. Did the health check run? Did it declare the primary path unhealthy? Was the backup route installed with a usable next hop? Did policy-based routing or a static route keep selected traffic on the failed link? Check timestamps for probes, route changes, and interface state transitions. A stale ARP entry or unresolved backup next hop can also make a route appear active while packets cannot leave the device.

Turn the diagnosis into a reliable fix

Test the backup path for the services users need, not only for basic reachability. Some applications depend on source addresses that are permitted only over the primary circuit, or on NAT rules that do not exist for the secondary path. Stateful sessions may not survive a change in public address, so existing connections can fail even when new sessions succeed. Separate “new connections work” from “existing sessions continue” when defining the expected behavior.

Include failback in the test plan. When the primary recovers, verify that monitoring confirms stability before traffic returns, and check whether route preference changes cause another interruption. Use clear hold-down or recovery behavior only as supported by the platform and design. Record which traffic is expected to use each path, who receives an alert, and how long a test should take. A controlled simulation during maintenance is safer and more informative than waiting for the next real outage.

Turn the first clues into a controlled test

Check whether the backup circuit is truly ready for use. A line can be physically up but lack provider-side routing, valid addressing, a functioning next hop, or the required security configuration. Test from the router using the intended source interface and from a user network after failover. Confirm that NAT, DNS forwarding, VPN termination, and critical service allowlists include the backup path where needed.

Understand the expected detection delay before calling it a failed failover. Health checks may use several missed probes or a hold-down timer to avoid reacting to brief packet loss. Compare the design’s documented thresholds with the actual probe logs and route change time. If detection is too slow or too permissive, adjust it only after assessing the effect of false positives and route flapping.

Schedule periodic failover exercises with the service owners. Test a small, agreed set of critical applications, document recovery time, and verify failback as well as failover. A backup link that has not been tested under controlled conditions may have stale credentials, expired certificates, or outdated firewall policy. Exercise results should create specific corrective work, not simply a checkbox that the circuit is present.

Finish with a repeatable user-level check

When both circuits appear healthy after the incident, do not assume the failover logic worked. The primary may have recovered before anyone inspected the backup, or monitoring may have missed a brief event. Compare provider timestamps with local route and probe history, and check whether traffic actually used the secondary path during the reported outage. If existing sessions were lost but new sessions worked, record that separately; it may be an expected limitation of changing public addresses rather than a failed route change.

Final thoughts

A backup circuit is useful only when the network can detect the right failure and carry a complete, bidirectional application session through the alternate path.

Frequently asked questions

Why can a circuit show up while the internet is down?

The physical interface may remain active even if the provider’s upstream path or a destination beyond the gateway is unavailable. Health checks should reflect the service users need.

How often should backup links be tested?

Follow the organization’s resilience policy and change windows. The important point is to test regularly enough that routing, NAT, security policy, and recovery timing remain known.

Why does traffic leave the backup link but not work?

The backup may lack correct NAT, firewall policy, DNS, or return routing. Check both directions and the translated addresses seen by the destination.

Should the network fail over on one failed ping?

Usually, a single missed probe can be too sensitive. Use an approved monitoring design with suitable targets and thresholds to balance quick detection against false failovers.

Join the discussion: When did you last test an actual failover?

For hands-on practice with real network troubleshooting, explore training with Atmic Networks.

SEO publishing details

SEO title: The Primary Link Failed. The Backup Didn’t Take Over. Why?

Meta description: Primary internet link failed and the backup did not take over? Check health detection, route priority, NAT, DNS, and the return path.

Focus keywords: network failover troubleshooting, backup internet link, primary link failure, WAN failover

URL slug: primary-link-failed-backup-did-not-take-over