Route Debugging

Companies: CSPs, Tier 1s, Wholesale Carriers

Roles: Network engineers, traffic engineers, IP/MPLS architects, peering coordinators, NOC leads

Internet Service Providers (ISPs) frequently rely on multiple direct and indirect interconnects to exchange traffic with major content providers and hyperscalers. In an ideal world, traffic takes the most efficient path (usually the direct interconnect) to ensure optimal latency, cost efficiency, and user experience. In reality, routing decisions are influenced by a combination of BGP policies, network topology, and remote peer behaviour. This can lead to situations where traffic is unexpectedly sent via indirect or transit routes even when a direct path exists. Such mismatches between the ‘theoretical’ best route and the ‘actual’ forwarding path often go unnoticed until they cause measurable performance degradation, increased costs, or customer complaints.

A European incumbent operator noticed that about 25% of traffic to a hyperscaler was routed via indirect paths, which is unusual for a peer with multiple direct interconnects.

• A lack of correlation between BGP visibility and actual forwarding paths.

• Standard routing diagnostics failed to identify the root cause.

The peering manager flagged the anomaly and sought to understand the discrepancy.

Fortunately, BENOCS Analytics had been deployed in the network since 2015, allowing for a deep dive into historical and real-time data.

1. Looking Glass – Checked all hyperscaler-announced prefixes:
   • 757 prefixes had a direct path
   • 910 prefixes had an indirect path
   • On its own, this didn’t reveal much, so traffic data was added.
2. Raw Network Analyzer (RNA) – Narrowed focus to prefixes actively carrying traffic:
   • Only 249 of 1,667 advertised prefixes carried outgoing traffic.
   • This allowed 1,447 unused prefixes to be excluded from further analysis.
3. Flow Explorer – Correlated ingress and egress flows for complete visibility:
   • Of the 249 active prefixes:
     • 60% (150 prefixes) used direct routes
     • 40% (100 prefixes) used indirect (transit) routes
   • Combining inbound and outbound views revealed symmetrical and asymmetrical paths, along with traffic volumes.

This combined view pinpointed the mismatch between theoretical BGP paths and real traffic flows.

• Identification of problematic prefixes
• Data-driven discussions with the hyperscaler could be established to improve path symmetry
• A recommendation for establishing PNI in the US for optimal delivery to affected destinations
• Reduced troubleshooting time thanks to correlation of multiple data layers within a single platform

BENOCS Analytics enables operators to troubleshoot asymmetric routing issues faster by correlating flow-level data with BGP visibility. The combination of Flow Explorer, Looking Glass, and RNA gives engineering teams a comprehensive toolset to resolve complex forwarding inconsistencies and optimize traffic delivery.