Battle-tested troubleshooting for BGP route flaps between Kubernetes egress and ISP networks with Python automation

BGP route flaps at the boundary between Kubernetes egress and ISP peers are rarely “just BGP.” They’re usually an interaction between routing policy, node/kernel behavior, CNI dataplane changes, and real physical circuit issues. The fastest path to root cause is disciplined symptom mapping plus automated, timestamped evidence collection across all domains.

This post focuses on advanced, practical troubleshooting with Python automation and correlation techniques that work whether your edge is Cisco/IOS-XE/IOS-XR, FRR, or a mix—feeding Kubernetes egress (e.g., eBPF CNI, kube-proxy, or dedicated egress gateways).

Symptom matrix: mapping observable failures to likely sources (CNI, node kernel, peering, or ISP circuits)

Use a symptom matrix to avoid chasing the wrong layer. Treat each symptom as a signal and look for clusters.

Symptom matrix

High-signal checks (do these early)

• Is the BGP session actually dropping?
• If session resets: focus on transport, timers, BFD, circuit.
• If session stable but routes churn: focus on policy/attributes/next-hop.
• Is impact node-scoped or cluster-wide?
• Node-scoped issues often implicate CNI/kernel/host networking.
• Are interface errors/discards rising during events?
• Physical or L2 problems surface here first.
• Does the ISP see the same?
• If ISP sees your neighbor bouncing from their side too, look at circuit/L2/transport.

Automated data collection: Python scripts to pull BGP state from Cisco/FRR, kube-proxy/CNI metrics, node syslogs and ISP MRT/RIB snapshots

The goal is repeatable capture of:
1) BGP neighbor state + reasons,
2) route update/withdraw timelines,
3) node networking signals (drops, conntrack, MTU),
4) Kubernetes events (rollouts, node readiness),
5) ISP routing perspective (MRT/RIB if you can get it).

Data to collect (minimum viable “flap bundle”)

Network edge / routing daemon
- Neighbor state transitions (Established/Idle/Active)
- BGP notifications / last reset reason
- BFD session state (if used)
- Prefix counts, max-prefix events
- Update/withdraw logs (if available)
- Interface counters on links to ISP (errors, flaps)

Kubernetes
- Node status transitions (Ready/NotReady)
- CNI daemon events (restarts, upgrades)
- kube-proxy restarts/mode changes
- Egress gateway/policy changes (CRDs if applicable)

Host / kernel
- dmesg around NIC resets
- conntrack -S (or nf_conntrack stats)
- ip -s link, ethtool -S for NIC-level errors
- tc -s qdisc (if shaping)
- CPU softirq spikes (network processing issues)

ISP / external
- If you can obtain: MRT/RIB snapshots for your prefixes or their view
- Otherwise: use public route collectors (limited) or ISP-provided graphs/logs

Correlation techniques: joining BGP updates, kube events and interface counters by timestamp/ifindex/AS-path to isolate root cause

You’re trying to answer:

• What changed first? (leading indicator)
• Is it control-plane (BGP) or data-plane (drops/latency)?
• Is it localized (one node / one link) or systemic?
• Do route attributes (AS-path, communities, MED) change with the flap?

Practical correlation keys

• Timestamp alignment: normalize everything to UTC and seconds (or milliseconds if you have it).
• ifindex / interface mapping: tie interface counters to the exact physical/logical interface carrying the peer.
• Neighbor IP + VRF: avoid mixing peers across VRFs/AFI-SAFI.
• Prefix + path attributes: track a “flapping prefix set” and compare stable prefixes.
• Node identity: nodeName + NIC name + PCI ID (if you suspect hardware resets).
• Event IDs: Kubernetes object UID for rollouts; device syslog message IDs.

A workable approach

1. Build an incident timeline
2. First customer-visible symptom (latency/drops)
3. First BGP event (update/withdraw/session change)
4. First interface anomaly (errors, link flap)

5. First kube/node/CNI event (pod restarted, DS rollout, node NotReady)

6. Classify the flap

7. Session flap: neighbor transitions; TCP reset; Hold timer; BFD down
8. Route flap: repeated withdrawals/announcements; attribute churn

9. Forwarding flap: stable routes but traffic loss

10. Confirm directionality

11. Is your edge withdrawing? Is the ISP withdrawing? Or is next-hop resolution changing?

Mitigation recipes: automated short-term fixes (route-filter adjustments, traffic-engineering, BGP dampening) and safe rollback steps

Mitigations should be:
- Reversible
- Scoped
- Auditable
- Time-bound (avoid “temporary” fixes that become permanent)

Recipe 1: Stabilize by steering traffic away (traffic-engineering)

Use communities/MED/local-pref (depending on relationship) to shift traffic to a stable link/POP.

• Inbound traffic shift (how others reach you): ISP communities to reduce preference on the flapping circuit.
• Outbound traffic shift (how you reach others): adjust local-pref on your edge to prefer alternate upstream.

Rollback plan:
- Keep a saved policy version
- Apply changes in a maintenance window if possible
- Verify route propagation (prefix + attributes) from at least two looking glasses / collectors

Recipe 2: Reduce route churn impact with dampening (cautiously)

BGP dampening can hide a real fault and prolong recovery if mis-tuned. Use it only when:
- A subset of prefixes is flapping (often due to redistribution instability), or
- You need to protect the control plane during an ongoing incident.

Rollback plan:
- Explicitly remove dampening config
- Clear dampened routes (platform-dependent)

Recipe 3: Tighten route filtering to stop pathological updates

If churn is caused by accidental redistribution (e.g., many host routes leaking), apply prefix-lists/route-maps to:
- Limit advertised prefixes to expected aggregates
- Prevent unintended next-hop changes
- Enforce max-prefix with alerting (not immediate teardown unless agreed)

Rollback plan:
- Apply filters in “permit expected first” order
- Validate Adj-RIB-Out before/after
- Have a console/OOBC path ready in case you lock yourself out

Recipe 4: Stabilize transport/timers (BFD/Hold-time sanity)

If circuit quality is marginal:
- Increase BFD timers or disable BFD temporarily (if business impact from false positives is worse than slower failover)
- Increase BGP keepalive/hold timers (less sensitive to transient loss)

Rollback plan:
- Document original timers
- Revert once circuit health is proven

Hardening and prevention: postmortem checklist, tests to simulate flaps in a lab, and continuous automation to detect and prevent recurrence (integration with NetBox inventory)

Postmortem checklist (routing + Kubernetes boundary)

Routing policy
- [ ] Was any route-map/prefix-list/community list changed near the incident?
- [ ] Were there any RPKI validation state changes (valid/invalid/not-found)?
- [ ] Did max-prefix thresholds trigger?
- [ ] Were next-hop tracking or recursive route dependencies unstable?

Transport/circuit
- [ ] Any interface flaps, CRC errors, or FCS increments?
- [ ] Any LACP member churn?
- [ ] Any optics alarms (LOS/LOF)?
- [ ] Any BFD down events without interface down?

Kubernetes / node
- [ ] Any CNI DaemonSet rollout, restart, or config change?
- [ ] Any node NotReady events or kubelet restarts?
- [ ] Any conntrack table pressure or drops?
- [ ] Any MTU mismatch introduced (overlay/underlay)?

Automation
- [ ] Did an automation job push routing config or restart agents?
- [ ] Were config diffs captured and approved?

Lab tests to simulate flaps (safely)

• Route churn simulation
• Inject/withdraw test prefixes from FRR (containerized) at controlled intervals.
• Validate dampening/policy behavior without impacting production.
• Session flap simulation
• Introduce controlled packet loss on the BGP TCP session using tc netem.
• Test BFD sensitivity and hold timer behavior.
• Kubernetes egress disruption
• Roll CNI DaemonSet versions in a staging cluster while monitoring BGP and dataplane drops.
• Simulate node NIC reset (where supported) or link down to verify failover.

Continuous detection and prevention

• Event-driven capture: when flap detected, automatically gather “flap bundle” artifacts for the last N minutes.
• Golden intent: validate that advertised prefixes match NetBox inventory (expected aggregates, allowed communities, peer ASNs).
• Baselines and SLOs:
• Allowed UPDATE rate per peer
• Max acceptable withdrawals/minute
• Interface error rate thresholds

Code: Python “flap bundle” collector (BGP + Kubernetes + syslog timeline)

The snippet below shows a practical pattern:

• Pull BGP neighbor summary from FRR (vtysh) and Cisco-like CLI via SSH
• Pull Kubernetes events for network-related objects
• Normalize timestamps and write a single JSONL timeline for correlation

Notes:
- In production, prefer structured APIs: gNMI/NETCONF/RESTCONF, FRR’s vtysh JSON outputs (where available), and Kubernetes API with field selectors.
- Store artifacts immutably (object storage) with an incident ID.

#!/usr/bin/env python3
import json
import subprocess
import datetime as dt
from pathlib import Path

import paramiko
from kubernetes import client, config

UTC = dt.timezone.utc

def now_utc():
    return dt.datetime.now(tz=UTC)

def iso(ts):
    if isinstance(ts, str):
        return ts
    return ts.astimezone(UTC).isoformat()

def run(cmd):
    p = subprocess.run(cmd, text=True, capture_output=True)
    return p.returncode, p.stdout.strip(), p.stderr.strip()

def frr_bgp_summary():
    # Works on a node/VM running FRR locally
    rc, out, err = run(["vtysh", "-c", "show bgp summary"])
    return {"source": "frr", "cmd": "show bgp summary", "rc": rc, "stdout": out, "stderr": err}

def ssh_cmd(host, username, password, cmd):
    c = paramiko.SSHClient()
    c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    c.connect(hostname=host, username=username, password=password, look_for_keys=False, allow_agent=False, timeout=10)
    stdin, stdout, stderr = c.exec_command(cmd)
    out = stdout.read().decode(errors="replace")
    err = stderr.read().decode(errors="replace")
    c.close()
    return {"source": "ssh", "host": host, "cmd": cmd, "stdout": out.strip(), "stderr": err.strip()}

def k8s_network_events(since_minutes=60, namespaces=None):
    # Requires kubeconfig or in-cluster config
    try:
        config.load_kube_config()
    except Exception:
        config.load_incluster_config()

    v1 = client.CoreV1Api()
    since = now_utc() - dt.timedelta(minutes=since_minutes)

    # Pull broad events then filter; for large clusters, use field selectors and paging.
    evs = v1.list_event_for_all_namespaces().items

    interesting = []
    for e in evs:
        # event_time is often None; fallback to last_timestamp/first_timestamp
        ts = e.event_time or e.last_timestamp or e.first_timestamp
        if not ts:
            continue
        ts = ts.replace(tzinfo=UTC) if ts.tzinfo is None else ts
        if ts < since:
            continue

        msg = (e.message or "").lower()
        involved = (e.involved_object.kind or "").lower()

        if any(k in msg for k in ["cni", "calico", "cilium", "kube-proxy", "iptables", "ipvs", "conntrack", "mtu", "route"]):
            interesting.append({
                "ts": iso(ts),
                "ns": e.metadata.namespace,
                "type": e.type,
                "reason": e.reason,
                "involved_kind": e.involved_object.kind,
                "involved_name": e.involved_object.name,
                "message": e.message
            })
        elif involved in ["node", "pod", "daemonset"]:
            # include node readiness and relevant infra objects
            if any(k in msg for k in ["notready", "ready", "reboot", "restart", "failed"]):
                interesting.append({
                    "ts": iso(ts),
                    "ns": e.metadata.namespace,
                    "type": e.type,
                    "reason": e.reason,
                    "involved_kind": e.involved_object.kind,
                    "involved_name": e.involved_object.name,
                    "message": e.message
                })
    return interesting

def write_jsonl(path, records):
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("w") as f:
        for r in records:
            f.write(json.dumps(r) + "\n")

def main():
    incident_id = now_utc().strftime("bgp-flap-%Y%m%dT%H%M%SZ")
    outdir = Path("artifacts") / incident_id
    timeline = []

    # 1) FRR local snapshot
    timeline.append({"ts": iso(now_utc()), "kind": "bgp_snapshot", **frr_bgp_summary()})

    # 2) Cisco-like BGP neighbor snapshot (example commands; adjust to platform)
    # Keep commands read-only; avoid "clear bgp" in automation.
    edge = {"host": "198.51.100.10", "username": "netops", "password": "REDACTED"}
    for cmd in [
        "show ip bgp summary",
        "show ip bgp neighbors | include BGP state|Last reset|Last read|Last write|Notification",
        "show interfaces counters errors",
        "show logging | include BGP|BFD|LINEPROTO|LINK-"
    ]:
        timeline.append({"ts": iso(now_utc()), "kind": "device_cli", **ssh_cmd(edge["host"], edge["username"], edge["password"], cmd)})

    # 3) Kubernetes events around networking
    for ev in k8s_network_events(since_minutes=120):
        timeline.append({"ts": ev["ts"], "kind": "k8s_event", "event": ev})

    # Sort by timestamp for quick inspection
    timeline.sort(key=lambda x: x["ts"])

    write_jsonl(outdir / "timeline.jsonl", timeline)
    print(f"Wrote {len(timeline)} records to {outdir}/timeline.jsonl")

if __name__ == "__main__":
    main()

How to use this in practice:

• Run it on-demand during an incident, or trigger it automatically when monitoring detects:
• neighbor state change
• withdrawals/minute above threshold
• interface errors jump
• Store output with an incident ID and attach it to your postmortem.

Common pitfalls

• Assuming a BGP flap implies an ISP fault: many “ISP flaps” are local control-plane overload, BFD over-aggressiveness, or LACP member churn.
• Not distinguishing session vs route churn: a stable session with high UPDATE volume points to policy/next-hop instability, not a dead circuit.
• Ignoring MTU and PMTUD: MTU blackholes can look like routing issues when sessions drop due to retransmits/timeouts.
• Collecting data without timestamps: evidence without precise time alignment makes correlation guesswork.
• Overusing dampening: can mask real faults and prolong convergence; tune carefully and remove when stable.
• Missing Kubernetes rollouts: CNI/kube-proxy changes can alter SNAT/ECMP behavior and trigger path changes that look like BGP instability.
• No immutable artifact storage: without saved “flap bundles,” every recurrence starts from scratch.

Key takeaways

• BGP flaps at the Kubernetes-to-ISP boundary often involve multiple layers: routing policy, kernel dataplane, CNI behavior, and physical circuits.
• The fastest path to root cause is an automated, timestamped multi-domain evidence bundle (BGP state, interface counters, syslogs, kube events).
• Correlate using time, interface identity (ifindex/name), and route attributes (AS-path/communities/next-hop) to isolate what changed first.
• Mitigations should be reversible and scoped: traffic-engineering, careful filtering, sane timers, and cautious dampening.
• Prevent recurrence with lab simulations, postmortem checklists, and continuous validation against inventory/intent (e.g., NetBox-backed expected prefixes/peers).