Battle-tested: Troubleshooting intermittent BGP route flaps in large Cisco datacenter with Python and Netmiko

Published on January 27, 2026

Battle-tested: Troubleshooting intermittent BGP route flaps in large Cisco datacenter with Python and Netmiko

Intermittent bgp route flaps in a large Cisco datacenter are rarely “just BGP.” They’re usually symptoms of one (or more) of these: control-plane stress, intermittent physical errors, MTU/PMTUD issues, software defects, policy churn, or unintended automation changes. The goal of troubleshooting is to reduce time-to-triage: quickly scope blast radius, collect high-signal evidence, correlate to changes, and apply mitigations safely.

This post focuses on advanced, practical workflows using Python + Netmiko, selective streaming telemetry, and automation guardrails suitable for spine/leaf fabrics and enterprise edge interconnects.

Symptoms, impact assessment and hypothesis framing: scoping who/what is affected

What intermittent BGP flaps look like (in practice)

You may see:

  • BGP neighbor sessions oscillating Established → Idle/Active → Established
  • Route churn: prefixes repeatedly withdrawn/announced
  • Application symptoms: asymmetric routing, intermittent drops, increased latency
  • Control plane side effects: CPU spikes, BGP process restarts, RIB/FIB programming delays
  • Secondary protocol instability (e.g., underlay IGP churn, BFD flaps)

First 15 minutes: determine blast radius

Start with the smallest number of questions that drives the next action:

  1. Where is the flap happening?
  2. Single leaf? entire pod? all spines? only border leafs? only DCI/edge?
  3. Is it a session flap or route flap?
  4. Neighbor down/up (transport/control plane) vs only certain prefixes (policy/route quality)
  5. Is it external or internal?
  6. eBGP to upstreams/DCI vs iBGP RR mesh vs EVPN overlay sessions
  7. Is it tied to time or events?
  8. A pattern (top of hour), link errors, config pushes, failovers, backup jobs
  9. How many prefixes and which critical services?
  10. Default route? anycast? shared services? storage/backup networks?

Impact assessment checklist

  • Which sites/pods are affected (map devices to fabric/pod and role).
  • Which peers (neighbor IP, AS, device role, interface).
  • What’s flapping:
  • Session state changes (Down/Up)
  • Prefix churn (withdraw/announce)
  • Path attribute changes (MED/localpref/community)
  • Customer impact proxy metrics:
  • ECMP path changes rate
  • drops on border interfaces
  • SLA/health checks failing correlated to flap timestamps

Hypothesis framing: build a short list before collecting everything

Use early symptoms to bias data collection:

  • Session resets with “Hold Timer Expired” → transport loss, CPU starvation, policing, MTU/PMTUD, microbursts on control-plane, linecard issues.
  • “Cease/Administrative reset” → automation changes, policy updates, neighbor config churn, session resets.
  • Only certain prefixes flap → route-map/policy changes, prefix-limit hits, dampening, RPKI/RTBH triggers, next-hop reachability changes.
  • BGP stable but traffic breaks → FIB programming, TCAM pressure, ECMP inconsistency, ARP/ND issues, VXLAN/EVPN control/data plane mismatch.

Fast data collection: Netmiko + telemetry to centralize BGP evidence

The key is to collect high-signal evidence quickly and consistently, then store it in a centralized place (files, object storage, time-series DB). Avoid “SSH-ing around” manually when the issue is intermittent.

What to pull from Cisco devices (minimum viable set)

For each affected device (and a couple adjacent devices for comparison), collect:

BGP state and reasons
- show bgp ipv4 unicast summary (or relevant AFI/SAFI: EVPN, vpnv4)
- show bgp ipv4 unicast neighbors <peer> | inc BGP state|Last reset|Hold|Keepalive|Notification|Last read|Last write
- show bgp ipv4 unicast neighbors <peer> received-routes (if soft-reconfig inbound enabled)
or show bgp ipv4 unicast neighbors <peer> routes / advertised-routes depending on platform

Underlying transport and interface health
- show interface <int> counters errors
- show interface <int> | inc line protocol|MTU|reliability|rxload|txload
- show ip cef <peer-ip> detail (next-hop resolution)
- ping <peer-ip> repeat 100 size <mtu-test> df-bit (when safe)

Control plane health
- show process cpu sorted | ex 0.00%__0.00%__0.00%
- show platform hardware capacity / TCAM utilization where available
- CoPP statistics if configured (platform-specific)

Logs
- show logging last 200 (or better: stream syslog centrally)

Netmiko collection pattern (practical and safe)

Below is a workable Python approach that:
- runs read-only commands,
- timestamps outputs,
- stores per-device bundles,
- and stays resilient to intermittent SSH issues.

python 
#!/usr/bin/env python3
import os
import json
import time
from datetime import datetime, timezone
from netmiko import ConnectHandler
from netmiko.ssh_exception import NetmikoTimeoutException, NetmikoAuthenticationException

COMMANDS = [
    "show clock",
    "show bgp ipv4 unicast summary",
    "show bgp ipv4 unicast neighbors | include BGP neighbor|BGP state|Last reset|Notification|hold time|keepalive",
    "show logging last 200",
    "show process cpu sorted | exclude 0.00%__0.00%__0.00%",
]

# Add per-interface commands dynamically when you know the peer interfaces
INT_COMMANDS_TMPL = [
    "show interface {intf} | include line protocol|MTU|reliability|rxload|txload|input error|CRC|overrun|ignored|output error|collisions",
    "show interface {intf} counters errors",
]

def collect_device(host, username, password, device_type="cisco_nxos", extra_cmds=None):
    ts = datetime.now(timezone.utc).isoformat()
    record = {"host": host, "timestamp_utc": ts, "device_type": device_type, "commands": {}}

    conn = None
    try:
        conn = ConnectHandler(
            device_type=device_type,
            host=host,
            username=username,
            password=password,
            fast_cli=False,          # more reliable under load
            conn_timeout=15,
            banner_timeout=15,
            auth_timeout=15,
            session_timeout=60,
        )

        # Avoid paging differences across platforms
        try:
            conn.send_command("terminal length 0")
        except Exception:
            pass

        cmds = COMMANDS + (extra_cmds or [])
        for cmd in cmds:
            out = conn.send_command(cmd, read_timeout=60)
            record["commands"][cmd] = out

    except (NetmikoTimeoutException, NetmikoAuthenticationException) as e:
        record["error"] = f"{type(e).__name__}: {e}"
    except Exception as e:
        record["error"] = f"Unhandled: {type(e).__name__}: {e}"
    finally:
        if conn:
            try:
                conn.disconnect()
            except Exception:
                pass

    return record

def write_bundle(out_dir, data):
    os.makedirs(out_dir, exist_ok=True)
    fname = f"{data['host']}_{data['timestamp_utc'].replace(':','_')}.json"
    path = os.path.join(out_dir, fname)
    with open(path, "w") as f:
        json.dump(data, f, indent=2)
    return path

if __name__ == "__main__":
    # Example inventory — in practice, load from NetBox/CMDB
    devices = [
        {"host": "10.0.10.11", "device_type": "cisco_nxos"},
        {"host": "10.0.10.12", "device_type": "cisco_nxos"},
    ]

    username = os.environ.get("NET_USER")
    password = os.environ.get("NET_PASS")
    out_dir = "./bgp_flap_bundles"

    for d in devices:
        rec = collect_device(d["host"], username, password, d["device_type"])
        path = write_bundle(out_dir, rec)
        print(f"Wrote {path}")
        time.sleep(0.5)  # light pacing to avoid self-inflicted control-plane load

Notes that matter in large fabrics

  • Pacing is not optional: blasting dozens of devices with SSH commands can worsen a control-plane issue and create false symptoms.
  • Favor read-only commands; never “clear bgp” during initial triage.
  • Use a central inventory (NetBox/CMDB) to group by pod/role and to identify the correct neighbor/interface.

Telemetry: stream the churn, don’t poll blindly

Polling is useful for snapshots; intermittent flaps need continuous time-series.

Options commonly deployed:
- gNMI/gRPC telemetry for interface counters, BGP neighbor state (platform support varies)
- Syslog to a central collector (most universally available)
- SNMP counters as a fallback (less detail, but ubiquitous)

A pragmatic hybrid:
- Use syslog as the “event trigger”
- When “BGP neighbor down” is observed, run the Netmiko bundle collector against:
- local device
- its neighbor
- and 1–2 upstream/downstream devices for context

Automated analysis with Python: correlate BGP evidence with changes and triggers

Once you have bundles, automate the “boring but critical” steps:
- parse neighbor resets and reasons,
- identify repeated patterns (same peer, same reason, same interval),
- correlate timestamps with config changes in git.

What to extract for prioritization

From show bgp neighbors / logs:
- Neighbor IP, AS, VRF/AFI
- Last reset time, last reset reason, notification codes/subcodes
- Hold/keepalive timers, BFD status (if used)
- Number of flaps per interval (last hour/day)

From route views:
- Prefixes most churned
- AS-path anomalies (unexpected prepends, route leaks)
- Next-hop changes and reachability

From platform data:
- Interface CRC/input errors aligning with flap time
- CPU spikes aligning with hold timer expiry
- CoPP drops aligning with BGP packet loss

Example: simple log correlation and “top talkers” for flaps

If your bundles include show logging last 200, you can mine the obvious patterns quickly. This is intentionally simple—enough to prioritize investigation.

python 
import json
import glob
import re
from collections import Counter, defaultdict
from datetime import datetime

NEIGH_DOWN_PAT = re.compile(r"BGP.*neighbor\s+(\S+)\s+.*(Down|down)", re.IGNORECASE)
NEIGH_UP_PAT   = re.compile(r"BGP.*neighbor\s+(\S+)\s+.*(Up|up|Established)", re.IGNORECASE)

def parse_time_prefix(line):
    # Varies by platform/log config; keep flexible and rely on bundle timestamp when needed
    return line

def analyze_bundles(path_glob="./bgp_flap_bundles/*.json"):
    flaps = Counter()
    per_host = defaultdict(Counter)

    for fname in glob.glob(path_glob):
        with open(fname) as f:
            rec = json.load(f)

        host = rec.get("host")
        log = rec.get("commands", {}).get("show logging last 200", "")
        for line in log.splitlines():
            m = NEIGH_DOWN_PAT.search(line)
            if m:
                nbr = m.group(1)
                flaps[(nbr, "down")] += 1
                per_host[host][(nbr, "down")] += 1
            m = NEIGH_UP_PAT.search(line)
            if m:
                nbr = m.group(1)
                flaps[(nbr, "up")] += 1
                per_host[host][(nbr, "up")] += 1

    print("Top flap-related log hits:")
    for (nbr, state), count in flaps.most_common(20):
        print(f"{nbr:>16} {state:>4}  {count}")

    print("\nPer-host hot spots:")
    for host, c in per_host.items():
        top = c.most_common(3)
        if top:
            print(host, top)

if __name__ == "__main__":
    analyze_bundles()

Correlate with config changes (git history)

If device configs are generated and stored in git (common in modern network automation), correlate flap windows with merges/deploys:

  • Identify deployment timestamp (pipeline logs, change tickets).
  • Compare “good known” vs “current” config:
  • neighbor settings (timers, BFD, update-source)
  • route-maps/prefix-lists (ordering and match conditions)
  • max-prefix thresholds
  • next-hop-self/route-reflector changes
  • EVPN/VRF import/export RTs

A practical gating question:
- Did the change increase churn potential? Example triggers:
- new policy that causes widespread re-advertisement (community rewrite)
- lowering max-prefix threshold close to actual received prefixes
- changing timers unevenly between peers
- enabling/disabling BFD asymmetrically

Prioritized alerts that engineers can act on

Avoid alert floods. Generate alerts like:

  • P1: Sessions down to border/DCI with “Hold Timer Expired” + interface errors rising
  • P1: Prefix-limit hit on critical peer (default route/provider)
  • P2: Route churn above threshold for specific prefixes (anycast VIPs)
  • P2: BGP stable but repeated bestpath changes (policy/attr thrash)
  • P3: Cosmetic (up/down logs without route impact)

Short-term mitigations and automated fixes: safe playbooks and execution patterns

Mitigation is about stopping the bleeding while preserving evidence. Keep actions reversible, scoped, and verified.

Safe execution patterns (non-negotiable in large fabrics)

  • Use two-phase changes:
    1) pre-check snapshot (state, routes, counters)
    2) change
    3) post-check snapshot + rollback if regression
  • Gate actions on conditions (don’t apply dampening everywhere “just in case”).
  • Prefer local changes over global policy modifications during an incident.
  • Always log: who/what/when/why, and store before/after outputs.

Common mitigations (choose based on evidence)

1) Route dampening (use carefully)

Helps when a neighbor is sending unstable routes, but can mask real issues and delay convergence. Use when:
- a peer is flapping many prefixes and impacting internal stability
- and you need to protect the fabric temporarily

Validate platform support and current policy; NX-OS/IOS-XE behavior differs.

2) Session reset (targeted and justified)

If the session is wedged (bad state machine, stuck attributes), a controlled reset can restore service.
- Prefer soft reset if supported (clear bgp ... soft in/out)
- Hard reset only when evidence suggests it’s necessary

3) Prefix-limits with sensible thresholds

Use to prevent table overflows or route leaks, but avoid thresholds too close to current counts.

4) Rapid config rollback (best when change-induced)

If correlation strongly suggests a recent policy change triggered churn, rollback can be the safest fix—especially if the change was broad.

Example Ansible play: pre-check → soft clear → verify

This is a simplified pattern; tailor commands to IOS-XE vs NX-OS and your AFI/VRF.

yaml 
---
- name: Mitigate intermittent BGP route flaps (targeted soft reset)
  hosts: border_leafs
  gather_facts: no
  serial: 5

  vars:
    bgp_neighbor_ip: "192.0.2.10"
    afi_cmds:
      - "show bgp ipv4 unicast summary"
      - "show bgp ipv4 unicast neighbors {{ bgp_neighbor_ip }} | include BGP state|Last reset|Notification|hold time|keepalive|Last read|Last write"
      - "show logging last 50"

  tasks:
    - name: Pre-check snapshot
      cisco.nxos.nxos_command:
        commands: "{{ afi_cmds }}"
      register: pre

    - name: Save pre-check to controller
      copy:
        content: "{{ pre.stdout | to_nice_yaml }}"
        dest: "./artifacts/{{ inventory_hostname }}_precheck.yml"

    - name: Soft clear inbound (minimize disruption)
      cisco.nxos.nxos_command:
        commands:
          - "clear ip bgp {{ bgp_neighbor_ip }} soft in"
      when: "'Established' in (pre.stdout[1] | default(''))"

    - name: Post-check snapshot
      cisco.nxos.nxos_command:
        commands: "{{ afi_cmds }}"
      register: post

    - name: Save post-check to controller
      copy:
        content: "{{ post.stdout | to_nice_yaml }}"
        dest: "./artifacts/{{ inventory_hostname }}_postcheck.yml"

    - name: Fail if session is not Established after mitigation
      fail:
        msg: "BGP neighbor not Established after soft clear; consider rollback/escalation."
      when: "'Established' not in (post.stdout[1] | default(''))"

Operational guardrails to add in real deployments
- Require a change ticket ID variable and embed it in artifact paths.
- Add per-neighbor route-count checks (don’t soft clear if it will trigger massive churn).
- Add rate limiting (serial + pause).
- Add platform-specific modules/commands for IOS-XE vs NX-OS vs IOS-XR.

Post-mortem hardening: validate root cause and prevent repeat incidents

Once the incident is stable, convert findings into durable controls.

Root-cause validation (evidence-based)

Build a timeline that includes:
- first observed flap (syslog/telemetry)
- interface counter inflections (CRC, drops)
- CPU/CoPP anomalies
- BGP notifications and reset reasons
- config/policy changes (git commit IDs + deployment logs)
- external provider events (if eBGP)

Then test the primary hypothesis:
- If it’s transport: reproduce with MTU tests, optical swaps, error counter monitoring under load, verify LACP member consistency.
- If it’s control-plane policing: confirm CoPP drops align with flap times; adjust policing carefully and monitor.
- If it’s policy churn: run policy diffs and route simulations; validate match conditions and ordering; add pre-deploy checks.
- If it’s a software defect: collect “show tech”/core evidence, isolate version, engage vendor TAC with precise artifacts.

Monitoring changes: BGP health checks that actually help

Move from “BGP neighbor is Established” to “BGP is healthy”:

  • Neighbor state + uptime
  • Keepalive/hold timer drift, last read/write
  • Prefix counts (received/accepted/advertised) with thresholds and rate-of-change
  • Route churn rate (withdraw/announce per minute)
  • Bestpath change rate for critical prefixes
  • Control-plane CPU + CoPP drops correlated to BGP events
  • Interface error deltas (not absolute counters)

CI/CD gating for control-plane changes

If configs are generated and deployed automatically, add gates for BGP-impacting changes:

  • Block merges that:
  • lower max-prefix below current received count + safety margin
  • change route-map ordering without explicit approval
  • modify communities/RTs widely (EVPN/VRF leakage risk)
  • change timers or BFD without matching peer compatibility notes
  • Require route-policy unit tests:
  • offline “expected routes in/out” for key neighbors
  • AS-path and community assertions
  • Canary deployment:
  • push to a single pod/edge first, validate churn metrics, then expand

Runbooks: codify decisions, not just commands

A good BGP flap runbook includes:
- “If Hold Timer Expired + interface errors → check optics/MTU/CoPP”
- “If Administrative reset coincides with pipeline deploy → rollback procedure”
- “If prefix-limit hit → identify offending prefixes and apply temporary filters”
- “When to escalate to provider/vendor and what artifacts to attach”

Common pitfalls

  • Treating all flaps as BGP problems: many are transport, ASIC/linecard, policing, or CPU scheduling issues.
  • Over-polling during an incident: aggressive SSH polling can amplify control-plane load and create misleading symptoms.
  • Ignoring timestamp consistency: device clocks drift; correlate using a central time source (NTP) and record collection timestamps in UTC.
  • Applying dampening globally: can hide the problem and create long recovery times.
  • Hard-clearing BGP too early: wipes evidence and can cause widespread reconvergence churn.
  • Prefix-limit misconfiguration: thresholds too tight trigger self-inflicted outages during normal growth or provider changes.
  • No config provenance: without git history and deployment logs, correlation becomes guesswork.

Key takeaways

  • Troubleshooting intermittent bgp route flaps is a scoping + evidence + correlation problem before it’s a “fix” problem.
  • Automate fast, repeatable data capture with Netmiko and store bundles centrally; use syslog/telemetry as event triggers.
  • Prioritize analysis: reset reasons, churn rate, prefix-impact, and correlation with interface/copp/cpu signals.
  • Mitigations should be targeted, reversible, and verified (pre-check/post-check), ideally executed via controlled playbooks.
  • Post-mortem hardening (health checks, CI/CD gates, policy tests, canaries) prevents a repeat—and makes the next incident shorter and safer.
Back to Blog