Checklist: Troubleshooting Kubernetes-to-Cisco BGP Peering in a Leaf-Spine Datacenter with Ansible Automation

Published on January 28, 2026

Checklist: Troubleshooting Kubernetes-to-Cisco BGP Peering in a Leaf-Spine Datacenter with Ansible Automation

This checklist targets kubernetes bgp troubleshooting when Kubernetes nodes (or a CNI/daemon on nodes) peer via BGP with Cisco leaf switches in a leaf–spine datacenter. It assumes you’re advertising Service VIPs (LoadBalancer IPs) and/or Pod CIDRs into the fabric, typically to EVPN/VXLAN or routed underlay.

Environment and assumptions checklist (align the design before debugging)

1) Identify which component is speaking BGP in Kubernetes

Pick one (mixed modes cause “it kind of works” outages):

  • MetalLB (BGP mode)
  • BGP speakers run as pods (daemonset) on selected nodes.
  • Advertises LoadBalancer IPs (and sometimes aggregates) to ToR/leaf.
  • kube-proxy still handles node traffic; BGP only for VIP reachability.
  • FRR inside the cluster (FRR-K8s / CNI-integrated)
  • FRR runs on nodes; can advertise Pod CIDRs and/or Service VIPs.
  • Often used with Calico, Cilium (BGP control plane), or FRR-K8s.
  • kube-proxy BGP mode (less common, highly implementation-dependent)
  • Confirm exact distro/feature; many environments don’t actually use kube-proxy for BGP.
  • If you’re not 100% sure, assume you’re using MetalLB or CNI BGP.

Checklist
- [ ] Document the BGP speaker: metallb-speaker, frr, calico-node, cilium-agent, etc.
- [ ] Confirm what routes should be advertised: Service VIPs, Pod CIDRs, or both.
- [ ] Confirm advertisement scope: per-node / per-rack / anycast / ECMP.

2) Cisco platform and feature assumptions

  • NX-OS (Nexus 9K/3K) vs IOS-XE (Catalyst/ASR) behave differently around:
  • eBGP multihop defaults and TTL security
  • BGP neighbor templates / peer policies
  • VRF vs global routing, EVPN AF handling
  • Route-map semantics on inbound/outbound

Checklist
- [ ] Record device OS and version: show version
- [ ] Verify BGP feature enabled and correct AFI/SAFI (IPv4 unicast vs EVPN):
- NX-OS: feature bgp, show bgp ipv4 unicast summary
- [ ] Confirm VRF context (default vs tenant VRF): show bgp vrf <vrf> ...

3) MTU and encapsulation (EVPN/VXLAN considerations)

BGP control plane can be fine while data plane blackholes due to MTU or VTEP encapsulation.

Checklist
- [ ] Underlay MTU sized for VXLAN/EVPN if pods/Services traverse overlay.
- [ ] Node NIC MTU aligned with CNI requirements (e.g., 1500 vs 9000).
- [ ] Confirm PMTUD isn’t blocked (ICMP fragmentation needed) on critical paths.
- [ ] If advertising VIPs from nodes, confirm return traffic path symmetry/asymmetry is acceptable (ECMP + conntrack implications).

4) Establish the peering topology (important for “route-reflector traps”)

Typical patterns:
- Kubernetes nodes ↔ leaf switches (eBGP): simplest and most common.
- Nodes ↔ route reflectors: possible, but adds next-hop and policy complexity.
- Nodes ↔ spines (rare): increases blast radius.

Checklist
- [ ] Map each node to its expected leaf neighbor(s).
- [ ] Determine eBGP vs iBGP and whether a route reflector is in play.
- [ ] Decide whether nodes should advertise with next-hop-self or preserve next-hop.

Top 8 failure modes to rule out first

1) ASN mismatches (local/remote AS confusion)

Symptoms:
- BGP stays in Active/Idle or oscillates.
- Cisco logs show “Bad peer AS” or “Open message error”.

Checklist
- [ ] Node BGP speaker ASN matches Cisco neighbor remote-as.
- [ ] Cisco local ASN matches node remote-as expectations.
- [ ] If using BGP “local-as” / “dual-as”, confirm both sides are consistent.

2) TCP/179 reachability and source interface issues

Symptoms:
- No SYN/SYN-ACK; session never reaches OpenSent.
- Works from one node but not another.

Checklist
- [ ] Routing to the neighbor IP is correct in the right VRF.
- [ ] Cisco neighbor is using the correct update-source (SVI/loopback) if applicable.
- [ ] Node uses the correct source IP (CNI/host network) for BGP sessions.

3) MD5 authentication mismatch (TCP-AO/MD5)

Symptoms:
- One side shows repeated connection attempts; the other logs “MD5 digest mismatch”.
- Session stuck in Active.

Checklist
- [ ] Same password on both ends; confirm no trailing spaces/secret mismatch.
- [ ] Ensure you’re using the same auth type supported by the platform/version.

4) TTL security / eBGP multihop mismatch

Symptoms:
- Peering to loopbacks or across a routed hop fails unless TTL is adjusted.
- Cisco might require ebgp-multihop or disable TTL security features.

Checklist
- [ ] If peering to a loopback, set ebgp-multihop appropriately on Cisco and the speaker.
- [ ] If using GTSM/TTL-security, make sure expected hop count matches reality.

5) Address-family mismatch (IPv4 unicast vs EVPN vs VPNv4)

Symptoms:
- Session is Established but 0 prefixes received/sent.
- show bgp summary looks fine, but route tables remain empty.

Checklist
- [ ] Both ends activate the same AFI/SAFI (usually ipv4 unicast for VIPs/Pod CIDRs).
- [ ] Avoid accidentally advertising into EVPN when you meant IPv4 unicast (or vice versa).
- [ ] In VRFs, ensure neighbor is activated under the correct VRF address-family.

6) Next-hop / reachability / “self-originate” issues

Symptoms:
- Cisco receives prefixes but marks them inaccessible (next-hop unreachable).
- Traffic is blackholed or only works from some subnets.

Checklist
- [ ] Confirm next-hop advertised by the node is reachable from leaf VRF/RIB.
- [ ] If using route reflectors or multi-hop topologies, set next-hop-self where appropriate.
- [ ] For MetalLB, confirm it announces with node IP as next-hop (typical) and that IP is reachable.

7) Policy filters and route-maps (silent drops)

Symptoms:
- Established session, but prefixes are not installed.
- received-routes empty or present but denied by policy.

Checklist
- [ ] Check inbound/outbound route-maps and prefix-lists.
- [ ] Verify maximum-prefix thresholds aren’t tripping.
- [ ] Confirm communities/extended communities aren’t required by policy (common in EVPN environments).

8) Route-reflector traps (when mixing iBGP, RR, and leaf peering)

Symptoms:
- Routes appear on RR but don’t reach leafs, or next-hop is wrong.
- Only one direction works (leaf learns VIP but return traffic fails).

Checklist
- [ ] Ensure RR reflects the correct AFI/SAFI and isn’t stripping next-hop attributes unexpectedly.
- [ ] For iBGP, remember: iBGP does not change next-hop by default.
- [ ] Validate that nodes are not unintentionally acting as transit for routes (leaking/redistribution).

Live verification commands and expected outputs (with examples)

Kubernetes: confirm speaker pods, node placement, and advertised IPs

MetalLB (example)

bash 
kubectl -n metallb-system get pods -o wide
kubectl -n metallb-system get configmap
kubectl get svc -A | grep LoadBalancer
kubectl -n metallb-system logs -l app=metallb,component=speaker --tail=200

Expected
- Speaker pods running on the nodes intended to peer.
- Logs contain BGP neighbor events: Established and announcements for VIPs.

FRR / CNI BGP (example checks)

bash 
kubectl -n kube-system get pods -o wide | egrep 'frr|calico|cilium'
kubectl -n kube-system logs -l k8s-app=calico-node --tail=200

Expected
- BGP control-plane shows peers as up and routes being advertised.

On Kubernetes nodes: inspect BGP state (FRR via vtysh)

If FRR is on the host (or in a privileged pod with host networking), use:

bash 
sudo vtysh -c "show bgp summary"
sudo vtysh -c "show bgp ipv4 unicast neighbors 10.0.0.1 advertised-routes"
sudo vtysh -c "show bgp ipv4 unicast neighbors 10.0.0.1 received-routes"
sudo vtysh -c "show ip route"

Expected (healthy cues)
- BGP state = Established
- Non-zero Prefixes in summary (for the AF you use)
- Advertised routes include expected VIPs or Pod CIDRs
- Next-hop is the node IP (or intended loopback), and that next-hop is in the node routing table

If BGP is established but prefixes are 0:
- Re-check AF activation, policy, and advertisement configuration.

Cisco leaf: verify BGP session, AF activation, and route installation

NX-OS examples

bash 
show bgp ipv4 unicast summary
show bgp ipv4 unicast neighbors 10.0.0.10
show bgp ipv4 unicast neighbors 10.0.0.10 routes
show bgp ipv4 unicast neighbors 10.0.0.10 received-routes
show ip route <vip>/32
show logging log | include BGP

Expected (healthy cues)
- Neighbor state Established
- Prefixes Received reflects your advertisements
- show ip route <vip> shows BGP-learned route with correct next-hop
- No repeated “authentication” / “TTL” / “bad AS” errors in logs

If routes are received but not in RIB:
- Look for route-map denies, next-hop unreachable, or RPKI/validation (if enabled).

IOS-XE examples

bash 
show ip bgp summary
show ip bgp neighbors 10.0.0.10 advertised-routes
show ip bgp neighbors 10.0.0.10 received-routes
show ip route <vip>
show logging | include BGP

Ansible remediation playbooks: atomically fixing configs, toggling BGP sessions, and rolling changes safely across leaf-spine

The goal is repeatable, minimal-diff remediation:
1) render intended neighbor config from inventory,
2) apply in a controlled window,
3) verify BGP established + prefixes,
4) roll forward or rollback.

Patterns that work well

  • Use neighbor templates / peer-policies on Cisco, so playbooks change one logical object.
  • Make changes idempotent (avoid “config replace everything” unless you have full tested golden configs).
  • Roll changes: one leaf at a time (or one rack) and validate before continuing.
  • Fail fast on verification tasks, and stop the rollout.

Example Ansible playbook (NX-OS): fix remote-as + password + AF activation, then bounce neighbor

Adjust to your topology (VRF, peer templates). This example uses cisco.nxos.nxos_config and nxos_command.

yaml 
---
- name: Remediate Kubernetes BGP peering on leafs (NX-OS)
  hosts: leafs
  serial: 1
  gather_facts: no
  vars:
    k8s_neighbors:
      - { ip: "10.0.0.10", remote_as: 65110, password: "S3cret", afi: "ipv4 unicast" }
      - { ip: "10.0.0.11", remote_as: 65110, password: "S3cret", afi: "ipv4 unicast" }

  tasks:
    - name: Precheck - capture BGP summary
      cisco.nxos.nxos_command:
        commands:
          - "show bgp ipv4 unicast summary"
      register: pre_bgp

    - name: Configure neighbors (idempotent lines)
      cisco.nxos.nxos_config:
        lines: |
          router bgp 65000
          {% for n in k8s_neighbors %}
          neighbor {{ n.ip }} remote-as {{ n.remote_as }}
          neighbor {{ n.ip }} password 0 {{ n.password }}
          address-family {{ n.afi }}
            neighbor {{ n.ip }} activate
          exit-address-family
          {% endfor %}
      register: cfg

    - name: Bounce neighbor sessions (soft reset first; if needed, shutdown/no shutdown)
      cisco.nxos.nxos_command:
        commands:
          - "clear bgp ipv4 unicast * soft in"
          - "clear bgp ipv4 unicast * soft out"

    - name: Postcheck - verify neighbors Established and prefixes non-zero
      cisco.nxos.nxos_command:
        commands:
          - "show bgp ipv4 unicast summary"
      register: post_bgp
      failed_when: "'Idle' in post_bgp.stdout[0] or 'Active' in post_bgp.stdout[0]"

Notes
- Use serial: 1 for safe rolling changes across leafs.
- Prefer soft clears before hard resets; hard resets can flap ECMP and trigger transient outages.
- Add targeted checks: parse prefix counts per neighbor rather than just string matching.

When you must do a hard reset safely

  • Bounce only the neighbor(s) facing K8s, not the entire BGP process.
  • Sequence changes:
    1) apply config,
    2) shutdown neighbor (optional),
    3) no shutdown,
    4) verify Established and prefix counts,
    5) verify RIB routes.

Atomicity and rollback

  • For NX-OS, consider configuration checkpoints:
  • create checkpoint pre-change
  • apply changes
  • rollback if verification fails
    (Implementation varies by platform and operational policy; test in lab first.)

Preventive automation and monitoring: NetBox hooks, session alerts, config drift detection, and test harnesses

NetBox inventory hooks (source of truth)

  • Track for each node or rack:
  • leaf pair, neighbor IPs, ASNs, VRF, BGP password secret reference
  • expected advertisements (VIP pools, Pod CIDRs)

Checklist
- [ ] Model BGP sessions as NetBox “connections” or custom fields.
- [ ] Generate Ansible inventory from NetBox to avoid hand-edited hostvars.
- [ ] Validate uniqueness constraints (duplicate neighbor IPs, mismatched ASNs).

BGP session monitoring and alerting

Practical signals:
- BGP state transitions (Established → Idle)
- Prefix count changes (sudden drop to 0)
- Next-hop unreachable events
- Excessive dampening / max-prefix hits

Checklist
- [ ] Poll show bgp ... summary and alert on state/prefix deltas.
- [ ] Export telemetry (gNMI/streaming telemetry where possible) for near-real-time session state.
- [ ] Alert on kube-level symptoms too: Service VIP not reachable, speaker restarts, etc.

Config drift detection

  • Nightly diff against intended neighbor templates/policies.
  • Detect “one leaf different” issues (common after emergency changes).

Checklist
- [ ] Run Ansible in check-mode + diff and store results.
- [ ] Track BGP neighbor stanza drift and route-map/prefix-list drift.
- [ ] Gate production changes with a preflight “no drift” requirement.

Test harnesses (prove control plane + data plane)

Minimum useful tests:
- Control plane: BGP Established + expected prefixes on leafs
- Data plane: reachability to a VIP from a test host in each VRF/tenant segment

Checklist
- [ ] Automated kubectl check: expected LB IPs exist and match pools.
- [ ] Automated leaf check: route exists in correct VRF with correct next-hop.
- [ ] Synthetic traffic: TCP/HTTP to VIP from a fabric-connected test pod/VM.

Common pitfalls

  • BGP Established != working: AF mismatch, policy deny, or next-hop unreachable can still break traffic.
  • Wrong VRF context on Cisco: neighbor configured in default VRF while the SVI/route lives in a tenant VRF (or vice versa).
  • Peering to loopbacks without multihop/TTL alignment: sessions stay down with little obvious clue.
  • Route-map “deny all” defaults: especially when adding a new prefix-list and forgetting a permit clause.
  • Asymmetric routing with ECMP + conntrack: return traffic may choose a different node/leaf path than expected.
  • MTU blackholes in VXLAN/EVPN fabrics: control plane looks perfect, data plane fails under load or with specific packet sizes.
  • Mixing multiple BGP speakers unintentionally: e.g., MetalLB and CNI both advertising the same VIP/CIDR, causing churn.
  • Overly broad aggregation: advertising a summary that steals traffic from more specific internal routes.

Key takeaways

  • Start with design alignment (who speaks BGP, which prefixes, which VRF/AFI) before diving into packet-level debugging.
  • Rule out the top failure modes quickly: ASN, TCP/179 reachability, MD5/TTL, AF activation, next-hop, and policy filters.
  • Verify from both ends: kubectl + vtysh (or CNI logs) + Cisco show bgp and RIB checks.
  • Automate remediation safely with Ansible: idempotent templates, rolling changes, and post-change verification gates.
  • Prevent recurrence via NetBox-driven inventory, continuous session monitoring, and config drift detection tied to a test harness.
Back to Blog