Battle-Tested Cookbook: Ansible for Cisco Leaf-Spine Changes with Kubernetes CNI Validation

Published on January 31, 2026

Battle-Tested Cookbook: Ansible for Cisco Leaf-Spine Changes with Kubernetes CNI Validation

Datacenter networking changes in a leaf-spine fabric are rarely “just a switch config push.” The blast radius includes routing symmetry, ECMP behavior, MTU consistency, and Kubernetes node-to-node paths that your CNI assumes are stable. This cookbook is an advanced, checklist-driven approach to executing Cisco NX-OS fabric changes with Ansible, then validating outcomes using Kubernetes CNI smoke tests plus telemetry-driven guardrails and rollback criteria.

Pre-change checklist: maintenance windows, NetBox inventory sync, circuit and port reservation, and impact analysis for Kubernetes node traffic paths

1) Maintenance window and change scope

  • Define scope (exact leaves/spines, VRFs, VLANs, BGP neighbors, port-channels, vPC domains).
  • Define success criteria (what must work after change):
  • All BGP sessions Established
  • No ECMP collapse (expected number of next-hops)
  • Kubernetes pod-to-pod and pod-to-service connectivity OK
  • No MTU regressions (especially for encapsulating CNIs)
  • Define rollback triggers (explicit, measurable):
  • Loss of BGP adjacency beyond N seconds
  • ECMP path count below threshold
  • CNI probes fail (specific tests listed below)
  • Telemetry shows interface errors/CRC spikes during window

2) NetBox source-of-truth sync (before touching the network)

  • Pull current state:
  • Devices (leaf/spine roles, mgmt IPs)
  • Interfaces + LAGs (vPC/port-channel membership)
  • VRFs, VLANs, prefixes, IP assignments
  • Cables/circuits (uplinks, DCI, firewall handoffs)
  • Freeze allocations during the change:
  • Reserve ports intended for new LAGs/hosts (avoid someone else consuming them mid-change).
  • Reserve VLAN/VRF IDs if your environment allocates dynamically.

Practical tips:
- Ensure the Ansible inventory is generated from NetBox (or at minimum validated against it).
- Normalize naming:
- Device names, interface names (Ethernet1/1 vs Eth1/1), VRF capitalization, VLAN naming conventions.

3) Circuit and port reservation

Checklist items (record in the change ticket/runbook):
- [ ] New links (ToR-to-leaf, leaf-to-spine, leaf-to-border) physically present and labeled
- [ ] Transceiver type and speed validated (10/25/40/100G) and supported on target ports
- [ ] Port-channel IDs reserved and consistent across vPC peers
- [ ] vPC peer-link and keepalive health validated pre-change

4) Impact analysis for Kubernetes traffic paths

Map how Kubernetes nodes traverse the fabric:
- Node-to-node: which leafs host worker nodes? Do they share the same VRF/VLAN?
- Pod-to-pod: depends on CNI mode (encap vs no-encap)
- Calico IP-in-IP / VXLAN: relies on underlay MTU, node reachability, and allowed protocol/UDP ports.
- Flannel VXLAN: similar MTU and underlay reachability dependencies.
- Service IP reachability:
- ClusterIP: usually kube-proxy / eBPF dataplane internal; depends on pod connectivity.
- NodePort/LoadBalancer: involves routes from external networks/border leafs and (often) BGP advertisement.
- North-south: ingress/egress, NAT gateways, firewalls, border leafs.

Concrete “gotchas” to check before change:
- Underlay MTU end-to-end for encapsulation (commonly 9000 underlay to preserve 1500 overlay).
- ACLs permitting required encapsulation traffic:
- VXLAN: UDP/4789 (plus BGP if used for control-plane)
- Calico BGP: TCP/179 between nodes if using BGP peering
- IP-in-IP: protocol 4 if enabled
- ECMP hashing consistency (5-tuple vs 2-tuple, and whether it affects certain flows like kube-proxy health checks).

Ansible playbook patterns for Cisco NX-OS: modular roles, transactional pushes, handling config merges vs replace, and vendor-specific idempotency traps

1) Use modular roles with clear separation

Recommended role layout:
- roles/nxos_baseline/ (NTP, syslog, SNMP, AAA, banners, feature toggles)
- roles/nxos_underlay/ (interfaces, OSPF/ISIS if used, BGP underlay)
- roles/nxos_overlay/ (EVPN/VXLAN, VRFs, VNIs, L2/L3 VTEP config)
- roles/nxos_edge/ (border leaf, DCI, firewall handoffs)
- roles/validate_fabric/ (pre/post checks, telemetry queries)

Design rules:
- Keep each role idempotent and scoped.
- Avoid “giant template replaces” except for tightly-controlled blocks.
- Make roles parameter-driven from NetBox-derived variables.

2) Transactional pushes: stage, apply, verify, commit/rollback

NX-OS supports checkpoints/rollback. Use them.

Pattern:
1. Pre-checks (state capture)
2. Create checkpoint
3. Apply change
4. Post-checks (must pass)
5. If failed, rollback to checkpoint

3) Merge vs replace: choose intentionally

  • Prefer merge for incremental changes (add VLAN, add BGP neighbor, add interface description).
  • Consider replace only for narrow sections where drift is unacceptable and you can safely own the whole block (e.g., a dedicated prefix-list generated entirely by automation).

Practical guidance:
- Replacing large portions of config increases risk when humans make emergency changes outside automation.
- Merging can hide drift unless you also run post-change audits.

4) Vendor-specific idempotency traps (NX-OS realities)

Common traps that break “run twice = no change”:
- Implicit defaults: NX-OS may not display default settings; pushing “no …” lines can create churn.
- Order sensitivity: some constructs (route-maps, prefix-lists) compare as different when re-ordered.
- Feature dependencies: configuring a component before feature is enabled causes failures or partial config.
- Interface state: switchport vs no switchport transitions can be disruptive and not safely idempotent if reused across different interface types.
- vPC consistency: applying mismatched allowed VLAN lists across vPC peers creates vPC consistency warnings and potential traffic impact.

Mitigations:
- Normalize ordering in templates (stable sort).
- Query the device state and gate tasks (when:) rather than blindly pushing.
- Use NX-OS “checkpoint” as a safety net even with idempotent tasks.

Example: transactional NX-OS change with checkpoint + post-check gates

yaml 
---
- name: Leaf-spine change with checkpoint and validation gates
  hosts: nxos_fabric
  gather_facts: false
  vars:
    checkpoint_name: "ansible_prechange_{{ lookup('pipe','date +%Y%m%d%H%M%S') }}"
    expected_bgp_established_min: 8

  tasks:
    - name: Pre-check - capture BGP summary
      cisco.nxos.nxos_command:
        commands:
          - "show bgp ipv4 unicast summary"
          - "show ip route vrf all summary"
      register: pre_state

    - name: Create rollback checkpoint
      cisco.nxos.nxos_command:
        commands:
          - "checkpoint {{ checkpoint_name }}"
      changed_when: false

    - name: Apply config (merge) - underlay/overlay snippets
      cisco.nxos.nxos_config:
        lines:
          - "router bgp {{ bgp_asn }}"
          - "address-family ipv4 unicast"
          - "maximum-paths ibgp 16"
        match: line
        save_when: modified

    - name: Post-check - ensure BGP sessions are up
      cisco.nxos.nxos_command:
        commands:
          - "show bgp ipv4 unicast summary | inc 'Estab|Idle|Active'"
      register: post_bgp

    - name: Gate - fail if BGP appears unhealthy (example heuristic)
      ansible.builtin.assert:
        that:
          - "'Idle' not in post_bgp.stdout[0]"
          - "'Active' not in post_bgp.stdout[0]"
        fail_msg: "BGP shows Idle/Active sessions after change; triggering rollback."

  rescue:
    - name: Roll back to checkpoint
      cisco.nxos.nxos_command:
        commands:
          - "rollback running-config checkpoint {{ checkpoint_name }}"
    - name: Fail the play after rollback
      ansible.builtin.fail:
        msg: "Change failed validation and was rolled back to {{ checkpoint_name }}."

Notes:
- This example uses a simplistic BGP health check. In practice, parse counts and require specific neighbors/prefix counts.
- For high-risk changes, add per-leaf sequencing and canary deployment (1 leaf pair at a time).

Kubernetes CNI smoke tests to run pre/post-change: test traffic flows, pod-to-pod connectivity, service IP reachability and CNI-specific probes (Calico/Flannel/DoK)

Run these tests before the change to establish a baseline, then run immediately after (and again after convergence timers).

1) Baseline cluster health

  • [ ] kubectl get nodes -o wide (all Ready, node IPs correct)
  • [ ] kubectl get pods -A (CNI pods Running)
  • [ ] kubectl get events -A --sort-by=.lastTimestamp | tail -n 50 (scan for CNI-related flaps)

2) Pod-to-pod connectivity across nodes

Goal: validate the fabric supports the overlay/underlay path assumptions.

Suggested approach:
- Deploy a test daemonset (one pod per node) that can respond to ICMP/TCP.
- From one node/pod, reach pods on other nodes by Pod IP.

Checklist:
- [ ] Pod IP reachability (ICMP if permitted, otherwise TCP)
- [ ] Cross-rack / cross-leaf reachability (ensure you hit different leaves)
- [ ] Latency sanity (watch for sudden increases due to suboptimal routing)

3) Service IP reachability (ClusterIP, NodePort as applicable)

  • [ ] ClusterIP access from another pod in a different node
  • [ ] DNS resolution (CoreDNS service) from a test pod
  • [ ] NodePort reachability from a node in a different rack (if used internally)
  • [ ] LoadBalancer IP reachability (if you run BGP-based LB like MetalLB or Cilium BGP)

4) CNI-specific probes (pick what matches your environment)

Calico

  • [ ] calicoctl node status (BGP peerings up if Calico BGP enabled)
  • [ ] If using IP-in-IP: validate protocol 4 permitted end-to-end where needed
  • [ ] If using VXLAN: validate UDP/4789 not blocked and MTU consistent
  • [ ] Optional: check Felix health endpoints/metrics if exposed

Flannel (VXLAN)

  • [ ] Confirm flannel.1 interface exists on nodes
  • [ ] Validate VXLAN traffic not filtered, MTU not causing fragmentation/blackholing
  • [ ] Verify kube-system flannel pods stable (no restarts during change)

“DoK” (vendor-specific / managed distributions)

  • Identify the dataplane type (iptables vs eBPF) and encapsulation mode.
  • [ ] Run distribution-specific diagnostics (often a kubectl -n kube-system logs pattern plus a health CLI).

Practical tip: keep a small “smoke test” namespace with:
- A netshoot-style toolbox pod (curl/dig/tcping)
- An echo server behind a ClusterIP service
- Optional NodePort service for north-south tests

Automated validation and observability: telemetry queries (gNMI/Streaming), BGP/ECMP path checks, and criteria that trigger automated rollback

1) Telemetry-driven pre/post checks (don’t rely only on CLI)

Use streaming telemetry or periodic gNMI queries to validate:
- Interface counters: CRC, drops, discards, pause frames
- Link status transitions during the change window
- BGP neighbor state and prefix counts
- ECMP next-hop counts for critical routes
- vPC consistency and peer-link health

Data sources commonly used:
- gNMI (OpenConfig where available; NX-OS support varies by release/features)
- Model-driven telemetry subscriptions to a collector (Prometheus, InfluxDB, ELK, etc.)
- Syslog correlation: link flaps, vPC warnings, BGP adjacency changes

2) BGP and EVPN checks that matter for Kubernetes

Depending on design:
- Underlay eBGP leaf-spine:
- [ ] All leaf-to-spine neighbors Established
- [ ] Expected number of prefixes received/advertised (watch for sudden drops)
- EVPN overlay:
- [ ] EVPN neighbors Established (route reflectors/spines if used)
- [ ] MAC/IP route counts stable (or trending correctly)
- Kubernetes-facing:
- If nodes or load balancers advertise routes via BGP:
- [ ] Those routes present in the correct VRF
- [ ] Next-hops distribute across expected leaf pairs

3) ECMP path checks (avoid silent capacity loss)

ECMP failures often don’t break connectivity; they reduce redundancy and throughput.

Checklist:
- [ ] For a set of critical prefixes (node subnets, pod CIDRs, service VIP ranges if routed), validate:
- next-hop count equals expected (e.g., 4 spines)
- next-hops are across distinct spines/paths
- [ ] Confirm hashing behavior isn’t unintentionally changed (platform defaults vs explicit config)

4) Define rollback criteria and automate it

Automated rollback works when criteria are:
- objective (counts, states, latency thresholds)
- measured quickly (minutes, not hours)
- scoped (rollback only when change is the likely cause)

Example rollback triggers:
- BGP neighbor(s) down for > 90 seconds after change
- ECMP next-hop count reduced below N for critical routes
- Packet loss > X% between test pods across racks
- CNI health endpoint fails on > Y nodes
- Interface error counters spike beyond baseline + threshold

Implementation approach:
- Gate Ansible playbook progression on validations (assert, fail, block/rescue)
- Store pre-change snapshots and compare post-change (structured parsing when possible)
- For multi-step changes, validate at each stage (e.g., after modifying one leaf pair)

Post-change audit and CI integration: update NetBox, emit change records, unit tests for playbooks, and techniques to keep automation safe in production

1) Update NetBox (close the loop)

  • [ ] Create/update:
  • interface/LAG membership changes
  • VLAN and VRF associations
  • IP assignments and anycast/VTEP IPs
  • circuit terminations and cable mappings
  • [ ] Attach artifacts:
  • pre/post device snapshots
  • validation results
  • rollback checkpoint names (even if not used)

2) Emit change records automatically

  • Store a machine-readable change summary:
  • devices touched
  • diffs (intended config vs applied, if available)
  • validation results + timestamps
  • operator and window metadata

Even if your org uses ITSM, keep a lightweight artifact in version control or an object store for quick incident correlation.

3) CI for Ansible (keep production safe)

Minimum practical CI for network automation:
- Linting: ansible-lint with a baseline ruleset
- Syntax checks: ansible-playbook --syntax-check
- Unit tests for templates/filters:
- Validate Jinja renders (no undefined vars)
- Validate generated configs match expectations (golden files)
- Static validation:
- Ensure VLAN IDs within allowed ranges
- Ensure no duplicate IPs / ASNs
- Ensure vPC peer parameters consistent across the pair
- Canary plan generation:
- Generate “intended config diff” per device before merge approval

4) Techniques that reduce risk in production

  • Canary deployment: one leaf pair at a time, validate, then proceed.
  • Rate limit changes: avoid simultaneously bouncing many links or BGP sessions.
  • Feature gating: do not enable disruptive NX-OS features mid-window unless required.
  • Config boundaries: reserve dedicated config blocks for automation-managed objects (prefix-lists/route-maps with clear naming).
  • Backout rehearsals: periodically test rollback in a lab or non-prod fabric.

Common pitfalls

  • Skipping baseline measurements: without pre-change CNI and ECMP baselines, post-change failures become guesswork.
  • Assuming “idempotent” means “safe”: an idempotent task can still be disruptive (e.g., toggling no switchport).
  • Template order churn: route-map/prefix-list reordering causes continual diffs and risky re-applies.
  • vPC drift: allowed VLAN mismatch, STP settings, or MTU inconsistencies between peers cause intermittent drops.
  • MTU mismatches: encapsulation CNIs fail in ways that look like random pod networking issues.
  • Partial telemetry: validating only control-plane (BGP up) can miss data-plane loss (drops, ECMP collapse, blackholing).
  • Over-broad rollbacks: rolling back the whole device config after unrelated transient failures can create bigger outages; scope changes and rollback domains carefully.

Key takeaways

  • Treat datacenter networking changes as end-to-end path changes: fabric + Kubernetes + CNI behavior.
  • Use transactional NX-OS workflows (checkpoint + gated validation + rollback).
  • Validate more than “BGP is up”: include ECMP path counts, interface error telemetry, and Kubernetes CNI smoke tests.
  • Close the loop with NetBox updates and CI checks so the next change starts from an accurate source-of-truth.
  • Safety comes from canaries, explicit rollback criteria, and staged validations, not from hoping idempotency will prevent outages.
Back to Blog