Checklist: Automating Kubernetes datacenter networking (EVPN‑VXLAN) on Cisco NX‑OS with Ansible

Published on January 29, 2026

Checklist: Automating Kubernetes datacenter networking (EVPN‑VXLAN) on Cisco NX‑OS with Ansible

Automating an EVPN‑VXLAN fabric for kubernetes is less about “pushing config” and more about enforcing repeatable decisions: MTU, underlay routing, VXLAN replication mode, VRF/VLAN mapping, and how services (LoadBalancer/NodePort) will exit the fabric. This checklist focuses on NX‑OS patterns that scale, plus Ansible workflows that are idempotent and testable.

Key takeaways

  • Treat EVPN‑VXLAN as a productized fabric: consistent templates, strict input variables, and automated validation.
  • Make MTU, replication mode (multicast vs ingress replication), and BGP policy explicit design decisions—they are the most common root causes of kubernetes connectivity issues.
  • Plan service routing early: MetalLB + BGP (or L2) needs deterministic L3/L2 adjacency and route policy to avoid leaks and flaps.
  • Build “day-2” into automation: health checks, diff-based deploys, and rollback workflows for rapid remediation.

Design decisions: underlay topology, MTU, multicast vs ingress replication, BGP EVPN control-plane considerations for Kubernetes CNI

Underlay topology checklist (spine-leaf best practices)

  • Routing: eBGP underlay (common) or OSPF/IS-IS; keep it simple and consistent.
  • Addressing:
  • Use /31 (IPv4) on P2P links where possible.
  • Use loopbacks for VTEP (NVE source) and BGP router-id.
  • ECMP:
  • Ensure equal-cost paths across spines.
  • Confirm hardware supports required ECMP scale for leaf-to-spine design.

MTU checklist (most kubernetes pain shows up here)

  • Decide the end-to-end MTU for the underlay and overlay:
  • If pods use 1500-byte MTU and you run VXLAN, you typically need ~1550+ on underlay (exact overhead varies; be consistent and validate).
  • Checklist:
  • [ ] Underlay interfaces MTU supports VXLAN encapsulation.
  • [ ] vPC peer-link and keepalive paths support the same MTU.
  • [ ] Any L3 out (firewalls/routers) supports your chosen MTU or clamps MSS.
  • [ ] Kubernetes CNI MTU is set appropriately (Calico/Antrea often configurable).

Multicast vs ingress replication (head-end replication)

  • Multicast replication:
  • Pros: efficient BUM replication at scale.
  • Cons: requires multicast in underlay (PIM, RP, etc.), more operational moving parts.
  • Ingress replication:
  • Pros: no multicast underlay.
  • Cons: replicates at VTEP; can stress CPU/TCAM in large BUM domains.
  • Checklist:
  • [ ] If using multicast: underlay PIM enabled, RP design validated, IGMP snooping behavior understood.
  • [ ] If using ingress replication: validate scale limits (VNI count, MAC/IP routes, BUM rate).

BGP EVPN control-plane considerations tuned for kubernetes

Kubernetes can generate bursty control-plane events (node churn, MetalLB announcements, service updates). EVPN control-plane should be stable and well-scoped.
- Checklist:
- [ ] Use stable router-id and update-source (loopback).
- [ ] Apply BGP policies: only export/import required RTs per VRF.
- [ ] Plan route scale: MAC/IP routes (type-2), IMET (type-3), and L3 routes (type-5) if used.
- [ ] Define clear L2/L3 boundary: where the default gateway lives (anycast SVI on leafs is typical).
- [ ] If MetalLB BGP is used, ensure BGP dampening/prefix-lists prevent accidental broad advertisements.

NX‑OS configuration patterns: example EVPN‑VXLAN, VLAN/VRF mapping, L3 out and BGP peering templates tuned for scale and Kubernetes services

Below is a pattern (not a complete fabric config). Adjust to your NX‑OS version and feature set.

Fabric feature checklist

  • [ ] feature nv overlay
  • [ ] feature vn-segment-vlan-based (if using vlan-based VNIs)
  • [ ] feature bgp
  • [ ] feature interface-vlan
  • [ ] feature fabric forwarding / related platform requirements (varies)
  • [ ] feature lacp and feature vpc if using vPC

VLAN/VRF/VNI mapping checklist

  • Create a predictable mapping strategy:
  • VLAN per tenant/app segment
  • VRF per tenant or per cluster domain
  • L2 VNI per VLAN, L3 VNI per VRF
  • Checklist:
  • [ ] Unique VNI ranges reserved per environment (prod/dev/etc.)
  • [ ] RT/RD scheme deterministic (e.g., ASN:VNI)
  • [ ] Anycast gateway IP/MAC consistent across leaf pair

Example NX‑OS (EVPN‑VXLAN essentials)

cli 
! --- Global features ---
feature bgp
feature nv overlay
feature interface-vlan
feature vn-segment-vlan-based

fabric forwarding anycast-gateway-mac 0001.0001.0001

! --- VRF + L3 VNI ---
vrf context K8S-PROD
  vni 100100
  rd auto
  address-family ipv4 unicast
    route-target both auto
  address-family ipv6 unicast
    route-target both auto

! --- VLAN + L2 VNI ---
vlan 110
  name K8S-PROD-PODS
  vn-segment 10110

interface Vlan110
  no shutdown
  vrf member K8S-PROD
  ip address 10.110.0.1/24
  fabric forwarding mode anycast-gateway

! --- NVE interface ---
interface nve1
  no shutdown
  source-interface loopback1
  host-reachability protocol bgp
  member vni 10110
    ingress-replication protocol bgp
  member vni 100100 associate-vrf

! --- BGP EVPN ---
router bgp 65001
  router-id 10.0.0.11
  address-family ipv4 unicast
  address-family l2vpn evpn

  neighbor 10.0.0.1
    remote-as 65000
    update-source loopback0
    address-family l2vpn evpn
      send-community extended

L3 out patterns for kubernetes services

Decide how cluster traffic exits the VRF:
- Centralized L3 out (border leafs): common for east-west inside fabric, north-south via border.
- Distributed L3 out (many leafs): reduces tromboning but increases routing adjacencies/policy points.

Checklist for services:
- [ ] Node subnets and pod subnets are routed as intended (depends on CNI mode).
- [ ] LoadBalancer IP pools (MetalLB) are either:
- routed in L3 with BGP advertisements, or
- bridged in L2 within a VLAN (less common in EVPN fabrics, but possible).
- [ ] Prefix-lists/route-maps ensure only the correct service pools are advertised.

BGP peering templates tuned for scale

  • Use peer templates where possible to standardize:
  • BFD (if supported/desired)
  • graceful restart
  • timers
  • max-prefix (guardrails)
  • Checklist:
  • [ ] max-prefix for BGP neighbors (especially if peering with MetalLB speakers).
  • [ ] Route policy to prevent leaking default routes into tenant VRFs unless required.
  • [ ] Standard communities (or large communities) for traffic engineering and filtering.

Ansible automation: roles, idempotent templates, inventory strategies, and test-driven deployment for NX‑OS configuration and validation

Role design checklist (practical layout)

Recommended role split:
- roles/nxos_underlay/ (interfaces, underlay routing, MTU)
- roles/nxos_evpn/ (BGP EVPN, NVE, VNIs, RT/RD policies)
- roles/nxos_tenants/ (VRFs, VLANs, SVIs, anycast GW)
- roles/nxos_border/ (L3 out, route-maps, prefix-lists)
- roles/nxos_validate/ (show commands + asserts)

Checklist:
- [ ] Roles accept structured vars (YAML dicts/lists), not raw CLI snippets.
- [ ] Jinja templates generate deterministic config.
- [ ] One device = one rendered configuration artifact (store in artifacts/).

Inventory strategies that work for fabrics

  • Use groups for roles:
  • spines, leafs, border_leafs
  • Use host_vars for per-device facts (loopbacks, ASN, vPC domain)
  • Use group_vars for fabric-wide constants (VNI ranges, RT base, MTU)

Checklist:
- [ ] Explicitly model: vtep_loopback, bgp_asn, underlay_links, vrfs, vlans.
- [ ] Keep a single source of truth for VLAN/VNI/VRF mappings (avoid duplicate definitions).

Idempotent deployment checklist (NX‑OS specifics)

  • Prefer declarative modules where feasible, but NX‑OS EVPN often requires templating.
  • Use nxos_config with:
  • src: to push a rendered config
  • diff: true in CI or pre-change checks
  • safe match: strategy (line vs strict) based on your standardization

Example Ansible playbook (deploy + validate)

yaml 
---
- name: Deploy EVPN-VXLAN fabric for kubernetes VRFs/VLANs
  hosts: leafs
  gather_facts: false
  connection: ansible.netcommon.network_cli

  vars:
    ansible_network_os: cisco.nxos.nxos

  tasks:
    - name: Render tenant config
      ansible.builtin.template:
        src: templates/tenants.nxos.j2
        dest: "artifacts/{{ inventory_hostname }}-tenants.cfg"

    - name: Push tenant config (idempotent)
      cisco.nxos.nxos_config:
        src: "artifacts/{{ inventory_hostname }}-tenants.cfg"
        diff: true
        save_when: modified

    - name: Validate NVE is up
      cisco.nxos.nxos_command:
        commands:
          - "show nve peers"
          - "show bgp l2vpn evpn summary"
      register: show_out

    - name: Assert EVPN BGP is established
      ansible.builtin.assert:
        that:
          - "'Estab' in show_out.stdout[1] or 'Established' in show_out.stdout[1]"
        fail_msg: "EVPN BGP not established on {{ inventory_hostname }}"

Test-driven deployment checklist

  • Pre-change:
  • [ ] Render configs and run diffs in CI (or locally) before touching devices.
  • [ ] Lint vars (schema validation helps: yamale/jsonschema).
  • Post-change:
  • [ ] Validate BGP sessions (underlay + EVPN).
  • [ ] Validate NVE peers and VNI state.
  • [ ] Validate MAC/IP route counts are within expected ranges.
  • Regression tests:
  • [ ] Keep a “golden outputs” approach for critical show commands (snapshot/compare).
  • [ ] Build a simple connectivity test plan aligned to kubernetes flows (pod-to-pod, pod-to-service, node-to-service, external-to-service).

CNI and service integration: validating CNI compatibility (Calico/Antrea/Flannel), MetalLB/service routing patterns, and health checks for pod-to-service connectivity

CNI compatibility checklist (what the fabric must support)

Different CNIs change what the network sees:
- Calico
- Can run with BGP to ToR or with overlay modes.
- If Calico advertises routes, ensure policies prevent exporting pod CIDRs to the wrong VRFs.
- Antrea
- Often uses Geneve overlay; consider double-encapsulation risks if misaligned with fabric design.
- Flannel
- VXLAN-based by default; be careful mixing with EVPN‑VXLAN fabrics (encapsulation on encapsulation).

Checklist:
- [ ] Decide whether pod CIDRs are:
- routed in the fabric (BGP from nodes/speakers), or
- hidden behind node IPs (NAT/overlay internal).
- [ ] Confirm whether CNI uses encapsulation; avoid accidental double overlay unless intentionally designed.
- [ ] Ensure kube-proxy mode (iptables vs IPVS) is accounted for in service troubleshooting.

MetalLB patterns (practical service routing)

Two common approaches:

1) MetalLB in BGP mode (recommended for routed fabrics)
- MetalLB speakers advertise service VIPs to border/leaf.
- Checklist:
- [ ] Establish BGP sessions from nodes (or MetalLB speakers) to leaf/border.
- [ ] Apply prefix-lists: only allow the LB pool(s).
- [ ] Consider BFD/timers to avoid long blackholes on node failure.
- [ ] Use max-prefix safety.

2) MetalLB in L2 mode (simpler, more broadcast)
- VIP is announced via ARP/ND on a VLAN.
- Checklist:
- [ ] EVPN BUM behavior is well understood (multicast/ingress replication).
- [ ] Keep LB L2 domains small; watch ARP/ND scaling.

Health checks for pod-to-service connectivity

Run checks that map to real flows:
- [ ] Pod → Pod (same node, different node)
- [ ] Pod → ClusterIP service
- [ ] Pod → LoadBalancer IP (MetalLB)
- [ ] External client → LoadBalancer IP
- [ ] NodePort reachability (if used)

Operational tips:
- Capture MTU/fragmentation signals early:
- ICMP “fragmentation needed” may be filtered; test with TCP MSS or controlled pings.
- Validate ECMP symmetry for flows that traverse stateful devices.

Real-world troubleshooting checklist: MTU-related packet loss, ARP/ND storms, asymmetric routing, BGP session flaps, and rollback/playbook remediation steps

MTU-related packet loss (classic symptoms in kubernetes)

Symptoms:
- Some services work, others hang (often TLS/large responses).
- Intermittent pod-to-service failures across nodes.

Checklist:
- [ ] Verify underlay interface MTU across leaf-spine links.
- [ ] Verify vPC peer-link MTU and consistency across vPC pair.
- [ ] Check CNI MTU settings align with fabric.
- [ ] Run targeted tests:
- From node network namespace or debug pod, test with DF set and increasing size.
- Remediation:
- Adjust underlay MTU and/or CNI MTU.
- If constrained on L3 out, clamp TCP MSS at border.

ARP/ND storms (EVPN flooding + kubernetes churn)

Symptoms:
- High CPU on leafs, elevated BUM traffic, intermittent neighbor resolution failures.

Checklist:
- [ ] Confirm ARP suppression/ND optimization settings (platform dependent).
- [ ] Check for large L2 domains (too many endpoints per VLAN/VNI).
- [ ] Inspect MAC move events (frequent moves can indicate loops or miswired ports).
- Remediation:
- Reduce L2 domain size; prefer L3 where possible.
- Fix duplicate IPs, loops, or misconfigured trunking.

Asymmetric routing (especially with stateful L3 out)

Symptoms:
- SYN seen one way, SYN/ACK returns another path; stateful firewall drops.

Checklist:
- [ ] Ensure consistent next-hop/ECMP hashing and symmetric forwarding where required.
- [ ] For services advertised by MetalLB BGP, confirm return traffic uses the same border path (policy-based routing or consistent advertisement).
- [ ] Confirm VRF route leaking is intentional and symmetrical.
- Remediation:
- Constrain egress/ingress via route policy.
- Consider dedicated border leafs for north-south to enforce symmetry.

BGP session flaps (EVPN peers or MetalLB)

Symptoms:
- Frequent EVPN peer down/up, service VIPs appear/disappear, route churn.

Checklist:
- [ ] Validate physical stability (optics, errors).
- [ ] Confirm BGP timers and keepalives match intent.
- [ ] If using BFD, ensure it’s stable end-to-end.
- [ ] Ensure max-prefix isn’t being hit (look for shutdown due to prefix limit).
- Remediation:
- Fix transport issues first.
- Add filtering and correct max-prefix.
- Stagger changes and use maintenance windows for spine/leaf BGP restarts.

Rollback and playbook remediation steps

Checklist:
- [ ] Always save pre-change configs (show running-config) to artifacts.
- [ ] Use Ansible --check and --diff before apply.
- [ ] Implement a “known-good” tag:
- deploy tag for pushing config
- validate tag for verification
- rollback tag to restore previous artifact
- [ ] Keep changes small: per VRF/VNI, per leaf pair.

Practical rollback approach:
- Store the last known-good rendered config per device.
- Use nxos_config to restore that file if validation fails (and alert).

Common pitfalls

  • MTU mismatches between underlay links, vPC peer-link, and CNI MTU (causes silent drops).
  • Treating EVPN‑VXLAN as “L2 everywhere”: oversized broadcast domains amplify ARP/ND and troubleshooting complexity.
  • Unscoped BGP advertisements (MetalLB or Calico) leaking prefixes across VRFs or into the wrong neighbors.
  • Mixing overlays unintentionally (e.g., CNI VXLAN over fabric VXLAN) without validating overhead and failure modes.
  • Not validating with kubernetes-realistic flows (ClusterIP, LoadBalancer, DNS, kube-apiserver access paths).

Final checklist (printable)

Design

  • [ ] Underlay routing and addressing plan locked (loopbacks, /31s, ECMP)
  • [ ] MTU strategy validated end-to-end (including L3 out)
  • [ ] Replication mode selected (multicast or ingress replication) with scale validation
  • [ ] EVPN RT/RD scheme deterministic; VRF/VLAN/VNI mapping documented

NX‑OS implementation

  • [ ] Anycast gateway consistent across leaf pairs
  • [ ] NVE peers stable; VNIs up; EVPN BGP established
  • [ ] L3 out and route policies defined (especially for service VIP pools)
  • [ ] Guardrails: prefix-lists, route-maps, max-prefix

Automation (Ansible)

  • [ ] Roles split by concern; vars are structured and validated
  • [ ] Idempotent config pushes with diffs and saved artifacts
  • [ ] Post-deploy validation and assertions for BGP/NVE/VNI state
  • [ ] Rollback workflow tested

Kubernetes integration

  • [ ] CNI mode understood (routing vs overlay); MTU aligned
  • [ ] MetalLB design chosen (BGP vs L2) with appropriate filtering
  • [ ] Connectivity tests cover pod↔pod, pod↔service, external↔service

Troubleshooting readiness

  • [ ] Runbooks for MTU, ARP/ND storms, asymmetry, BGP flaps
  • [ ] Telemetry/monitoring for BGP/NVE peer state and route churn
  • [ ] Safe remediation playbooks and staged rollout plan
Back to Blog