How I Automated BGP Provisioning for 100+ ISP Customers with NetBox, Ansible and Python

Published on January 31, 2026

How I Automated BGP Provisioning for 100+ ISP Customers with NetBox, Ansible and Python

Provisioning bgp for ISP customers sounds repetitive until it isn’t: every customer wants slightly different routing policy, addressing, SLAs, security posture, and turn-up timeline. At 100+ customers, “copy/paste and pray” turns into an outage generator.

This post walks through a battle-tested, automation-first approach using NetBox as the source of truth, Ansible for idempotent configuration, and Python for validation and safe-change gates—built for multi-tenant ISP edge environments where zero-downtime and policy correctness matter.

Constraints and objectives: multi-tenant addressing, numbering policies, IRR vs RPKI, and zero-downtime expectations for ISP networks

Before writing code, the constraints define the shape of the system.

Multi-tenant realities you must model

  • Multiple customers per PE (or ToR/leaf acting as edge)
  • Per-customer VRFs (L3VPN-ish) and/or global routing with strict policy separation
  • Multiple session types:
  • eBGP to customer (most common)
  • iBGP within the ISP core
  • eBGP to upstreams/peers (often different policy and validation)
  • Dual-stack: IPv4 + IPv6 neighbors, separate policies and max-prefix thresholds
  • Redundancy: two CEs, two PEs, or active/standby CE; BFD frequently expected

Numbering and addressing policies (standardize or suffer)

Document and enforce:
- ASN allocation policy:
- Public ASN for customers that announce their own space
- Private ASN for “managed” customers (with remove-private-as toward upstream)
- Per-customer max-prefix defaults (v4/v6) based on product tier
- Interface addressing:
- /31 for IPv4 p2p links
- /127 for IPv6 p2p links
- BGP passwords:
- Rotatable secrets, no plaintext in Git
- Communities and routing policy taxonomy:
- Standard communities for blackhole, localpref tiers, prepend, no-export, etc.

IRR vs RPKI: decide how you will validate and what you will enforce

In ISP networks, these come up immediately:

  • IRR (Internet Routing Registry)
  • Pros: broad legacy adoption, supports “route objects” workflows
  • Cons: data quality varies, auth can be messy, stale objects common
  • RPKI
  • Pros: cryptographically verifiable origin validation
  • Cons: customers may not have ROAs; enforcement can be politically sensitive

Practical approach:
- Treat validation as policy tiers:
- Tier 0 (internal / managed): strict allow-lists (exact prefixes)
- Tier 1 (good hygiene): require RPKI valid where ROA exists, warn on not-found
- Tier 2 (legacy): IRR-derived prefix-lists with monitoring, not hard-drop initially

Zero-downtime expectation: automate for “safe change”

Goals that matter operationally:
- Idempotent changes: reruns shouldn’t flap sessions or rewrite configs unnecessarily
- Staged rollout: build config first, attach/activate second
- Pre-flight checks: validate data + policy before touching devices
- Rollback: deterministic and fast (within minutes), triggered by telemetry/health checks

Modeling BGP in NetBox: device, interface, VRF, ASN and prefix objects, custom fields and relationship design for automation-first workflows

NetBox can represent BGP in different ways depending on version and plugins. The key is to model what you need to generate config safely and keep it consistent across vendors.

Core objects to use

  • Tenants: represent customers (and optionally upstreams/peers as “tenants” too)
  • VRFs: per-customer VRF or shared VRF models
  • Devices: PEs, CEs (if managed), route-reflectors, border routers
  • Interfaces: physical/subinterfaces/SVIs where BGP neighbors live
  • IP Addresses and Prefixes:
  • Link addresses (p2p)
  • Customer assigned space (aggregate and subnets)
  • ASNs:
  • Customer ASN (public/private)
  • Provider ASN(s)

If you have NetBox BGP modeling available (native or plugin), it’s helpful, but the design below works even if you keep BGP-specific fields in custom fields/relationships.

Automation-first data model: what the role needs

For each customer BGP session, automation typically needs:
- Local device + local interface
- Remote neighbor IP (v4/v6)
- Remote ASN
- VRF (or global routing)
- Policy inputs:
- inbound/outbound prefix limits
- allowed prefixes (static list, IRR-derived, or “customer owned prefixes”)
- communities, localpref, MED behavior
- BFD settings
- authentication password (optional)
- Operational toggles:
- admin state (planned/active/suspended)
- deployment stage (render-only vs apply)

Recommended custom fields (examples)

Attach these at sensible scopes (Tenant, VRF, Interface, Prefix, ASN, Device). Examples:

Tenant (Customer)
- customer_asn (integer, if not using ASN objects)
- routing_policy_tier (choice: strict / rpki_enforce / irr_warn / legacy)
- max_prefix_v4 / max_prefix_v6 (integer)
- bgp_password_ref (string reference to secret store path; not the password)

VRF
- import_rt / export_rt (if doing MPLS L3VPN-style)
- leak_to_global (bool, if you do controlled leaking)

Interface (PE-facing)
- bgp_session (bool)
- bfd (bool)
- session_role (choice: customer / upstream / peer)

Prefix (customer-owned announcements)
- announce (bool)
- max_length (int; useful for RPKI maxLength alignment)
- policy_group (optional: “default”, “blackhole”, etc.)

Relationship design: make “session” a first-class concept

A reliable pattern is to create a custom object (via a plugin) or approximate it with relationships:

BGP Session (logical entity)
- tenant → customer
- device → PE device
- vrf → VRF (nullable for global)
- local_asn / remote_asn
- local_ip / neighbor_ip (v4/v6)
- local_interface
- address_family (ipv4/ipv6)
- policy_in / policy_out (names or references to templates)
- enabled (bool)
- bfd (bool)
- password_ref (string reference to secret store)

Why this matters:
- You generate config from sessions, not from “interfaces that might imply sessions”.
- You can validate session completeness and avoid surprises.

Naming conventions that scale

Pick naming rules and enforce them in validation:
- Route-maps / policy statements:
- RM-IN-CUST-<tenant_slug>-V4
- RM-OUT-CUST-<tenant_slug>-V6
- Prefix-lists:
- PL-CUST-<tenant_slug>-ANNOUNCE-V4
- BGP neighbor groups/peer-groups:
- PG-CUST-<tenant_slug>-V4

Consistency makes diffs readable and troubleshooting faster.

Ansible role architecture: idempotent templates for neighbor sessions, route-maps, prefix-lists and BFD across Cisco and mixed vendors, including secrets management

For advanced ISP automation, avoid “one massive playbook.” Use a role structure that separates:
- data gathering
- rendering
- validation
- deployment
- verification
- rollback

Suggested role layout

code 
roles/
  bgp_provision/
    defaults/main.yml
    vars/main.yml
    tasks/
      main.yml
      fetch_netbox.yml
      build_model.yml
      render.yml
      preflight.yml
      deploy.yml
      verify.yml
      rollback.yml
    templates/
      iosxe_bgp.j2
      nxos_bgp.j2
      junos_bgp.j2
      common_policies.j2

Cross-vendor strategy: normalize data, then render vendor templates

  • Build a vendor-neutral session model in build_model.yml.
  • Templates per platform consume the same model.

Example session model (conceptual):
- sessions:
- tenant: acme
- vrf: CUST-ACME
- afi: ipv4
- neighbor_ip: 192.0.2.10
- remote_asn: 64501
- max_prefix: 200
- bfd: true
- policy_in: RM-IN-CUST-acme-V4
- policy_out: RM-OUT-CUST-acme-V4

Secrets management: never store BGP passwords in NetBox or Git

Battle-tested options:
- Ansible Vault for small/medium environments
- HashiCorp Vault or AWS Secrets Manager for large-scale and rotation
- Store only a reference in NetBox (bgp_password_ref), e.g.:
- secret/network/bgp/acme/v4

Then resolve it at runtime.

Idempotent deployment: prefer native modules where possible

  • Cisco IOS-XE/NX-OS: cisco.ios.ios_config, cisco.nxos.nxos_config with match and replace controls
  • Junos: junipernetworks.junos.junos_config with update: merge or check_commit

Use:
- candidate config (when supported)
- diff mode
- commit confirmed (Junos) or staged replace blocks

Example: Ansible tasks (NetBox → render → deploy)

yaml 
---
- name: Fetch BGP sessions from NetBox (example via netbox.netbox lookup)
  ansible.builtin.set_fact:
    nb_bgp_sessions: "{{ query('netbox.netbox.nb_lookup', 'custom_field_data', api_endpoint=netbox_url, token=netbox_token) }}"
  tags: [always]

- name: Build vendor-neutral session model (simplified)
  ansible.builtin.set_fact:
    bgp_model:
      device: "{{ inventory_hostname }}"
      local_asn: "{{ hostvars[inventory_hostname].local_asn }}"
      sessions: "{{ sessions_normalized }}"
  vars:
    sessions_normalized: >-
      {{ nb_bgp_sessions
         | selectattr('device.name', 'equalto', inventory_hostname)
         | map('extract_session_fields')
         | list }}
  tags: [render]

- name: Render IOS-XE BGP config
  ansible.builtin.template:
    src: iosxe_bgp.j2
    dest: "rendered/{{ inventory_hostname }}-bgp.cfg"
  when: ansible_network_os == 'cisco.ios.ios'
  tags: [render]

- name: Deploy IOS-XE BGP config (idempotent block replace)
  cisco.ios.ios_config:
    src: "rendered/{{ inventory_hostname }}-bgp.cfg"
    match: line
    replace: block
    diff_against: running
  when: ansible_network_os == 'cisco.ios.ios'
  tags: [deploy]

Notes:
- extract_session_fields would be a filter plugin (recommended) to keep Jinja and tasks clean.
- For Junos, prefer commit confirmed to reduce risk.

BFD and session grouping

Practical improvements:
- Use peer-groups / neighbor-groups where platform supports it.
- Enable BFD per neighbor only when customer equipment supports it.
- Tie BFD config to the session model (bfd: true/false) rather than guessing from interface type.

Python validation and pre-flight checks: automated linting of NetBox data, route policy simulation, and safe-change verification scripts

Automation fails in two ways: bad data and correct data with bad policy outcomes. Python gates catch both before Ansible changes devices.

What to validate (minimum viable gates)

Data linting
- Every session has:
- device, neighbor IP, remote ASN, VRF/global, AFI
- Neighbor IP is on the correct connected subnet (/31 or /127)
- No duplicate neighbor IPs on the same device/VRF/AFI
- Max-prefix present and within sensible bounds
- Naming conventions for policies are consistent and resolvable

Policy linting
- Outbound announce set is non-empty only when expected
- No “default-originate” unless explicitly allowed
- No accidental transit:
- customer-to-upstream route leaking prevented by tags/policy tiers
- Communities applied as expected

RPKI / IRR pre-checks
- For each customer prefix reminder:
- RPKI state (valid/invalid/not-found) captured
- IRR route objects fetched if in IRR mode
- Flag invalids; decide whether to block or warn based on tier

Route policy simulation (what “battle-tested” looks like)

Don’t rely only on config rendering. Simulate intended outcomes:
- Build expected prefix-lists from NetBox Prefix objects (announce=true)
- Compare to:
- IRR-generated prefix set (if used)
- ROA maxLength (for warnings)
- Ensure route-map/policy will:
- permit only expected prefixes
- attach required communities
- set localpref appropriately

If you can, run offline tests with Batfish for multi-vendor policy analysis. Even partial adoption (edge only) catches painful mistakes (e.g., permit-any statements, wrong direction application).

Example Python pre-flight: NetBox session sanity + duplication check

python 
#!/usr/bin/env python3
import ipaddress
import sys
from collections import defaultdict

def die(msg):
    print(f"ERROR: {msg}", file=sys.stderr)
    sys.exit(2)

def warn(msg):
    print(f"WARN: {msg}", file=sys.stderr)

def validate_sessions(sessions):
    seen = defaultdict(list)

    for s in sessions:
        required = ["device", "afi", "neighbor_ip", "remote_asn", "enabled"]
        for k in required:
            if k not in s or s[k] in (None, "", []):
                die(f"Missing required field {k} in session: {s}")

        # Parse IP and normalize key for duplicates
        try:
            n_ip = ipaddress.ip_address(s["neighbor_ip"])
        except ValueError:
            die(f"Invalid neighbor_ip {s['neighbor_ip']} in session: {s}")

        vrf = s.get("vrf") or "global"
        key = (s["device"], vrf, s["afi"], str(n_ip))
        seen[key].append(s)

        # Policy guardrails
        maxp = s.get("max_prefix")
        if maxp is None:
            warn(f"{key}: max_prefix missing (consider blocking this in strict mode)")
        elif int(maxp) < 1 or int(maxp) > 500000:
            die(f"{key}: max_prefix out of bounds: {maxp}")

        tier = s.get("routing_policy_tier", "legacy")
        if tier not in ("strict", "rpki_enforce", "irr_warn", "legacy"):
            die(f"{key}: unknown routing_policy_tier={tier}")

    # Duplicate neighbor detection
    dups = [k for k, v in seen.items() if len(v) > 1]
    if dups:
        for k in dups:
            die(f"Duplicate neighbor session detected for {k}: {seen[k]}")

    print(f"OK: validated {len(sessions)} sessions")

if __name__ == "__main__":
    # sessions would come from NetBox API; keep example minimal here
    example_sessions = [
        {"device": "pe1", "vrf": "CUST-ACME", "afi": "ipv4", "neighbor_ip": "192.0.2.10", "remote_asn": 64501, "enabled": True, "max_prefix": 200, "routing_policy_tier": "strict"},
    ]
    validate_sessions(example_sessions)

Where this fits:
- Run this script in CI and again as an Ansible pre-task.
- Fail fast before rendering or deployment.

Safe-change verification scripts (post-deploy gates)

After config push, use Python (or Ansible modules) to verify:
- BGP session state: Established within expected time window
- Prefix counts: received/advertised within expected bounds (max-prefix not tripped)
- No unexpected route changes (diff RIB snapshots if available)
- No CPU/memory alarms on edge devices during churn

This becomes the trigger for automated rollback (next section).

Rollout, monitoring and rollback: staged cutover playbooks, realtime telemetry checks, automated rollback triggers and post-change reconciliation with NetBox

At scale, “deploy and eyeball” doesn’t work. Treat turn-ups like controlled releases.

Staged cutover playbooks

A practical sequence:
1. Stage 0 — Render only
- Pull NetBox data
- Validate
- Render configs
- Store artifacts (timestamped)
2. Stage 1 — Pre-config
- Create prefix-lists / policy statements (not applied yet)
- Create peer-groups / neighbor-groups
3. Stage 2 — Activate neighbor
- Add neighbor under BGP, apply policies, enable BFD if needed
4. Stage 3 — Verify
- Check session state, prefix counters, logs
5. Stage 4 — Reconcile
- Update NetBox session status (active) and record change metadata

Realtime telemetry checks

Pick checks that are fast and meaningful:
- BGP session state (per neighbor)
- Max-prefix warnings (platform-dependent counters/logs)
- Interface errors (CRC, drops) that can masquerade as routing issues
- Control-plane health (CPU spikes)
- Streaming telemetry or polling (gNMI/SNMP/API) if available

Keep the verification window realistic:
- eBGP should establish quickly, but prefix propagation and customer filtering changes can take longer.

Automated rollback triggers

Rollback shouldn’t be emotional; it should be rule-based.

Common triggers:
- Session doesn’t reach Established within X minutes
- Prefix received exceeds max-prefix threshold (or close to it, e.g., >90% unexpectedly)
- Advertised prefixes mismatch expected set (empty when it shouldn’t be, or unexpectedly large)
- Device health degradation beyond threshold during the change window

Rollback options by platform:
- Junos: commit confirmed is ideal (auto-revert if verify fails)
- IOS/NX-OS:
- Keep a pre-change checkpoint (where supported)
- Or maintain an “automation block” with replace: block so you can revert the block deterministically

Post-change reconciliation with NetBox

NetBox must reflect reality, or future automation will break.

Reconcile:
- Mark session object as active (or failed with reason)
- Store:
- last deployed timestamp
- config artifact hash
- neighbor state at completion
- If the device diverges (hotfixes), detect and either:
- import the change intentionally, or
- overwrite via automation after review

Common pitfalls

  • Modeling BGP implicitly (e.g., “if interface is in VRF, assume session exists”): this creates ghost sessions and surprises.
  • Storing secrets in NetBox: even “restricted” fields tend to leak into exports, logs, or backups. Store references only.
  • Not separating data normalization from rendering: Jinja becomes a programming language, tests become harder, and cross-vendor support becomes painful.
  • Prefix-lists without source-of-truth prefixes: if you can’t point to the authoritative list of customer announcements, your policy will drift.
  • No “tiered enforcement” for RPKI/IRR: going from warn → drop overnight often breaks customers; policy needs explicit, staged intent.
  • Verification that checks only session state: Established can still mean “routing is wrong.” Validate advertised/received routes and policy outcomes.
  • Non-idempotent config pushes: rewriting large sections can flap BGP, reset timers, or trigger churn on every run.
  • Rollback that depends on humans: if rollback is manual, it won’t be fast enough during a widespread turn-up window.

Key takeaways

  • Treat bgp sessions as first-class objects in NetBox (or an equivalent relationship model), not as side effects of interfaces.
  • Normalize NetBox data into a vendor-neutral model, then render per-platform templates.
  • Use Python pre-flights to catch bad data, duplicates, and policy hazards before touching routers.
  • Build rollout playbooks with staging, verification gates, and deterministic rollback—that’s what makes automation safe at ISP scale.
  • Make RPKI/IRR handling explicit and tiered, so enforcement is predictable and operationally manageable.
Back to Blog