Tutorial: Build a GitLab CI/CD GitOps Pipeline for Network Automation with Ansible, NetBox and Netmiko

This tutorial walks through an intermediate, production-minded approach to gitlab ci cd network automation using a GitOps workflow: NetBox as the source of truth, Ansible for orchestration and templating, and Netmiko for network device transport. The goal is repeatable, testable, and auditable configuration delivery with safe promotion from feature branches to production.

Prereqs and repo layout: NetBox models, inventory design, Ansible roles, and GitLab repo structure

Prerequisites

• GitLab (SaaS or self-managed) with CI runners that can run Docker jobs
• NetBox (v3.x or v4.x) with:
• Devices, device roles, sites, platforms
• Interfaces/IPs/VLANs/VRFs as applicable
• Custom fields/tags if needed for automation flags (e.g., automation_enabled)
• Ansible (2.15+ recommended) plus:
• netbox.netbox collection (to query NetBox)
• ansible.utils, ansible.netcommon
• Optional: vendor collections (e.g., cisco.ios, arista.eos) if you use native modules
• Netmiko (Python package) for CLI-based configuration transport
• A lab environment for testing (e.g., containerlab, EVE-NG, CML) or non-production devices

NetBox modeling checklist (keep it automation-friendly)

• Use device roles and platforms consistently (e.g., leaf, spine, platform iosxe, eos)
• Ensure each device has a primary IP (or clearly defined management IP)
• Capture “intent” in structured data:
• VLANs/VRFs/Prefixes in NetBox IPAM
• Interface mode (access/trunk), VLAN assignment, LAG membership
• Tags for scoping (edge, datacenter, region-us-east)
• Define automation boundaries:
• A tag like managed-by-ansible
• Or a custom field automation_enabled=true

Repository layout (practical GitOps structure)

A clean layout makes CI predictable:

.
├── ansible/
│   ├── ansible.cfg
│   ├── inventories/
│   │   ├── netbox.yml
│   │   └── group_vars/
│   │       ├── all.yml
│   │       └── dc1.yml
│   ├── playbooks/
│   │   ├── render.yml
│   │   ├── deploy.yml
│   │   ├── validate.yml
│   │   └── rollback.yml
│   ├── roles/
│   │   └── campus_access/
│   │       ├── templates/
│   │       │   └── ios_access.j2
│   │       ├── tasks/
│   │       │   ├── render.yml
│   │       │   └── deploy_netmiko.yml
│   │       └── defaults/main.yml
│   └── filter_plugins/
├── configs/
│   ├── rendered/          # generated artifacts (often CI artifacts, not committed)
│   └── golden/            # optional reference configs, carefully controlled
├── tests/
│   ├── unit/
│   └── integration/
├── .gitlab-ci.yml
└── pyproject.toml / requirements.txt

Inventory design: NetBox dynamic inventory

Prefer dynamic inventory so device sources are consistent and changes flow from NetBox → pipeline.

Example ansible/inventories/netbox.yml:

plugin: netbox.netbox.nb_inventory
api_endpoint: "{{ lookup('env','NETBOX_URL') }}"
token: "{{ lookup('env','NETBOX_TOKEN') }}"
validate_certs: true

# Pull only devices intended for automation
query_filters:
  - tag: managed-by-ansible
  - status: active

group_by:
  - device_roles
  - sites
  - platforms

# Use primary_ip4 for management, fall back to primary_ip
compose:
  ansible_host: primary_ip4.address | default(primary_ip.address) | regex_replace('/.*$', '')

Mapping intent to device config: templates, Jinja2, and pulling authoritative data from NetBox

The GitOps pattern works best when the repository stores how to render config (templates/roles), while NetBox stores what should be rendered (intent/state).

Recommended approach

1. NetBox holds intent: VLANs, interface assignments, IPs, BGP neighbors, etc.
2. Ansible pulls NetBox data and normalizes it into structured variables.
3. Jinja2 templates render device configs deterministically.
4. Rendered config is:
5. stored as CI artifacts
6. diffed vs current/golden
7. deployed after approvals

Data extraction pattern (NetBox → Ansible facts)

Use netbox.netbox.nb_lookup or inventory plugin + hostvars to gather device details. A common pattern is to create a “normalized facts” task that builds variables the templates consume.

Example task snippet (conceptual):

- name: Build intended VLAN list from NetBox
  set_fact:
    intended_vlans: >-
      {{
        query('netbox.netbox.nb_lookup', 'vlans',
              api_endpoint=lookup('env','NETBOX_URL'),
              token=lookup('env','NETBOX_TOKEN'),
              filter={'site': hostvars[inventory_hostname].site.slug})
        | map(attribute='value')
        | list
      }}

Jinja2 templating: keep templates deterministic

Guidelines that reduce surprise diffs:

• Sort lists (VLAN IDs, interface names, prefix lists)
• Normalize strings (e.g., lower-case tags, consistent spacing)
• Avoid environment-dependent rendering (timestamps, random IDs)
• Keep platform-specific logic isolated (separate templates or role vars per platform)

A minimal Jinja2 example (VLAN block):

{% for vlan in intended_vlans | sort(attribute='vid') %}
vlan {{ vlan.vid }}
 name {{ vlan.name }}
{% endfor %}

“Intent to config” checklist

• [ ] Define a small, stable schema of variables templates expect (intended_vlans, intended_interfaces, routing)
• [ ] Validate required fields exist (fail early in CI)
• [ ] Render configs in CI and review diffs before deploy
• [ ] Keep secrets out of templates (use vault/CI variables)

GitLab CI examples: pipelines for linting, unit tests, change validation, and feature branch previews

A good gitlab ci cd network automation pipeline should do four things reliably:

1. Lint and static checks (fast feedback)
2. Render configs and produce diffs (reviewable output)
3. Validate intent (schema + policy checks)
4. Support feature-branch previews (non-prod/lab or “render-only”)

Example .gitlab-ci.yml (practical baseline)

This example uses a Python image, installs requirements, renders configs, and produces artifacts and diffs. Adjust paths and tooling to match your repo.

stages:
  - lint
  - test
  - render
  - validate
  - deploy

default:
  image: python:3.11-slim
  before_script:
    - python -V
    - pip install -r requirements.txt
    - ansible-galaxy collection install -r ansible/collections/requirements.yml

variables:
  ANSIBLE_STDOUT_CALLBACK: yaml
  PIP_DISABLE_PIP_VERSION_CHECK: "1"

lint:ansible:
  stage: lint
  script:
    - pip install ansible-lint yamllint
    - yamllint .
    - ansible-lint ansible/playbooks ansible/roles
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH

test:unit:
  stage: test
  script:
    - pip install pytest
    - pytest -q tests/unit
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH

render:configs:
  stage: render
  script:
    - mkdir -p configs/rendered
    - ansible-playbook -i ansible/inventories/netbox.yml ansible/playbooks/render.yml
    - find configs/rendered -type f -maxdepth 2 | sort
  artifacts:
    when: always
    expire_in: 7 days
    paths:
      - configs/rendered/
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH

validate:diff:
  stage: validate
  script:
    - apt-get update && apt-get install -y git diffutils
    # Example: compare rendered configs to a committed baseline (optional pattern)
    - |
      if [ -d "configs/golden" ]; then
        diff -ruN configs/golden configs/rendered || true
      else
        echo "No golden configs directory; skipping diff."
      fi
  artifacts:
    when: always
    expire_in: 7 days
    paths:
      - configs/rendered/
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

deploy:staging:
  stage: deploy
  script:
    - ansible-playbook -i ansible/inventories/netbox.yml ansible/playbooks/deploy.yml --limit "tag_staging"
  environment:
    name: staging
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      when: manual
      allow_failure: false

Feature branch previews (MR pipelines)

Effective MR workflows typically include:

• Render configs and attach artifacts to the MR
• Post a summary (GitLab job output, or a comment via API)
• Optionally deploy to a virtual lab for functional validation

Recommended MR checks:

• [ ] ansible-lint, yamllint
• [ ] Unit tests for filters/helpers (Python)
• [ ] Render-only job with artifacts
• [ ] Policy validation (see below)

Change validation: policy checks and schema validation

Add a lightweight validator job to catch unsafe changes:

• Ensure only approved roles can touch certain device groups
• Prevent VLAN deletions without approvals
• Require ticket references in commit/MR title for production changes

Implementation options:

• Simple Python policy script that inspects NetBox data + rendered config
• Ansible task that asserts invariants (assert:)
• Custom CI job that runs pyyaml schema validation on generated data

Deploying changes safely: Ansible+Netmiko playbooks, transactional deploy patterns, testing on virtual labs

Netmiko is often chosen when:
- vendor native modules are incomplete
- you need consistent CLI behavior across platforms
- you want to push config snippets with explicit logging

Deployment pattern: render → diff → deploy

Avoid “generate on the fly” during deploy. Render first, then deploy exactly what was reviewed.

A typical flow:

1. render.yml generates configs/rendered/<device>.cfg
2. validate.yml checks syntax/policy and optionally diffs current state
3. deploy.yml pushes config in a controlled way

Transactional deploy patterns (practical options)

Different platforms support different safety mechanisms:

• Candidate configuration + commit (best): Junos, IOS-XR (commit replace), some NOSs
• Checkpoint/rollback features: some platforms support config archive and rollback commands
• Atomic blocks: push per feature with strict ordering and fail-fast behavior
• Maintenance windows: limit concurrency and enforce approvals

Even with Netmiko, you can implement a “transaction-like” workflow:

• Pre-check: capture running config and save artifact
• Apply: push changes
• Post-check: verify intended state (interfaces up, BGP sessions, VLAN presence)
• If failed: revert using saved config / rollback commands

Example Ansible playbook using Netmiko (conceptual but deployable)

This example uses the ansible.netcommon.net_put-style patterns conceptually, but for Netmiko you’ll commonly call a custom module, a role task using ansible.builtin.command with a Python wrapper, or ansible.builtin.script. A practical approach is to run a small Python script that uses Netmiko and is driven by Ansible.

ansible/playbooks/deploy.yml:

- name: Deploy rendered configs via Netmiko
  hosts: all
  gather_facts: false
  serial: 5   # limit blast radius
  vars:
    rendered_dir: "{{ playbook_dir }}/../../configs/rendered"
  tasks:
    - name: Ensure rendered config exists
      ansible.builtin.stat:
        path: "{{ rendered_dir }}/{{ inventory_hostname }}.cfg"
      register: cfg_file

    - name: Fail if missing rendered config
      ansible.builtin.fail:
        msg: "Missing rendered config for {{ inventory_hostname }}"
      when: not cfg_file.stat.exists

    - name: Backup running config (device-side command)
      ansible.builtin.command: >
        python3 ansible/scripts/netmiko_exec.py
        --host {{ ansible_host }}
        --platform {{ platform | default('cisco_ios') }}
        --cmd "show running-config"
      register: running
      changed_when: false

    - name: Save running config as artifact (runner filesystem)
      ansible.builtin.copy:
        content: "{{ running.stdout }}"
        dest: "{{ rendered_dir }}/backups/{{ inventory_hostname }}-running.txt"

    - name: Push config snippet
      ansible.builtin.command: >
        python3 ansible/scripts/netmiko_push_config.py
        --host {{ ansible_host }}
        --platform {{ platform | default('cisco_ios') }}
        --config {{ rendered_dir }}/{{ inventory_hostname }}.cfg
      register: push_result

    - name: Basic post-check (example)
      ansible.builtin.command: >
        python3 ansible/scripts/netmiko_exec.py
        --host {{ ansible_host }}
        --platform {{ platform | default('cisco_ios') }}
        --cmd "show ip interface brief"
      register: post
      changed_when: false

Notes for production hardening:
- Use GitLab masked CI variables for credentials (or SSH keys) and avoid printing secrets
- Prefer SSH keys + device AAA over passwords
- Set serial and/or max_fail_percentage
- Add explicit “device readiness” checks (SSH reachable, correct prompt, correct platform)

Testing on virtual labs

A lab pipeline is a high-ROI investment:

• Use containerlab to spin up EOS/SONiC (or other supported images) where licensing allows
• Use EVE-NG/CML for IOS-XE/IOSv testing
• Run a CI stage that:
• boots the topology
• loads base config
• runs the same render/validate/deploy workflow
• runs post-change verification tests

This provides confidence that template changes won’t brick access or break routing at scale.

Rollback, audit and promotion: merge strategies, drift detection, and production hardening for enterprise networks

Merge/promotion strategy

For network GitOps, keep promotion predictable:

• Feature branches: render + validate + lab deploy (optional)
• Merge request: requires review from network peers
• Main branch: represents “approved intent”
• Tagged releases: represent “approved deployments” (useful for change windows)

Common patterns:

• Squash merges to keep history readable per change
• Require MR approvals for main
• Use protected branches/environments in GitLab:
• deploy:staging manual
• deploy:prod manual with additional approvals

Rollback strategies

Rollbacks are platform-specific; don’t wait until an outage to design them.

Practical rollback options:

1. Device-native rollback (preferred where supported)
2. Restore from captured running config (push previous known-good)
3. Re-render from last release tag and re-deploy

Implementation checklist:

• [ ] Always capture pre-change state (running config, key show commands)
• [ ] Store backups as CI artifacts and/or in an immutable object store
• [ ] Define rollback time objectives (how quickly should you be able to revert?)
• [ ] Practice rollback in lab at least quarterly

Drift detection

Drift happens when someone changes the device outside GitOps.

Effective drift detection loop:

• On a schedule (nightly):
• Pull device running config (Netmiko “show run”)
• Render intended config from NetBox + repo
• Diff and alert when differences exceed a threshold

In GitLab, implement as a scheduled pipeline:
- drift:collect job saves current configs as artifacts
- drift:diff job compares against rendered intent
- If drift detected: open an issue or notify a channel (keep notifications actionable)

Enterprise hardening

For larger environments:

• Concurrency control: use serial, batch by site/role, and maintenance windows
• Change scope controls:
• Only allow deploy to production for tagged devices or site group
• Require explicit --limit or CI variable DEPLOY_SCOPE
• Observability:
• Log every command pushed (minus secrets)
• Capture device prompt and session metadata for audit
• Secrets management:
• GitLab CI masked variables + protected variables
• Consider HashiCorp Vault integration for short-lived credentials
• Idempotence:
• Templates should produce stable output
• Avoid “append-only” CLI changes that accumulate duplicates

Common pitfalls

• NetBox data quality is inconsistent: missing primary IPs, mismatched platforms, or stale interface data causes render failures or wrong config.
• Templates grow into untestable monoliths: split by feature/role and normalize data first.
• Pushing config without preview: always render and diff in CI; avoid “live rendering” during deploy.
• No blast-radius controls: lack of serial, limits, or environment protections can turn a small change into a large outage.
• Secrets leak into logs: ensure scripts never print passwords/tokens; redact command output where needed.
• Drift is ignored: without scheduled drift detection, GitOps becomes “best effort” instead of authoritative.

Key takeaways

• NetBox works best as an authoritative intent source, not a loose inventory list—model consistently and validate early.
• A reliable gitlab ci cd network automation pipeline should lint, test, render, diff, and only then deploy with approvals.
• Treat deployment as rendered-artifact promotion: what’s reviewed is what’s pushed.
• Use transactional patterns (pre-check/backup/apply/post-check/rollback) to reduce risk even when using CLI transport via Netmiko.
• Add drift detection and protected environment promotion to make the workflow durable for enterprise networks.