Wozney Enterprises Limited

My journey into network automation

YAML

Jinja2

Python

Netmiko

All of these came together to help solve a problem for me. What inspired me to create this mess of code?

I had a job to update the configuration and firmware on over 250 VyOS routers. Coming soon is another job to build 40+ switch stacks (on about 180 switches) so I know that I’m going to be using this again.

In the past I’ve done configuration by hand and it has worked for a long time. Most of the time text configurations don’t change too much, I take a technically working configuration and modify it for a new job and then slap on whatever security template I think makes sense.

I’ve become proficient at scripting within Excel and I can generate single-shot configurations that follow strict rules, it’s helpful but not really practical for big jobs. I needed a tool that could pump out configurations quickly.

Excel Tricks

This post isn’t supposed to be about Excel tricks, but I figured I throw in the few things I’ve used in the past. It might help.

  • Concatenate – merge a string with a excel value:
    =CONCATENATE(“interface “, E1)
  • Vlookup – search a column of data for a match.
    =VLOOKUP(lookup_value, lookup_range, column, match_type)
  • Index/Match – search any data for a match. This is far faster and more flexible than vlookup.
    =INDEX(range, MATCH(lookup_value, lookup_range, match_type))

Once you’ve got your config output into one line per-row, then copy all that into a text file and swap the TABs with spaces and you’re done.

You should also watch You Suck at Excel by Joel Spolsky.

YAML

YAML Ain’t Markup Language is a human friendly data serialization standard for all programming languages. It’s basically a data storage system that you can read, but is easily parsed by Python into a data structure like a dictionary.

The hierarchy in the file is defined by indents which makes it easy to see the relationships between values.

Here’s a quick example of a YAML file:

SERVER: 1.1.1.1
OUTSIDE_NET: 2.2.2.0/22
INSIDE_NET: 192.168.0.0/24

PORTGROUP:
  - name: RTP
    description: SIP-RTP
    port: 10000-65535
  - name: SIP
    description: SIP
    port: 5060
  - name: RTSP
    description: RTSP
    port: 10000-65535
POLICIES:
 - Client-LAN:
  - rule: 1
    action: accept
    description: SIP
    destAdd: SERVER
    destGroup: SIP
    log: disable
    protocol: tcp_udp
  - rule: 2
    action: accept
    description: SIP-RTP
    destAdd: OUTSIDE_NET
    destGroup: RTP
    log: disable
    protocol: udp

You can see that even just looking at this, you can get a sense for what the data represents.

Jinja2

This Python module programmatically generates text blocks, from within a text file. It is like a programming language in a config file – so you can build up logical structures in a template file that gets interpreted and spits out a sensible configuration.

Jinja2 Text Blocks

Jinja2 lets you create blocks of text and silently insert programming control structures within that text.

Jinja2 control structures are marked using {% stuff %} and variables are accessed using {{ variable_name }}.

Use the hyphen like this {%- stuff %} so your jinja2 commands don’t mess up your line breaks or white space.

Jinja2 For/endfor

There’s other loop types, but this is all I used. If you need different tools then you can dig into the documentation.

Here’s an example:

firewall {
  group {
{%- for PG in PORTGROUP %}
    port-group name {
      description "give me all your cash”
      port 12345
    }
{%- endfor %}
}

So that section after the “firewall” statement could potentially repeat many times. In the context of this VyOS configuration it is building each firewall rule one at a time.

Indentation doesn’t matter for jinja2, but you can and probably should use it to keep your code readable.

Jinja2 If/else/elif/endif

{%- if ITEM.destAdd == "SERVER" %}
    address 10.10.10.10
{%- elif ITEM.destAdd %}
    address 30.30.30.30
{%- endif %}

Jinja2 Variables

firewall {
  group {
{%- for PG in PORTGROUP %}
    port-group {{ PG.name }} {
      description "{{ PG.description }}"
      port {{ PG.port }}
    }
{%- endfor %}
}

Here the variable PORTGROUP references the YAML definition PORTGROUP. The for loop cycles through the upper level entries of each group stored in the temporary variable (jinja2 only) PG.

The variable PG contains the data for ONE entry from YAML, and its own data is accessed with dot notation also taken from YAML.

Python

It is a bit of an understatement to say that Python is very popular. But it is and there are many code examples on the Internet. Many modules are available and can be easily installed with ‘pip’ while others can be downloaded from github.

Python Import objects

from jinja2 import Environment, FileSystemLoader
import yaml
import csv
from netmiko import ConnectHandler, SCPConn
from netmiko.vyos import VyOSSSH
import os
import scp
import subprocess

This pulls modules into play, allowing your script to access the objects and code within them.

Python Load data structures

with open("config.yaml") as _:
    config = yaml.load(_)

offices = csv.DictReader(open("offices.csv"))

This loads the YAML file into memory, and interprets it into a dictionary called ‘config’. I also imported the office room numbers/IP addresses into Python too. These two data sources are referenced by Jinja2 to build the configurations themselves.

Python Loops

offices = csv.DictReader(open("offices.csv"))

for row in offices:
  s_file = row['IPA'] + '-config.boot'
  d_file = '/config/' + s_file

There are multiple loop types, but I only used the for loop in my script.

Python Error handling

for row in offices:
  response = os.system("ping -c 1" + row['IPA'])
  if response == 0:
    print "Success, code goes here"
  else:
    print row['Office'], ' is down!'

I used the ‘os.system’ library to make sure I can ping a router before I attempt to connect to it and load a configuration file.

Python File writing and transfer

with open("config.yaml") as _:
    yamlfile = yaml.load(_)

for row in offices:
  s_file = row['IPA'] + '-config.boot'
  d_file = '/config/' + s_file

  configfile = open(s_file, 'w')

  rowdict['Office'] = row['Office']
  rowdict['IPA'] = row['IPA'] 
  rowdict['IPB'] = row['IPB'] 

  yamlfile.update(rowdict)
  template = ENV.get_template("jinja2.text")

  configfile.write(template.render(yamlfile))
  configfile.close()

Once I generated the configuration, I wanted to save it to a unique file for future reference.

Netmiko

Netmiko was based on paramiko (a ssh library for Python) but it simplifies using SSH with Python. It creates an object that is referenced by ssh functions within your program and allows your script to be less vendor specific because you don’t have to deal with the madness of screen scraping to get the job done.

Netmiko object

Before you can do anything with netmiko, you have to create the object:

from netmiko.vyos import VyOSSSH

for row in offices:
  rtr = {
      'device_type': 'vyos',
      'ip': row['IPA'],
      'username': 'admin',
      'password': 'strong-password,
  }

In my example, as the loop iterates through the list of offices it builds an object that netmiko can use for SSH or SCP.

Netmiko SCP

for row in offices:
  s_file = row['IPA'] + '-config.boot'
  d_file = '/config/' + s_file

  ssh_conn = ConnectHandler(**rtr)
  scp_conn = SCPConn(ssh_conn)
  scp_conn.scp_transfer_file(s_file, d_file)
  scp_conn.close()
        }

First Netmiko transfers the configuration file to the router.

Netmiko SSH

for row in offices:
  s_file = row['IPA'] + '-config.boot'
  d_file = '/config/' + s_file

  ssh_conn = ConnectHandler(**rtr)
  vsum = ssh_conn.send_command(
'md5sum config.boot').split()
  hsum = subprocess.check_output(
['md5sum', s_file]).split()

  if vsum[0] == hsum[0]:
    print row['IPA'] + " success!
  else:
    print "Configuration transfer failed"
        }

Then it uses SSH to log into the router to do a MD5 hash on the file to make sure it transferred correctly.

Useful Links

YAML

Jinja2

Python

Netmiko

Netmiko Github



Leave a reply

« Previous post