Ethernet Link Checker (multi-vendor/using Schprokits Table output)

eth_nei_table.yml(for junos)

# for hosts wtih _os = 'junos'
---
eth_nei_table:
  rpc: get-lldp-neighbors-information
  item: lldp-neighbor-information
  key: lldp-local-interface
  view: eth_nei_record

eth_nei_record:
    fields:
        device: lldp-remote-system-name
        port: lldp-remote-port-description

eth_nei_table.yml(for nxos)

# for hosts with _os = 'nxos'
---
eth_nei_table:
    command: show cdp neighbors
    item: .//ROW_cdp_neighbor_brief_info
    key: intf_id
    view: eth_nei_record

eth_nei_record:
    fields:
        device: device_id
        port: port_id

get_eth_nei

---
- info: Retrieve Ethernet port neighbor information, store to files
  actions:
    - info: Get ethernet neighbor data
      get_table: name=eth_nei_table file="tables/{{ _os }}/eth_nei_table.yml"
      opipe_file: file=eth_nei.json indent=2

link_checker.py

# Copyright 2014 Schprokits, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""
Example "Link checker" program that uses the output tables created by
Schprokits in order to match actual ethernet links against a "known good"
definition file.

In this file are three class definitions:

    (*) LinkspecDB - used to store/model the EXPECTED link neighbors
    (*) HostsDB - used to store/model the ACTUAL link neighbors
    (*) LinkChecker - used to compare the two and print a report

See the help() on each of these for further deteails.

For demo purposes, the actual filename used to store things are hard-coded;
but you could easily add command-line parsing.  For the demo, the following
files are defined:

    (*) 'hosts' is a flat-file/line-separated list of host-names used
        during the data collection process

    (*) 'linkcheck.yml' contains the EXPECTED link neighbors.  See
        help(LinkspecDB) for example/format of this file.

    (*) 'eth_nei.json' is the file-name that stores each of the ACTUAL
        host specific data.  So if there are four hosts collected, there
        are four eth_nei.json files that were generated by Schprokits
        See help(HostsDB) for example/format of this file.
"""

import yaml
from os import path
import re
from itertools import ifilter

HOSTS_FILE = 'hosts'
LINKSPEC_FILE = 'linkspec.yml'
NEIDATA_FILE = 'eth_nei.json'

##### ========================================================================
#####                   LinkspecDB
##### ========================================================================


class LinkspecDB(object):
    """
    LinkpecDB - Contains the connection map that *SHOULD-BE*

    The purpose of `LinkspecDB` is to load in a structured set of data
    that represents the mapping between a "host", it's ports and the
    expected remote (host, port) tuple.  Here is an example expressed
    in YAML format:

        ---
        ex1:
            ge-1/0/24.0: ex2, ge-0/0/24.0
            ge-0/0/30.0: ex2, ge-0/0/38.0
            ge-2/0/8.0: ex2, ge-0/0/2.0

        n9k1:
            ethernet1/48: n9k2, ethernet1/1
            ethernet2/11: n9k2, ethernet2/2
            ethernet2/12: n9k2, ethernet2/1

    The above says, for example:

        host "ex1", port "ge-1/0/24.0" should be connected to
        remote-host "ex1" on its port "ge-0/0/24.0"

    NOTES:

        You do not need to add duplicate pairs, i.e.
        (n9k1, ethernet1/48), (n9k2, ethernet1/1)
        needs only to be defined once either by
        the 'n9k1' section (shown above) or by
        a section for 'n9k2' (not shown|not needed)

    USAGE:

        spec_db = LinkspecDB(filename='linkspec.yml')
        spec_db.create_db()

    RESULTS:

        The contents of the "spec-file" are loaded into `LinkspecDB`
        as a list of sets that can be used for comparison against
        the actual data.  The resulting set list would look something
        like the following:

        `spec_db.links`, for example, results in a list-of-sets:
          [
            set([('n9k1', 'ethernet1/48'), ('n9k2', 'ethernet1/1')]),
            set([('n9k1', 'ethernet2/11'), ('n9k2', 'ethernet2/2')]),
            set([('n9k1', 'ethernet2/12'), ('n9k2', 'ethernet2/1')]),
            set([('ex1', 'ge-1/0/24.0'), ('ex2', 'ge-0/0/24.0')]),
            set([('ex1', 'ge-0/0/30.0'), ('ex2', 'ge-0/0/38.0')]),
            set([('ex1', 'ge-2/0/8.0'), ('ex2', 'ge-0/0/2.0')])
          ]
    """

    def __init__(self, filename):
        if not path.isfile(filename):
            raise IOError('file not found: ' + filename)
        self._filename = filename          # name of file
        self._contents = None              # contents of file
        self._db = []                      # will store list of port-tuples

    ### ----------------------------------------------------------------------
    ###                 PROPERTIES
    ### ----------------------------------------------------------------------

    @property
    def links(self):
        return self._db

    ### ----------------------------------------------------------------------
    ###                 PUBLIC METHODS
    ### ----------------------------------------------------------------------

    def create_db(self):
        self._load()

        rdata_re = re.compile('(?P<host>.+?),\s+(?P<port>.+)')
        rdata_sub = lambda got: got.group('host').lower() + ',' + got.group('port').lower()

        rdata_get = lambda v: tuple(rdata_re.sub(rdata_sub, v).split(','))

        def mkset_host(host):
            return [
                set([(host, k), rdata_get(v)])
                for k, v in self._contents[host].items()]

        for host in self._contents:
            self._db.extend(mkset_host(host))

    ### ----------------------------------------------------------------------
    ###                 PRIVATE METHODS
    ### ----------------------------------------------------------------------

    def _load(self):
        try:
            self._contents = yaml.load(open(self._filename, 'r'))
        except Exception as exc:
            # YAML loader error most likely.  re-wrap as IOError
            raise IOError(exc.message)


##### ========================================================================
#####                   HostsDB
##### ========================================================================

class HostsDB(object):
    """
    HostsDB - Contains the *ACTUAL* link DB for the hosts

    The purpose of the `HostsDB` is to load the "eth_nei" table
    data that was created via Schprokits and stored within the
    $workdir/store/$host/$json_file area.  Each host data
    is the *ACTUAL* port neighbor information.  This information
    will be compared against the `LinkspecDB` file using
    an instance of `Wirechecker`.

    Each host specific file will be loaded into `HostsDB`
    and stored in the samelist-of-sets style as described
    in the `LinkspecDB`.

    For example, a "eth_nei.json" file for "n9k1" looks like
    this:

        {
          "num_items": 4,
          "name": "eth_nei_table",
          "contents": {
            "Ethernet1/48": {
              "device": "N9K2.cisconxapi.com(SNABC123)",
              "port": "Ethernet1/1"
            },
            "Ethernet2/11": {
              "device": "N9K2.cisconxapi.com(SNABC123)",
              "port": "Ethernet2/2"
            },
            "mgmt0": {
              "device": "c3550",
              "port": "FastEthernet0/22"
            },
            "Ethernet2/12": {
              "device": "N9K2.cisconxapi.com(SNABC123)",
              "port": "Ethernet2/1"
            }
          }
        }

    The above says, for example:

        host "n9k1", port "Ethernet1/48" *IS CONNECTED*
        to remote device "n9k2" port "Ethernet1/1

    USAGE:

        hosts_list = ['ex1','ex2','n9k1','n9k2']
        hosts_db = HostsDB(filename='eth_nei.json')
        hosts_db.create_db(hosts_list)

    RESULTS:

        Same lins-of-sets for each of the hosts provided
        in the call to :meth:`create_db`

    """
    def __init__(self, filename):
        self._basepath = path.join('work', 'store')
        self._filename = filename
        self._db = []

    @property
    def links(self):
        return self._db

    def load(self, host):
        filepath = path.join(self._basepath, host, self._filename)
        return yaml.load(open(filepath, 'r'))

    def create_db(self, hostlist):
        rdata_re = re.compile('(?P<host>[^\.]+)(?P<rest>.*)')
        rdata_sub = lambda got: got.group('host').lower()

        def mkset_host(host):
            data = self.load(host)
            contents = data['contents']
            return [
                set([(host.lower(), k.lower()),
                    (rdata_re.sub(rdata_sub, v['device']), v['port'].lower())])
                for k, v in contents.items()]

        for host in hostlist:
            self._db.extend(mkset_host(host))


##### ========================================================================
#####                   LinkChecker
##### ========================================================================


class LinkChecker(object):
    """
    LinkChecker - Compare the LinkSpec and Hosts link databases

    Once you have created and loaded both the LinkspecDB and the
    HostsDB data, you then pass these to the LinkChcker and run
    the report as shown in the example usage:

    USAGE:

        checker = LinkChecker(spec_db, hosts_db)
        checker.run()

    NOTES:

        You can customize the output of the report by creating
        a new sub-class of LinkChecker and overloading the
        methods that begin with "when_", see code for details.

    SAMPLE OUTPUT:

        jeremy@Jeremys-MacBook-Pro-2$ py checker.py
        ------------------------------------------------------------------------------
        OK:       n9k2 ethernet1/1     <----->       n9k1 ethernet1/48
        FAIL:       n9k1 ethernet2/11    <-!!!->       n9k2 ethernet2/21
           FOUND       n9k1 ethernet2/11    <----->       n9k2 ethernet2/2
        OK:       n9k2 ethernet2/1     <----->       n9k1 ethernet2/12
        OK:        ex2 ge-0/0/24.0     <----->        ex1 ge-1/0/24.0
        OK:        ex2 ge-0/0/38.0     <----->        ex1 ge-0/0/30.0
        OK:        ex2 ge-0/0/2.0      <----->        ex1 ge-2/0/8.0
        ------------------------------------------------------------------------------
    """

    def __init__(self, spec_db, hosts_db):
        self._spec_db = spec_db
        self._hosts_db = hosts_db
        self.num_ok = 0
        self.num_fail = 0

    # class method for formatting the (host, port) tuple
    strlink = lambda cls, l: "{0:>10} {1:<15}".format(l[0], l[1])

    # class method for line_sep
    line_sep = '-' * 78

    ### ----------------------------------------------------------------------
    ###                 PUBLIC METHODS
    ### ----------------------------------------------------------------------

    def run(self):
        self.num_ok = 0
        self.num_fail = 0

        self.when_run_start()

        for link in self._spec_db.links:
            side_a, side_b = link
            if link in self._hosts_db.links:
                self.num_ok += 1
                self.when_ok(side_a, side_b)
            else:
                self.num_fail += 1
                mate_a = self._find_mate(side_a)
                mate_b = self._find_mate(side_b)
                self.when_fail(side_a, side_b, mate_a, mate_b)

        self.when_run_end()

    ## --------------------------------------------------------------------
    ## 'when' methods could be overloaded by subclass should you want
    ## to change the default behavior, yo!
    ## --------------------------------------------------------------------

    def when_ok(self, side_a, side_b):
        """ called for each link that matches EXPECTED """
        fmt = self.strlink
        print "OK: " + fmt(side_a) + " <-----> " + fmt(side_b)

    def when_fail(self, side_a, side_b, mate_a, mate_b):
        """ called for each link that fails match of EXPECTED """
        fmt = self.strlink
        str_side_a = fmt(side_a)
        str_side_b = fmt(side_b)
        print "FAIL: " + str_side_a + " <-!!!-> " + str_side_b
        if mate_a:
            print "   FOUND {0} <-----> {1}".format(str_side_a, fmt(mate_a))
        if mate_b:
            print "   FOUND {0} <-----> {1}".format(str_side_b, fmt(mate_b))

    def when_run_start(self):
        """ called before comparing links, good for report banner start """
        print self.line_sep

    def when_run_end(self):
        """ called after comparing links, good for final report banner """
        total = self.num_ok + self.num_fail
        print self.line_sep
        print(
            "{total} LINKS CHECKED: OK={ok}, FAIL={fail}"
            .format(total=total, ok=self.num_ok, fail=self.num_fail))

    ### ----------------------------------------------------------------------
    ###                 PRIVATE METHODS
    ### ----------------------------------------------------------------------

    def _find_mate(self, this_side):
        """
        Used to find the `this_side` link in the hosts (ACTUAL) database and
        return back the other side ('mate') of the link.  If the 'this_side'
        does not exist, then return None.
        """
        match = lambda link: this_side in link
        found = next(ifilter(match, self._hosts_db.links), None)
        if not found: return None
        mate = found ^ set((this_side,))
        return mate.pop()


##### ------------------------------------------------------------------------
#####                   MAIN BLOCK
##### ------------------------------------------------------------------------

if __name__ == '__main__':
    # loads the hosts file into a list
    hosts_list = open(HOSTS_FILE, 'r').read().splitlines()

    # loads the EXPECTED link database
    spec_db = LinkspecDB(filename=LINKSPEC_FILE)
    spec_db.create_db()

    # loads the ACTUAL link databases
    hosts_db = HostsDB(filename=NEIDATA_FILE)
    hosts_db.create_db(hosts_list)

    # runs the report against the two
    checker = LinkChecker(spec_db, hosts_db)
    checker.run()

sample_output

jeremy@Jeremys-MacBook-Pro-2$ py checker.py 
------------------------------------------------------------------------------
OK:       n9k2 ethernet1/1     <----->       n9k1 ethernet1/48   
FAIL:       n9k1 ethernet2/11    <-!!!->       n9k2 ethernet2/21   
   FOUND       n9k1 ethernet2/11    <----->       n9k2 ethernet2/2    
OK:       n9k2 ethernet2/1     <----->       n9k1 ethernet2/12   
OK:        ex2 ge-0/0/24.0     <----->        ex1 ge-1/0/24.0    
OK:        ex2 ge-0/0/38.0     <----->        ex1 ge-0/0/30.0    
OK:        ex2 ge-0/0/2.0      <----->        ex1 ge-2/0/8.0     
------------------------------------------------------------------------------
6 LINKS CHECKED: OK=5, FAIL=1