Using text finite state machines (Part 1)

Before networking vendors became API friendly we were stuck with expect/screen-scraping. Untold hours of chunking busybox data into dictionaries, splitting lines based on spaces, tabs, or colons, then trying to skip lines/headers/footers etc. And while this can be done effectively and accurately, I'd like to introduce a package to hopefully save you some time.

I have a project where I need to get some information from a Calix E7-20 but the information is unreliable when retrieved from CMS. Furthermore, the CMS connection may be down when the information is needed the most.

Background. We are upgrading from GPON to XGSPON which requires a massive amount of fiber rework. One stage of the rework is to hot cut all the ONTs from one shelf to another in a maintenance window.

Problem. In order for the teams to know they got everyone back online, they need to be able to perform a snapshot before and after the maintenance window. My first thought is to use the Calix Cloud service with SNMP traps or CMS reports. For some reason it seems that neither of these options provide accurate results. In my testing, if I have 4,000 ONTs on a shelf, I may only get half of those in the reports.

Solution. Run a few commands on the OLT before and after, compare the results, tell me who is missing.

Example. Below is an example of the results to the command "show ont detail", only, imagine you had 4,000 entries of these.

ONT                     : 1
Profile                 : 844G
Serial #                : abcdef
Manufacturing Serial #  : 
Registration ID         : 
Subscriber ID           : 
Description             : hut
Vendor ID               : 
PON port                : 20/16
PWE3 profile            : N/A
Alarm Thresholds
  Low NE Rx Pwr         : -30 dBm
  High NE Rx Pwr        : -7 dBm
  ONT SDBER rate US     : 5
  PON port SDBER DS rate: 5
ONT information         : <unavailable>
SDBER Rate US Current   : <unavailable>
SDBER Rate DS Current   : <unavailable>
Battery present alarm   : disabled
Admin status            : enabled
Operational status      : system disabled
Additional status info  : child provisioned, missing
PSE maximum power budget: 30 watts
Poe high power mode     : disabled

My first thought used to be to simply split based on the colons, count the columns, skip the blank lines, headers, footers, etc. And this would work and is what you may be doing today. But heres another thought.

Proposed solution. Use Google's text finite state machine to parse semi formatted text, like the example above. Lets use a smaller version of the example data above and create a textfsm.

Example data.

ONT                     : 1
Profile                 : 844G
Serial #                : abcdef
Manufacturing Serial #  : 

Corresponding template.

Value ONT (\d+)
Value Profile (\S+)
Value Serial (\S+)
Value ManufacturingSerial (\S+)

Start
  ^ONT\s*:\s*${ONT}
  ^Profile\s*:\s*${Profile}
  ^Serial #\s*:\s*${Serial}
  ^Manufacturing Serial #\s*:\s*${ManufacturingSerial} -> Record

Results.

['1','844G','88256D','']

Lets break down whats going on here.

1. We need structured data. Something we can utilize regular expressions to parse. Exactly the same requirement as screenscraping
2. We need to define the columns of data we expect to store for each row
3. We need to define where/how to find the columns and where to store the found data
4. We need to define the end of one row. In otherwords, when should we stop associating values with the current group and move to another group.

I think we can skip step one and move straight to step two. Defining the columns. In my example above, we can clearly see that the columns of data we want to store are ONT, Profile, Serial, and Manufacturing Serial. So we tell textfsm about these columns by using the "Value" statement, then the name, then the regex to match the data being stored there.

Step three is where most of the magic happens. Here we define the regex that will pull the data from the source text and build a row in the table. To tell textfsm that we are starting the State Definitions, we use the keyword "Start.". Then we create a state definition, separated by a new line and indented, for each column. The formula for ONT is

^ONT\s*:\s*${ONT}

which expands to

^ONT\s*:\s*(\d+)

because that is what we defined in the Value definitions section. (${ONT} = (\d+))

Step four. We need to tell textfsm when to stop grouping values into a single row. In our case, the last piece of data that should be grouped with an ONT is

Poe high power mode     : disabled

After we store that data, anything parsed after that should go to the next ONT. We do this with the Record Action.

^Manufacturing Serial #\s*:\s*${ManufacturingSerial} -> Record

In the end, we end up with a nicely parsed version of our source text that we can use pragmatically. But why all the hassle if we end up with the same results?

1. TextFSM provides a template based method to interpret data vs a complicated custom logic function which makes adapting to change much easier. If the vendor changes the output, updating the template can be much easier than updating the screenscraper
2. TextFSM provides a easily reusable interface to parsing the data. You can imagine that as you start to use this package, you'll develop a library of template files. These files can be easily updated, understood, and adapted as needed.
3. TextFSM provides advanced textual operations that can be done by updating the template file instead of writing custom code. Check out actions on their docs.

In the end we are really just trading the custom code complexity for essentially a Python DSL. As usual, the more we abstract away from the lower level components and operate in a higher language, the less flexibility we have. But I think in this case the pros far out weigh the cons.

Github link to this example: https://github.com/MajesticFalcon/textfsms

Github link to textfsm: https://github.com/google/textfsm

Textfsm doc link: https://github.com/google/textfsm/wiki/TextFSM