From guesswork to framework; firewall filters

Intro

When routing and forwarding tables fail to reveal a connectivity issue, there is an often overlooked tool that can improve mean-time-to-innocence: the firewall filter.

Flexible-match-mask

The star of the show in this post will be flexible-match-mask which will allow us to examine fields inside the frame, packet, payload, etc, and find a specific value to match on. The only time you need to use flexible masks is when you are matching on fields that aren't used enough for Juniper to build them in or are too dynamic or custom in nature.

flexible-match-mask {
    match-start
    byte-offset
    bit-length
    prefix
}

In order to take advantage of flexible matches, you need to be very familiar with the structure of Ethernet frames in hex.

1. Determine what layer in the OSI model you want start at. See here for available match-starts and their corresponding families
2. Determine how far into that layer you need to look. To match a DST MAC you don't need to skip anything because that is the first field in layer-2. For 802.1Q tag, you skip 14 bytes, 12 bytes for the MACs and 2 bytes for the ether-type
3. Determine how large the field is in bits.
4. Determine the data you want to match for in hex

Let's take a look at some various real life examples below.

Vanilla layer 2

In this real life scenario, we have an NNI with two customers where we utilize a bridge to connect their networks and don't have any layer 3 interfaces in their bridge domain.

Problem report: Customer A reports they cannot ping Customer B when using Q-in-Q.

There isn't a lot of options for troubleshooting here which makes the firewall a very useful tool. For purposes of this discussion, lets look at a hex dump of a Q-in-Q frame and discuss each of the fields.

After talking with the customer, they are using outer-tag 5 and inner-tag 3. We can use this information to plug into our flexible firewall filter and count packets at each end of the circuit. The test here is to see if both customers are sending bi-directional traffic on the correct VLANs.

family bridge {
    filter FM-QNQ-V5-V3 {
        interface-specific;
        term OUTER-V5 {
            from {
                flexible-match-mask {
                    match-start layer-2; <---- Starting layer
                    byte-offset 14;      <---- Jump to spot in layer
                    bit-length 32;       <---- Length of field to process
                    prefix 0x5;          <---- Value to match (5)
                }
            }
            then {
                count CNTR-V5;
                next term;               <---- Jump to inner VLAN matcher
            }
        }
        term INNER-V3 {
            from {
                flexible-match-mask {
                    match-start layer-2; 
                    byte-offset 18;
                    bit-length 32;
                    prefix 0x3;          <---- Same process as above
                }
            }
            then {
                count CNTR-V3;
                accept;
            }
        }
        term CATCH-ALL {
            then accept;
        }
    }
}

Now that we have our filter, lets see where the traffic is dropping.

So CUSTOMERA's traffic ingresses ge-0/0/0.0 and egresses ge-0/0/1.0 with the right tags but CUSTOMERB never responds; see the 0 packets on the input chain of ge-0/0/1.0. This is actionable evidence!

Root cause, customer misconfiguration.

Vanilla layer 3

There are of course a dozen ways to troubleshoot here, but lets assume no ability to get data from either customer except "this specific traffic flow doesn't work." After checking route tables, mac tables, firewall rules, etc and even sourcing IP traffic from one side to the other on your routers, you conclude it must their problem.

However, using firewall filters, we can take the expected traffic flow and match it bi-directionally on each end and confirm packets are being delivered as expected.

family inet {
    filter FF-INET-CUSTOMERA-to-CUSTOMERB-FLOW-1 {
        interface-specific;
        term INTERESTING-TRAFFIC-D1 {
            from {
                source-address {
                    198.51.100.2/32;
                }
                destination-address {
                    198.51.100.6/32;
                }
                destination-port 443;
            }
            then {
                count COUNTER-CUSTOMERA-to-CUSTOMERB-FLOW-1;
                accept;
            }
        }
        term INTERESTING-TRAFFIC-D2 {
            from {
                source-address {
                    198.51.100.6/32;
                }
                destination-address {
                    198.51.100.2/32;
                }
                source-port 443;
            }
            then {
                count COUNTER-CUSTOMERB-to-CUSTOMERA-FLOW-1;
                accept;
            }
        }
        term CATCH-ALL {
            then accept;
        }
    }
}

As the customer continues to attempt to access the resources, the firewall will reveal the issue.

As the screenshot above shows, packets are leaving our network but CUSTOMERB doesn't respond. This is actionable evidence!

Root cause, customer misconfiguration.

Okay so what, we can count some bytes in a few simple L2/L3 scenarios.. not super useful.

True.. Lets step it up

MPLS EVPN-VPWS

In order to trace this packet on the core side, we need to know what MPLS labels we are looking for. Lets take a look at the EVPN and LSPs.

Now that we know what we are looking for, lets create our filter!

firewall {
    family mpls {
        filter FM-MPLS-EVPN {
            term INTERESTING-TRAFFIC-EVPN {
                from {
                    flexible-match-mask {
                        match-start layer-3;
                        byte-offset 0;
                        bit-length 20;
                        prefix 0x49310;
                    }
                }
                then {
                    count CNT-EVPN;
                    next term;
                }
            }
            term INTERESTING-TRAFFIC-VPWS {
                from {
                    flexible-match-mask {
                        match-start layer-3;
                        byte-offset 4;
                        bit-length 20;
                        prefix 0x49300;
                    }
                }
                then {
                    count CNT-VPWS;
                    accept;
                }
            }
            term CATCH-ALL {
                then accept;
            }
        }
    }
}

I recommend taking advantage of filter templates if you think you will be debugging often.

family mpls {
    filter FM-MPLS-EVPN {
        term INTERESTING-TRAFFIC-EVPN {
            from {
                flexible-match-mask {
                    prefix 0x49301;
                    flexible-mask-name FM-MPLS-HEADER; <-- template
                }
            }
            then {
                count CNT-EVPN;
                next term;
            }
        }
        term INTERESTING-TRAFFIC-VPWS {
            from {
                flexible-match-mask {
                    match-start layer-3;
                    byte-offset 4;
                    bit-length 20;
                    prefix 0x49300;
                }
            }
            then {
                count CNT-VPWS;
                accept;
            }
        }
        term CATCH-ALL {
            then accept;
        }
    }
}
flexible-match FM-MPLS-HEADER {  <------- template definition
    match-start layer-3;
    byte-offset 0;
    bit-length 20;
}

One last trick

There are certain scenarios where you simply can't/don't know the traffic definition. For instance, if you shipped a router to remote location but it isn't coming online, you can filter for what you EXPECT to be arriving on your interface, but that doesn't help you actually fix anything.

In scenarios like this, there is a method to very carefully inspect traffic as it passes through the internals of the router.

Below is the MX204 (EA TRIO) functional block diagram. As data arrives from the WAN interface (PIC/MIC) and traverses the MQSS and LUSS, there is an opportunity to dump that traffic into RAM and then retreive it if you know at least something about the data.

In this example, I know that my non working device has a MAC of "0c:00:9d:d7:73:01"

start shell pfe network fpc0
test jnh 0 packet-via-dmem enable
test jnh 0 packet-via-dmem capture 0x3 0c009dd77301

Wait from some traffic to pass. In this case, I forced a single ICMP packet.

test jnh 0 packet-via-dmem capture 0x0
test jnh 0 packet-via-dmem decode

Below you'll see any packets that matched the filter you create above. You will see two entries for any given matched packet. One as the packet arrives at the MQSS block from the WAN interface (PIC/MIC), and another as the packet arrives at the MQSS block from the LUSS block.

Taking a look at the packet, lets see if we can figure out why we can't talk to our router

-1-2-3-4-5-6-7-8   <--- adding byte counter for visibility

0c00427f85010c00   <--- 6 bytes of destination mac 0c00427f8501
9dd7730181000064   <--- 6 bytes of source mac      0c009dd77301
        ^-------------- 2 bytes of ether-type (8100 = 802.1q protocol)
            ^---------- 4 bits of PCP/CFI
              ^-------- 12 bits of vlan-id. 0x64 is 100 in decimal
                        Now we know the vlan-id is 100
810000c808004500
^---------------------- 2 bytes of ether-type (8100 = 802.1q)
    ^------------------ 4 bits of PCP/CFI
      ^---------------- 12 bits of vlan-id. 0xc8 is 200 in decimal
         ^              Now we know we have a QnQ with 200 as the inner tag
         ^------------- 2 bytes of ether-type (0800 = IP)
            ^---------- 1 nibble denoting IP version (0x04 = IPv4)
             ^--------- 1 nibble denoting header length (0x05)
              ^-------- 1 byte DSCP (0x00)
005447d000004001
de6ec6336402c633
640108000f7ac94d
00006690a0f7000d
2ca008090a0b0c0d
0e0f101112131415
161718191a1b1c1d
1e1f202122232425
262728292a2b2c2d
2e2f303132333435
3637

So we've learned that this device is expecting 2x 0x8100 tags. If we kept going, we would learn the SRC and DST IP as well. These are useful and actionable pieces of evidence that may help you resolve your connectivity issues.

Obviously attempting to manually decode packets is completely unscalable. Luckily, there are several tools online that can help you.

Closing

Regardless of the protocol you need to troubleshoot, firewall filters can greatly reduce the MTTI. Instead of guessing at possible answers to the problem you're facing, use your network tools, write some firewall rules, and systematically walk the TCP/IP stack!