NAT Masquerade/VRF Issue(?)

jg159357 · October 22, 2023, 1:48pm

This is a continuation of my first post. Thanks @kraal

tl;dr - I either have a misconfiguration or found a bug on snat/vrf setups; please help.

I’m not sure I’m doing this right, but I’m trying to use VRFs to segment my home network from my “datacenter” lab. I hope the firewall diagram provides enough information to follow along; treat the interfaces as directly connected to end devices for testing. Off of the bond I have a Catalyst 3850 also with VRFs and I’m using loopbackk7724 (10.255.24.2) on the cisco to generate traffic through VLAN 1700 (172.17.0.11). Off of ether3 is also the 3850 using VRF Temp1 (203.0.113.1) and loopback1111 (1.1.1.1) to simulate the internet. I had the default firewall config to start with, but have deleted everything trying to find why the srcnat chain isn’t matching the outbound traffic. If I remove the interface match statement, ping through the firewall works, but it’s doing SNAT on every interface hop (as you’d expect with no filter).

I’ll add more posts with the tests to keep this first section “shorter”

/interface bridge
add frame-types=admit-only-vlan-tagged name=br_main vlan-filtering=yes
add name=lo_tun_vrf-datacenter protocol-mode=none
add name=lo_tun_vrf-edge protocol-mode=none
add name=lo_tun_vrf-home protocol-mode=none
add name=lo_vrf-datacenter protocol-mode=none
add name=lo_vrf-edge protocol-mode=none
add name=lo_vrf-home protocol-mode=none
/interface ethernet
set [ find default-name=ether3 ] comment=vrf-edge_wan1
set [ find default-name=ether4 ] comment=vrf-edge_wan2
set [ find default-name=ether5 ] l2mtu=9198
set [ find default-name=ether6 ] l2mtu=9198
set [ find default-name=ether7 ] l2mtu=9198
set [ find default-name=ether8 ] l2mtu=9198
/interface ipip
add comment=vrf-datacenter local-address=198.51.100.3 name=ipip_vrf-datacenter_vrf-edge remote-address=198.51.100.2
add comment=vrf-edge local-address=198.51.100.2 name=ipip_vrf-edge_vrf-datacenter remote-address=198.51.100.3
add comment=vrf-edge local-address=198.51.100.2 name=ipip_vrf-edge_vrf-home remote-address=198.51.100.1
add comment=vrf-home local-address=198.51.100.1 name=ipip_vrf-home_vrf-edge remote-address=198.51.100.2
/interface vlan
add comment=vrf-home interface=br_main name=vlan255 vlan-id=255
add comment=vrf-datacenter interface=br_main name=vlan1700 vlan-id=1700
add comment=vrf-datacenter interface=br_main name=vlan1710 vlan-id=1710
add comment=vrf-edge interface=br_main name=vlan2540 vlan-id=2540
/interface bonding
add lacp-rate=1sec min-links=1 mode=802.3ad mtu=9198 name=bond_3850-1_po10 slaves=ether5,ether6 transmit-hash-policy=layer-3-and-4 up-delay=250ms
add lacp-rate=1sec min-links=1 mode=802.3ad mtu=9198 name=bond_3850-2_po10 slaves=ether7,ether8 transmit-hash-policy=layer-3-and-4 up-delay=250ms
/interface vrrp
add interface=vlan1710 name=vrrp1710 priority=250 remote-address=172.17.10.3 sync-connection-tracking=yes vrid=2
/interface list
add comment=vrf-edge name=WAN
add comment=vrf-edge name=vrf-edge
add comment=vrf-datacenter name=vrf-datacenter
add comment=vrf-home name=vrf-home
add comment=vrf-edge name=LAN
add exclude=vrf-home,vrf-datacenter,WAN name=vrf-edge_inside
/interface bridge msti
add bridge=br_main comment=vrf-edge identifier=2 priority=0xF000 vlan-mapping=2540-2549
add bridge=br_main comment=vrf-home identifier=1 priority=0xF000 vlan-mapping=128-255
add bridge=br_main comment=vrf-datacenter identifier=3 priority=0xF000 vlan-mapping=1700-1799
/interface bridge port
add bridge=br_main frame-types=admit-only-vlan-tagged interface=bond_3850-1_po10
add bridge=br_main frame-types=admit-only-vlan-tagged interface=bond_3850-2_po10
add bridge=br_main fast-leave=yes frame-types=admit-only-untagged-and-priority-tagged interface=ether2 point-to-point=yes pvid=1710 trusted=yes
/interface bridge vlan
add bridge=br_main tagged=bond_3850-1_po10,bond_3850-2_po10,br_main vlan-ids=2540
add bridge=br_main tagged=bond_3850-1_po10,bond_3850-2_po10,br_main vlan-ids=1700
add bridge=br_main tagged=br_main,bond_3850-1_po10,bond_3850-2_po10 vlan-ids=255
add bridge=br_main tagged=br_main untagged=ether2 vlan-ids=1710
/interface list member
add interface=ether3 list=WAN
add interface=ipip_vrf-datacenter_vrf-edge list=vrf-datacenter
add interface=vlan1700 list=vrf-datacenter
add interface=vlan1710 list=vrf-datacenter
add interface=lo_vrf-datacenter list=vrf-datacenter
add interface=vrrp1710 list=vrf-datacenter
add interface=ether3 list=vrf-edge
add interface=ether4 list=vrf-edge
add interface=ipip_vrf-edge_vrf-datacenter list=vrf-edge
add interface=ipip_vrf-edge_vrf-home list=vrf-edge
add interface=lo_vrf-edge list=vrf-edge
add interface=vlan2540 list=vrf-edge
add interface=ipip_vrf-home_vrf-edge list=vrf-home
add interface=vlan255 list=vrf-home
add interface=lo_vrf-home list=vrf-home
add interface=br_main list=LAN


/ip vrf
add interfaces=lo_vrf-datacenter,ipip_vrf-datacenter_vrf-edge,vlan1700,vlan1710,vrrp1710 name=vrf-datacenter
add interfaces=lo_vrf-home,ipip_vrf-home_vrf-edge,vlan255 name=vrf-home
add interfaces=ether3,ether4,lo_vrf-edge,ipip_vrf-edge_vrf-home,ipip_vrf-edge_vrf-datacenter,vlan2540 name=vrf-edge
/ip address
add address=198.51.100.1 comment=main interface=lo_tun_vrf-home network=198.51.100.1
add address=198.51.100.2 comment=main interface=lo_tun_vrf-edge network=198.51.100.2
add address=198.51.100.3 comment=main interface=lo_tun_vrf-datacenter network=198.51.100.3
add address=10.254.7.1/30 comment=vrf-edge interface=ipip_vrf-edge_vrf-home network=10.254.7.0
add address=10.254.7.5/30 comment=vrf-edge interface=ipip_vrf-edge_vrf-datacenter network=10.254.7.4
add address=10.254.7.2/30 comment=vrf-home interface=ipip_vrf-home_vrf-edge network=10.254.7.0
add address=10.254.7.6/30 comment=vrf-datacenter interface=ipip_vrf-datacenter_vrf-edge network=10.254.7.4
add address=172.17.10.2/24 comment=vrf-datacenter interface=vlan1710 network=172.17.10.0
add address=10.254.0.1/24 comment=vrf-edge interface=vlan2540 network=10.254.0.0
add address=172.17.0.10/24 comment=vrf-datacenter interface=vlan1700 network=172.17.0.0
add address=192.168.255.1/26 comment=vrf-home interface=vlan255 network=192.168.255.0
add address=10.254.9.255 comment=vrf-edge interface=lo_vrf-edge network=10.254.9.255
add address=172.17.10.1/24 interface=vrrp1710 network=172.17.10.0
add address=172.17.99.255 comment=vrf-datacenter interface=lo_vrf-datacenter network=172.17.99.255
/ip dhcp-client
add interface=ether3 use-peer-dns=no use-peer-ntp=no
/ip dns
set servers=4.2.2.1,1.1.1.1
/ip firewall address-list
add address=192.168.128.0/17 list=vrf-home_192.168.128.0-192.168.255.255
add address=172.17.0.0/16 list=vrf-datacenter_172.17.0.0-172.17.255.255
add address=10.254.0.0/21 list=vrf-edge_10.254.0.0-10.254.7.255
add address=198.51.100.0/24 list=main_198.51.100.0-198.51.100.255
add address=0.0.0.0/8 list=not_internet_routable-RFC6890
add address=172.16.0.0/12 list=not_internet_routable-RFC6890
add address=192.168.0.0/16 list=not_internet_routable-RFC6890
add address=10.0.0.0/8 list=not_internet_routable-RFC6890
add address=169.254.0.0/16 list=not_internet_routable-RFC6890
add address=127.0.0.0/8 list=not_internet_routable-RFC6890
add address=224.0.0.0/4 list=not_internet_routable-RFC6890
add address=198.18.0.0/15 list=not_internet_routable-RFC6890
add address=192.0.0.0/24 list=not_internet_routable-RFC6890
add address=192.0.2.0/24 list=not_internet_routable-RFC6890
add address=198.51.100.0/24 disabled=yes list=not_internet_routable-RFC6890
add address=203.0.113.0/24 list=not_internet_routable-RFC6890
add address=100.64.0.0/10 list=not_internet_routable-RFC6890
add address=240.0.0.0/4 list=not_internet_routable-RFC6890
add address=192.88.99.0/24 list=not_internet_routable-RFC6890
add address=172.17.10.10 list=firewall_interfaces
add address=192.168.255.1 list=firewall_interfaces
add address=10.254.9.255 list=firewall_interfaces
add address=10.254.0.1 list=firewall_interfaces
add address=127.0.0.0/8 comment="defconf: RFC6890" list=bad_ipv4
add address=192.0.0.0/24 comment="defconf: RFC6890" list=bad_ipv4
add address=192.0.2.0/24 comment="defconf: RFC6890 documentation" list=bad_ipv4
add address=198.51.100.0/24 comment="defconf: RFC6890 documentation" disabled=yes list=bad_ipv4
add address=203.0.113.0/24 comment="defconf: RFC6890 documentation" list=bad_ipv4
add address=240.0.0.0/4 comment="defconf: RFC6890 reserved" list=bad_ipv4
add address=0.0.0.0/8 comment="defconf: RFC6890" list=not_global_ipv4
add address=10.0.0.0/8 comment="defconf: RFC6890" list=not_global_ipv4
add address=100.64.0.0/10 comment="defconf: RFC6890" list=not_global_ipv4
add address=169.254.0.0/16 comment="defconf: RFC6890" list=not_global_ipv4
add address=172.16.0.0/12 comment="defconf: RFC6890" list=not_global_ipv4
add address=192.0.0.0/29 comment="defconf: RFC6890" list=not_global_ipv4
add address=192.168.0.0/16 comment="defconf: RFC6890" list=not_global_ipv4
add address=198.18.0.0/15 comment="defconf: RFC6890 benchmark" list=not_global_ipv4
add address=255.255.255.255 comment="defconf: RFC6890" list=not_global_ipv4
add address=224.0.0.0/4 comment="defconf: multicast" list=bad_src_ipv4
add address=255.255.255.255 comment="defconf: RFC6890" list=bad_src_ipv4
add address=0.0.0.0/8 comment="defconf: RFC6890" list=bad_dst_ipv4
add address=224.0.0.0/4 comment="defconf: RFC6890" list=bad_dst_ipv4
add address=10.254.7.6 list=vrf-datacenter_172.17.0.0-172.17.255.255
/ip firewall nat
add action=masquerade chain=srcnat log=yes out-interface=ether3
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=10.254.7.5%ipip_vrf-datacenter_vrf-edge@vrf-datacenter pref-src="" routing-table=vrf-datacenter scope=30 \
    suppress-hw-offload=no target-scope=10 vrf-interface=ipip_vrf-datacenter_vrf-edge
add disabled=no dst-address=0.0.0.0/0 gateway=10.254.7.1%ipip_vrf-home_vrf-edge@vrf-home routing-table=vrf-home suppress-hw-offload=no
add disabled=yes distance=1 dst-address=0.0.0.0/0 gateway=10.254.7.2%ipip_vrf-edge_vrf-home@vrf-edge pref-src="" routing-table=vrf-edge scope=30 suppress-hw-offload=no target-scope=\
    10
add disabled=no dst-address=172.17.0.0/16 gateway=10.254.7.6%ipip_vrf-edge_vrf-datacenter@vrf-edge routing-table=vrf-edge suppress-hw-offload=no

jg159357 · October 22, 2023, 2:07pm

Test results from above config with no modifications. All sniffs are from interface gi1/0/13 on the catalyst 3850 that is plugged into ether3.

Ping from Catalyst vrf Temp1 to ether3

#ping vrf Temp1 ip 203.0.113.5 source 1.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 203.0.113.5, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1
!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 18/19/20 ms

*Oct 22 14:01:04.343: IP: tableid=5, s=1.1.1.1 (local), d=203.0.113.5 (GigabitEthernet1/0/13) nexthop=203.0.113.5, routed via FIB
*Oct 22 14:01:04.344: IP: s=1.1.1.1 (local), d=203.0.113.5 (GigabitEthernet1/0/13), len 100, sending
*Oct 22 14:01:04.344: IP: s=1.1.1.1 (local), d=203.0.113.5 (GigabitEthernet1/0/13), len 100, output feature, feature skipped, QoS Classification(40), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSEOutput intf Gi1/0/13 matches debug filer

*Oct 22 14:01:04.345: IOSXE-INJECT: set pak datagramstart(from 0xFF94BBC1B0 to 0xFF94BBC1B0), datagramsize(from 100 to 100);
*Oct 22 14:01:04.345: IOSXE-INJECT: add L3 inject hdr
*Oct 22 14:01:04.346: IOSXE-INJECT: inject hdr len 32, feature hdr len 0, l2-enc len 0, link type ip, pak len 100, total len 132, inject type normal, seqnum 167092
*Oct 22 14:01:04.346: IOSXE-INJECT: inject_sb inject_flag=0x0, subtype=0, type_flags=0x0,
*Oct 22 14:01:04.346: send out the l3(0x5) packet to cpp:
FF94BBC190: 01010000 00000064 00840000 20010000 …d… …
FF94BBC1A0: 00000005 00000036 01000001 00100200 …6…
FF94BBC1B0: 45000064 00DA0000 FF017CB7 01010101 E..d.Z…|7…
FF94BBC1C0: CB007105 08007D6E 002E0000 00000000 K.q…}n…
FF94BBC1D0: 175FE94E ABCDABCD ABCDABCD ABCDABCD ._iN+M+M+M+M+M+M
FF94BBC1E0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD +M+M+M+M+M+M+M+M
FF94BBC1F0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD +M+M+M+M+M+M+M+M
FF94BBC200: ABCDABCD ABCDABCD ABCDABCD ABCDABCD +M+M+M+M+M+M+M+M
FF94BBC210:
*Oct 22 14:01:04.362: IP protocol ICMP …
*Oct 22 14:01:04.362: IOSXE-INJECT: pak inject type normal, ready for tx, seqnum 167092
*Oct 22 14:01:04.362: IOSXE-INJECT: inject_hdr pal_if_handle=0x36, flags=0x10, cause=2, sub_cause=0
*Oct 22 14:01:04.365: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, input feature, QoS Classification(37), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:01:04.365: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, input feature, QoS Marking(62), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:01:04.366: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, input feature, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:01:04.367: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, rcvd 2
*Oct 22 14:01:04.367: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, stop process pak for forus packet

>
\
<br>
> Ping from vrf datacenter to 1.1.1.1
>
> ```text
#ping vrf datacenter ip 1.1.1.1 source 172.17.0.11 repeat 1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 172.17.0.11
.
Success rate is 0 percent (0/1)


*Oct 22 14:04:15.742: IP: s=172.17.0.11 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, input feature, QoS Classification(37), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:04:15.743: IP: s=172.17.0.11 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, input feature, QoS Marking(62), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:04:15.743: IP: s=172.17.0.11 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, input feature, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:04:15.744: IP: s=172.17.0.11 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, rcvd 2
*Oct 22 14:04:15.745: IP: s=172.17.0.11 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, stop process pak for forus packet
*Oct 22 14:04:15.746: IP: s=1.1.1.1 (local), d=172.17.0.11 (nil), len 100, unroutable
*Oct 22 14:04:16.707: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=255.255.255.255 (nil), len 184, input feature, QoS Classification(37), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:04:16.708: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=255.255.255.255 (nil), len 184, input feature, QoS Marking(62), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:04:16.709: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=255.255.255.255 (nil), len 184, input feature, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:04:16.710: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=255.255.255.255 (nil), len 184, rcvd 2
*Oct 22 14:04:16.710: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=255.255.255.255 (nil), len 184, stop process pak for forus packet
*Oct 22 14:04:16.711: IP: s=172.17.0.10 (Vlan1700), d=255.255.255.255 (nil), len 186, stop process pak for forus packet

jg159357 · October 22, 2023, 2:23pm

I then remove the ether3 match rule and repeat.

/ip firewall nat
add action=masquerade chain=srcnat log=yes log-prefix=Masq

Ping from vrf datacenter to 1.1.1.1

#ping vrf datacenter ip 1.1.1.1 source 172.17.0.11 repeat 1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 172.17.0.11
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 28/28/28 ms

*Oct 22 14:10:16.176: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, input feature, QoS Classification(37), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:10:16.177: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, input feature, QoS Marking(62), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:10:16.177: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, input feature, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:10:16.179: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, rcvd 2
*Oct 22 14:10:16.179: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=1.1.1.1 (nil), len 100, stop process pak for forus packet
*Oct 22 14:10:16.180: IP: tableid=5, s=1.1.1.1 (local), d=203.0.113.5 (GigabitEthernet1/0/13) nexthop=203.0.113.5, routed via FIB
*Oct 22 14:10:16.180: IP: s=1.1.1.1 (local), d=203.0.113.5 (GigabitEthernet1/0/13), len 100, sending
*Oct 22 14:10:16.181: IP: s=1.1.1.1 (local), d=203.0.113.5 (GigabitEthernet1/0/13), len 100, output feature, feature skipped, QoS Classification(40), rtype 1, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSEOutput intf Gi1/0/13 matches debug filer

*Oct 22 14:10:16.182: IOSXE-INJECT: set pak datagramstart(from 0xFF94E20372 to 0xFF94E20380), datagramsize(from 114 to 100);
*Oct 22 14:10:16.182: IOSXE-INJECT: add L3 inject hdr
*Oct 22 14:10:16.182: IOSXE-INJECT: inject hdr len 32, feature hdr len 0, l2-enc len 0, link type ip, pak len 100, total len 132, inject type normal, seqnum 167684
*Oct 22 14:10:16.182: IOSXE-INJECT: inject_sb inject_flag=0x0, subtype=0, type_flags=0x0,
*Oct 22 14:10:16.183: send out the l3(0x5) packet to cpp:
FF94E20360: 01010000 00000064 00840000 20010000 …d… …
FF94E20370: 00000005 00000036 01000001 00100200 …6…
FF94E20380: 45000064 00E30000 FF017CAE 01010101 E..d.c…|…
FF94E20390: CB007105 0000C36D 568F0000 00000000 K.q…CmV…
FF94E203A0: 176854E5 ABCDABCD ABCDABCD ABCDABCD .hTe+M+M+M+M+M+M
FF94E203B0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD +M+M+M+M+M+M+M+M
FF94E203C0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD +M+M+M+M+M+M+M+M
FF94E203D0: ABCDABCD ABCDABCD ABCDABCD ABCDABCD +M+M+M+M+M+M+M+M
FF94E203E0:
*Oct 22 14:10:16.198: IP protocol ICMP …
*Oct 22 14:10:16.198: IOSXE-INJECT: pak inject type normal, ready for tx, seqnum 167684
*Oct 22 14:10:16.199: IOSXE-INJECT: inject_hdr pal_if_handle=0x36, flags=0x10, cause=2, sub_cause=0
*Oct 22 14:10:16.201: IP: s=1.1.1.1 (Vlan1700), d=172.17.0.11 (nil), len 100, stop process pak for forus packet
*Oct 22 14:10:16.706: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=255.255.255.255 (nil), len 184, input feature, QoS Classification(37), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:10:16.707: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=255.255.255.255 (nil), len 184, input feature, QoS Marking(62), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:10:16.708: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=255.255.255.255 (nil), len 184, input feature, MCI Check(109), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
*Oct 22 14:10:16.709: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=255.255.255.255 (nil), len 184, rcvd 2
*Oct 22 14:10:16.709: IP: s=203.0.113.5 (GigabitEthernet1/0/13), d=255.255.255.255 (nil), len 184, stop process pak for forus packet
*Oct 22 14:10:16.710: IP: s=172.17.0.10 (Vlan1700), d=255.255.255.255 (nil), len 186, stop process pak for forus packet
*Oct 22 14:10:16.711: IP: s=192.168.255.1 (Vlan255), d=255.255.255.255 (nil), len 185, stop process pak for forus packet

>
\
<br>
> Ping from vrf datacenter to vlan 2540 in vrf-edge
>
> ```text
ping vrf datacenter ip 10.254.0.1 source 172.17.0.11 repeat 1
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 10.254.0.1, timeout is 2 seconds:
Packet sent with a source address of 172.17.0.11
.
Success rate is 0 percent (0/1)

Log entries for masquerade match

Masq srcnat: in:vlan1700 out:ipip_vrf-datacenter_vrf-edge, connection-state:new src-mac ec:1d:8b:bf:67:54, proto ICMP (type 8, code 0), 172.17.0.11->1.1.1.1, len 100
Masq srcnat: in:ipip_vrf-edge_vrf-datacenter out:ether3, connection-state:new,dnat proto ICMP (type 8, code 0), 10.254.7.6->1.1.1.1, len 100
Masq srcnat: in:vlan1700 out:ipip_vrf-datacenter_vrf-edge, connection-state:new src-mac ec:1d:8b:bf:67:54, proto ICMP (type 8, code 0), 172.17.0.11->10.254.0.1, len 100

jg159357 · October 26, 2023, 11:15am

Does anyone have any suggestions on what else to try or suggestions on how to find out why the Masquerade rule isn’t firing?

jg159357 · November 12, 2023, 3:37am

I finally got some more time to test this and I still can’t nail down the actual cause. I upgraded to 7.12 just before testing as well.
If I only use destination IP address the srcnat rule fires and I get this log entry. (the destination list is ‘exclude all not_global_ipv4_routable’ addresses)

srcnat: in:ipip_vrf-edge_vrf-datacenter out:ether3, packet-mark:vrf-datacenter-internet connection-state:new,dnat proto ICMP (type 8, code 0), 10.254.7.6->1.1.1.1, len 100

This shows that it knows it’s going out Interface ether3, has a packet mark vrf-datacenter-internet and a protocol of ICMP

However, if I enable those options (one at a time) in the srcnat chain rule, then the packet doesn’t match anymore for the out interface and the packet mark. It does still work for protocol filter though.