Q-in-Q 802.1ad MTU Issue

I am wracking my head against the wall on this issue for two days now.

I cannot get 1500 byte MTU without fragmentation. 1496 is the cap. 4-byte difference has got to be q-in-q. I think? the resolution right now is to have customers set WAN port MTU to 1496. But, I’d like to nix this issue so that isn’t required.

We take 10G handoff from fiber provider at a CCR1072 on interface sfp-sfpplus1. They give us VLAN 202 for transport. I use VLAN 20 and VLAN 30 with service-tag=yes stacked in VLAN 202 from fiber provider. On customer end we have a hEX behind a Calix ONT connected on ETHER1. Calix is GPON provisioned from fiber provider and we do not have access to them.

CCR1072 Configuration

/interface vlan add comment="BF Business CoS" interface=sfp-sfpplus1 mtu=1500 name=v_BF_BUS vlan-id=202
/interface vlan add interface=v_BF_BUS name=v_BF_BUS_GPON-CUST use-service-tag=yes vlan-id=30
/interface vlan add interface=v_BF_BUS name=v_BF_BUS_GPON-MGMT use-service-tag=yes vlan-id=20

Customer hEX Configuration

/interface vlan add interface=ether1 name=v_BF-CUST use-service-tag=yes vlan-id=30
/interface vlan add interface=ether1 name=v_BF-MGMT use-service-tag=yes vlan-id=20

Using default MTU=1500 on physical interfaces and VLAN interfaces.

Packet capture of ping from CCR1072 to hEX

Frame 1: 1522 bytes on wire (12176 bits), 1522 bytes captured (12176 bits)
Ethernet II, Src: Routerbo_fe:37:ca (6c:3b:6b:fe:37:ca), Dst: Routerbo_71:23:12 (6c:3b:6b:71:23:12)
802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 202
    000. .... .... .... = Priority: Best Effort (default) (0)
    ...0 .... .... .... = CFI: Canonical (0)
    .... 0000 1100 1010 = ID: 202
    Type: 802.1ad Provider Bridge (Q-in-Q) (0x88a8)
IEEE 802.1ad, ID: 20
    000. .... .... .... = Priority: 0
    ...0 .... .... .... = DEI: 0
    .... 0000 0001 0100 = ID: 20
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 1.1.1.1, Dst: 2.2.2.2
Internet Control Message Protocol



Frame 2: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits)
Ethernet II, Src: Routerbo_71:23:12 (6c:3b:6b:71:23:12), Dst: Routerbo_fe:37:ca (6c:3b:6b:fe:37:ca)
802.1Q Virtual LAN, PRI: 2, CFI: 0, ID: 202
    010. .... .... .... = Priority: Spare (2)
    ...0 .... .... .... = CFI: Canonical (0)
    .... 0000 1100 1010 = ID: 202
    Type: 802.1ad Provider Bridge (Q-in-Q) (0x88a8)
IEEE 802.1ad, ID: 20
    000. .... .... .... = Priority: 0
    ...0 .... .... .... = DEI: 0
    .... 0000 0001 0100 = ID: 20
    Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 2.2.2.2, Dst: 1.1.1.1
Data (1472 bytes)



Frame 3: 64 bytes on wire (512 bits), 64 bytes captured (512 bits)
Ethernet II, Src: Routerbo_71:23:12 (6c:3b:6b:71:23:12), Dst: Routerbo_fe:37:ca (6c:3b:6b:fe:37:ca)
802.1Q Virtual LAN, PRI: 2, CFI: 0, ID: 202
    010. .... .... .... = Priority: Spare (2)
    ...0 .... .... .... = CFI: Canonical (0)
    .... 0000 1100 1010 = ID: 202
    Type: 802.1ad Provider Bridge (Q-in-Q) (0x88a8)
IEEE 802.1ad, ID: 20
    000. .... .... .... = Priority: 0
    ...0 .... .... .... = DEI: 0
    .... 0000 0001 0100 = ID: 20
    Type: IPv4 (0x0800)
    Padding: 00000000000000000000
    Trailer: 00000000
Internet Protocol Version 4, Src: 2.2.2.2, Dst: 1.1.1.1
Internet Control Message Protocol

If I can provide anymore information just let me know. Thanks in advance!

Well, Calix just informed us that their 801g ONT is fixed at 1496. So, that explains why i’m losing 4-bytes and fragmenting.