Community discussions

MikroTik App
 
lctn
Member Candidate
Member Candidate
Topic Author
Posts: 176
Joined: Tue Apr 04, 2006 3:51 pm

Bandwidth pinched through VxLAN tunnel

Thu Apr 14, 2022 10:13 pm

ROS: 7.1.3
Devices: CCR1036-8G-2S

I had a production VxLAN tunnel running with 2 CCR1036-8G-2S Cloud Core Routers. The tunnel was for a k12 school district and I would often get complaints about web traffic being painfully slow. If left alone the issue would resolve itself but repeat several times throughout the day. I finally had to pull the Mikrotiks and put the district back on our Aruba VxLAN tunnel, which resolved the issue for them. My 1st choice is to use the Mikrotiks but I need to resolve the issue before implementing again.

In duplicating the setup, with 2 cloud routers directly connected, I discovered the following:
1. With a VxLAN configuration, a laptop pulled a 3Gb ISO down at 19MBs
2. With the same configuration a laptop and PC pulled the same file down at 11MBs, each
3. Removing VXLAN and configuring the devices with only L2 VLANs, the laptop pulled the file down at 34MBs.
4. The laptop and PC pulled the file down at the same time at 34MBs, each.

Obviously, there is a major degrade in performance when using VxLAN, especially when more than 1 client is downloading a file.

I have contacted support concerning this but have not received instruction on how to remedy the problem yet, outside of a recommendation to change the mss to 1300, but that rule did not have any hits, regardless of the interface used. Ideas??

Attaching all configs:

VxLAN:
Master:
# apr/13/2022 12:29:37 by RouterOS 7.1.3
# software id = IHD4-H4MR
#
# model = CCR1036-8G-2S+

/interface bridge
add ingress-filtering=no name=BRIDGE-VxLAN-VNI-102 vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] l2mtu=8000
set [ find default-name=ether2 ] l2mtu=8000
set [ find default-name=ether4 ] l2mtu=8000
/interface vxlan
add group=224.0.0.1 interface=ether1 mtu=1400 name=vxlan-vni-102 port=8473 \
    vni=102
/interface vlan
add interface=ether1 name=vlan703 vlan-id=703
/interface lte apn
set [ find default=yes ] ip-type=ipv4
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool0 ranges=172.169.0.10-172.169.0.30
/ip dhcp-server
add address-pool=dhcp_pool0 name=dhcp1
/port
set 0 name=serial0
set 1 name=serial1
/interface bridge port
add bridge=BRIDGE-VxLAN-VNI-102 interface=ether2
add bridge=BRIDGE-VxLAN-VNI-102 interface=vxlan-vni-102
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface bridge vlan
add bridge=BRIDGE-VxLAN-VNI-102 tagged=vxlan-vni-102,ether2 vlan-ids=703
add bridge=BRIDGE-VxLAN-VNI-102 tagged=vxlan-vni-102,ether2 vlan-ids=704
/interface vxlan vteps
add interface=vxlan-vni-102 port=8572 remote-ip=172.169.0.2
/ip address
add address=172.169.0.1/24 interface=ether1 network=172.169.0.0
/ip dhcp-client
add interface=ether8
/ip dhcp-server network
add address=172.169.0.0/24 gateway=172.169.0.1
/ip firewall mangle
add action=change-mss chain=forward log=yes new-mss=1300 out-interface=\
    BRIDGE-VxLAN-VNI-102 passthrough=yes protocol=tcp tcp-flags=syn tcp-mss=\
    1301-65535
/system clock
set time-zone-name=America/Chicago
/system identity
set name=VxLAN-Master
Client:
# apr/14/2022 10:20:18 by RouterOS 7.1.3
# software id = MPIL-B0WN
#
# model = CCR1036-8G-2S+

/interface bridge
add ingress-filtering=no name=BRIDGE-VxLAN-VNI-102 vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] l2mtu=8000
set [ find default-name=ether2 ] l2mtu=8000
set [ find default-name=ether4 ] l2mtu=8000
/interface vxlan
add group=224.0.0.1 interface=ether1 mtu=1400 name=vxlan-vni-102 port=8473 \
    vni=102
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool0 ranges=172.168.0.10-172.168.0.30
/ip dhcp-server
add address-pool=dhcp_pool0 name=dhcp1
/port
set 0 name=serial0
set 1 name=serial1
/interface bridge port
add bridge=BRIDGE-VxLAN-VNI-102 interface=vxlan-vni-102
add bridge=BRIDGE-VxLAN-VNI-102 interface=ether2
add bridge=BRIDGE-VxLAN-VNI-102 interface=ether4 pvid=703
/interface bridge vlan
add bridge=BRIDGE-VxLAN-VNI-102 tagged=vxlan-vni-102,ether2 vlan-ids=703
/ip address
add address=172.169.0.2/24 interface=ether1 network=172.169.0.0
/ip dhcp-client
add interface=ether8
/ip firewall mangle
add action=change-mss chain=forward new-mss=1300 out-interface=\
    BRIDGE-VxLAN-VNI-102 protocol=tcp tcp-flags=syn tcp-mss=1301-65535
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=172.168.0.1 routing-table=main \
    suppress-hw-offload=no
add disabled=no dst-address=0.0.0.0/0 gateway=172.169.0.1 routing-table=main \
    suppress-hw-offload=no
/system clock
set time-zone-name=America/Chicago
/system identity
set name=LCTN-Rm424
/tool sniffer
set filter-interface=ether1
L2 VLAN Only:
Master:
# jan/02/1970 00:54:26 by RouterOS 7.1.3
# software id = IHD4-H4MR
#
# model = CCR1036-8G-2S+

/interface bridge
add ingress-filtering=no name=B703 vlan-filtering=yes
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/port
set 0 name=serial0
set 1 name=serial1
/interface bridge port
add bridge=B703 interface=ether1
add bridge=B703 interface=ether2
add bridge=B703 interface=ether3 pvid=703
add bridge=B703 interface=ether4 pvid=703
/interface bridge vlan
add bridge=B703 tagged=ether1,ether2 vlan-ids=703
add bridge=B703 tagged=ether1,ether2 vlan-ids=704
/ip address
add address=172.169.0.1/24 interface=ether1 network=172.169.0.0
add address=10.7.3.99/24 interface=ether2 network=10.7.3.0
Client:
# apr/14/2022 13:40:15 by RouterOS 7.1.3
# software id = MPIL-B0WN
#
# model = CCR1036-8G-2S+

/interface bridge
add ingress-filtering=no name=B703 vlan-filtering=yes
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/port
set 0 name=serial0
set 1 name=serial1
/interface bridge port
add bridge=B703 interface=ether1
add bridge=B703 interface=ether3 pvid=703
add bridge=B703 interface=ether4 pvid=703
/interface bridge vlan
add bridge=B703 tagged=ether1 untagged=ether3 vlan-ids=703
/ip address
add address=172.169.0.2/24 interface=ether1 network=172.169.0.0
add address=10.7.3.100/24 interface=B703 network=10.7.3.0
/ip dhcp-client
add interface=ether8
/system clock
set time-zone-name=America/Chicago
Last edited by lctn on Tue Apr 26, 2022 7:46 pm, edited 3 times in total.
 
User avatar
smyers119
Member Candidate
Member Candidate
Posts: 232
Joined: Sat Feb 27, 2021 8:16 pm
Location: USA

Re: Bandwidth pinched through VxLAN tunnel

Fri Apr 15, 2022 7:59 am

Until you hear back about the speed problem are you interested in alternative solutions to your problem? There is plenty of other ways to extend your layer 2 network that may not take as much of a speed deficit as vxlan, assuming there's nothing you can do about that.
 
tangent
Forum Guru
Forum Guru
Posts: 1329
Joined: Thu Jul 01, 2021 3:15 pm
Contact:

Re: Bandwidth pinched through VxLAN tunnel

Fri Apr 15, 2022 12:46 pm

The tunnel was for a k12 school district

Why then are you using an interior slice of Oath/Yahoo/AOL’s IP space? Unless your K12 school is, unaccountably, owned by the parent company of Yahoo, this choice is likely to break access to Yahoo properties.

4. The laptop and PC pulled the file down at the same time at 34MBs, each.

Please use Mbit/sec. I’ll never understand why browsers use bytes when the entire rest of the networking world uses bits.

What’s the size of your upstream pipe? You show at least 544 Mbit/sec, but how large is that compared to the claimed maximum?

a recommendation to change the mss to 1300, but that rule did not have any hits

Since that mangle rule you have…

/ip firewall mangle
add action=change-mss chain=forward log=yes new-mss=1300 out-interface=\
    BRIDGE-VxLAN-VNI-102 passthrough=yes protocol=tcp tcp-flags=syn tcp-mss=\
    1301-65535

…is up at the top, there should be no way to skip it unless you've misspecified it.

My guess is that your out-interface rule is wrong. If we temporarily set aside my inference above that your LAN IP space is badly-chosen, I think you want to replace it with something like dst-address=!172.168.0.0/15, since you want this rule to affect only traffic forwarded to the Internet, not in-LAN traffic. It doesn't matter which interface it's going out, it matters which IP it's going to.

If you do decide you need to specify this by interface name, I believe they're case-sensitive in RouterOS filters. You've got a mix of all-lowercase names and mixed-case names in this config.

Drop the log=yes bit. That's needlessly expensive here: logging is expensive in general, and you've got it on a rule that should be hit by every TCP SYN packet, of which there are often dozens per HTTP hit. For your purposes, simply watching the packet counter in WinBox should suffice to tell you when it's working.

/interface vxlan add group=224.0.0.1

The 224.0.0.0/24 range is for well-known services, registered with IANA. The ".1" address in that range is a particularly bad choice, meaning "all multicast hosts".

You should be using something in 239.0.0.0/8 that you've pre-cleared as unused locally.
 
lctn
Member Candidate
Member Candidate
Topic Author
Posts: 176
Joined: Tue Apr 04, 2006 3:51 pm

Re: Bandwidth pinched through VxLAN tunnel

Sat Apr 16, 2022 4:36 am

Thank you for the replies..

Just a reminder: The Aruba VxLAN tunnel does not have any of the same limitations and runs across the same WAN for 18 other school districts.

My design is as follows (10 Gb WAN)

School district edge switch > Mikrotik Cloud Router (VxLAN) > Cisco 3400 WAN switch > Cisco 3600 WAN Switch > Mikrotik Cloud Router > Fortigate Vdom> Internet


@smyers119

I would be very interested in learning of other options that have better throughput.


@sweetcandysp

I left things as is so all things could be considered when responding.


@tangent

My IPs came from something similar I found on a tutorial. None of the IPs shown ever get beyond our 10Gb WAN

There are no other rules on the devices. The mss is all I have. If it should be configured differently, I would love to know which of the listed interfaces should be used. In my troubleshooting, I believe I tried every interface that was not a slave interface.


I am the WAN admin, the UDP addresses are available. I actually had started with 239.x.x.x but had 2 more testing tunnels on the original equipment and wanted to be sure I was not causing an issue between them. In my current test, I have a single tunnel on two isolated Mikrotik routers. The results are the same, regardless.
 
tangent
Forum Guru
Forum Guru
Posts: 1329
Joined: Thu Jul 01, 2021 3:15 pm
Contact:

Re: Bandwidth pinched through VxLAN tunnel

Sat Apr 16, 2022 6:56 am

The Aruba VxLAN tunnel does not have any of the same limitations

Part of the problem may be that you're trying to treat MikroTik as Aruba.

There's a lot of foundational stuff you can port from one vendor to another, but take that too far and you'll run into trouble no matter which direction you're going. At some point, you have to take each vendor's offering on its own merits.

My IPs came from something similar

Yeah, it looks like you saw something done with a 172.16.x.y through 172.31.x.y scheme and thought you could make up anything in the 172 space. You can't. Same as how you're making up multicast IPs without knowing what you're doing and running into trouble with that, too.

None of the IPs shown ever get beyond our 10Gb WAN

Not the point: if anyone on your LAN tries to go to any Oath Holdings property that happens to resolve via DNS to an IP in your bogarted range, your routers will say, "Oh hey, that's a local IP! Let me direct you!"

If you're lucky, no local machine will be at the referenced IP, and you'll just get an apparent DNS failure.

If you're unlucky, it'll resolve to something with a server running on it and your user will start griping about how they went to frobozz.yahoo.com and got the district SAN's management web app instead.

If you're really unlucky, the same thing will happen, but it'll affect only one critical element on the page out of 100 that work just fine, with the result that you won't figure out why it's failing until you spend two hours with the browser developer tools, a copy of the Necronomicon, and a ouija board to summon the ghost of Postel to serve as your spririt guide into the bowels of someone else's overcomplicated enterprisey web app.

If it should be configured differently, I would love to know which of the listed interfaces should be used.

I told you: you don't want an interface name, you want a direction of some kind.

Look at your bridge configuration: you've got the L2 MTU set to 8000. Why would you do that and then clamp MSS to 1300? A wise network admin wouldn't, because that'd be silly. Just because the TCP SYN transits the LAN doesn't mean it must be clamped. Where it's going affects what MSS you need.

Instead of clamping MSS to the worst case and making everything suffer, a better config might use three different MSS values:

1. In-LAN destinations, where you leave the MTU/MSS unclamped, so your jumbo packet config (MTU=8000) can take effect.

2. Cross-district destinations where you have to take the VXLAN overhead into account. Something like 1450 might suffice here.

3. Internet destinations where you also have to take into account another bottleneck down the line, where a smaller MSS still might be needed.

It's all conditional. You have to know your network.

I am the WAN admin, the UDP addresses are available.

If you're trying to justify 224.0.0.1, it doesn't matter that it's "unused" on your LAN. It's wrong, period. It's a type of broadcast address, so you can expect trouble if you try to use it for VXWAN broadcast signalling. That's broadcast over broadcast, see?

I actually had started with 239.x.x.x but had 2 more testing tunnels on the original equipment and wanted to be sure I was not causing an issue between them.

In multicast space, there's nothing like a subnet.† We use CIDR notation to talk about blocks of multicast IPs, but 239.0.1.2 is no closer to 239.0.1.3 than to 239.253.254.255. You can have two services right next to each other in this space and they won't create a conflict.

—◆—◆—◆—

† Okay, okay, there's the 1:32 Ethernet to IP chunking, but the stack takes care of that. If you want to be paranoid, add 32 to each IP to be extra-sure they don't conflict, even down at the L2 level.
 
lctn
Member Candidate
Member Candidate
Topic Author
Posts: 176
Joined: Tue Apr 04, 2006 3:51 pm

Re: Bandwidth pinched through VxLAN tunnel

Sat Apr 16, 2022 5:32 pm

Thanks for the feedback. I will make some changes next week and report my findings.
 
lctn
Member Candidate
Member Candidate
Topic Author
Posts: 176
Joined: Tue Apr 04, 2006 3:51 pm

Re: Bandwidth pinched through VxLAN tunnel

Tue Apr 26, 2022 7:18 pm

I followed the recommendations but really could not get VxLAN to perform any differently than it did with my posted configs. I finally moved to an EOIP solution which performs quite a bit better with the configs below. I will have 300+ users getting their Internet across 20 VLANs through this tunnel. Is there anything else I can modify to inure the best throughput possible?

Master
# apr/26/2022 10:56:02 by RouterOS 7.1.3
# software id = IHD4-H4MR
#
# model = CCR1036-8G-2S+

/interface bridge
add name=B703
add name=B704
add ingress-filtering=no name=trunk vlan-filtering=yes
/interface eoip
add loop-protect=off mac-address=02:97:91:21:41:0E mtu=1500 name=eoip-tunnel1 \
    remote-address=10.1.1.2 tunnel-id=7
/interface vlan
add interface=trunk name=vlan703 use-service-tag=yes vlan-id=703
add interface=ether2 name=vlan703-acc vlan-id=703
add interface=trunk name=vlan704 use-service-tag=yes vlan-id=704
add interface=ether2 name=vlan704-acc vlan-id=704
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/port
set 0 name=serial0
set 1 name=serial1
/interface bridge port
add bridge=trunk interface=eoip-tunnel1
add bridge=trunk interface=ether2
add bridge=B703 interface=vlan703
add bridge=B703 interface=ether3
add bridge=B703 interface=vlan703-acc
add bridge=B704 interface=vlan704
add bridge=B704 interface=vlan704-acc
add bridge=B704 interface=ether4
/interface bridge vlan
add bridge=trunk tagged=trunk,ether2 vlan-ids=703,704
/ip address
add address=10.1.1.1/27 interface=ether1 network=10.1.1.0
/ip dhcp-client
add interface=ether8
/system clock
set time-zone-name=America/Chicago
/system identity
set name=Master-EOIP

Client
# apr/26/2022 10:55:55 by RouterOS 7.1.3
# software id = MPIL-B0WN
#
# model = CCR1036-8G-2S+

/interface bridge
add name=B703 protocol-mode=none
add name=B704 protocol-mode=none
add ingress-filtering=no name=trunk protocol-mode=none vlan-filtering=yes
/interface eoip
add loop-protect=off mac-address=FE:35:A9:BE:03:2F mtu=1500 name=eoip-tunnel1 \
    remote-address=10.1.1.1 tunnel-id=7
/interface vlan
add interface=trunk name=vlan703 use-service-tag=yes vlan-id=703
add interface=ether2 name=vlan703-acc vlan-id=703
add interface=trunk name=vlan704 use-service-tag=yes vlan-id=704
add interface=ether2 name=vlan704-acc vlan-id=704
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/port
set 0 name=serial0
set 1 name=serial1
/snmp community
add addresses=::/0 name=lctnmrtg
/interface bridge port
add bridge=trunk interface=eoip-tunnel1
add bridge=trunk interface=ether2
add bridge=B703 interface=vlan703
add bridge=B703 interface=ether3
add bridge=B703 interface=vlan703-acc
add bridge=B704 interface=vlan704
add bridge=B704 interface=vlan704-acc
add bridge=B704 interface=ether4
/interface bridge vlan
add bridge=trunk tagged=trunk,ether2 vlan-ids=703,704
/ip address
add address=10.1.1.2/27 interface=ether1 network=10.1.1.0
/system identity
set name=Client-EOIP

Last edited by lctn on Tue Apr 26, 2022 7:34 pm, edited 1 time in total.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Bandwidth pinched through VxLAN tunnel

Tue Apr 26, 2022 7:26 pm

Remove serial numbers from export, everytime...
 
lctn
Member Candidate
Member Candidate
Topic Author
Posts: 176
Joined: Tue Apr 04, 2006 3:51 pm

Re: Bandwidth pinched through VxLAN tunnel

Tue Apr 26, 2022 7:35 pm

Thank you!
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Bandwidth pinched through VxLAN tunnel

Tue Apr 26, 2022 7:38 pm

Ehm... also from your other posts...
search.php?keywords=%22serial+number%22&author=lctn
 
lctn
Member Candidate
Member Candidate
Topic Author
Posts: 176
Joined: Tue Apr 04, 2006 3:51 pm

Re: Bandwidth pinched through VxLAN tunnel

Tue Apr 26, 2022 7:52 pm

Got it. Thanks
 
mada3k
Long time Member
Long time Member
Posts: 682
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: Bandwidth pinched through VxLAN tunnel

Tue Apr 26, 2022 9:52 pm

Smells like MTU issues. What the the maximum allowed MTU on the line between the CCR'es?
 
lctn
Member Candidate
Member Candidate
Topic Author
Posts: 176
Joined: Tue Apr 04, 2006 3:51 pm

Re: Bandwidth pinched through VxLAN tunnel

Tue Apr 26, 2022 10:31 pm

If you are referring to the VxLAN config, we finally settled on 1400 (tried 1450 too) on the tunnel interface and 1500 for all other interfaces. In my test environment, the two Mikrotiks were directly connected to each other, via ether1

Who is online

Users browsing this forum: Majestic-12 [Bot], nuwang13, Rhydu and 60 guests