Debug slow L2TP/IPsec

hpet · March 4, 2021, 10:31am

For some reason my L2TP/IPsec client/server connection runs very very slow… in kbs instead of Mbs. I tested internet performance on both ends and is 600/100 Mbps. This was working fine till today and nothing realy changed. Before I was getting around 80Mbps over tunnel. Today couple hundred kbs. I rebooted client, not yet server router.

How and where do I start searching for the cause?

Edit
L2TP/IPsec client:

Actual MTU: 1596
Max MTU: 1450
Max MRU: 1450
MRRU: 1600
Change TCP MSS: default

L2TP/IPsec server:

Max MTU: 1450
Max MRU: 1450
MRRU: 1600
Change TCP MSS: default

L2TP/IPsec server client interface:

Actual MTU: 1596

sindy · March 4, 2021, 12:09pm

Under what conditions do you test the throughput? I remember you use L2 tunneling, maybe some L2 flood consumes the bandwidth of the tunnel?

What does /interface monitor l2tp-interface-name show when you send no test traffic, and when you do?

hpet · March 4, 2021, 12:44pm

I am testing throughput usng iperf. I run iperf server on client side of l2tp and iperf -c on server side of l2tp. Before running iperf traffic is from 0-100kbps, when running iperf it varies arround 400kbps.

I don’t recall of any “major” changes on routers (both RB4011). Just added l2tp client interface as member of BASE interface list so that I am able to access remote router from central office (we discussed this a couple of days ago). It appears as something happened “over night”. Didn’t restart server router yet though.

Before generating traffic:

/interface monitor-traffic <l2tp-STUDENCI> 
                         name:  <l2tp-STUDENCI>
        rx-packets-per-second:               10
           rx-bits-per-second:         14.7kbps
     fp-rx-packets-per-second:                0
        fp-rx-bits-per-second:             0bps
          rx-drops-per-second:                0
         rx-errors-per-second:                0
        tx-packets-per-second:                9
           tx-bits-per-second:         27.3kbps
     fp-tx-packets-per-second:                0
        fp-tx-bits-per-second:             0bps
          tx-drops-per-second:                0
    tx-queue-drops-per-second:                0
         tx-errors-per-second:                0

Running iperf:

/interface monitor-traffic <l2tp-STUDENCI> 
                         name:  <l2tp-STUDENCI>
        rx-packets-per-second:               40
           rx-bits-per-second:         24.3kbps
     fp-rx-packets-per-second:                0
        fp-rx-bits-per-second:             0bps
          rx-drops-per-second:                0
         rx-errors-per-second:                0
        tx-packets-per-second:               34
           tx-bits-per-second:        314.1kbps
     fp-tx-packets-per-second:                0
        fp-tx-bits-per-second:             0bps
          tx-drops-per-second:                0
    tx-queue-drops-per-second:                0
         tx-errors-per-second:                0

Almost nothing…

Edit:
my tunnel is l2tp/ipsec bcp enabled to “stretch voip vlan”, but otherwise I also use it to route between networks.

sindy · March 4, 2021, 12:51pm

Does the iperf run in TCP or UDP mode? What is the round-trip delay of ping through the tunnel?

hpet · March 4, 2021, 12:57pm

iperf tcp mode.
udp mode is just a bit higher, varies around 1Mbps, no major difference.
round trip time varies between 11-15ms

sindy · March 4, 2021, 1:13pm

Is the H (hardware encryption) indicator shown at server side in /ip ipsec installed-sa print output?

Also, I’m afraid the 300 kBit/s Tx indicated in the /interface monitor output suggests that the stream from the iperf gets throttled before reaching the L2TP processing, as that is an input point to the L2TP processing. No QoS rules, no policy routing (routing-mark etc.) there?

hpet · March 4, 2021, 1:23pm

HE - indicator is shown.

I just rebooted both routers and problem remains.
No QoS rules, no policy routing.
Preaty much vanila setup.

I used iperf before and it was showing expected figures. I am not throttling anything, at least not on purpose

I will post config, maybe something shows up in there. Let me clean it up.

hpet · March 4, 2021, 2:03pm

I excluded wireless to cut down on config.
I did notice some “left overs” on remote site, like under ppp profile *default-encryption had some “unknown” interface… removed that. Or under /ip dns static had some invalid dns… removed that. Or under BR1 had ether1 (wan) member, which should not be there… removed that. I think these were left over from previous EoIP config. Cleared that up. Restarted l2tp tunnel, but still no love.

central site:

# mar/04/2021 14:24:29 by RouterOS 6.48
# software id = VIVR-KH3V
#
# model = RB4011iGS+5HacQ2HnD

/interface bridge
add name=BR1 protocol-mode=none vlan-filtering=yes
add name=BR20 protocol-mode=none

/interface vlan
add interface=BR1 name=PC-VLAN vlan-id=10
add interface=BR1 name=VOIP-VLAN vlan-id=20

/interface list
add name=WAN
add name=VLAN
add name=BASE

/ip pool
add name=PC-POOL ranges=172.31.1.50-172.31.1.99
add name=L2TP-POOL ranges=10.10.10.50-10.10.10.99

/ip dhcp-server
add address-pool=PC-POOL disabled=no interface=PC-VLAN name=PC-DHCP

/ppp profile
add bridge=BR20 comment="bcp bridged voip network" name=L2TP-VOIP-BCP use-encryption=required

/interface bridge port
add bridge=BR1 comment="trunk/uplink: V10, V20" interface=ether2
add bridge=BR1 comment="ingress/hybrid: access V10, trunk V20" ingress-filtering=yes interface=ether3 pvid=10
add bridge=BR1 comment="ingress/hybrid: access V10, trunk V20" ingress-filtering=yes interface=ether4 pvid=10
add bridge=BR1 comment="ingress/hybrid: access V10, trunk V20" ingress-filtering=yes interface=ether5 pvid=10
add bridge=BR1 comment="ingress/hybrid: access V10, trunk V20" ingress-filtering=yes interface=ether6 pvid=10
add bridge=BR1 comment="ingress/hybrid: access V10, trunk V20" ingress-filtering=yes interface=ether7 pvid=10
add bridge=BR1 comment="ingress/hybrid: access V10, trunk V20" ingress-filtering=yes interface=ether8 pvid=10
add bridge=BR1 comment="ingress/hybrid: access V10, trunk V20" ingress-filtering=yes interface=ether9 pvid=10
add bridge=BR1 comment="ingress/PBX: access V20" frame-types=admit-only-untagged-and-priority-tagged interface=ether10 pvid=20
add bridge=BR20 comment="V20 (voip) over BR20 (L2TP/BCP)" interface=VOIP-VLAN

/ip neighbor discovery-settings
set discover-interface-list=BASE

/interface bridge vlan
add bridge=BR1 comment="Egress/Hybrid: tagged BR1, ether2; untagged ether3-9" tagged=BR1,ether2 untagged=ether3,ether4,ether5,ether6,ether7,ether8,ether9 vlan-ids=10
add bridge=BR1 comment="Egress/PBX: tagged BR1, ether2-9; untagged ether10" tagged=ether2,ether3,ether4,ether5,ether6,ether7,ether8,ether9,BR1 untagged=ether10 vlan-ids=20

/interface l2tp-server server
set authentication=mschap2 default-profile=L2TP-DEFAULT enabled=yes ipsec-secret=**** mrru=1600 use-ipsec=required

/interface list member
add interface=ether1 list=WAN
add interface=VOIP-VLAN list=VLAN
add interface=PC-VLAN list=VLAN
add interface=PC-VLAN list=BASE

/ip address
add address=172.31.1.254/24 interface=PC-VLAN network=172.31.1.0

/ip dhcp-client
add disabled=no interface=ether1

/ip dhcp-server network
add address=172.31.1.0/24 dns-server=172.31.1.1 gateway=172.31.1.254

/ip dns
set allow-remote-requests=yes

/ip firewall filter
add action=accept chain=input comment="accept established,related" connection-state=established,related
add action=drop chain=input comment="drop invalid" connection-state=invalid
add action=accept chain=input comment="accept ICMP" protocol=icmp
add action=accept chain=input comment="accept IPsec ESP" protocol=ipsec-esp
add action=accept chain=input comment="accept IPsec NAT" dst-port=4500 protocol=udp
add action=accept chain=input comment="accept IKE" dst-port=500 protocol=udp
add action=accept chain=input comment="accept L2TP ipsec encapsulated" dst-port=1701 ipsec-policy=in,ipsec protocol=udp
add action=accept chain=input comment="accept PC_VLAN (base)" in-interface=PC-VLAN
add action=drop chain=input comment="drop all not comming from BASE" in-interface-list=!BASE
add action=drop chain=input comment="drop all"
add action=accept chain=forward comment="accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment=fasttrack connection-state=established,related
add action=accept chain=forward comment="accept established,related" connection-state=established,related
add action=accept chain=forward comment="accept PC_VLAN internet access" connection-state=new in-interface=PC-VLAN out-interface-list=WAN
add action=drop chain=forward comment="drop invalid" connection-state=invalid
add action=drop chain=forward comment="drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN

/ip firewall nat
add action=masquerade chain=srcnat comment="Default masquerade" out-interface-list=WAN

/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes

/ppp secret
add comment="routed network" local-address=10.10.10.1 name=STUDENCI profile=L2TP-VOIP-BCP remote-address=10.10.10.2 routes="172.31.2.0/24 10.10.10.2 1" service=l2tp

remote site:

# mar/04/2021 14:25:13 by RouterOS 6.48
# software id = YDGE-MDDX
#
# model = RB4011iGS+5HacQ2HnD

/interface bridge
add name=BR1 protocol-mode=none vlan-filtering=yes
add name=BR10 protocol-mode=none
add name=BR20 protocol-mode=none

/interface vlan
add interface=BR1 name=PC-VLAN vlan-id=10
add interface=BR1 name=VOIP-VLAN vlan-id=20

/interface list
add name=WAN
add name=VLAN
add name=BASE

/ip pool
add name=PR-POOL ranges=192.168.1.10-192.168.1.254
add name=PC-POOL ranges=172.31.2.50-172.31.2.99

/ip dhcp-server
add address-pool=PR-POOL disabled=no interface=BR10 name=PR-DHCP
add address-pool=PC-POOL disabled=no interface=PC-VLAN name=PC-DHCP

/ppp profile
add bridge=BR20 interface-list=BASE name=L2TP-VOIP-BCP use-encryption=required

/interface l2tp-client
add allow=mschap2 connect-to=******* disabled=no ipsec-secret=***** mrru=1600 name=L2TP-CENTER password=**** profile=L2TP-VOIP-BCP use-ipsec=yes user=STUDENCI

/interface bridge port
add bridge=BR1 comment="ingress/hybrid: access V10, trunk V20" ingress-filtering=yes interface=ether2 pvid=10
add bridge=BR1 comment="ingress/hybrid: access V10, trunk V20" ingress-filtering=yes interface=ether3 pvid=10
add bridge=BR1 comment="ingress/hybrid: access V10, trunk V20" ingress-filtering=yes interface=ether4 pvid=10
add bridge=BR1 comment="ingress/hybrid: access V10, trunk V20" ingress-filtering=yes interface=ether5 pvid=10
add bridge=BR1 comment="ingress/hybrid: access V10, trunk V20" ingress-filtering=yes interface=ether6 pvid=10
add bridge=BR1 comment="ingress/hybrid: access V10, trunk V20" ingress-filtering=yes interface=ether7 pvid=10
add bridge=BR1 interface=ether8
add bridge=BR1 comment="ingress: access V10" ingress-filtering=yes interface=wlan1 pvid=10
add bridge=BR1 comment="ingress: access V10" ingress-filtering=yes interface=wlan2 pvid=10
add bridge=BR10 comment=home interface=ether10
add bridge=BR20 comment="V20 (voip) over BR20 (L2TP/BCP)" interface=VOIP-VLAN

/ip neighbor discovery-settings
set discover-interface-list=BASE

/interface bridge vlan
add bridge=BR1 comment="egress/hybrid: tagged BR1, untagged e2-e9" tagged=BR1 untagged=ether2,ether3,ether4,ether5,ether6,ether7 vlan-ids=10
add bridge=BR1 comment="egress/hybrid: tagged BR1, e2-e9" tagged=BR1,ether2,ether3,ether4,ether5,ether6,ether7 vlan-ids=20

/interface list member
add interface=ether1 list=WAN
add interface=PC-VLAN list=BASE
add interface=VOIP-VLAN list=VLAN
add interface=PC-VLAN list=VLAN

/ip address
add address=192.168.1.1/24 interface=BR10 network=192.168.1.0
add address=172.31.2.1/24 interface=PC-VLAN network=172.31.2.0

/ip dhcp-client
add disabled=no interface=ether1

/ip dhcp-server network
add address=172.31.2.0/24 comment="studenci network" dns-server=172.31.1.1 gateway=172.31.2.1	
add address=192.168.1.0/24 comment="home network" dns-server=77.111.1.1,77.111.1.77,8.8.8.8,8.8.4.4 gateway=192.168.1.1
	
/ip dns
set allow-remote-requests=yes

/ip firewall filter
add action=accept chain=input comment="accept established,related" connection-state=established,related
add action=drop chain=input comment="drop invalid" connection-state=invalid
add action=accept chain=input comment="accept ICMP" protocol=icmp
add action=accept chain=input comment="accept IPsec ESP" protocol=ipsec-esp
add action=accept chain=input comment="accept IPsec NAT" dst-port=4500 protocol=udp
add action=accept chain=input comment="accept IKE" dst-port=500 protocol=udp
add action=accept chain=input comment="accept L2TP ipsec encapsulated" dst-port=1701 ipsec-policy=in,ipsec protocol=udp
add action=accept chain=input comment="accept PC_VLAN (base)" in-interface=PC-VLAN
add action=accept chain=input comment="winbox wan port" dst-port=8291 in-interface-list=WAN protocol=tcp src-address=82.149.19.89
add action=drop chain=input comment="drop all not comming from BASE" in-interface-list=!BASE
add action=drop chain=input comment="drop all"
add action=accept chain=forward comment="accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment=fasttrack connection-state=established,related
add action=accept chain=forward comment="accept established,related" connection-state=established,related
add action=accept chain=forward comment="accept PC_VLAN internet access" in-interface=PC-VLAN out-interface-list=WAN
add action=drop chain=forward comment="drop invalid" connection-state=invalid
add action=drop chain=forward comment="drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
add action=drop chain=forward comment="drop all from BR10 (home) to BR1 (company)" in-interface=BR10 out-interface=BR1

/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN

/ip route
add distance=1 dst-address=172.31.1.0/24 gateway=10.10.10.1

/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes

hpet · March 4, 2021, 2:21pm

Ok. I can point finger to IPSec now.
I disabled IPSec on L2TP client and iperf jump to 70Mbps+ as would be expected.

I remeber couple of days ago we discussed L2TP to happily continue if IPSec session terminates.
We discussed some firewall rules… and then you pitched in with additional ipsec policy order.
I didn’t implement this part yet, but did made a minor change in firewall on central (l2tp server) router:

add action=accept chain=input comment=“accept L2TP ipsec encapsulated” dst-port=1701 ipsec-policy=in,ipsec protocol=udp

hpet · March 4, 2021, 2:27pm

removed that, but still the same.
it works only when I remove ipsec on l2tp.

sindy · March 4, 2021, 3:01pm

Would you mind moving to 6.48.1 (or to 6.47.9)? The 6.48 seems to be one of the worse ones.

And yes, your test indicates clearly that IPsec is responsible, the question is how exactly.

Regarding that action=accept dst-port=1701 … rule - given the overall setup of your firewall, adding ipsec-policy=in,ipsec to that rule is functionally equivalent to setting use-ipsec=required in /interface l2tp-server server. Both that rule and that setting only affect the establishing of the L2TP connection, and once it establishes, they don’t interfere any more.

I.e. the modification of this rule has nothing to do with the speed, all mid-connection L2TP transport packets are handled by the action=accept connection-state=established,related rule as before.

I cannot spot anything in the configurations that would explain the speed drop.

With IPsec on, can you sniff at both devices into a file for either ipsec-esp packets if both devices have a public IP, or of UDP packets with port=4500 if at least one is behind a NAT, while running the iperf test, to be sure that what leaves the server side also arrives to the client one?

And when IPsec is off, what does the monitor-traffic show while the test is running at 70 Mbit/s?

hpet · March 4, 2021, 3:14pm

Silly question, how do I downgrade?
…but will try to sniff packets before doing that.

sindy · March 4, 2021, 3:19pm

In general, you download the older .npk package (or set of packages if you don’t use the bundle) to the router using /tool fetch or upload it there using Winbox, WebFig, or sftp client like WinSCP. Then, run /system package downgrade and confirm the reboot.

But in this case, switching the channel from stable to long-term and then asking for an upgrade might work too.

hpet · March 4, 2021, 7:05pm

Just started sniffing packets etc. when I received a call from one person on remote location and she said everything is working well for her. I immediately ran iperf against her computer and on my surprise everything is running as expected. Now I am lost.
iperf yielding 65Mbps against her and 300kbps against another computer on the same remote network.

What could this new finding mean?
Testing this pc against internet is showing full bandwidth as expected.

sindy · March 4, 2021, 9:09pm

Pojma nemam… Especially as against the test pc the speed depends on use of IPsec.

nichky · March 5, 2021, 4:18am

@sindy -molimte reci mi ovo:

i got router with hardware encryption, (hEX S RB760iGS)
and from /ip ipsec installed-sa print - i can see only E , can’t see H. what im missing?

nichky · March 5, 2021, 4:26am

it was hidden ..all good

hpet · March 5, 2021, 8:58am

Are there any kind of ipsec caching tables I could flush/reset?
It is obviously something with IPsec.
Asked remote user to connect to a different port (doesn’t hurt to try but still the same.
Also changed cable, just in case. No joy.

sindy · March 5, 2021, 11:02am

You can flush the “installed SA”, but that changes nothing - first, they are short-lived (a rekey takes place every 30 minutes by defaut, and it actually means a creation of a new SA, a switchover to it, and removal of the old one), and second, all the differences between the two remote devices are hidden inside the L2TP transport packets. So the IPsec security association can see only UDP packets from port 1701 to port 1701 and doesn’t care about their payload. So it would have to be some bit pattern in the payload, i.e. the inner IP header inside the PPP packets, which would cause the encryption hardware to slow down - I’ve seen too much to use the word “impossible” too often, but it doesn’t seem a likely scenario to me.

Long time ago, there was a special patch for RB3011 (also ARM architecture) which, if I got it right, took care of processing all packets transported using a single IPsec SA by the same CPU core, as the receiving side had throughput problems when the packet arrived in swapped order.

So you can try another recipe which a user here has published recently for SSTP, which is to make the virtual interface a parent of a queue in /queue tree, without even forcing any traffic into that queue. But there is a significant difference between SSTP (which encrypts the payload packets in a single step and uses TCP as transport, so missequenced packets are a headache already for TCP) and L2TP/IPsec, where the IPsec SA takes the L2TP transport packets as they arrive and if the order of the L2TP transport packets is shuflled, the IPsec SA transports them in that shuffled order without even noticing that.

So application of that recipe (it may be easier to reserve a static interface name for the LT2P user at the server side than to use the on-up and on-down scripts on /ppp profile row) may prevent L2TP transport packets from getting shuffled, and thus the re-establishment of the payload stream at receiving side will not have to wait for the previous packet), but the relationship to IPsec would then be only indirect (if IPsec processing is not used, the number of L2TP encapsulations processed by other core may be lower). So nothing sharp and clear, just assumptions.

hpet · March 5, 2021, 7:01pm

I downgraded both routers to long-term 6.47.9, clear reset all configuration, reset both routers… same problem. Only one PC on remote network seem unaffected… everything else is running unusably slow over l2tp/ipsec. Created new tunnel, nothing. If I turn off ipsec, everything works. That is just crazy. I had high hopes on whiping all out…