Community discussions

MikroTik App
 
vikinggeek
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 54
Joined: Sat Aug 02, 2014 4:14 am

L2TP cannot connect across US border on AT&T network

Tue Sep 20, 2022 9:35 am

All,

We are running into a very strange problem with our network. It has multiple L2TP/IP Sec links between our nodes and routing management is done via OSPF. We are still running 6.49.6 throughout our network. All good.

Last year, the L2TP links between AT&T fiber and our European nodes would not any longer connect. No change to configuration. We traced down the problem to an Internet exchange in New York. The problem seems to be that a server response is lost in the network. Please refer to the L2TP model as described in Wikipedia here https://en.wikipedia.org/wiki/Layer_2_T ... g_Protocol see section "L2TP packet exchange”.

When connection from <Client IP> to <Server IP>, the following interaction takes place (as observed in the packet sniffer on both sides):
  1. Client issue a SCCRQ call to <Server IP> (141 bytes)
  2. The server respons with a SCCRP response to <Client IP> (140 bytes)
  3. The client sends a SCCCN call (64 bytes)
  4. The server responds to the SCCCN call with a 54 byte reply, but that reply never reaches the client.

The net result of the missing packet is that the client does not go to the next step and request a session as it thinks that the server is “dead”. It therefore restart on 1) instead of issuing a ICRQ call.

The strange thing is that from Comcast, Optimum, etc (DOCSIS ISPs) we don't have this problem. When connection with an L2TP/IPSec client e.g. Mac it works, and AT&T therefore claim that the problem is with our Tiks.

The workaround has been to check the "Use IPsec" flag in the L2TP dial-out dialog. The encoding on the L2TP connection changes from "MPPE128 stateless" to "cbc(aes) + hmac(sha1)". It uses the default IPSec policy that prevents us from assigning static IP addresses to the tunnel and therefore making the OSPF management harder. We can also create a PPTP link, but this has obviously other security related problems.

Has anyone else experienced this problem? Is there another workaround possible so that we can statically assign IP addresses to the tunnel endpoints in the IPSec Policy and not rely upon the default policy?

Any help or insight into this challenge would be appreciated.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: L2TP cannot connect across US border on AT&T network

Tue Sep 20, 2022 1:38 pm

When connection with an L2TP/IPSec client e.g. Mac it works, and AT&T therefore claim that the problem is with our Tiks.
From what you wrote it is no proof, as the other clients use L2TP/IPsec whereas your Mikrotik clients normally use bare L2TP without IPsec. So even the 54-byte SCCCN response gets encapsulated into a larger packet if IPsec is used.

Is there another workaround possible so that we can statically assign IP addresses to the tunnel endpoints in the IPSec Policy and not rely upon the default policy?
I'm not sure I understand the issue, as normally the dynamically created IPsec policy inherits all the address information from the L2TP configuration, but I don't need to understand everything.
By setting use-ipsec to yes, you tell RouterOS to generate the complete IPsec configuration (peer & policy) necessary to transport the L2TP control&transport packets using the default profile and the policy template(s) in policy group default (which typically refer to the default proposal).

So the easiest way is to copy both these dynamically created items into static ones changing their names to non-conflicting ones and keeping the internal references (i.e. the manually configured policy must refer to the manually configured peer), then set use-ipsec back to no and continue with the manually configured IPsec.

This way you can also step away from the default settings (profile, proposal, policy template group) if you find it useful.
 
vikinggeek
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 54
Joined: Sat Aug 02, 2014 4:14 am

Re: L2TP cannot connect across US border on AT&T network

Wed Sep 21, 2022 2:13 am

Thank you Sindy for the quick response!

From what you wrote it is no proof, as the other clients use L2TP/IPsec whereas your Mikrotik clients normally use bare L2TP without IPsec. So even the 54-byte SCCCN response gets encapsulated into a larger packet if IPsec is used.
Agree, we have argued this to AT&T as well, but obviously AT&T knows better. They claim that if they were dropping packets, they would get thousands of complaints. :roll:
So the easiest way is to copy both these dynamically created items into static ones
This is what we have had all along. Though it is painful to maintain, for each L2TP connection we have policy, proposal, peer, and identities static configurations. The policy is using transport mode.

What I don't understand is how setting the "Use IPsec" flag in the L2TP dial-out dialog changes the encoding from "MPPE128 stateless" to "cbc(aes) + hmac(sha1)". To me, this indicates that Mikrotik changes the connection from a pure point-to-point using L2TP to a regular IPSec. My understanding is that an IPSec connection does not have an endpoint in the router (read interface) which we can use for our OSPF routing, but I may be wrong on that.

Furthermore, the original static policy for the L2TP connection (without setting the "Use IPsec" flag) is still active showing "established" in the PH2 state column. Both policies (static and dynamically generated) show up in "Installed SAs". Does not that indicate that there is a double encryption going on?

What other tunneling options do we have in this case that a) is IPSec or encrypted and b) provides an interface for our OSPF routing?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: L2TP cannot connect across US border on AT&T network

Wed Sep 21, 2022 9:14 am

Though it is painful to maintain, for each L2TP connection we have policy, proposal, peer, and identities static configurations. The policy is using transport mode.
Given the following, maybe you could actually return to the automatically generated peers&policies?

What I don't understand is how setting the "Use IPsec" flag in the L2TP dial-out dialog changes the encoding from "MPPE128 stateless" to "cbc(aes) + hmac(sha1)". To me, this indicates that Mikrotik changes the connection from a pure point-to-point using L2TP to a regular IPSec.
It is just a confusing way how the monitor works. Same like setting use-ipsec to yes causes RouterOS to dynamically generate the peer and policy, it also makes it show different information in monitor. When bare L2TP is used, monitor shows information about the PPP encryption (or its absence); when IPsec is used to encrypt L2TP, monitor indicates the IPsec encryption&authentication parameters.

My understanding is that an IPSec connection does not have an endpoint in the router (read interface) which we can use for our OSPF routing, but I may be wrong on that.
Your understanding of bare IPsec behavior is correct, your understanding of what use-ipsec=yes does is not. It doesn't mean "use IPsec instead of L2TP" nor "use IPsec to encrypt the payload traffic before sending it down the L2TP tunnel", it means "use IPsec in transport mode to encrypt the L2TP transport&control packets". So the tunnel interface (/interface l2tp-client or /interface l2tp-server) is there in both cases.

Furthermore, the original static policy for the L2TP connection (without setting the "Use IPsec" flag) is still active showing "established" in the PH2 state column. Both policies (static and dynamically generated) show up in "Installed SAs". Does not that indicate that there is a double encryption going on?
It depends on what the traffic selector (src-address, dst-address, protocol, src-port, dst-port), mode (tunnel=yes or no), and peer of the manually configured policy are.

If the manually configured policy was a verbatim copy of the one created dynamically thanks to use-ipsec=yes, one of them should normally be marked as invalid, because it would be shadowed by the other one. Already two peers with identical local-address, remote-address, and exchange-mode should cause an error indication, as one of them would be shadowed by the other for incoming connection requests. It could be that there is an issue with this, though - Mikrotik is constantly doing subtle changes in IPsec handling and I don't check unusual configurations like this with every new release.

But if you've manually configured a tunnel-mode policy with traffic selector matching whole subnets, you have indeed superseded the results of the normal routing and although the L2TP tunnel is established, it doesn't carry any actual payload.

If uncertain even after this explanation, post /ip ipsec peer print, /ip ipsec active-peers print, and /ip ipsec policy print detail when the system is in the state above, after systematically replacing all occurrences of any given public IP address by a unique symbolic name (so that the relationship between policies and peers remains visible).

BTW, when you use IPsec to encrypt L2TP, there is no point in using encryption on PPP level, it just adds extra load to the CPU. I'm not aware that use-ipsec=yes would disable PPP encryption automatically, but as stated above, this may have been added without me noticing it.
 
vikinggeek
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 54
Joined: Sat Aug 02, 2014 4:14 am

Re: L2TP cannot connect across US border on AT&T network

Thu Sep 22, 2022 3:20 am

This is a lot of good information that I wish was covered in the manual!

Given the following, maybe you could actually return to the automatically generated peers&policies?
The point of assigning static /32 addresses (10.0.X.Y) to each link, is that we can then assign weight (i.e. distance) to those link and we avoid OSPF creating routes that take you on a trip to another continent. I guess we could have used BGP, but OSPF seems to be better suited for internal routing.

It is just a confusing way how the monitor works.
I learned something new here. Thank you!

it means "use IPsec in transport mode to encrypt the L2TP transport&control packets".
Ah, another revelation. Thank you!

If the manually configured policy was a verbatim copy of the one created dynamically
As explain above, each link is given an 10.0.x.y/32 address. In some countries we are not able to obtain static addresses for some sites and the DHCP provided address to the sites changes too often to be practical to be used for the IPSec policy. This is how we ended up with the manual configuration and pseudo addressing scheme.

But if you've manually configured a tunnel-mode policy with traffic selector matching whole subnets
OSPF is handling the route to a subnet, the L2TP/IPSec links are just "highways" between nodes/sites and given stable "name" via the 10.0.x.y/32 addressing scheme.

Since there are many links, I have just provided the info for the AT&T link we are having issues with.

/ip ipsec peer print (server)
name="treskeveien" address=10.0.16.25/32 local-address=10.0.16.26 profile=actinet_def exchange-mode=ike2 send-initial-contact=yes

/ip ipsec peer print (client)
name="newport" address=10.0.16.26/32 local-address=10.0.16.25 profile=actinet_def exchange-mode=ike2 send-initial-contact=yes 

/ip ipsec active-peers print (server)
2    10.0.16.25           established        2d22h14m28s             1 10.0.16.25                                                                                                                    
4 R                       established        12h38m36s               1 84.209.x.x                                                                                                                 

/ip ipsec active-peers print (client)
 2 R  10.0.16.26           established        2d22h18m40s             1 10.0.16.26                                                                       
 4                         established        12h42m49s               1 99.88.x.x                                                                     

/ip ipsec policy print (server)
4   DA  vpn                                                       no     99.88.x.x/32                                                                                84.209.x.x/32                                                                               udp        encrypt unique           1
13   A  ;;; <Client Site Name>
        treskeveien                                               no     10.0.16.26/32                                                                                  10.0.16.25/32                                                                                  all        encrypt require          1

/ip ipsec policy print (client)
 0   DA  <Server Site Name from L2TP connection>           no     84.209.101.22/32                                                  99.88.41.105/32                                                   udp        encrypt require          2
 2   A  ;;;<Server Site Name>
        newport                    no     10.0.16.25/32                                                     10.0.16.26/32                                                     all        encrypt require          1

BTW, when you use IPsec to encrypt L2TP, there is no point in using encryption on PPP level
So, we should uncheck the MSChap 1 and 2 options since this is not needed when we use IPSec?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: L2TP cannot connect across US border on AT&T network

Thu Sep 22, 2022 9:18 am

I must say I am confused by the topology and even by the fact that it works the way you expect. What your /ipsec ... print outputs show is that you use IKEv2 IPsec policies in transport mode to link the pairs of /32 private addresses, which (probably) run inside the L2TP tunnels. So with L2TP/IPsec, you end up with three nested tunnels (the IKEv2 one is transported via the L2TP one which itself is transported via an IKE(v1) one).

You've mentioned OSPF in your initial post. I cannot exclude that it is possible to transport the OSPF communication itself using an IPsec policy, it never came to my mind to try that, but the network the OSPF "drives" must consist of interfaces and subnets attached to them. So my assumption is that 10.0.16.25/32 and 10.0.16.26/26 are the addresses you use as the routers' own addresses in the OSPF communication. But as the inner IKEv2 policies ("tunnels") only link these addresses, the general traffic doesn't flow through these and enters the L2TP links in plaintext.

This would still work for me, even though encryption of OSPF messages seems pointless to me, but the statement
The point of assigning static /32 addresses (10.0.X.Y) to each link, is that we can then assign weight (i.e. distance) to those link and we avoid OSPF creating routes that take you on a trip to another continent.
suggests a deep misunderstanding at my side or yours. I fully understand that you want to affect the choices made by the OSPF, as a path through 3 other routers in the same country is still better than a path through a single router in an overseas country due to the much higher delay on the latter. But I don't get how the distance values on routes to the 10.0.16.x destnations should affect the path calculations made by OSPF. The distance parameter only decides which of the routes to exactly the same destination will be active in a local routing table if multiple such routes are available. To control the preference of the individual paths in OSPF calculations, the cost parameter of /routing ospf interface object is used.

Maybe a voice call would be better, as it seems there's a lot of information missing and the 24h round-trip delay is not practical?

So, we should uncheck the MSChap 1 and 2 options since this is not needed when we use IPSec?
The use of MPPE (encryption of payload data in L2TP) is controlled by the use-encryption parameter on the /ppp profile row. mschapX are options for user authentication during establishing of the L2TP tunel. So if you use L2TP/IPsec, you can use ppp profile default rather than default-encryption (unless you use custom profiles for some reason).
 
vikinggeek
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 54
Joined: Sat Aug 02, 2014 4:14 am

Re: L2TP cannot connect across US border on AT&T network

Fri Sep 23, 2022 3:33 am

I must say I am confused by the topology and even by the fact that it works the way you expect.
The architecture has evolved, but for the past 5+ years it has served us well. Since OSPF is dynamically updating the routing tables, the hardest part is automatic. In cases where we loose a trans Pacific link due to a typhoon or in this case a misconfiguration in the NY Internet exchange, our network automatically reconfigures. In fact, the AT&T problem was only discovered after a user complained about poor performance.

suggests a deep misunderstanding at my side or yours
Let me share the architecture, and matters may become more clear:
  1. The first step bringing up a site is to determine the type of connection, speed, and distance. Latency is obviously very important, but symmetrical vs. asymmetrical bandwidth also plays an important role. This is reflected in the as you noted the cost parameter of /routing ospf interface object
  2. The next step is to define the tunnel. Up to now, his was to create a simple L2TP connection between the two routers. On the server side, we specify the 10.0.x.y. local and remote addresses which the router correctly assign to the interfaces on the client and server side.
  3. The /32 addresses are then used to configure the IPSec parameters to encrypt the traffic on the L2TP tunnel using IPSec transport mode (documented above with IKEv2). They are also used by the firewall to allow traffic (/32s in an address list). This tunnel can obviously only see the router on the other end, but that's enough for OSPF to make is discovery of another OSPF deamon and the routers are starting to exchange topology information (which can be seen by an increase in version number)
  4. We update the firewall address list and include the new subnet which for us is typically /21 IP range.
  5. The final step is to add the /32 named interface to the OSPF configuration including setting the cost parameter value like this
    cost=<as described>  interface=<interface name> network-type=point-to-point
There is obviously some OSPF setup work like defining the instance value. We also only use one area (backbone) as we don't yet exceed recommended values for a single area.

The bottom line is that we only need to set up the tunnels between routers and OSPF takes care of the rest. Though we need to put in work to define the "physical" layer, all the routing is now automatic.
But as the inner IKEv2 policies ("tunnels") only link these addresses, the general traffic doesn't flow through these and enters the L2TP links in plaintext.
Reading through your comments, I'm getting a little worried. I have always assumed that our transport configuration of IPSec encrypted the payload of the tunnel. That's something I need to revisit.
you can use ppp profile default rather than default-encryption
Got it, the configuration is set to default-encryption at the moment, but it can easily switch to default.

Maybe a voice call would be better, as it seems there's a lot of information missing and the 24h round-trip delay is not practical?
I would very much like to talk to you. Currently, I'm in California, but I can easily do a call late at night to reach a European destination. DM how to reach you?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: L2TP cannot connect across US border on AT&T network

Fri Sep 23, 2022 9:46 am

  • The final step is to add the /32 named interface to the OSPF configuration including setting the cost parameter value like this
    cost=<as described>  interface=<interface name> network-type=point-to-point
This shows how a precise wording is important for understanding :)

I've also learned something new, my colleagues kept telling me that Mikrotik was bad for not supporting /31 networks as OSPF "didn't work on that 'emulation' of /31 subnets using one /32 as the address and another /32 as the network". Based on your description, I will have to revisit the topic with them.


Reading through your comments, I'm getting a little worried. I have always assumed that our transport configuration of IPSec encrypted the payload of the tunnel. That's something I need to revisit.
The only cheer up I can give you is that there is at least the MPPE for the non-OSPF traffic.

This cannot be resolved by using "wider" policies for the IKEv2 sessions, but you can enable the "automatic" IPsec encryption of the L2TP control&transport packets for all links, not just the AT&T ones. Or, if you don't like IKE(v1) and/or the pre-shared key authentication, you can use IKEv2 with some more advanced authentication setup to encrypt the L2TP C&T packets instead, but you'd have to configure it manually.

I would very much like to talk to you. Currently, I'm in California, but I can easily do a call late at night to reach a European destination. DM how to reach you?
It seems the need to talk became less urgent as the misunderstandings have been clarified by now, but still no problem. Since DM is long disabled here, use this. Your 9-12 AM maps to my 6-9 PM so not a big deal; if you are an owl, your 11PM-1AM is my 8-10 AM, also not a big deal.
 
vikinggeek
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 54
Joined: Sat Aug 02, 2014 4:14 am

Re: L2TP cannot connect across US border on AT&T network

Thu Sep 29, 2022 4:54 am

It has taken me a couple of days to test through the options:
  • Setting the L2TP client and server profile to "default" is ok since the IPSec tunnel already provides encryption
  • There is no need to have a manually configured policy since the default template creates a good configuration if the "Use IPSec" flag is checked in the L2TP dial-out
  • The setting of the values in the default policy template does not make sense. According to the table in https://wiki.mikrotik.com/wiki/Manual:I ... celeration, both my routers for the test (CCR1009 and RB750Gr3) support a combination of AES and SHA256 encryption. However, when I set the value to SHA256, the IPSec tunnel is not established. Setting it back to SHA1, the tunnel comes up and the L2TP tunnel is established.
  • OSPF is working as intended with the /32 addresses if the L2TP tunnel is up

I can accept the AES-256 CBD with SHA1, I just cannot understand why SHA256 is not working?

My second question is regarding the use of group in the default policy. Since both src and dst is set to 0.0.0.0/0, it would have been nice to be able to configure the group for the L2TP dial-out parameters? This way we could invoke different policies if I'm not mistaken? This would allow us to use more secure connection(s) if needed for specific links. Or is there another way to do this?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: L2TP cannot connect across US border on AT&T network

Thu Sep 29, 2022 11:17 am

I can accept the AES-256 CBD with SHA1, I just cannot understand why SHA256 is not working?
That sounds weird indeed. No explanation comes to my mind.

My second question is regarding the use of group in the default policy. Since both src and dst is set to 0.0.0.0/0, it would have been nice to be able to configure the group for the L2TP dial-out parameters? This way we could invoke different policies if I'm not mistaken? This would allow us to use more secure connection(s) if needed for specific links. Or is there another way to do this?
It's the other way round. The identity item used to identify and authenticate the remote peer defines the group within which the templates for Phase 2 of that session are searched for. The identity item the L2TP server dynamically creates uses policy-template-group=default, and when the L2TP/IPsec client initiates the IPsec connection, a policy is generated using one of the template policies in that group (default). So you could set up multiple templates with different dst-address to link the proposal to the public IP of the client, but I don't think such an approach is significantly simpler than configuring the whole IPsec hierarchy manually rather than dynamically.
 
vikinggeek
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 54
Joined: Sat Aug 02, 2014 4:14 am

Re: L2TP cannot connect across US border on AT&T network

Thu Sep 29, 2022 11:43 am

That sounds weird indeed. No explanation comes to my mind.
Could it be that I didn't wait long enough and some token was still hanging around?

It's the other way round.
Ah, :idea: Thank you for clearing my confusion.

So you could set up multiple templates with different dst-address to link the proposal to the public IP of the client
Since we are not always able to obtain a static IP address for a client, matching a public address is not really feasible. I guess we will have to accept that the our default policy is going to be SHA1 for authentication and AES-256 CBD for encryption unless I can figure out why SHA256 is not working.

Thank you for your help Sindy, you gave me get a better understanding of how to set this up with IPSec since AT&T is continuing to block the L2TP connection. Our previous scheme to only encrypt the L2TP payload is now replaced with a full L2TP/IPSec implementation where we also encrypt the L2TP header. Hopefully our use of OSPF gave you some ideas into a better routing architecture as well.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: L2TP cannot connect across US border on AT&T network

Thu Sep 29, 2022 11:56 am

Could it be that I didn't wait long enough and some token was still hanging around?
No tokens. Once you change the proposal at one peer, that peer terminates the SAs using that proposal and starts re-negotiating them, which keeps failing until you change the proposal at the other peer. So unless you've made some mistake, two minutes at worst should be sufficient even if there are some load reducing time gaps (which do exist for L2TP itself and Phase 1, not sure about Phase 2).

Since we are not always able to obtain a static IP address for a client, matching a public address is not really feasible.
The dst-address in a policy template may be a prefix (subnet), so what I had in mind was a dedicated template policy for a prefix rather than for an individual address. The prefix would cover the whole range used by a particular ISP in a particular country. But that's just a clarification remark, as said I don't think this approach has any significant advantage.
 
vikinggeek
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 54
Joined: Sat Aug 02, 2014 4:14 am

Re: L2TP cannot connect across US border on AT&T network  [SOLVED]

Wed Oct 05, 2022 10:35 am

I filed a ticket with Mikrotik support and learned a few relevant facts regarding the sha256 challenge:
  • The default proposal under
    /ip ipsec proposal print
    controls the authentication and authorization parameters used for the L2TP/IPSec connection
  • The policy is not utilized i.e. whatever proposal is set for the default policy is ignored

I would also like to mention that we are tunneling IPv6 over this link using OSPFv3. The configuration is easier than IPv4 as you only need to chose the interface in OSPFv3 and ROS will assign a link local address. However, you will need to set MTU to 1290 on this link as the size of the IPv6 headers reduces the effective payload.

With the help from Sidney, we have found a good work around for the AT&T problem, and I hope this thread will help others in the same situation.

Who is online

Users browsing this forum: Amazon [Bot], danriis, sebus46 and 35 guests