Cisco VPN 3002 and Mikrotik

RouterOS General
1

Ok, I have a mikrotik network routers and wireless devices.. One of my users is having a hard time with his Cisco VPN3002 concentrator. It connects and establishes the IPSEC VPN. The user can ping devices in his internal network and a tracert shows that all packets are tunneled to his network and leaving through his internet gateway.

Here’s the catch, the user can not authenticate with his windows 2003 domain, nor can he RDP (Remote Desktop) to his servers. And none of his network based applications function.

I have a Mikrotik as my edge router and all devices in my network all mikrotik (wireless bridges). The edge router is also doing minor firewalling. The WAN port has a Point to Point link to my upstream provider (/30 subnet). The only ports I’m currently blocking are incoming 25 (tcp), 135-139 (tcp/udp), and 445 (tcp).

Now before any of you blurt out and say “well thats why he can’t authenticate, you’re blocking NetBIOS and Microsoft Directory Services” he is connecting to his network through a VPN Tunnel. Aside from the Mikrotik ripping apart the VPN tunnel and dropping those packets within the tunnel (which I’ve yet to see a firewall do), I can’t seem to explain this. And just to be sure I put in an Accept rule for his IP address so that all packets destined for his static IP address were accepted.

I want to pass it off to the client and say it’s a problem with his internal network, but when he plugs the same device into another ISP it works flawlessly.

Here’s the layout of the network. Client’s Cisco VPN3002 → Tranzeo/Mikrotik/Zcomax CPE (i’ve tried all three same results) → Mikrotik AP (2.4) → switch —> Mikrotik WDS-Station (5.8 ) —> Mikrotik AP (5.8 ) —> Layer3 switch —> Allot Netenforcer —> Mikrotik Edge Router —> upstream provider fiber MPLS network.

I’ve been beating my head up against a wall on this one, any suggestions?

I even had him take his Cisco 3002 from the client location to the next hop up to rule out the 2.4 link, plugged in and got the same results. This all worked for two months straight, we made some network upgrades. We swapped out a 5.8 PtP link that was underutilized for a 5.8 PtMP, removed are cisco 2600 series edge router and replaced with a mikrotik… And now he has problems… My brain is fried, I’ll check back tomorrow.

-Bill

P.S. I’m using 2.9.17 on all mikrotik devices..

and I guess I should mention there’s no NAT being done on my network, all nodes get publicly routeable IP addresses.

2

that’s far to complex for an easy answer.
i would check end-to-end connectivity by using portscanner and packet-analyzer, if possible.

for a complicated mixture of protocols and applications, maybe installing a direct EoIP-tunnel through IPSec would be the easiest solution.

3

Yeah the packet analyzer was my next move. Just thought I’d check with the forum to see if anyone else had encountered a similar problem.

4

This sounds like an MTU problem. Ping works because the packets are small. As soon as you start to put anything more complicated through, it falls over.

I had some problems with VPN tunnels with earlier versions of 2.9. I don’t know about 2.9.17 because I skipped that release. Some tips:

Make sure that the MT interface MTUs are set to 1500.

Iptables requires connection tracking turned on to do fragment re-assembly. MT is probably the same.

Adjust the MTU on the client downwards.

Make sure that you’re not blocking all ICMP at the gateway. This will cause problems with path discovery.

Regards

Andrew

5

MTU’s are all set to 1500, ICMP is allowed throughout the network, and we’ve tried lowering the MTU, but I’ll suggest that again.

I believe we tried MTU values of 1300, 1400, 1480 on the Cisco 3002. I’ll re-check and verify.

Connection Tracking had been on, I disabled today as I thought it may have been contributing to the problem. Turned out not to be the case so I re-enabled.

I’ve tried pinging with a load here are the results…

/ping 66.xx.xxx.1 size=1500
66.xx.xxx.1 1500 byte ping: ttl=64 time=70 ms
66.xx.xxx.1 1500 byte ping: ttl=64 time=12 ms
66.xx.xxx.1 1500 byte ping: ttl=64 time=10 ms
66.xx.xxx.1 1500 byte ping: ttl=64 time=12 ms
66.xx.xxx.1 1500 byte ping: ttl=64 time=11 ms
66.xx.xxx.1 1500 byte ping: ttl=64 time=25 ms
66.xx.xxx.1 1500 byte ping: ttl=64 time=34 ms
66.xx.xxx.1 1500 byte ping: ttl=64 time=14 ms
66.xx.xxx.1 1500 byte ping: ttl=64 time=12 ms
66.xx.xxx.1 1500 byte ping: ttl=64 time=12 ms
66.xx.xxx.1 1500 byte ping: ttl=64 time=12 ms
66.xx.xxx.1 1500 byte ping: ttl=64 time=48 ms
66.xx.xxx.1 1500 byte ping: ttl=64 time=13 ms

That’s pinging from the 2.4 AP through to the gateway with a 1500 byte packet.

Here’s another fun fact to throw in, when using a Cisco software VPN client everything works fine.

6

I was assuming that the client device was using the software Cisco VPN client and the server was the 3002. Didn’t pick up that wasn’t the case from your original post.

Is the software client doing encapsulation? Is this ESP/AH?

Your 1500 byte ping will be fragmented. What’s the maximum size packet you can get through with the DF bit set?

What does logging on the VPN server have to say? Does it show packets being discarded?

So many questions :slight_smile: I must say, I hate IPSEC! I love what it does when it’s working, but that’s as far as it goes.

Regards

Andrew

7

Yeah it’s a hardware 3002 VPN Access Gateway that connects back to their 3005 VPN Access Concentrator.

Here’s a 1500 byte ping from the AP to the gateway with the do not fragment bit set. (this not going through the 2.4 link, im not on site so I can’t do the ping from the actual client site.)

I believe the Cisco 3002 successfully negotiates phase1 and phase2, but I’ll see if I can get a peak at the logs..

/ping 66.60.188.1 size=1500 do-not-fragment 
66.60.188.1 1500 byte ping: ttl=64 time=55 ms
66.60.188.1 1500 byte ping: ttl=64 time=32 ms
66.60.188.1 1500 byte ping: ttl=64 time=12 ms
66.60.188.1 1500 byte ping: ttl=64 time=55 ms
66.60.188.1 1500 byte ping: ttl=64 time=45 ms
66.60.188.1 1500 byte ping: ttl=64 time=14 ms
66.60.188.1 1500 byte ping: ttl=64 time=26 ms
66.60.188.1 1500 byte ping: ttl=64 time=10 ms
66.60.188.1 1500 byte ping: ttl=64 time=12 ms
66.60.188.1 1500 byte ping: ttl=64 time=13 ms
66.60.188.1 1500 byte ping: ttl=64 time=15 ms
66.60.188.1 1500 byte ping: ttl=64 time=81 ms
12 packets transmitted, 12 packets received, 0% packet loss
round-trip min/avg/max = 10/30.8/81 ms
8

The other thing that occurs to me is to repeat this from a client on their site through the VPN to see what size packet will actually get through the tunnel. Then reduce the MTU on their clients accordingly.

Regards

Andrew

9

You think its a problem with path MTU discovery? I’ll look into it, the client will be at a different site today that we just hooked up on a different segment and he will test there. We’ll see what happens. I’ll have him play around with MTU sizes.

10

OK its definately looking like an MTU problem… but I can’t figure out where.

The max packet size he can ping to his VPN concentrator through the 3002 Hardware VPN client is 1418 bytes. I can ping his IP address from my router w/ a 1500 byte packet on the internal network.

I had him lower his MTU to 1400, but no change.

I’m thinking this is a problem with our WAN/LAN ports on the router. When I open the properties for those ports and check the status the auto negotiation says “incomplete”, the rate is “unknown”, and the fullduplex light is grey. They’re the internal motherboard ports which are 10/100/1000, I’m going up to the tower right now to switch them over to two of the RB44 10/100 ports to see if this solves the problem. Which at this point I’m fairly confident it will.

11

The gigabit ports are the “SysKonnect SK-98xx/SK-95xx Gigabit Ethernet” at least that’s the driver thats listed in RouterOS.

They’re built into an ASUS motherboard, I can’t remember the model, but if its the root cause of this problem I’ll be sure to get all that information so this can be corrected, or avoided in the future!

-Bill

12

OK,

switching interfaces didn’t do anything. Swapped out the Mikrotik edge router for the old Cisco router, still couldn’t get the client working..

The only other change we’ve made that I haven’t switched back was the upgrade to 2.9.17… I would try downgrading, but the “downgrade” button in winbox doesn’t seem to be worth its code.

I’ve noticed I can’t ping any of my Mikrotik routers from the internet with a 1500byte packet.. At least the ones running 2.9.17. A few of my old RB230’s that are out of their upgrade period are still running 2.8.xx and I can ping those devices with a 1500 byte packet.

Any idea’s why? is there something in 2.9.17 that would prohibit the router from accepting a 1500 byte ping?

13

I’ve heard that there was a fragment reassembly problem with some versions of 2.9.

I can ping my 2.9.18 router from the internal lan with 1500 and 1600 byte packets. The 1600 byte ones must be fragmented so I presume this version doesn’t have the problem.

Regards

Andrew

14

Yeah I can to.. From the internal network. It’s when packets come from the WAN interface through the edge router is when it times out.

I can ping the router with a 1500 byte packet from the WAN. The Switch, my servers, and every other non-mikrotik device from the WAN. I can also ping every mikrotik device pre-2.9.xx, I can’t ping 2.9.17 routers with 1500 byte packet from the WAN. I upgraded one of the devices to 2.9.18 and still it wont accept the 1500 byte ping..

I’ll be forwarding this thread to support.

15

This ended up being a client side issue. They installed the Cisco software VPN client on all the host machines and it resolved the issue… My only conclusion to this is that when they installed the VPN client it fixed a problem with the TCP/IP stack or MTU on the local machine. They just installed the client, they didn’t use it to connect to the VPN and then the hardware client started passing traffic normally, so go figure. I think this goes down in the books as a WTF…

16

You’re right, WTF. Glad you got it fixed.

Sorted my TCP encapsulation problems as well. It was a duff policy route configuration on a Cisco 2600.

Regards

Andrew