IPv6 issues via HE tunnel

halacs · Sun Dec 01, 2019 12:04 pm

Hi,

I have a CCR1009-7G-1C-1S+ router with HE IPv6 tunnel. Ping6 works fine, but seemingly nothing else.
Do you have any idea what could be the problem? I could setup successfully HE tunnel in my other flat.
Full router config can be found here: https://gist.github.com/halacs/7194ff37 ... e7ffbfe73e

ICMPv6 is allowed in IPv6 firewall.

I have read that HTTPS issues usually happens in case of wrong MTU settings so this was my first suspect. Previously I had IPv4 MTU issues causing HTTPS connection problems, but I have solved it before.

Might useful info:
- HE tunnel MTU is 1280.
- I have a EoIP tunnel between my two flats. IPv6 works fine in flat "B" but in flat "A".
- I have PPPoE internet connection in flat "A" where IPv6 problem happens. PPPoE MTU is 1480 and I couldn't set higher.

fflo · Mon Dec 02, 2019 1:27 am

Did you check that protocol 41 is not blocked in-transit (in- and outbound to HE)?
Have you cross-checked the IPv4 addresses to be static on both ends?

halacs · Mon Dec 02, 2019 11:11 am

If protocol 41 is blocked then no IPv6 ping should work.
IPv4 addresses at both end of the tunnel should be also fine as IPv6 ping works fine. If I disable the tunnel interface no ping anymore.
Please correct me if I'm wrong.

Anumrak · Thu Dec 12, 2019 4:26 pm

Hey. You should advertise your IPv6 /64 prefixes in your LAN. And in IPv6 - ND you should enable advertise mac address and DNS. Also you should write ipv6 dns servers in ip - dns. You don't have them either.

halacs · Thu Dec 12, 2019 4:36 pm

ah, sorry, it seems I attached configuration when I really disabled IPv6 address advertisement. IPv6 advertisement was on on the vlan-local interface where IPv6 address itself is now disabled to avoid full internet outage at the clients.
I have tried to add IPv6 DNS servers and enable MAC and DNS advertisements in ND, but I will double test it. Thanks for the hint! Lets see!

Anumrak · Fri Dec 13, 2019 9:23 am

ah, sorry, it seems I attached configuration when I really disabled IPv6 address advertisement. IPv6 advertisement was on on the vlan-local interface where IPv6 address itself is now disabled to avoid full internet outage at the clients.
I have tried to add IPv6 DNS servers and enable MAC and DNS advertisements in ND, but I will double test it. Thanks for the hint! Lets see!

Sure! Hope you will find the issue.

halacs · Sat Dec 14, 2019 9:21 am

Well, it didn't help.
I have set what you suggested but problem still exists: ipv6 connection is slow and not all the https pages are loading.
Downloading google.com on ipv6 is wrong, but on ipv4: curl -6 -v -L google.com is stuck at https handshake but curl -4 -v -L google.com quickly gives back the html code.

$ ping6 google.com
PING google.com(bud02s27-in-x0e.1e100.net (2a00:1450:400d:804::200e)) 56 data bytes
64 bytes from bud02s27-in-x0e.1e100.net (2a00:1450:400d:804::200e): icmp_seq=1 ttl=57 time=14.7 ms
64 bytes from bud02s27-in-x0e.1e100.net (2a00:1450:400d:804::200e): icmp_seq=2 ttl=57 time=20.3 ms
64 bytes from bud02s27-in-x0e.1e100.net (2a00:1450:400d:804::200e): icmp_seq=3 ttl=57 time=16.8 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.786/17.346/20.376/2.306 ms

$ curl -6 ipv6-test.com -vL
* Rebuilt URL to: ipv6-test.com/
*   Trying 2001:41d0:701:1100::29c8...
* TCP_NODELAY set
* Connected to ipv6-test.com (2001:41d0:701:1100::29c8) port 80 (#0)
> GET / HTTP/1.1
> Host: ipv6-test.com
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Date: Sat, 14 Dec 2019 07:11:20 GMT
< Server: Apache/2.4.25 (Debian)
< Location: https://ipv6-test.com/
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host ipv6-test.com left intact
* Issue another request to this URL: 'https://ipv6-test.com/'
* Connection 0 seems to be dead!
* Closing connection 0
*   Trying 2001:41d0:701:1100::29c8...
* TCP_NODELAY set
* Connected to ipv6-test.com (2001:41d0:701:1100::29c8) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
^C

halacs · Sat Dec 14, 2019 9:26 am

Auch, I have pasted wrong curl call. Curl with google.com is not working even in case of http.

$ time curl -6 -v -L google.com
* Rebuilt URL to: google.com/
*   Trying 2a00:1450:400d:808::200e...
* TCP_NODELAY set
* Connected to google.com (2a00:1450:400d:808::200e) port 80 (#0)
> GET / HTTP/1.1
> Host: google.com
> User-Agent: curl/7.58.0
> Accept: */*
>
^C

real    0m29.183s
user    0m0.005s
sys     0m0.009s

pe1chl · Sat Dec 14, 2019 5:46 pm

When your MTU is not 1500 you should enter the correct MTU in the ND settings or else you will have problems like this.
It is better to make a native IPv6 connection instead of such a tunnel.

halacs · Sat Dec 14, 2019 6:15 pm

Thanks for the ideas!

I have changed MTU in ND as showed on the below picture but didn't help, even after a reboot on the client PC side then on the router side as well.

Yes, native IPv6 would be better but as a Hungarian Telecom subscriber I can get native IPv6 address only via zero configuration. My Windows PC can get IPv6 address directly from the modem but I could not configure my Mikrotik router to do so.

pe1chl · Sat Dec 14, 2019 6:30 pm

You could try to make a packet sniffer trace on the tunnel and see if the TCP SYN packets contain the correct MSS value.

halacs · Sat Dec 14, 2019 6:58 pm

I started a packet sniffer on the tunnel interface.

I can see a TCP SYN MSS=1220 packet then a TCP SYN ACK mss=1220 (between my PC and google.com)
This seems to be good if 1280 is the MTU on the tunnel. Right?

What is interesting for me is the TCP Retransmission part.

pe1chl · Sat Dec 14, 2019 7:13 pm

I started a packet sniffer on the tunnel interface.

I can see a TCP SYN MSS=1220 packet then a TCP SYN ACK mss=1220 (between my PC and google.com)
This seems to be good if 1280 is the MTU on the tunnel. Right?

Yes, this is what you want to see.

What is interesting for me is the TCP Retransmission part.

That is not good! The other end apparently does not respect the MSS, or the datapackets are dropped for some other reason.

halacs · Sat Dec 14, 2019 7:16 pm

So this is the point when I have to contact the tunnel support. Okay, thanks. I will contact them and come back to share the solution.

halacs · Sun Dec 15, 2019 10:50 am

I got answer from the support:

We don't have any firewall rules on our side configured for any tunnel, except SMTP and IRC filters as exist by default.
-- -H U R R I C A N E - E L E C T R I C-

pe1chl · Sun Dec 15, 2019 11:55 am

I have no idea. MTU 1280 on the IPv6 interface should be low enough to pass the largest packet through a tunnel even with the limited MTU of 1480 you have.
That is why I suggested that the other side may ignore the MSS and still send full-size (1500 byte) packets which do not fit the 1480 byte internet MTU.
I have lots of experience with different types of tunnels in combination with such limited MTU sizes imposed by PPPoE and it always is a headache, but lowering the MSS normally solves it for TCP.
No idea why it still does not work for you.

halacs · Sun Dec 15, 2019 11:58 am

Thanks a lot pe1chl!

Anumrak · Mon Dec 16, 2019 12:36 pm

Hey again. Did you try to connect to another web sites?

halacs · Mon Dec 16, 2019 1:14 pm

Hi,

yes, sure! I pase few other tests below.

I have an Ubuntu server with working ipv6 somewhere else. I have curl-ed it while tcpdump was writing pcap file. Here I also found "TCP Retransmission" lines.
In contrast with the previous sniff what happened locally in my Mikrotik router, SYN ACK MSS is not as expected:
SYN MSS=1220
SYN ACK MSS=1440

gabor@home:~$ time curl -vL6 index.hu
* Rebuilt URL to: index.hu/
*   Trying 2a02:730::1860...
* TCP_NODELAY set
* Connected to index.hu (2a02:730::1860) port 80 (#0)
> GET / HTTP/1.1
> Host: index.hu
> User-Agent: curl/7.58.0
> Accept: */*
>
^C

real    0m40.441s
user    0m0.007s
sys     0m0.007s
130 gabor@home:~$ time curl -vL6 origo.hu
* Rebuilt URL to: origo.hu/
*   Trying 2001:4c48:16:6::2:1c...
* TCP_NODELAY set
* Connected to origo.hu (2001:4c48:16:6::2:1c) port 80 (#0)
> GET / HTTP/1.1
> Host: origo.hu
> User-Agent: curl/7.58.0
> Accept: */*
>
^C

 gabor@home:~$ time curl -vL6 forum.mikrotik.com
* Rebuilt URL to: forum.mikrotik.com/
*   Trying 2a02:610:7501:1000::205...
* TCP_NODELAY set
* Connected to forum.mikrotik.com (2a02:610:7501:1000::205) port 80 (#0)
> GET / HTTP/1.1
> Host: forum.mikrotik.com
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: nginx
< Date: Mon, 16 Dec 2019 10:59:57 GMT
< Content-Type: text/html
< Content-Length: 178
< Connection: keep-alive
< Keep-Alive: timeout=5
< Location: https://forum.mikrotik.com/
< Strict-Transport-Security: max-age=31536000
<
* Ignoring the response-body
* Connection #0 to host forum.mikrotik.com left intact
* Issue another request to this URL: 'https://forum.mikrotik.com/'
* Connection 0 seems to be dead!
* Closing connection 0
*   Trying 2a02:610:7501:1000::205...
* TCP_NODELAY set
* Connected to forum.mikrotik.com (2a02:610:7501:1000::205) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
^C

real    0m25.049s
user    0m0.028s
sys     0m0.000s

130 gabor@home:~$ ping ipv6.google.com
PING ipv6.google.com(bud02s26-in-x0e.1e100.net (2a00:1450:400d:809::200e)) 56 data bytes
64 bytes from bud02s26-in-x0e.1e100.net (2a00:1450:400d:809::200e): icmp_seq=1 ttl=57 time=15.7 ms
64 bytes from bud02s26-in-x0e.1e100.net (2a00:1450:400d:809::200e): icmp_seq=2 ttl=57 time=15.3 ms
^C
--- ipv6.google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 15.332/15.530/15.729/0.234 ms

gabor@home:~$ ping forum.mikrotik.com
PING forum.mikrotik.com(forum.mikrotik.com (2a02:610:7501:1000::205)) 56 data bytes
^C
--- forum.mikrotik.com ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6130ms

Anumrak · Mon Dec 16, 2019 3:12 pm

I will try to catch my tcp exchange today on normal web serfing via ipv6, but for now I think it is high delay between ack segments in your exchange. Try to sniff that on ipv4(if your server has ipv4), will you find the difference? Also you have spur retransmissions, which means that you've already saw that ack segment from server, but server received your first ack too late - delayed and server had to send it again.

What mtu and tcp mss do you have in your server config?

halacs · Mon Dec 16, 2019 5:22 pm

On my remote Ubuntu server, I have never changed MTU and MSS.

$ cat /sys/class/net/ens3/mtu
1500

ping looks good on IPv4. Curl gets back very fast as expected. On the picture you can see how I curl-ing my remote server. There an 1460 MSS in the SYN and 1440 in the SYN ACK.

ping [my-remote-ubuntu]
PING [my-remote-ubuntu] 56(84) bytes of data.
64 bytes from [my-remote-ubuntu]: icmp_seq=1 ttl=55 time=40.1 ms
64 bytes from [my-remote-ubuntu]: icmp_seq=2 ttl=55 time=14.1 ms
64 bytes from [my-remote-ubuntu]: icmp_seq=3 ttl=55 time=52.1 ms
64 bytes from [my-remote-ubuntu]: icmp_seq=4 ttl=55 time=16.6 ms
64 bytes from [my-remote-ubuntu]: icmp_seq=5 ttl=55 time=13.0 ms
64 bytes from [my-remote-ubuntu]: icmp_seq=6 ttl=55 time=13.1 ms
64 bytes from [my-remote-ubuntu]: icmp_seq=7 ttl=55 time=13.2 ms
^C
--- [my-remote-ubuntu] ping statistics ---
7 packets transmitted, 7 received, 0% packet loss, time 6007ms
rtt min/avg/max/mdev = 13.007/23.221/52.171/14.918 ms

pe1chl · Mon Dec 16, 2019 5:26 pm

There an 1460 MSS in the SYN and 1440 in the SYN ACK.

That (the 1460 in the SYN) must be a bug in your client!

halacs · Mon Dec 16, 2019 5:40 pm

hmm, why? What should I change in my Mikrotik?
I have a PPPoE Internet connection with 1480 MTU.

pe1chl · Mon Dec 16, 2019 5:53 pm

The client should not send MSS=1460 when it has a MTU of 1500. That is a bug. 1440 is OK.
This is not something you can change in the MikroTik.

For a PPPoE internet connection with 1480 byte MTU the MSS should be 1420 (1480-60).
(assuming native IPv6)

halacs · Mon Dec 16, 2019 6:08 pm

Okay, so it is a bug in Ubuntu 18.04.3 LTS or Curl 7.58.0 released on 2018-01-24.

If I know correctly, if remote server responds smaller MSS, that smaller MSS will be used in the connection. This is why I have internet at all. Right? But what about the 6to4 tunnel?
Plus, if my PC sends too big MSS, should I be able to override it with a "change MSS" action in Firewall \ Mangle?

pe1chl · Mon Dec 16, 2019 6:43 pm

The MSS is negotiated independently for each direction. It is only used to tell the other side what is the max size of the segments you expect.
Sending a too-large value can cause problems e.g. when the other side locally is on a jumbo-size network (MTU 9000) and obeys your request to send 1460-byte segments which require a 1520-byte MTU.
Wen Ubuntu really does that, without setting a >1500 byte MTU, I would say it is a bug yes. You can report it.
(I use Debian Buster and it does not exhibit this behavior, so it may already be fixed in a later Ubuntu release)

Of course you can clamp the MSS value in the MikroTik, yes. Maybe it helps (I don't think so because when you set the ND MTU to 1280 before it did not work either)

halacs · Mon Dec 16, 2019 7:03 pm

In the meanwhile I tried to clamp MSS both in IPv6 firewall/mangle and IPv4 firewall / mangle. It didn't help doesn't matter how small I set (e.g. 1100). You were right.

I'm wondering then why even my Windows 7 PC doesn't work if it is a linux bug.

Anumrak · Mon Dec 16, 2019 11:04 pm

Hey folks. So, answer of remote side about 1220 MSS is correct. First syn of 1440 is correct for ipv6. And I have no such huge amount of retransmissions like you. So you need to try your connections to other ipv6 resources as much closest to you as you can.

If your client sends tcp syn with 1440 - it means that it's interface has ip mtu of 1500 and it's adjusting to 1220 because of ipv6 mtu of 1280 on it's way, it's ok. And 1460 MSS on syn should be a bug... Which write ipv4 MSS in TCP segment of ipv6 header, I believe(1500-20-20).

halacs · Mon Dec 16, 2019 11:21 pm

LAN interface MTU is 1500, true. In IPv6, ND MTU is set to 1280 for all interfaces.

I'm trying to connect servers within Hungary such as the biggest news portal (6ms ping on IPv4, 0% packet loss).

Thanks for your effort!

Anumrak · Tue Dec 17, 2019 12:27 pm

LAN interface MTU is 1500, true. In IPv6, ND MTU is set to 1280 for all interfaces.

I'm trying to connect servers within Hungary such as the biggest news portal (6ms ping on IPv4, 0% packet loss).

Thanks for your effort!

In order to try it to others with ipv6 connectivity, you can print here its dns A record.

IPv6 issues via HE tunnel

IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Re: IPv6 issues via HE tunnel

Who is online