I'm unable to upgrade RouterOS on MikroTik hardware, as I get bad HTTP response in WinBox. Trying to upgrade from stable version 7.23.2 to version 7.23.3
The check for update never worked for me, during initial setup I downloaded the latest RouterOS to install manually as the internet was disconnected. Previous upgrade failed as well, and I had to manually download the npk package and upload it to the router files via WinBox
I know it's possible to upgrade manually, but I would like to resolve this issue if possible. What I found out:
I'm using stable channel with HTTPS mode and Auto IP version
DNS resolves upgrade.mikrotik.com to 159.148.147.251
Ping upgrade.mikrotik.com works
Tried with IPv4 only and without checking certificates, same result
I'm able to fetch the NPK package using cli fetch command /tool fetch url="https://upgrade.mikrotik.com/routeros/7.23.3/routeros-7.23.3-arm64.npk"
Clock is already synced with NTP Client
Certificates already have ISRG Root X1 + ISRG Root X2 as trusted (the root CA)
That is known to cause issues. Make sure there is an "MTU clamping" rule or the PPPoE interface is setup with a PPP profile that has "Set TCP MSS" configured as Yes.
PPP profile assinged to the PPPoE interface already has Change TCP MSS to yes and I have IPv4 firewall rule under Mangle to clamp MSS to PMTU. Interface has the correct 1492 MTU as well.
Would the fetch tool work if the issue is with MTU?
Well, when the outgoing TCP connect is with an MSS that is too large for 1492 MTU, it is known to fail.
But when the MSS is changed in the PPPoE interface (PPP profile) that should not be the reason.
At the moment the upgrade server is heavily loaded and people report issues, but as you say it never worked that probably is not it.
You can try to setup a "tools -> packet sniffer" session on the pppoe interface with log to file, start capture, attempt the upgrade, stop capture download the file and read in Wireshark.
You don't need the IPv4 mangle rule if you have change-tcp-mss=yes in the PPP profile. But if you have enabled IPv6 on the PPPoE interface, you'll need an IPv6 mangle rule. I had all kinds of issues with IPv6 enabled sites until I added the IPv6 mangle rule.
I didn't know that for IPv4 as I saw increasing packets and kept the rule, I thought it was needed for IPv4. For IPv6 I noticed the MTU issues until I applied it there also.
Now regardless the update seems to fail on both IPv4/IPv6 for me, so I only focused on getting the update via V4 first
I will try the packet sniffer method suggested by pe1chl but it seems the feature is disabled and I need to reboot first
Edit: I've managed to fix this with the help of LLM by adding a clamp rule on IPv4 firewall mangle but on the output chain (the two rules I had earlier were on the forward chain). So it turned out to be MTU issue eventually, thank you @pe1chl
/ip/firewall/mangle/add chain=output protocol=tcp tcp-flags=syn action=change-mss new-mss=clamp-to-pmtu comment="Clamp local router traffic for PPPoE"
Interesting that this fixes it, while the PPP setting for the same thing does not work. I think that has been broken before but it was long ago. I usually disable that and use the mangle rule (both in output and forward chain) instead. Which apparently still works.
MikroTik could help their users by setting a smaller MSS at their end (like 1280) and thereby avoid this kind of issues with PPPoE, VPN, etc.
BTW, you can see if you can get the PPPoE MTU to 1500. Several ISPs support that nowadays and in avoids endless hassle.
I have the exact same problem (bad HTTP response) but have never had an upgrade issue via Winbox until now. I am running 7.24rc2 from the testing channel, which was installed via winbox upgrade. Now I cannot upgrade to 7.24.rc3 because of the problem. It does not matter what channel I pick; they all result in the same problem. My 7 access points are running 7.24rc1 and can upgrade via winbox to rc3. BTW, no PPPoE, IPv6 or default fastrack mangle rules here.
So it seems to be a recently introduced issue, as I said (but cannot easily test that here): the PPP setting that fixes this maybe no longer works and you need that mangle rule.
I added the mangle rules to output and forward chains. Winbox would not let me add it to the input chain. That did not fix the issue, but I got a chance to reboot the router then the issue was gone. I disabled the mangle rules and rebooted, and the issue is still gone. Go figure.
