Weird bug in RB450

Hello!

We have several RB450 (OS v. 3.10) connected to the same network 192.168.1.0/24. One of routers serving as main router between our network and Internet, doing NAT, band shaping, …
Before this we did
/system reset-configuration
for this router, then configured it from scratch.
Now the router working, doing NAT, everything looks ok, EXCEPT:

  • it can NOT ping other MT routers in the same network. It pings all the other network IPs, except MTs.
  • there is NO any kind of access from this routers to other MTs, except mac-address access, so we can NOT do anything from it (telnet,ssh,bandwidth test, etc…).

I was trying to compare router settings with original factory settings, I did “export” for whole system then compared it to similar export text file which was saved before “system reset-configuration”. Then I compared two export files with Linux “diff” tool – I have not found any critical points in the settings.
I’ve spent a lot of time trying to solve this trouble – no luck!

Please help me, asap.
Thank you.

Not much to go on here. Can you be more specific without compromising your security?
What interface are the unpingable MT boxes on, and the IP/netmask?
What interface is connected to the internet? No ip or netmask needed.
The ip/netmask of the MT boxes would help, along with any “/ip route” entries in those boxes.

SurferTim,

ether1 – connected to Internet, ether2 – to our network.
Here are router’s settings ( IPs are changed for security reasons :wink: ):

ip addr pri
Flags: X - disabled, I - invalid, D - dynamic

ADDRESS NETWORK BROADCAST INTERFACE

0 222.11.123.3/24 222.11.123.0 222.11.123.255 ether1
1 192.168.54.3/24 192.168.54.0 192.168.54.255 ether2
2 222.11.123.2/24 222.11.123.0 222.11.123.255 ether1
3 111.22.231.66/26 111.22.231.64 111.22.231.127 ether1

222.11.123.2 and 111.22.231.66 are NATed to the server (192.168.54.254) inside our network connected to (ether2). The server is pingable and accessible as well as some other hardware boxes with IPs 192.168.54.*/24, EXCEPT other Mikrotik routers. :frowning:

ip route pri
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit

DST-ADDRESS PREF-SRC GATEWAY-STATE GATEWAY DISTANCE INTERFACE

0 A S 0.0.0.0/0 reachable 222.11.123.1 1 ether1
1 ADC 192.168.54.0/24 192.168.54.3 0 ether2
2 ADC 111.22.231.64/26 111.22.231.66 0 ether1
3 ADC 222.11.123.0/24 222.11.123.3 0 ether1

ip firewall nat exp
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=222.11.123.3 dst-port=9223
protocol=tcp to-addresses=192.168.54.223 to-ports=80
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=222.11.123.3 dst-port=9253
protocol=tcp to-addresses=192.168.54.253 to-ports=80
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=222.11.123.2 to-addresses=
192.168.54.254 to-ports=0-65535
add action=dst-nat chain=dstnat comment=“” disabled=no dst-address=111.22.231.66 to-addresses=
192.168.54.254 to-ports=0-65535
add action=src-nat chain=srcnat comment=“” disabled=no src-address=192.168.54.254 to-addresses=
222.11.123.2 to-ports=0-65535
add action=src-nat chain=srcnat comment=“” disabled=no src-address=192.168.54.254 to-addresses=
111.22.231.66 to-ports=0-65535
add action=src-nat chain=srcnat comment=“” disabled=no src-address=192.168.54.0/24 to-addresses=
222.11.123.3 to-ports=0-65535

try to ping the neigbour MT router

[admin@RouterXXX] > ping 192.168.54.4
192.168.54.4 ping timeout
192.168.54.4 ping timeout
192.168.54.4 ping timeout
4 packets transmitted, 0 packets received, 100% packet loss

there are 2 more MT routers in the same network, the result for them is the same.. > :frowning:

Let’s try to ping the server

[admin@RouterXXX] > ping 192.168.54.254
192.168.54.254 64 byte ping: ttl=64 time<1 ms
192.168.54.254 64 byte ping: ttl=64 time<1 ms
192.168.54.254 64 byte ping: ttl=64 time<1 ms
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0/0.0/0 ms

Let’s try to ping something else:

[admin@RouterXXX] > ping 192.168.54.223
192.168.54.223 64 byte ping: ttl=64 time=14 ms
192.168.54.223 64 byte ping: ttl=64 time=13 ms
192.168.54.223 64 byte ping: ttl=64 time=13 ms
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 13/13.3/14 ms

All the other kind of hardware can see and ping each other in this net
This is ping test from server to this router and to mentioned above (192.168.54.4):

~# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.54.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
0.0.0.0 192.168.54.3 0.0.0.0 UG 0 0 0 eth0
~# ping 192.168.54.3
PING 192.168.54.3 (192.168.54.3) 56(84) bytes of data.
64 bytes from 192.168.54.3: icmp_seq=1 ttl=64 time=0.538 ms
64 bytes from 192.168.54.3: icmp_seq=2 ttl=64 time=0.406 ms

— 192.168.54.3 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.406/0.472/0.538/0.066 ms

:~# ping 192.168.54.4
PING 192.168.54.4 (192.168.54.4) 56(84) bytes of data.
64 bytes from 192.168.54.4: icmp_seq=1 ttl=64 time=2.05 ms
64 bytes from 192.168.54.4: icmp_seq=2 ttl=64 time=2.39 ms

— 192.168.54.4 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
rtt min/avg/max/mdev = 2.059/2.224/2.390/0.172 ms

This info is from one of neighbours router 192.168.54.4

ip route pri
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit

DST-ADDRESS PREF-SRC GATEWAY-STATE GATEWAY DISTANCE INTERFACE

0 ADC 192.168.54.0/24 192.168.54.4 0 ether2
[admin@RouterZZZ] > ip addr pri
Flags: X - disabled, I - invalid, D - dynamic

ADDRESS NETWORK BROADCAST INTERFACE

0 192.168.54.4/24 192.168.54.0 192.168.54.255 ether2
[admin@RouterZZZ] > ping 192.168.54.3
192.168.54.3 64 byte ping: ttl=64 time=13 ms
192.168.54.3 64 byte ping: ttl=64 time=2 ms
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 2/7.5/13 ms

p.s. the are no more critical settings in this router, as the network is in transitional state.

For now this router serves for bandwith testing purposes inside our net.

Thank you for reading.

That is better! We will try a little at a time, so you will know where the challenge was.

Insure your gateways in the MT boxes (and all others too) on the 192.168.54.x/24 net have a gateway set as 192.168.54.3. Normally, I set the gateway interface to 192.168.54.1/24, but it will work as long as you know that!

Is there anything in
/ip firerwall filter
in the MT boxes that would stop a ping?

Can you show me the “/ip address” and '/ip route" in one of the unpingable MT boxes?

It is responding to some pings from the server:
:~# ping 192.168.54.4
PING 192.168.54.4 (192.168.54.4) 56(84) bytes of data.
64 bytes from 192.168.54.4: icmp_seq=1 ttl=64 time=2.05 ms
64 bytes from 192.168.54.4: icmp_seq=2 ttl=64 time=2.39 ms

It’s in main router (192.168.54.3) (just to prevent ssh and ftp brute force attacks, as in wiki..):

[admin@RouterZZZ] > ip firewall filter pri
Flags: X - disabled, I - invalid, D - dynamic
0 ;;; drop ftp brute forcers
chain=input action=drop src-address-list=ftp_blacklist dst-port=21 protocol=tcp

1 chain=output action=accept protocol=tcp content=530 Login incorrect dst-limit=1/1m,9,dst-address/1m

2 chain=output action=add-dst-to-address-list address-list=ftp_blacklist address-list-timeout=3h protocol=tcp
content=530 Login incorrect

3 ;;; drop ssh brute forcers
chain=input action=drop src-address-list=ssh_blacklist dst-port=22 protocol=tcp

4 chain=input action=add-src-to-address-list connection-state=new src-address-list=ssh_stage3
address-list=ssh_blacklist address-list-timeout=1w3d dst-port=22 protocol=tcp

5 chain=input action=add-src-to-address-list connection-state=new src-address-list=ssh_stage2
address-list=ssh_stage3 address-list-timeout=1m dst-port=22 protocol=tcp

6 chain=input action=add-src-to-address-list connection-state=new src-address-list=ssh_stage1
address-list=ssh_stage2 address-list-timeout=1m dst-port=22 protocol=tcp

7 chain=input action=add-src-to-address-list connection-state=new address-list=ssh_stage1 address-list-timeout=1m
dst-port=22 protocol=tcp

All the rest of routers have nothing in “ip firewall filter”, as they are not accessible from I-net directly.

Pls see previous message for 192.168.54.4. And here is for another one (192.168.54.10):

[admin@RouterYYY] > ip firewall filter pri
Flags: X - disabled, I - invalid, D - dynamic
[admin@RouterYYY] > ip addr pri
Flags: X - disabled, I - invalid, D - dynamic

ADDRESS NETWORK BROADCAST INTERFACE

0 192.168.54.10/24 192.168.54.0 192.168.54.255 ether2
[admin@RouterYYY] > ip route pri
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit

DST-ADDRESS PREF-SRC GATEWAY-STATE GATEWAY DISTANCE INTERFACE

0 ADC 192.168.54.0/24 192.168.54.10


This one is a neighbour, not the router which has trouble(192.168.54.3 – let’s call it “Main” router).
The Main (192.168.54.3 ) is pingable from everywhere, and, as I told, it can ping everything EXCEPT neighbour MT routers ( 192.168.54.{4,10,5…} ). There is another router .5 - has similar settings like .10 …
All the other members of network can ping each other.

I don’t know what you mean by neighbor. that implies peer. If they are on a private net, they are more like “children”. So “Mom” has ether1 on the internet, and ether2 is the 192.168.54.3/24 local net. That is where the “children” servers and routers, including the unpingables, are connected.

The 192.168.54.10/24 address on that “child” router is on ether2. That is where the cable that goes to “Mom” is connected, correct? Not ether1.

Ok. Correct.
Saying “neighbors” I also mean that all these routers are wisible from “Mom”
with Winbox:IP->Neighbors or :

[admin@RouterZZZ] > ip neighbor pri

INTERFACE ADDRESS MAC-ADDRESS IDENTITY VERSION UNPACK

1 ether2 192.168.54.5 00:0C:42:2F:1D:65 RouterNNN 3.10 none
2 ether2 192.168.54.10 00:0C:42:37:15:C8 RouterYYY 3.10 none
3 ether2 192.168.54.4 00:0C:42:2E:B3:E6 RouterXXX 3.10 none

[admin@RouterZZZ] > ping 192.168.54.4 interface=ether2
192.168.54.4 with hw-addr 00:0C:42:2E:B3:E6 ping time=2 ms
192.168.54.4 with hw-addr 00:0C:42:2E:B3:E6 ping time=2 ms
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 2/2.0/2 ms
[admin@RouterZZZ] > ping 192.168.54.4
192.168.54.4 ping timeout
192.168.54.4 ping timeout
3 packets transmitted, 0 packets received, 100% packet loss

They are all pingable from Mom by mac-address, but not by IP.

The 192.168.54.10/24 address on that “child” router is on ether2. That is where the cable that goes to “Mom” is connected, correct? Not ether1.

yes.

So if you tell “Mom” which interface to use, the ping is successful, but if not, it fails?

As you see…
I think, by adding “interface” parameter to ping command it involves MAC ping instead of TCP (…What MT people can tell about this? ).
Anyways, normal ping doesn’t works, as well as all the other services like SSH, Telnet…

MAC ping always works well.

[admin@RouterXXX] > ping 00:0C:42:2E:B3:E6
00:0C:42:2E:B3:E6 64 byte ping time=2 ms
00:0C:42:2E:B3:E6 64 byte ping time=5 ms
00:0C:42:2E:B3:E6 64 byte ping time=5 ms
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 2/4.0/5 ms
[admin@RouterXXX] > ping 192.168.54.4 interface=ether2
192.168.54.4 with hw-addr 00:0C:42:2E:B3:E6 ping time=10 ms
192.168.54.4 with hw-addr 00:0C:42:2E:B3:E6 ping time=2 ms
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 2/6.0/10 ms

My bad! I should have been more specific. Try
ping 192.168.54.4 interface=ether2
from Mom without the mac part, and see if that works. If it does, it is probably a routing challenge in Mom.

Add: And check this on the 192.168.54.10 child:

/ip route pri
# DST-ADDRESS PREF-SRC GATEWAY-STATE GATEWAY DISTANCE INTERFACE
0 ADC 192.168.54.0/24 192.168.54.10

It should look like this:

0 ADC  192.168.54.0/24     192.168.54.10      0     ether2

Insure there is a distance and interface on there.

EDIT: OK. No more dancing. I won’t step on your new post. But I have no problem with that nat rule here.
But as I recall, srcnat rules are only applied on forward chains, not input chains. like this should be. That, along with the inability to ping except by mac address, almost implies the port the cable is connected into is not assigned 192.168.54.3.

Stop all the dancing, my friend. :slight_smile:
I’ve found THE BUG!

I have two spare routers at my office, so I configured similar environment and was testing, playing, etc… Now here is the evil:

/ip firewall nat
add action=src-nat chain=srcnat comment=“” disabled=no src-address=192.168.54.0/24
to-addresses=222.11.123.3

As soon as you remove this one, everything(ping, telnet, ssh, bandwidth testing to another MT box) starts to work like it should. All the rest of NAT rules doesn’t matters.

Well, it is time to hear something about this from MT PEOPLE. Any solutions, workarounds, etc..?

did you tried

/ip firewall nat
add action=src-nat chain=srcnat comment=“” disabled=no > out-interface=ether1 > src-address=192.168.54.0/24
to-addresses=222.11.123.3

?

p.s. MT people anwer on support@mikrotik.com :wink:

yes.

Well, it’s solved now by upgrading to ROS 3.20 and adding the IP of eth2 of “Mom” as a default gw to others (child routers). At least TCP bandwidth test is working well now, but the UDP bandtest works only half-way… And, for example, /sys ssh 192.168.54.4

  • doesn’t work, but for now it doesn’t annoy me much…

OK, one more dance, just for old time’s sake. :smiley:

I would find that more than annoying. It is malfunctioning.
The srcnat rules are still being applied? (implies non-localnet port)
Unable to ping except by mac? (implies non-localnet port)
Needs gateway on what should be a localnet connection? (implies non-localnet port).

“Mom” should only have two cables connected. One in ether1 to the internet. The only other cable should connect to ether2, and go to a hub/switch, then divided up to the localnet devices. That is correct?

oh, yes :slight_smile:

I would find that more than annoying. It is malfunctioning.

I agree.
I don’t know if MT ROS developers consider it as a bug…

The srcnat rules are still being applied? (implies non-localnet port)

yes.

Unable to ping except by mac? (implies non-localnet port)

No. Normal ping is working well as soon as you upgrade to ROS 3.20, then configure 192.168.54.3 as default gateway for a child box. Pls see my reply to Chupaka.

Needs gateway on what should be a localnet connection? (implies non-localnet port).

yes.
..at least it solves my problem for now.

“Mom” should only have > two cables > connected. One in ether1 to the internet. The only other cable should connect to ether2, and go to a hub/switch, then divided up to the localnet devices. That is correct?

yes. Correct.

Here are the steps to reconstruct the situation simple way (I don’t know if the problem will appear for the others MT models):

Connect RB450 by eth1 to some network (your local network, let’s say
192.168.1.0/24), then

/ip address
add address=192.168.1.11/24 interface=ether1
add address=192.168.54.3/24 interface=ether2

  • connect eth2 to a switch

  • connect to the same switch one more (second) RB450 with an interface having addr
    say 192.168.54.5 (the same subnet 192.168.54.0/24)

  • connect to the same switch a PC or any non-Mikrotik box which could be able to respond ping.
    Use the same subnet 192.168.54.0/24 for the interface of this PC, say IP: 192.168.54.6 .

  • log in to console of first RB450 and try:
    ping 192.168.54.5
    ping 192.168.54.6

  • everything works OK.

  • Now, to the first RB450 router add this rule
    /ip firewall nat
    add action=src-nat chain=srcnat src-address=192.168.54.0/24 to-addresses=192.168.1.11

now try:
ping 192.168.54.5 (2nd RB450) – it won’t work!!
ping 192.168.54.6 (PC) – it works OK.

If you are certain of the cable placement, then the “non-localnet” port must be on the other end. Can you please show me your “/ip route print” from one of the child boxes? Pick one of them, so we can get the bugs out of it.

[admin@RouterXXX] > ip route pri
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit

DST-ADDRESS PREF-SRC GATEWAY-STATE GATEWAY DISTANCE INTERFACE

0 A S 0.0.0.0/0 reachable 192.168.54.3 1 bridge1
1 ADC 192.168.54.0/24 192.168.54.5

That is what I thought. If we would have continued that last dance, this would probably be solved now. This is what you need:
/ip route
set 1 distance=0 interface=ether2

If ether2 is not correct, change it to the interface assigned the ip/netmask. Without that, it does not know what interface to use.

EDIT: I did not see the bridge before. “/int bridge print” and “/int bridge port print” might help.
I see the ip 192.168.54.5 as the ip of that interface. That is the box we will stay with until working.

deleted

hmm, for me it looks at least strange.
Why do we need to mention these settings..? Well, I understand that it might be a workaround to solve the problem, but anyway it looks weird! In that route output of .54.5 (rule 1) it does not mention distance at all. Logically it does mean that the distance is the minimal (eq 0)… The interface is also should be applied correctly by default as there are no more other IPs or interfaces configured or involved. These routes the system configures automatically by default as soon as you add new IP addresses. Normally you don’t need to do something special about this. It should work “as is”.
I think it would be not bad to try to use packet sniffer to see what kind of information the “Mom” sends in the headers of packets, but MT ROS native tools are not yet comfortable for me (as for example tcpdump of Linux). I will play with it as soon as I’ll have time. I feel the answer should be in packet headers…

EDIT: I did not see the bridge before. “/int bridge print” and “/int bridge port print” might help.
I see the ip 192.168.54.5 as the ip of that interface. That is the box we will stay with until working.

I didn’t use bridges.