Cannot Use Multiple IPs

tb704 · Thu Feb 25, 2021 1:50 am

I am having issues using multiple IPs with my ISP. They have provided me a /29 subnet with 8 useable IPs. I have all 8 IPs set up on my WAN port (ethernet 8). With nothing else configured, I can ping using the first two IPs, while the last 3 time out. I connected directly to the demarc and was able to ping using my laptop on all 5 IPs. The ISP told me it was a misconfiguration with my Mikrotik and that I was passing my internal VLAN traffic to the internet. I figured if that were happening, I wouldn't get any IP to work as they'd be expecting my traffic to be untagged, when in reality it is tagged. Furthermore, there aren't any VLANs configured on the WAN port.

To prove this I wiped the config from my Mikrotik and configured my WAN with the IPs and a gateway. No VLANs, no security, no nothing. I ran the same ping tests and got the same result; the first two IPs work fine, the final three time out. I bypassed the router one last time, configured each IP on my laptop, and all 5 worked without any issue. Can anyone see any reason why this would be happening?

Here is my config with the IPs mostly removed. As you can see, I left it as vanilla as I could manage to, it doesn't even have the right date and time. When not testing, I have it set up to pull NTP time.

[admin@MikroTik] > /export
# jan/02/1970 00:06:12 by RouterOS 6.45.3
# software id = HT9Y-4QA6
#
# model = CCR1036-8G-2S+
# serial number = 5AAD02D2FDAA
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip address
add address=192.168.88.1/24 comment=defconf interface=ether1 network=192.168.88.0
add address=XXX.XXX.XXX.250/29 interface=ether8 network=XXX.XXX.XXX.248
add address=XXX.XXX.XXX.251/29 interface=ether8 network=XXX.XXX.XXX.248
add address=XXX.XXX.XXX.252/29 interface=ether8 network=XXX.XXX.XXX.248
add address=XXX.XXX.XXX.253/29 interface=ether8 network=XXX.XXX.XXX.248
add address=XXX.XXX.XXX.254/29 interface=ether8 network=XXX.XXX.XXX.248
/ip route
add distance=1 gateway=XXX.XXX.XXX.249


[admin@MikroTik] > ping count=5 src-address=XXX.XXX.XXX.250 8.8.8.8
  SEQ HOST                                     SIZE TTL TIME  STATUS                                                                                                                                                                                       
    0 8.8.8.8                                    56 117 7ms  
    1 8.8.8.8                                    56 117 6ms  
    2 8.8.8.8                                    56 117 6ms  
    3 8.8.8.8                                    56 117 6ms  
    4 8.8.8.8                                    56 117 6ms  
    sent=5 received=5 packet-loss=0% min-rtt=6ms avg-rtt=6ms max-rtt=7ms 

[admin@MikroTik] > ping count=5 src-address=XXX.XXX.XXX.251 8.8.8.8 
  SEQ HOST                                     SIZE TTL TIME  STATUS                                                                                                                                                                                       
    0 8.8.8.8                                    56 117 6ms  
    1 8.8.8.8                                    56 117 6ms  
    2 8.8.8.8                                    56 117 6ms  
    3 8.8.8.8                                    56 117 6ms  
    4 8.8.8.8                                    56 117 6ms  
    sent=5 received=5 packet-loss=0% min-rtt=6ms avg-rtt=6ms max-rtt=6ms 

[admin@MikroTik] > ping count=5 src-address=XXX.XXX.XXX.252 8.8.8.8 
  SEQ HOST                                     SIZE TTL TIME  STATUS                                                                                                                                                                                       
    0 8.8.8.8                                                 timeout                                                                                                                                                                                      
    1 8.8.8.8                                                 timeout                                                                                                                                                                                      
    2 8.8.8.8                                                 timeout                                                                                                                                                                                      
    3 8.8.8.8                                                 timeout                                                                                                                                                                                      
    4 8.8.8.8                                                 timeout                                                                                                                                                                                      
    sent=5 received=0 packet-loss=100% 

[admin@MikroTik] > ping count=5 src-address=XXX.XXX.XXX.253 8.8.8.8 
  SEQ HOST                                     SIZE TTL TIME  STATUS                                                                                                                                                                                       
    0 8.8.8.8                                                 timeout                                                                                                                                                                                      
    1 8.8.8.8                                                 timeout                                                                                                                                                                                      
    2 8.8.8.8                                                 timeout                                                                                                                                                                                      
    3 8.8.8.8                                                 timeout                                                                                                                                                                                      
    4 8.8.8.8                                                 timeout                                                                                                                                                                                      
    sent=5 received=0 packet-loss=100% 
[admin@MikroTik] > ping count=5 src-address=XXX.XXX.XXX.254 8.8.8.8
  SEQ HOST                                     SIZE TTL TIME  STATUS                                                                                                                                                                                       
    0 8.8.8.8                                                 timeout                                                                                                                                                                                      
    1 8.8.8.8                                                 timeout                                                                                                                                                                                      
    2 8.8.8.8                                                 timeout                                                                                                                                                                                      
    3 8.8.8.8                                                 timeout                                                                                                                                                                                      
    4 8.8.8.8                                                 timeout                                                                                                                                                                                      
    sent=5 received=0 packet-loss=100%

satman1w · Thu Feb 25, 2021 1:37 pm

Is it possible that the subnet given by ISP is not /29 but /30 ?

sindy · Thu Feb 25, 2021 5:49 pm

You say you've tried the addresses one by one with the PC but all 5 were configured simultaneously at the Tik.

Can you try to configure just one of the malfunctioning ones on the Tik and try again, while sniffing at ether8?

Make the command line window as wide as your screen allows. Then run
/tool sniffer set file-name=ether8.pcap
/tool sniffer quick interface=ether8

Now ping from that single malfunctioning address (in another window) and then stop the /tool sniffer quick ....

If you cannot see what happens from the text output of /tool sniffer quick ..., download the ether8.pcap and open it using Wireshark.

If it eventually works that way, configure all 5 and try the same again for each of them.

@satman1w, I don't think they have a /30 mask at their side as in such case, even the .251 should have problems, and the failure would have to happen on the PC as well.

satman1w · Thu Feb 25, 2021 6:07 pm

@satman1w, I don't think they have a /30 mask at their side as in such case, even the .251 should have problems, and the failure would have to happen on the PC as well.

No, it should not... If you try to ping "internet" from your broadcast address it works!
...and this looks exactly like the /30 subnet at the ISP..
I would try the traceroute to all of the IP adressess in the range (from outside) and I suspect that last 3 would show different route path...

:-)

sindy · Thu Feb 25, 2021 7:26 pm

If you try to ping "internet" from your broadcast address it works!

You've made me investigate :)

The thing is I do remember a recent issue where a colleague has made an mistake when calculating the mask and has set a broadcast address as the own one on the Tik, and the Tik didn't respond to incoming ARP requests to this broadcast address. So it wasn't clear to me how the ping responses sent from a broadcast address could arrive, although it's actually an inverse scenario.

So I've first done the same like in that older case (set 10.30.50.1/24 at one device and 10.30.50.2/28 on another one), and tried to ping 10.30.50.15 from the first one while sniffing at the second one; as expected, the ARP requests were coming but no responses were leaving. When I pinged the 10.30.50.2, the ARP response was there, so not a sniffing problem.

But yes, you are right that you do get responses if you ask ping to use a broadcast address in your connected network as src-address, but it's actually because the RouterOS ping silently ignores that parameter (whereas if you ask it to use some host address from a connected subnet which is hovewer none of its own ones, the ping says "could not make socket").

Everything above doesn't depend on whether the broadcast address is actually assigned as an own one or not.

Next, I have tried the reverse, matching your suspected scenario: I've changed the mask to /28 at the first device and to /24 on the second, and started pinging 8.8.8.8 from 10.30.50.15 on the second one. In this case, the first device did not attempt to send an ARP request for the address it considered a broadcast one.

I guess the behaviour may differ per vendor, so it is still possible that if the ISP has a /30 mask there, it does send an ARP request, or learns the IP to MAC binding from the incoming packet, so pings from .251 can be responded, but it still doesn't explain why .252 to .254 also get ping responses if they are up on a PC.

satman1w · Thu Feb 25, 2021 9:15 pm

If you try to ping "internet" from your broadcast address it works!
You've made me investigate :)

Nice job :-)

I was preparing to do the same... just for fun, but now... you ruined it :-D

anyway, the traceroute test will give the right answer, don't you think?

regards

sindy · Thu Feb 25, 2021 9:24 pm

It may or may not. If both the suspected /30 subnets are on the same device at ISP side, the traceroute from outside to both will show the address of the internet-facing interface of that device for both target subnets.

joegoldman · Thu Feb 25, 2021 11:04 pm

No, it should not... If you try to ping "internet" from your broadcast address it works!

Hrmm - can depend on vendor - some simply won't pass traffic to the defined network or broadcast address when actually configured as that subnet.

If x.x.x.248/29 was actually routed via an interconnect, then yes the whole /29 would be usable (including 'network' and 'broadcast' addresses) as long as you configured them correctly - but if the ISP router is specifically holding x.x.x.249/29 (or suspected in this case, .249/30) then it is entirely possible it won't accept ARP for broadcast address and know to send it to OP's router for response. The OP does not have a /29 routed to them, they are being handed-off within a /29 which is a very big difference.

tb704 · Fri Feb 26, 2021 2:48 am

Going to try to answer in the order I was asked.

It is not a /30 subnet. I neglected to mention that I work for the ISP and can look at how this subnet is configured in our router. The mask on the VLAN is 255.255.255.248. But if it were a /30 none of the IPs would work due to mask mismatches.

I did try to run pings to 8.8.8.8 with packet sniffer running. What I noticed is that using .252 .253 or .254 sees no response back. So I ran a ping test to my gateway .249 and got the same results. Only .250 and .251 saw responses from the gateway.

When running traceroute using .250 or .251 as the source address, it shows my default gateway .249 being the first hop. When using any of the other three IPs, that first hop times out. This was my HA it has to be misconfigured in the ISP router, but if that were the case, it wouldn't have worked directly from the PC.

As far as using only one IP at a time, I kind of can't right now. The way my NAT is set up, it'll kill the internet completely and my wife and kids will go ballistic. If I can manage to, I will try this suggestion after everyone is asleep.

As far as your discussion on pinging broadcast traffic, that should always work as long as SOMETHING responds. So if i ping using the .255 IP (which is still part of my subnet), it works just fine.

tb704 · Fri Feb 26, 2021 3:01 am

So for the first time in recorded history using Winbox fixed it. I went into Winbox and disabled all of the IPs it the IP list. I then enable and tested them individually and they all worked. I turned all 5 back on and now they're working. This is the strangest behavior I've ever seen.

satman1w · Fri Feb 26, 2021 11:53 am

When running traceroute using .250 or .251 as the source address, it shows my default gateway .249 being the first hop.

... the idea was to trace route to your public subnet from internet, not from your router, but it does not matter now..

accarda · Tue Mar 23, 2021 3:30 pm

I was reading this post and I’m wondering whether one issue that I have with an IP address availability could be explained by the exchange here.
My case is different, in the sense that I’ve got a /29 from ISP, which I haven’t assigned to a LAN interface, but instead I’m setting things so that I can use all 8 IPs with DST-NAT and SRC-NAT for the internal hosts which need to be reached from outside on specific IP.
As of now I have assigned all those addr to a loopback interface and when I ping each of them from outside, I can ping only the first 7 but not the last one.
Also using traceroute on the 8th addr I can reach until it gets to the data center with the same path as for the other 7, but then it goes in timeout.

Could be this some issue related to how that address is handled and considered the broadcast addr for that /29 subnet ?
Thanks in advance for any opinion that you can share.

sindy · Tue Mar 23, 2021 3:47 pm

That's strange. If they had it as a local subnet on one of their interfaces, the network address (.0,.8, etc. depending on the prefix) and one of the other addresses should also not reach your router.

I assume you've assigned those addresses as individual /32 ones to the loopback interface, correct?

accarda · Tue Mar 23, 2021 4:14 pm

Yes,
your assumption is correct, I have assigned them as /32 and I have all listed from x.x.x.24 to x.x.x.31.

Actually I tried again this morning few hours ago, and the address .31 didn't work at all.
At this very moment I tried from the router to make a ping to 8.8.8.8 an using source addr the .31 and the outgoing ping went fine.
Now I have tried from an external CHR to ping directly .31 and it's fine (it was not few hours ago); now I can also complete traceroute.
This morning I raised the question to the ISP, so I guess by now they must have done something.

Like for the OP, it looks like by posting here on this thread creates some magic bonding and the issue got fixed by itself.
Thanks again for your time though.
Armando

Cannot Use Multiple IPs

Cannot Use Multiple IPs

Re: Cannot Use Multiple IPs

Re: Cannot Use Multiple IPs

Re: Cannot Use Multiple IPs

Re: Cannot Use Multiple IPs

Re: Cannot Use Multiple IPs

Re: Cannot Use Multiple IPs

Re: Cannot Use Multiple IPs

Re: Cannot Use Multiple IPs

Re: Cannot Use Multiple IPs

Re: Cannot Use Multiple IPs

Re: Cannot Use Multiple IPs

Re: Cannot Use Multiple IPs

Re: Cannot Use Multiple IPs

Who is online