problems routing

ejay · March 28, 2011, 8:22pm

I have layer 2 switches configured for 3 vlans. I have a single RB750G.

I have a ISP modem connected to wan port, and it gets its IP via DHCP client.

vlan 1 is just the switches.
vlan 2 is supposed to have my fast ethernet devices, and get their IPs by the DHCP server.
vlan 3 are my gige machines.

Routing between vlans is accomplished by OSPF.

The RB750G was configured for port 4 no-ip, and 3 vlans. (1-3) OSPF, and DHCP server configured for the vlan 2 interface. On the L2 switch side port 4 is on a trunk port, and uses tagged traffic only.

port 4 = no-ip
vlan 1 = 10.0.1.254
vlan 2 = 10.1.0.254 = dhcp service
vlan 3 = 10.2.0.254
port 5 = addy by DHCP client/nat

Everything sometimes worked, but other stuff stopped working? So I moved vlan 2 to another port, tagged/trunked.

port 3 = no-ip
vlan 2 = 10.1.0.254 = dhcp service
port 4 = no-ip
vlan 1 = 10.0.1.254
vlan 3 = 10.2.0.254
port 5 = addy by DHCP client/nat

Now all members of vlan 3 can route to the Internet. I moved my laptop back to vlan 2, and service is spotty? It sometimes sees the Interenet. DHCP server is working. When it works web is slow on vlan 2? Vlan 3 web is normal, and responsive? Well something just came up, but it took a long time to bring up a web page?

fewi · March 28, 2011, 8:29pm

The RB750G by default has a switch chip turned on that bridges port 2-5 in hardware. For your set up that would have to be turned off. Did you turn it off? To do so, you’d set the master-port of interfaces 3-5 to ‘none’. At that point all ports are simple routed ports that can be used as you intend.

If you need further assistance post the output of “/ip address print detail”, “/interface print”, “/ip route print”, “/interface vlan print detail”, and “/ip firewall export”.

ejay · March 28, 2011, 8:33pm

Yeah it’s no master port. It won’t work at all if that is on. If I give a page about 20min it does come up?

ejay · March 28, 2011, 8:46pm

/ip address print detail
Flags: X - disabled, I - invalid, D - dynamic
0 address=10.1.0.2/16 network=10.1.0.0 broadcast=10.1.255.255
interface=ether4-local-master actual-interface=ether4-local-master

1 address=10.0.1.3/16 network=10.0.0.0 broadcast=10.0.255.255
interface=loopback actual-interface=loopback

2 address=10.0.1.254/16 network=10.0.0.0 broadcast=10.0.255.255
interface=vlan1 actual-interface=vlan1

3 address=10.1.0.254/16 network=10.1.0.0 broadcast=10.1.255.255
interface=vlan2 actual-interface=vlan2

4 address=10.2.0.254/16 network=10.2.0.0 broadcast=10.2.255.255
interface=vlan3 actual-interface=vlan3

5 D address=192.168.1.64/24 network=192.168.1.0 broadcast=192.168.1.255
interface=ether5-gateway actual-interface=ether5-gateway

/interface print
Flags: D - dynamic, X - disabled, R - running, S - slave

NAME TYPE MTU L2MTU

0 ether1-local-slave ether 1500 1524
1 R ether2-local-slave ether 1500 1524
2 R ether3-local-failover ether 1500 1524
3 ether4-local-master ether 1500 1524
4 R ether5-gateway ether 1500 1524
5 R loopback bridge 1500 65535
6 R vlan1 vlan 1500 1520
7 R vlan3 vlan 1500 1520
8 R vlan2 vlan 1500 1520

/ip route print
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit

DST-ADDRESS PREF-SRC GATEWAY DISTANCE

0 ADS 0.0.0.0/0 192.168.1.254 0
1 ADC 10.0.0.0/16 10.0.1.3 loopback 0
vlan1
2 ADC 10.1.0.0/16 10.1.0.2 ether4-local-ma... 0
vlan2
3 ADC 10.2.0.0/16 10.2.0.254 vlan3 0
4 ADC 192.168.1.0/24 192.168.1.64 ether5-gateway 0

/interface vlan print detail
Flags: X - disabled, R - running, S - slave
0 R name="vlan1" mtu=1500 l2mtu=1520 mac-address=00:0C:42:70:2F:8C
arp=enabled vlan-id=1 interface=ether3-local-failover
use-service-tag=no

1 R name="vlan3" mtu=1500 l2mtu=1520 mac-address=00:0C:42:70:2F:8C
arp=enabled vlan-id=3 interface=ether3-local-failover
use-service-tag=no

2 R name="vlan2" mtu=1500 l2mtu=1520 mac-address=00:0C:42:70:2F:8B
arp=enabled vlan-id=2 interface=ether2-local-slave use-service-tag=no

/ip firewall export

jan/02/1970 01:34:40 by RouterOS 4.16

software id = C6CJ-J4JY

/ip firewall connection tracking
set enabled=yes generic-timeout=10m icmp-timeout=10s tcp-close-timeout=10s
tcp-close-wait-timeout=10s tcp-established-timeout=1d
tcp-fin-wait-timeout=10s tcp-last-ack-timeout=10s
tcp-syn-received-timeout=5s tcp-syn-sent-timeout=5s tcp-syncookie=no
tcp-time-wait-timeout=10s udp-stream-timeout=3m udp-timeout=10s
/ip firewall filter
add action=accept chain=input comment="Added by webbox" disabled=no protocol=
icmp
add action=accept chain=input comment="Added by webbox" connection-state=
established disabled=no in-interface=ether5-gateway
add action=accept chain=input comment="Added by webbox" connection-state=
related disabled=no in-interface=ether5-gateway
add action=drop chain=input comment="Added by webbox" disabled=no
in-interface=ether5-gateway
/ip firewall nat
add action=masquerade chain=srcnat comment="Added by webbox" disabled=yes
out-interface=ether5-gateway
add action=masquerade chain=srcnat comment="Added by webbox" disabled=no
out-interface=ether5-gateway
/ip firewall service-port
set ftp disabled=no ports=21
set tftp disabled=no ports=69
set irc disabled=no ports=6667
set h323 disabled=no
set sip disabled=no ports=5060,5061
set pptp disabled=no

fewi · March 28, 2011, 9:02pm

You have an IP address within 10.0.0.0/16 on two interfaces: 10.0.1.3/16 on the loopback interface, and 10.0.1.254/16 on the VLAN 1 interface. You also have an overlay between VLAN 2 and ether4. That is not a valid configuration. Ensure each IP subnet is only reachable through one interface.

ejay · March 28, 2011, 11:03pm

If you look closely the two 10.0.0.0/16s one is in use, and other other is not. (just deleted the loopback as a distraction) Ether 4 is also not in use.

Just so you know it is not possible to route vlan 1 between trunk ports. They are 2-3 only when tagged.

fewi · March 28, 2011, 11:20pm

While that is true, here a copy/paste of your post above:

/ip route print
Flags: X - disabled, A - active, D - dynamic, 
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 ADS 0.0.0.0/0 192.168.1.254 0 
1 ADC 10.0.0.0/16 10.0.1.3 loopback 0 
vlan1 
2 ADC 10.1.0.0/16 10.1.0.2 ether4-local-ma... 0 
vlan2 
3 ADC 10.2.0.0/16 10.2.0.254 vlan3 0 
4 ADC 192.168.1.0/24 192.168.1.64 ether5-gateway 0

That lists the loopback interface as the gateway for 10.0.0.0/16. Similarly, 10.1.0.0/16 is listed with a gateway of ether4-local-master in the above output.

ejay · March 28, 2011, 11:27pm

In the winbox it says under "routes" ether 4 unreachable, vlan 2 reachable.

I'm sorry the output is truncated, there seems no way to strech the window for you. Also 10.1.0.2 is not a gateway, it's under pref-src instead in that output. As I said it's not even plugged in. I deleted loopback, as it was also a distraction.

/ip route print
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit

DST-ADDRESS PREF-SRC GATEWAY DISTANCE

0 ADS 0.0.0.0/0 192.168.1.254 0
1 ADC 10.0.0.0/16 10.0.1.254 vlan1 0
2 ADC 10.1.0.0/16 10.1.0.254 vlan2 0
3 ADC 10.2.0.0/16 10.2.0.254 vlan3 0
4 ADC 192.168.1.0/24 192.168.1.64 ether5-gateway 0

You were correct sir! Once I removed the IP off the other unused port, and after I brought the OSPF network back up it seems the network actually routes as it should. The software is a little buggy, there is no way an unused IP should be factored into system routing.

fewi · March 29, 2011, 2:23am

Editing post with examples.

It’s a Linux thing. I agree that it is counterintuitive, but in Linux interfaces that are enabled are eligible for routing even if nothing is connected (there is no concept of down/down like in IOS). RouterOS is built on Linux, so it behaves the same.

Here an example on a laptop that is running the latest Debian.

# ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 00:27:13:b4:c1:6b brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:26:c6:c6:44:8e brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.5/24 brd 172.17.0.255 scope global wlan0
    inet6 2001:470:f037:2:226:c6ff:fec6:448e/64 scope global dynamic 
       valid_lft 2591971sec preferred_lft 604771sec
    inet6 fe80::226:c6ff:fec6:448e/64 scope link 
       valid_lft forever preferred_lft forever
4: pan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN 
    link/ether 5a:a8:ce:aa:97:02 brd ff:ff:ff:ff:ff:ff

# ip route show
172.17.0.0/24 dev wlan0  proto kernel  scope link  src 172.17.0.5  metric 2 
169.254.0.0/16 dev wlan0  scope link  metric 1000 
default via 172.17.0.1 dev wlan0  proto static

As you can see eth0 (the wired interface built in) is down. No IPs assigned.

Now I add an IP address to eth0. I do not plug anything in.

# ip address add 10.0.0.1/24 dev eth0
# ip address show eth0
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 00:27:13:b4:c1:6b brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 scope global eth0
# ip route show
10.0.0.0/24 dev eth0  proto kernel  scope link  src 10.0.0.1 
172.17.0.0/24 dev wlan0  proto kernel  scope link  src 172.17.0.5  metric 2 
169.254.0.0/16 dev wlan0  scope link  metric 1000 
default via 172.17.0.1 dev wlan0  proto static

The interface is still down, but is the path to 10.0.0.0/24 regardless.

Next I shut down the interface - this basically shuts down the driver for the interface.

# ifconfig eth0 down
# ip route show
172.17.0.0/24 dev wlan0  proto kernel  scope link  src 172.17.0.5  metric 2 
169.254.0.0/16 dev wlan0  scope link  metric 1000 
default via 172.17.0.1 dev wlan0  proto static

And the route is gone. Next I bring the interface back up, and the route is back despite the interface still being down:

# ifconfig eth0 up
# ip address show eth0
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 00:27:13:b4:c1:6b brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 scope global eth0
# ip route show
10.0.0.0/24 dev eth0  proto kernel  scope link  src 10.0.0.1 
172.17.0.0/24 dev wlan0  proto kernel  scope link  src 172.17.0.5  metric 2 
169.254.0.0/16 dev wlan0  scope link  metric 1000 
default via 172.17.0.1 dev wlan0  proto static 
# ip address flush dev eth0
#

Again - really counterintuitive when coming from other routing platforms, but expected on Linux, and not a bug. Makes it even more important of a best practice to shut down (disable via “/interface ethernet disable etherX”) any interfaces that aren’t in use on a router.

ejay · March 29, 2011, 2:42am

Ty sir, and very informative. It really is an oddity, but understandable in the context of your examples. OSPF still seems a little slow, but once I change providers I’ll see if this is more to do with the odd hardware I have beyond the ether5-gateway. My xbox 360 says it doesn’t like its connection?

Otherwise everything is working now.

fewi · March 29, 2011, 2:56am

Wild guess for the XBox is a NAT issue - you’d have to either turn on UPnP (see wiki manual), or look up the ports and do manual port forwarding (see IP firewall NAT wiki for an example).
FWIW my XBox 360 says the same, and it is working fine for downloads and online interactions in some games like NFS:HP for leader boards. I don’t have Gold and don’t play online, I suspect that I couldn’t without one of the two options above, or that at least I would not be able to host games.

Have fun with your router. They have their kinks and oddities but are quite fun once you get used to the idiosyncrasies.

ejay · March 29, 2011, 3:44am

Like I said once I swap providers, I’ll see if there is still an issue. I am behind two NATs at the moment. I did this to prepare for the new service while I’m in classes again (we were on break), and don’t have the time to trouble shoot this. Since I have this all working it’ll be as easy as unplug, and replug. Then it’ll be a real cable modem, and no extra NAT involved.

I am impressed how intuitive the router os was. I did quite an advanced thing for the first time, and it didn’t take very long to make it work. Only one, or two gottchas. They should make bigger switches like this RB750G. They have so many features. I was happy to figure out I can do DHCP service per vlan. That is a very handy tool.

I still want to know if I put vlan 2 back on the other port 4 (with vlan1 & vlan 3), if sub interfaces would work? I don’t like wasting the extra port I’m using. (2 really)

fewi · March 29, 2011, 3:55am

Moving the VLAN 2 interface back should be no problem, and should be as easy as editing the “/interface vlan” section and setting a different parent interface - one step. That’s one nice part about all configuration for that network being attached to the VLAN interface by reference, and the physical interface that it is tied to just being a parameter.

ejay · March 29, 2011, 4:08am

I almost forgot. I saw some “jumps through hoops” examples on the wiki, how to set a trunk port on an interface. I just removed the IP, and set a vlan as a sub interface. No other hoop jumping needed.