Community discussions

MikroTik App
 
dark3phoenix
just joined
Topic Author
Posts: 3
Joined: Mon Feb 20, 2023 1:05 am

Help with pinging between only "existing" VLANs, not any I create now

Sun Mar 12, 2023 8:58 pm

Hello,

My old Cisco SG350 died this week and I decided to replace my whole physical network (fortunately I had been planning to do that so I had gear already in-house). I just finished rebuilding my network with an RB5009, CRS328, and a pair of CRS312s for 10gb networking. My physical connections are:

RB5009-1---sfp-sffplus1--------sfp-sfpplus1--CRS328-1
CRS-328-1--sfp-sfpplus3------combo4--CRS312-1
CRS-328-1--sfp-sfpplus4------combo4--CRS312-2

Each switch is running RouterOS 7.8 and I was able to regenerate the (albeit crazy) configuration that I had in my old setup. I have about 10 VLANs established (which I created earlier in the week during the initial bring-up) and everything has been working well for the past 3 - 4 days. Traffic is passing normally between all devices/VLANs and all my home and lab services are running without error. I am allowing all traffic between all VLANs right now (plan to change that in the future) using the RB5009 as the main router for the network.

In order to restore NSX-T to my home lab (the last thing on my list), I need to get BGP going again. To get started, I created a new (not part of the original setup) VLAN 31 to connect the switches together with BGP. I believe I followed the same configuration as I have for all the other VLANs, but I cannot get the two switches to ping to each other. In both switches, I have the VLAN interfaces defined to the bridge, on the bridge I have the VLAN tagged on the appropriate port for that switch and the bridge, and I have created local TCP/IP addresses on the VLAN (just like all my other working VLANs).

When I ping from RB5009->CRS328 using the interfaces I defined to VLAN31 I see the arp entry on both sides, but it is in "D" and never "DC". When I ping from CRS32->RB5009 I don't see any entries added to either ARP table. I have deleted the stale ARP entries and tried pinging from both directions, I deleted the interfaces and VLAN on both sides and started again, I tried changing the VLAN numbers (originally I tried using VLAN 2711, then 27, then 31), but no change. Just to ensure I wasn't completely crazy I created interfaces in 328 on all the existing VLANs and I can ping from them to the corresponding RB5009 interfaces with no issue.

This seems so utterly basic to me, so I'm assuming I just forgetting a step or something equally noobish. I have rewatched about 40 videos on setting up VLANs and reading the docs and forums but I just can't figure out what I'm doing wrong.

Can someone spot what I'm doing wrong here?
You do not have the required permissions to view the files attached to this post.
 
dark3phoenix
just joined
Topic Author
Posts: 3
Joined: Mon Feb 20, 2023 1:05 am

Re: Help with pinging between only "existing" VLANs, not any I create now

Mon Mar 13, 2023 5:19 am

So after posting this entry and walking away for a few hours I came back and started looking at it. When I ping from RB5009 -> CRS328 I can see the packet going out on the traffic page, then I see 1 sent and 1 received on the 328 via the same panel on the CRS328 (traffic on the VLAN). Arp is right on the 328, missing mack on RB5009. I then moved the interfaces (172.28.31.1/2) to the bridge interfaces and when pinging everything works. Move them back to VLAN 31 and the same symptoms - no ping but the arp is correct on the CRS328. Tried reversing the ping CRS328 -> RB5009 and the arp shows correctly on the CRS328 but no packets at all make it to the 328. Setup logging all the drop lines in the firewall config and I don't see anything. I don't think i'm any closer to understanding what's going on but thought maybe this additional information would help diagnose what is going on and what I misconfigured.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19099
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Help with pinging between only "existing" VLANs, not any I create now

Mon Mar 13, 2023 12:51 pm

(1) So many vlans but you only really define 4 of them..............

(2) you can shorten up your /interface bridge vlan
/interface bridge vlan
add bridge=bridge tagged=bridge,sfp-sfpplus1-CRS328-1 untagged="ether4 - CMM P\
C,ether5 - Living Room,ether3 - Google Wifi,ether6 - Chris Work laptop" \
vlan-ids=101
add bridge=bridge tagged=bridge,sfp-sfpplus1-CRS328-1 vlan-ids=5
add bridge=bridge tagged=bridge,sfp-sfpplus1-CRS328-1 vlan-ids=6
add bridge=bridge tagged=bridge,sfp-sfpplus1-CRS328-1 vlan-ids=7
add bridge=bridge tagged=bridge,sfp-sfpplus1-CRS328-1 vlan-ids=10
add bridge=bridge tagged=sfp-sfpplus1-CRS328-1,bridge vlan-ids=15
add bridge=bridge tagged=bridge,sfp-sfpplus1-CRS328-1 vlan-ids=20
add bridge=bridge tagged="bridge,sfp-sfpplus1-CRS328-1,ether5 - Living Room" \
vlan-ids=25
add bridge=bridge tagged=bridge,sfp-sfpplus1-CRS328-1 vlan-ids=30
add bridge=bridge tagged=bridge,sfp-sfpplus1-CRS328-1 vlan-ids=35
add bridge=bridge tagged=bridge,sfp-sfpplus1-CRS328-1 vlan-ids=40
add bridge=bridge tagged=sfp-sfpplus1-CRS328-1,bridge vlan-ids=31


TO:
add bridge=bridge tagged=bridge,sfp-sfpplus1-CRS328-1 vlan-ids=5,6,7,10,15,20,30,35,40,31
add bridge=bridge tagged="bridge,sfp-sfpplus1-CRS328-1,ether5 - Living Room" \
vlan-ids=25


(3) It looks like ether5 is a hybrid port ??

(4) In reality with only 4 vlans defined, it should look quite different

(5) Only one dhcp server assigned

(6) Five dhcp server networks assigned........

(7) For some reason you opted to give the bridge an address vice just leaving it to bridge............... no dhcp no nthing
suggest turn 192.168.88.0 into VLAN2 etc............. much less confusing.

All in all a confused config that is not complete.
 
dark3phoenix
just joined
Topic Author
Posts: 3
Joined: Mon Feb 20, 2023 1:05 am

Re: Help with pinging between only "existing" VLANs, not any I create now

Mon Mar 13, 2023 6:35 pm

Thank you for the detailed analysis. Let me ensure that I understand what you've laid out:

1) Each VLAN has a use, just not in RB5009-1 or in the CRS328-1 - they are used in the two CRS312s that connect to the CRS328-1 (I did not include those configs but I certainly can if it clears anything up). I am passing the VLANs up to the router because I thought the router should be doing the routing. I suppose I could push them down to the CRS328-1 if that is a better design since it is the closest to 2 CRS312s. When you say "only really define 4 of them" do you mean I'm missing configuration for many of the VLANs? If so, what did I misconfigure?
2) I thought you were supposed to create the VLAN tagging definitions based on the VLAN number, not the port. I can easily swap that around, I was just coming at it backward.
3) Yes, it currently is a hybrid port due to some problems I had with the SG350-10 switch on the other end. I'm replacing that switch this week with a CSS610 so I'll change it to a proper trunk port then.
4) I agree, once I swap from VLAN -> Port as the focus for the tagging it will get much smaller on the RB5009 and the CRS328.
5) I'm confused by this. When I go to "DHCP Server" in the DHCP tab in Winbox I see 5 DHCP entries - one for each network I give out DHCP addresses for. What is the proper configuration to change to?
6) Is the number of networks an issue? I used to use my Synology as the DHCP server for everything but thought it was better to have the router doing it when I rebuilt everything.
7) I was using the bridge network as the default management network thinking that if I ever messed up the configuration any port I connect it to should be able to reach the management IP. I can certainly move them to VLAN 2.

I will look to make these changes tonight or tomorrow night when the network isn't in use. Thank you again for taking the time to analyze the configuration.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19099
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Help with pinging between only "existing" VLANs, not any I create now

Mon Mar 13, 2023 7:46 pm

To manage the configuration and possible burps with bridge and/or vlan configuration, I prefer to create an off-bridge port.

Lets say ether5, ensure its off the bridge.
create an IP address for it.
192.168.55.1/24 interface=ether5 network=192.168.55.0
Done!
Now you can plug in a laptop to ether5 set any ipv4 address you want on the ethernet card settings eg. 192.168.55.5
and gain entry to the router!
To be able to config the roiuter, make sure the subnet or IP is included in any source list or interface list allowed on the routers input chain.
add chain=input action=accept in-interface-list=MGMT src-address-list=Authorized.
/interface list
add name=MGMT
/interface list memberrs
add interface=trusted vlan list=MGMT
add interface=ether5 list=MGMT
/ip firewall address list
add ip-address=AdminDesktop_IP list=Authorized
add ip-address=AdminLaptop_IP list=Authorized
Add ip-address=AdminIPAD/iphone_IP list=Authorized
Add ip-address=AdminRemoteWG_IP list=Authorized
add ip-address=192.16.55.0/24 list=Authorized.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19099
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Help with pinging between only "existing" VLANs, not any I create now

Mon Mar 13, 2023 7:47 pm

So are you saying you define vlans elsewhere (other devices) but they need to traverse this router enroute somewhere else be it internet or other devices on the network????
 
User avatar
Buckeye
Forum Veteran
Forum Veteran
Posts: 887
Joined: Tue Sep 11, 2018 2:03 am
Location: Ohio, USA

Re: Help with pinging between only "existing" VLANs, not any I create now

Tue Mar 14, 2023 2:46 pm

1) Each VLAN has a use, just not in RB5009-1 or in the CRS328-1 - they are used in the two CRS312s that connect to the CRS328-1 (I did not include those configs but I certainly can if it clears anything up). I am passing the VLANs up to the router because I thought the router should be doing the routing. I suppose I could push them down to the CRS328-1 if that is a better design since it is the closest to 2 CRS312s.
What is the purpose of creating all the vlan interfaces on the CRS328 switch? i.e. the
/interface vlan
add interface=bridge mtu=9000 name=5-HomeManagemet vlan-id=5
add interface=bridge mtu=9000 name=6-Lab-Management vlan-id=6
add interface=bridge mtu=9000 name=7-Lab-Machines vlan-id=7
add interface=bridge mtu=9000 name=10-Client-Network vlan-id=10
add interface=bridge mtu=9000 name=15-Storage vlan-id=15
add interface=bridge mtu=9000 name=20-vMotion vlan-id=20
add interface=bridge name=25-Raw-Internet vlan-id=25
add interface=bridge mtu=9000 name=30-Horizon-View vlan-id=30
add interface=bridge mtu=9000 name="31-BGP RB5009-1 <=> CRS328-1" vlan-id=31
add interface=bridge mtu=9000 name=35-NSX-Host-Tep vlan-id=35
add interface=bridge mtu=9000 name=40-NSX-Edge-Tep vlan-id=40
add interface=bridge mtu=9000 name="101-General Network" vlan-id=101

These create a layer 3 interface for the CPU and if you then add an ip address, it will create a connected route, and that route will be preferred over the default gateway. They are not needed to create vlans on the switch. They are just a "connector" for the CPU in the CRS328 to be able to tap into one of the vlans on the switch.

You are creating the following addresses, so if you issue a /ip route print command on the CRS328 you will see connected routes for those as long as there is an "up" interface. If there is a way to disable connected routes, I am not aware of it (other than disabling the interface). If someone know a way, speak up.

/ip address
add address=192.168.88.2/24 comment=defconf interface=bridge network=\
192.168.88.0
add address=172.28.1.254/22 interface="101-General Network" network=\
172.28.0.0
add address=172.28.31.2/30 interface="31-BGP RB5009-1 <=> CRS328-1" network=\
172.28.31.0

I think you should have only one ip network defined on the switches, the one you will use for management (i.e. the address you will connect to with ssh or winbox). The vlans don't need a connection to the CPU in the CRS switches.

What network do you want to use for management access to the switches? The 172.28.1.254 address? What is the purpose of the 172.28.31.0/30 network (with the name "31-BGP RB5009-1 <=> CRS328-1" that suggests you are using BGP between the switch and the router. Why would the switch be participating in BGP, won't that just be the router and the VM running NSX-T? (I have no knowledge of NSX-T, it just googled it). The CPU in the CRS328 isn't meant to do much more than menial tasks, like SNMP and ssh and winbox server, and while it "can" route and do other things, if you look at the specs you can it it has a single core 32bit ARM processor running at 800Mhz thats built into the switch ASIC. On the other hand the RB5009 has a much beefier CPU and if there is a CPU in its switch ASIC it isn't hosting ROS. Look at the differences in performance on the CRS328 for switching vs (software) bridging/routing (which are done by the CPU).

The point is we don't want to let the CRS CPU have the opportunity to try to route, because it will do a poor job of it. And the easiest way to avoid that is to have only one L3 interface on the CRS328 with an ip address, then it can't route, and will create only a single connected route (other than loopback) for the vlan interface you want to use for management. Any traffic to/from other networks will go through the default route (which should be your RB5009). It may be possible to do hw assisted layer 3 on the CRS3* but if that is the case that wasn't being wasn't being shown in the routing performance numbers. Also, if you want a consistent traffic cop between vlans, you will need to let the firewall in the RB5009 be the only path between the vlans, then the inter-vlan traffic will have to go through the RB5009 CPU and allow it to apply firewall rules to the traffic. The only case this wouldn't be true is if you have a host with multiple interfaces (possibly vlan interfaces) that are directly connected to more than one of the vlans. So my recommendation is let the switches do switching, and the RB5009 router do routing (there is also a switch chip in the RB5009 that is quite capable as well, although probably not as feature full as the one in the CRS328).

Who is online

Users browsing this forum: No registered users and 63 guests