VLAN with Router - Basic Setup

penguinpages · October 8, 2019, 11:31pm

I have a MikroTik CRS305-1G-4S+in

three servers with up-link to 24port 1Gb switch

By default the VLAN is 1 and management I set to 172.16.100.x/24 management IP for network.

Played around for a bit and came to conclusion I want to do following

Router mode (vs switch which was default)
All links to switch are “trunk allow VLAN all” so each physical interface has ONLY tagged packets and those are on each physical port
VLAN 1 is left untagged but ONLY on the 1Gb RJ45 port with the default IP as a backup to “get in if something going bad” and need to get into admin portal

Diagram:

I upgraded router, tried to put it into “router mode” and created VLANs that I think are all linked via “bridge”

In all other switch vendors I work with you define VLAN and bind those to be “tagged” on ports. I think in this switch you bind VLANs to bridge and interfaces to bridge and that flows the packets…

In this switch I am still trying to work out process. I did watch some you-tube videos but they are reflecting much older OS versions where UI does not match up or examples are port based VLAN.

One thing I have an issue with is that one server (server in port 3) keeps dropping off network and I found that the port flips from “RS” to “S” (I assume this means Router Vs Switch?? )

Questions:

If goal is simply to create the five VLANs and bind IP to each and then set that VLAN tagged on all five interfaces, is there an example (CLI is fine an I think may be only way to do this)
Why does that one port keep flipping from RS to S and what does this mode change mean?
Are there any threads or examples of integration with KVM systems working OVS 2.9
How can I display via the CLI “running Configuration” vs “saved configuration” .. I poked around wiki and this I did not find.

Thanks

penguinpages · October 9, 2019, 1:54pm

Update: One question I did figure out was how to show running switch configuration and also save it off.

Show running configuration:
[admin@sw3] > export file=

oct/08/2019 20:16:50 by RouterOS 6.45.6

software id = 9NJL-Q743

model = CRS305-1G-4S+

set address-acquisition-mode=static allow-from-vlan=100 identity=sw3 static-ip-address=172.16.100.250

[admin@sw3] >

How to export if
[admin@sw3] > export file=sw320191008

If anyone has an example of the other "trunk tagged" interface.. I have to believe this is very typical. If I can see the CLI it is way easier to wrap head around then you tubes and screen shots of UI

penguinpages · October 9, 2019, 2:15pm

Here is output of switch configuration that I think I need adjustment to:
<<COMMENTS/ QUESTION in line that I think I did wrong and need to adjust>>>>

[admin@sw3] > export compact
# oct/09/2019 09:55:04 by RouterOS 6.45.6
# software id = 9NJL-Q743
#
# model = CRS305-1G-4S+
# serial number = AB5C0AA84802
/interface bridge
add admin-mac=74:4D:28:C1:6E:B4 auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=sfp-sfpplus3 ] l2mtu=9216 mtu=9216 name=medusa-enp3s0f0 speed=10Gbps
set [ find default-name=sfp-sfpplus2 ] l2mtu=9216 mtu=9216 name=odin-enp3s0f0 speed=10Gbps
set [ find default-name=sfp-sfpplus4 ] l2mtu=9216 mtu=9216 speed=10Gbps
set [ find default-name=ether1 ] l2mtu=9216 mtu=9216 name=sw0-24 speed=100Mbps
set [ find default-name=sfp-sfpplus1 ] l2mtu=9216 mtu=9216 name=thor-enp3s0f0 speed=10Gbps
/interface vlan
<<< COMMENT:  No VLAN1 defined so no way to bind mgmt to a defined VLAN 1... not a big deal>>>
add interface=bridge name=Containerpriv vlan-id=103
add interface=bridge name=DMZ vlan-id=102
add interface=bridge name=Production vlan-id=100
add interface=bridge name=RED vlan-id=666
add interface=bridge name=Storage vlan-id=101
add interface=bridge name=VPN vlan-id=104
add interface=bridge name=wise_guest vlan-id=105
/interface ethernet switch
set 0 name=sw03
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot

<<< Question: This defined a logical "bridge" called "bridge" to create L2 repeat segment.  Into this I place interfaces physical as well as VLAN logical ones..... I think....>>>>
/interface bridge port
add bridge=bridge comment=defconf interface=sw0-24
add bridge=bridge comment=defconf interface=thor-enp3s0f0
add bridge=bridge comment=defconf interface=odin-enp3s0f0
add bridge=bridge comment=defconf interface=medusa-enp3s0f0
add bridge=bridge comment=defconf interface=sfp-sfpplus4
/interface bridge vlan
add bridge=bridge comment=100_105_VLANs tagged=Storage,VPN,DMZ,Containerpriv,wise_guest,RED untagged=Production vlan-ids=""
/interface list member
add comment="sw3-1 SFP to Port0 on 10Gb NIC" interface=thor-enp3s0f0 list=LAN
add comment="sw3-2 SFP to Port0 on 10Gb NIC" interface=odin-enp3s0f0 list=LAN
add comment="sw3-3 SFP to Port0 on 10Gb NIC" interface=medusa-enp3s0f0 list=LAN
add interface=sfp-sfpplus4 list=LAN
add interface=sw0-24 list=WAN

<<<< COMMENT:  this binds IPs for both managment and router interfaces to each VLAN.  Seems the mgmt IP 172.16.100.250 which is the GUI mgmt IP to the first 10Gb port????
/ip address
add address=172.16.100.250/24 comment=defconf interface=thor-enp3s0f0 network=172.16.100.0
add address=172.16.101.1/24 comment="Storage VLAN IP" interface=Storage network=172.16.101.0
add address=172.16.102.1/24 comment="DMZ IP" interface=DMZ network=172.16.102.0
add address=172.16.103.1/24 comment="Container Segment Private " interface=Containerpriv network=172.16.103.0
add address=172.16.104.1/24 comment="VPN Managment IP" interface=VPN network=172.16.104.0
add address=172.16.105.1/24 comment="wise_guest Wifi VLAN" interface=wise_guest network=172.16.105.0
/ip dns
set servers=172.16.100.40,8.8.8.8
/ip route
add distance=1 gateway=172.16.100.1
/ip smb
set comment="sw3 " domain=sw3
/ip ssh
set allow-none-crypto=yes forwarding-enabled=remote
/routing rip network
add network=172.16.101.0/24
add network=172.16.102.0/24
add network=172.16.103.0/24
add network=172.16.104.0/24
add network=172.16.105.0/24
/system clock
set time-zone-name=America/New_York
/system identity
set name=sw3
/system routerboard settings
set boot-os=router-os
/system swos
<<< QUESTION: Not sure what this is about. I think this is saying to use static to bind to tagged VLAN 100 172.16.100.250 but not sure what interface that binds to. I will eventually.>>>
set address-acquisition-mode=static allow-from-vlan=100 identity=sw3 static-ip-address=172.16.100.250
[admin@sw3] >

Here is logical diagram of goal

mkx · October 9, 2019, 4:00pm

The VLAN setup is pretty flawed. I suggest you to go through this tutorial. Then come back with questions.

BTW, VID=1 is used as default in many places. So try to avoid using VLAN with this VID to avoid some surprises. And try to avoid using “hybrid” ports (a port with a few tagged and one untagged VLAN) if possible, running untagged VLAN over trunk can also bring some surprise.

penguinpages · October 10, 2019, 11:03pm

Thanks for link.

Good review of basics and it is very helpful where they post examples of configuration. Both sides of switch helps

I believe for my example my environment is a bit of a mess now in that the managment VLAN 1
Question: Is there a command to enable or disable mangment access on a given interface. Ex: bind IP to VLAN (as router interface then) is that by default capable of response to mgmt traffic?

Second issue is that mgmt IP is today bound to my first 10Gb NIC port which is bound to bridge "bridge" and so that is how mangment is working. All physical ports on the same logical bridge called "Bridge" is how it is functioning, but I would prefer mgmt was bound to a VLAN and no matter which VLAN I am on, I can get to manage the switch. Move default VLAN to 100 and VLAN 1 would be the only "untagged" interface and go only on the 1Gb. Question this does bring up is do BDPUs for this switch still get pinned to VLAN 1 or are they per VLAN? This plays into OVS configuration with LLDP and virtual switches where I hope that I could later on do dynamic VLAN scripting via OVS.

Command for default Mgmt on VLAN 1 but I read bellow as also allowing mgmt on VLAN 100

set address-acquisition-mode=static allow-from-vlan=100 identity=sw3
static-ip-address=172.16.100.250

\

Commands to fix "trunks" (assuming I get mgmt over all IPs validated to be ok)

Create a new logical bridge

/interface bridge port
add bridge=BR1 interface=sfp-sfpplus1
add bridge=BR1 interface=sfp-sfpplus2
add bridge=BR1 interface=sfp-sfpplus3
add bridge=BR1 interface=sfp-sfpplus4
add bridge=BR1 interface=ether1

Bind List of VLANs to the new BR1 logical bridge for each interface

add bridge=BR1 tagged=BR1,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,ether1 vlan-ids=100
add bridge=BR1 tagged=BR1,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,ether1 vlan-ids=101
add bridge=BR1 tagged=BR1,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,ether1 vlan-ids=102
add bridge=BR1 tagged=BR1,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,ether1 vlan-ids=103
add bridge=BR1 tagged=BR1,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,ether1 vlan-ids=104
add bridge=BR1 tagged=BR1,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,ether1 vlan-ids=105

Create VLANs with better layout to have DHCP pools as well as. Repeat below for each VLAN

/interface vlan add interface=BR1 name=Production vlan-id=100
/ip address add interface=Production address=172.16.100.1/24
/ip pool add name=Production ranges=172.16.100.50-172.16.100.200
/ip dhcp-server add address-pool=Production_Pool interface=Production name=Production_DHCP disabled=no
/ip dhcp-server network add address=172.16.100.0/24 dns-server=172.16.100.41 gateway=172.16.100.42

Questions:

What will this do for VLANs I had already defined and have bound to bridge "bridge"?
I am glad DHCP is a service offering on this switch, but is there a link to when I need to pass more advanced options such as bootp / NTP parameters? A link to example documentation is likly all I need.
This switch does not seem to have a function of apply to running vs boot. AKA.. all commands are real time and if I scew it up I am finding a paper clip to do a hard reset (vs power cycle back to last saved configuration state.
Also open to rec

I saw that there is a OVS of a virtual machine switch simulator. I assume that this would be a good place to run testing /simulations with but how close can I match hardware.. Is there a good site / documentation on how to set these up. I searched site / youtube about that and did not find much.

mkx · October 11, 2019, 6:07am

In RouterOS there’s no a-priori distinction between L3 interfaces (those with IP address bound) with regard to management access (or most of other functionalities for that matter). The difference is made by firewall filter rules (and bridge filter rules). Default setup uses two interface lists (WAN and LAN) and allows management access from LAN interface list. It’s up to router admin to maintain interface list membership current. This approach is slightly simplistic (also other functionality is bound to LAN interface list) so you might want to introduce another interface list (named for example management) and construct appropriate firewall filter rules which will allow management access from interfaces members of that interface list (and don’t forget about /tool mac-server settings). After that it’s a fairly simple act of adding or removing interface to/from interface list which will enable or disable management via certain interface.
Note that interface list should contain L3 interfaces, when using VLANs that means appropriate vlan interface (and not physical interface carrying that VLAN).

Personally I don’t have experience with xSTP … but other members of this forum shared the knowledge which goes approximately like this: when using bridge vlan-filtering way of configuring VLANs, xSTP works correctly over VLANs. Contrast this to old school (dumb bridge and VLANs set on switch hardware) where bridge is unaware of VLANs and xSTP works using untagged frames (which apparently is not the right way).
BTW, apparently it’s not possible to configure CRS3xx the old shool way …

This should work as expected … bridges are independent of each other and it should be just fine to run VLAN with same VID on several bridges, those should still be separated. However, none of Routerboards can run HW accelerated more than single bridge per switch chip. Meaning the second (and third and …) bridge on your CRS will not be HW accelerated and all data between ports belonging to same bridge (even same VLAN) will pass CPU. Which will become a major bottleneck.
Of course it’s not possible to add an interface as slave interface to more than one bridge.

DHCP server doc There are DHCP options that are configurable in a “user friendly” way, the rest can be configured the dhcp-option … but that’s messy.

Indeed.
There’s safe mode (available both in Winbox and CLI) which helps a bit … if you enable safe mode and enter command which breaks management connection, device will revert configuration to the state before enabling safe mode. When you’re done with configuration and you still have management connectivity, exit safe mode and config gets “commited”.
There are commands that are supposed to break management connectivity (i.e. change of management interface IP address) and those obviously can’t be executed while in safe mode. So one has to be really sure about those commands (or else the paper clip becomes a best friend again).

I’ve no idea about simulators, perhaps some other user can shed a light?

penguinpages · October 12, 2019, 1:15pm

Thanks for detail.

I am trying to learn the new OS syntax. It has a quazi directory structure so I can't just write script and past and it applies (unless there is some mode I am missing)

I can't see how to get the GUI to defined each inteface to bind to a list of VLANs.. so this is CLI only work...which is fine.

I want to use bridge "bridge" to maintain performance per note above.. that is fine. (I always have my friend the paper clip to back me up)

Goal:Define for interface "bridge" for VLAN tagged a list of all physical interfeaces for "vlan 105"
add bridge=bridge tagged=bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,ether1 vlan-ids=105

That would work if no "bridge" existed. But it does... and I need to learn how to edit not just wipe / rebuild

[admin@sw3] /interface> export

serial number = AB5C0AA84802

/interface bridge
add admin-mac=74:4D:28:C1:6E:B4 auto-mac=no comment=defconf name=bridge

[admin@sw3] /interface> /
[admin@sw3] > /interface bridge
[admin@sw3] /interface bridge>
[admin@sw3] /interface bridge> edit
number: bridge
value-name: tagged=bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,ether1 vlan-ids=105
Script Error: action cancelled
[admin@sw3] /interface bridge> exit
bad command name exit (line 1 column 1)
[admin@sw3] /interface bridge> edit bridge
value-name: vlan-ids=105
Script Error: action cancelled
[admin@sw3] /interface bridge>

So I think what I am not understanding is the sysnax. RTFM is ok so long as FM is in form of URL I can read. I did find a few youtube videos on the CLI but they were very basic, and green field vs change. I am sure it is something stupid.

Second question:
When I set switch in "safe mode" by selecting the button in the GUI.. and I export the current configuration in the CLI, I see no difference. I don't trust GUI buttons (especially java). Is there a CLI means to set / unset this?

Third Question:
DHCP option to set static reservations. I read that page and good for the basics but typical tech docmentation, no examples. I have to believe somone has some example of a basic DHCP server with a set of static mac based reservations.

mkx · October 12, 2019, 3:18pm

The configuration script would be a series of CLI commands, written in plain text file and usual name extension is .rsc. You can push a script file to the routerboard and then execute /import . Actually output of /export is exactly such a script and if it was imported after device was reset with no defaults, this script would, more or less, regenerate your current config (resulting script doesn’t contain usernames and passwords nor does it contain SSL certificates).

Vast majority of things can be set up through both GUI and CLI. GUI (both Webfig and winbox) more or less follows the CLI hierarchical structure. Just look around GUI and you’ll probably find what you’re looking for.
Hint: in webfig, look under Bridge → VLANs …

Most probably the command above fails because there’s already defined list of ports for given combination of bridge and vlan-ids. (It is possible to have plenty of such stanzas per bridge as long as bridge&vlan-ids property combination is unique). However, if there’s already such a list and you want to add or remove interfaces, you can always use set like this:

set [ find bridge=bridge and vlan-ids=105 ] tagged=bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,ether1,ether2

Or, if you use print command right before executing set, you could replace the [ find … ] construct by row number, displayed by print … so assuming the config stanza you want to change is printed under #5, the command would be

set 5 tagged=bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4,ether1,ether2

You can change any property using set (including name of bridge or vlan-ids).
If you want to delete some configuratuion detail, use remove command.

Use CTRL-x to toggle safe mode on/off … when safe mode is enabled, the command prompt changes (it gets “” at the end of prompt prefix).

The easiest way of creating static leases is to let device obtain a dynamic one. Then convert it to static (while in /ip dhcp-server lease by using make-static … and you can replce by a construct something like [ find address= ] or something like that). After that you can change things (such as assigned address) using set … command.

Surely you can construct a static lease manually, you have to enter at least address, mac-address and server properties (server being the name of one of configured DHCP server instances run on your RB).

anav · October 12, 2019, 8:32pm

highly recommend not using vlan1 for any purpose, just leave it in defaults where it exists.
you will notice the examples in the excellent dont use it either.

penguinpages · October 13, 2019, 10:35pm

I am continuing to poke around to learn UI and CLI. I tried 'add" vs "set" and like noted that within context of / you can't just "paste" the commands back from export. Maybe it is expected your within "enable" kind of mode. Is there any manual or document of taking via CLI NIB router and programming and then making changes.

Current state. After changes below. I can ping gateway but no longer ping between hosts.

[admin@sw3] > add bridge=bridge comment="Bridge With All VLANs" tagged=Production,Storage,DMZ,VPN,wise_guest,RED,thor-enp3s0f0,odin-enp3s0f0,medusa-enp3s0f0,sfp-sfpplus4,sw0-24 vlan-ids="100,101,102,103,104,105,666"
bad command name add (line 1 column 1)
[admin@sw3] > set bridge=bridge comment="Bridge With All VLANs" tagged=Production,Storage,DMZ,VPN,wise_guest,RED,thor-enp3s0f0,odin-enp3s0f0,medusa-enp3s0f0,sfp-sfpplus4,sw0-24 vlan-ids="100,101,102,103,104,105,666"
syntax error (line 1 column 5)

I know the 4 SFP ports are working on same VLAN / Bridge ad they were working (and can work if I unbind from "VLAN 101"
but not able from logical router interface to ping IPs bound to those interfaces... aaka. they are not on same VLAN.

From the two hosts I can ping the router interfaces with VLAN tagged interface

58: storage@enp3s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:0e:1e:06:f4:7c brd ff:ff:ff:ff:ff:ff
inet 172.16.101.101/24 brd 172.16.101.255 scope global noprefixroute storage
valid_lft forever preferred_lft forever
inet6 fe80::20e:1eff:fe06:f47c/64 scope link
valid_lft forever preferred_lft forever
[root@thor network-scripts]# ping 172.16.101.101
PING 172.16.101.101 (172.16.101.101) 56(84) bytes of data.
64 bytes from 172.16.101.101: icmp_seq=1 ttl=64 time=0.045 ms
^C
--- 172.16.101.101 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.045/0.045/0.045/0.000 ms
[root@thor network-scripts]# ping 172.16.101.102
PING 172.16.101.102 (172.16.101.102) 56(84) bytes of data.
From 172.16.101.101 icmp_seq=1 Destination Host Unreachable
From 172.16.101.101 icmp_seq=2 Destination Host Unreachable
From 172.16.101.101 icmp_seq=3 Destination Host Unreachable
From 172.16.101.101 icmp_seq=4 Destination Host Unreachable
^C
--- 172.16.101.102 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3998ms
pipe 3
[root@thor network-scripts]#

The VLAN shows traffic so it is tagging the tagged packets but it is like the two hosts are on different VLANs.. or bridge is not accepting tagging between ports. This is a L2 issue. I will keep poking at it and see if something jumps out.

mkx · October 14, 2019, 5:32am

vlan-ids should be either single value or value list (comma-separated), not a quoted string. In short: don’t use (double) quotes to enclose list of values.

I’m sure you’re aware that you added all ports as tagged … which means thet connected devices have to tag frames with proper VLAN tags themselves. If one of devices doesn’t do that, then … well, they’re not part of same VLAN.

In addition to that: I don’t see bridge itself added as tagged member of all of those VLANs… hence no traffic can pass between different VLANs as device’s CPU doesn’t see them.

And: don’t add vlan interfaces as members of VLANs. vlan interfaces are kind of selective pipes: one end is anchored on multi-vlan interface (in your case that’s bridge “interface”), the other end is actually untagged and should be used as such. Making vlan interfaces members of VLANs is similar to throwing anchor into the sea and tie the chain to … the sea. Such vlan-interface passes only frames with defined VID in direction from anchor towards the untagged end and get untagged during this travel. When untagged frames travel in the other direction, they get tagged with defined VID and get flushed to the “sea”.
Actually it is possible to throw tagged frames at the “untagged” end of vlan interface, in this case another VLAN tag gets added (so called tag-stacking) … and if doubly-tagged frame enters from the “anchor” side, only the outer tag gets stripped, the inner one remains. But this kind of operation is not exactly common …

I’m more or less in the dark regarding current config of your device so it’s really hard to comment the problems you see. Perhaps you could post complete configuration as shown by /export hide-sensitive (enclose it in [__code][/code] environment for better readability), screenshots are king of hard to read (and many times one can’t show all the information in single picture).

penguinpages · October 14, 2019, 11:24am

That fixed it!!

First:
“…vlan-ids should be either single value or value list (comma-separated), not a quoted string. In short: don’t use (double) quotes to enclose list of values…”

The GUI does that based on value of VLANs allowed set to 1-4094 which then it build the common seperated list in the actual CLI command.

But as for my L2 issue.. It was that the bridge called “bridge” was no in the list of tagged logical interfaces.

I A$$umed… that as i was “under bridge section” under “VLANS” and selected the Bridge called “bridge” it was included (as a parent device to bind all the other logical interfaces under it. And as such was with that L2 domain.

I really do appreciate your help and postings. I hope to learn more about this switch OS and become more competent and spare the support community newbie questions.