seperate 10G subnet to existing 1G

RouterOS Beginner Basics
1

Hy to the forum,
it’s my first post here as I’m complete new to Mikrotik
No network Pro, but quite familiar with stuff :slight_smile:

I got a Mikrotik CRS317-1G-16S+ and updated it to RouterOS 7.11
Since 2 days,I try to realize a seperate 10G Network for Proxmox with an dedicated iSCSI Storage
(see Diagramm below), but somwhere i have a fault or a knot in my brain and i cant find the error.

Basically the 10G subnet (192.168.88.0/24) should only talk to 1G Network (192.168.178.0/24)

  • when it comes to access from/to 1G network
  • 10G network needs internet connectivity
  • The Dell Servers in Diagramm and the NAS should only talk over 10G

I connected 1G Port1 from CRS317 to 1G Switch, which is connected to my pfsense.
The IP on the Mikrotik is 192.168.88.250. As Gateway I set 192.168.178.1 (pfsense)

I connected 10G Port2, from CRS317, to Dell Server and configured the bridge with 192.178.88.250/24

From the Dell Server, I’m able to ping 192.168.88.250 (Port1 1G CRS317) and 192.168.178.250 (Port2 10G CRS317) via 10G NIC. I have 1G NIC on Server
but communication should, happen over the 10G Interface.

Network config on Server:
192.178.88.2/24 gw 192.178.88.250 (10G Mikrotik Port 2)

For some reason I’m not able to internet access, as I’m not reaching the pfsense firewall.
Here are some addtional screens from the bridge / interfaces and routes

pings from CRS31 via bridge interface to following devices are successful:
192.168.178.1 (pfsense)
192.168.178.250 (port1 1G CRS317)
192.168.88.250 (port2 10G CRS317)
192.168.88.1 (10G Port Server)

I would highly appreciate, if the pros here could give me a hint, where my thinking fault is.
If addtional information is needed (cli, screens, etc) please let me know.

2

Reading screenshots is for ChatGPT, we network enthusiasts prefer text exports of config :wink:

So please: open terminal window (or ssh to device), execute /export file=anynameyouwish, fetch file off device, open it with favourite text editor, redact any sensitive information (such as serial number or any passwords or public IP addresses) and copy-paste it in a post, enclosed in [__code] [/code] environment (the button on top of post edit window).

And, BTW, how’s routing on those 10Gbps set up? And does your main router know anything about additional IP subnet?

3

Hy, normally I’m a terminal guy too, but only familiar with Juniper Devices and Linux :slight_smile:
Sorry for the screens - I tought that would help more. Here is the output.

On the firewall there is a static route from LAN to MikrotikGW (192.168.178.250) which is connected to sfp1G on CRS317

interface bridge
add admin-mac=78:9A:18:4B:D1:8F auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=sfp-sfpplus2 ] name=sfp-sfpplus2-10G
set [ find default-name=sfp-sfpplus1 ] name=sfp1G
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/port
set 0 name=serial0
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=sfp1G
add bridge=bridge comment=defconf interface=sfp-sfpplus2-10G
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
add bridge=bridge comment=defconf interface=sfp-sfpplus5
add bridge=bridge comment=defconf interface=sfp-sfpplus6
add bridge=bridge comment=defconf interface=sfp-sfpplus7
add bridge=bridge comment=defconf interface=sfp-sfpplus8
add bridge=bridge comment=defconf interface=sfp-sfpplus9
add bridge=bridge comment=defconf interface=sfp-sfpplus10
add bridge=bridge comment=defconf interface=sfp-sfpplus11
add bridge=bridge comment=defconf interface=sfp-sfpplus12
add bridge=bridge comment=defconf interface=sfp-sfpplus13
add bridge=bridge comment=defconf interface=sfp-sfpplus14
add bridge=bridge comment=defconf interface=sfp-sfpplus15
add bridge=bridge comment=defconf interface=sfp-sfpplus16
/interface list member
add interface=ether1 list=WAN
add interface=sfp1G list=LAN
add interface=sfp-sfpplus2-10G list=LAN
add interface=sfp-sfpplus3 list=LAN
add interface=sfp-sfpplus4 list=LAN
add interface=sfp-sfpplus5 list=LAN
add interface=sfp-sfpplus6 list=LAN
add interface=sfp-sfpplus7 list=LAN
add interface=sfp-sfpplus8 list=LAN
add interface=sfp-sfpplus9 list=LAN
add interface=sfp-sfpplus10 list=LAN
add interface=sfp-sfpplus11 list=LAN
add interface=sfp-sfpplus12 list=LAN
add interface=sfp-sfpplus13 list=LAN
add interface=sfp-sfpplus14 list=LAN
add interface=sfp-sfpplus15 list=LAN
add interface=sfp-sfpplus16 list=LAN
/ip address
add address=192.168.178.250/24 comment=defconf interface=sfp1G network=\
    192.168.178.0
add address=192.168.88.250/24 interface=bridge network=192.168.88.0
/ip dns
set servers=1.1.1.1
/system clock
set time-zone-name=Europe/Vienna
/system note
set show-at-login=no
/system routerboard settings
set boot-os=router-os]

meanwhile i got it to work, and tested with dd / scp, if routing 10G works as expected - it does at the moment
and seems to bottleneck on RAID1 SSD, but I dont know if its best practice on Mikrotik or someone has input how do do it more efficent or better :slight_smile:

thanks in advance

4

A few (not so) minor details:

If a port is member of bridge (sfp1G in your case), then it really shouldn’t be used as interface (i.e. to set IP address on it) … bridge offers its own interface (inconveniently also named bridge, which you’re already using), member ports are in the same L2 broadcast domain (and that includes also bridge CPU-facing port). So move the address 192.168.178.250 to bridge interface.

CRS317 is also missing default route. This is either minor (if CRS is not supposed to route towards internet, then its only purpose is to give CRS access to internet … e.g. for software updates or for NTP) or major flaw (if CRS is supposed to route between 192.168.88.0/24 and internet via upstream gateway).

Regarding moving IP address 192.168.178.250/24 to bridge: currently device is still switching between all ports, including sfp1G. If you want to contain 192.168.88.0/24 in your 10Gbps network (you probably do, you don’t want broadcasts to escape to the rest of LAN), then you really should isolate sfp1G from other ports. There are several ways to do it:

  1. using horizon values on ports
  2. using VLAN
  3. removing sfp1G from bridge

The last one disables L3HW offload and since you don’t have any firewall running on CRS between both subnets, this is not what you want (CRS would revert to routing in CPU and CPU in this device really can’t route at 1Gbps).
I’m not sure about option #1 … it might disable L3HW as well.

So it leaves you with option #2. Now, you can configure VLANs on CRS and that doesn’t mean that any frame will ever exit any port tagged. Simply set sfp1G as access port of one VLAN and the rest of ports as access port of another VLAN. Then add a couple of VLAN interfaces and move IP addresses to those VLAN interfaces.
I suggest you to take opportunity to take ether1 off bridge and configure it as OOB management port … because re-configuring device’s layer 2 can easily mean loss of connectivity.

The changes regarding VLANs (relative to your current config) are something like this:

/interface/bridge/port
set [ find name=sfp1G ] pvid=10 frame-types=admit-only-untagged-and-priority-tagged
set [ find name=sfp-sfpplus2-10G ] pvid=20 frame-types=admit-only-untagged-and-priority-tagged
set [ find name=sfp-sfpplus3 ] pvid=20 frame-types=admit-only-untagged-and-priority-tagged
# same for the rest of sfp-sfpplus ports
/interface/bridge/vlan
# this sets brdige interface a tagged member of VLAN IDs with which ROS has to communicate
add bridge=bridge tagged=bridge vlan-ids=10
add bridge=bridge tagged=bridge vlan-ids=20
/interface/vlan
add name=LAN1G interface=bridge vlan-id=10
add name=LAN10G interface=bridge vlan-id=20

# Without first configuring ether1 as OOB management port ... and doing this config vvia ether1, 
# you'll loose management access performing next step. The loss should be temporary, you should be
# able to reconnect. If you can't, use winbox with MAC connectivity.
/ip address
set [ find address=192.168.178.250/24 ] interface=LAN1G
set [ find address=192.168.88.250/24 ] interface=LAN10G

# The last step. If not using ether1 OOB management access, you'll loose access (again).
# Again you should be able to reconnect afterwards.
/interface bridge
set [ find name=bridge ] vlan-filtering=yes

And some light ( :laughing: ) reading:

5

Hy and many thanks for your reply and effort.
Sorry for late answer but was busy with job, family, etc.
I will try these things tomorrow and come back tou you ASAP :slight_smile:

6

Hy there, so i began testing a bit around but there are a few problems left i have to solve.
The first one is, i configured everything as you told me, configured vlan tagging at my test debian11 and meanwhile Im able to ping
the gateway of vlan 20 from a machine in this vlan. However, if I enable vlan filtering like this:

/interface bridge
set [ find name=bridge ] vlan-filtering=yes

Im not able to ping the gatewy anymore and I dont know why, because without this option its working.
I found a few tutorials about this, but tbh - i cant solve the riddle. Any help on thjis I would highly appreciate

7

When bridge property vlan-filtering is set to no (default), then bridge doesn’t do anything with VLAN tags. So it doesn’t add VLAN tag on ingress if frame is untagged (pvid setting), it doesn’t remove tag on egress (port is set as untagged member of VLAN), nothing. Essentially bridge ports act as they were all hybrid ports (allowed both untagged frames and tagged frames, all VLANs allowed). And it’s important to remember that one of bridge personalities is CPU-facing bridge port which allows CPU to communicate with bridged networks. And bridge port also behaves as promiscuous hybrid port.
If you configure connected device (your debian host) with VLANs, that means that frames for those VLANs have VLAN tags attached. They pass ingress bridge port just fine and they also pass bridge (CPU-facing) port as well. There those vlan interfaces grab tagged frames and pass them (untagged) to IP layer of router.

As soon as you set bridge property vlan-filtering to yes, bridge starts to deal with (or mess with :wink:) VLAN tags. Now, if your CRS is confiugred according to the example I posted in my previous post, all tagged frames get dropped on ingress (because of “frame-types=admit-only-untagged-and-priority-tagged” setting). You really have to set up each CRS port according to how it’s going to be used. If VLAN tags will be added/removed by connected device (either a downstream switch or end device capable of VLAN tag handling), then that port has to be configured as trunk port, something like this:

/interface/bridge/port
set [ find name=sfp-sfpplus3 ] pvid=1 frame-types=admit-only-vlan-tagged ingress-filtering=yes

# the next part is impossible to write as a working recipe without knowing current setting of "tagged" property.
# I'm not going into details about it at this point
/interface/bridge/vlan
set [ find vlan-ids=20 ] tagged=bridge,sfp-sfpplus3

The ingress-filtering=yes setting instructs bridge to check VLAN ID of ingress frames and drop all frames with VLAN IDs other than one of VIDs where port is member of (as configured under /interface/bridge/vlan. Even without it, a two way communication is not possible (because frames with VID of non-member VLANs won’t egress that port), so it’s a security feature, preventing some malicious host from injecting frames to “forbidden” VLAN. BTW, same setting is completely valid even for hybrid and access ports (the check is done after PVID setting is taken into account).

N.b.: setting of pvid on true trunk ports (such as sfp-sfpplus3 in example above) is meaningless, I included particular setting in the command expressly to set it back to default value (from pvid=20 from my example in previous post).