If device is going to be used as internal core router, then you can (or should) remove all config under /ip firewall (and sub-tree).
Hi mkx thx for response first 
if i disable all irules and nat i have no connection to firewall plus internet
Couldn’t remove Firewall rule <> Connot remove builtin (6) says on some and let not remove it. Special dummy rules this
There are two possibilities:
- keep firewall config as is, but you have to keep using NAT on core router for traffic towards firewall (I assume it’s behind ether1).
This means firewall will keep seeing router’s (dynamic) IP address as source for all traffic - configure number of static routes on firewall … direct all internal subnets using router’s (dynamic) IP address as gateway.
In this case having router dynamic address on any of interfaces is not acceptable.
If I understand you right, option #1 is not an option, so you’ll have to assign static address on “WAN” interface of router and add static routes on firewall. And verify that firewall does whatever it needs to do (e.g. NAT) also for subnets behind core router.
My firewall have the ip 172.16.16.1 and mikrotik have fixep ip of 172.16.16.2 and all the routes have to mikrotik on firewall side are configured so routes going to 172.16.16.2 Mikrotik have default route to 172.16.16.1 to firewall. This is the right way or?
ROS won’t let you remove the dummy rule (which is for counting fasttracked traffic), but will disappear when you reboot the router if there is no “normal” fast track rule.
Re. static routing: right. If your firewall was Mikrotik, it would need the following routes:
/ip route
add dst-address=172.16.1.1/24 gateway=172.16.16.2
add dst-address=172.16.5.1/24 gateway=172.16.16.2
add dst-address=172.16.30.1/24 gateway=172.16.16.2
add dst-address=172.16.40.1/24 gateway=172.16.16.2
add dst-address=172.16.10.1/24 gateway=172.16.16.2
add dst-address=172.16.50.1/24 gateway=172.16.16.2
add dst-address=172.16.60.1/24 gateway=172.16.16.2
add dst-address=172.16.70.1/24 gateway=172.16.16.2
add dst-address=172.16.80.1/24 gateway=172.16.16.2
add dst-address=172.16.90.1/24 gateway=172.16.16.2
add dst-address=172.16.20.1/24 gateway=172.16.16.2
add dst-address=172.16.100.1/24 gateway=172.16.16.2
If it’s not mikrotik, adjust the commands above accordingly.
Even easier would be to run a BGP/OSPF between core router and firewall if firewall can do it, this way core router would push routes to firewall automatically (e.g. when you add another LAN subnet, firewall routes would be updated automatically). I’m not intimate with routing protocols so if you decide to go that way, somebody else will have to jump in with some guidance.
My firewall is on opnsense it is like pfsense need to make this ospf seems good idea than i check if its working
This is exactly what is configured on the Firewall all routes and gateways are pingable from firewall side let say my monitoring server are monitoring my firewall but firewall gets always 172.16.16.2 adresses whatever is going out to wan or firewall side so Mikrotik is natting but i need the real adress what connecting to this service so the server ip need to be here.
I’m pretty sure that if you set static IP address on ether1 (the last configuration you showed had DHCP client running on that interface) nad then you remove all /ip firewall setup, then things should work … perhaps a good reboot of CRS has to be performed to get rid of any non-removeable entries in firewall section. The most important being getting rid of the /ip firewall nat entry.
Later on you might want to add a few firewall filter rules to protect router. That’s the rules for chain=input. But nothing like the current rules you have, they are canibalized fragment of default SOHO config which doesn’t really apply in your case.
Sry mate forget to mention, this is the config right now.
# sep/09/2021 15:43:47 by RouterOS 6.48.4
# software id = xxxx
#
# model = CCR2004-1G-12S+2XS
# serial number = xxxxx
/interface bridge
add name=Trunk_bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] comment=WAN
set [ find default-name=sfp-sfpplus1 ] l2mtu=9578 mtu=9000
set [ find default-name=sfp-sfpplus2 ] l2mtu=9578 mtu=9000
set [ find default-name=sfp-sfpplus3 ] l2mtu=9578 mtu=9000
set [ find default-name=sfp-sfpplus4 ] l2mtu=9578 mtu=9000
set [ find default-name=sfp-sfpplus5 ] l2mtu=9578 mtu=9000
set [ find default-name=sfp-sfpplus6 ] l2mtu=9578 mtu=9000
set [ find default-name=sfp-sfpplus7 ] l2mtu=9578 mtu=9000
set [ find default-name=sfp-sfpplus8 ] l2mtu=9578 mtu=9000
set [ find default-name=sfp-sfpplus9 ] l2mtu=9578 mtu=9000
set [ find default-name=sfp-sfpplus10 ] l2mtu=9578 mtu=9000
set [ find default-name=sfp-sfpplus11 ] l2mtu=9578 mtu=9000
set [ find default-name=sfp-sfpplus12 ] l2mtu=9578 mtu=9000
set [ find default-name=sfp28-1 ] l2mtu=9578 mtu=9000
set [ find default-name=sfp28-2 ] l2mtu=9578 mtu=9000
/interface vlan
add interface=Trunk_bridge mtu=9000 name=vlan5 vlan-id=5
add interface=Trunk_bridge mtu=9000 name=vlan10 vlan-id=10
add interface=Trunk_bridge mtu=9000 name=vlan20 vlan-id=20
add interface=Trunk_bridge mtu=9000 name=vlan30 vlan-id=30
add interface=Trunk_bridge mtu=9000 name=vlan40 vlan-id=40
add interface=Trunk_bridge mtu=9000 name=vlan50 vlan-id=50
add interface=Trunk_bridge mtu=9000 name=vlan60 vlan-id=60
add interface=Trunk_bridge mtu=9000 name=vlan70 vlan-id=70
add interface=Trunk_bridge mtu=9000 name=vlan80 vlan-id=80
add interface=Trunk_bridge mtu=9000 name=vlan90 vlan-id=90
add interface=Trunk_bridge mtu=9000 name=vlan100 vlan-id=100
/interface bonding
add comment="T6202 bond" mtu=9000 name=bonding1 slaves=\
sfp-sfpplus9,sfp-sfpplus10
add comment="t620 bond" mtu=9000 name=bonding2 slaves=\
sfp-sfpplus7,sfp-sfpplus8
add comment="t320 bond" mtu=9000 name=bonding3 slaves=\
sfp-sfpplus5,sfp-sfpplus6
/interface list
add name=WAN
add name=LAN
add name=Gast
add name=Server
add name=WLAN
add name=IOT
add name=SECURITY
add name=IPMI
add name=MGMT
add name=DMZ
add name=Cluster
add name=HA
/interface lte apn
set [ find default=yes ] ip-type=ipv4-ipv6
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool_ipmi ranges=172.16.5.100-172.16.5.254
add name=dhcp_pool_security ranges=172.16.30.100-172.16.30.254
add name=dhcp_pool_IoT ranges=172.16.40.100-172.16.40.254
add name=dhcp_pool_mgmt ranges=172.16.10.100-172.16.10.254
add name=dhcp_pool_guest ranges=172.16.50.2-172.16.50.254
add name=dhcp_pool_dmz ranges=172.16.60.100-172.16.60.254
add name=dhcp_pool_ha ranges=172.16.70.100-172.16.70.254
add name=dhcp_pool_cluster ranges=172.16.80.100-172.16.80.254
add name=dhcp_pool_wlan ranges=172.16.90.50-172.16.90.254
add name=dhcp_pool_LAN ranges=172.16.100.2-172.16.100.254
add name=dhcp_pool_server ranges=172.16.20.11-172.16.20.20
/ip dhcp-server
add address-pool=dhcp_pool_ipmi disabled=no interface=vlan5 lease-time=2h \
name=dhcp_ipmi
add address-pool=dhcp_pool_security disabled=no interface=vlan30 lease-time=\
2h name=dhcp_security
add address-pool=dhcp_pool_IoT disabled=no interface=vlan40 lease-time=2h \
name=dhcp_IoT
add address-pool=dhcp_pool_mgmt disabled=no interface=vlan10 lease-time=2h \
name=dhcp_mgmt
add address-pool=dhcp_pool_guest disabled=no interface=vlan50 lease-time=2h \
name=dhcp_guest
add address-pool=dhcp_pool_dmz disabled=no interface=vlan60 lease-time=2h \
name=dhcp_dmz
add address-pool=dhcp_pool_ha disabled=no interface=vlan70 lease-time=2h \
name=dhcp_ha
add address-pool=dhcp_pool_cluster disabled=no interface=vlan80 lease-time=2h \
name=dhcp_cluster
add address-pool=dhcp_pool_wlan disabled=no interface=vlan90 lease-time=8h \
name=dhcp_wlan
add address-pool=dhcp_pool_LAN disabled=no interface=vlan100 lease-time=8h \
name=dhcp_LAN
add address-pool=dhcp_pool_server disabled=no interface=vlan20 lease-time=8h \
name=dhcp_server
/interface bridge port
add bridge=Trunk_bridge interface=sfp-sfpplus1
add bridge=Trunk_bridge interface=sfp-sfpplus2
add bridge=Trunk_bridge interface=sfp-sfpplus3
add bridge=Trunk_bridge interface=sfp-sfpplus4
add bridge=Trunk_bridge interface=sfp-sfpplus11
add bridge=Trunk_bridge interface=sfp28-1
add bridge=Trunk_bridge interface=sfp28-2
add bridge=Trunk_bridge interface=sfp-sfpplus12
add bridge=Trunk_bridge interface=bonding1
add bridge=Trunk_bridge interface=bonding2
add bridge=Trunk_bridge interface=bonding3
/interface bridge vlan
add bridge=Trunk_bridge tagged="Trunk_bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfp\
plus3,sfp-sfpplus4,sfp-sfpplus12,bonding1,bonding2,bonding3,sfp28-1,sfp28-\
2,sfp-sfpplus11" vlan-ids=10
add bridge=Trunk_bridge tagged="Trunk_bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfp\
plus3,sfp-sfpplus4,bonding3,sfp-sfpplus11,sfp-sfpplus12,bonding1,bonding2,\
sfp28-1,sfp28-2" vlan-ids=5
add bridge=Trunk_bridge tagged="Trunk_bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfp\
plus3,sfp-sfpplus4,bonding3,sfp-sfpplus9,sfp-sfpplus10,sfp-sfpplus11,sfp-s\
fpplus12,sfp28-1,sfp28-2,bonding1,bonding2" vlan-ids=20
add bridge=Trunk_bridge tagged="Trunk_bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfp\
plus3,sfp-sfpplus4,bonding3,sfp-sfpplus11,sfp-sfpplus12,bonding1,bonding2,\
sfp28-1,sfp28-2" vlan-ids=30
add bridge=Trunk_bridge tagged="Trunk_bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfp\
plus3,sfp-sfpplus4,bonding3,sfp-sfpplus11,sfp-sfpplus12,bonding1,bonding2,\
sfp28-1,sfp28-2" vlan-ids=40
add bridge=Trunk_bridge tagged="Trunk_bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfp\
plus3,sfp-sfpplus4,bonding3,sfp-sfpplus11,sfp-sfpplus12,bonding1,bonding2,\
sfp28-1,sfp28-2" vlan-ids=50
add bridge=Trunk_bridge tagged="Trunk_bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfp\
plus3,sfp-sfpplus4,bonding3,sfp-sfpplus11,sfp-sfpplus12,bonding1,bonding2,\
sfp28-1,sfp28-2" vlan-ids=60
add bridge=Trunk_bridge tagged="Trunk_bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfp\
plus3,sfp-sfpplus4,bonding3,sfp-sfpplus11,sfp-sfpplus12,bonding1,bonding2,\
sfp28-1,sfp28-2" vlan-ids=70
add bridge=Trunk_bridge tagged="Trunk_bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfp\
plus3,sfp-sfpplus4,bonding3,sfp-sfpplus11,sfp-sfpplus12,bonding1,bonding2,\
sfp28-1,sfp28-2" vlan-ids=80
add bridge=Trunk_bridge tagged="Trunk_bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfp\
plus3,sfp-sfpplus4,bonding3,sfp-sfpplus11,sfp-sfpplus12,bonding1,bonding2,\
sfp28-1,sfp28-2" vlan-ids=90
add bridge=Trunk_bridge tagged="Trunk_bridge,sfp-sfpplus1,sfp-sfpplus2,sfp-sfp\
plus3,sfp-sfpplus4,bonding3,sfp-sfpplus11,sfp-sfpplus12,sfp28-1,sfp28-2,bo\
nding1,bonding2" vlan-ids=100
/interface list member
add interface=ether1 list=WAN
add interface=vlan100 list=LAN
add interface=vlan50 list=Gast
add interface=vlan30 list=SECURITY
add interface=vlan90 list=WLAN
add interface=vlan40 list=IOT
add interface=vlan5 list=IPMI
add interface=vlan60 list=DMZ
add interface=vlan80 list=Cluster
add interface=vlan70 list=HA
/ip address
add address=172.16.5.1/24 interface=vlan5 network=172.16.5.0
add address=172.16.30.1/24 interface=vlan30 network=172.16.30.0
add address=172.16.40.1/24 interface=vlan40 network=172.16.40.0
add address=172.16.10.1/24 interface=vlan10 network=172.16.10.0
add address=172.16.50.1/24 interface=vlan50 network=172.16.50.0
add address=172.16.60.1/24 interface=vlan60 network=172.16.60.0
add address=172.16.70.1/24 interface=vlan70 network=172.16.70.0
add address=172.16.80.1/24 interface=vlan80 network=172.16.80.0
add address=172.16.90.1/24 interface=vlan90 network=172.16.90.0
add address=172.16.20.1/24 interface=vlan20 network=172.16.20.0
add address=172.16.100.1/24 interface=vlan100 network=172.16.100.0
add address=172.16.16.2/30 interface=ether1 network=172.16.16.0
/ip dhcp-server network
add address=172.16.1.0/24 gateway=172.16.1.1
add address=172.16.5.0/24 gateway=172.16.5.1
add address=172.16.10.0/24 gateway=172.16.10.1
add address=172.16.20.0/24 gateway=172.16.20.1
add address=172.16.30.0/24 gateway=172.16.30.1
add address=172.16.40.0/24 gateway=172.16.40.1
add address=172.16.50.0/24 gateway=172.16.50.1
add address=172.16.60.0/24 gateway=172.16.60.1
add address=172.16.70.0/24 gateway=172.16.70.1
add address=172.16.80.0/24 gateway=172.16.80.1
add address=172.16.90.0/24 dns-server=172.16.90.1,172.16.16.1 gateway=\
172.16.90.1
add address=172.16.100.0/24 dns-server=172.16.20.5,172.16.100.1 domain=\
xxx.local gateway=172.16.100.1
/ip dns
set allow-remote-requests=yes servers=172.16.16.1,1.1.1.1
/ip firewall filter
add action=accept chain=input connection-state=established,related,untracked
add action=drop chain=input comment=" defconf: drop invalid" \
connection-state=invalid disabled=yes
add action=accept chain=input protocol=icmp
add action=drop chain=input disabled=yes in-interface-list=!LAN
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
connection-state=established,related
add action=accept chain=forward comment=\
"defconf: accept established,related, untracked" connection-state=\
established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
connection-state=invalid disabled=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set sip disabled=yes
set pptp disabled=yes
set dccp disabled=yes
/ip route
add distance=1 gateway=172.16.16.1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=Europe/Vienna
/system identity
set name=cr1
/system leds
set 18 disabled=yes
set 19 disabled=yes
set 20 disabled=yes
set 21 disabled=yes
set 22 disabled=yes
set 23 disabled=yes
set 24 disabled=yes
set 25 disabled=yes
/system ntp client
set enabled=yes primary-ntp=xxxxxxxx secondary-ntp=xxxxxxxx
/system ntp server
set broadcast=yes enabled=yes
/system resource irq rps
set ether1 disabled=yes
/tool mac-server
set allowed-interface-list=MGMT
/tool mac-server mac-winbox
set allowed-interface-list=MGMT
/tool mac-server ping
set enabled=no
and in Firewall the routes for them
172.16.50.0/24 cr1 - 172.16.16.2
172.16.10.0/24 cr1 - 172.16.16.2
172.16.30.0/24 cr1 - 172.16.16.2
172.16.80.0/24 cr1 - 172.16.16.2
172.16.40.0/24 cr1 - 172.16.16.2
172.16.60.0/24 cr1 - 172.16.16.2
172.16.90.0/24 cr1 - 172.16.16.2
everything is perfect working except
failed to accept an incoming connection: connection from “172.16.16.2” rejected, allowed hosts: “172.16.20.3”
because monitoring servers ip is 172.16.16.2 because of natting it should be the original ip 172.16.20.3
So the relevant (if I didn’t forget to include anything else) configuration part, which does things you don’t want to see, is this:
/interface list
add name=WAN
/interface list member
add interface=ether1 list=WAN
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN
Essentially whatever leaves router through ether1 (towards your firewall), regardless the source (any of VLAN subnets), and including traffic originating router itself (any of own IP addresses), gets SRC NATed. And per your discussion so far you really don’t want to have SRC NAT.
So either remove NAT rule or remove ether1 from WAN interface list. As I mentioned earlier, you’ll have to reboot router so that connection state clears (configuration changes often don’t affect existing connections).
The both alternatives from previous sentence are not exactly identical, but the effect with your particular firewall filter rules will be the same. I still think you should remove all filter rules for chain=forward unless you really want to filter traffic between different subnets. If you don’t care about that, and additionally you don’t want/need to protect router from LAN users, you can remove all filter rules (including those for chain=input) … if you have any filter rules (regardless for which chain), firewall will perform connection tracking and that is quite a burden on CPU. Even though CCR2004 has a pretty powerful CPU, routing speed is not exactly wire-speed (real-life performance might peak around 10Gbps on all ports combined) and removing part of burden will definitely help.
Hi mate yes i just wanna route between those local subnets as performance as max as possible for storage area network and for my home lab have 4 proxmox server in a cluster so i decide to get as maximum routing performance 20 gbit and up would be fine. No firewall in plan mybe i try later to cut vlans from talking each other with rules but first need to be working ether1 side and many many thx for your suggestions I will try it now hope not loose internet again.
I disabled the all firewall rules removed NAT restart no connections to firewall side and to Internet. Put the NAT again back all working but not understand why withount nat not works
I strongly suspect that some bit of configuration is off on firewall. Can you perform traceroute from firewall towards one of LAN servers to see whether packets actually reach as far as CCR?
BTW, even though for now you seem to need NAT, remove firewall filter rules so that they don’t interfere with traffic.
Another test is from router’s side:
/tool traceroute 172.16.16.1
/tool traceroute 172.16.16.1 src-address=172.16.50.1
The first line should succeed since all will stay inside “routing subnet”. The second one (src-address should be one of router’s addresses otger than 172.16.16.2) will succeed if firewall is correctly configured for the rest of LAN subnets. If the second one succeeds while you can’t access firewall from other hosts in same VLAN, then …
From firewall
traceroute to 172.16.20.3 (172.16.20.3) from 172.16.16.1, 18 hops max, 40 byte packets
1 172.16.16.2 0.303 ms 0.204 ms 0.160 ms
2 172.16.20.3 1.372 ms 3.121 ms 2.303 ms
Right. So from connectivity point of view everything works without NAT. Which means you should review firewall rules on your firewall … does it allow input (ping) and forward from LAN interface where src address is not covered by LAN interface address/netmask?
oh this was with nat
But wasn’t needed. SRC-NAT only does the trick for traffic leaving router (through that particular interface) of connections marked for NAT and only un-NATs traffic identified as being part of nat-ed connection (or “connection” in case of stateless protocols such as ICMP or UDP). And only marks connection for NATing when it’s new.
here the rules on firewal LAN where the Mikrotik is connected
Protocol Source Port Destination Port Gateway Schedule Description
Automatically generated rules
IPv4 * LAN net * * * * * Default allow LAN to any rule
IPv4 * LocalNetwork * * * * * Allow Local LAN to IN any rule
IPv4 * LocalNetwork * * * * * Allow Local LAN to OUT any rule
LocalNetwork is Alias where the local Networks are
I’ve no idea about how to configure opensense, sorry.
On firewall lan site is all to all allowed and as the says it is expecting the server ip but get Mikrotik ip
I’ll write it once again: as long as you have that NAT rule enabled, firewall won’t see anything but router’s address. However, when you disable (or remove) that rule, nothing in router’s config blocks traffic from flowing between firewall and any of subnets. So if you remove NAT rule and traffic doesn’t flow while you can traceroute from firewall to LAN server (but not in the other direction), then it’s almost 100% something on firewall blocking traffic (not routing config).
Did you try traceroute from router the way I explained in my post #32 above?