Community discussions

MikroTik App
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

src-nat stops working after some time

Mon May 16, 2022 7:52 pm

Hi!

Scenario:

Got 5 static IPs (/29) from my ISP. All come in via ether1-WAN (from ISPs modem in bridge mode) on my HEX-S (v7.2.3), let's call them addrA, addrB, addrC, addrD, addrE

got some VMs configured on a separate vlan, few matching dst-nat rules, and a few completely generic src-nat rules like so:
add chain=srcnat src-address=10.1.10.234 out-interface=ether1-WAN action=src-nat to-addresses=addrA
and so on, with every VM having a different public IP.

When I restart my router, all these src-nat rules work (in the VM console I check public IP address and it matches the src-nat); I can change them to any public IP address configured on ether1-WAN and after a fwe moments the assigned public address is reported (from ifconfig.io) as the one I set in the rule.

BUT.

after a few minutes, all VMs report only addrE. Changing the src-nat rules to any other address removes outbound internet access on these VMs.

I'm completely baffled as to why it behaves like that.

Of course, default masquerade works, defaulting to addrA.

I have no custom routing addedd (only manually added ISP's GW address), no mangle rules, nothing out of the ordinary.

Any help will be greatly appreciated.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: src-nat stops working after some time

Tue May 17, 2022 1:00 am

The rule you posted is fine, it must be something else (maybe if you post your whole config, someone will see it). It seems like some communication problem between those other addresses and ISP's router. The weird part is that it works for a while and then it stops.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Wed May 18, 2022 11:43 am

this is my firewall export.

The various subnets are VLANs, but the communication both ways between router/subnets is fine, so I omitted that part of config.
/ip firewall address-list
add address=addrA list=INEA
add address=addrB list=INEA
add address=addrC list=INEA
add address=addrD list=INEA
add address=addrE list=INEA
add address=10.1.1.0/24 list=LANs
add address=10.1.10.0/24 list=LANs
add address=10.1.20.0/24 list=LANs
add address=10.1.30.0/24 list=LANs
/ip firewall connection tracking
set enabled=yes
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=accept chain=forward comment="Accept DST NAT" \
    connection-nat-state=dstnat connection-state=new
add action=accept chain=forward comment="Accept SRC NAT" \
    connection-nat-state=srcnat connection-state=""
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid log=yes log-prefix=invalid
add action=accept chain=input comment="defconf: accept ICMP" disabled=yes \
    protocol=icmp
add action=accept chain=input comment="SSH HEXs" dst-address=addrE \
    dst-port=22222 protocol=tcp
add action=accept chain=input comment=WireGuard dst-address=addrE \
    dst-port=13231,13232 protocol=udp
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid log=yes log-prefix=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface=ether1-WAN
/ip firewall nat
add action=src-nat chain=srcnat comment="Anchorage SRC (email)" \
    out-interface=ether1-WAN src-address=10.1.10.234 to-addresses=\
    addrA
add action=src-nat chain=srcnat comment="Buckland SRC (eto)" out-interface=\
    ether1-WAN src-address=10.1.10.235 to-addresses=addrB
add action=src-nat chain=srcnat comment="Coldbay SRC (Adam)" out-interface=\
    ether1-WAN src-address=10.1.11.0/24 to-addresses=addrC
add action=src-nat chain=srcnat comment="Deering SRC (git)" out-interface=\
    ether1-WAN src-address=10.1.10.237 to-addresses=addrD
add action=src-nat chain=srcnat comment="LAN SRC" out-interface=ether1-WAN \
    src-address=10.1.1.0/24 to-addresses=addrE
add action=src-nat chain=srcnat comment="LAN SRC" out-interface=ether1-WAN \
    src-address=10.1.10.0/24 to-addresses=addrE
add action=src-nat chain=srcnat comment="LAN SRC" out-interface=ether1-WAN \
    src-address=10.1.20.0/24 to-addresses=addrE
add action=masquerade chain=srcnat comment="defconf: masquerade" disabled=yes \
    ipsec-policy=out,none out-interface=ether1-WAN
add action=dst-nat chain=dstnat comment="Anchorage E-mail" dst-address=\
    addrA dst-port=80,443,25,587,110,143,993,995 protocol=tcp \
    to-addresses=10.1.10.234
add action=dst-nat chain=dstnat comment="Anchorage SSH" dst-address=\
    addrA dst-port=22222 protocol=tcp to-addresses=10.1.10.234 \
    to-ports=22
add action=dst-nat chain=dstnat comment="Buckland HTTPS" dst-address=\
    addrB dst-port=80,443 protocol=tcp to-addresses=10.1.10.235
add action=dst-nat chain=dstnat comment="Buckland SSH" dst-address=\
    addrB dst-port=22222 protocol=tcp to-addresses=10.1.10.235 \
    to-ports=22
add action=dst-nat chain=dstnat comment="Coldbay Adam" dst-address=\
    addrC dst-port=8444 protocol=tcp to-addresses=10.1.11.10
add action=dst-nat chain=dstnat comment="Coldbay Adam" dst-address=\
    addrC dst-port=22,80,443,1922,25565 protocol=tcp to-addresses=\
    10.1.11.11
add action=dst-nat chain=dstnat comment="Deering GIT, HTTPS" dst-address=\
    addrD dst-port=22,80,443 protocol=tcp to-addresses=10.1.10.237
add action=dst-nat chain=dstnat comment=CCTV disabled=yes dst-port=8000 \
    in-interface=*9 protocol=tcp to-addresses=10.1.10.200 to-ports=8000
add action=dst-nat chain=dstnat comment="R720 PVE webui" disabled=yes \
    dst-port=38006 in-interface=*9 protocol=tcp to-addresses=10.1.1.11 \
    to-ports=8006
add action=dst-nat chain=dstnat comment="Transmission RPC" dst-address=\
    addrE dst-port=59091 protocol=tcp to-addresses=10.1.10.2 \
    to-ports=9091
add action=dst-nat chain=dstnat comment="Transmission TCP" dst-address=\
    addrE dst-port=51413 protocol=tcp to-addresses=10.1.10.2 \
    to-ports=51413
add action=dst-nat chain=dstnat comment="Transmission UDP" dst-address=\
    addrE dst-port=51413 protocol=udp to-addresses=10.1.10.2 \
    to-ports=51413
add action=dst-nat chain=dstnat comment="Elim TRAEFIK" dst-address=\
    addrE dst-port=80,443,8080,8088 in-interface=ether1-WAN \
    protocol=tcp to-addresses=10.1.10.3
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: src-nat stops working after some time

Wed May 18, 2022 2:49 pm

This doesn't explain it. If it didn't work at all, I'd say that you assigned only addrE to router and not others, and so ISP's router can't reach them. But that can't be the case if it works at first.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Wed May 18, 2022 4:48 pm

I think I'm going to grab another MT device with software 6.x, replicate this config and check if it works. I've used this exact scenario countless times with setups having multiple public IPs and it just worked.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Sat May 21, 2022 2:36 pm

Of course I didn't have time to check with another device, but I went a step back and started testing from step 1.

And it's weird :)

I'm doing ping from mikrotik, checking all my public IPs with `src-address`.

And only 3 of 5 are working. I'm confused even more.

I've called my ISP and there were no changes in any way on their side. Nada.
 
tangent
Forum Guru
Forum Guru
Posts: 1330
Joined: Thu Jul 01, 2021 3:15 pm
Contact:

Re: src-nat stops working after some time

Sat May 21, 2022 5:11 pm

Why are you using src NAT with public static IPs at all? Assign the IPs to the VMs and be happy.

If you want the VMs on a private VLAN as well, that’s fine. Server OSes have supported multiple interfaces and multiple IPs per interface since approximately forever.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Thu May 26, 2022 7:33 pm

Why are you using src NAT with public static IPs at all? Assign the IPs to the VMs and be happy.
Because I have a few hosts that are on a VLAN and I want all of them to respond with a specific SRC address. I know I can set the public IPs to hosts directly.

Other than that, I found time to dig into this issue and I'm 99% it's a bug in MT.

I've reset firmware to default settings, the behavior is the same. src-nat to addresses A, B, E is working, addresses C and D are not. And I found something.

For some weird reason, FastTrack isn't enabling for these not working addresses.

This is what I did: Reset the device to default configuration. Setup public addresses, ISP GW etc.
Then I created the same generic src-nat rule and ran ping from the src-natted host.

Observing the ICMP connections in IP->Firewall->Connections with src-address set to my src-natted host, shows that FastTrack flag is enabled only for outgoing addresses A, B, E, not C, D.

Worth noting is that I modify the src-nat rule to another address, not creating any new rules etc.

Side note, I've plugged my laptop directly to IPS modem and all public addresses are working as expected. The problem shows with mikrotik.

So, should I escalate this to MT support?
 
tangent
Forum Guru
Forum Guru
Posts: 1330
Joined: Thu Jul 01, 2021 3:15 pm
Contact:

Re: src-nat stops working after some time

Fri May 27, 2022 5:32 am

Your "action=fasttrack-connection" line is below the srcnat lines in your firewall, so the "action=accept" on them bypasses the fasttrack action for NAT connections. I suspect this is what distinguishes your C/D address schemes, but I didn't dig deeply in to check.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Fri May 27, 2022 12:31 pm

If that mattered, it would behave the same for all 5 addresses, not only 2 of them.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Fri May 27, 2022 6:10 pm

Ok, it gets weirder more and more.

I took more steps backs and started checking if the public IPs are reachable (ping) from outside.

When connected to my laptop (all 5 IPs on the ethernet interface) all 5 IPs are reachable from the internet.

Connected to mikrotik, at most 3 of them work. After last restart, only one.
 
User avatar
k6ccc
Forum Guru
Forum Guru
Posts: 1490
Joined: Fri May 13, 2016 12:01 am
Location: Glendora, CA, USA (near Los Angeles)
Contact:

Re: src-nat stops working after some time

Fri May 27, 2022 6:31 pm

There is something amiss all right. You only posted a part of your configuration earlier. Please post all of your configuration - /export hide-sensitive file=any-filename-you-wish
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Fri May 27, 2022 7:38 pm

Attached
# may/27/2022 18:31:08 by RouterOS 7.2.3
# software id = TRGR-BPA8
#
# model = RB760iGS
# serial number = A36A0B8BDDB0
/caps-man channel
add band=2ghz-b/g control-channel-width=20mhz name=24g
add band=5ghz-a/n/ac name=5g
/interface bridge
add admin-mac=C4:AD:34:E6:00:07 auto-mac=no comment=defconf \
    ingress-filtering=no name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] arp=proxy-arp l2mtu=1598 name=ether1-WAN
set [ find default-name=ether2 ] name=ether2-SG1024DE
set [ find default-name=ether4 ] name=ether4-SG108PE
set [ find default-name=ether5 ] name=ether5-RB951
/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard1
add listen-port=13232 mtu=1420 name=wireguard2
/interface vlan
add interface=bridge name=vlan10-Public vlan-id=10
add interface=bridge name=vlan11-Adam vlan-id=11
add interface=bridge name=vlan20-AP vlan-id=20
add interface=bridge name=vlan30-IOT vlan-id=30
/caps-man security
add authentication-types=wpa2-psk name=ap
add authentication-types=wpa2-psk encryption=aes-ccm group-encryption=aes-ccm \
    name=iot
/caps-man configuration
add channel=24g country=poland datapath.bridge=bridge .local-forwarding=yes \
    .vlan-id=20 .vlan-mode=use-tag installation=indoor name=public \
    security=ap ssid=public
add channel=24g country=poland datapath.bridge=bridge .local-forwarding=yes \
    .vlan-id=30 .vlan-mode=use-tag installation=indoor name=iot security=iot \
    ssid=oz14iot
add channel=5g country=poland datapath.bridge=bridge .local-forwarding=yes \
    .vlan-id=20 .vlan-mode=use-tag installation=indoor name=public-5g \
    security=ap ssid="public 5G"
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=pool-LAN ranges=10.1.1.190-10.1.1.199
add name=pool-AP ranges=10.1.20.100-10.1.20.253
add name=pool-Public ranges=10.1.10.200-10.1.10.253
add name=pool-IOT ranges=10.1.30.100-10.1.30.253
add name=dhcp_pool5 ranges=10.1.11.10-10.1.11.19
/ip dhcp-server
add address-pool=pool-LAN interface=bridge name=dhcp-LAN
add address-pool=pool-Public interface=vlan10-Public name=dhcp-Public
add address-pool=pool-AP interface=vlan20-AP name=dhcp-AP
add address-pool=pool-IOT interface=vlan30-IOT name=dhcp-IOT
add address-pool=dhcp_pool5 interface=vlan11-Adam name=dhcp-Adam
/port
set 0 name=serial0
/routing bgp template
set default disabled=no output.network=bgp-networks
/routing ospf instance
add disabled=no name=default-v2
/routing ospf area
add disabled=yes instance=default-v2 name=backbone-v2
/user group
set full policy="local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,pas\
    sword,web,sniff,sensitive,api,romon,dude,rest-api"
/caps-man manager
set enabled=yes
/caps-man provisioning
add action=create-dynamic-enabled hw-supported-modes=a master-configuration=\
    public-5g
add action=create-dynamic-enabled hw-supported-modes=g master-configuration=\
    public slave-configurations=iot
/interface bridge port
add bridge=bridge ingress-filtering=no interface=ether3
add bridge=bridge ingress-filtering=no interface=ether4-SG108PE
add bridge=bridge ingress-filtering=no interface=ether5-RB951
add bridge=bridge interface=ether2-SG1024DE
/ip firewall connection tracking
set enabled=yes
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ip settings
set accept-source-route=yes
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=8192
/interface bridge vlan
add bridge=bridge tagged=bridge,ether4-SG108PE,ether5-RB951,ether2-SG1024DE \
    vlan-ids=10,20,30,11
/interface list member
add comment=defconf interface=bridge list=LAN
add interface=vlan20-AP list=LAN
add interface=vlan30-IOT list=LAN
add interface=vlan10-Public list=LAN
add interface=ether1-WAN list=WAN
add interface=wireguard1 list=LAN
add interface=vlan11-Adam list=LAN
add interface=wireguard2 list=LAN
/ip address
add address=10.1.1.254/24 interface=bridge network=10.1.1.0
add address=10.1.20.254/24 interface=vlan20-AP network=10.1.20.0
add address=10.1.10.254/24 interface=vlan10-Public network=10.1.10.0
add address=10.1.30.254/24 interface=vlan30-IOT network=10.1.30.0
add address=172.16.0.1/29 interface=wireguard1 network=172.16.0.0
add address=addrE/29 comment="ELIM (piwnica)" interface=ether1-WAN \
    network=ISP_NETWORK
add address=addrA/29 comment=ANCHORAGE interface=ether1-WAN network=\
    ISP_NETWORK
add address=addrB/29 comment=BUCKLAND interface=ether1-WAN network=\
    ISP_NETWORK
add address=addrC/29 comment=COLDBAY interface=ether1-WAN network=\
    ISP_NETWORK
add address=addrD/29 comment=DEERING interface=ether1-WAN network=\
    ISP_NETWORK
add address=10.1.11.254/24 interface=vlan11-Adam network=10.1.11.0
add address=172.16.11.1/29 interface=wireguard2 network=172.16.11.0
/ip cloud
set ddns-enabled=yes
/ip dhcp-client
add comment=defconf disabled=yes interface=ether1-WAN
/ip dhcp-server network
add address=10.1.1.0/24 comment=LAN dns-server=10.1.1.254 domain=oz gateway=\
    10.1.1.254 ntp-server=10.1.1.254
add address=10.1.10.0/24 comment="Public servers" dns-server=10.1.10.254 \
    domain=public gateway=10.1.10.254
add address=10.1.11.0/24 gateway=10.1.11.254
add address=10.1.20.0/24 comment=AP dns-server=10.1.20.254 domain=wifi \
    gateway=10.1.20.254
add address=10.1.30.0/24 comment=IOT dns-server=10.1.30.254 domain=iot \
    gateway=10.1.30.254
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip firewall address-list
add address=addrA list=INEA
add address=addrB list=INEA
add address=addrC list=INEA
add address=addrD list=INEA
add address=addrE list=INEA
add address=10.1.1.0/24 disabled=yes list=LANs
add address=10.1.10.0/24 disabled=yes list=LANs
add address=10.1.20.0/24 disabled=yes list=LANs
add address=10.1.30.0/24 disabled=yes list=LANs
/ip firewall filter
add action=accept chain=input comment="defconf: accept ICMP" \
    dst-address-list=INEA protocol=icmp
add action=accept chain=forward comment="Accept DST NAT" \
    connection-nat-state=dstnat connection-state="" disabled=yes \
    in-interface=ether1-WAN
add action=accept chain=forward comment="Accept SRC NAT" \
    connection-nat-state=srcnat connection-state="" disabled=yes
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid log=yes log-prefix=INVALID
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface=ether1-WAN log=yes log-prefix=INVALID
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid log=yes log-prefix=INVALID
add action=accept chain=input comment="SSH HEXs" dst-address=addrE \
    dst-port=22222 protocol=tcp
add action=accept chain=input comment=WireGuard dst-address=addrE \
    dst-port=13231,13232 protocol=udp
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN log=yes log-prefix="NOT LAN"
/ip firewall nat
add action=src-nat chain=srcnat comment="Anchorage SRC (email)" log-prefix=\
    "ANCHORAGE SRC" out-interface=ether1-WAN src-address=10.1.10.234 \
    to-addresses=addrA
add action=src-nat chain=srcnat comment="Buckland SRC (eto)" log-prefix=\
    "BUCKLAND SRC" out-interface=ether1-WAN src-address=10.1.10.235 \
    to-addresses=addrB
add action=src-nat chain=srcnat comment="Coldbay SRC (Adam)" disabled=yes \
    out-interface=ether1-WAN src-address=10.1.11.0/24 to-addresses=\
    addrC
add action=src-nat chain=srcnat comment="Deering SRC (git)" log-prefix=\
    "DEERING SRC" out-interface=ether1-WAN src-address=10.1.10.237 \
    to-addresses=addrD
add action=src-nat chain=srcnat comment="LAN SRC" out-interface=ether1-WAN \
    src-address=10.1.1.0/24 to-addresses=addrA
add action=src-nat chain=srcnat comment="LAN SRC" out-interface=\
    ether1-WAN src-address=10.1.10.0/24 to-addresses=addrB
add action=src-nat chain=srcnat comment="LAN SRC" out-interface=ether1-WAN \
    src-address=10.1.20.0/24 to-addresses=addrE
add action=masquerade chain=srcnat comment="defconf: masquerade" disabled=yes \
    out-interface=ether1-WAN
add action=dst-nat chain=dstnat comment="Anchorage E-mail" dst-address=\
    addrA dst-port=80,443,25,587,110,143,993,995 protocol=tcp \
    to-addresses=10.1.10.234
add action=dst-nat chain=dstnat comment="Anchorage SSH" dst-address=\
    addrA dst-port=22222 protocol=tcp to-addresses=10.1.10.234 \
    to-ports=22
add action=dst-nat chain=dstnat comment="Buckland HTTPS" dst-address=\
    addrB dst-port=80,443 log-prefix="BUCKLAND DST" protocol=tcp \
    to-addresses=10.1.10.235
add action=dst-nat chain=dstnat comment="Buckland SSH" dst-address=\
    addrB dst-port=22222 protocol=tcp to-addresses=10.1.10.235 \
    to-ports=22
add action=dst-nat chain=dstnat comment="Coldbay Adam" dst-address=\
    addrC dst-port=8444 protocol=tcp to-addresses=10.1.11.10
add action=dst-nat chain=dstnat comment="Coldbay Adam" dst-address=\
    addrC dst-port=22,80,443,1922,25565 protocol=tcp to-addresses=\
    10.1.11.11
add action=dst-nat chain=dstnat comment="Deering GIT, HTTPS" dst-address=\
    addrD dst-port=22,80 log-prefix="DEERING DST" protocol=tcp \
    to-addresses=10.1.10.237
add action=dst-nat chain=dstnat comment="Deering GIT, HTTPS" dst-address=\
    addrD dst-port=443 log-prefix="DEERING DST" protocol=tcp \
    to-addresses=10.1.10.237 to-ports=443
add action=dst-nat chain=dstnat comment=CCTV disabled=yes dst-port=8000 \
    in-interface=*9 protocol=tcp to-addresses=10.1.10.200 to-ports=8000
add action=dst-nat chain=dstnat comment="R720 PVE webui" disabled=yes \
    dst-port=38006 in-interface=*9 protocol=tcp to-addresses=10.1.1.11 \
    to-ports=8006
add action=dst-nat chain=dstnat comment="Transmission TCP" disabled=yes \
    dst-address=addrE dst-port=51413,59091 protocol=tcp \
    to-addresses=10.1.10.2
add action=dst-nat chain=dstnat comment="Transmission UDP" disabled=yes \
    dst-address=addrE dst-port=51413 protocol=udp to-addresses=\
    10.1.10.2
add action=dst-nat chain=dstnat comment="Elim TRAEFIK" dst-address=\
    addrE dst-port=80 in-interface=ether1-WAN protocol=tcp \
    to-addresses=10.1.10.3 to-ports=8088
add action=dst-nat chain=dstnat comment="Elim TRAEFIK" dst-address=\
    addrE dst-port=443 in-interface=ether1-WAN protocol=tcp \
    to-addresses=10.1.10.3 to-ports=8443
/ip firewall service-port
set ftp disabled=yes
/ip route
add disabled=yes distance=1 dst-address=10.3.10.0/24 gateway=172.16.0.3 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=yes dst-address=10.2.10.0/24 gateway=172.16.0.2
add disabled=yes distance=1 dst-address=10.2.30.0/24 gateway=172.16.0.2 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ISP_GW \
    pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
/ip service
set telnet disabled=yes
set ssh port=22222
/system clock
set time-zone-name=Europe/Warsaw
/system logging
add topics=caps,debug
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool romon
set enabled=yes
You do not have the required permissions to view the files attached to this post.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Fri May 27, 2022 8:21 pm

Another update. Torch on WAN interface with protocol ICMP shows incoming pings on all addresses, replying only on addrA. The default firewall rule to accept ICMP on input is in place and enabled.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Fri Jun 10, 2022 7:26 pm

Anyone has any ideas? I'm completely baffled.

Right now I'm on clean install of 7.3

I can ping from mikrotik using all 5 addresses. But src-nat works only for the first one.
WHen I disable addressA, then src-nat starts working for addressB, and so on.

Please help!
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: src-nat stops working after some time

Fri Jun 10, 2022 8:30 pm

You can remove proxy ARP from ether1-WAN, there's no reason why you should need that with current config. Then accept-source-route=yes in IP setting is probably not something you need either. But neither of those should be breaking anything.

Srcnat must work, it's the simplest thing. You can try to enable logging for those rules and I'm sure you'll get hits for them. Or add some other logging rules, play with packets sniffer, ... there must be something.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Fri Jun 10, 2022 9:53 pm

I've added logging to forward rule (the default one with accept established/related/untracked) and in logs it shows the whole (correct!) path of NAT end-to-end BUT despite having src-nat set as addrB, in logs it's always addrA.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Fri Jun 10, 2022 9:55 pm

forward: in:bridge out:ether1, src-mac 48:2a:e3:9d:29:f1, proto TCP (ACK), 192.168.88.254:45821->116.202.225.122:80, NAT (192.168.88.254:45821->addrA:45821)->116.202.225.122:80, len 52
This is with masq off, and only this single src-nat entry.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: src-nat stops working after some time

Sat Jun 11, 2022 2:07 am

Where does 192.168.88.254 come from? There was no such subnet anywhere in posted config.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Sat Jun 11, 2022 2:19 am

Sigh, yes. I took the device again, reset it again to MT defaults and that was the IP of my laptop.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: src-nat stops working after some time

Sat Jun 11, 2022 3:21 am

I don't know how much traffic you have, so for testing try to add rules like these (at the beginning of chains; select unique testing address for each of your public addresses):
/ip firewall mangle
add action=log chain=prerouting dst-address=1.1.1.1 protocol=icmp log-prefix="request A1:"
add action=log chain=postrouting dst-address=1.1.1.1 protocol=icmp log-prefix="request A2:"
add action=log chain=prerouting src-address=1.1.1.1 protocol=icmp log-prefix="response A1:"
add action=log chain=postrouting src-address=1.1.1.1 protocol=icmp log-prefix="response A2:"
add action=log chain=prerouting dst-address=1.0.0.1 protocol=icmp log-prefix="request B1:"
add action=log chain=postrouting dst-address=1.0.0.1 protocol=icmp log-prefix="request B2:"
add action=log chain=prerouting src-address=1.0.0.1 protocol=icmp log-prefix="response B1:"
add action=log chain=postrouting src-address=1.0.0.1 protocol=icmp log-prefix="response B2:"
...
/ip firewall nat
add action=src-nat chain=srcnat dst-address=1.1.1.1 log=yes log-prefix="nat A:" protocol=icmp to-addresses=addrA
add action=src-nat chain=srcnat dst-address=1.0.0.1 log=yes log-prefix="nat B:" protocol=icmp to-addresses=addrB
...
Then ping those external addresses (1.1.1.1, 1.0.0.1, ...) and check logs. E.g. for the two I've shown you should see:
firewall,info request A1: prerouting: in:<LAN> out:(unknown 0), ... proto ICMP (type 8, code 0), 192.168.x.x->1.1.1.1, ...
firewall,info request A2: postrouting: in:(unknown 0) out:<WAN>, ... proto ICMP (type 8, code 0), 192.168.x.x->1.1.1.1, ...
firewall,info nat A: srcnat: in:(unknown 0) out:<WAN>, ... proto ICMP (type 8, code 0), 192.168.x.x->1.1.1.1, ...
firewall,info response A1: prerouting: in:<WAN> out:(unknown 0), ... proto ICMP (type 0, code 0), 1.1.1.1->addrA, NAT 1.1.1.1->(addrA->192.168.x.x), ...
firewall,info response A2: postrouting: in:(unknown 0) out:<LAN>, ... proto ICMP (type 0, code 0), 1.1.1.1->192.168.x.x, NAT 1.1.1.1->(addrA->192.168.x.x), ...
...
firewall,info request B1: prerouting: in:<LAN> out:(unknown 0), ... proto ICMP (type 8, code 0), 192.168.x.x->1.0.0.1, ...
firewall,info request B2: postrouting: in:(unknown 0) out:<WAN>, ... proto ICMP (type 8, code 0), 192.168.x.x->1.0.0.1, ...
firewall,info nat B: srcnat: in:(unknown 0) out:<WAN>, ... proto ICMP (type 8, code 0), 192.168.x.x->1.0.0.1, ...
firewall,info response B1: prerouting: in:<WAN> out:(unknown 0), ... proto ICMP (type 0, code 0), 1.0.0.1->addrB, NAT 1.0.0.1->(addrB->192.168.x.x), ...
firewall,info response B2: postrouting: in:(unknown 0) out:<LAN>, ... proto ICMP (type 0, code 0), 1.0.0.1->192.168.x.x, NAT 1.0.0.1->(addrB->192.168.x.x), ...
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Sat Jun 11, 2022 2:17 pm

Thanks for the suggestion, I'll try it on Monday.

So for all of my 5 addresses, I should have specific unique address, like 1.0.0.1, 1.0.0.2, etc - just something to enable the mangle rules to mark these for logging, correct?

I did something somewhat similar yesterday. I've added logging to the forwarding of related connections and spotted something weird.
For connections src-natted as addrB, the response was coming to addrA. At first I ignored it, but then did another test. I've connected ISP modem to a switch and created an untagged VLAN out of it. Then I connected one of the servers directly to the switch (bypassing my router) and set public IP directly on server interface. And it doesn't work either. So with ISP support team we'll investigate a possibility of some misconfiguration of routing on their side. On monday I will connect my laptop again to the ISP modem and will check if the responses are coming from the same address, or are they overwritten to addrA as well.

Weird stuff.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: src-nat stops working after some time

Sat Jun 11, 2022 4:33 pm

You just need some addresses in internet that will respond to ping. I chose addresses of well-known public DNS resolvers as example, but it can be anything. If you'd happen to use these as resolvers, then choose something else. The point is to have something distinct for testing that won't be triggered by anything else, only by your testing. So the log won't become long unreadable mess.

And if you did logging for packets matching connection-state=related, it sounds like nonsense, related state is result of conntrack magic, it doesn't necessarily stick to one address (but in most cases it probably does). Keep it simple, start with ping. Otherwise changing address in responses sounds weird too, that would have to be very creative misconfiguration. I don't know if I'd be able to do that even if I wanted to.
 
emunt6
Frequent Visitor
Frequent Visitor
Posts: 87
Joined: Fri Feb 02, 2018 7:00 pm

Re: src-nat stops working after some time

Sat Jun 11, 2022 5:17 pm

Hi!

This is a normal behavior like in Linux:
> If you have an interface (eth1) with multiple addresses the first assigned address will be the default ( Example: "arp response srcip=first-assigned-ip"- even if it was request for 2nd ip arp) - You cannot change this behavior - You can add some custom fixes: adding rule to every answer correctly each request ( if dst-ip=2nd-ip then set snat=2nd-ip ) - this is a real pain.

Linux has fixed this in a different way its called "ipvlan driver" - Using eth1 as a parent interface with subinterfaces ( each subinterface has the same mac-address - its like a "bridgeless bridge" ).
I think mikrotik added this in the v7 series of RouterOS - add vlan interface to the WAN interface "type=ipvlan mode=L2" - so you need to create ipvlan-interfaces for every WAN-address ( 5ip - 5interface ) - Using this and using the old config will work.

More about the "ipvlan driver":
https://developers.redhat.com/blog/2018 ... networking

I dont have v7 series of Mikrotik hardver, I only use Linux boxes right now.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: src-nat stops working after some time

Sat Jun 11, 2022 6:34 pm

It's true that when there are multiple IP addresses on one interface, system will use one as default (and yes, I think it's the first one) when it initiates new connection with unspecified source. But if source is set to specific address (and that happens also for action=src-nat), then it uses that address. ARP doesn't have much to do with this, there's one physical interface with one MAC address, so that's used for all IP addresses on that interface, be it one or hundered. It's problem if you need to get them from DHCP, but not when they are static. I use configs like this for years and it works fine. And AFAIK there's no ipvlan in RouterOS.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Mon Jun 13, 2022 5:59 pm

addrA = x.x.x.234
addrB = x.x.x.235
addrC = x.x.x.236
addrD = x.x.x.237
addrE = x.x.x.238
10.1.1.196 = my laptop
/ip firewall mangle
add action=log chain=prerouting dst-address=1.1.1.1 protocol=icmp log-prefix="request A1:"
add action=log chain=postrouting dst-address=1.1.1.1 protocol=icmp log-prefix="request A2:"
add action=log chain=prerouting src-address=1.1.1.1 protocol=icmp log-prefix="response A1:"
add action=log chain=postrouting src-address=1.1.1.1 protocol=icmp log-prefix="response A2:"
add action=log chain=prerouting dst-address=1.0.0.1 protocol=icmp log-prefix="request B1:"
add action=log chain=postrouting dst-address=1.0.0.1 protocol=icmp log-prefix="request B2:"
add action=log chain=prerouting src-address=1.0.0.1 protocol=icmp log-prefix="response B1:"
add action=log chain=postrouting src-address=1.0.0.1 protocol=icmp log-prefix="response B2:"
add action=log chain=prerouting dst-address=9.9.9.9 protocol=icmp log-prefix="request C1:"
add action=log chain=postrouting dst-address=9.9.9.9 protocol=icmp log-prefix="request C2:"
add action=log chain=prerouting src-address=9.9.9.9 protocol=icmp log-prefix="response C1:"
add action=log chain=postrouting src-address=9.9.9.9 protocol=icmp log-prefix="response C2:"
add action=log chain=prerouting dst-address=149.112.112.112 protocol=icmp log-prefix="request D1:"
add action=log chain=postrouting dst-address=149.112.112.112 protocol=icmp log-prefix="request D2:"
add action=log chain=prerouting src-address=149.112.112.112 protocol=icmp log-prefix="response D1:"
add action=log chain=postrouting src-address=149.112.112.112 protocol=icmp log-prefix="response D2:"
add action=log chain=prerouting dst-address=208.67.222.222 protocol=icmp log-prefix="request E1:"
add action=log chain=postrouting dst-address=208.67.222.222 protocol=icmp log-prefix="request E2:"
add action=log chain=prerouting src-address=208.67.222.222 protocol=icmp log-prefix="response E1:"
add action=log chain=postrouting src-address=208.67.222.222 protocol=icmp log-prefix="response E2:"

/ip firewall nat
add action=src-nat chain=srcnat dst-address=1.1.1.1 log=yes log-prefix="nat A:" protocol=icmp to-addresses=x.x.x.234
add action=src-nat chain=srcnat dst-address=1.0.0.1 log=yes log-prefix="nat B:" protocol=icmp to-addresses=x.x.x.235
add action=src-nat chain=srcnat dst-address=9.9.9.9 log=yes log-prefix="nat C:" protocol=icmp to-addresses=x.x.x.236
add action=src-nat chain=srcnat dst-address=149.112.112.112 log=yes log-prefix="nat D:" protocol=icmp to-addresses=x.x.x.237
add action=src-nat chain=srcnat dst-address=208.67.222.222 log=yes log-prefix="nat E:" protocol=icmp to-addresses=x.x.x.238
command
ping -c 1 1.1.1.1
returns
16:53:29 firewall,info request A1: prerouting: in:bridge out:(unknown 0), src-mac 48:2a:e3:9d:29:f1, proto ICMP (type 8, code 0), 10.1.1.196->1.1.1.1, len 84
16:53:29 firewall,info request A2: postrouting: in:bridge out:ether1-WAN, src-mac 48:2a:e3:9d:29:f1, proto ICMP (type 8, code 0), 10.1.1.196->1.1.1.1, len 84
16:53:29 firewall,info nat A: srcnat: in:bridge out:ether1-WAN, src-mac 48:2a:e3:9d:29:f1, proto ICMP (type 8, code 0), 10.1.1.196->1.1.1.1, len 84
16:53:29 firewall,info response A1: prerouting: in:ether1-WAN out:(unknown 0), src-mac 7c:a2:3e:fd:6e:77, proto ICMP (type 0, code 0), 1.1.1.1->x.x.x.234, NAT 1.1.1.1->(x.x.x.234->10.1.1.196), len 84
16:53:29 firewall,info response A2: postrouting: in:ether1-WAN out:bridge, src-mac 7c:a2:3e:fd:6e:77, proto ICMP (type 0, code 0), 1.1.1.1->10.1.1.196, NAT 1.1.1.1->(x.x.x.234->10.1.1.196), len 84
However,
ping -c 1 1.0.0.1
returns only
16:54:39 firewall,info request B1: prerouting: in:bridge out:(unknown 0), src-mac 48:2a:e3:9d:29:f1, proto ICMP (type 8, code 0), 10.1.1.196->1.0.0.1, len 84
16:54:39 firewall,info request B2: postrouting: in:bridge out:ether1-WAN, src-mac 48:2a:e3:9d:29:f1, proto ICMP (type 8, code 0), 10.1.1.196->1.0.0.1, len 84
16:54:39 firewall,info nat B: srcnat: in:bridge out:ether1-WAN, src-mac 48:2a:e3:9d:29:f1, proto ICMP (type 8, code 0), 10.1.1.196->1.0.0.1, len 84
I moved the src-nat rules above masqurade.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Mon Jun 13, 2022 6:03 pm

Also, using ping tool in WinBox (as long as these new src-nat rules are in place) i'm unable to ping any of the IPs other than 1.1.1.1

However in this case the log output is different:
17:04:43 firewall,info request B2: postrouting: in:(unknown 0) out:ether1-WAN, proto ICMP (type 8, code 0), x.x.x.234->1.0.0.1, len 50
17:04:43 firewall,info nat B: srcnat: in:(unknown 0) out:ether1-WAN, proto ICMP (type 8, code 0), x.x.x.234->1.0.0.1, len 50
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: src-nat stops working after some time

Mon Jun 13, 2022 9:31 pm

Yes, from router it's different. The first step (in prerouting) is not there, and original source being x.x.x.234 is fine too. But the rest should work. You have log from srcnat rule, so that happened. You can use Tools->Torch and you should be able to see outgoing packets with correct x.x.x.23X source addresses. It still looks like some communication problem between your and ISP's router. I was re-reading the thread and e.g. the part where all five addresses work when they are on laptop connected directly to ISP, but don't on router, it suggests something like that. But it doesn't make any sense why it would happen.

I guess the next step is packet sniffer on ether1-WAN, watch for ARP requests or anything suspicious. Or before that, if you can test blank config (after resetting router), with only those five addresses on WAN, default route, and nothing else, no firewall or anything, in case it's some mistake that neither of us is seeing. Do that, open five terminals and run these in them:
ping src-address=x.x.x.234 address=8.8.8.8
ping src-address=x.x.x.235 address=8.8.8.8
...etc
It it works, then start adding things, step by step. And if it doesn't, then something is really wrong, but I don't have any idea what it might be.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Tue Jun 14, 2022 12:09 am

The crazy thing is that I tested this exact setup on two separate routers (HEX S and 951), with default config, both on latest 6.x and two versions of 7.x. And it's always .234 that is working.

Of course the ISP says that "since all five addresses work when connected directly, then it's customer issue". While I understand their stance, I'm so, so frustrated with this, because all signs on the skies say that it all should work. But for whatever reason doesn't.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: src-nat stops working after some time

Tue Jun 14, 2022 1:52 am

But they are directly connected here as well, right? There isn't anything I'm missing? When you test it with laptop, you simply add five addresses on one interface, and connect it to the cable that comes from ISP. And when you test it with router, you add same five addresses to one of router's interfaces and connect that to the same cable that comes from ISP? Correct? So there's no difference, only that it's two different devices with different systems, but basically the same config.

If you have another free MikroTik router, then make it into transparent bridge, put it between ISP and the other router or laptop, and watch what happens there. You'll be able to see what's on the wire, if main router sends correct addresses, if ARP between router and ISP works correctly, etc.
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Thu Jun 16, 2022 1:10 am

I'll be able to test on Monday. Meanwhile, someone on Reddit proposed me something interesting: one test would be to disable fasttrack on ether1 (I remember one of my tests showing that fasttrack was not working of srcnatted addresses, so I'm interesting to test this one). Second thing (a bit of a potshot) would be to create a new bridge and assign ether1 to it, and setup public addresses on that new bridge.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: src-nat stops working after some time

Thu Jun 16, 2022 3:39 am

I wouldn't bet on either, because unless it's some really weird bug, fasttrack shouldn't have any problem with multiple addresses, and ethernet interface "wrapped in bridge" doesn't change anything. Of course if it is some weird bug, then anything is possible.

I like the idea with another router as transparent bridge (temporarily for testing), and not only because it's mine. :) It would allow to see what exactly happens, what each router is sending. So if your main router sends something wrong, you'd see it. If it receives something correct, but claims it doesn't, you'd know. Same for ISP's router.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: src-nat stops working after some time

Thu Jun 16, 2022 1:51 pm

Is the /29 routed or a link prefix?

If it's /29 link prefixes, you need to assign each individual /32 out of it to the WAN interface directly PTP to the ISP to ensure ARP works right.

The better solution is ask the ISP to slice it into /30 and from there, use a /31 for PTP and route the remaining with next-hop the /31 address on your end. It now becomes a "LAN" prefix, and you can use it natting more efficiently, however still, stop using src nat and get on netmap:
viewtopic.php?t=176358
 
yaczesuaff
just joined
Topic Author
Posts: 19
Joined: Fri Mar 04, 2022 9:35 pm

Re: src-nat stops working after some time

Fri Jun 17, 2022 12:42 pm

Is the /29 routed or a link prefix?

If it's /29 link prefixes, you need to assign each individual /32 out of it to the WAN interface directly PTP to the ISP to ensure ARP works right.

The better solution is ask the ISP to slice it into /30 and from there, use a /31 for PTP and route the remaining with next-hop the /31 address on your end. It now becomes a "LAN" prefix, and you can use it natting more efficiently, however still, stop using src nat and get on netmap:
viewtopic.php?t=176358
It's a proper /29. I tried using each address with /32 and it doesn't work at all. Also the ISP won't slice it into /30.

As for netmap it's a no go for me, I've read about it.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: src-nat stops working after some time

Sat Jun 18, 2022 3:26 pm

Is the /29 routed or a link prefix?

If it's /29 link prefixes, you need to assign each individual /32 out of it to the WAN interface directly PTP to the ISP to ensure ARP works right.

The better solution is ask the ISP to slice it into /30 and from there, use a /31 for PTP and route the remaining with next-hop the /31 address on your end. It now becomes a "LAN" prefix, and you can use it natting more efficiently, however still, stop using src nat and get on netmap:
viewtopic.php?t=176358
It's a proper /29. I tried using each address with /32 and it doesn't work at all. Also the ISP won't slice it into /30.

As for netmap it's a no go for me, I've read about it.
If it does not work, then it is not a routed prefix, that's a link prefix whereby on their end it is something like 8.0.0.1/29 eth1 and on your end it is 8.0.0.2/29 eth2.

The only work around here is 8.0.0.3/32 and so on directly on eth2.

Why is "netmap" a no-go? It is in fact the default behaviour of src nat on modern day Linux kernels, so if MT decided to upgrade the kernel to latest LTS, src nat and netmap would do the same thing and ensure persistent IP:port mapping.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: src-nat stops working after some time

Sat Jun 18, 2022 10:07 pm

It was already posted, all addresses are on WAN interface, with some srcnat/dstnat rules. So simple config that should just work.

Who is online

Users browsing this forum: No registered users and 69 guests