Community discussions

MikroTik App
 
ErkDog
just joined
Topic Author
Posts: 23
Joined: Thu Dec 02, 2021 5:51 pm

NAT Issues every 10-14 days

Thu Jun 02, 2022 3:04 am

I have a 3Gbps Comcast fiber circuit and a 1 Gbps FiOS on a CCR1072-1G-8S+ - RouterOS 7.2.3

I do have RemoteWinBox currently setup to pull config backups, that's about all I do with it.

Every 10 - 14 days, NAT seems to stop working.

All servers inside the network become unavailable, and all internal computers stop being able to get out.

The Mikrotik is up, because I solve it by logging in with WinBox and telling it to reboot.

Every time there is a router os upgrade, I hope that I see something about NAT in patch notes, and the issue goes away, but it does not.

If I'm not at the office when it happens, I'm able to Wireguard in, and connect to the Mikrotik and reboot it remotely.

I don't really have time to "dig into" why it's not working when it stops, because I'm constantly getting offline alerts and phone calls and have to resolve the issue as promptly as possible to restore connectivity to customers.

One thing of note, is that if I mark the Comcast Gateway as disabled, everything works over FIOS.

I haven't tried re-enabling the Comcast Gateway to see if it comes back up, because I just reboot the mikrotik, and reenable it to get things back to normal as soon as possible.

IPv6 traffic seems to continue to work as well, which of course is not using NAT either.

Any ideas or thoughts?
 
ErkDog
just joined
Topic Author
Posts: 23
Joined: Thu Dec 02, 2021 5:51 pm

Re: NAT Issues every 10-14 days

Tue Jun 14, 2022 3:10 am

It just did all this again.

Any Ideas?
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: NAT Issues every 10-14 days

Tue Jun 14, 2022 3:41 am

i had similar issue with ccr1072 some months ago with routeros 6.48.6

in my case many hosts being src-natted with a mascarade rule wich uses a public ip, that same public ip is the same of another scr-nat rules for other groups of hosts using deterministic nat 444 cg-nat

the "solution" was to use independent public ip for src-nat mascarade and src-nat nat 44 cgnat rules

lets call this issue some kind of src-nat collision

as you say the problem affect customer service so i dont have opportunity to test it too much, since i make that changes the issue is gone

i hope my case can help you

Please confirm if you can solve it in the same way
 
User avatar
Znevna
Forum Guru
Forum Guru
Posts: 1347
Joined: Mon Sep 23, 2019 1:04 pm

Re: NAT Issues every 10-14 days

Tue Jun 14, 2022 7:22 am

My money is on a butchered config.
 
ErkDog
just joined
Topic Author
Posts: 23
Joined: Thu Dec 02, 2021 5:51 pm

Re: NAT Issues every 10-14 days

Thu Jul 28, 2022 8:11 am

Thank you for your reply @chechito.

I'm having a little trouble understanding exactly what you did to resolve the issue though.

I have a public routable IP from my ISP not behind carrier-grade NAT or anything.

Also, @Znevna, just saying "I bet your config is screwed up." is not helpful at all, nor does it make any sense.

If my overall config was broken, then things wouldn't work right at all.

All NAT wouldn't just stop working every 10-14 days.

As this is an enterprise piece of equipment, that was not cheap, it should be able to handle anything I throw at it, and if it can't it should tell me why.

Neither of those two things are happening.

I guess I'll open an official ticket and refer to this thread for context since no clear path to resolution has been provided.

Thanks,
Matt
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: NAT Issues every 10-14 days

Thu Jul 28, 2022 8:30 am

Also, @Znevna, just saying "I bet your config is screwed up." is not helpful at all, nor does it make any sense.
It's as helpful as it gets. The problem you vaguely report (NAT stopping to work every now and then) isn't well known, that's for sure.
Most problems we see in this forum are due to configuration, only a few are actual bugs. Statistics thus says it's more probable your config is screwed up than that you hit an actual bug. But it's impossible to tell without seeing actual config you're running (and you didn't provide it). And if it was a bug, it's obviously rarely hit. MT support can't fix it if they can't replicate it ... and you can't (or won't) provide instructions on how to (reliably) replicate the problem.
 
holvoetn
Forum Guru
Forum Guru
Posts: 5317
Joined: Tue Apr 13, 2021 2:14 am
Location: Belgium

Re: NAT Issues every 10-14 days

Thu Jul 28, 2022 9:02 am

@ErkDog:

"My car doesn't run, what's wrong with it ?"
You see the problem here ?
How would anyone be able to help in an efficient way with such a problem description ?
Can be anything.

To get to the point where we need to be:
the only way anyone is able to verify what might be wrong with your config, is for you to show it.
It might also help to make a small drawing on what's connected where, with indication of used IP address (obfuscated, see below).

terminal:
/export hide-sensitive file=anynameyouwish
Review export for remaining sensitive info (check for secret keys, remove serial number!, change public IPs to PubIP1- PubIP2-... ) and post here between [code] quotes.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: NAT Issues every 10-14 days

Thu Jul 28, 2022 6:12 pm

+1 for export
 
ErkDog
just joined
Topic Author
Posts: 23
Joined: Thu Dec 02, 2021 5:51 pm

Re: NAT Issues every 10-14 days

Wed Aug 10, 2022 11:00 pm

Here is config.

<REMOVED>

<ADDED>
# aug/10/2022 15:56:32 by RouterOS 7.4
# software id = <CENSORED>
#
# model = CCR1072-1G-8S+
# serial number = <CENSORED>
/interface sstp-client
add comment="Remote Winbox connection for WilsonAve" connect-to=<CENSORED> disabled=no name=RemoteWinboxVPN3 user=no
/interface ethernet
set [ find default-name=ether1 ] name=ETH1MGMT
set [ find default-name=sfp-sfpplus1 ] name=SFP1-LAN
set [ find default-name=sfp-sfpplus2 ] advertise=1000M-full disabled=yes name=SFP2-FIOSWAN speed=1Gbps
set [ find default-name=sfp-sfpplus3 ] name=SFP3-CCWAN
set [ find default-name=sfp-sfpplus4 ] advertise=1000M-full name=SFP4-SDWAN
/interface wireguard
add listen-port=13231 mtu=1420 name=WireGuard
/interface list
add include=static name=WANS
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool0 ranges=10.8.8.100-10.8.8.200
/ip dhcp-server
add address-pool=dhcp_pool0 interface=SFP1-LAN lease-time=12h name=dhcp1
/ipv6 dhcp-server
add address-pool=CCast interface=SFP1-LAN lease-time=4w2d name=CCast
/ipv6 pool
add name=CCast prefix=<CENSORED>:/64 prefix-length=64
/port
set 0 name=serial0
set 1 name=serial1
/routing bgp template
set default disabled=no output.network=bgp-networks
/routing ospf instance
add disabled=no name=default-v2
add disabled=no name=default-v3 version=3
/routing ospf area
add disabled=yes instance=default-v2 name=backbone-v2
add disabled=yes instance=default-v3 name=backbone-v3
/routing table
add fib name=ROUTE2FIOS
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set max-neighbor-entries=8192
/interface detect-internet
set internet-interface-list=WANS wan-interface-list=WANS
/interface list member
add interface=SFP3-CCWAN list=WANS
add interface=SFP2-FIOSWAN list=WANS
/interface ovpn-server server
set auth=sha1,md5
/interface wireguard peers
add allowed-address=192.168.253.101/32 comment=Erk-Legion2 interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=192.168.253.100/32 comment=Erk-S20 interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=192.168.0.0/24,192.168.253.1/32 comment="CrtrCreek-Router (Uses 192.168.253.1)" endpoint-address=<CENSORED> endpoint-port=13231 interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=192.168.253.102/32 comment="Dawn - Android19" interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=192.168.253.105/32 comment=John-Desktop interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=192.168.253.106/32 comment=Antonizoon-Phone interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=192.168.253.107/32 comment=doukaina interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=192.168.253.108/32 comment=Keeter-Phone interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=10.7.7.0/24,192.168.253.2/32 comment="KetterTIK (Uses 192.168.253.2)" endpoint-address=<CENSORED> endpoint-port=13231 interface=WireGuard persistent-keepalive=5m public-key="no"
/ip address
add address=192.168.88.1/24 comment=defconf interface=ETH1MGMT network=192.168.88.0
add address=10.8.8.5/24 interface=SFP1-LAN network=10.8.8.0
add address=<CENSORED>.26/30 interface=SFP3-CCWAN network=<CENSORED>.24
add address=192.168.253.5/24 interface=WireGuard network=192.168.253.0
add address=<CENSORED>.38/27 interface=SFP4-SDWAN network=<CENSORED>.32
/ip cloud
set ddns-enabled=yes ddns-update-interval=3m
/ip dhcp-client
add default-route-distance=3 disabled=yes interface=SFP2-FIOSWAN use-peer-dns=no use-peer-ntp=no
add default-route-distance=4 disabled=yes interface=SFP4-SDWAN use-peer-dns=no use-peer-ntp=no
/ip dhcp-server config
set store-leases-disk=15m
/ip dhcp-server lease
add address=10.8.8.62 client-id=1:a0:ce:c8:e3:6f:82 mac-address=A0:CE:C8:E3:6F:82 server=dhcp1
add address=10.8.8.98 client-id=ff:f7:f6:49:34:0:2:0:0:ab:11:3a:e3:f4:36:4b:0:a6:70 mac-address=DC:A6:32:07:D0:B1 server=dhcp1
add address=10.8.8.7 mac-address=80:CC:9C:82:E7:08 server=dhcp1
add address=10.8.8.97 client-id=1:e0:d5:5e:87:c7:e0 mac-address=E0:D5:5E:87:C7:E0 server=dhcp1
add address=10.8.8.41 mac-address=00:0C:15:04:30:57 server=dhcp1
add address=10.8.8.40 mac-address=00:0C:15:04:2F:EC server=dhcp1
add address=10.8.8.49 client-id=1:34:9f:7b:a4:3:eb mac-address=34:9F:7B:A4:03:EB server=dhcp1
add address=10.8.8.34 mac-address=00:09:F5:27:48:66 server=dhcp1
add address=10.8.8.60 client-id=ff:7a:f:84:9a:0:1:0:1:28:da:ac:5d:1c:69:7a:f:84:9a mac-address=1C:69:7A:0F:84:9A server=dhcp1
add address=10.8.8.61 client-id=1:e8:ea:6a:9:65:54 mac-address=E8:EA:6A:09:65:54 server=dhcp1
add address=10.8.8.50 client-id=1:e4:5f:1:37:56:e7 mac-address=E4:5F:01:37:56:E7 server=dhcp1
add address=10.8.8.35 mac-address=00:09:F5:2A:C0:D3 server=dhcp1
add address=10.8.8.81 client-id=1:8c:85:80:d6:ad:b2 comment=EUFY-LROOM mac-address=8C:85:80:D6:AD:B2 server=dhcp1
add address=10.8.8.80 client-id=1:8c:85:80:d4:ab:46 comment=EUFY-SERVEROOM mac-address=8C:85:80:D4:AB:46 server=dhcp1
/ip dhcp-server network
add address=10.8.8.0/24 dns-server=10.8.8.4 domain=ecansol.loc gateway=10.8.8.5 netmask=24 ntp-server=10.8.8.4
/ip dns
set servers=10.8.8.4
/ip firewall address-list
add address=<CENSORED>.26 list=WANIPS
add address=<CENSORED>.38 list=WANIPS
/ip firewall filter
add action=accept chain=input protocol=icmp
add action=accept chain=input dst-port=13231 protocol=udp
add action=accept chain=input dst-port=13231 protocol=tcp
add action=accept chain=forward dst-address=10.8.8.0/24 src-address=192.168.0.0/24
add action=accept chain=forward dst-address=192.168.0.0/24 src-address=10.8.8.0/24
add action=accept chain=forward dst-address=10.7.7.0/24 src-address=10.8.8.0/24
add action=accept chain=forward dst-address=10.8.8.0/24 src-address=10.7.7.0/24
add action=accept chain=forward dst-address=0.0.0.0/0 src-address=10.7.7.0/24
add action=accept chain=forward dst-address=10.7.7.0/24 src-address=0.0.0.0/0
add action=accept chain=input comment="Allow Remote Winbox" in-interface=RemoteWinboxVPN3
add action=reject chain=input dst-address-list=WANIPS dst-port=2000 protocol=tcp reject-with=icmp-network-unreachable
add action=reject chain=input dst-address-list=WANIPS dst-port=2000 protocol=udp reject-with=icmp-network-unreachable
add action=reject chain=input dst-address-list=WANIPS dst-port=5678 protocol=udp reject-with=icmp-network-unreachable
add action=reject chain=input dst-address-list=WANIPS dst-port=5678 protocol=tcp reject-with=icmp-network-unreachable
add action=drop chain=input dst-address-list=WANIPS dst-port=53 protocol=tcp
add action=drop chain=input dst-address-list=WANIPS dst-port=53 protocol=udp
add action=accept chain=forward dst-address=0.0.0.0/0 src-address=0.0.0.0/0
/ip firewall mangle
add action=mark-routing chain=prerouting comment=NetMgmt-VZFios new-routing-mark=ROUTE2FIOS passthrough=yes src-address=10.8.8.65
add action=mark-routing chain=prerouting comment=Ops-Skull-SDWAN new-routing-mark=ROUTE2FIOS passthrough=yes src-address=10.8.8.101
/ip firewall nat
add action=masquerade chain=srcnat comment="NAT FOR SDWAN" out-interface=SFP4-SDWAN
add action=masquerade chain=srcnat comment="NAT FOR CC" out-interface=SFP3-CCWAN
add action=dst-nat chain=dstnat comment=NetMGMT dst-address-list=WANIPS dst-port=9443 protocol=tcp to-addresses=10.8.8.4 to-ports=9443
add action=dst-nat chain=dstnat comment="Master - SRV" dst-address-list=WANIPS dst-port=60059 protocol=tcp to-addresses=10.8.8.21 to-ports=60059
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=80 protocol=tcp to-addresses=10.8.8.21 to-ports=80
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=443 protocol=tcp to-addresses=10.8.8.21 to-ports=443
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2083 protocol=tcp to-addresses=10.8.8.21 to-ports=2083
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2087 protocol=tcp to-addresses=10.8.8.21 to-ports=2087
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2096 protocol=tcp to-addresses=10.8.8.21 to-ports=2096
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=53 protocol=tcp to-addresses=10.8.8.21 to-ports=53
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=53 protocol=udp to-addresses=10.8.8.21 to-ports=53
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25 protocol=tcp to-addresses=10.8.8.21 to-ports=25
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=993 protocol=tcp to-addresses=10.8.8.21 to-ports=993
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=995 protocol=tcp to-addresses=10.8.8.21 to-ports=995
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=587 protocol=tcp to-addresses=10.8.8.21 to-ports=587
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=465 protocol=tcp to-addresses=10.8.8.21 to-ports=465
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=30033 protocol=tcp to-addresses=10.8.8.21 to-ports=30033
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=30033 protocol=udp to-addresses=10.8.8.21 to-ports=30033
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=10011 protocol=tcp to-addresses=10.8.8.21 to-ports=10011
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=10011 protocol=udp to-addresses=10.8.8.21 to-ports=10011
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=9987 protocol=tcp to-addresses=10.8.8.21 to-ports=9987
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=9987 protocol=udp to-addresses=10.8.8.21 to-ports=9987
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8111 protocol=tcp to-addresses=10.8.8.21 to-ports=8111
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8111 protocol=udp to-addresses=10.8.8.21 to-ports=8111
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8110 protocol=tcp to-addresses=10.8.8.21 to-ports=8110
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8110 protocol=udp to-addresses=10.8.8.21 to-ports=8110
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8610 protocol=tcp to-addresses=10.8.8.21 to-ports=8610
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8610 protocol=udp to-addresses=10.8.8.21 to-ports=8610
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8611 protocol=tcp to-addresses=10.8.8.21 to-ports=8611
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8611 protocol=udp to-addresses=10.8.8.21 to-ports=8611
add action=dst-nat chain=dstnat comment="Anton Desktop - ketilfastr" dst-address-list=WANIPS dst-port=43030 protocol=tcp to-addresses=10.8.8.97 to-ports=43030
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2053 protocol=tcp to-addresses=10.8.8.97 to-ports=443
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8880 protocol=tcp to-addresses=10.8.8.97 to-ports=80
add action=dst-nat chain=dstnat comment="BA-NUC - SSH" dst-address-list=WANIPS dst-port=43028 protocol=tcp to-addresses=10.8.8.60 to-ports=43028
add action=dst-nat chain=dstnat comment="BA - rPI" dst-address-list=WANIPS dst-port=43029 protocol=tcp to-addresses=10.8.8.98 to-ports=43029
add action=dst-nat chain=dstnat comment=DIFFDEV dst-address-list=WANIPS dst-port=60070 protocol=tcp to-addresses=10.8.8.62 to-ports=60070
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7000 protocol=tcp to-addresses=10.8.8.62 to-ports=7000
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7001 protocol=tcp to-addresses=10.8.8.62 to-ports=7001
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7002 protocol=tcp to-addresses=10.8.8.62 to-ports=7002
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7003 protocol=tcp to-addresses=10.8.8.62 to-ports=7003
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7004 protocol=tcp to-addresses=10.8.8.62 to-ports=7004
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7005 protocol=tcp to-addresses=10.8.8.62 to-ports=7005
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=3000 protocol=tcp to-addresses=10.8.8.62 to-ports=3000
add action=dst-nat chain=dstnat comment=MYTHVA1 dst-address-list=WANIPS dst-port=8085 protocol=tcp to-addresses=10.8.8.61 to-ports=8085
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=17777 protocol=tcp to-addresses=10.8.8.61 to-ports=17777
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=17777 protocol=udp to-addresses=10.8.8.61 to-ports=17777
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=17778 protocol=tcp to-addresses=10.8.8.61 to-ports=17778
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=17778 protocol=udp to-addresses=10.8.8.61 to-ports=17778
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=37015 protocol=tcp to-addresses=10.8.8.61 to-ports=37015
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=37015 protocol=udp to-addresses=10.8.8.61 to-ports=37015
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=19132 protocol=tcp to-addresses=10.8.8.61 to-ports=19132
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=19132 protocol=udp to-addresses=10.8.8.61 to-ports=19132
add action=dst-nat chain=dstnat comment="Matrix - SRV" dst-address-list=WANIPS dst-port=60065 protocol=tcp to-addresses=10.8.8.15 to-ports=60065
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8448 protocol=tcp to-addresses=10.8.8.15 to-ports=8448
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8448 protocol=udp to-addresses=10.8.8.15 to-ports=8448
add action=dst-nat chain=dstnat comment=GOKU dst-address-list=WANIPS dst-port=60052 protocol=tcp to-addresses=10.8.8.17 to-ports=60052
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8080 protocol=tcp to-addresses=10.8.8.17 to-ports=8080
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=4233 protocol=tcp to-addresses=10.8.8.17 to-ports=4233
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2224 protocol=tcp to-addresses=10.8.8.17 to-ports=2224
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=4234 protocol=tcp to-addresses=10.8.8.17 to-ports=4234
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=4235 protocol=tcp to-addresses=10.8.8.17 to-ports=4235
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=6499 protocol=udp to-addresses=10.8.8.17 to-ports=6499
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=6499 protocol=tcp to-addresses=10.8.8.17 to-ports=6499
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=6599 protocol=udp to-addresses=10.8.8.17 to-ports=6599
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=6599 protocol=tcp to-addresses=10.8.8.17 to-ports=6599
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=6597 protocol=udp to-addresses=10.8.8.17 to-ports=6597
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=6597 protocol=tcp to-addresses=10.8.8.17 to-ports=6597
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25566 protocol=udp to-addresses=10.8.8.17 to-ports=25566
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25566 protocol=tcp to-addresses=10.8.8.17 to-ports=25566
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25501 protocol=udp to-addresses=10.8.8.17 to-ports=25501
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25501 protocol=tcp to-addresses=10.8.8.17 to-ports=25501
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25565 protocol=udp to-addresses=10.8.8.17 to-ports=25565
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25565 protocol=tcp to-addresses=10.8.8.17 to-ports=25565
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25591 protocol=udp to-addresses=10.8.8.17 to-ports=25591
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25591 protocol=tcp to-addresses=10.8.8.17 to-ports=25591
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7777 protocol=udp to-addresses=10.8.8.17 to-ports=7777
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7778 protocol=tcp to-addresses=10.8.8.17 to-ports=7778
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7778 protocol=udp to-addresses=10.8.8.17 to-ports=7778
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7777 protocol=tcp to-addresses=10.8.8.17 to-ports=7777
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=15000 protocol=udp to-addresses=10.8.8.17 to-ports=15000
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=15000 protocol=tcp to-addresses=10.8.8.17 to-ports=15000
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=15777 protocol=udp to-addresses=10.8.8.17 to-ports=15777
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=15777 protocol=tcp to-addresses=10.8.8.17 to-ports=15777
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=5678 protocol=udp to-addresses=10.8.8.17 to-ports=5678
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=5678 protocol=tcp to-addresses=10.8.8.17 to-ports=5678
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=5679 protocol=tcp to-addresses=10.8.8.17 to-ports=5679
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=5679 protocol=udp to-addresses=10.8.8.17 to-ports=5679
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2226 protocol=tcp to-addresses=10.8.8.17 to-ports=2226
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2226 protocol=udp to-addresses=10.8.8.17 to-ports=2226
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8123 protocol=tcp to-addresses=10.8.8.17 to-ports=8123
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8123 protocol=udp to-addresses=10.8.8.17 to-ports=8123
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=34197 protocol=tcp to-addresses=10.8.8.17 to-ports=34197
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=34197 protocol=udp to-addresses=10.8.8.17 to-ports=34197
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=34198 protocol=tcp to-addresses=10.8.8.17 to-ports=34198
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=34198 protocol=udp to-addresses=10.8.8.17 to-ports=34198
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=34199 protocol=tcp to-addresses=10.8.8.17 to-ports=34199
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=34199 protocol=udp to-addresses=10.8.8.17 to-ports=34199
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2230 protocol=tcp to-addresses=10.8.8.17 to-ports=2230
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2230 protocol=udp to-addresses=10.8.8.17 to-ports=2230
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=26015 protocol=tcp to-addresses=10.8.8.17 to-ports=26015
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=26015 protocol=udp to-addresses=10.8.8.17 to-ports=26015
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=26016 protocol=tcp to-addresses=10.8.8.17 to-ports=26016
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=26016 protocol=udp to-addresses=10.8.8.17 to-ports=26016
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25567 protocol=tcp to-addresses=10.8.8.17 to-ports=25567
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25567 protocol=udp to-addresses=10.8.8.17 to-ports=25567
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=29335 protocol=tcp to-addresses=10.8.8.17 to-ports=29335
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=29335 protocol=udp to-addresses=10.8.8.17 to-ports=29335
add action=dst-nat chain=dstnat comment=GIRU dst-address-list=WANIPS dst-port=24388 protocol=tcp to-addresses=10.8.8.25 to-ports=24388
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=24388 protocol=udp to-addresses=10.8.8.25 to-ports=24388
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7781 protocol=udp to-addresses=10.8.8.25 to-ports=7781
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7781 protocol=tcp to-addresses=10.8.8.25 to-ports=7781
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=45882 protocol=tcp to-addresses=10.8.8.25 to-ports=45882
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=45882 protocol=udp to-addresses=10.8.8.25 to-ports=45882
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=45883 protocol=tcp to-addresses=10.8.8.25 to-ports=45883
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=45883 protocol=udp to-addresses=10.8.8.25 to-ports=45883
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=45884 protocol=tcp to-addresses=10.8.8.25 to-ports=45884
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=45884 protocol=udp to-addresses=10.8.8.25 to-ports=45884
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=28967 protocol=tcp to-addresses=10.8.8.25 to-ports=28967
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=28967 protocol=udp to-addresses=10.8.8.25 to-ports=28967
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=32400 protocol=tcp to-addresses=10.8.8.25 to-ports=32400
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=32400 protocol=udp to-addresses=10.8.8.25 to-ports=32400
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=16261 protocol=tcp to-addresses=10.8.8.25 to-ports=16261
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=16261 protocol=udp to-addresses=10.8.8.25 to-ports=16261
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8766 protocol=tcp to-addresses=10.8.8.25 to-ports=8766
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8766 protocol=udp to-addresses=10.8.8.25 to-ports=8766
/ip route
add check-gateway=ping comment="Pri CC Gateway-Disable if Forcing FiOS" disabled=yes distance=1 dst-address=0.0.0.0/0 gateway=<CENSORED>.25 pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=5 dst-address=192.168.0.0/24 gateway=192.168.253.1 pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=2 dst-address=0.0.0.0/0 gateway=<CENSORED>.33 pref-src=0.0.0.0 routing-table=ROUTE2FIOS scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=5 dst-address=192.168.0.0/24 gateway=192.168.253.1 pref-src=0.0.0.0 routing-table=ROUTE2FIOS scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=5 dst-address=192.168.253.0/24 gateway=WireGuard routing-table=ROUTE2FIOS scope=10 suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=3 dst-address=0.0.0.0/0 gateway=<CENSORED>.25 pref-src=0.0.0.0 routing-table=ROUTE2FIOS scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=10 dst-address=0.0.0.0/0 gateway=<CENSORED>.33 pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=5 dst-address=10.7.7.0/24 gateway=192.168.253.2 pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=5 dst-address=10.7.7.0/24 gateway=192.168.253.2 pref-src=0.0.0.0 routing-table=ROUTE2FIOS scope=30 suppress-hw-offload=no target-scope=10
/ipv6 route
add check-gateway=ping disabled=no distance=1 dst-address=/0 gateway=<CENSORED>:5a05 scope=30 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=/0 gateway=<CENSORED>:5a05 scope=30 target-scope=10
add gateway=<CENSORED>:5a05%SFP3-CCWAN
/ip service
set telnet address=10.8.8.0/24,192.168.88.0/24,<CENSORED>.237/32 disabled=yes
set ftp address=10.8.8.0/24,192.168.88.0/24,<CENSORED>.237/32 disabled=yes
set www address=10.8.8.0/24,192.168.88.0/24,<CENSORED>.237/32 disabled=yes
set ssh address=10.8.8.0/24,192.168.88.0/24,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
set www-ssl address=10.8.8.0/24,192.168.88.0/24,<CENSORED>.237/32
set api address=10.8.8.0/24,192.168.88.0/24,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
set winbox address="10.8.8.0/24,192.168.88.0/24,192.168.0.0/24,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,192.168.253.0/24"
set api-ssl address=10.8.8.0/24,192.168.88.0/24,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
/ip smb shares
add comment="default share" directory=/pub name=pub
add comment="default share" directory=/pub name=pub
/ip smb users
add name=guest
add name=guest
/ip upnp
set enabled=yes
/ip upnp interfaces
add interface=SFP1-LAN type=internal
add interface=SFP3-CCWAN type=external
/ipv6 address
add address=<CENSORED>:5a06/126 advertise=no interface=SFP3-CCWAN
add address=<CENSORED>:1 interface=SFP1-LAN
/ipv6 nd
set [ find default=yes ] dns=<CENSORED>:4 hop-limit=64 interface=SFP1-LAN managed-address-configuration=yes
/lcd
set backlight-timeout=5m default-screen=stats-all
/lcd pin
set pin-number=<CENSORED>
/system clock
set time-zone-name=America/New_York
/system identity
set name=ECANWA
/system ntp client
set enabled=yes
/system ntp client servers
add address=10.8.8.4
Last edited by rextended on Thu Aug 11, 2022 12:38 am, edited 2 times in total.
Reason: <R/A>
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: NAT Issues every 10-14 days

Thu Aug 11, 2022 12:29 am

Here is config.
please edit your public ip address data for privacy reasons
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: NAT Issues every 10-14 days

Thu Aug 11, 2022 12:31 am

One moment, I censore the config and repost
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: NAT Issues every 10-14 days

Thu Aug 11, 2022 12:36 am

i read your config and does not match my scenario so no luck with that

the only idea i have is

maybe if you change action from mascarade to scr-nat specifying outbound public ip address in your src-nat rules the issue maybe solves

if you want to use multiple public ip address use action same and mark the option not by dst specifying public ip address range to use (need to be contiguous ip addresses)
Last edited by chechito on Thu Aug 11, 2022 12:39 am, edited 1 time in total.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: NAT Issues every 10-14 days

Thu Aug 11, 2022 12:39 am

Replaced with local censored version.

:)

I don't know if I have removed all sensitive data,
but it is the poster's responsibility not to disclose them.
 
User avatar
loloski
Member Candidate
Member Candidate
Posts: 276
Joined: Mon Mar 15, 2021 9:10 pm

Re: NAT Issues every 10-14 days

Thu Aug 11, 2022 1:14 am

In a plain linux system nat issues happen if conntrack entries is too much for your system to handle, so when it happen again please post this before you reboot the system, just my 0.2$
/ip firewall/connection/tracking/print
and also post system logs here during the onset of the issue because we might see some peculiar entries in the logs pertains to conntrack entries, the suggestion of @chehito is also good to snat your connection to your specific public ip
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: NAT Issues every 10-14 days

Thu Aug 11, 2022 3:04 am

+1 on

/ip firewall/connection/tracking/print
at peak hour of traffic
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18959
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: NAT Issues every 10-14 days

Thu Aug 11, 2022 3:27 am

Jajajaj are you saying his router almost has a period? But every 12-14 days?
If the router has a peak period it would be daily or weekly LOL........
 
ErkDog
just joined
Topic Author
Posts: 23
Joined: Thu Dec 02, 2021 5:51 pm

Re: NAT Issues every 10-14 days

Thu Aug 11, 2022 5:14 pm

Thanks for removing IPs, I wasn't too worried about it though, I run production business services off this device so they are not a secret :-D.

I would think that a $3,000 router touted as a 'flagship' device, wouldn't have an upward limitation on "the number of connections it can handle" and if it did it would be absurdly high.

I will collect this information next time it hangs up.

Thanks all.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: NAT Issues every 10-14 days

Thu Aug 11, 2022 5:50 pm

Is not necessary to wait until hangs, check conn-track print at your peak hour to have an idea of your connections load to see if can be the source of the problem

Connection tracking can be tuned to solve it if it is necessary
 
ErkDog
just joined
Topic Author
Posts: 23
Joined: Thu Dec 02, 2021 5:51 pm

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 7:46 am

Well this is what it looks like right now:

[admin@ECANWA] > /ip firewall/connection/tracking/print
enabled: auto
tcp-syn-sent-timeout: 5s
tcp-syn-received-timeout: 5s
tcp-established-timeout: 1d
tcp-fin-wait-timeout: 10s
tcp-close-wait-timeout: 10s
tcp-last-ack-timeout: 10s
tcp-time-wait-timeout: 10s
tcp-close-timeout: 10s
tcp-max-retrans-timeout: 5m
tcp-unacked-timeout: 5m
loose-tcp-tracking: yes
udp-timeout: 10s
udp-stream-timeout: 3m
icmp-timeout: 10s
generic-timeout: 10m
max-entries: 1048576
total-entries: 1512


There's not really any time I can think of it's peak vs not peak.

Is connection tracking related to NAT?

Again, this seems like kind of a peculiar limitation on a $3,000 router with 16G of ram and the processor on this thing is so absurd that it's almost always at 0%.

This is a CCR1072-1G-8S+ :-/
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 10:49 am

Actually is useless 1 day timeout on one TCP connection.
(I use on all my core routers
/ip firewall connection tracking set loose-tcp-tracking=no tcp-established-timeout=30m
and I never have problems or complains from 4000+ users)

The timeout is triggered after the time specified.
For default, after 1 day no one TCP packet pass over that tracked connection, the conntrack remove the track and free port and resources.

The port numbers used for NAT are limited to 32767 (ok, after some config 65535, but is not this the point) and when all port are busy
for already tracked connection, the NAT stop working.

Example:
Against your Public IP 100.64.3.6 100 natted users use Google.
The DNS on that moment solve for everyone google.com to 172.23.25.14
The NAT can not reuse same ports, because on that case all the incoming packet directedo to 172.23.25.14 is coming all to port 46789 (and all from 443):
how distinguish on conntrack to what local IP must be sended the packet?
NAT is forced everytime to use different ports.
NAT can reuse same port, but with different IP, conntrack do the rest.

I do not know how many internal IPs you have, but probably you deplete all the available port combination for NAT,
changing gateway you have another IP usable, and the NAT work because have another group of 32767 ports usable.

Also when something is not working, mus be checked if only TCP services are stopped or UDP services.
Connecting directly the RB work because you use directly the IP of the device without use NAT.

Try this when all appear locked, and see if all working again:
/ip fire conn
:foreach idc in=[find where (timeout>60)] do={
    remove [find where .id=$idc]
}

Sorry for my english, I hope I have explained what I mean.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18959
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 1:30 pm

I didnt understand a word, but can I ask.

Are you pointing out a config issue or a more serious limitation in RoS where we should all flush them down the toilet and ask for our money back??????
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 1:38 pm

Simply written: A single Public IP has limits on how many connections can be NATtated (and tracked) at the same time.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18959
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 1:41 pm

Simply responded, what makes this person, the only person on gods green earth using MT devices to trip over this fact???
(the sceptical llama with occasional heartburn and flatulence)
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 1:47 pm

My guess is based on the fact that it has 3Gbps, just one IPv4 address and several tunnels.
I don't think he uses those 3Gbps just to keep his nephew calm by hypnotizing him with youtube... or not? :?

A fast way to deplete resources: open bittorrent or similar on one or more devices....

However it remains a hypothesis, there is no security.

If he gives that command and everything starts up again...
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18959
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 3:03 pm

I see said the blind deaf mute to the horse............
All you needed to say was that it was stab in the dark, and thus lets wait to see if blood was spilled. :-)
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 3:27 pm

[...] One thing of note, is that if I mark the Comcast Gateway as disabled, everything works over FIOS.
I haven't tried re-enabling the Comcast Gateway to see if it comes back up [...]

[...] but probably you deplete all the available port combination for NAT,
changing gateway you have another IP usable, and the NAT work [...]
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 3:33 pm


The port numbers used for NAT are limited to 32767 (ok, after some config 65535, but is not this the point)

can you provide more info about this? i think is very interesting

Thank You
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 3:48 pm

Nothing special, TCP and UDP use 65536 ports, from 0 to 65535 (for now ignoring exceptions, assigned etc.)
Usually the first 1024 ports from 0 to 1023 are reserved from IANA,
From 1024 to 32767 are usually not-ufficially-reserved for other services, like sql, remote desktop, proxy, upnp, etc.
and the last group from 32766 to 65535 are usd for NAT.
This behaviour can be changed without problem, like assign port interval for each internal IP, etc.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 6:18 pm

The port numbers used for NAT are limited to 32767 (ok, after some config 65535, but is not this the point) and when all port are busy
for already tracked connection, the NAT stop working.

I'll hypotheticize here as I've no idea how NAT actually works in linux kernel or ROS. But: for connection tracking machinery it's a quartet of addressing metadata that matters: src_address, src_port, dst_address, dst_port, if any of those changes, it's an entirely different connection (and additionally throw in protocol type to expand the possibilities). When it comes to NAT, one has to work with single src_address (in most cases) and limited number of ports (let's assume @rextended is right about 32767) ... however we get to work with plethora of dst_addresses and theoretically NAT could use same pair of src_address,src_port to connect to different remote addresses (or even to same remote address if remote port number is different). Which would mean that SRC NAT port pool exhaustion is not as likely as @rextended would like us to believe. If I'm right about src_port re-use.
Number of tracked connections is a different beast, I guess it's easy to skyrocket those. Specially with UDP "connections" (e.g. bit torrent) which don't have any connection "management" handshakes (unlike TCP which has the 3-way init handshake and a handfull of ways to terminate connection), so any firewall in the way has to rely on its own inactivity timers.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 6:32 pm

On my example I suppose only:
Only one unique Public IP,
Multiple internal IPs
Only one DNS used, that give the same resolved IP to all the machines.
The HTTPS use everytime same ports.

What the conntrack have to distinguish, for example tcp connections:
for example google.com = 172.23.23.23

On internal side, 3 devices, and can happen that the device use same ports that other devices, but the internal IPs port number not count on WAN traffic:
10.0.0.1:5678 -> 172.23.23.23:443
10.0.0.1:5679 -> 172.23.23.23:443
10.0.0.1:5680 -> 172.23.23.23:443
10.0.0.1:5681 -> 172.23.23.23:443
10.0.0.1:5682 -> 172.23.23.23:443

10.0.0.2:5678 -> 172.23.23.23:443
10.0.0.2:5679 -> 172.23.23.23:443
10.0.0.2:5680 -> 172.23.23.23:443
10.0.0.2:5681 -> 172.23.23.23:443
10.0.0.2:5682 -> 172.23.23.23:443

10.0.0.3:5678 -> 172.23.23.23:443
10.0.0.3:5679 -> 172.23.23.23:443
10.0.0.3:5680 -> 172.23.23.23:443
10.0.0.3:5681 -> 172.23.23.23:443
10.0.0.3:5682 -> 172.23.23.23:443

now NAT take control:
10.0.0.1:5678 -> 172.23.23.23:443 = 100.64.1.1:54879 -> 172.23.23.23:443
10.0.0.1:5679 -> 172.23.23.23:443 = 100.64.1.1:54880 -> 172.23.23.23:443
10.0.0.1:5680 -> 172.23.23.23:443 = 100.64.1.1:54881 -> 172.23.23.23:443
10.0.0.1:5681 -> 172.23.23.23:443 = 100.64.1.1:54882 -> 172.23.23.23:443
10.0.0.1:5682 -> 172.23.23.23:443 = 100.64.1.1:54883 -> 172.23.23.23:443

10.0.0.2:5678 -> 172.23.23.23:443 = 100.64.1.1:54884 -> 172.23.23.23:443
10.0.0.2:5679 -> 172.23.23.23:443 = 100.64.1.1:54885 -> 172.23.23.23:443
10.0.0.2:5680 -> 172.23.23.23:443 = 100.64.1.1:54886 -> 172.23.23.23:443
10.0.0.2:5681 -> 172.23.23.23:443 = 100.64.1.1:54887 -> 172.23.23.23:443
10.0.0.2:5682 -> 172.23.23.23:443 = 100.64.1.1:54888 -> 172.23.23.23:443

10.0.0.3:5678 -> 172.23.23.23:443 = 100.64.1.1:54889 -> 172.23.23.23:443
10.0.0.3:5679 -> 172.23.23.23:443 = 100.64.1.1:54890 -> 172.23.23.23:443
10.0.0.3:5680 -> 172.23.23.23:443 = 100.64.1.1:54891 -> 172.23.23.23:443
10.0.0.3:5681 -> 172.23.23.23:443 = 100.64.1.1:54892 -> 172.23.23.23:443
10.0.0.3:5682 -> 172.23.23.23:443 = 100.64.1.1:54893 -> 172.23.23.23:443

Do you notice than you have only the PublicIP port than change and local Public IP, Remote IP and remote port are the same?

When one host guest multiple services, and those services rely on same htttps ports, only one variable can change...

And this example consider only one service and 3 devices, just with 5 connections each...
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 6:44 pm

It doesn't look like it's hitting the limit:
max-entries: 1048576
total-entries: 1512
Plus the manual says about it:
Max amount of entries that connection tracking table can hold. This value depends on installed amount of RAM. Note that system does not create maximum size connection tracking table when it starts, maximum entry amount can increase if situation demands it and router still has free ram left.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11967
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 6:48 pm

That print was not done when the connection hangs,
but my hypothesis, which remains so, is about the ports that can only be used on a single unique Public IP to mask everything on the NAT.

Surely conntrack has room for a thousand addresses.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 7:14 pm

What the conntrack have to distinguish, for example tcp connections:
for example google.com = 172.23.23.23
Agree. But if other users connect e.g. mikrotik at 159.148.147.196, same port numbers with WAN IP address can be used.

In extreme case (bit torrent), where single client communicates with gazzilion of peers (tens of peers for every active torrent), it's enough to use one (or few) port numbers.

Just like DST-NAT, where single pair of WAN address + port (e.g. 443) is used to service potentialy milions of clients. It doesn't matter which NAT it is (SRC or DST), it's still the address/port quartet that identifies connection.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 7:26 pm

There would have to be one extremely popular IP address that everyone is connecting to (and to same port). Then yes, number of those connections would be limited to at most 65k (I'm not sure how many ports RouterOS uses for srcnat) for one local public address. I guess something like 8.8.8.8:53 could do it if too many internal devices used it.

Otherwise there's no problem with reusing local ports, you can have multiple connections from local public address and fixed port for all, to different remote addresses and/or ports, and it will works just fine, e.g.:
 0  SAC  s  protocol=tcp src-address=192.168.80.10:57397 dst-address=re.mo.te.138:8291 reply-src-address=re.mo.te.138:8291 
            reply-dst-address=lo.ca.l.134:666 tcp-state=established timeout=23h59m59s orig-packets=3 404 orig-bytes=208 405 orig-fasttrack-packets=0 
            orig-fasttrack-bytes=0 repl-packets=5 503 repl-bytes=7 693 869 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=43.4kbps 
            repl-rate=26.9kbps 

 1  SAC  s  protocol=tcp src-address=192.168.80.10:57400 dst-address=re.mo.te.139:8291 reply-src-address=re.mo.te.139:8291 
            reply-dst-address=lo.ca.l.134:666 tcp-state=established timeout=23h59m59s orig-packets=2 784 orig-bytes=171 781 orig-fasttrack-packets=0 
            orig-fasttrack-bytes=0 repl-packets=4 506 repl-bytes=6 275 963 repl-fasttrack-packets=0 repl-fasttrack-bytes=0 orig-rate=38.8kbps 
            repl-rate=279.5kbps
Same local lo.ca.l.134:666, no problem.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: NAT Issues every 10-14 days

Fri Aug 12, 2022 10:16 pm

There would have to be one extremely popular IP address that everyone is connecting to (and to same port). Then yes, number of those connections would be limited to at most 65k (I'm not sure how many ports RouterOS uses for srcnat) for one local public address. I guess something like 8.8.8.8:53 could do it if too many internal devices used it.

Otherwise there's no problem with reusing local ports, you can have multiple connections from local public address and fixed port for all, to different remote addresses and/or ports, and it will works just fine
I agree, that's the reason why on CG-NAT (with tunned conn-track timeouts) a subscriber can be translated to a small range of SRC-ports like 64 ports (that's the smaller amount in some other vendor's CG-NAT solutions) and works OK
 
ErkDog
just joined
Topic Author
Posts: 23
Joined: Thu Dec 02, 2021 5:51 pm

Re: NAT Issues every 10-14 days

Sun Aug 14, 2022 8:14 am

So the consensus seems to be that something is causing all my available nat ports to get used up and then NAT subsequently generally hangs.

And that to mitigate it I can run that command to see what happens when it hangs up, or lower the TCP timeout to something lower.

I don't want to make it too low cause then actual persistent connections that don't trade much information very often will drop.

What would the best command be to drop it to say 10 hours?

Thanks,
Matt
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: NAT Issues every 10-14 days

Sun Aug 14, 2022 10:39 am

@rextended wrote command to shorten connection tracking timeout in post #20 above.

I wouldn't care about idle connections too much. Most proper software using persistent connections implement keepalive functionality exactly for such purpose and everybody is welcome to enable it. So something around 1 hour should be fine (with app keepalive interval of half hour to match).
 
User avatar
fischerdouglas
Frequent Visitor
Frequent Visitor
Posts: 55
Joined: Thu Mar 07, 2019 6:38 pm
Location: Brazil
Contact:

Re: NAT Issues every 10-14 days

Thu Dec 29, 2022 8:51 pm

RFC5382 REQ-5 suggests that TCP Established Timeout should not be less than 2h4min.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: NAT Issues every 10-14 days

Thu Dec 29, 2022 9:44 pm

2h4min

i think that's a good start point from 1 day default setting
 
User avatar
sirbryan
Member Candidate
Member Candidate
Posts: 298
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: NAT Issues every 10-14 days

Thu Dec 29, 2022 10:01 pm

I had it set to 1h on my 1036 CGNAT router, which has worked well for the past year and a half for 400+ households. But I'll take 2:04:00 (and round it up to 2:05:00 for good measure).
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: NAT Issues every 10-14 days

Thu Dec 29, 2022 10:46 pm

@sirbryan

check your connection tracking table size using
/ip firewall connection tracking print 
check total-entries: value at peak hour

i have seen Routers Working OK with around 700k-800k
 
R1CH
Forum Guru
Forum Guru
Posts: 1098
Joined: Sun Oct 01, 2006 11:44 pm

Re: NAT Issues every 10-14 days

Fri Dec 30, 2022 2:58 am

I hope I never have to use such ISPs that mess with the TCP established timeout... 1 day is already significantly lower than the Linux default (5 days). Unless you're actually running out of memory due to conntrack entries, I really don't recommend touching this. Sure, most home users who do simple web browsing or Netflix will not notice or care, but some of us like TCP/IP to actually behave as-designed. I use a lot of SSH connections and can immediately tell when I'm behind a shitty hotel NAT that's screwed around with the timeouts, suddenly my sessions start to die since the server tried to send me data but the NAT had destroyed my binding.
 
User avatar
sirbryan
Member Candidate
Member Candidate
Posts: 298
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: NAT Issues every 10-14 days

Fri Dec 30, 2022 6:22 am

Call me a crappy ISP that messes with stuff. I also use CGNAT (the horror), and my IPV6 network is torn apart at the moment. Nevertheless, these settings have worked well on this particular CGNAT router for roughly a year with zero complaints from users.
      tcp-syn-sent-timeout: 5s
  tcp-syn-received-timeout: 5s
   tcp-established-timeout: 2h5m
      tcp-fin-wait-timeout: 10s
    tcp-close-wait-timeout: 10s
      tcp-last-ack-timeout: 10s
     tcp-time-wait-timeout: 10s
         tcp-close-timeout: 10s
   tcp-max-retrans-timeout: 2m
       tcp-unacked-timeout: 1m
        loose-tcp-tracking: yes
               udp-timeout: 17s
        udp-stream-timeout: 3m
              icmp-timeout: 10s
           generic-timeout: 10m
               max-entries: 1048576
             total-entries: 43234
             
It seems I have room for 100K more entries. I can see the total-entries being higher if I had left the tcp-established-timeout to the default of 24h.

But back to OP. It would be interesting to see what his 1072's total-entries looks like every 8-10 days.

Who is online

Users browsing this forum: No registered users and 81 guests