NAT Issues every 10-14 days

ErkDog · June 2, 2022, 12:04am

I have a 3Gbps Comcast fiber circuit and a 1 Gbps FiOS on a CCR1072-1G-8S+ - RouterOS 7.2.3

I do have RemoteWinBox currently setup to pull config backups, that’s about all I do with it.

Every 10 - 14 days, NAT seems to stop working.

All servers inside the network become unavailable, and all internal computers stop being able to get out.

The Mikrotik is up, because I solve it by logging in with WinBox and telling it to reboot.

Every time there is a router os upgrade, I hope that I see something about NAT in patch notes, and the issue goes away, but it does not.

If I’m not at the office when it happens, I’m able to Wireguard in, and connect to the Mikrotik and reboot it remotely.

I don’t really have time to “dig into” why it’s not working when it stops, because I’m constantly getting offline alerts and phone calls and have to resolve the issue as promptly as possible to restore connectivity to customers.

One thing of note, is that if I mark the Comcast Gateway as disabled, everything works over FIOS.

I haven’t tried re-enabling the Comcast Gateway to see if it comes back up, because I just reboot the mikrotik, and reenable it to get things back to normal as soon as possible.

IPv6 traffic seems to continue to work as well, which of course is not using NAT either.

Any ideas or thoughts?

ErkDog · June 14, 2022, 12:10am

It just did all this again.

Any Ideas?

chechito · June 14, 2022, 12:41am

i had similar issue with ccr1072 some months ago with routeros 6.48.6

in my case many hosts being src-natted with a mascarade rule wich uses a public ip, that same public ip is the same of another scr-nat rules for other groups of hosts using deterministic nat 444 cg-nat

the “solution” was to use independent public ip for src-nat mascarade and src-nat nat 44 cgnat rules

lets call this issue some kind of src-nat collision

as you say the problem affect customer service so i dont have opportunity to test it too much, since i make that changes the issue is gone

i hope my case can help you

Please confirm if you can solve it in the same way

Znevna · June 14, 2022, 4:22am

My money is on a butchered config.

ErkDog · July 28, 2022, 5:11am

Thank you for your reply @chechito.

I’m having a little trouble understanding exactly what you did to resolve the issue though.

I have a public routable IP from my ISP not behind carrier-grade NAT or anything.

Also, @Znevna, just saying “I bet your config is screwed up.” is not helpful at all, nor does it make any sense.

If my overall config was broken, then things wouldn’t work right at all.

All NAT wouldn’t just stop working every 10-14 days.

As this is an enterprise piece of equipment, that was not cheap, it should be able to handle anything I throw at it, and if it can’t it should tell me why.

Neither of those two things are happening.

I guess I’ll open an official ticket and refer to this thread for context since no clear path to resolution has been provided.

Thanks,
Matt

mkx · July 28, 2022, 5:30am

It’s as helpful as it gets. The problem you vaguely report (NAT stopping to work every now and then) isn’t well known, that’s for sure.
Most problems we see in this forum are due to configuration, only a few are actual bugs. Statistics thus says it’s more probable your config is screwed up than that you hit an actual bug. But it’s impossible to tell without seeing actual config you’re running (and you didn’t provide it). And if it was a bug, it’s obviously rarely hit. MT support can’t fix it if they can’t replicate it … and you can’t (or won’t) provide instructions on how to (reliably) replicate the problem.

holvoetn · July 28, 2022, 6:02am

@ErkDog:

“My car doesn’t run, what’s wrong with it ?”
You see the problem here ?
How would anyone be able to help in an efficient way with such a problem description ?
Can be anything.

To get to the point where we need to be:
the only way anyone is able to verify what might be wrong with your config, is for you to show it.
It might also help to make a small drawing on what’s connected where, with indication of used IP address (obfuscated, see below).

terminal:
/export hide-sensitive file=anynameyouwish
Review export for remaining sensitive info (check for secret keys, remove serial number!, change public IPs to PubIP1- PubIP2-… ) and post here between [__code] quotes.

chechito · July 28, 2022, 3:12pm

+1 for export

ErkDog · August 10, 2022, 8:00pm

Here is config.

# aug/10/2022 15:56:32 by RouterOS 7.4
# software id = <CENSORED>
#
# model = CCR1072-1G-8S+
# serial number = <CENSORED>
/interface sstp-client
add comment="Remote Winbox connection for WilsonAve" connect-to=<CENSORED> disabled=no name=RemoteWinboxVPN3 user=no
/interface ethernet
set [ find default-name=ether1 ] name=ETH1MGMT
set [ find default-name=sfp-sfpplus1 ] name=SFP1-LAN
set [ find default-name=sfp-sfpplus2 ] advertise=1000M-full disabled=yes name=SFP2-FIOSWAN speed=1Gbps
set [ find default-name=sfp-sfpplus3 ] name=SFP3-CCWAN
set [ find default-name=sfp-sfpplus4 ] advertise=1000M-full name=SFP4-SDWAN
/interface wireguard
add listen-port=13231 mtu=1420 name=WireGuard
/interface list
add include=static name=WANS
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool0 ranges=10.8.8.100-10.8.8.200
/ip dhcp-server
add address-pool=dhcp_pool0 interface=SFP1-LAN lease-time=12h name=dhcp1
/ipv6 dhcp-server
add address-pool=CCast interface=SFP1-LAN lease-time=4w2d name=CCast
/ipv6 pool
add name=CCast prefix=<CENSORED>:/64 prefix-length=64
/port
set 0 name=serial0
set 1 name=serial1
/routing bgp template
set default disabled=no output.network=bgp-networks
/routing ospf instance
add disabled=no name=default-v2
add disabled=no name=default-v3 version=3
/routing ospf area
add disabled=yes instance=default-v2 name=backbone-v2
add disabled=yes instance=default-v3 name=backbone-v3
/routing table
add fib name=ROUTE2FIOS
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set max-neighbor-entries=8192
/interface detect-internet
set internet-interface-list=WANS wan-interface-list=WANS
/interface list member
add interface=SFP3-CCWAN list=WANS
add interface=SFP2-FIOSWAN list=WANS
/interface ovpn-server server
set auth=sha1,md5
/interface wireguard peers
add allowed-address=192.168.253.101/32 comment=Erk-Legion2 interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=192.168.253.100/32 comment=Erk-S20 interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=192.168.0.0/24,192.168.253.1/32 comment="CrtrCreek-Router (Uses 192.168.253.1)" endpoint-address=<CENSORED> endpoint-port=13231 interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=192.168.253.102/32 comment="Dawn - Android19" interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=192.168.253.105/32 comment=John-Desktop interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=192.168.253.106/32 comment=Antonizoon-Phone interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=192.168.253.107/32 comment=doukaina interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=192.168.253.108/32 comment=Keeter-Phone interface=WireGuard persistent-keepalive=5m public-key="no"
add allowed-address=10.7.7.0/24,192.168.253.2/32 comment="KetterTIK (Uses 192.168.253.2)" endpoint-address=<CENSORED> endpoint-port=13231 interface=WireGuard persistent-keepalive=5m public-key="no"
/ip address
add address=192.168.88.1/24 comment=defconf interface=ETH1MGMT network=192.168.88.0
add address=10.8.8.5/24 interface=SFP1-LAN network=10.8.8.0
add address=<CENSORED>.26/30 interface=SFP3-CCWAN network=<CENSORED>.24
add address=192.168.253.5/24 interface=WireGuard network=192.168.253.0
add address=<CENSORED>.38/27 interface=SFP4-SDWAN network=<CENSORED>.32
/ip cloud
set ddns-enabled=yes ddns-update-interval=3m
/ip dhcp-client
add default-route-distance=3 disabled=yes interface=SFP2-FIOSWAN use-peer-dns=no use-peer-ntp=no
add default-route-distance=4 disabled=yes interface=SFP4-SDWAN use-peer-dns=no use-peer-ntp=no
/ip dhcp-server config
set store-leases-disk=15m
/ip dhcp-server lease
add address=10.8.8.62 client-id=1:a0:ce:c8:e3:6f:82 mac-address=A0:CE:C8:E3:6F:82 server=dhcp1
add address=10.8.8.98 client-id=ff:f7:f6:49:34:0:2:0:0:ab:11:3a:e3:f4:36:4b:0:a6:70 mac-address=DC:A6:32:07:D0:B1 server=dhcp1
add address=10.8.8.7 mac-address=80:CC:9C:82:E7:08 server=dhcp1
add address=10.8.8.97 client-id=1:e0:d5:5e:87:c7:e0 mac-address=E0:D5:5E:87:C7:E0 server=dhcp1
add address=10.8.8.41 mac-address=00:0C:15:04:30:57 server=dhcp1
add address=10.8.8.40 mac-address=00:0C:15:04:2F:EC server=dhcp1
add address=10.8.8.49 client-id=1:34:9f:7b:a4:3:eb mac-address=34:9F:7B:A4:03:EB server=dhcp1
add address=10.8.8.34 mac-address=00:09:F5:27:48:66 server=dhcp1
add address=10.8.8.60 client-id=ff:7a:f:84:9a:0:1:0:1:28:da:ac:5d:1c:69:7a:f:84:9a mac-address=1C:69:7A:0F:84:9A server=dhcp1
add address=10.8.8.61 client-id=1:e8:ea:6a:9:65:54 mac-address=E8:EA:6A:09:65:54 server=dhcp1
add address=10.8.8.50 client-id=1:e4:5f:1:37:56:e7 mac-address=E4:5F:01:37:56:E7 server=dhcp1
add address=10.8.8.35 mac-address=00:09:F5:2A:C0:D3 server=dhcp1
add address=10.8.8.81 client-id=1:8c:85:80:d6:ad:b2 comment=EUFY-LROOM mac-address=8C:85:80:D6:AD:B2 server=dhcp1
add address=10.8.8.80 client-id=1:8c:85:80:d4:ab:46 comment=EUFY-SERVEROOM mac-address=8C:85:80:D4:AB:46 server=dhcp1
/ip dhcp-server network
add address=10.8.8.0/24 dns-server=10.8.8.4 domain=ecansol.loc gateway=10.8.8.5 netmask=24 ntp-server=10.8.8.4
/ip dns
set servers=10.8.8.4
/ip firewall address-list
add address=<CENSORED>.26 list=WANIPS
add address=<CENSORED>.38 list=WANIPS
/ip firewall filter
add action=accept chain=input protocol=icmp
add action=accept chain=input dst-port=13231 protocol=udp
add action=accept chain=input dst-port=13231 protocol=tcp
add action=accept chain=forward dst-address=10.8.8.0/24 src-address=192.168.0.0/24
add action=accept chain=forward dst-address=192.168.0.0/24 src-address=10.8.8.0/24
add action=accept chain=forward dst-address=10.7.7.0/24 src-address=10.8.8.0/24
add action=accept chain=forward dst-address=10.8.8.0/24 src-address=10.7.7.0/24
add action=accept chain=forward dst-address=0.0.0.0/0 src-address=10.7.7.0/24
add action=accept chain=forward dst-address=10.7.7.0/24 src-address=0.0.0.0/0
add action=accept chain=input comment="Allow Remote Winbox" in-interface=RemoteWinboxVPN3
add action=reject chain=input dst-address-list=WANIPS dst-port=2000 protocol=tcp reject-with=icmp-network-unreachable
add action=reject chain=input dst-address-list=WANIPS dst-port=2000 protocol=udp reject-with=icmp-network-unreachable
add action=reject chain=input dst-address-list=WANIPS dst-port=5678 protocol=udp reject-with=icmp-network-unreachable
add action=reject chain=input dst-address-list=WANIPS dst-port=5678 protocol=tcp reject-with=icmp-network-unreachable
add action=drop chain=input dst-address-list=WANIPS dst-port=53 protocol=tcp
add action=drop chain=input dst-address-list=WANIPS dst-port=53 protocol=udp
add action=accept chain=forward dst-address=0.0.0.0/0 src-address=0.0.0.0/0
/ip firewall mangle
add action=mark-routing chain=prerouting comment=NetMgmt-VZFios new-routing-mark=ROUTE2FIOS passthrough=yes src-address=10.8.8.65
add action=mark-routing chain=prerouting comment=Ops-Skull-SDWAN new-routing-mark=ROUTE2FIOS passthrough=yes src-address=10.8.8.101
/ip firewall nat
add action=masquerade chain=srcnat comment="NAT FOR SDWAN" out-interface=SFP4-SDWAN
add action=masquerade chain=srcnat comment="NAT FOR CC" out-interface=SFP3-CCWAN
add action=dst-nat chain=dstnat comment=NetMGMT dst-address-list=WANIPS dst-port=9443 protocol=tcp to-addresses=10.8.8.4 to-ports=9443
add action=dst-nat chain=dstnat comment="Master - SRV" dst-address-list=WANIPS dst-port=60059 protocol=tcp to-addresses=10.8.8.21 to-ports=60059
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=80 protocol=tcp to-addresses=10.8.8.21 to-ports=80
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=443 protocol=tcp to-addresses=10.8.8.21 to-ports=443
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2083 protocol=tcp to-addresses=10.8.8.21 to-ports=2083
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2087 protocol=tcp to-addresses=10.8.8.21 to-ports=2087
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2096 protocol=tcp to-addresses=10.8.8.21 to-ports=2096
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=53 protocol=tcp to-addresses=10.8.8.21 to-ports=53
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=53 protocol=udp to-addresses=10.8.8.21 to-ports=53
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25 protocol=tcp to-addresses=10.8.8.21 to-ports=25
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=993 protocol=tcp to-addresses=10.8.8.21 to-ports=993
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=995 protocol=tcp to-addresses=10.8.8.21 to-ports=995
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=587 protocol=tcp to-addresses=10.8.8.21 to-ports=587
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=465 protocol=tcp to-addresses=10.8.8.21 to-ports=465
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=30033 protocol=tcp to-addresses=10.8.8.21 to-ports=30033
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=30033 protocol=udp to-addresses=10.8.8.21 to-ports=30033
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=10011 protocol=tcp to-addresses=10.8.8.21 to-ports=10011
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=10011 protocol=udp to-addresses=10.8.8.21 to-ports=10011
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=9987 protocol=tcp to-addresses=10.8.8.21 to-ports=9987
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=9987 protocol=udp to-addresses=10.8.8.21 to-ports=9987
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8111 protocol=tcp to-addresses=10.8.8.21 to-ports=8111
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8111 protocol=udp to-addresses=10.8.8.21 to-ports=8111
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8110 protocol=tcp to-addresses=10.8.8.21 to-ports=8110
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8110 protocol=udp to-addresses=10.8.8.21 to-ports=8110
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8610 protocol=tcp to-addresses=10.8.8.21 to-ports=8610
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8610 protocol=udp to-addresses=10.8.8.21 to-ports=8610
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8611 protocol=tcp to-addresses=10.8.8.21 to-ports=8611
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8611 protocol=udp to-addresses=10.8.8.21 to-ports=8611
add action=dst-nat chain=dstnat comment="Anton Desktop - ketilfastr" dst-address-list=WANIPS dst-port=43030 protocol=tcp to-addresses=10.8.8.97 to-ports=43030
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2053 protocol=tcp to-addresses=10.8.8.97 to-ports=443
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8880 protocol=tcp to-addresses=10.8.8.97 to-ports=80
add action=dst-nat chain=dstnat comment="BA-NUC - SSH" dst-address-list=WANIPS dst-port=43028 protocol=tcp to-addresses=10.8.8.60 to-ports=43028
add action=dst-nat chain=dstnat comment="BA - rPI" dst-address-list=WANIPS dst-port=43029 protocol=tcp to-addresses=10.8.8.98 to-ports=43029
add action=dst-nat chain=dstnat comment=DIFFDEV dst-address-list=WANIPS dst-port=60070 protocol=tcp to-addresses=10.8.8.62 to-ports=60070
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7000 protocol=tcp to-addresses=10.8.8.62 to-ports=7000
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7001 protocol=tcp to-addresses=10.8.8.62 to-ports=7001
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7002 protocol=tcp to-addresses=10.8.8.62 to-ports=7002
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7003 protocol=tcp to-addresses=10.8.8.62 to-ports=7003
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7004 protocol=tcp to-addresses=10.8.8.62 to-ports=7004
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7005 protocol=tcp to-addresses=10.8.8.62 to-ports=7005
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=3000 protocol=tcp to-addresses=10.8.8.62 to-ports=3000
add action=dst-nat chain=dstnat comment=MYTHVA1 dst-address-list=WANIPS dst-port=8085 protocol=tcp to-addresses=10.8.8.61 to-ports=8085
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=17777 protocol=tcp to-addresses=10.8.8.61 to-ports=17777
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=17777 protocol=udp to-addresses=10.8.8.61 to-ports=17777
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=17778 protocol=tcp to-addresses=10.8.8.61 to-ports=17778
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=17778 protocol=udp to-addresses=10.8.8.61 to-ports=17778
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=37015 protocol=tcp to-addresses=10.8.8.61 to-ports=37015
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=37015 protocol=udp to-addresses=10.8.8.61 to-ports=37015
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=19132 protocol=tcp to-addresses=10.8.8.61 to-ports=19132
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=19132 protocol=udp to-addresses=10.8.8.61 to-ports=19132
add action=dst-nat chain=dstnat comment="Matrix - SRV" dst-address-list=WANIPS dst-port=60065 protocol=tcp to-addresses=10.8.8.15 to-ports=60065
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8448 protocol=tcp to-addresses=10.8.8.15 to-ports=8448
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8448 protocol=udp to-addresses=10.8.8.15 to-ports=8448
add action=dst-nat chain=dstnat comment=GOKU dst-address-list=WANIPS dst-port=60052 protocol=tcp to-addresses=10.8.8.17 to-ports=60052
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8080 protocol=tcp to-addresses=10.8.8.17 to-ports=8080
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=4233 protocol=tcp to-addresses=10.8.8.17 to-ports=4233
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2224 protocol=tcp to-addresses=10.8.8.17 to-ports=2224
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=4234 protocol=tcp to-addresses=10.8.8.17 to-ports=4234
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=4235 protocol=tcp to-addresses=10.8.8.17 to-ports=4235
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=6499 protocol=udp to-addresses=10.8.8.17 to-ports=6499
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=6499 protocol=tcp to-addresses=10.8.8.17 to-ports=6499
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=6599 protocol=udp to-addresses=10.8.8.17 to-ports=6599
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=6599 protocol=tcp to-addresses=10.8.8.17 to-ports=6599
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=6597 protocol=udp to-addresses=10.8.8.17 to-ports=6597
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=6597 protocol=tcp to-addresses=10.8.8.17 to-ports=6597
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25566 protocol=udp to-addresses=10.8.8.17 to-ports=25566
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25566 protocol=tcp to-addresses=10.8.8.17 to-ports=25566
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25501 protocol=udp to-addresses=10.8.8.17 to-ports=25501
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25501 protocol=tcp to-addresses=10.8.8.17 to-ports=25501
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25565 protocol=udp to-addresses=10.8.8.17 to-ports=25565
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25565 protocol=tcp to-addresses=10.8.8.17 to-ports=25565
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25591 protocol=udp to-addresses=10.8.8.17 to-ports=25591
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25591 protocol=tcp to-addresses=10.8.8.17 to-ports=25591
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7777 protocol=udp to-addresses=10.8.8.17 to-ports=7777
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7778 protocol=tcp to-addresses=10.8.8.17 to-ports=7778
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7778 protocol=udp to-addresses=10.8.8.17 to-ports=7778
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7777 protocol=tcp to-addresses=10.8.8.17 to-ports=7777
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=15000 protocol=udp to-addresses=10.8.8.17 to-ports=15000
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=15000 protocol=tcp to-addresses=10.8.8.17 to-ports=15000
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=15777 protocol=udp to-addresses=10.8.8.17 to-ports=15777
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=15777 protocol=tcp to-addresses=10.8.8.17 to-ports=15777
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=5678 protocol=udp to-addresses=10.8.8.17 to-ports=5678
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=5678 protocol=tcp to-addresses=10.8.8.17 to-ports=5678
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=5679 protocol=tcp to-addresses=10.8.8.17 to-ports=5679
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=5679 protocol=udp to-addresses=10.8.8.17 to-ports=5679
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2226 protocol=tcp to-addresses=10.8.8.17 to-ports=2226
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2226 protocol=udp to-addresses=10.8.8.17 to-ports=2226
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8123 protocol=tcp to-addresses=10.8.8.17 to-ports=8123
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8123 protocol=udp to-addresses=10.8.8.17 to-ports=8123
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=34197 protocol=tcp to-addresses=10.8.8.17 to-ports=34197
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=34197 protocol=udp to-addresses=10.8.8.17 to-ports=34197
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=34198 protocol=tcp to-addresses=10.8.8.17 to-ports=34198
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=34198 protocol=udp to-addresses=10.8.8.17 to-ports=34198
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=34199 protocol=tcp to-addresses=10.8.8.17 to-ports=34199
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=34199 protocol=udp to-addresses=10.8.8.17 to-ports=34199
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2230 protocol=tcp to-addresses=10.8.8.17 to-ports=2230
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=2230 protocol=udp to-addresses=10.8.8.17 to-ports=2230
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=26015 protocol=tcp to-addresses=10.8.8.17 to-ports=26015
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=26015 protocol=udp to-addresses=10.8.8.17 to-ports=26015
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=26016 protocol=tcp to-addresses=10.8.8.17 to-ports=26016
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=26016 protocol=udp to-addresses=10.8.8.17 to-ports=26016
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25567 protocol=tcp to-addresses=10.8.8.17 to-ports=25567
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=25567 protocol=udp to-addresses=10.8.8.17 to-ports=25567
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=29335 protocol=tcp to-addresses=10.8.8.17 to-ports=29335
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=29335 protocol=udp to-addresses=10.8.8.17 to-ports=29335
add action=dst-nat chain=dstnat comment=GIRU dst-address-list=WANIPS dst-port=24388 protocol=tcp to-addresses=10.8.8.25 to-ports=24388
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=24388 protocol=udp to-addresses=10.8.8.25 to-ports=24388
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7781 protocol=udp to-addresses=10.8.8.25 to-ports=7781
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=7781 protocol=tcp to-addresses=10.8.8.25 to-ports=7781
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=45882 protocol=tcp to-addresses=10.8.8.25 to-ports=45882
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=45882 protocol=udp to-addresses=10.8.8.25 to-ports=45882
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=45883 protocol=tcp to-addresses=10.8.8.25 to-ports=45883
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=45883 protocol=udp to-addresses=10.8.8.25 to-ports=45883
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=45884 protocol=tcp to-addresses=10.8.8.25 to-ports=45884
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=45884 protocol=udp to-addresses=10.8.8.25 to-ports=45884
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=28967 protocol=tcp to-addresses=10.8.8.25 to-ports=28967
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=28967 protocol=udp to-addresses=10.8.8.25 to-ports=28967
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=32400 protocol=tcp to-addresses=10.8.8.25 to-ports=32400
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=32400 protocol=udp to-addresses=10.8.8.25 to-ports=32400
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=16261 protocol=tcp to-addresses=10.8.8.25 to-ports=16261
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=16261 protocol=udp to-addresses=10.8.8.25 to-ports=16261
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8766 protocol=tcp to-addresses=10.8.8.25 to-ports=8766
add action=dst-nat chain=dstnat dst-address-list=WANIPS dst-port=8766 protocol=udp to-addresses=10.8.8.25 to-ports=8766
/ip route
add check-gateway=ping comment="Pri CC Gateway-Disable if Forcing FiOS" disabled=yes distance=1 dst-address=0.0.0.0/0 gateway=<CENSORED>.25 pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=5 dst-address=192.168.0.0/24 gateway=192.168.253.1 pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=2 dst-address=0.0.0.0/0 gateway=<CENSORED>.33 pref-src=0.0.0.0 routing-table=ROUTE2FIOS scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=5 dst-address=192.168.0.0/24 gateway=192.168.253.1 pref-src=0.0.0.0 routing-table=ROUTE2FIOS scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=5 dst-address=192.168.253.0/24 gateway=WireGuard routing-table=ROUTE2FIOS scope=10 suppress-hw-offload=no
add check-gateway=ping disabled=yes distance=3 dst-address=0.0.0.0/0 gateway=<CENSORED>.25 pref-src=0.0.0.0 routing-table=ROUTE2FIOS scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping disabled=no distance=10 dst-address=0.0.0.0/0 gateway=<CENSORED>.33 pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=5 dst-address=10.7.7.0/24 gateway=192.168.253.2 pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=5 dst-address=10.7.7.0/24 gateway=192.168.253.2 pref-src=0.0.0.0 routing-table=ROUTE2FIOS scope=30 suppress-hw-offload=no target-scope=10
/ipv6 route
add check-gateway=ping disabled=no distance=1 dst-address=/0 gateway=<CENSORED>:5a05 scope=30 target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=/0 gateway=<CENSORED>:5a05 scope=30 target-scope=10
add gateway=<CENSORED>:5a05%SFP3-CCWAN
/ip service
set telnet address=10.8.8.0/24,192.168.88.0/24,<CENSORED>.237/32 disabled=yes
set ftp address=10.8.8.0/24,192.168.88.0/24,<CENSORED>.237/32 disabled=yes
set www address=10.8.8.0/24,192.168.88.0/24,<CENSORED>.237/32 disabled=yes
set ssh address=10.8.8.0/24,192.168.88.0/24,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
set www-ssl address=10.8.8.0/24,192.168.88.0/24,<CENSORED>.237/32
set api address=10.8.8.0/24,192.168.88.0/24,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
set winbox address="10.8.8.0/24,192.168.88.0/24,192.168.0.0/24,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,192.168.253.0/24"
set api-ssl address=10.8.8.0/24,192.168.88.0/24,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
/ip smb shares
add comment="default share" directory=/pub name=pub
add comment="default share" directory=/pub name=pub
/ip smb users
add name=guest
add name=guest
/ip upnp
set enabled=yes
/ip upnp interfaces
add interface=SFP1-LAN type=internal
add interface=SFP3-CCWAN type=external
/ipv6 address
add address=<CENSORED>:5a06/126 advertise=no interface=SFP3-CCWAN
add address=<CENSORED>:1 interface=SFP1-LAN
/ipv6 nd
set [ find default=yes ] dns=<CENSORED>:4 hop-limit=64 interface=SFP1-LAN managed-address-configuration=yes
/lcd
set backlight-timeout=5m default-screen=stats-all
/lcd pin
set pin-number=<CENSORED>
/system clock
set time-zone-name=America/New_York
/system identity
set name=ECANWA
/system ntp client
set enabled=yes
/system ntp client servers
add address=10.8.8.4

chechito · August 10, 2022, 9:29pm

please edit your public ip address data for privacy reasons

rextended · August 10, 2022, 9:31pm

One moment, I censore the config and repost

chechito · August 10, 2022, 9:36pm

i read your config and does not match my scenario so no luck with that

the only idea i have is

maybe if you change action from mascarade to scr-nat specifying outbound public ip address in your src-nat rules the issue maybe solves

if you want to use multiple public ip address use action same and mark the option not by dst specifying public ip address range to use (need to be contiguous ip addresses)

rextended · August 10, 2022, 9:39pm

Replaced with local censored version.

I don’t know if I have removed all sensitive data,
but it is the poster’s responsibility not to disclose them.

loloski · August 10, 2022, 10:14pm

In a plain linux system nat issues happen if conntrack entries is too much for your system to handle, so when it happen again please post this before you reboot the system, just my 0.2$

/ip firewall/connection/tracking/print

and also post system logs here during the onset of the issue because we might see some peculiar entries in the logs pertains to conntrack entries, the suggestion of @chehito is also good to snat your connection to your specific public ip

chechito · August 11, 2022, 12:04am

+1 on

/ip firewall/connection/tracking/print

at peak hour of traffic

anav · August 11, 2022, 12:27am

Jajajaj are you saying his router almost has a period? But every 12-14 days?
If the router has a peak period it would be daily or weekly LOL…

ErkDog · August 11, 2022, 2:14pm

Thanks for removing IPs, I wasn’t too worried about it though, I run production business services off this device so they are not a secret .

I would think that a $3,000 router touted as a ‘flagship’ device, wouldn’t have an upward limitation on “the number of connections it can handle” and if it did it would be absurdly high.

I will collect this information next time it hangs up.

Thanks all.

chechito · August 11, 2022, 2:50pm

Is not necessary to wait until hangs, check conn-track print at your peak hour to have an idea of your connections load to see if can be the source of the problem

Connection tracking can be tuned to solve it if it is necessary

ErkDog · August 12, 2022, 4:46am

Well this is what it looks like right now:

[admin@ECANWA] > /ip firewall/connection/tracking/print
enabled: auto
tcp-syn-sent-timeout: 5s
tcp-syn-received-timeout: 5s
tcp-established-timeout: 1d
tcp-fin-wait-timeout: 10s
tcp-close-wait-timeout: 10s
tcp-last-ack-timeout: 10s
tcp-time-wait-timeout: 10s
tcp-close-timeout: 10s
tcp-max-retrans-timeout: 5m
tcp-unacked-timeout: 5m
loose-tcp-tracking: yes
udp-timeout: 10s
udp-stream-timeout: 3m
icmp-timeout: 10s
generic-timeout: 10m
max-entries: 1048576
total-entries: 1512

There’s not really any time I can think of it’s peak vs not peak.

Is connection tracking related to NAT?

Again, this seems like kind of a peculiar limitation on a $3,000 router with 16G of ram and the processor on this thing is so absurd that it’s almost always at 0%.

This is a CCR1072-1G-8S+ :-/

rextended · August 12, 2022, 7:49am

Actually is useless 1 day timeout on one TCP connection.
(I use on all my core routers
/ip firewall connection tracking set loose-tcp-tracking=no tcp-established-timeout=30m
and I never have problems or complains from 4000+ users)

The timeout is triggered after the time specified.
For default, after 1 day no one TCP packet pass over that tracked connection, the conntrack remove the track and free port and resources.

The port numbers used for NAT are limited to 32767 (ok, after some config 65535, but is not this the point) and when all port are busy
for already tracked connection, the NAT stop working.

Example:
Against your Public IP 100.64.3.6 100 natted users use Google.
The DNS on that moment solve for everyone google.com to 172.23.25.14
The NAT can not reuse same ports, because on that case all the incoming packet directedo to 172.23.25.14 is coming all to port 46789 (and all from 443):
how distinguish on conntrack to what local IP must be sended the packet?
NAT is forced everytime to use different ports.
NAT can reuse same port, but with different IP, conntrack do the rest.

I do not know how many internal IPs you have, but probably you deplete all the available port combination for NAT,
changing gateway you have another IP usable, and the NAT work because have another group of 32767 ports usable.

Also when something is not working, mus be checked if only TCP services are stopped or UDP services.
Connecting directly the RB work because you use directly the IP of the device without use NAT.

Try this when all appear locked, and see if all working again:

/ip fire conn
:foreach idc in=[find where (timeout>60)] do={
    remove [find where .id=$idc]
}

Sorry for my english, I hope I have explained what I mean.