Community discussions

MikroTik App
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Inconsistent WG connection via 2 ISPs

Wed Jul 17, 2024 2:07 am

ok, I'm stumped on this.
I have a situation where I've had WG working on a specific 5009 for months. The site added a 2nd ISP. Wanted to be able to use WG on both connections.
I enabled ECMP on both default routes, and added the following mangles to keep ISP1 and ISP2 connections to their own lane.
ip firewall mangle
add action=mark-connection chain=input comment="Keep eth1 inbound connections to eth1" connection-mark=no-mark in-interface=ether1 new-connection-mark=ISP1-IN passthrough=yes
add action=mark-routing chain=output connection-mark=ISP1-IN new-routing-mark=ISP1 passthrough=yes
add action=mark-connection chain=input comment="Keep eth2 inbound connections to eth2" connection-mark=no-mark in-interface=ether2 new-connection-mark=ISP2-IN passthrough=yes
add action=mark-routing chain=output connection-mark=ISP2-IN new-routing-mark=ISP2 passthrough=yes
If I try to connect to this WG instance via ISP1, via my wired ISP, handshake fails. If I try to connect to this WG instance via ISP1, via my LTE data, handshake works everytime.

If I try to connect to this WG instance via ISP2, via my wired ISP, handshake works. If I try to connect to this WG instance via ISP2, via my LTE data, handshake works everytime.

To make sure I did not have a WG config error or any kind of other user overlap. I created a brand new WG instance, seperate subnet, allowed it out, allowed the WG ports in... and the exact same symptoms.

I'm struggling to find why this would block me on WG on my hard wired ISP vs say my mobile data.
Any suggestions on what to check?
Last edited by CTSsean on Wed Jul 17, 2024 10:32 pm, edited 1 time in total.
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12323
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Inconsistent WG connection via 2 ISPs

Wed Jul 17, 2024 2:35 am

Since the mangle alone do not do all the works, all the other part of the config must be analyzed, like missing interface on WAN list, error on routing table, etc. etc.

If you need to have your car checked out, do you just bring only the tank cap to the mechanic?
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Inconsistent WG connection via 2 ISPs

Wed Jul 17, 2024 2:36 am

 
CGGXANNX
Member Candidate
Member Candidate
Posts: 170
Joined: Thu Dec 21, 2023 6:45 pm

Re: Inconsistent WG connection via 2 ISPs

Wed Jul 17, 2024 6:09 am

I can confirm that this is a bug reproducible on multiple installations using ECMP. And the workaround is the dstnat rule as posted by @anav.
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Wed Jul 17, 2024 6:13 pm

Since the mangle alone do not do all the works, all the other part of the config must be analyzed, like missing interface on WAN list, error on routing table, etc. etc.

If you need to have your car checked out, do you just bring only the tank cap to the mechanic?
Unlike a car, I can't bring you the physical device. There's a lot of superfluous config on this device that is outside the purpose of WG and I'd be gladly posts the parts that are requested. To sanitize the entire config would take hours.

Both ISP interface are on the WANlist. I'm not flawless and I'm more than willing to admit its possible I made a config mistake. If you have specific sections you'd like to see, I'll sanitize those and post them.
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Wed Jul 17, 2024 6:14 pm

Thank you kind sir! Is this an issue with Wireguard itself, or just the way RouterOS implements it?
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Wed Jul 17, 2024 7:27 pm

I can confirm that this is a bug reproducible on multiple installations using ECMP. And the workaround is the dstnat rule as posted by @anav.
I have a question about this.
/ip firewall nat
add action=dstnat chain=dst-nat in-interface=ether2 dst-address-type=local dst-port=13231 protocol=udp to-address=172.16.0.1
172.16.0.1 looks to be the default gateway for ISP1.

Why would DST natting a connection coming in ISP2 (eth2) to the gateway on ISP1 (eth1)?
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Inconsistent WG connection via 2 ISPs

Wed Jul 17, 2024 10:06 pm

I can confirm that this is a bug reproducible on multiple installations using ECMP. And the workaround is the dstnat rule as posted by @anav.
Why would DST natting a connection coming in ISP2 (eth2) to the gateway on ISP1 (eth1)?
The gist of is, that the response to a query is already bleeding out of WAN1 instead of WAN2, for the initial handshake.
Therefore we tell the router that all traffic coming from ether2 (wan2) is destinatted to WAN1.

Thus when WAN1 incorrectly replies to traffic from that port ithe router sends the traffic UN Destinationatted back via WAn2. (ether2).
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Wed Jul 17, 2024 10:19 pm



Why would DST natting a connection coming in ISP2 (eth2) to the gateway on ISP1 (eth1)?
The gist of is, that the response to a query is already bleeding out of WAN1 instead of WAN2, for the initial handshake.
Therefore we tell the router that all traffic coming from ether2 (wan2) is destinatted to WAN1.

Thus when WAN1 incorrectly replies to traffic from that port ithe router sends the traffic UN Destinationatted back via WAn2. (ether2).
This breaks my brain. WoW. How did anyone figure this out? Lots of packet captures?

In my case, where I have 2 WG instances. One designated for WAN1 and one WAN2 (different ports). Should I do the criss cross of its?

Example:
/ip firewall nat
add action=dstnat chain=dst-nat in-interface=ether2 dst-address-type=local dst-port=13232 protocol=udp to-address=WAN1Gateway

/ip firewall nat
add action=dstnat chain=dst-nat in-interface=ether1 dst-address-type=local dst-port=13231 protocol=udp to-address=WAN2Gateway
Now, here's a thing what we've noticed in our testing. Reboots cause cause the 'active' WG interface to change. So for my issue, yesterday WAN1 would not work hardly at all for WG. However after a reboot last night, WG works fine on WAN1, but doesn't work at all for WAN2 today.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Inconsistent WG connection via 2 ISPs

Wed Jul 17, 2024 10:23 pm

No this only affects the WAN that is second in natural priority ( the failover wan so to speak).
You still need some mangling going on, and so I would have to see the config to comment further
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Wed Jul 17, 2024 10:30 pm

No this only affects the WAN that is second in natural priority ( the failover wan so to speak).
You still need some mangling going on, and so I would have to see the config to comment further
What parts of the config do you need and I'll gather those?
My thoughts are...
interface wireguard (i'll need to hide the private keys)
ip route
ip addresses (i'll change the public IP)
ip firewall filter
ip firewall nat
ip firewall mangle

Anything else?
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Inconsistent WG connection via 2 ISPs

Thu Jul 18, 2024 12:20 am

/export file=anynameyouwish ( minus router serial number, any public WANIP info, keys etc.)
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Thu Jul 18, 2024 12:41 am

/export file=anynameyouwish ( minus router serial number, any public WANIP info, keys etc.)
Understood. I'll start working on that.
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Thu Jul 18, 2024 3:39 am

/export file=anynameyouwish ( minus router serial number, any public WANIP info, keys etc.)
Here you go. I tried to organize it a bit with some spacing.

https://pastebin.thenetwork.pro/?a116c9 ... 6rcnA5FqDk


I have tried this
/ip firewall mangle
add action=mark-connection chain=input comment="Keep eth1 inbound connections to eth1" connection-mark=no-mark in-interface=ether1 new-connection-mark=ISP1-IN passthrough=yes
add action=mark-routing chain=prerouting connection-mark=ISP1-IN new-routing-mark=ISP1 passthrough=no
add action=mark-connection chain=input comment="Keep eth2 inbound connections to eth2" connection-mark=no-mark in-interface=ether2 new-connection-mark=ISP2-IN passthrough=yes
add action=mark-routing chain=prerouting connection-mark=ISP2-IN new-routing-mark=ISP2 passthrough=no
instead of
/ip firewall mangle
add action=mark-connection chain=input comment="Keep eth1 inbound connections to eth1" connection-mark=no-mark in-interface=ether1 new-connection-mark=ISP1-IN passthrough=yes
add action=mark-routing chain=output connection-mark=ISP1-IN new-routing-mark=ISP1 passthrough=no
add action=mark-connection chain=input comment="Keep eth2 inbound connections to eth2" connection-mark=no-mark in-interface=ether2 new-connection-mark=ISP2-IN passthrough=yes
add action=mark-routing chain=output connection-mark=ISP2-IN new-routing-mark=ISP2 passthrough=no
but everytime I do, all external connections drop to the router. Can't ping, can't ssh, can't vpn, winbox etc.
Not sure why
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Fri Jul 19, 2024 4:37 pm

/export file=anynameyouwish ( minus router serial number, any public WANIP info, keys etc.)
Any thoughts?
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Fri Jul 19, 2024 11:42 pm

Tested the fix... didn't work for me.
/ip firewall nat
add action=dstnat chain=dst-nat in-interface=ether2 dst-address-type=local dst-port=AllWGPorts protocol=udp to-address=ISPGateway1
add action=dstnat chain=dst-nat in-interface=ether1 dst-address-type=local dst-port=AllWGPorts protocol=udp to-address=ISPGateway2
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Sat Jul 20, 2024 12:20 am

I can confirm that this is a bug reproducible on multiple installations using ECMP. And the workaround is the dstnat rule as posted by @anav.
Any chance you'd be willing to share the rules that you used? I can't seem to make it work with my understanding.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Inconsistent WG connection via 2 ISPs

Sat Jul 20, 2024 1:47 am

Why did you completely disregard the advice provided??

No this only affects the WAN that is second in natural priority ( the failover wan so to speak).
You still need some mangling going on, and so I would have to see the config to comment further


To be fair, I didnt look at your whole config but will look tonight.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Inconsistent WG connection via 2 ISPs

Sat Jul 20, 2024 2:02 am

(1) Why the four or five wireguard interfaces. I like simple and clean. Unless there is a reason to have four or five you only need one interface!

You can actually define and use multiple IP subnets to a single wireguard interface.
The only reason you would need multiple INTERFACES is if there was any router traffic that needed to go out the internet on the remote end.
Then you would have to use 0.0.0.0/0 on the allowed peer settings, and thus not possible to have more than one peer effectively.

(2) What a mess, you have bridge but dont use bridge vlan filtering............................... as I stated simplify.
Nothing to be gained by the over complex structure.

(3) For example why do you have two management subnets..........makes no sense
/ip pool
add name=dhcp_pool_mavico ranges=10.1.10.50-10.1.10.254
]add name=dhcp_pool_administradores ranges=10.1.60.2-10.1.60.254
add name=dhcp_pool_management ranges=10.1.50.10-10.1.50.253
add name=dhcp_pool_alquiler1 ranges=10.1.1.2-10.1.1.254
add name=dhcp_pool_VPN ranges=192.168.89.10-192.168.89.254
add name=dhcp_pool_servidores ranges=10.0.0.2-10.0.0.254
add name=dhcp_pool_alquiler2 ranges=10.1.2.2-10.1.2.254
add name=dhcp_pool_alquiler3 ranges=10.1.3.2-10.1.3.254
add name=dhcp_pool_invitados ranges=10.1.100.2-10.1.100.254
add name=dhcp_pool_wifi_mavico ranges=10.1.80.2-10.1.80.254
add name=dhcp_pool_ILO ranges=10.0.1.2-10.0.1.6
add name=dhcp_pool_can&t ranges=10.1.15.2-10.1.15.18
add name=dhcp_pool_management_network ranges=10.0.2.2-10.0.2.254
add name=dhcp_pool_MINER ranges=10.0.250.2-10.0.250.4
add name=dhcp_pool_productora ranges=10.1.150.25-10.1.150.254


(4) Rp filter strict is a no no especially in multiwan scenario, should be set to LOOSE

++++++++++++++++++++++++++++++++++++++++++++
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Sat Jul 20, 2024 6:56 pm

(1) Why the four or five wireguard interfaces. I like simple and clean. Unless there is a reason to have four or five you only need one interface!

You can actually define and use multiple IP subnets to a single wireguard interface.
The only reason you would need multiple INTERFACES is if there was any router traffic that needed to go out the internet on the remote end.
Then you would have to use 0.0.0.0/0 on the allowed peer settings, and thus not possible to have more than one peer effectively.

(2) What a mess, you have bridge but dont use bridge vlan filtering............................... as I stated simplify.
Nothing to be gained by the over complex structure.

(3) For example why do you have two management subnets..........makes no sense
(4) Rp filter strict is a no no especially in multiwan scenario, should be set to LOOSE

++++++++++++++++++++++++++++++++++++++++++++
This is a client router and I'm coming in after its been deployed and their last network guy left.
I agree the vlans need to be cleaned up, its a mess along with a bunch of other things.
I'd like the is thread to focus on the WG issues with multi wan.

After our last discussion, I did find and change the RP filter to loose.

The client has had multiple WG interfaces since before I took them on.
wireguard1 - has both roadwarrior and a site to site tunnels. Meant to come in on WAN1
wirenetlife - has both roadwarrior and a site to site tunnels. Meant to come in on WAN2
wiregrelive - has only roadwarrior tunnels, meant to come in on WAN1.
wireguard2-test - was set up as a brand new test for me to see if the problem stemmed from prior setup. This will be deleted

While I'm confident I can merge wiregrelive and wireguard1 together, that doesn't solve the core issues I've reported in this thread.
There are bandwidth and latency reasons why certain connections need to come in WAN1 and not WAN2, and vice versa. Once I have this issues sorted, WG ports will only be allowed in via their respective WAN interfaces.

You asked for the config, I've provided it.
I listed the mangles above, and in the config. I'll list them again in case you missed them.
/ip firewall mangle
add action=mark-connection chain=input comment="Keep eth1 inbound connections to eth1" connection-mark=no-mark in-interface=ether1 new-connection-mark=ISP1-IN passthrough=yes
add action=mark-routing chain=output comment="Keep eth1 connections to eth1 - mark route" connection-mark=ISP1-IN new-routing-mark=ISP1 passthrough=no
add action=mark-connection chain=input comment="Keep eth2 inbound connections to eth2" connection-mark=no-mark in-interface=ether2 new-connection-mark=ISP2-IN passthrough=yes
add action=mark-routing chain=output comment="Keep eth2 connections to eth2 - mark route" connection-mark=ISP2-IN new-routing-mark=ISP2 passthrough=no
I've added these DSTnat rules per your recommendation.
/ip firewall nat
add action=dst-nat chain=dstnat comment="WG Fix for Eth2" dst-address-type=local dst-port=23232-23239 in-interface=ether2 protocol=udp to-addresses=WAN1IP
add action=dst-nat chain=dstnat comment="WG Fix for Eth1" dst-address-type=local dst-port=23232-23239 in-interface=ether1 protocol=udp to-addresses=WAN2IP
During my tests, from my home ISP, I can handshake with all WG instances via WAN1 and WAN2, however, from my home ISP, I cannot route any actual traffic. (like checking ipinfo.io)
However, if I test using my cell phone with the exact same WG peer profiles, WAN2 instances work, but, and only wiregrelive works via ISP1.

It's completely inconsistent.

My only guess is that there's some kind of hashing algo related to ECMP that's happening with each WG instance in RouterOS, and once that has has been made, it stay resident in memory until the unit is rebooted. If we reboot the device, WG could work fine via WAN1, or only work via WAN2.
Last edited by CTSsean on Sat Jul 20, 2024 7:10 pm, edited 2 times in total.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Inconsistent WG connection via 2 ISPs

Sat Jul 20, 2024 7:04 pm

Understand the requirements better all makes sense, concur the two wireguard WAN1s could be combined, just use same interface two different subnets for clarity. The more the requirements are understood the better a config can be developed.

From experience issues stem from conflicting parts of a config so looking at them in isolation is not always fruitful.
In this case its too messy to ignore. In fact your mangles ( look fine to me but may be incomplete based on understanding the whole config) and other things may not be the problem but the complex mess is creating the problem.
Its far easier NOT to try and look for the needle in the haystack, but to remove the hay, break out the metal detector and find the needle on the floor!
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Sat Jul 20, 2024 7:12 pm

Understand the requirements better all makes sense, concur the two wireguard WAN1s could be combined, just use same interface two different subnets for clarity. The more the requirements are understood the better a config can be developed.

From experience issues stem from conflicting parts of a config so looking at them in isolation is not always fruitful.
In this case its too messy to ignore. In fact your mangles ( look fine to me but may be incomplete based on understanding the whole config) and other things may not be the problem but the complex mess is creating the problem.
Its far easier NOT to try and look for the needle in the haystack, but to remove the hay, break out the metal detector and find the needle on the floor!
Makes a lot of sense. Any suggestions on what the next step is? My brain cells are burned out on this one.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Inconsistent WG connection via 2 ISPs

Sat Jul 20, 2024 8:11 pm

What means ( translation and purpose of list)
BLUE
DROP ILO
BLOQUEOS
NOBLOQUEARSCAN
RESTRICCION

and why is not
vlan80,vlan90,vlan100,vlan110,vlan250 not part of LANlist?

also
why is vlan250 not part of any list???

why does this address exist??
add address=10.1.255.255 interface=Loopback network=10.1.255.255

What is the purpose of this address ( not affiliated to any vlan ?? )
add address=10.0.1.1/29 comment="RED ILO -- SERVER" interface=ether5 network=10.0.1.0


Assuming its a public static IP just not showing the numbers?
add address=WAN_IP_1/24 comment="ISP1 IP" interface=ether1 network=WAN_IP_1_NETWORK

You have 14 vlans, 14 dhcp-servers, WHY 15 pools ??? and 15 dhcp-server networks??


I see multiple VPN networks.
from the 'you should get rid of ppp, insecure protocol, ovpn, l2TP, wireguard.
which can we delete.......... ( is wireguard enough ? )
I see sstp also enabled is it used between two MT devices??


Who is using ssh access to the router??
Who is using ssl access to the router??

+++++++++++++++++++++
Rationalizing a cleaner leaner approach :-)
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Sat Jul 20, 2024 8:49 pm

What means ( translation and purpose of list)
BLUE
DROP ILO
BLOQUEOS
NOBLOQUEARSCAN
RESTRICCION

and why is not
vlan80,vlan90,vlan100,vlan110,vlan250 not part of LANlist?

also
why is vlan250 not part of any list???

why does this address exist??
add address=10.1.255.255 interface=Loopback network=10.1.255.255

What is the purpose of this address ( not affiliated to any vlan ?? )
add address=10.0.1.1/29 comment="RED ILO -- SERVER" interface=ether5 network=10.0.1.0


Assuming its a public static IP just not showing the numbers?
add address=WAN_IP_1/24 comment="ISP1 IP" interface=ether1 network=WAN_IP_1_NETWORK

You have 14 vlans, 14 dhcp-servers, WHY 15 pools ??? and 15 dhcp-server networks??


I see multiple VPN networks.
from the 'you should get rid of ppp, insecure protocol, ovpn, l2TP, wireguard.
which can we delete.......... ( is wireguard enough ? )
I see sstp also enabled is it used between two MT devices??


Who is using ssh access to the router??
Who is using ssl access to the router??

+++++++++++++++++++++
Rationalizing a cleaner leaner approach :-)
This client is in Brazil, so I can imagine most of the wording is Spanish/Portuguese. I'm unsure of the naming convention. All before me.

BLOQUEOS = BLOCKS
NOBLOQUEARSCAN = UNBLOCKSCAN
RESTRICCION = RESTRICTION
I do not know the purpose of the vlans yet. That's to be determined when I start the vlan cleanup process.
------------------------
I do not know why the different vlans are not in the LANlist. The client does not know either.

DHCP Pools, there's a pool for OpenVPN. Not sure why there's 15 DHCP networks. Probably left over from a previous config.
I'm using OpenVPN as an emergency connection to the router via WAN2 in case something breaks.

There's lots of everything, VPN networks, vlans, filters, you name it. This will be cleaned up over the next 3 months.
I need to focus on one problem at a time. The wireguard specific problem.

Would you be willing to review the mangle, the recommended NAT rules, and the routes?
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Inconsistent WG connection via 2 ISPs

Sat Jul 20, 2024 9:07 pm

First I have to organize the config so it makes sense.
One cannot detect errors that are not obvious with such a convoluted mess.
I am making slow progress.
Okay so will have to use firewall rules to discern purpose of lists.

To preserve wireguard is probably best leave as is, so that why client settings do not change..........
Just need to add few connections for you,

------------ at the end ensure you document the config sufficiently such that your replacement isnt so screwed over ;-)
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Inconsistent WG connection via 2 ISPs

Sun Jul 21, 2024 12:09 am

Trying to sort out the WAN situation...

There appears to be an address for ISP1
add address=WAN_IP_1/24 comment="ISP1 IP" interface=ether1 network=WAN_IP_1_NETWORK
But nothing for ISP2.

When looking at routes, there are no numbers for ISP1 gateway which leads me to believe its DHCP or PPOE dynamic so an entry in IP address for WAN1 doesnt make sense.
Then in routes the gateway for ISP2 is a private IP address 192.168.2.1 which tends one to think that one is getting a static private IP for WAN2,
BUT................... ISP2 is set up under DHCP CLIENT and there is no expected entry in IP address for wan2 ????

WAN1
/ip address
add address=WAN_IP_1/24 comment="ISP1 IP" interface=ether1 network=WAN_IP_1_NETWORK
add check-gateway=ping comment="ISP1 default route" disabled=no distance=10 dst-address=0.0.0.0/0 gateway=WAN1_GATEWAY pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10



WAN2
set [ find default-name=ether2 ] comment="ISP2 Netlife"
/ip dhcp-client
add add-default-route=no interface=ether2 use-peer-dns=no
add disabled=no distance=10 dst-address=0.0.0.0/0 gateway=192.168.2.1 pref-src="" routing-table=ISP2 scope=30 suppress-hw-offload=no target-scope=10
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Inconsistent WG connection via 2 ISPs

Sun Jul 21, 2024 12:10 am

Need to know facts
WAN1 - static or dynamic IP, public or private IP
WAN2 - static or dynamic IP, public or private IP
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Sun Jul 21, 2024 12:57 am

Need to know facts
WAN1 - static or dynamic IP, public or private IP
WAN2 - static or dynamic IP, public or private IP
WAN1 - static pub
WAN2 - static Pub, behind NAT without the option of bridging the gateway.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Inconsistent WG connection via 2 ISPs

Sun Jul 21, 2024 1:51 am

Okay have to assume that the WAN2 works for wireguard and servers because the Upstream router providing a private IP for wan can forward ports to the LANIP of the MT.
Just in case I kick the bucket tomorrow this encapsulates all I can do with the provided info.
If you want to buy me a ticket to Brazil I can do some onsite work LOL........
.........
interface bridge
add bridge=bridge  vlan-filtering=no  { Change to yes at very end of config }

/interface ethernet
set [ find default-name=ether1 ] comment="ISP1 Movistar" 
set [ find default-name=ether2 ] comment="ISP2 Netlife" 
set [ find default-name=ether3 ] comment="PUERTO TRONCAL MK SWPAT"
set [ find default-name=ether4 ] comment="ENLACE MK - SWPA2"
set [ find default-name=ether5 ] comment="ILO -- AISLADO"  <--- UNKNOWN purpose
set [ find default-name=ether6 ] comment=ESXi
set [ find default-name=ether7 ] comment=ESXi
set [ find default-name=ether8 ] comment="PUERTO TRONCAL MK SWPA1"
set [ find default-name=sfp-sfpplus1 ] disabled=yes

/interface wireguard
add listen-port=23231 mtu=1420 name=wireguard1
add listen-port=23232 mtu=1420 name=wiregrelive
add listen-port=23239 mtu=1420 name=wirenetlife

/interface vlan
add interface=bridge name=vlan10-admin-mavico  vlan-id=10
add interface=bridge name=vlan20-Aquiler1  vlan-id=20
add interface=bridge name=vlan30-Aquiler2  vlan-id=30
add interface=bridge name=vlan40-Aquiler3 vlan-id=40
add interface=bridge name=vlan50-Gestion vlan-id=50
add interface=bridge name=vlan60-ADMINSTRA vlan-id=60
add interface=bridge name=vlan70-SERVI  vlan-id=70
add interface=bridge name=vlan80-WIFI-mavico vlan-id=80
add interface=bridge name=vlan90-CAN&T vlan-id=90
add interface=bridge name=vlan100-WIFI-guest vlan-id=100
add interface=bridge name=vlan110-MANAGE vlan-id=110
add interface=bridge name=vlan150-PRODUCT vlan-id=150
add interface=bridge name=vlan200-CARDANO vlan-id=200
add interface=bridge name=vlan250-MINER vlan-id=250

/interface list
add name=WAN
add name=LAN
add name=BASE
add name=DNS-Force
add name=Internet
add name=VPN

/ip dhcp-server option
add code=43 name=microsoft-disable-netbios-option value=0x010400000002

/ip pool  
add name=dhcp_pool_mavico ranges=10.1.10.50-10.1.10.254
add name=dhcp_pool_alquiler1 ranges=10.1.1.2-10.1.1.254
add name=dhcp_pool_alquiler2 ranges=10.1.2.2-10.1.2.254
add name=dhcp_pool_alquiler3 ranges=10.1.3.2-10.1.3.254
add name=dhcp_pool_management ranges=10.1.50.10-10.1.50.253
add name=dhcp_pool_administradores ranges=10.1.60.2-10.1.60.254
add name=dhcp_pool_servidores ranges=10.0.0.2-10.0.0.254
add name=dhcp_pool_wifi_mavico ranges=10.1.80.2-10.1.80.254
add name=dhcp_pool_can&t ranges=10.1.15.2-10.1.15.18
add name=dhcp_pool_invitados ranges=10.1.100.2-10.1.100.254
add name=dhcp_pool_management_network ranges=10.0.2.2-10.0.2.254
add name=dhcp_pool_productora ranges=10.1.150.25-10.1.150.254
add name=dhcp_pool_MINER ranges=10.0.250.2-10.0.250.4

{ unknown pools }
add name=dhcp_pool_VPN ranges=192.168.89.10-192.168.89.254
add name=dhcp_pool_ILO ranges=10.0.1.2-10.0.1.6

/ip dhcp-server
add address-pool=dhcp_pool_mavico interface=vlan10-admin-mavico lease-time=1h name="DHCP RED mavico"
add address-pool=dhcp_pool_alquiler1  interface=vlan20-Aquiler1 lease-time=1h name="DHCP RED ALQUILER 1"
add address-pool=dhcp_pool_alquiler2 interface=vlan30-Aquiler2  lease-time=1h name="DHCP RED ALQUILER 2"
add address-pool=dhcp_pool_alquiler3 interface=vlan40-Aquiler3 lease-time=1h name="DHCP RED ALQUILER 3"
add address-pool=dhcp_pool_management interface=vlan50-Gestion  lease-time=1h name="DHCP RED GESTION"
add address-pool=dhcp_pool_administradores interface=vlan60-ADMINSTRA lease-time=1h name="DHCP RED ADMINISTRADORES"
add address-pool=dhcp_pool_servidores interface=vlan70-SERVI   lease-time=1h name="DHCP RED SERVIDORES"
add address-pool=dhcp_pool_wifi_mavico interface=vlan80-WIFI-mavico  lease-time=1h name="DHCP WIFI mavico VLAN 80"
add address-pool=dhcp_pool_can&t interface=vlan90-CAN&T lease-time=1h name="DHCP RED CAN&T"
add address-pool=dhcp_pool_invitados interface=vlan100-WIFI-guest lease-time=1h name="DHCP WIFI INVITADOS"
add address-pool=dhcp_pool_management_network interface=vlan110-MANAGE lease-time=10m name="DHCP RED MANAGEMENT NETWORK"
add address-pool=dhcp_pool_productora interface=vlan150-PRODUCT lease-time=10m name="DHCP PRODUCTORA"
add address-pool=dhcp_pool_MINER interface=vlan250-MINER lease-time=10m name="DHCP RED MINER"

{ note vlan200 does not have a pool or or dhcp-server }

/ppp profile
add change-tcp-mss=yes interface-list=LANList local-address=192.168.89.1 name=DesarrolloFab remote-address=dhcp_pool_VPN use-encryption=yes
set *FFFFFFFE interface-list=LANList local-address=192.168.89.1 remote-address=dhcp_pool_VPN

/routing table
add disabled=no fib name=to-ISP2
add disabled=no fib name=to-ISP1

/ip neighbor discovery-settings
set discover-interface-list=BASE

/ip settings
set max-neighbor-entries=8192 rp-filter=loose tcp-syncookies=yes

/ipv6 settings
set max-neighbor-entries=8192

/interface l2tp-server server
set allow-fast-path=yes enabled=yes use-ipsec=yes

/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=WAN
add interface=vlan10-admin-mavico  list=LAN
add interface=vlan20-Aquiler1 list=LAN
add interface=vlan30-Aquiler2 list=LAN
add interface=vlan40-Aquiler3 list=LAN
add interface=vlan50-Gestion list=LAN
add interface=vlan60-ADMINSTRA list=LAN
add interface=vlan70-SERVI list=LAN
add interface=vlan80-WIFI-mavicot list=LAN
add interface=vlan90-CAN&T list=LAN
add interface=vlan100-WIFI-guest list=LAN
add interface=vlan110-NETMANAGE list=LAN
add interface=vlan150-PRODUCT list=LAN
add interface=vlan200-CARDANO list=LAN
add interface=wireguard1 list=LAN
add interface=wiregrelive list=LAN
add interface=wirenetlife list=LAN

add interface=vlan110-NETMANAGE list=BASE
add interface=wireguard1 list=BASE
add interface=wiregrelive list=BASE
add interface=wirenetlife list=BASE

add interface=vlan10-admin-mavico list=DNS-Force
add interface=vlan20-Aquiler1 list=DNS-Force
add interface=vlan30-Aquiler2 list=DNS-Force
add interface=vlan40-Aquiler3 list=DNS-Force
add interface=vlan80-WIFI-mavicot=DNS-Force
add interface=vlan90-CAN&T list=DNS-Force
add interface=vlan100-WIFI-guest list=DNS-Force
add interface=vlan200-CARDANO list=DNS-Force

add interface=vlan10-admin-mavico  list=Internet
add interface=vlan20-Aquiler1 list=Internet
add interface=vlan30-Aquiler2 list=Internet
add interface=vlan40-Aquiler3 list=Internet
add interface=vlan50-Gestion list=Internet
add interface=vlan60-ADMINSTRA list=Internet
add interface=vlan80-WIFI-mavicot=Internet
add interface=vlan90-CAN&T list=Internet
add interface=vlan100-WIFI-guest list=Internet
add interface=vlan110-NETMANAGE list=Internet
add interface=vlan150-PRODUCT list=Internet
add interface=vlan200-CARDANO list=Internet

add interface=vlan200-CARDANO list=VPN
add interface=vlan250-MINER list=VPN

PURPOSE NOT CLEAR AND THUS NO LIST AFFILIATION  ----> ETHER5.

/interface ovpn-server server
set auth=sha1 certificate=server@RB-mavico cipher=aes256-gcm default-profile=default-encryption enabled=yes protocol=udp

/interface sstp-server server
set default-profile=default-encryption enabled=yes

/interface bridge port  { only ports I could find identified }
add bridge=bridge ingress-filtering=yes frame-types=admit-only-vlan-tagged interface=ether3
add bridge=bridge ingress-filtering=yes frame-types=admit-only-vlan-tagged interface=ether6
add bridge=bridge ingress-filtering=yes frame-types=admit-only-vlan-tagged interface=ether7
add bridge=bridge ingress-filtering=yes frame-types=admit-only-vlan-tagged interface=ether8

/interface bridge vlan
add bridge=bridge tagged=bridge,ether3,ether8 vlan-ids=10
add bridge=bridge tagged=bridge,ether3,ether6,ether7,ether8 vlan-ids=50
add bridge=bridge tagged=bridge,ether6,ether7,ether8 vlan-ids=70
add bridge=bridge tagged=bridge,ether3  vlan-ids=90
add bridge=bridge tagged=bridge,ether6,ether7 vlan-ids=110,200
add bridge=bridge tagged=bridge,ether8 vlan-ids=20,30,40,60,80,100,150,250

/interface wireguard peers
add allowed-address=192.168.20.2/32 interface=wireguard1  name=Alf  public-key="VxyTPMB5qCwj9yER1PRnRxItNp4ES+dWRNtQropIgDA="
add allowed-address=10.0.201.2/32 interface=wiregrelive  name="GRELIVE WG" public-key="r681usDuWxOJIVObo91UB/QLTX8+tYk4dFf4FIAjq3I="
add allowed-address=192.168.20.5/32 interface=wireguard1  name="Alex Movistar"  public-key="Dlk+Z6H1/z5PAkTnwPDtdFedg5iLIOVc+XfCltp021I="
add allowed-address=192.168.20.3/32 interface=wireguard1  name="Router VE Movistar"  public-key="sh2GvC4MdMwgx82dJQ3GBHbPB9iDrv5ktvMWAweuojo="
add allowed-address=10.0.201.100/32 interface=wiregrelive  name="GRELIVE WG Muhammad"  public-key="BmFa/wrFvTbc98sUz9NejBTYwssSND9+VwF7kG1Y4jQ="
add allowed-address=10.0.201.200/32 interface=wiregrelive  name="GRELIVE WG Clyde Nodes"  public-key="D6nFNxcau5CwLIee9+tXMvxhABQ2674id4trj2ryOxQ="
add allowed-address=192.168.30.246/32 interface=wirenetlife  name="Alf Netlife Phone"  public-key="aBWTZbeBte+83WkxfT8O+Ge6gJJMtgr5Fpos67mEakE="
add allowed-address=192.168.30.3/32 interface=wirenetlife i name="Router VE Netlife"  public-key="xBozfwwpg8m+++dwnphR/NZCDkmWwtWUNAybI3XHwxc="
add allowed-address=192.168.20.247/32  name=TNP-Sean1-wireguard1-ISP1  public-key="pIbPjd4FezJk+WO99Evg76UzBMLL3tUosqnsMPrROAo="
add allowed-address=192.168.30.5/32 interface=wirenetlife name="Alex Netlife"  public-key="Dlk+Z6H1/z5PAkTnwPDtdFedg5iLIOVc+XfCltp021I="
add allowed-address=192.168.20.7/32 interface=wireguard1 name=jdcassin public-key="k1DHuMZ44bks2hD+SLkwmIEVVz20BxPdsvKTxYl6IDM="
add allowed-address=192.168.30.7/32 interface=wirenetlife name=jdcassinNetlife public-key="PZgDGJtdhnD7KqMAkkvvlDNXRaJ45nGfWwrSx3GRNEU="
++++++++++++++
add allowed address=192.168.20.247/32 interface=wireguard1 comment="sean ISP1 remote laptop"   public-key="xxxxxxxx"
add allowed address=192.168.20.6/32 interface=wireguard1 comment="sean ISP1 from work or home or smartphone?"   public-key="aaaaaaaaa"
add allowed-address=10.0.201.247/32 interface=wiregrelive comment="sean ISP1 remote laptop"  public-key="AIjNIMUe5ZqBzw9bcoCVZb6iRtpSufgA3WLqg4AqCg8="
add allowed address=10.0.201.6/32 interface=wiregrelive comment="sean ISP1 from work or home or smartphone?"   public-key="zzzzzzz"
add allowed-address=192.168.30.249/32 interface=wirenetlife comment="sean ISP2 remote laptop" public-key="b7BPD4C24pkRqlX4/vcvWYHXyEfoxAh1XdVo7wauq2U="
add allowed address=192.168.30.6/32 interface=wirenetlifee comment="sean ISP2 from work or home or smartphone?"   public-key="yyyyyyy"

/ip address
add address=10.1.10.1/24 comment="RED mavico (VID=10)" interface=vlan10-admin-mavico   network=10.1.10.0
add address=10.1.1.1/24 comment="RED ALQUILER 1 (VID=20)" interface=vlan20-Aquiler1 network=10.1.1.0
add address=10.1.2.1/24 comment="RED ALQUILER 2 (VID=30)" interface=vlan30-Aquiler2 network=10.1.2.0
add address=10.1.3.1/25 comment="RED ALQUILER 3 (VID=40)" interface=vlan40-Aquiler3 network=10.1.3.0
add address=10.1.50.1/25 comment="RED GESTION (VID=50)" interface=vlan50-Gestion network=10.1.50.0
add address=10.1.60.1/24 comment="RED ADMINISTRADORES (VID=60)" interface=vlan60-ADMINSTRA network=10.1.60.0
add address=10.0.0.1/24 comment="RED SERVIDORES VLAN 70" interface=vlan70-SERVI network=10.0.0.0
add address=10.1.80.1/24 comment="RED WIFI mavico (VID=80)" interface=vlan80-WIFI-mavico network=10.1.80.0
add address=10.1.15.1/27 comment="RED CANT&T (VID=90)" interface=vlan90-CAN&T network=10.1.15.0
add address=10.1.100.1/24 comment="RED INVITADOS  (VID=100)" interface=vlan100-WIFI-guest  network=10.1.100.0
add address=10.0.2.1/24 comment="RED NETWORK MANAGEMENT" interface=vlan110-NETMANAGE network=10.0.2.0
add address=10.1.150.1/24 comment="RED PRODUCTORA (VID=150)" interface=vlan150-PRODUCT network=10.1.150.0
add address=10.0.200.1/24 comment="RED CARDANO NODOS" interface=vlan200-CARDANO network=10.0.200.0
add address=10.0.250.1/24 comment="RED MINER" interface=vlan250-MINER network=10.0.250.0
+++++++++++
add address=192.168.20.1/24 comment="RED WIREGUARD1" interface=wireguard1 network=192.168.20.0
add address=192.168.30.1/24 comment="RED WIREGUARD1" interface=wirenetlife network=192.168.30.0
add address=10.0.201.1/24 comment=GRELIVE interface=wiregrelive network=10.0.201.0
++++++++++++

add address=WAN_IP_1/24 comment="ISP1 IP" interface=ether1 network=WAN_IP_1_NETWORK
add address=10.0.1.1/29 comment="RED ILO -- SERVER" interface=ether5 network=10.0.1.0  comment="NOT ON BRIDGE" 

/ip cloud
set ddns-enabled=yes update-time=no

/ip dhcp-client
add add-default-route=no interface=ether2 use-peer-dns=no

/ip dhcp-server network
add address=10.1.10.0/24 comment="RED mavico VID=10" dns-server=10.0.0.254 domain=mavico.com gateway=10.1.10.1 wins-server=10.0.0.100
add address=10.1.1.0/24 comment="RED ALQUILER 1 VID=20" dns-server=10.0.0.254 domain=mavico.com gateway=10.1.1.1 wins-server=10.0.0.100
add address=10.1.2.0/24 comment="RED ALQUILER 2 VID=30" dns-server=10.0.0.254 domain=mavico.com gateway=10.1.2.1 wins-server=10.0.0.100
add address=10.1.3.0/24 comment="RED ALQUILER 3 VID=40" dns-server=10.0.0.254 domain=mavico.com gateway=10.1.3.1 wins-server=10.0.0.100
add address=10.1.50.0/24 comment="RED GESTION VID=50" dhcp-option=microsoft-disable-netbios-option dns-server=10.1.50.1 domain=mavico.com gateway=10.1.50.1 wins-server=10.0.0.100
add address=10.1.60.0/24 comment="RED ADMINISTRADORES VID=60" dns-server=10.1.60.1 domain=mavico.com gateway=10.1.60.1 wins-server=10.0.0.100
add address=10.0.0.0/24 comment="RED SERVIDORES VID=70" dns-server=10.0.0.1 gateway=10.0.0.1
add address=10.1.80.0/24 comment="RED WIFI mavico VID=80" dhcp-option=microsoft-disable-netbios-option dns-server=10.0.0.254 domain=mavico.com gateway=10.1.80.1
add address=10.1.15.0/27 comment="RED CANT VID=90" dns-server=10.0.0.254 domain=mavico.com gateway=10.1.15.1 wins-server=10.0.0.100
add address=10.1.100.0/24 comment="RED WIFI INVITADOS VID=100" dhcp-option=microsoft-disable-netbios-option dns-server=10.0.0.254 domain=mavico.com gateway=10.1.100.1 wins-server=10.0.0.100
add address=10.0.2.0/24 comment="RED MANAGEMENT NETWORK VID=110" dns-server=10.0.2.1 domain=mavico.com gateway=10.0.2.1
add address=10.1.150.0/24 comment="RED PRODUCTORA VID=150" dns-server=10.1.150.1 domain=mavico.com gateway=10.1.150.1 wins-server=10.0.0.100
add address=10.0.200.0/24 comment="RED CARDANO NETWORK VID=200" dns-server=10.0.0.254 domain=mavico.com gateway=10.0.200.1
add address=10.0.250.0/24 comment="RED KDA MINER NETWORK VID=250" gateway=10.0.250.1

UKNOWN
add address=10.0.1.0/29 comment="RED ILO" dns-server=10.0.1.1 domain=mavico.com gateway=10.0.1.1

/ip dns
set allow-remote-requests=yes servers=1.1.1.1,8.8.4.4

/ip firewall address-list
add address=10.0.2.0/24 list=Authorized comment="local admin network"
add addres=192.168.20.247  list=Authorized comment="sean isp1 laptop wg-1"
add addres=192.168.20.6  list=Authorized comment="sean isp1  work, home, or smartphone wg-1"
add addres=10.0.201.247 list=Authorized comment="sean isp1 laptop  wg-live"
add addres=10.0.201.6 list=Authorized comment="sean isp1  work, home, or smartphone wg-live"
add addres=192.168.30.247  list=Authorized comment="sean isp2 laptop wg-life"
add addres=192.168.30.6  list=Authorized comment="sean isp2  work, home, or smartphone wg-life"
add address=192.168.89.247  list=Authorized  comment="OpenVPN - SEAN"
+++++
add address=pool.ntp.org list=ntplist
add address=ec.pool.ntp.org list=ntplist
add address=10.1.50.13 list=grelive200-2-50-13
add address=10.0.200.2 list=grelive200-2-50-13
add address=10.0.0.197 list=grelive200-2-50-13
add address=10.0.0.240 list=grelive200-2-50-13
add address=10.0.0.199 list=grelive200-2-50-13
add address=10.0.0.198 list=grelive200-2-50-13
add address=10.0.200.3 list=grelive200-2-50-13
add address=10.0.0.200 list=grelive200-2-50-13
add address=10.0.201.2 list=VPNs
add address=192.168.20.2 list=VPNs
add address=192.168.20.3 list=VPNs
add address=192.168.30.2 list=VPNs
add address=192.168.30.3 list=VPNs
add address=192.168.89.254 list=VPNs
add address=107.175.134.123 list=grafana-celestia-vps-gala

/ip firewall filter
#Input Chain
add action=accept chain=input connection-state=established,related,untracked
add action=drop chain=input connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
++++++++++++
add action=accept chain=input comment="allow wireguard1 and wiregrelive" connection-state=new dst-address=WAN_IP_1 dst-port=23231-23232 log=yes log-prefix=wireguard1-wiregreliveLOG protocol=udp
add action=accept chain=input comment="allow wirenetlife" connection-state=new dst-port=23239 in-interface=ether2 log=yes log-prefix=wirenetlifeLOG protocol=udp
add action=accept chain=input comment="allow ovpn" dst-port=1194 protocol=udp
add action=accept chain=input comment="allow ovpn" dst-port=1194 protocol=tcp
add action=accept chain=input comment="allow IPsec NAT" dst-port=4500 protocol=udp
add action=accept chain=input comment="allow IKE" dst-port=500 protocol=udp
add action=accept chain=input comment="allow l2tp" dst-port=1701 protocol=udp
add action=accept chain=input comment="Admin Access"  in-interface-list=BASE src-address-list=Authorized
add action=accept chain=input comment="users to services"  in-interface-list=LAN dst-port=53,123 protocol=udp
add action=accept chain=input comment="users to services"  in-interface-list=LAN dst-port=53 protocol=tcp
add action=drop chain=input comment="drop all else"

#Forward Chain
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward connection-state=established,related connection-mark=no-mark
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward connetion-state=invalid
add action=accept chain=forward comment="internet traffic"  in-interface-list=Internet   out-interface-list=WAN
add action=accept chain=forward comment="allow to DNS server"  in-interface-list=DNS-Force dst-address=10.0.0.254
add action=accept chain=forward comment="admin access"  in-interface-list=BASE src-address-list=Authorized
add action=accept chain=forward comment="wg-life to all subnets except network management"  in-interface=list=wirenetlife out-interface-list=LAN dst-address=!10.0.2.0/24  log=yes log-prefix="LogWGwirenetlife"
add action=accept chain=forward comment="Gestion vlan50  to Adminstr vlan60 "  src-address=10.1.50.0/24  dst-address=10.1.60/0/24
add action=accept chain=forward comment="Deniega peticiones al Vicidial" src-address=10.1.10.0/24 dst-address=10.0.0.197 dst-port=3306,5038 protocol=tcp log=yes log-prefix=TRAIDORMYSQL
add action=accept chain=forward comment="Servers to NTP" src-address=10.0.0.0/24 out-interface-list=WAN dst-address-lst=ntplist
add action=accept chain=forward comment="Vlan200/250 only local access"  in-interface-list=VPN dst-address-list=VPNs
add action=accept chain=forward comment="Allow wg-live to list"  in-interface=wiregrelive dst-address=list=grelive200-2-50-13
add action=accept chain=forward comment="port forwarding"  connection-state=dstnat
add action=drop chain=forward comment="drop all else"

UNUSED RULES

add action=accept chain=forward dst-address=10.1.50.0/24 protocol=tcp src-address=10.1.10.95   PURPOSE NOT CLEAR -- why does one user in vlan10 have access to the entire Gestion subnet??
add action=accept chain=forward connection-limit=100,32 in-interface="BRIDGE VLAN 60" src-address-list=10.1.10.169   PURPOSE NOT CLEAR traffic to where why limit as well???

/ip firewall mangle
{ Ensure incoming traffic to the router, goes out same ISP }
add action=mark-connection chain=input comment="Keep eth1 inbound connections to eth1" connection-mark=no-mark in-interface=ether1 new-connection-mark=ISP1-IN passthrough=yes
add action=mark-routing chain=output connection-mark=ISP1-IN new-routing-mark=to-ISP1 passthrough=no
add action=mark-connection chain=input comment="Keep eth2 inbound connections to eth2" connection-mark=no-mark in-interface=ether2 new-connection-mark=ISP2-IN passthrough=yes
add action=mark-routing chain=output connection-mark=ISP2-IN new-routing-mark=to-ISP2 passthrough=no
{ Ensure any port forwardings coming in wanX, go back on wanX }
add action=mark-connection chain=forward comment="Incoming for Servers WAN1" connection-mark=no-mark in-interface=ether1 new-connection-mark=incoming-WAN1 passthrough=yes
add action=mark-routing chain=prerouting in-interface=vlan70-SERVI connection-mark=incoming-WAN1 new-routing-mark=to-ISP1 passthrough=no
add action=mark-connection chain=forward comment="Incoming for Servers WAN2" connection-mark=no-mark in-interface=ether2 new-connection-mark=incoming-WAN2 passthrough=yes
add action=mark-routing chain=prerouting in-interface=vlan70-SERVI connection-mark=incoming-WAN2 new-routing-mark=to-ISP2 passthrough=no

/ip firewall nat
add action=masquerade chain=srcnat comment="Main Masqerade" out-interface-list=WAN
add action=dst-nat chain=dstnat comment="Use Local DNS Server" in-interface-list=DNS-Force dst-port=53 protocol=ucp to-ports=10.0.0.254
add action=dst-nat chain=dstnat comment="Use Local DNS Server" in-interface-list=DNS-Force dst-port=53 protocol=tcp to-ports=10.0.0.254
add action=dst-nat chain=dstsnat comment="WG-Wirenetlife fix ISP2" in-interface=ether2 dst-address-type=local dst-port=23239 protocol=udp to-address=current-wan1-ip-address
{ will need to make a script  to keep this updated everytime IP changes }
+++++++++
add action=dst-nat chain=dstnat comment="DST-NAT SSH ViciDial" dst-address=192.168.2.2 dst-port=2242 in-interface=ether2 log-prefix=ViciDialSSH protocol=tcp to-addresses=10.0.0.197 to-ports=22
add action=dst-nat chain=dstnat comment="DST-NAT ClaroConsultaISP1Movi" disabled=yes dst-port=5500 in-interface=ether1 log=yes log-prefix=ClaroConsultaMovi protocol=tcp to-addresses=10.1.50.13 
add action=dst-nat chain=dstnat comment="DST-NAT WEB443 GreLive-STG" dst-address=192.168.2.2 dst-port=443 in-interface=ether2 log-prefix=GreLive-STG protocol=tcp to-addresses=10.0.200.2 
add action=dst-nat chain=dstnat comment="DST-NAT BeraChain-Node Ubuntu Movistar (P2P)" dst-port=26646 in-interface=ether1 protocol=tcp to-addresses=10.0.0.199
add action=dst-nat chain=dstnat comment="DST-NAT BeraChain-Node Ubuntu Netlife (P2P)" dst-address=192.168.2.2 dst-port=26646 in-interface=ether2 protocol=tcp to-addresses=10.0.0.199
add action=dst-nat chain=dstnat comment="DST-NAT BeraChain-Node Ubuntu Movistar (P2P)" dst-port=5000 in-interface=ether1 protocol=tcp to-addresses=10.0.0.199 
add action=dst-nat chain=dstnat comment="DST-NAT BeraChain-Node Ubuntu Netlife (P2P)" dst-address=192.168.2.2 dst-port=5000 in-interface=ether2 protocol=tcp to-addresses=10.0.0.199
add action=dst-nat chain=dstnat comment="DST-NAT Celestia-Node Ubuntu Movistar (P2P)" dst-port=26656 in-interface=ether1 protocol=tcp to-addresses=10.0.0.240 
add action=dst-nat chain=dstnat comment="DST-NAT Celestia-Node Ubuntu Movistar GRAFANA (P2P)" dst-port=9100 in-interface=ether1 protocol=tcp src-address-list=grafana-celestia-vps-gala to-addresses=10.0.0.240 
add action=dst-nat chain=dstnat comment="DST-NAT Celestia-Node Ubuntu Movistar" dst-port=1317 in-interface=ether1 protocol=tcp to-addresses=10.0.0.240 
add action=dst-nat chain=dstnat comment="DST-NAT Celestia-Node Ubuntu Movistar" dst-port=2121 in-interface=ether1 protocol=tcp to-addresses=10.0.0.240 
add action=dst-nat chain=dstnat comment="DST-NAT Celestia-Node Ubuntu Movistar" dst-port=2121 in-interface=ether1 protocol=udp to-addresses=10.0.0.240 
add action=dst-nat chain=dstnat comment="DST-NAT Celestia-Node Ubuntu Movistar" dst-port=2022 in-interface=ether1 protocol=tcp to-addresses=10.0.0.240 to-ports=22
add action=dst-nat chain=dstnat comment="DST-NAT Celestia-Node Ubuntu ORCHESTOR Movistar" dst-port=30000 in-interface=ether1 protocol=tcp to-addresses=10.0.0.240 
add action=dst-nat chain=dstnat comment="DST-NAT Celestia-Node Mocha-4 Netlife" dst-address=192.168.2.2 dst-port=26656 in-interface=ether2 protocol=tcp to-addresses=10.0.0.198
add action=dst-nat chain=dstnat comment="DST-NAT BTFS SVRWIN19" dst-port=4001 in-interface=ether1 limit=300,5:packet protocol=tcp to-addresses=10.0.0.253 
add action=dst-nat chain=dstnat comment="DST-NAT BTFS SVRWIN19" dst-port=4001 in-interface=ether1 protocol=udp to-addresses=10.0.0.253 

DSTNAT RULES ADDED dst-address (192.168.2.2)  thus in WRONG FORMAT NOT INCLUDED ABOVE
add action=dst-nat chain=dstnat comment="DST-NAT FullStorageCelestia Netlife" dst-address=192.168.2.2 dst-port=2121,2122,2123,9090 in-interface=ether2 protocol=tcp to-addresses=10.0.0.195
add action=dst-nat chain=dstnat comment="DST-NAT FullStorageCelestia Netlife" dst-address=192.168.2.2 dst-port=2121,2122,2123 in-interface=ether2 protocol=udp to-addresses=10.0.0.195 
add action=dst-nat chain=dstnat comment="DST-NAT FullStorageCelestia Netlife" dst-address=192.168.2.2 dst-port=26658 in-interface=ether2 protocol=tcp to-addresses=10.0.0.195 
add action=dst-nat chain=dstnat comment="DST-NAT FullStorageCelestia Netlife" dst-address=192.168.2.2 dst-port=26659 in-interface=ether2 protocol=tcp to-addresses=10.0.0.195 


UNUSED DSTNAT RULES as WHY ALLOWING port forwarding from COUNTRY BLOCK LIST????? [ disabled rules removed entirely }
add action=dst-nat chain=dstnat comment="DST-NAT 1437 GreLive-STG" dst-address=192.168.2.2 dst-port=1437 in-interface=ether2 log-prefix=GreLive-STG protocol=tcp src-address-list=CountryIPBlocks to-addresses=10.0.200.2 to-ports=1437
add action=dst-nat chain=dstnat comment="DST-NAT SSH SuiteCRM" dst-port=2226 in-interface=ether1 log-prefix=SuiteCRM protocol=tcp src-address-list=CountryIPBlocks to-addresses=10.0.0.238 to-ports=22
add action=dst-nat chain=dstnat comment="DST-NAT SSH DashDepurador" dst-port=2240 in-interface=ether1 log=yes log-prefix=DashDepuradorSSH protocol=tcp src-address-list=CountryIPBlocks to-addresses=10.0.0.200 to-ports=22
add action=dst-nat chain=dstnat comment="DST-NAT WEB8443 ViciDial" dst-address=192.168.2.2 dst-port=8443 in-interface=ether2 log-prefix=ViciDialWEB8443 protocol=tcp src-address-list=CountryIPBlocks to-addresses=10.0.0.197 to-ports=443
add action=dst-nat chain=dstnat comment="DST-NAT WEB8089 ViciDial" dst-address=192.168.2.2 dst-port=8089 in-interface=ether2 log-prefix=ViciDial8090 protocol=tcp src-address-list=CountryIPBlocks to-addresses=10.0.0.197 to-ports=8089
add action=dst-nat chain=dstnat comment="DST-NAT UDP ViciDial" dst-address=192.168.2.2 dst-port=19000-20000 in-interface=ether2 log-prefix=ViciDialUDP protocol=udp src-address-list=CountryIPBlocks to-addresses=10.0.0.197 to-ports=19000-20000
add action=dst-nat chain=dstnat comment="DST-NAT WEB8443 SuiteCRM" disabled=yes dst-port=8443 in-interface=ether1 log-prefix=SuiteCRM protocol=tcp src-address-list=CountryIPBlocks to-addresses=10.0.0.238 to-ports=443
add action=dst-nat chain=dstnat comment="DST-NAT SSH GreLive-STG" dst-address=192.168.2.2 dst-port=2230 in-interface=ether2 log-prefix=GreLive-STG protocol=tcp src-address-list=CountryIPBlocks to-addresses=10.0.200.2 to-ports=22
add action=dst-nat chain=dstnat comment="DST-NAT JENKINS GreLive-STG" dst-address=192.168.2.2 dst-port=9090 in-interface=ether2 log-prefix=GreLive-STG protocol=tcp src-address-list=CountryIPBlocks to-addresses=10.0.200.2 to-ports=9090
add action=dst-nat chain=dstnat comment="DST-NAT PUSHER CHAT GreLive-STG" dst-address=192.168.2.2 dst-port=3001 in-interface=ether2 log-prefix=GreLive-STG protocol=tcp src-address-list=CountryIPBlocks to-addresses=10.0.200.2 to-ports=3001
add action=dst-nat chain=dstnat comment="DST-NAT 3366 GreLive-STG" disabled=yes dst-address=192.168.2.2 dst-port=3366 in-interface=ether2 log=yes log-prefix=GreLive-STG protocol=tcp src-address-list=CountryIPBlocks to-addresses=10.0.200.2 to-ports=3366


/ip route
{ main table routes }
add check-gateway=ping comment=Check-ISP1 distance=2 dst-address=0.0.0.0/0  gateway=8.8.8.8 routing-table=main scope=10 target-scope=12 
add distance=2 dst-address=8.8.8.8/32 gateway=WAN1_GATEWAY routing-table=main scope=10 target-scope=11
add check-gateway=ping comment=Check-ISP2 distance=4 dst-address=0.0.0.0/0  gateway=9.9.9.9 routing-table=main scope=10 target-scope=12 
add distance=4 dst-address=9.9.9.9/32 gateway=WAN2_GATEWAY routing-table=main scope=10 target-scope=11
++++++++
{ special routes }
add dst-address=0.0.0.0/0 gateway=WAN1_GATEWAY  routing-table=to-ISP1
add dst-address=0.0.0.0/0 gateway=WAN2_GATEWAY  routing-table=to-ISP2

/ip service
set telnet disabled=yes
set ftp disabled=yes
set www address=0.0.0.0/0 disabled=yes port=8088
set ssh address="10.1.50.0/24,10.1.60.0/24,10.1.89.0/24,192.168.89.0/24,192.168.10.0/24,192.168.20.0/24,192.168.30.0/24,200.75.100.0/24,68.84.240.117/32"
set www-ssl address=0.0.0.0/0 port=8888
set winbox address=10.0.2.0/24,192.168.20.0/24,10.0.201.0/24,192.168.30.0/24,192.168.89.0/24

/ip ssh
set strong-crypto=yes

/routing rule
add action=lookup-only-in-table main-prefix=0 comment="ensures all local traffic permitted prior to rest of rules"
add action=lookup comment="Mocha-4 Celestia"  src-address=10.0.0.198/32 table=ISP2
add action=lookup comment=Depurador  src-address=10.0.0.200/32 table=ISP2
add action=lookup comment="ViciDial Netlife"  src-address=10.0.0.197/32 table=ISP2
add action=lookup comment="Artio Berachain Node"  src-address=10.0.0.199/32 table=ISP2
add action=lookup comment="FullStorage Celestia"  src-address=10.0.0.195/32 table=ISP2
add action=lookup comment=Depurador-Kali disabled=no src-address=10.1.50.13/32 table=ISP2
add action=lookup disabled=no src-address=10.1.10.0/24 table=ISP2
add action=lookup disabled=no src-address=10.1.80.0/24 table=ISP2
add action=lookup disabled=no src-address=10.1.90.0/24 table=ISP2
add action=lookup disabled=no src-address=10.1.100.0/24 table=ISP2
add action=lookup comment=PRODUCTORA src-address=10.1.150.0/24 table=ISP2

/system ntp client
set enabled=yes
/system ntp client servers
add address=ec.pool.ntp.org
add address=pool.ntp.org
Last edited by anav on Sun Jul 21, 2024 3:05 am, edited 1 time in total.
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Sun Jul 21, 2024 2:38 am

Groovy. I’ll review later today. Ty
 
CTSsean
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 79
Joined: Fri Sep 15, 2017 12:56 pm

Re: Inconsistent WG connection via 2 ISPs

Tue Jul 23, 2024 1:48 am

Okay have to assume that the WAN2 works for wireguard and servers because the Upstream router providing a private IP for wan can forward ports to the LANIP of the MT.
Just in case I kick the bucket tomorrow this encapsulates all I can do with the provided info.
If you want to buy me a ticket to Brazil I can do some onsite work LOL........
I've reviewed the config. Seems that the only thing I'm missing are the mangle rules for port forwards.

I do see that you've set the default routes to not be ECMP. Is that by design?
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 20818
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Inconsistent WG connection via 2 ISPs

Tue Jul 23, 2024 2:16 am

What is ECMP? Equal Cost or something, like the router randomly decides which WAN is primary for each session???
Never used it before.

Okay it appears to be a load balancing type.

Did you also want to load up load balancing so the WANS are equally used for all LAN traffic""

Otherwise I would not monkey with ECMP unless its needed and I dont see the case for it at the moment.
We have two things going on, wireguard and port forwarding to deal with.

Who is online

Users browsing this forum: Ahrefs [Bot], GoogleOther [Bot] and 30 guests