Problems on routing to second gateway

GiovanniG · November 8, 2023, 7:19pm

Hi, I’ve a gateway at version 6.49 which I have troubles to configure, I need your suggestion, thank you.

The router has a default route on a provider which gives a private IP, for this reason I’m using a second gateway, I’t also a private IP but there are NATted TCP ports I need to reach some devices on the LAN. I would use the second gateway only for port NAT, keeping it free from all the others LAN requests, the story is this:
a packet is coming form internet (from second gateway) and properly destnat on the LAN target, the problem is that the answer front he device is routed to the first gateway and the connection doesn’t work.
I’ve used mangle to route mark incoming packets from interface of gateway2, with that dest IP address, and on IP route I have created a defaut route on that IP, with distance 5, and routing mark name from the mangle. It doesn’t work, I’ve also added an srcnat with this IP, so the LAN device can reply even if the LAN network doen’t know the gateway2. I’ve tried with masquerade, accept, src nat, same.
It looks there still a problem of routing, on version 7 there is routing-rules, I don’t know if it may help, but on ver.6 there isn’t. What can I do? Thanks

My logic says that to don’t mess with ports the masquerade shoud be avoid, a simple source nat is needed, maybe can I do this and use the same routing mark on the srcnat entry? The goal can be to mark everything coming from the second gateway and using this mark to srcnat it and route it to the proper gateway..

mkx · November 8, 2023, 7:46pm

If I understand you correctly, you’re trying to use dual WAN with some policy-based routing? If that’s right, then … PBR works best if the device enforcing policy is the default/only gateway for LAN hosts. In case of your prefered provider that will mean double NAT but most of time this won’t hurt your users in any way.

GiovanniG · November 8, 2023, 8:46pm

Thanks for answer ) sorry I don’t understand what you meant, what is PBR?
I just need a suggestion to make the thing work, is it right I mark the routing? Maybe I need to mark packets? Route+srcnat is ok?
Thanks

anav · November 8, 2023, 8:56pm

Mangle incoming on WAN2 to go out WAN2

add chain=prerouting add action=mark-connection mark=no-mark
in-interface=WAN2 new connection-mark=W2-Incoming passthrough=yes
add chain=output add action=mark-routing connection-mark=W2-Incoming
new-routing-mark=useWAN2

Routing rules for server traffic… lets say at 192.168.0.5

Version7

add dst-address=other-local-subnet1 action=lookup-only-in-table table=main
add dst-address=other-local-subnet2 action=lookup-only-in-table table=main
add src-address=serverLANIP action=lookup-only-in-table table=useWAN2

/ip route
add dst-address=0.0.0.0/0 gateway=ISP2 table=useWAN2

/routing table add fib name=useWAN2

Version6

add dst-address=other-local-subnet1 action=lookup-only-in-table table=main
add dst-address=other-local-subnet2 action=lookup-only-in-table table=main
add src-address=serverLANIP action=lookup-only-in-table table=useWAN2

/ip route
add dst-address=0.0.0.0/0 gateway=ISP2 routing-mark=useWAN2

GiovanniG · November 9, 2023, 11:38pm

I guess there are more than one way/solution to solve this issue, I’ve tried yours but it doesn’t work, I?m not 100% sure I’ve created it, I prefer much more to use Winbox instead of command line because I can understand more and learn.. not only paste.
it’s a bit complicated me to understand your approach, why you use chain output?

With my approach I was able to reply to the dstnat on the proper WAN2 connection, but it outputs the source IP address of the LAN target, and the next router doesn’t probably understand the socket (he believes to receive the reverse socket it creates, as reply, which is: IPWAN2:destTCP, internetIPthatGeneratedRequest:theTCPitUsed. Actually I have IPLANtarget:destTCP, internetIPthatGeneratedRequest:theTCPitUsed.
So the second default router works after mangle, I’ve tried to srcnat all pacjets coming from IPLANtarget with IPWAN2 but there is nothing to do! I’ve used also post routing with mangle and marked packets or connections, no way to change that IP! it sounds crazy..

Maybe because I defined the interface with WAN2 not a WAN.. but a LAN, since there are other LANs there and it can be smart to publish the local LAN also there, so the computers can see each other. I’ll try with WAN

GiovanniG · November 10, 2023, 12:24am

It looks like it routes the reply from LAN target before it completes the dstnat on the reply. In theory it should do this:
Incoming packet, dstnat to LAN target, target replies, dstnat change the source target LAN with its wan2 IP, then routes the packet to wan2.
In my case I have dstnat, target replies, routing on wan2
I’ll try to focus on it, I’ll try to mark routing with mangle for packets with source IP wan2 and then route them, in this case the dstnat will be completed

GiovanniG · November 10, 2023, 8:51am

Talking about LAN.. probably what I need is just a srcnat at “wan2”, no routing then is needed.
Packet come with InternetsourceIP:randomTCP - publicIPWAN2:target TCP

this is destnat on my router, where I need to build config
InternetsourceIP:randomTCP - internalrouterIPWAN2:target TCP

now, placing a dstnat on target IP
InternetsourceIP:randomTCP - targetIPLAN:target TCP
and also a srcnat
routerIPLAN:randomTCP - targetIPLAN:target TCP

the target LAN shuld answer to router, which rebuilt the back-path, it should work like this, without routing

anav · November 10, 2023, 1:05pm

Clearly you dont need help then, you know how things work…

In any case, enough chit chat about what you dont know.
Please post a config, for some facts, tired of opinion.

GiovanniG · November 10, 2023, 3:32pm

Unfortunately it doesn’t work, it still output on WAN2 the LAN address as source, and the packet die in the next hop. It’s like it doesn’t complete the dstnat procedure, if it does, then it should have the WAN2 source IP. I don’t know why mangle mark routing, and it routes before the packet can be changed by dstnat, the packet exit form WAN2 as it comes from LAN, how can I postpone the routing after the competion of dstnat?

this is the config:

# 2023-11-10 18:31:13 by RouterOS 7.10.1
# software id = U31Z-TEGH
#
# model = RB750r2
# serial number = 8B3808E10A62
/interface bridge
add admin-mac=CC:2D:E0:69:19:FA auto-mac=no comment=defconf name=bridge
add name=bridgeOVPN
/interface ethernet
set [ find default-name=ether1 ] l2mtu=1600
set [ find default-name=ether2 ] l2mtu=1600
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip dhcp-client option
add code=55 name=parameter_request_list value=0x010306212A79F9
/ip dhcp-server option
add code=6 name="DNS Alternative" value="'64.225.68.86'"
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip pool
add name=default-dhcp ranges=192.168.101.100-192.168.101.200
/ip dhcp-server
add address-pool=default-dhcp interface=bridge lease-time=1w name=defconf
/ppp profile
add bridge=bridgeOVPN local-address=192.168.2.3 name=OVPN2 remote-address=\
    192.168.2.1 use-ipv6=no use-mpls=no
/interface ovpn-client
add cipher=aes256-cbc connect-to=146.66.181.172 mac-address=02:A3:5E:EB:58:64 \
    max-mtu=1460 name=Vdv port=44301 profile=OVPN2 use-peer-dns=no user=OVPN2
/interface l2tp-client
add add-default-route=yes connect-to=tp.internet.beeline.ru \
    default-route-distance=2 disabled=no max-mtu=1460 name=l2tp-beeline \
    profile=default use-peer-dns=exclusively user=0892090319
/queue simple
add name=queue4 priority=5/5 target=192.168.101.100/32
add name=queue5 priority=5/5 target=192.168.101.101/32
add disabled=yes max-limit=20M/20M name=queue1 target=192.168.0.112/32
add disabled=yes max-limit=5M/5M name=queue2 target=192.168.0.104/32
add limit-at=50M/50M max-limit=70M/70M name=queue3 target=192.168.101.116/32
/routing table
add disabled=no fib name=vdv
/snmp community
set [ find default=yes ] addresses=192.168.0.0/24 write-access=yes
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
/ip neighbor discovery-settings
set discover-interface-list=LAN
/ipv6 settings
set disable-ipv6=yes
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf disabled=yes interface=ether1 list=WAN
add interface=l2tp-beeline list=WAN
add interface=ether2 list=LAN
add interface=bridgeOVPN list=LAN
add interface=Vdv list=WAN
/ip address
add address=192.168.0.10/24 comment=defconf interface=bridge network=\
    192.168.0.0
add address=192.168.99.1/24 comment=defconf disabled=yes interface=bridge \
    network=192.168.99.0
add address=192.168.101.10/24 comment=defconf interface=bridge network=\
    192.168.101.0
/ip dhcp-client
add comment=defconf default-route-distance=5 dhcp-options=\
    parameter_request_list,hostname interface=ether1
/ip dhcp-server lease
add address=192.168.101.100 mac-address=10:F0:05:CA:8A:0A server=defconf
add address=192.168.99.101 disabled=yes mac-address=D8:CE:3A:FC:C8:7D server=\
    defconf
add address=192.168.101.252 mac-address=98:CD:AC:30:AF:9F server=defconf
add address=192.168.101.253 client-id=1:12:42:7d:5c:19:78 mac-address=\
    12:42:7D:5C:19:78 server=defconf
add address=192.168.101.101 client-id=1:d8:ce:3a:fc:c8:7d mac-address=\
    D8:CE:3A:FC:C8:7D server=defconf
add address=192.168.101.116 client-id=1:b4:2e:99:e5:95:6d mac-address=\
    B4:2E:99:E5:95:6D server=defconf
add address=192.168.101.250 client-id=1:80:f3:ef:8e:32:3 mac-address=\
    80:F3:EF:8E:32:03 server=defconf
/ip dhcp-server network
add address=192.168.101.0/24 comment=defconf dns-server=\
    8.8.8.8,1.1.1.1,192.168.101.10 gateway=192.168.101.10 netmask=24
/ip dns
set allow-remote-requests=yes servers=208.67.222.222,64.225.68.86
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan
add address=192.168.0.100 name=vdv.vdv
/ip firewall address-list
add address=192.168.0.100 list=test
add address=192.168.0.101 list=test
/ip firewall filter
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" disabled=yes \
    dst-address=127.0.0.1
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=drop chain=forward disabled=yes dst-address=192.168.2.0/24 \
    in-interface=bridge src-address-list=!test
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    connection-state=new in-interface-list=!LAN
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,new,untracked disabled=yes in-interface=bridgeOVPN
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
/ip firewall mangle
add action=mark-routing chain=prerouting disabled=yes dst-address=\
    !192.168.0.0/16 new-routing-mark=vdv passthrough=no protocol=!udp \
    src-address=192.168.0.250
add action=mark-routing chain=prerouting disabled=yes dst-address=\
    !192.168.0.0/16 new-routing-mark=vdv passthrough=no protocol=!udp \
    src-address=192.168.101.100
add action=mark-routing chain=prerouting disabled=yes dst-address=\
    !192.168.0.0/16 new-routing-mark=vdv passthrough=no protocol=!udp \
    src-address=192.168.101.101
add action=mark-routing chain=prerouting disabled=yes dst-address=\
    !192.168.0.0/16 new-routing-mark=vdv passthrough=no protocol=!udp \
    src-address=192.168.101.106
add action=mark-routing chain=prerouting connection-nat-state=dstnat \
    dst-address=192.168.2.3 dst-port=8001 new-routing-mark=vdv passthrough=\
    yes protocol=tcp
/ip firewall nat
add action=src-nat chain=srcnat out-interface=Vdv routing-mark=vdv \
    to-addresses=192.168.2.3
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
add action=src-nat chain=srcnat dst-address=192.168.2.2 to-addresses=\
    192.168.2.3
add action=src-nat chain=srcnat dst-address=192.168.2.4 to-addresses=\
    192.168.2.3
add action=src-nat chain=srcnat dst-port=8291 protocol=tcp src-address=\
    192.168.0.100 to-addresses=192.168.2.3
add action=redirect chain=dstnat dst-address=192.168.0.100 dst-port=80 \
    protocol=tcp src-address=192.168.0.0/24 to-ports=1880
add action=src-nat chain=srcnat out-interface=Vdv routing-mark=vdv \
    src-address=192.168.101.100 to-addresses=192.168.2.3
add action=dst-nat chain=dstnat dst-address=192.168.2.3 dst-port=8001 \
    protocol=tcp to-addresses=192.168.101.100 to-ports=8001
/ip route
add disabled=yes distance=1 dst-address=192.168.88.0/24 gateway=192.168.0.111 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=192.168.1.0/24 gateway=192.168.2.1 \
    routing-table=main scope=10 suppress-hw-offload=no
add disabled=no distance=8 dst-address=0.0.0.0/0 gateway=192.168.2.1 \
    pref-src=192.168.2.3 routing-table=vdv scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no dst-address=192.168.3.0/24 gateway=192.168.2.2 routing-table=\
    main suppress-hw-offload=no
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=\
    33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
/ppp secret
add name=OVPN2 profile=OVPN2 service=ovpn
/snmp
set enabled=yes
/system clock
set time-zone-name=Europe/Moscow
/system identity
set name=Baklazhan
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=193.204.114.232
add address=132.163.97.5
/system scheduler
add interval=1m name="Renew gateway" on-event="/ip dhcp-client renew ether1" \
    policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=1970-01-01 start-time=00:00:01
add interval=1d name="Public IP renew" on-event=\
    "/interface l2tp-client disable 0\r\
    \n/interface l2tp-client enable 0\r\
    \n" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=2023-03-25 start-time=06:30:00
/system script
add dont-require-permissions=no name="Public IP renew" owner=admin policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=\
    "/interface l2tp-client disable 0\r\
    \n/interface l2tp-client enable 0\r\
    \n"
/system watchdog
set auto-send-supout=yes ping-start-after-boot=10m send-email-to=\
    aa@bb.cc watch-address=8.8.8.8
/tool e-mail
set address=mail.sicurezza.ru from=aa@bb.cc port=2525 user=\
    service@sicurezza.ru
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool sniffer
set filter-interface=bridge filter-mac-protocol=arp

anav · November 10, 2023, 3:55pm

Draw a diagram to hard to understand what you need as you focus on what doesnt work and need more context…
Also have no idea of the actual user traffic requirements
Ensure you detail the OVPN structure, the L2TP structure in the diagram…

I see on the bridge you have assigned 3 IP addresses, one gets DCHP responsibilities.
You are client for OVPN on the 192.168.2.0 subnet.
Default static rule can be removed → /ip dns static
add address=192.168.88.1 comment=defconf name=router.lan

Looking at routes…
/ip route
add disabled=yes distance=1 dst-address=192.168.88.0/24 gateway=192.168.0.111
pref-src=“” routing-table=main scope=30 suppress-hw-offload=no
target-scope=10
add disabled=no distance=1 dst-address=192.168.1.0/24 gateway=192.168.2.1
routing-table=main scope=10 suppress-hw-offload=no
add disabled=no distance=8 dst-address=0.0.0.0/0 gateway=192.168.2.1
pref-src=192.168.2.3 routing-table=vdv scope=30 suppress-hw-offload=no
target-scope=10
add disabled=no dst-address=192.168.3.0/24 gateway=192.168.2.2 routing-table=
main suppress-hw-offload=no

a. Is the **192.168.88.**0 pointing to a non-existant network?? Is the device at 192.168.0.111 a natted router??
b. Is the 192.168.1.0/24 pointing to a non-existent network? Or is this a subnet at the other end of the OVPN tunnel?
c. I see you want to send some local users out OVPN for internet… no need to use distance its a different table than main!!!

Its the last route that concerns me.
If the internet rule and route to 192.168.1.0 above are is correct, the GATEWAY for OVPN is 192.168.2.1, so why do you use 192.168.2.2 for the last rule???
(Assuming that 192.168.3.0 is a subnet at the other end of the OVPN tunnel?)

in fact, all three rules are different ???
to 192.168.1.0/24 gwy=192.168.2.1
to internet gwy=192.168.2.1 prefsrc=192.168.2.3
to 192.168.3.0/24 gwy=192.168.2.2

What is the purpose of this rule… ? which basically says destinated traffic heading to 192.168.2.3 on port 8001 protocol tcp gets the applicable routing mark. Assuming that traffic as indicated goes out the IP route for OVPN tunnel but if the dst is 192.168.2.3 why in that route is presrce 192.168.2.3 ??? Coming or going?

add action=mark-routing chain=prerouting connection-nat-state=dstnat
dst-address=192.168.2.3 dst-port=8001 new-routing-mark=vdv passthrough=
yes protocol=tcp

The associated sourcenat rules simply add confusion…

GiovanniG · November 10, 2023, 4:08pm

I think I’ve described it in detail already, here is my goal:

Packet come with InternetsourceIP:randomTCP - publicIPWAN2:target TCP

the border router I cannot touch do the first dstnat
2) InternetsourceIP:randomTCP - myrouterIPWAN2:target TCP

the destnat on myrouter, target the host on LAN
3) InternetsourceIP:randomTCP - targetIPLAN:target TCP

now the LAN host replies
4) targetIPLAN:target TCP - InternetsourceIP:randomTCP

and the dstnat it’s supposed to finish its job
5) myrouterIPWAN2:target TCP - InternetsourceIP:randomTCP

and then route the packet on the right gateway (WAN2), but it’s not, the routing comes before the finish of the dstnat, so on WAN2 I have output:
targetIPLAN:target TCP - InternetsourceIP:randomTCP
instead of: myrouterIPWAN2:target TCP - InternetsourceIP:randomTCP

How can I solve this?

anav · November 10, 2023, 4:20pm

THose are shit requirements, I dont care about packets, packets dont make money, packets dont blow kisses,
PEOPLE use the router… what do people need to do… browse, bank, access servers, etc…
PEOPLE can be directed to use certain traffic paths, group of users X ( a vlan) should go out WAN1
PEOPLE access servers, external users must come in WAN2 and access a server on vlanyy

Or DEVICES need traffic flow so what do the devices need to d??.

Unless you can describe requirements properly and provided a detailed network diagram, I am wasting my time.

GiovanniG · November 10, 2023, 5:36pm

Probaly you have another way to see the problems.
I told it in the first post, the ain router has to provide internet and load the WAN1. WAN1 doesn’t have a public IP, so to target from internet some host in LAN I use WAN2, from another router which has public IP. WAN2 should not be loaded by the local trafic. What is difficult here? It’s a common situation.

I’ve managed to reach th goal in this way:

incoming packet is dstnat to host
I have a general srcnat (placed over all) with out interface WAN2, routing mark “go to WAN2” and action srcnat with the WAN2 IP
this time I mangle the replied packet from the LAN target, so source IP the LAN target and source port what was the destination TCP port (the reversed socket), in this way the dstnat is able to complete its job on the reversed socket, then the packet is srcnatted correctly with the WAN2 IP and routed to the proper WAN2 link according to mangle.

In the end I can say that it’ needed to focus on the reply from the host, and not from the first request coming from internet