Cannot get address list created automatically

ouyuan · Sun Jan 02, 2022 1:33 pm

Hello there,
my RouterOS is v7.1.1
I am trying to create a dynamic ip list with action add-dst-to-address-list, but cannot get the address list created, even cannot see any log about it. Below is the command line I used. It works in my old router (RouterOS v6.48.4).

/ip firewall filter add chain=forward action=add-dst-to-address-list protocol=tcp address-list=discord_ips address-list-timeout=30d dst-port=443 log=yes log-prefix="%%IPLIST-I-" tls-host=*discord.com* comment="Dynamically create ip list: discord_ips - discordapp.net" place-before=1

Please advise.
Thanks,

Ou

ConnyMercier · Sun Jan 02, 2022 3:30 pm

I did some quick test ....
It seams the "place-before" argument isn`t ALWAYS working in 7.1.1

probably just a bug...

ouyuan · Sun Jan 02, 2022 3:42 pm

place-before works for me.

The rule created by the command is supposed to automatically created an address list "discord_ips" when user accesses to discord.com, but it didn't.

I downgrade RouterOS to v6.46.8, it still doesn't create address list. Not sure if it is hardware or firmware related.

My router is hAP ac3 Wireless Dual-Band Router.
Router Model: RBD53iG-5HacD2HnD

ConnyMercier · Sun Jan 02, 2022 3:55 pm

jan/02/2022 14:53:16 by RouterOS 7.1.1
# model = RBD25G-5HPacQD2HPnD

ip firewall filter add chain=forward action=add-dst-to-address-list protocol=tcp address-list=discord_ips address-list-timeout=30d dst-port=443 log=yes log-prefix="%%IPLIST-I-" tls-host=*discord.com* comment="Dynamically create ip list: discord_ips - discordapp.net"

It works without a problem.

2022-01-02_14-54-38.jpg

ouyuan · Sun Jan 02, 2022 4:27 pm

It works on my old router: model number: RBD52G-5HacD2HnD, but not new one.

sindy · Sun Jan 02, 2022 5:26 pm

The issue needs to be narrowed down to a particular item (a single match condition or the add-dst-to-address-list operation itself).

So place an action=passthrough rule with same match conditions before the action=add-dst-to-address-list one, and see whether it counts. If it doesn't, remove one of the match conditions and check again; if nothing changes, put that match condition back and remove another one.

ouyuan · Sun Jan 02, 2022 5:53 pm

Thanks, Sindy.
Added passthrough rule before the forward rule with same condition, there was still no traffic pass the rule. Then, I remove "TLS Host" condition, the traffic went through the rule.
So "TLS Host" condition doesn't work like before. It is either a bug, or some other combinations are required.

sindy · Sun Jan 02, 2022 6:12 pm

Or the discord client uses some other port than 443, or it uses TLS1.3 so the tls-host is not present in the packet... packet sniffing to your help here. It may or may not be related to 7.1.

ouyuan · Sun Jan 02, 2022 6:25 pm

The problem is still outstanding even I downgraded the OS to v6.48.5. Let's say it is no V7.1 related.
By the way, I removed dst port, there is still no traffic went through the rule.

[admin@MikroTik] > /ip f f print
0 D ;;; special dummy rule to show fasttrack counters
chain=forward action=passthrough

1 ;;; Dynamically create ip list: discord_ips - discord.com
chain=forward action=passthrough protocol=tcp log=yes log-prefix="%%IPLIST-I-" tls-host=*discord.com*

2 ;;; Dynamically create ip list: discord_ips - discord.com
chain=forward action=add-dst-to-address-list protocol=tcp address-list=discord_ips address-list-timeout=4w2d
dst-port=443 log=yes log-prefix="%%IPLIST-I-" tls-host=*discord.com*

ConnyMercier · Sun Jan 02, 2022 6:31 pm

How do you test your Rules ?
Did you try opening a Web-Browser and entering www.discord.com?

ouyuan · Sun Jan 02, 2022 6:34 pm

How do you test your Rules ?
Did you try opening a Web-Browser and entering www.discord.com?

Yes, that is how I am testing it.

ConnyMercier · Sun Jan 02, 2022 6:41 pm

Is it possible to Post your Config ?
(/export hide-sensitive file=anynameyouwish)

I am not able to reproduce your Problem in my LAB
As soon as I add your Firewall-Rule, the Address-List gets populated,
as soon as i open the Browser and enter discord.com

ouyuan · Sun Jan 02, 2022 6:54 pm

Is it possible to Post your Config ?
(/export hide-sensitive file=anynameyouwish)

I am not able to reproduce your Problem in my LAB
As soon as I add your Firewall-Rule, the Address-List gets populated,
as soon as i open the Browser and enter discord.com

Thanks, here is the dump.

# jan/02/2022 11:50:11 by RouterOS 6.48.5
# software id = DKSE-JYW3
#
# model = RBD53iG-5HacD2HnD
# serial number = F3550F029581
/interface bridge
add admin-mac=DC:2C:4E:03:55:2A auto-mac=no comment=defconf name=bridge
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-XX \
    disabled=no distance=indoors frequency=auto installation=indoor mode=\
    ap-bridge ssid=MikroTik-03512E wireless-protocol=802.11
set [ find default-name=wlan2 ] band=5ghz-a/n/ac channel-width=\
    20/40/80mhz-XXXX disabled=no distance=indoors frequency=auto \
    installation=indoor mode=ap-bridge ssid=Os-Man-Thus wireless-protocol=\
    802.11
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] authentication-types=wpa-psk,wpa2-psk mode=\
    dynamic-keys supplicant-identity=MikroTik
/ip firewall layer7-protocol
add name=discord regexp="^.+(discord.com).*\$"
/ip pool
add name=dhcp ranges=192.168.66.100-192.168.66.254
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge name=defconf
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=wlan1
add bridge=bridge comment=defconf interface=wlan2
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
/ip address
add address=192.168.66.1/24 comment=defconf interface=ether2 network=\
    192.168.66.0
add address=192.168.88.32/8 interface=ether1 network=192.0.0.0
/ip dhcp-client
add comment=defconf interface=ether1
/ip dhcp-server network
add address=192.168.66.0/24 comment=defconf gateway=192.168.66.1 netmask=24
/ip dns
set allow-remote-requests=yes servers=208.67.222.222,208.67.220.220
/ip dns static
add address=192.168.66.1 comment=defconf name=router.lan
/ip firewall filter
add action=passthrough chain=forward comment=\
    "Dynamically create ip list: discord_ips - discord.com" log=yes \
    log-prefix=%%IPLIST-I- protocol=tcp tls-host=*discord.com*
add action=add-dst-to-address-list address-list=discord_ips \
    address-list-timeout=4w2d chain=forward comment=\
    "Dynamically create ip list: discord_ips - discord.com" dst-port=443 log=\
    yes log-prefix=%%IPLIST-I- protocol=tcp tls-host=*discord.com*
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
/ip route
add distance=1 gateway=192.168.88.1
/system clock
set time-zone-name=America/Toronto
/system leds
set 0 interface=wlan1 leds=led1,led2,led3,led4,led5 type=\
    wireless-signal-strength
set 1 leds=poe-led type=poe-out

ConnyMercier · Sun Jan 02, 2022 7:12 pm

Your Config is....let say Unique =)

Problem 1:
You have created a Bridge and assigned your Interfaces to the LAN.
But for some reason you have assigned IP-Address and in turn DHCP-Server to a "Slave" Interface.

This can create Problems

Wrong:

/ip address
add address=192.168.66.1/24 comment=defconf interface=ether2 network=192.168.66.0

Problem 2:
You assigned your Ether1 (WAN) a Static IP 192.168.88.32/8 AND DHCP-Client

This can create Problems as well.
+ The /8 Network-Range is wrong and can make problem in your config

ConnyMercier · Sun Jan 02, 2022 7:22 pm

Do you need help to rectify the Config?

ouyuan · Sun Jan 02, 2022 7:38 pm

Your Config is....let say Unique =)

Problem 1:
You have created a Bridge and assigned your Interfaces to the LAN.
But for some reason you have assigned IP-Address and in turn DHCP-Server to a "Slave" Interface.

This can create Problems

Wrong:
Code: Select all
/ip address
add address=192.168.66.1/24 comment=defconf interface=ether2 network=192.168.66.0
Problem 2:
You assigned your Ether1 (WAN) a Static IP 192.168.88.32/8 AND DHCP-Client

This can create Problems as well.
+ The /8 Network-Range is wrong and can make problem in your config

Mikrotik hap ac3 is my 2nd level router behind the ISP router.

ouyuan · Sun Jan 02, 2022 7:39 pm

Do you need help to rectify the Config?

That will be great.

Thanks in advance.

ConnyMercier · Sun Jan 02, 2022 8:02 pm

If you enter this in a CLI, it should rectify your Setup

/ip address set [find address="192.168.66.1/24"] interface=bridge
/ip address set [find address="192.168.88.32/8"] address="192.168.88.32/24"
/ip dhcp-client disable [find interface=ether1]

ouyuan · Sun Jan 02, 2022 8:22 pm

If you enter this in a CLI, it should rectify your Setup

/ip address set [find address="192.168.66.1/24"] interface=bridge
/ip address set [find address="192.168.88.32/8"] address="192.168.88.32/24"
/ip dhcp-client disable [find interface=ether1]

Thanks, ConnyMercier

It has been patched.

There is still no traffic goes through :
1 ;;; Dynamically create ip list: discord_ips - discord.com
chain=forward action=passthrough protocol=tcp log=yes log-prefix="%%IPLIST-I-" tls-host=*discord.com*

ConnyMercier · Sun Jan 02, 2022 8:24 pm

Can you please Post the latest Config ?
(/export hide-sensitive file=anynameyouwish)

ouyuan · Sun Jan 02, 2022 8:46 pm

Can you please Post the latest Config ?
(/export hide-sensitive file=anynameyouwish)

Please check the attachment.

hapac3-1.rsc

ConnyMercier · Sun Jan 02, 2022 8:52 pm

For some reason or another, the DHCP-Client is still active .....
Can you please reenter this in a CLI

/ip dhcp-client disable [find interface=ether1]

ouyuan · Sun Jan 02, 2022 9:01 pm

For some reason or another, the DHCP-Client is still active .....
Can you please reenter this in a CLI
Code: Select all
/ip dhcp-client disable [find interface=ether1]

The dhcp-client shows dissabled.

hapac3-2.rsc

ConnyMercier · Sun Jan 02, 2022 9:12 pm

That is weird ....
In the Export the DHCP-Client isn`t disabled

/ip dhcp-client
add comment=defconf interface=ether1

ConnyMercier · Sun Jan 02, 2022 9:18 pm

Q1: Do your device behind the Router have internet?
Q2: What Network-Cable are connected to your Router and where do they go ?

ouyuan · Sun Jan 02, 2022 9:19 pm

That is weird ....
In the Export the DHCP-Client isn`t disabled
Code: Select all
/ip dhcp-client
add comment=defconf interface=ether1

I just deleted the entry. And did a quick test, still no traffic goes through the rule.

hapac3-3.rsc

ouyuan · Sun Jan 02, 2022 9:24 pm

Q1: Do your device behind the Router have internet?

Q2: What Network-Cable are connected to your Router and where do they go ?

Q1: Do your device behind the Router have internet?
Yes, there is internet. I am using the router to communicate with you right now.
Q2: What Network-Cable are connected to your Router and where do they go ?
The router is connected to My ISP Router thru a switch with CAT 5 cable.

Mikrotik Router <===> Dlink Switch <===> ISP Router <===> Internet

ConnyMercier · Sun Jan 02, 2022 9:33 pm

So the only cable connected to the Mikrotik is ether1 ?

And can you try adding these Firewall Rule,
And tell me if the Rules get Traffic , when you are surfing the Web and accessing www.discord.com
Thanks!

/ip firewall filter
add action=passthrough chain=forward comment="TEMP Internet Traffic-Counter (LAN --> WAN)" dst-address=!192.168.88.0/24 in-interface-list=LAN out-interface-list=WAN place-before=0
add action=add-dst-to-address-list address-list=discord_ips address-list-timeout=4w2d chain=forward comment="TEMP Dynamically create ip list: discord_ips - discordapp.net" log=yes log-prefix=%%IPLIST-I- protocol=tcp tls-host=*discord.com* place-before=1

sindy · Sun Jan 02, 2022 9:37 pm

That is weird ....
In the Export the DHCP-Client isn`t disabled

The /interface dhcp-client items are disabled by default, hence disabled=yes is not shown in the output of export without the verbose modifier.

ouyuan · Sun Jan 02, 2022 9:41 pm

So the only cable connected to the Mikrotik is ether1 ?

And can you try adding these Firewall Rule,
And tell me if the Rules get Traffic , when you are surfing the Web and accessing www.discord.com
Thanks!
Code: Select all
/ip firewall filter
add action=passthrough chain=forward comment="TEMP Internet Traffic-Counter (LAN --> WAN)" dst-address=!192.168.88.0/24 in-interface-list=LAN out-interface-list=WAN place-before=0
add action=add-dst-to-address-list address-list=discord_ips address-list-timeout=4w2d chain=forward comment="TEMP Dynamically create ip list: discord_ips - discordapp.net" log=yes log-prefix=%%IPLIST-I- protocol=tcp tls-host=*discord.com* place-before=1

Here is a screen shot. I tried discord.com, there is no traffic went through rule 2.

sindy · Sun Jan 02, 2022 9:49 pm

There's a remark in the manual:
Note that matcher will not be able to match hostname if TLS handshake frame is fragmented into multiple TCP segments (packets).
Maybe the MTU is different on this router than on the one where it works, so the BPDU carrying the TLS SNI hostname is split into two packets?

ConnyMercier · Sun Jan 02, 2022 9:53 pm

Thanks @Sindy for the Backup =)

Could the DNS-Servers be the Problem?

ouyuan · Sun Jan 02, 2022 9:54 pm

There's a remark in the manual:
Note that matcher will not be able to match hostname if TLS handshake frame is fragmented into multiple TCP segments (packets).
Maybe the MTU is different on this router than on the one where it works, so the BPDU carrying the TLS SNI hostname is split into two packets?

The MTU on the interfaces are all 1500.

Screen Shot 2022-01-02 at 2.53.38 PM.png

sindy · Sun Jan 02, 2022 9:57 pm

Could the DNS-Servers be the Problem?

How? The client does get the domain name resolved, so the DNS server works. So the issue must be with either the tls-host matcher or with the port, if by some strange reason the connection establishes via another port than 443.

ConnyMercier · Sun Jan 02, 2022 10:02 pm

@sindy , your are right !
I still testet both DNS-Server .... they work fine !

I am out of deas....
Imported the Config in a Lab-Router
It is also double NATed

My Laptop and Cellphone are able to connect to Discord and
both the Android-App AND the Website (discord.com) trigger the Filter-Rules

ouyuan · Sun Jan 02, 2022 10:05 pm

@sindy , your are right !
I still testet both DNS-Server .... they work fine !

I am out of deas....
Imported the Config in a Lab-Router
It is also double NATed

My Laptop and Cellphone are able to connect to Discord and
both the Android-App AND the Website (discord.com) trigger the Filter-Rules

The difference could be the Router model. Mine is hap ac3.

By the way, my another hap ac2 works.

sindy · Sun Jan 02, 2022 10:19 pm

The difference could be the Router model. Mine is hap ac3.

By the way, my another hap ac2 works.

These two routers have the same CPU architecture, therefore the same software package. That's not it.

Local MTU of 1500 doesn't mean that there cannot be some bottleneck between the router and the remote server, so the size of TCP packets can get reduced by the Path MTU Discovery process.

I would recommend you to run packet sniffer and use Wireshark to see how the Client Hello packet the client sends to the Discord server looks like at the WAN interface.

Or you can try a content matcher instead - it does an exact match (no wildcards), but anywhere in the packet, i.e. it doesn't look for the Client Hello BPDU and within it for the *discord.com* at an exact place. So content=disc and content=cord might give some hint if you want to avoid sniffing and Wireshark analysis.

ConnyMercier · Sun Jan 02, 2022 10:38 pm

So i did some more Test....
I don`t understand exactly why, but if you delete Fasttrack and Restart the Router it may work

enter this in a CLI and then Reboot

/ip firewall filter remove [find action=fasttrack-connection]

sindy · Sun Jan 02, 2022 11:30 pm

I don`t understand exactly why, but if you delete Fasttrack and Restart the Router it may work

It is well possible that the connection is already fasttracked when the Client Hello is being sent, so the packet bypasses the firewall thanks to that.

And it should not be even necessary to restart the router - each HTTPS connection is separate, so it won't get fasttracked once the action=fasttrack-connection rule is disabled or removed.

ouyuan · Mon Jan 03, 2022 2:45 am

So i did some more Test....
I don`t understand exactly why, but if you delete Fasttrack and Restart the Router it may work

enter this in a CLI and then Reboot
Code: Select all
/ip firewall filter remove [find action=fasttrack-connection]

it works like a chime. No reboot required like Sindy said.
Appreciate your time and effort to help me resolve the problem.

ouyuan · Mon Jan 03, 2022 2:46 am

I don`t understand exactly why, but if you delete Fasttrack and Restart the Router it may work
It is well possible that the connection is already fasttracked when the Client Hello is being sent, so the packet bypasses the firewall thanks to that.

And it should not be even necessary to restart the router - each HTTPS connection is separate, so it won't get fasttracked once the action=fasttrack-connection rule is disabled or removed.

Thanks, Sindy

Who is online