are this rules on the top mandatory?

tomislav91 · Fri Oct 30, 2020 11:17 pm

/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=fasttrack-connection chain=forward

what do you think about this 3 rules on the top of the firewall? If I dont put fasttrack my cpu goes to 50,60%. It is ccr 1008.

CZFan · Sat Oct 31, 2020 12:16 am

First two rules are for input chain, the 3rd, fasttrack is for forward chain and has nothing to do with first 2 rules.

Also not sure I understand your question?

tomislav91 · Sat Oct 31, 2020 3:14 am

First two rules are for input chain, the 3rd, fasttrack is for forward chain and has nothing to do with first 2 rules.

Also not sure I understand your question?

my q was, does it something thats a MUST HAVE this input chain rules to have most of the benefits of the firewall. Is there some updated firewall rules for preventing intrusions?Maybe someone to share their firewall without sensitive data ofc

Pea · Sat Oct 31, 2020 12:16 pm

https://wiki.mikrotik.com/wiki/Manual:S ... our_Router

sindy · Sat Oct 31, 2020 1:04 pm

my q was, does it something thats a MUST HAVE this input chain rules to have most of the benefits of the firewall. Is there some updated firewall rules for preventing intrusions?Maybe someone to share their firewall without sensitive data ofc

Short answer: yes, it is a must have.

Long answer:
Fasttracking means that packets belonging to existing connections bypass

all the rest of the firewall handling following the connection tracking
IPsec policy matching
queues except the interface ones.

So if fasttracking rule lowers the CPU load significantly, something in the above list is causing the CPU load.

Without fasttracking, the action=accept connection-state=established,related,untracked accepts the part of forwarded traffic which would otherwise be accepted due to fasttracking. Only initial packets of forwarded connections will make it past this rule to the subsequent filter ones; these subsequent rules decide which connections will be permitted and which will not.

If connection tracking is active, every single packet is compared to the complete list of existing connections. And if it matches any already existing one, it is labeled with connection-state "established", and the rule you mention accepts it (or fasttracking accepts it instead if enabled).

The connection tracking itself is also quite CPU intensive, and fasttracking makes no sense in input chain. So as in your case, a lot of traffic is probably IPsec one, you could use
chain=prerouting dst-address-type=local protocol=udp dst-port=500,4500 action=notrack
chain=prerouting dst-address-type=local protocol=ipsec-esp action=notrack
chain=output protocol=udp src-port=500,4500 action=notrack
chain=output protocol=ipsec-esp action=notrack
rules in /ip firewall raw to prevent IPsec packets to and from your router itself from being connection-tracked. The same rule in filter will accept them as it matches also on connection-state "untracked", but the amount of CPU spent on those raw rules may be lower than the one otherwise spent by connection-tracking on the same packets. The actual benefit of this depends on the amount of tracked connections in total.

If you eventually don't need to use the router as a firewall for forwarded traffic, you may stop using connection tracking at all and only use rules in input chain to protect the router itself. Connection tracking is common for all traffic (both local and forwarded), so stopping to refer to connection-state and connection-nat-state only in input chain of filter will not help lower the CPU load. But in that case, the order of the rules in the filter is very important - most packets should pass through the least number of rules, so accept rules for the most frequent packets must be at the top of the list (probably ipsec-esp).

anav · Sat Oct 31, 2020 3:16 pm

Why dont you ask the real question which is.

I don't have a sweet effing clue about what the MT firewall rules do or mean, so can somebody please explain them.!

(which is basically the question Sindy answered because he is a very kind man and wicked smart).

tomislav91 · Sat Oct 31, 2020 6:07 pm

my q was, does it something thats a MUST HAVE this input chain rules to have most of the benefits of the firewall. Is there some updated firewall rules for preventing intrusions?Maybe someone to share their firewall without sensitive data ofc
Short answer: yes, it is a must have.

Long answer:
Fasttracking means that packets belonging to existing connections bypass
all the rest of the firewall handling following the connection tracking

IPsec policy matching

queues except the interface ones.
So if fasttracking rule lowers the CPU load significantly, something in the above list is causing the CPU load.

Without fasttracking, the action=accept connection-state=established,related,untracked accepts the part of forwarded traffic which would otherwise be accepted due to fasttracking. Only initial packets of forwarded connections will make it past this rule to the subsequent filter ones; these subsequent rules decide which connections will be permitted and which will not.

If connection tracking is active, every single packet is compared to the complete list of existing connections. And if it matches any already existing one, it is labeled with connection-state "established", and the rule you mention accepts it (or fasttracking accepts it instead if enabled).

The connection tracking itself is also quite CPU intensive, and fasttracking makes no sense in input chain. So as in your case, a lot of traffic is probably IPsec one, you could use
chain=prerouting dst-address-type=local protocol=udp dst-port=500,4500 action=notrack
chain=prerouting dst-address-type=local protocol=ipsec-esp action=notrack
chain=output protocol=udp src-port=500,4500 action=notrack
chain=output protocol=ipsec-esp action=notrack
rules in /ip firewall raw to prevent IPsec packets to and from your router itself from being connection-tracked. The same rule in filter will accept them as it matches also on connection-state "untracked", but the amount of CPU spent on those raw rules may be lower than the one otherwise spent by connection-tracking on the same packets. The actual benefit of this depends on the amount of tracked connections in total.

If you eventually don't need to use the router as a firewall for forwarded traffic, you may stop using connection tracking at all and only use rules in input chain to protect the router itself. Connection tracking is common for all traffic (both local and forwarded), so stopping to refer to connection-state and connection-nat-state only in input chain of filter will not help lower the CPU load. But in that case, the order of the rules in the filter is very important - most packets should pass through the least number of rules, so accept rules for the most frequent packets must be at the top of the list (probably ipsec-esp).

So you think this is the best rules to put it on top

/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=fasttrack-connection chain=forward

and add this ti raw

add action=notrack chain=prerouting dst-address-type=local dst-port=500,4500 protocol=udp
add action=notrack chain=prerouting dst-address-type=local protocol=ipsec-esp
add action=notrack chain=output protocol=udp src-port=500,4500
add action=notrack chain=output protocol=ipsec-esp

sindy · Sat Oct 31, 2020 6:47 pm

So you think this is the best rules to put it on top

There is no such thing as "the top of filter". There is the top of chain input in filter, and there is the top of chain forward in filter. Packets always only go through one of these rule chains, not both.
So the default firewall of the SOHO grade devices is the optimum one:
add action=accept chain=input connection-state=established,related,untracked
add action=drop chain=input connection-state=invalid
... individual permissive rules if necessary for IPsec and other VPNs and/or for direct remote management via secure protocols...
add action=drop chain=input in-interface-list=WAN
If you don't want to permit access to router management from anywhere in the LAN, add permissive rules for authorized LAN devices where indicated and remove the in-interface-list=WAN from the last rule in input.

Adding the rules I've suggested to raw may reduce the CPU load. You have to try, it depends on your traffic structure whether the difference will be noticeable or not.

In forward, the structure is basically the same, except that the action=fasttrack-connection rule must be followed by action=accept chain=forward connection-state=established,related,untracked because some small amount of packets belonging to fasttracked connections takes the full path through the firewall, and if you use bare IPsec with policy matching, it must be preceded by action=accept chain=forward connection-state=established,related,untracked ipsec-policy=in,ipsec and action=accept chain=forward connection-state=established,related,untracked ipsec-policy=out,ipsec as connections which need to be handled by IPsec policies must not get fasttracked. In special cases, these rules exempting connections from fasttracking have to be more complex (where NAT is necessary in order that the packet matched the IPsec policy, it doesn't match the ipsec-policy=in|out,ipsec while being handled in filter), so you need to use src-address(-list) and dst-address(-list) matching, connection-mark matching etc. there.

tomislav91 · Sat Oct 31, 2020 8:22 pm

So you think this is the best rules to put it on top
There is no such thing as "the top of filter". There is the top of chain input in filter, and there is the top of chain forward in filter. Packets always only go through one of these rule chains, not both.
So the default firewall of the SOHO grade devices is the optimum one:
add action=accept chain=input connection-state=established,related,untracked
add action=drop chain=input connection-state=invalid
... individual permissive rules if necessary for IPsec and other VPNs and/or for direct remote management via secure protocols...
add action=drop chain=input in-interface-list=WAN
If you don't want to permit access to router management from anywhere in the LAN, add permissive rules for authorized LAN devices where indicated and remove the in-interface-list=WAN from the last rule in input.

Adding the rules I've suggested to raw may reduce the CPU load. You have to try, it depends on your traffic structure whether the difference will be noticeable or not.

In forward, the structure is basically the same, except that the action=fasttrack-connection rule must be followed by action=accept chain=forward connection-state=established,related,untracked because some small amount of packets belonging to fasttracked connections takes the full path through the firewall, and if you use bare IPsec with policy matching, it must be preceded by action=accept chain=forward connection-state=established,related,untracked ipsec-policy=in,ipsec and action=accept chain=forward connection-state=established,related,untracked ipsec-policy=out,ipsec as connections which need to be handled by IPsec policies must not get fasttracked. In special cases, these rules exempting connections from fasttracking have to be more complex (where NAT is necessary in order that the packet matched the IPsec policy, it doesn't match the ipsec-policy=in|out,ipsec while being handled in filter), so you need to use src-address(-list) and dst-address(-list) matching, connection-mark matching etc. there.

2020-10-31 19_16_28-Window.png

but then i got problem then with dude just "getting stuff" but nothing happend than, and when disable fasttrackk it came and everything is ok. Dude is a mikrotik which is located on another HQ and been connected via ipsec to this mikrotik where i put these comands.
And router is going up to 70% cpu, and when put fasttrack it came below 30. I see in tool profile that firewall consume about 30% of cpu but cant find why.

/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked disabled=yes
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid disabled=yes
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked disabled=yes
add action=fasttrack-connection chain=forward disabled=yes

add action=accept chain=forward dst-address=10.11.0.0/16 src-address=192.168.30.0/24
add action=accept chain=input dst-address=10.11.0.0/16 src-address=192.168.30.0/24

add action=drop chain=forward comment="Drop Inv." connection-state=invalid
add action=accept chain=forward dst-address=10.98.0.0/24 src-address=192.168.2.0/24
add action=accept chain=input in-interface=vrrp_WAN1 protocol=icmp
add action=accept chain=input comment="permit from local ip DNS reuqest udp 53" dst-port=53 protocol=udp src-address-list=local_ip


add action=accept chain=forward dst-address=192.168.200.0/24 src-address=192.168.23.0/24
add action=accept chain=forward dst-address=192.168.200.0/24 src-address=172.16.2.0/24
add action=accept chain=forward dst-address=192.168.200.0/24 src-address=192.168.30.0/24
add action=reject chain=input comment="Drop !DNS" dst-port=53 protocol=udp reject-with=icmp-network-unreachable
add action=reject chain=input comment="Drop !DNS" dst-port=53 protocol=tcp reject-with=icmp-network-unreachable
add action=accept chain=input comment=SSH dst-port=4777 protocol=tcp src-address-list=HQ2Addresses
add action=accept chain=input comment=SSH dst-port=4777 protocol=tcp src-address-list=HQ1Addresses
add action=accept chain=input comment=WinBox dst-port=8291 protocol=tcp src-address-list=HQ1Addresses
add action=accept chain=input comment=WinBox dst-port=8291 protocol=tcp src-address-list=HQ2Addresses
add action=accept chain=input comment="Allow ICMP" protocol=icmp src-address-list=HQ2Addresses
add action=accept chain=input comment="Allow ICMP" protocol=icmp src-address-list=HQ1Addresses
add action=drop chain=input comment="Drop Inv." connection-state=invalid
add action=reject chain=input comment="Drop !DNS" dst-port=53 protocol=tcp reject-with=icmp-network-unreachable src-address-list=!DNS
add action=reject chain=input comment="Drop !DNS" dst-port=53 protocol=udp reject-with=icmp-network-unreachable src-address-list=!DNS
add action=accept chain=output comment="OUT PMTUD" icmp-options=3:4 protocol=icmp
add action=accept chain=input comment="IN PMTUD" icmp-options=3:4 protocol=icmp
add action=accept chain=input comment="IN-Allow ping 1468b up to 5 in sec" limit=5,1:packet packet-size=1468 protocol=icmp
add action=add-src-to-address-list address-list=pingers address-list-timeout=1d chain=input comment="IN-List ICMP which dont match criteria" dst-address-list=!HQ1Addresses in-interface-list=WAN log-prefix=Ping@IN protocol=icmp src-address-list=\
    !HQ1Addresses
add action=add-src-to-address-list address-list=@Services_Phase1 address-list-timeout=30m chain=input comment=IN-Services_Phase1 dst-address-list=!HQ1Addresses dst-port=21,22,23,69,80,443,5060,8080 in-interface-list=WAN protocol=tcp src-address-list=\
    !HQ1Addresses
add action=add-src-to-address-list address-list=@Services_Phase1 address-list-timeout=30m chain=input comment=IN-Services_Phase1-UDP dst-address-list=!HQ1Addresses dst-port=21,22,23,69,80,443,5060,8080 in-interface-list=WAN protocol=udp src-address-list=\
    !HQ1Addresses
add action=add-src-to-address-list address-list=@Services_Phase2 address-list-timeout=30m chain=input comment=IN-Services_Phase2 dst-port=21,22,23,69,80,443,5060,8080 in-interface-list=WAN protocol=tcp src-address-list=@Services_Phase1
add action=add-src-to-address-list address-list=@Services_Phase2 address-list-timeout=30m chain=input comment=IN-Services_Phase2-UDP dst-port=21,22,23,69,80,443,5060,8080 in-interface-list=WAN protocol=udp src-address-list=@Services_Phase1
add action=add-src-to-address-list address-list=@Services_Phase3 address-list-timeout=1w chain=input comment=IN-Services_Phase3 dst-port=21,22,23,69,80,443,5060,8080 in-interface-list=WAN protocol=tcp src-address-list=@Services_Phase2
add action=add-src-to-address-list address-list=@Services_Phase3 address-list-timeout=1w chain=input comment=IN-Services_Phase3-UDP dst-port=21,22,23,69,80,443,5060,8080 in-interface-list=WAN protocol=udp src-address-list=@Services_Phase2
add action=drop chain=input comment=IN-Phase3_dropRAW src-address-list=@Services_Phase3
add action=drop chain=input comment="IN-Block Shodan" src-address-list=shodan
add action=drop chain=input comment="IN-Defend from pingers" src-address-list=pingers

add action=accept chain=input dst-port=161 protocol=udp
add action=accept chain=forward dst-port=161 protocol=udp
add action=accept chain=output dst-port=161 protocol=ud

anav · Sat Oct 31, 2020 8:35 pm

Gluck with that bloated mess Sindy. ;-P

sindy · Sat Oct 31, 2020 10:04 pm

but then i got problem then with dude just "getting stuff" but nothing happend than, and when disable fasttrackk it came and everything is ok.

I have explained this in the previous post, which you've quoted as a whole (no idea why you do that) but apparently haven't read or understood it.

The connections transported using IPsec must not get fasttracked. As your current action=fasttrack-connection rule matches on all forwarded traffic, it prevents Dude requests from being forwarded to destinations accessible via IPsec, and prevents responses from devices which are accessible without IPsec from being delivered back to the Dude.

You have to prevent traffic which needs to be transported using IPsec from matching that rule, or from even reaching it. I've told you how to do that - by placing rules that accept packets beloging to both directions of that traffic in filter before these packets reach the action=fasttrack-connection rule.

But with the fasttrack rule fasttracking only what it should, the CPU load may not decrease too much, as the difference in CPU load with the fasttracking rule disabled and enabled may be caused by the fact that part of the traffic simply doesn't exist - if the requests don't reach the destination, there are no responses, and if the initial query doesn't succeed, the subsequent ones are not sent at all. So the actual decrease of CPU load depends on the ratio between the traffic transported using IPsec and the traffic which doesn't need IPsec.

Another noteworthy point is that the firewall rules you currently have in filter chain forward effectively do nothing, because the traffic which doesn't match to any of the rules gets accepted. The rules in output chain are the same case - everything the router sends out is sent, but each packet is inspected by two rules.

tomislav91 · Sat Oct 31, 2020 11:37 pm

but then i got problem then with dude just "getting stuff" but nothing happend than, and when disable fasttrackk it came and everything is ok.
I have explained this in the previous post, which you've quoted as a whole (no idea why you do that) but apparently haven't read or understood it.

The connections transported using IPsec must not get fasttracked. As your current action=fasttrack-connection rule matches on all forwarded traffic, it prevents Dude requests from being forwarded to destinations accessible via IPsec, and prevents responses from devices which are accessible without IPsec from being delivered back to the Dude.

You have to prevent traffic which needs to be transported using IPsec from matching that rule, or from even reaching it. I've told you how to do that - by placing rules that accept packets beloging to both directions of that traffic in filter before these packets reach the action=fasttrack-connection rule.

But with the fasttrack rule fasttracking only what it should, the CPU load may not decrease too much, as the difference in CPU load with the fasttracking rule disabled and enabled may be caused by the fact that part of the traffic simply doesn't exist - if the requests don't reach the destination, there are no responses, and if the initial query doesn't succeed, the subsequent ones are not sent at all. So the actual decrease of CPU load depends on the ratio between the traffic transported using IPsec and the traffic which doesn't need IPsec.

Another noteworthy point is that the firewall rules you currently have in filter chain forward effectively do nothing, because the traffic which doesn't match to any of the rules gets accepted. The rules in output chain are the same case - everything the router sends out is sent, but each packet is inspected by two rules.

is this better? i missread the ipsec in and out in forward.
first i add mangle to capture ipsec, router is now at 25,30%

/ip firewall mangle add action=mark-connection chain=forward comment="mark ipsec connections" ipsec-policy=out,ipsec new-connection-mark=ipsec
/ip firewall mangle add action=mark-connection chain=forward comment="mark ipsec connections" ipsec-policy=in,ipsec new-connection-mark=ipsec

than do what i think you told me

/ip firewall filter
/ip firewall filter
add action=fasttrack-connection chain=forward connection-mark=!ipsec connection-state=established,related
add action=accept chain=forward connection-state=established,related
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid

because the traffic which doesn't match to any of the rules gets accepted

how to fix it? just to add drop rule?

sindy · Sun Nov 01, 2020 10:33 am

is this better? i missread the ipsec in and out in forward.
first i add mangle to capture ipsec, router is now at 25,30%

Yes, this is yet another way how to do that. With this setup, a packet transported using IPsec is inspected by 1.5 mangle rule on average (those matching on ipsec-policy=out,ipsec are handled by one, those matching on ipsec-policy=in,ipsec are handled by two). You can reduce that to a single rule for each direction by placing action=accept chain=forward connection-state=!new before them in mangle. It is sufficient to assign the connection-mark to the connection when handling its initial packet, so all subsequent packets of the same connection don't need to be inspected for a match on the ipsec-policy again. So all mid-connection packets will match already on the connection-state=!new.

because the traffic which doesn't match to any of the rules gets accepted
how to fix it? just to add drop rule?

Yes. Plus you may want to place an action=log rule before the action=drop one to see what the drop one drops. It will help you notice which "legal" traffic you forgot to permit, and what is the illegal traffic trying to get through your router.

tomislav91 · Sun Nov 01, 2020 3:49 pm

is this better? i missread the ipsec in and out in forward.
first i add mangle to capture ipsec, router is now at 25,30%
Yes, this is yet another way how to do that. With this setup, a packet transported using IPsec is inspected by 1.5 mangle rule on average (those matching on ipsec-policy=out,ipsec are handled by one, those matching on ipsec-policy=in,ipsec are handled by two). You can reduce that to a single rule for each direction by placing action=accept chain=forward connection-state=!new before them in mangle. It is sufficient to assign the connection-mark to the connection when handling its initial packet, so all subsequent packets of the same connection don't need to be inspected for a match on the ipsec-policy again. So all mid-connection packets will match already on the connection-state=!new.

because the traffic which doesn't match to any of the rules gets accepted
how to fix it? just to add drop rule?
Yes. Plus you may want to place an action=log rule before the action=drop one to see what the drop one drops. It will help you notice which "legal" traffic you forgot to permit, and what is the illegal traffic trying to get through your router.

i added that firewall mangle rule before those ipsec.
I got aproblem that tunnels goes down, msg1 sent error and i must disable all that i newly created and restart peer and than tunnels go up.

Yes

2020-11-01 14_47_00-Window.png

you think about these 3 rules that dont have a drop one? i cant see any other from my post under.

sindy · Sun Nov 01, 2020 4:52 pm

i added that firewall mangle rule before those ipsec.
I got aproblem that tunnels goes down, msg1 sent error and i must disable all that i newly created and restart peer and than tunnels go up.

I can't get how a rule in forward chain of mangle should break IPsec transport and control traffic which is handled by chains input and output. Post the export of the mangle.

Yes

2020-11-01 14_47_00-Window.png
you think about these 3 rules that dont have a drop one? i cant see any other from my post under.
[/quote]
I was talking about adding the "drop the rest" rule solely to chain forward in filter. If you've added a "drop the rest" rule also to chain output in flter, there's no wonder it has broken the tunnels. A drop in output of filter is only used for very specific purposes. Regarding chain input of filter, I didn't dive deep into what you've set up there.

tomislav91 · Sun Nov 01, 2020 5:11 pm

i added that firewall mangle rule before those ipsec.
I got aproblem that tunnels goes down, msg1 sent error and i must disable all that i newly created and restart peer and than tunnels go up.
I can't get how a rule in forward chain of mangle should break IPsec transport and control traffic which is handled by chains input and output. Post the export of the mangle.

Yes
2020-11-01 14_47_00-Window.png
you think about these 3 rules that dont have a drop one? i cant see any other from my post under.

I was talking about adding the "drop the rest" rule solely to chain forward in filter. If you've added a "drop the rest" rule also to chain output in flter, there's no wonder it has broken the tunnels. A drop in output of filter is only used for very specific purposes. Regarding chain input of filter, I didn't dive deep into what you've set up there.
[/quote]

This is mangle now

/ip firewall mangle
add action=mark-packet chain=forward new-packet-mark=VOIP passthrough=yes \
    src-address=VOIP
add action=mark-packet chain=forward dst-address=VOIP new-packet-mark=\
    VOIP passthrough=yes
add action=mark-packet chain=prerouting comment=office-subnet dst-address-list=\
    !VIDEO new-packet-mark=Marketing passthrough=yes src-address-list=\
    ISP2
add action=mark-routing chain=prerouting comment=office-subnet dst-address-list=\
    !VIDEO new-routing-mark=Marketing packet-mark=Marketing passthrough=\
    yes src-address-list=ISP2
add action=mark-connection chain=prerouting comment=Raspberry connection-mark=\
    no-mark disabled=yes new-connection-mark=raspTemp passthrough=yes \
    src-address=192.168.50.144
add action=mark-routing chain=prerouting comment=Raspberry connection-mark=\
    raspTemp disabled=yes new-routing-mark=raspTemp passthrough=yes \
    src-address=192.168.50.144
add action=mark-connection chain=prerouting connection-mark=no-mark dst-port=53 \
    layer7-protocol=*4 new-connection-mark=block_connection passthrough=yes \
    protocol=udp
add action=mark-packet chain=prerouting connection-mark=block_connection \
    new-packet-mark=block_packet
add action=accept chain=forward connection-state=!new disabled=yes
add action=mark-connection chain=forward comment="mark ipsec connections" \
    disabled=yes ipsec-policy=out,ipsec new-connection-mark=ipsec passthrough=\
    yes
add action=mark-connection chain=forward comment="mark ipsec connections" \
    disabled=yes ipsec-policy=in,ipsec new-connection-mark=ipsec passthrough=\
    yes

any drop that i have is

 0 X  ;;; defconf: drop invalid
      chain=forward action=drop connection-state=invalid log=no log-prefix="" 

 1 X  ;;; defconf: drop invalid
      chain=input action=drop connection-state=invalid log=no log-prefix="" 

 2    ;;; @Dongla Server drop Locations
      chain=forward action=drop src-address=192.168.50.144 
      dst-address=10.0.0.0/8 log=no log-prefix="" 

 3 X  chain=forward action=drop protocol=icmp packet-size=200-65535 log=no 
      log-prefix="" 

 4    ;;; Drop Inv.
      chain=forward action=drop connection-state=invalid 

 5    ;;; Drop Inv.
      chain=input action=drop connection-state=invalid 

 6    ;;; IN-Phase_dropRAW
      chain=input action=drop src-address-list=@Servisi_Phase log=no 
      log-prefix="" 

 7    ;;; IN-Block shodan
      chain=input action=drop src-address-list=shodan 

 8    ;;; IN-Defend from pingers
      chain=input action=drop src-address-list=pingers log=no log-prefix=""

I have posted in thread tunnel troubleshot a log and ip ipsec installed-sa print results at the time tunnel dont work. But what is strange.
I have LAN subnet which are all created by MikroTik and anothers which are on the Cisco L3, maybe the drawing make you easier to understand.
So, subnets which are creating a tunnel is maked on mikrotik and on cisco. And someday one have a problem and not the another, and so on. Then, one subnet for example 192.168.50.0/24 dont see 10.11.0.0/16 even if status is established and have access to 10.10.0.0/16, and so on.
There are more ipsec and on the other side there is some directly connected subnets (not all are openvpn, some is directly connected to pfsense).

10.10.0.0/16 and 10.11.0.0/16 are remote locations not phisically in the building where is pfsense

sindy · Sun Nov 01, 2020 9:22 pm

Let's keep the threads (firewall and IPsec) separate.

Here, try to enable only the three mangle rules you've added, but keep those drop ones, which you've eventually added since the point in time when it was working, disabled, and tell me how it works.

tomislav91 · Mon Nov 02, 2020 11:22 am

Let's keep the threads (firewall and IPsec) separate.

Here, try to enable only the three mangle rules you've added, but keep those drop ones, which you've eventually added since the point in time when it was working, disabled, and tell me how it works.

i cant enable JUST those three in mangle, because here i mangle some subnet and force it to another ISP in routes and also using for VOIP, so i cant disable it.

sindy · Mon Nov 02, 2020 11:36 am

i cant enable JUST those three in mangle, because here i mangle some subnet and force it to another ISP in routes and also using for VOIP, so i cant disable it.

I had in mind "out of the rules added by my recommendation, enable only the three added to mangle, not the drop ones in filter". Of course all your pre-existing rules must stay in place.

Also, from your print of drop rules, it seemed to me that the drop one in forward in filter was not the very last one in that chain. The rules are matched in sequence, first towards last until first match to a rule providing a final verdict, in the chain chosen depending on packet type: incoming to router itself, i.e. no out-interface used / forwarded from interface to interface / outgoing from router itself, i.e. no in-interface used.

tomislav91 · Mon Nov 02, 2020 12:06 pm

i cant enable JUST those three in mangle, because here i mangle some subnet and force it to another ISP in routes and also using for VOIP, so i cant disable it.
I had in mind "out of the rules added by my recommendation, enable only the three added to mangle, not the drop ones in filter". Of course all your pre-existing rules must stay in place.

Also, from your print of drop rules, it seemed to me that the drop one in forward in filter was not the very last one in that chain. The rules are matched in sequence, first towards last until first match to a rule providing a final verdict, in the chain chosen depending on packet type: incoming to router itself, i.e. no out-interface used / forwarded from interface to interface / outgoing from router itself, i.e. no in-interface used.

it is how it was earlier, i just added that three in mangle if you think about these rules

add action=accept chain=forward connection-state=!new disabled=yes
add action=mark-connection chain=forward comment="mark ipsec connections" \
    disabled=yes ipsec-policy=out,ipsec new-connection-mark=ipsec passthrough=\
    yes
add action=mark-connection chain=forward comment="mark ipsec connections" \
    disabled=yes ipsec-policy=in,ipsec new-connection-mark=ipsec passthrough=\
    yes

and you saw drop rules in filter, dont sure what you point, i have the rule number four same as number one, and i will remove it.

   ;;; Drop Inv.
      chain=forward action=drop connection-state=invalid

sindy · Mon Nov 02, 2020 12:13 pm

OK, and with these three mangle rules enabled, everything works fine? What about the difference in CPU load when the "accept connection-state=!new" is enabled and when it is disabled, is there any?

tomislav91 · Mon Nov 02, 2020 1:13 pm

OK, and with these three mangle rules enabled, everything works fine? What about the difference in CPU load when the "accept connection-state=!new" is enabled and when it is disabled, is there any?

no no, it wasnt fine. I added this with

/ip firewall filter
add action=fasttrack-connection chain=forward connection-mark=!ipsec \
    connection-state=established,related disabled=yes
add action=accept chain=forward connection-state=established,related disabled=\
    yes
add action=drop chain=forward comment="defconf: drop invalid" connection-state=\
    invalid disabled=yes
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked disabled=yes
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid disabled=yes

and when i enable these rules (ofcourse with mangle rules) tunnel got into msg1 sent or just established and dont work.I posted that earlier

i added that firewall mangle rule before those ipsec.
I got aproblem that tunnels goes down, msg1 sent error and i must disable all that i newly created and restart peer and than tunnels go up.

sindy · Mon Nov 02, 2020 1:44 pm

That's magic.

First, the IPsec control packets and transport packets are handled by chains input and output. Payload packets from/to external devices, which get extracted from IPsec transport packets, and which are going to be encapsulated into IPsec transport packets, are handled by chain forward. So whatever you do in chain forward cannot cause the IPsec tunnels to get to "msg1 sent" state.

Second, out of the five rules you've listed above, for the two for chain input I can see no way how they could break anything, regardless where exactly you place them.

So enable only the "accept" one of them in chain input, make an export of the complete firewall filter after that, and if that breaks IPsec, post that export. If it doesn't break anything, enable also the "drop invalid" one in chain input, export the filter, and again if it breaks anything, post that export.

tomislav91 · Mon Nov 02, 2020 2:31 pm

That's magic.

First, the IPsec control packets and transport packets are handled by chains input and output. Payload packets from/to external devices, which get extracted from IPsec transport packets, and which are going to be encapsulated into IPsec transport packets, are handled by chain forward. So whatever you do in chain forward cannot cause the IPsec tunnels to get to "msg1 sent" state.

Second, out of the five rules you've listed above, for the two for chain input I can see no way how they could break anything, regardless where exactly you place them.

So enable only the "accept" one of them in chain input, make an export of the complete firewall filter after that, and if that breaks IPsec, post that export. If it doesn't break anything, enable also the "drop invalid" one in chain input, export the filter, and again if it breaks anything, post that export.

with the mangles rules for ipsec or without?

sindy · Mon Nov 02, 2020 2:34 pm

With. They are in forward, aren't they?

tomislav91 · Mon Nov 02, 2020 2:39 pm

With. They are in forward, aren't they?

yes, they are

add action=mark-connection chain=prerouting connection-mark=no-mark dst-port=53 \
    layer7-protocol=*4 new-connection-mark=block_connection passthrough=yes \
    protocol=udp
add action=mark-packet chain=prerouting connection-mark=block_connection \
    new-packet-mark=block_packet
add action=accept chain=forward connection-state=!new disabled=yes
add action=mark-connection chain=forward comment="mark ipsec connections" \
    disabled=yes ipsec-policy=out,ipsec new-connection-mark=ipsec passthrough=\
    yes
add action=mark-connection chain=forward comment="mark ipsec connections" \
    disabled=yes ipsec-policy=in,ipsec new-connection-mark=ipsec passthrough=\
    yes

but fasttrack connection for that is disabled, should i enable that also?

2020-11-02 13_38_39-Window.png

sindy · Mon Nov 02, 2020 2:48 pm

but fasttrack connection for that is disabled, should i enable that also?

No, don't enable the fasttrack connection. The steps need to be taken one by one to see where is the issue. So first you enable only the additional mangle rules and the additional rules in input in filter. If that works, we can proceed.

tomislav91 · Mon Nov 02, 2020 2:50 pm

but fasttrack connection for that is disabled, should i enable that also?
No, don't enable the fasttrack connection. The steps need to be taken one by one to see where is the issue. So first you enable only the additional mangle rules and the additional rules in input in filter. If that works, we can proceed.

i enable it. Waiting now, i will edit this post when finds out!

tomislav91 · Fri Nov 06, 2020 3:30 pm

I am getting cpu about 70%.
I notice that one bridge when disable cpu goes regular about 10%.

2020-11-06 14_28_09-Window.png

is it possibly because link is more than 100mb and pc are only 100mb and thats what suffocate a router cpu?

sindy · Fri Nov 06, 2020 7:16 pm

Što si htio da rekneš ovom rječenicom: "I notice that one bridge when disable cpu goes regular about 10%."?

tomislav91 · Sat Nov 07, 2020 12:32 am

Što si htio da rekneš ovom rječenicom: "I notice that one bridge when disable cpu goes regular about 10%."?

i have one bridge with ports and 200Mb+ bandwidth within.
when i disable it, cpu % is ok, there is no loop in network, log also dont write anything. can be problem within 100mb switches? i have 1gbit sw also but 100mb is present and they complain about slow network, and i notice in speed test there is no more download, 1 or 2 mb.
can that be bad for cpu intensity? i am planning to change them to gigabit, no problem but i am just wondering why cpu is that high when that bridge interface is enabled.
nadam se da je jasnije, ako ne eo i ovako. taj interfejs bridge je jedan deo zgrade i vuku oko 200mb,ali imamo i tih obicnih sviceva od 100mb
moze li to biti uzrok da skoci tako cpu load

sindy · Sat Nov 07, 2020 7:46 pm

Are we still talking about the machine where the IPsec tunnels are running? I'm asking because in its configuration export, I've found just an empty bridge with no ports at all.

It is theoretically possible that there is so much traffic towards the devices connected to the switches with only 100 Mbit/s ports that most of that traffic gets dropped due to insufficient bandwidth, and this causes retransmissions at source, so even more traffic.

tomislav91 · Sun Nov 08, 2020 4:48 pm

Are we still talking about the machine where the IPsec tunnels are running? I'm asking because in its configuration export, I've found just an empty bridge with no ports at all.

It is theoretically possible that there is so much traffic towards the devices connected to the switches with only 100 Mbit/s ports that most of that traffic gets dropped due to insufficient bandwidth, and this causes retransmissions at source, so even more traffic.

yes, in the picture i sent you earlier the subnet 192.168.90.0/24 which "lives" on CCR itselft there is a bridge with that subnet and also a tunnel with 10.10 and 10.11.
Another subnets are just created on cisco core (on mikrotik are only dhcp pool and networks for relay).
I will change that to make sure cpu is going lower. Also, maybe high % of cpu cause some tunnel to break.
When i get a problem with tunnels i will post you conf, as you said earlier in the post.

sindy · Sun Nov 08, 2020 5:47 pm

Then why doesn't the picture match the configuration? 192.168.90.0/24 is attached to ether3_HQCAM in that configuration; the bridge exists there but has no member ports and no IP configuration is attached to it. Are you constantly updating the configuration?

tomislav91 · Sun Nov 08, 2020 8:29 pm

Then why doesn't the picture match the configuration? 192.168.90.0/24 is attached to ether3_HQCAM in that configuration; the bridge exists there but has no member ports and no IP configuration is attached to it. Are you constantly updating the configuration?

Yeah, i am seeing that now in the my post. Somehow there is no bridge members.
I dont change anything. I just add something what I need, that is some firewall rules for blocking-allowing something, nothing else.

sindy · Sun Nov 08, 2020 8:52 pm

Officially, the 192.168.90.1/24 should be attached to bridgeVN rather than ether3_HQCAM. Practically I have never seen this wrong setup to cause any issues, and even the ROS upgrade script migrating configurations from old "master port" to current "bridge with hardware acceleration" doesn't move the IP configuration from the former master port to the bridge.

But I rarely see CPU loads above 20 %, so it might help something to do it properly.

Looking at half of the total load to be spent on "networking" and just a small bit on "bridging", I suspect (and I may easily be wrong) that this load comes from the processing of the interrupts asserted on arrivals and departures of Ethernet frames. So if it is related to the 100 Mbit/s speed on bridge ports, I expect it to be caused by retransmissions induced by high loss rate as suggested several posts above.

tomislav91 · Mon Nov 09, 2020 9:31 am

Officially, the 192.168.90.1/24 should be attached to bridgeVN rather than ether3_HQCAM. Practically I have never seen this wrong setup to cause any issues, and even the ROS upgrade script migrating configurations from old "master port" to current "bridge with hardware acceleration" doesn't move the IP configuration from the former master port to the bridge.

But I rarely see CPU loads above 20 %, so it might help something to do it properly.

Looking at half of the total load to be spent on "networking" and just a small bit on "bridging", I suspect (and I may easily be wrong) that this load comes from the processing of the interrupts asserted on arrivals and departures of Ethernet frames. So if it is related to the 100 Mbit/s speed on bridge ports, I expect it to be caused by retransmissions induced by high loss rate as suggested several posts above.

i change it now to bridge_VN port.
I will change my switches, and we will see after that.

tomislav91 · Fri Nov 13, 2020 11:35 pm

i changed switches, but no signifcally changes

2020-11-13 22_34_55-Window.png

tomislav91 · Sun Nov 15, 2020 4:49 pm

can you explain, in the bridge i can see that Tx (transmit=upload) is 246Mb and in queue for that subnet Download is 233Mbps.
Like is vice versa? Or I missunderstand that tx and rx? Also when try to speedtest from that 192.168.90.0 subnet upload is 0 (all switches changed)

2020-11-15 15_46_43-Window.png

2020-11-15 15_46_59-Window.png

sindy · Sun Nov 15, 2020 5:53 pm

You may misunderstand that tx and rx. In the list of interfaces, the bridge is an interface through which the router (L3) part of the software sends data to external devices connected to the physical ports included into the bridge, which is the download direction for those devices.

Or in generic case - download from internet to a device in LAN is seen as Rx on the WAN port and as Tx on the LAN port.

Most speedtests test each direction separately, so it should show almost 0 in upload while doing the download test and almost 0 in download while doing the upload test.

During normal operation, does any significant amount of L2 traffic run between different ports of the bridge, i.e. will the total amount of traffic running through the CCR change siginficantly if you connect one of the switches currently connected directly to the CCR to a free port on the other switch instead?

tomislav91 · Mon Nov 16, 2020 1:04 am

You may misunderstand that tx and rx. In the list of interfaces, the bridge is an interface through which the router (L3) part of the software sends data to external devices connected to the physical ports included into the bridge, which is the download direction for those devices.

Or in generic case - download from internet to a device in LAN is seen as Rx on the WAN port and as Tx on the LAN port.

Most speedtests test each direction separately, so it should show almost 0 in upload while doing the download test and almost 0 in download while doing the upload test.

During normal operation, does any significant amount of L2 traffic run between different ports of the bridge, i.e. will the total amount of traffic running through the CCR change siginficantly if you connect one of the switches currently connected directly to the CCR to a free port on the other switch instead?

no, that bridge is for video surveillance center. So there is no much l2 traffic in the bridge there. Funny thing is that when i disable that bridge or just pull out cables, cpu level is OK, about 10-20%.
But i cant figure out two things. First, why speed tests shows upload 0 (download goes under 100mb). Here are examples from different pcs.

viber_image_2020-11-15_10-22-40.jpg

viber_image_2020-11-15_10-40-23.jpg

sindy · Mon Nov 16, 2020 9:45 am

no, that bridge is for video surveillance center. So there is no much l2 traffic in the bridge there. Funny thing is that when i disable that bridge or just pull out cables, cpu level is OK, about 10-20%.

As you've mentioned video surveillance, two things come to my mind - the cameras may be sending multicast traffic, and/or they may be sending short packets, i.e. relatively low bit rate but high packet rate. The high CPU percentage spent on networking vs. almost none spent on bridging suggests rather the latter (high rate of small packets), but I have no experience with the TILE architecture so it is just an assumption.

So the next step is to run /interface monitor-traffic etherX,etherY,.., where etherX, etherY, ... is the list of actual member ports of the bridge, to show the current traffic every second, and /interface ethernet print stats where name~"etherX|etherY" which shows the counters just one-time, but the counters are more detailed - by frame size group, and also broadcast and multicast frames are counted separately.

With /interface ethernet print stats sent several times with a fixed time between the attempts, you can get the traffic volumes per second in each size/type group by subtraction and averaging, something like

/interface ethernet print stats file=ether-stats where name~"etherX|etherY" ; /delay 50s ; /interface ethernet print stats file=ether-stats append where name~"etherX|etherY" ; /delay 50s ; /interface ethernet print stats file=ether-stats append where name~"etherX|etherY"

But i cant figure out two things. First, why speed tests shows upload 0 (download goes under 100mb). Here are examples from different pcs.

These two PCs are in the same subnet?

atakacs · Mon Nov 16, 2020 10:30 am

Just wanted to say that I find this thread both fascinating and instructive :)

tomislav91 · Mon Nov 16, 2020 4:03 pm

no, that bridge is for video surveillance center. So there is no much l2 traffic in the bridge there. Funny thing is that when i disable that bridge or just pull out cables, cpu level is OK, about 10-20%.
As you've mentioned video surveillance, two things come to my mind - the cameras may be sending multicast traffic, and/or they may be sending short packets, i.e. relatively low bit rate but high packet rate. The high CPU percentage spent on networking vs. almost none spent on bridging suggests rather the latter (high rate of small packets), but I have no experience with the TILE architecture so it is just an assumption.

So the next step is to run /interface monitor-traffic etherX,etherY,.., where etherX, etherY, ... is the list of actual member ports of the bridge, to show the current traffic every second, and /interface ethernet print stats where name~"etherX|etherY" which shows the counters just one-time, but the counters are more detailed - by frame size group, and also broadcast and multicast frames are counted separately.

With /interface ethernet print stats sent several times with a fixed time between the attempts, you can get the traffic volumes per second in each size/type group by subtraction and averaging, something like

/interface ethernet print stats file=ether-stats where name~"etherX|etherY" ; /delay 50s ; /interface ethernet print stats file=ether-stats append where name~"etherX|etherY" ; /delay 50s ; /interface ethernet print stats file=ether-stats append where name~"etherX|etherY"

But i cant figure out two things. First, why speed tests shows upload 0 (download goes under 100mb). Here are examples from different pcs.
These two PCs are in the same subnet?

As you've mentioned video surveillance, two things come to my mind - the cameras may be sending multicast traffic

they are watching cameras from those tunnels and some mikrotik ip cloud IPs.
here you are ether-stats.txt and results of commands you asked.

ether-stats.txt

These two PCs are in the same subnet?

yeah, its one big office.

ccr1.png

ccr2.png

sindy · Tue Nov 17, 2020 12:01 am

Looking at the file, those about 250 Mbit/s sent out via ether3 are sent as 21000 packets/s in 1500-byte packets, hence I assume the cameras send the video using TCP. The ACK traffic in the opposite direction takes some 9000 packets/s in 64-byte frames (which is fine, TCP doesn't necessarily acknowledge every single packet received). So I assume that if you disconnect the switch from the port, this traffic stops being sent as the cameras stop getting the ACK, and hence the CPU load decreases.

What surprises me is that it worked at least somehow while a 100 Mbit/s switch was connected to a port which pushes 250 Mbit/s out when connected to a gigabit peer. Do the cameras adjust the bit rate to the available bandwidth (which they learn from the feedback in the ACK)?

I'd say you should use /interface ethernet monitor-traffic ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8 once with and without ether3 connected. The sum of Rx bytes across all ports should be roughly the same like the sum of Tx bytes (it will not be exactly the same, at least due to IPsec encapsulation and decapsulation). So you'll see the total traffic volume the router has to handle while the camera streams are getting through and while they are squelched as the recipient(s) do(es)n't ACK them.

tomislav91 · Tue Nov 17, 2020 11:16 am

Looking at the file, those about 250 Mbit/s sent out via ether3 are sent as 21000 packets/s in 1500-byte packets, hence I assume the cameras send the video using TCP. The ACK traffic in the opposite direction takes some 9000 packets/s in 64-byte frames (which is fine, TCP doesn't necessarily acknowledge every single packet received). So I assume that if you disconnect the switch from the port, this traffic stops being sent as the cameras stop getting the ACK, and hence the CPU load decreases.

What surprises me is that it worked at least somehow while a 100 Mbit/s switch was connected to a port which pushes 250 Mbit/s out when connected to a gigabit peer. Do the cameras adjust the bit rate to the available bandwidth (which they learn from the feedback in the ACK)?

I'd say you should use /interface ethernet monitor-traffic ether1,ether2,ether3,ether4,ether5,ether6,ether7,ether8 once with and without ether3 connected. The sum of Rx bytes across all ports should be roughly the same like the sum of Tx bytes (it will not be exactly the same, at least due to IPsec encapsulation and decapsulation). So you'll see the total traffic volume the router has to handle while the camera streams are getting through and while they are squelched as the recipient(s) do(es)n't ACK them.

there are too many cameras and i cant say if they properly configured with a bandwitdth in the remote location if thats you asking.
All routers on the locations are also mikrotik and send via tcp port to our center (via ipsec tunnel or via cloud ip)

here are result before/after ether3 and on the right is resource after removing ether3.

ccrPLUG.png

sindy · Tue Nov 17, 2020 2:40 pm

I don't get why you are so fond of images where text is much more useful for post-processing.

So roughly, when the DVR/monitor wall receiving the streams from the cameras is reachable (when the external switch is connected to ether3), the machine routes about 410 Mbit/s of traffic (and decrypts a good deal of it); when the video stream destination is unavailable, the total traffic volume is just 160 Mbit/s, so 2.5 times less.
In packets per second, it is 65 k packets/s when video destination is reachable and 25 k packets/s when it is not, so also about 2.5 times less.

So if everything was proportional, the CPU load should raise from 12 % to some 30 % when you connect the video destination(s). But it obviously isn't, so the question is what is so different about the video traffic that it causes so much extra load. Is the other (non-video) traffic coming IPsec-encrypted as well? Can you show the output of /tool profile in both states?

tomislav91 · Tue Nov 17, 2020 3:41 pm

I don't get why you are so fond of images where text is much more useful for post-processing.

So roughly, when the DVR/monitor wall receiving the streams from the cameras is reachable (when the external switch is connected to ether3), the machine routes about 410 Mbit/s of traffic (and decrypts a good deal of it); when the video stream destination is unavailable, the total traffic volume is just 160 Mbit/s, so 2.5 times less.
In packets per second, it is 65 k packets/s when video destination is reachable and 25 k packets/s when it is not, so also about 2.5 times less.

So if everything was proportional, the CPU load should raise from 12 % to some 30 % when you connect the video destination(s). But it obviously isn't, so the question is what is so different about the video traffic that it causes so much extra load. Is the other (non-video) traffic coming IPsec-encrypted as well? Can you show the output of /tool profile in both states?

Is the other (non-video) traffic coming IPsec-encrypted as well?

they are just in terms of ipsec watching cameras. Any other job is standard job as in any office. My first idea was that somewhere is a problem with network loop, but router in the logs tell me nothing in terms of loops.
But my technicians claims that there is no loops. So from router cable from ether3 comes into that office to gigabit switches (1 16port and 3 8 port).

I dont have complains about ipsec for now (or support dont tell me :D )

sindy · Tue Nov 17, 2020 5:21 pm

Is the other (non-video) traffic coming IPsec-encrypted as well?
they are just in terms of ipsec watching cameras. Any other job is standard job as in any office.

Sorry, I understand every single word but not the answer as a whole. So again - does any other traffic except the one from cameras need IPsec decryption at the CCR?

My first idea was that somewhere is a problem with network loop, but router in the logs tell me nothing in terms of loops.
But my technicians claims that there is no loops. So from router cable from ether3 comes into that office to gigabit switches (1 16port and 3 8 port).

If there was an L2 loop on the switches connected to ether3 and ether4, which are bridged together, the traffic volumes would be about the same on both. A routing loop is a different case, but for it to exist, there would have to exist a physical path from the switch connected to ether3 and something connected via ether1 or ether2. Related to this, I actually am surprised that part of the traffic from the cameras is coming via ether1 although the bulk of it is coming via ether2 - do you have any explanation of this?

I dont have complains about ipsec for now (or support dont tell me :D )

When asking about IPsec above, I had in mind that forwarding of a packet that arrives encrypted is a bit more CPU-intensive than forwarding a packet which doesn't require decryption.

tomislav91 · Tue Nov 17, 2020 6:53 pm

Is the other (non-video) traffic coming IPsec-encrypted as well?
they are just in terms of ipsec watching cameras. Any other job is standard job as in any office.
Sorry, I understand every single word but not the answer as a whole. So again - does any other traffic except the one from cameras need IPsec decryption at the CCR?

My first idea was that somewhere is a problem with network loop, but router in the logs tell me nothing in terms of loops.
But my technicians claims that there is no loops. So from router cable from ether3 comes into that office to gigabit switches (1 16port and 3 8 port).
If there was an L2 loop on the switches connected to ether3 and ether4, which are bridged together, the traffic volumes would be about the same on both. A routing loop is a different case, but for it to exist, there would have to exist a physical path from the switch connected to ether3 and something connected via ether1 or ether2. Related to this, I actually am surprised that part of the traffic from the cameras is coming via ether1 although the bulk of it is coming via ether2 - do you have any explanation of this?

I dont have complains about ipsec for now (or support dont tell me :D )
When asking about IPsec above, I had in mind that forwarding of a packet that arrives encrypted is a bit more CPU-intensive than forwarding a packet which doesn't require decryption.

Sorry, I understand every single word but not the answer as a whole. So again - does any other traffic except the one from cameras need IPsec  decryption at the CCR?

YES. Several things. 192.168.50.0/24 using ipsec for connect to remote routers 10.10, 10.11 and so on. Also for connecting to 192.168.200 (some inhouse services for company).
Only video surveillance (192.168.90.0) use ipsec for watching cameras.

Also there is a some other subnets (192.168.30.0,192.168.23.0,etc) and also some of them connect as subnet 50 for connecting to routers (or some kind of stuff from locations, internal things).

Not only 192.168.90 use ipsec, thats a point.
Only difference that 90 subnet is whole created on CCR, others mainly on cisco L3 switches (ccr is just a relay and pool created there).

do you have any explanation of this?

ether1 is WAN from all subnets, main uplink.
ether2 is also WAN and used primarly for 192.168.90 (so video surveillance) and for backup route if ether1 goes down.

add action=mark-packet chain=prerouting comment=VN-Subnet dst-address-list=!VIDEO new-packet-mark=Video_Surv passthrough=yes src-address-list=WAN2
add action=mark-routing chain=prerouting comment=VN-Subnet dst-address-list=!VIDEO new-routing-mark=Video_Surv packet-mark=Video_Surv passthrough=yes src-address-list=WAN2

and in WAN2 address list is 192.168.90.0/24

sindy · Tue Nov 17, 2020 10:06 pm

So the additional CPU spent on IPsec encryption and decryption per packet does not explain the non-linear growth of the CPU load as the video traffic is added to the mix. At this moment I've got no further ideas.

The product page provides no information regarding IPsec throughput of your CCR1009-8G-1S, but the CPU in it should be the same one like in the newer CCR1009-7G-1C-1S+, and for that one, the IPsec performance is shown and doesn't seem to be a limiting factor for your traffic volume. But there's a more interesting information in the product list, which is the fact that on your model, ether1 to ether4 are connected to a switch chip which is connected to the CPU using a single 1 Gbit/s lane. Since ether5 and ether6 appear to be unused in your configuration, I'd recommend to migrate ether1 configuration and cable to ether5 and ether2 configuration and cable to ether6 and see what happens. At least it won't be worse.

tomislav91 · Tue Nov 17, 2020 11:02 pm

So the additional CPU spent on IPsec encryption and decryption per packet does not explain the non-linear growth of the CPU load as the video traffic is added to the mix. At this moment I've got no further ideas.

The product page provides no information regarding IPsec throughput of your CCR1009-8G-1S, but the CPU in it should be the same one like in the newer CCR1009-7G-1C-1S+, and for that one, the IPsec performance is shown and doesn't seem to be a limiting factor for your traffic volume. But there's a more interesting information in the product list, which is the fact that on your model, ether1 to ether4 are connected to a switch chip which is connected to the CPU using a single 1 Gbit/s lane. Since ether5 and ether6 appear to be unused in your configuration, I'd recommend to migrate ether1 configuration and cable to ether5 and ether2 configuration and cable to ether6 and see what happens. At least it won't be worse.

i will do it. Get back with info!

tomislav91 · Wed Nov 18, 2020 8:09 pm

That's magic.

First, the IPsec control packets and transport packets are handled by chains input and output. Payload packets from/to external devices, which get extracted from IPsec transport packets, and which are going to be encapsulated into IPsec transport packets, are handled by chain forward. So whatever you do in chain forward cannot cause the IPsec tunnels to get to "msg1 sent" state.

Second, out of the five rules you've listed above, for the two for chain input I can see no way how they could break anything, regardless where exactly you place them.

So enable only the "accept" one of them in chain input, make an export of the complete firewall filter after that, and if that breaks IPsec, post that export. If it doesn't break anything, enable also the "drop invalid" one in chain input, export the filter, and again if it breaks anything, post that export.

just to have a word from this post, this is a drops now

Flags: X - disabled, I - invalid, D - dynamic 
 0 X  ;;; for test ipsec
      chain=input action=drop connection-state=invalid log=no log-prefix="" 

 1    ;;;  Server drop Locations
      chain=forward action=drop dst-address=10.0.0.0/8 src-address-list=!ALLOW-LOCATIONS log=no log-prefix="" 

 2    ;;; Drop Inv.
      chain=forward action=drop connection-state=invalid 

 3    ;;; IN-Phase3_dropRAW
      chain=input action=drop src-address-list=@Servisi_Faza3 log=no log-prefix="" 

 4    ;;; IN-block shodan
      chain=input action=drop src-address-list=shodan 

 5    ;;; IN-Defend from pingers
      chain=input action=drop src-address-list=pingeri log=no log-prefix=""

tomislav91 · Wed Nov 18, 2020 8:10 pm

That's magic.

First, the IPsec control packets and transport packets are handled by chains input and output. Payload packets from/to external devices, which get extracted from IPsec transport packets, and which are going to be encapsulated into IPsec transport packets, are handled by chain forward. So whatever you do in chain forward cannot cause the IPsec tunnels to get to "msg1 sent" state.

Second, out of the five rules you've listed above, for the two for chain input I can see no way how they could break anything, regardless where exactly you place them.

So enable only the "accept" one of them in chain input, make an export of the complete firewall filter after that, and if that breaks IPsec, post that export. If it doesn't break anything, enable also the "drop invalid" one in chain input, export the filter, and again if it breaks anything, post that export.

just to have a word from this post, this is a drops now, and seems ok in term of tunnels.

Flags: X - disabled, I - invalid, D - dynamic 
 0 X  ;;; for test ipsec
      chain=input action=drop connection-state=invalid log=no log-prefix="" 

 1    ;;;  Server drop Locations
      chain=forward action=drop dst-address=10.0.0.0/8 src-address-list=!ALLOW-LOCATIONS log=no log-prefix="" 

 2    ;;; Drop Inv.
      chain=forward action=drop connection-state=invalid 

 3    ;;; IN-Phase3_dropRAW
      chain=input action=drop src-address-list=@Servisi_Faza3 log=no log-prefix="" 

 4    ;;; IN-block shodan
      chain=input action=drop src-address-list=shodan 

 5    ;;; IN-Defend from pingers
      chain=input action=drop src-address-list=pingeri log=no log-prefix=""

tomislav91 · Wed Nov 18, 2020 8:24 pm

I don't get why you are so fond of images where text is much more useful for post-processing.

So roughly, when the DVR/monitor wall receiving the streams from the cameras is reachable (when the external switch is connected to ether3), the machine routes about 410 Mbit/s of traffic (and decrypts a good deal of it); when the video stream destination is unavailable, the total traffic volume is just 160 Mbit/s, so 2.5 times less.
In packets per second, it is 65 k packets/s when video destination is reachable and 25 k packets/s when it is not, so also about 2.5 times less.

So if everything was proportional, the CPU load should raise from 12 % to some 30 % when you connect the video destination(s). But it obviously isn't, so the question is what is so different about the video traffic that it causes so much extra load. Is the other (non-video) traffic coming IPsec-encrypted as well? Can you show the output of /tool profile in both states?

just for info, how you get 410 or 160mbit/s if total traffic is inclued? What you summarize here? ( in first case of 410Mbit i cant get that if i summarize ether1+ether2 tx and rx bits /s)

sindy · Wed Nov 18, 2020 9:04 pm

just for info, how you get 410 or 160mbit/s if total traffic is inclued? What you summarize here? ( in first case of 410Mbit i cant get that if i summarize ether1+ether2 tx and rx bits /s)

Assuming that the own traffic of the router (sent by the router itself or received by the router itself) is negligible, almost each packet which comes in through one interface must get out through another one. So the sum of the rx speeds on all interfaces is roughly equal to the sum of tx speeds on all interfaces, and either of them is the total throughput of the router. So in both cases I was summarizing the same direction of flows on all ports, ether1 to ether8. sfp1 was unused in the configuration you've posted and you haven't included it either, so I assume there is no traffic on it.

tomislav91 · Thu Nov 19, 2020 11:37 am

just for info, how you get 410 or 160mbit/s if total traffic is inclued? What you summarize here? ( in first case of 410Mbit i cant get that if i summarize ether1+ether2 tx and rx bits /s)
Assuming that the own traffic of the router (sent by the router itself or received by the router itself) is negligible, almost each packet which comes in through one interface must get out through another one. So the sum of the rx speeds on all interfaces is roughly equal to the sum of tx speeds on all interfaces, and either of them is the total throughput of the router. So in both cases I was summarizing the same direction of flows on all ports, ether1 to ether8. sfp1 was unused in the configuration you've posted and you haven't included it either, so I assume there is no traffic on it.

yeah now i know. Can you a just please more explanation. When you have traffic IN and OUT, OUT is what that port, or server connected to that port on switch,router sends. What is bandwitdth IN? What comes from OUTSIDE to that server, some queries or?

sindy · Thu Nov 19, 2020 1:10 pm

Rx is what comes from the wire to the router interface from an external device.
Tx is what the router sends to the wire.

tomislav91 · Fri Nov 20, 2020 9:43 pm

I migrate ether1 to ether5 and ether2 to ether6 and now seems ok. I will be waiting to Monday to check all, but it seems ok for now.
Thanks sindy.
Can we somehow improve firewall rules?

# nov/02/2020 13:32:24 by RouterOS 6.47.4

add action=fasttrack-connection chain=forward connection-mark=!ipsec \
    connection-state=established,related disabled=yes
add action=accept chain=forward connection-state=established,related \
    disabled=yes
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid disabled=yes
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid disabled=yes
add action=drop chain=forward comment="@Server drop Locations" \
    dst-address=10.0.0.0/8 src-address=192.168.50.144
add action=accept chain=forward dst-address=10.11.0.0/16 src-address=\
    192.168.30.0/24
add action=accept chain=input dst-address=10.11.0.0/16 src-address=\
    192.168.30.0/24
add action=drop chain=forward disabled=yes packet-size=200-65535 protocol=\
    icmp
add action=drop chain=forward comment="Drop Inv." connection-state=invalid
add action=accept chain=forward dst-address=10.98.0.0/24 src-address=\
    192.168.2.0/24
add action=accept chain=input in-interface=vrrp_WAN1 protocol=icmp
add action=accept chain=input comment=\
    "permit from local ip DNS reuqest udp 53" dst-port=53 protocol=udp \
    src-address-list=local_ip
add action=accept chain=forward dst-address=192.168.200.0/24 src-address=\
    192.168.23.0/24
add action=accept chain=forward dst-address=192.168.200.0/24 src-address=\
    172.16.2.0/24
add action=accept chain=forward dst-address=192.168.200.0/24 src-address=\
    192.168.30.0/24
add action=reject chain=input comment="Drop !DNS" dst-port=53 protocol=udp \
    reject-with=icmp-network-unreachable
add action=reject chain=input comment="Drop !DNS" dst-port=53 protocol=tcp \
    reject-with=icmp-network-unreachable
add action=accept chain=input comment=SSH dst-port=4777 protocol=tcp \
    src-address-list=HQ2Addresses
add action=accept chain=input comment=SSH dst-port=4777 protocol=tcp \
    src-address-list=HQ1Addresses
add action=accept chain=input comment=WinBox dst-port=8291 protocol=tcp \
    src-address-list=HQ1Addresses
add action=accept chain=input comment=WinBox dst-port=8291 protocol=tcp \
    src-address-list=HQ2Addresses
add action=accept chain=input comment="Allow ICMP" protocol=icmp \
    src-address-list=HQ2Addresses
add action=accept chain=input comment="Allow ICMP" protocol=icmp \
    src-address-list=HQ1Addresses
add action=drop chain=input comment="Drop Inv." connection-state=invalid
add action=reject chain=input comment="Drop !DNS" dst-port=53 protocol=tcp \
    reject-with=icmp-network-unreachable src-address-list=!DNS
add action=reject chain=input comment="Drop !DNS" dst-port=53 protocol=udp \
    reject-with=icmp-network-unreachable src-address-list=!DNS
add action=accept chain=output comment="OUT-Allow PMTUD" icmp-options=3:4 \
    protocol=icmp
add action=accept chain=input comment="IN-Allow PMTUD" icmp-options=3:4 \
    protocol=icmp
add action=accept chain=input comment="IN-ProAllow ping 1468b do 5 u sekundi" \
    limit=5,1:packet packet-size=1468 protocol=icmp
add action=add-src-to-address-list address-list=pingers address-list-timeout=\
    1d chain=input comment="IN-List ICMP which dont match criteria" \
    dst-address-list=!HQ1Addresses in-interface-list=WAN log-prefix=Ping@IN \
    protocol=icmp src-address-list=!HQ1Addresses
add action=add-src-to-address-list address-list=@Services_Phase1 \
    address-list-timeout=30m chain=input comment=IN-Services_Phase1 \
    dst-address-list=!HQ1Addresses dst-port=21,22,23,69,80,443,5060,8080 \
    in-interface-list=WAN protocol=tcp src-address-list=!HQ1Addresses
add action=add-src-to-address-list address-list=@Services_Phase1 \
    address-list-timeout=30m chain=input comment=IN-Services_Phase1-UDP \
    dst-address-list=!HQ1Addresses dst-port=21,22,23,69,80,443,5060,8080 \
    in-interface-list=WAN protocol=udp src-address-list=!HQ1Addresses
add action=add-src-to-address-list address-list=@Services_Phase2 \
    address-list-timeout=30m chain=input comment=IN-Services_Phase2 dst-port=\
    21,22,23,69,80,443,5060,8080 in-interface-list=WAN protocol=tcp \
    src-address-list=@Services_Phase1
add action=add-src-to-address-list address-list=@Services_Phase2 \
    address-list-timeout=30m chain=input comment=IN-Services_Phase2-UDP \
    dst-port=21,22,23,69,80,443,5060,8080 in-interface-list=WAN protocol=udp \
    src-address-list=@Services_Phase1
add action=add-src-to-address-list address-list=@Services_Phase3 \
    address-list-timeout=1w chain=input comment=IN-Services_Phase3 dst-port=\
    21,22,23,69,80,443,5060,8080 in-interface-list=WAN protocol=tcp \
    src-address-list=@Services_Phase2
add action=add-src-to-address-list address-list=@Services_Phase3 \
    address-list-timeout=1w chain=input comment=IN-Services_Phase3-UDP \
    dst-port=21,22,23,69,80,443,5060,8080 in-interface-list=WAN protocol=udp \
    src-address-list=@Services_Phase2
add action=drop chain=input comment=IN-Phase3_dropRAW src-address-list=\
    @Services_Phase3
add action=drop chain=input comment="IN-Block Shodan" src-address-list=\
    shodan
add action=drop chain=input comment="IN-Defend from pingers" \
    src-address-list=pingers
add action=accept chain=input dst-port=161 protocol=udp
add action=accept chain=forward dst-port=161 protocol=udp
add action=accept chain=output dst-port=161 protocol=udp

tomislav91 · Tue Dec 01, 2020 10:09 pm

That's magic.

First, the IPsec control packets and transport packets are handled by chains input and output. Payload packets from/to external devices, which get extracted from IPsec transport packets, and which are going to be encapsulated into IPsec transport packets, are handled by chain forward. So whatever you do in chain forward cannot cause the IPsec tunnels to get to "msg1 sent" state.

Second, out of the five rules you've listed above, for the two for chain input I can see no way how they could break anything, regardless where exactly you place them.

So enable only the "accept" one of them in chain input, make an export of the complete firewall filter after that, and if that breaks IPsec, post that export. If it doesn't break anything, enable also the "drop invalid" one in chain input, export the filter, and again if it breaks anything, post that export.

ipsec breaks.
its established, but there is no traffic, i cant communicate with that subnets.

ipsec.txt

exportWITHinputCHAIN.rsc

sindy · Tue Dec 01, 2020 10:48 pm

ipsec breaks.
its established, but there is no traffic, i cant communicate with that subnets.

Is this statement valid with the firewall filter rules exactly as in the posted .rsc file, including the disabled=yes at some of them?
What is the difference in configuration between the "working" and "non-working" states?

sindy · Tue Dec 01, 2020 11:16 pm

None of those firewall rules blocks IPsec as such, nor do they block traffic to/from the subnets reachable via IPsec (unless you've added them to the shodan or pingers or @Services_Phase3 address lists). In fact, the firewall rules block almost nothing.

In the printout of the security associations, I can see that traffic to/from the VoIP thing is bi-directional, but the traffic to/from pfsense is unidirectional from the HQ router to pfsense, except a single SA which shows packets received from pfsense side.

This confirms that firewall rules are unrelated to the issue. Whether the SA rekeying has failed on all SAs or whether there is another reason can only be found out if you can visualize the encryption keys used at pfsense side. What does /ip ipsec statistics print show in this state?

tomislav91 · Tue Dec 01, 2020 11:44 pm

None of those firewall rules blocks IPsec as such, nor do they block traffic to/from the subnets reachable via IPsec (unless you've added them to the shodan or pingers or @Services_Phase3 address lists). In fact, the firewall rules block almost nothing.

In the printout of the security associations, I can see that traffic to/from the VoIP thing is bi-directional, but the traffic to/from pfsense is unidirectional from the HQ router to pfsense, except a single SA which shows packets received from pfsense side.

This confirms that firewall rules are unrelated to the issue. Whether the SA rekeying has failed on all SAs or whether there is another reason can only be found out if you can visualize the encryption keys used at pfsense side. What does /ip ipsec statistics print show in this state?

 /ip ipsec statistics print
     out-state-expired: 157
 out-no-states: 67342
 in-template-mismatches: 16

I cant get back it to work. Some subnets are established, some no phase2 (when disable enable it came to established).But nor of them cant work.
Cant figure out what happend.

update:
when i try to change tunnel to go mkt-mkt, also established, but not working

Who is online