Community discussions

MikroTik App
 
sstory
just joined
Topic Author
Posts: 15
Joined: Thu Jul 01, 2010 11:03 pm

High number of established connections for one address

Mon Jan 28, 2019 4:55 pm

I am seeing pages of connections on the Firewall->Connections tab from a information TV in a lobby to rssweather.com on port 80 TCP. The state remains at established and the timeout apparently 24hours. Orig./Repl. Rate 0bps/0bps and Orig./Repl. Bytes 40 B/0B They seem to last until timed out.
Is there a way to allow the device to do whatever it needs in connnecting to that address but somehow close those connections after so long? Say close those that are over 5 minutes from those two addresses? Script if fine.
There is apparently nothing going on, on them, but they hang open. The Src address and Dst address and port are fixed.
I don't want to change the Tracking TCP Established timeout value because there may be other things users are a doing that might need to run and not get cut off--thought I can't think of what off the top of my head.
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1782
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: High number of established connections for one address

Mon Jan 28, 2019 11:52 pm

You could modify the settings under "/ip firewall connection tracking". Most connections will implement a keep-alive if they need it open for a long time.
I'm using est timeout of 30m with no issues
 
User avatar
pcunite
Forum Guru
Forum Guru
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: High number of established connections for one address

Tue Jan 29, 2019 4:43 am

You could modify the settings under "/ip firewall connection tracking". Most connections will implement a keep-alive if they need it open for a long time. I'm using est timeout of 30m with no issues

This works well. I use a 5 minute time and have no issues. There is almost nothing doing a keep-alive quicker than that.
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1782
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: High number of established connections for one address

Tue Jan 29, 2019 12:10 pm

This works well. I use a 5 minute time and have no issues. There is almost nothing doing a keep-alive quicker than that.
... slower ... ;-)
 
sstory
just joined
Topic Author
Posts: 15
Joined: Thu Jul 01, 2010 11:03 pm

Re: High number of established connections for one address

Tue Jan 29, 2019 8:03 pm

Thanks for the replies! So if I am downloading a file from the Internet, say CentOS 7 DVD or something larger that takes a few hours on slower Internet, would it kill that after 30 minutes? If not, maybe it would work.
 
Redmor
Member Candidate
Member Candidate
Posts: 256
Joined: Wed May 31, 2017 7:40 pm
Location: Italy

Re: High number of established connections for one address

Tue Jan 29, 2019 9:02 pm

Thanks for the replies! So if I am downloading a file from the Internet, say CentOS 7 DVD or something larger that takes a few hours on slower Internet, would it kill that after 30 minutes? If not, maybe it would work.
No TCP established timeout doesn't close connection every 30 minutes.
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1782
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: High number of established connections for one address

Wed Jan 30, 2019 12:02 am

As said I've a time out of 30m, yet I can maintain an established connection far longer, as long as there is data flowing over that connection.
It's not documented, but I think it's <timeout> from last packet seen...That is also corroborated by connection list in firewall -> see timeout field
 
sstory
just joined
Topic Author
Posts: 15
Joined: Thu Jul 01, 2010 11:03 pm

Re: High number of established connections for one address

Thu Jan 31, 2019 7:34 pm

OK. I have put 30min in place and am trying it out. Connections dropped from 2800 to 600 so that is a good thing. Only time will tell.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10221
Joined: Mon Jun 08, 2015 12:09 pm

Re: High number of established connections for one address

Thu Jan 31, 2019 8:27 pm

This works well. I use a 5 minute time and have no issues. There is almost nothing doing a keep-alive quicker than that.
It depends on your use of the network.
For the typical "browsing and mailing" use of internet it is absolutely fine, but users who use SSH to get a terminal on a remote system will hate you for it.
It is possible to set a keepalive in the server config but usually it isn't done by default.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11593
Joined: Thu Mar 03, 2016 10:23 pm

Re: High number of established connections for one address

Fri Feb 01, 2019 8:23 am

... but users who use SSH to get a terminal on a remote system will hate you for it.
It is possible to set a keepalive in the server config but usually it isn't done by default.

Ah, we ssh users hate us router/FW admins already :wink:

ssh -o ServerAliveInterval=30
(without any change on server side) will send keep-alive packet every 30 seconds. Keeps even the most zealous firewalls from breaking idle connection.
N.b.: the above command works with OpenSSH, I'm not sure about other implementations.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10221
Joined: Mon Jun 08, 2015 12:09 pm

Re: High number of established connections for one address

Fri Feb 01, 2019 11:16 am

ssh -o ServerAliveInterval=30
(without any change on server side) will send keep-alive packet every 30 seconds. Keeps even the most zealous firewalls from breaking idle connection.
N.b.: the above command works with OpenSSH, I'm not sure about other implementations.
I normally set ClientAliveInterval and ClientAliveCountMax in the server config so it also closes sessions when the client has become unreachable.
(so you don't get a pile of orphaned sessions when the network is not 100% reliable)

Of course not with a ridiculously low time/count like winbox uses...
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1782
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: High number of established connections for one address

Fri Feb 01, 2019 11:39 am

What is the use-case here of opening a ssh session and letting it sit for 30 minutes with NO data flowing in either direction? The established timeout is after last packet sent...

Edit: actually that would even be a security issue!
 
pe1chl
Forum Guru
Forum Guru
Posts: 10221
Joined: Mon Jun 08, 2015 12:09 pm

Re: High number of established connections for one address

Fri Feb 01, 2019 12:45 pm

AH... the "you should not want that" response. Typical BOFH sysadmin. I won't even reply to that.
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1782
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: High number of established connections for one address

Fri Feb 01, 2019 12:51 pm

You just did...

Anyway, why complain about imaginary issues if you can't substantiate them? Focusing on one specific element of answer to discredit the whole message??? That's just BOFH...

Ps: and lets keep it civil and on facts, so it's useful and productive.
 
R1CH
Forum Guru
Forum Guru
Posts: 1101
Joined: Sun Oct 01, 2006 11:44 pm

Re: High number of established connections for one address

Fri Feb 01, 2019 2:13 pm

TCP sessions should be able to last days without a router breaking them. I personally have many active SSH connections that sometimes remain idle for days until a log event is triggered or similar. I would hate to be a user of a network where such connections are broken after just 30 minutes. This only encourages protocols to use more and more keepalive, sending useless data back and forth for no other reason than to stop routers breaking things, and the end result is still the same (you have a bunch of established connections, except now they're using extra data).

Unless you're actually running out of resources, I would not worry about this at all. RouterOS supposedly lowers the limits by itself once it is close to running out of memory in any case.
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1782
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: High number of established connections for one address

Fri Feb 01, 2019 3:21 pm

In today's world of cloud, auto balancing & fail-over, auto upgrades and deployments, Agile release cycles of only few minutes, I don't think that expecting TCP to live "forever" is realistic.
Ssh is still a different beast, but we also do much more patching today than ever.

Since the infrastructure we operate on changes and the processes to control change are different, the usage scenarios are bound to change too.
 
User avatar
pcunite
Forum Guru
Forum Guru
Posts: 1345
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: High number of established connections for one address

Fri Feb 01, 2019 3:51 pm

TCP sessions should be able to last days without a router breaking them.

There is no way I'm going to hold a connection open for that long. Sorry. If you're alive, prove it. A tiny packet every 4m 30secs is not going to break the internet.
 
R1CH
Forum Guru
Forum Guru
Posts: 1101
Joined: Sun Oct 01, 2006 11:44 pm

Re: High number of established connections for one address

Fri Feb 01, 2019 4:57 pm

TCP session state is based on the endpoints, as long as you pass packets back and forth correctly the session will be fine, there's no state necessary on the router. If you actively break this process by introducing NAT then you should accept that it's your responsibility not to break things for the endpoints. I've seen far too many broken NAT setups in the wild, and bad advice like reducing the timeout to 5 minutes (!) isn't helping things. Again, unless you're actually running into resource exhaustion, you shouldn't need to mess with these timeouts. A big list of connections in your winbox does not indicate a problem.

(On a side note, the current RouterOS implementation of TCP NAT is bugged and also requires the unacked timeout to be increased for proper operation of long-lived sessions)
 
pe1chl
Forum Guru
Forum Guru
Posts: 10221
Joined: Mon Jun 08, 2015 12:09 pm

Re: High number of established connections for one address

Fri Feb 01, 2019 6:23 pm

It is not strictly related to NAT, the use of a connection-tracking firewall is enough to cause those issues, even in a network without NAT.
For example, when you make a "fault tolerant" network by installing several routers with different redundant connections, e.g. using BGP or OSPF to autoroute around outages, and you have connection-tracking firewalls on them, the existing connections will fail due to mismatch with the tracked connections.
(or there even will be problems establishing connections when the routes are not symmetrical)
 
nostromog
Member Candidate
Member Candidate
Posts: 226
Joined: Wed Jul 18, 2018 3:39 pm

Re: High number of established connections for one address

Sat Feb 02, 2019 4:44 am

What is the use-case here of opening a ssh session and letting it sit for 30 minutes with NO data flowing in either direction? The established timeout is after last packet sent...

Edit: actually that would even be a security issue!
My use cases for this are:
  • the boss interrupting me for half an hour in the middle of a long task. By far the most common.
  • monitoring of a log that has little activity. So I do a less xxx ->F or a tail -F xxx and wait for something nasty to happen, looking into it from time to time in the hope to react on time to something bad.
  • Slow ssh session establishment due to GSS API timeouts or other flukes... and I have a standby session to be ready for intervention. Not so common.
My problem typically is not so much TCP connection tracking as VPN idle timeout, that cuts the connection and then gives me a new IP when I reconnect, thus killing all my TCP sessions. Against this byobu/screen/tmux are usually your friends...
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1782
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: High number of established connections for one address

Sat Feb 02, 2019 3:11 pm

you can work around (some of) these issues with "screen" https://www.rackaid.com/blog/linux-scre ... nd-how-to/

(and I don't mean the boss-issue ;-) )
 
dmitris
Member Candidate
Member Candidate
Posts: 127
Joined: Mon Oct 09, 2017 1:08 pm

Re: High number of established connections for one address

Wed Apr 29, 2020 11:44 am

Hello, i would like to extend this topic further, i have similar situation where lots of connections are established toward my client with 0/0 orig rate and bytes. I see that these connections are established backward only when client established connection to some https server. How i can filter such behavior and not brake client connection?
Any ideas why it's happening?
Personally i suppose, that this is some malicious behavior on client side and i want hold this number of unneeded connections low as possible on my router.
fw1.PNG
You do not have the required permissions to view the files attached to this post.
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: High number of established connections for one address

Wed Apr 29, 2020 2:04 pm

Hello, i would like to extend this topic further, i have similar situation where lots of connections are established toward my client with 0/0 orig rate and bytes. I see that these connections are established backward only when client established connection to some https server. How i can filter such behavior and not brake client connection?
Any ideas why it's happening?
Personally i suppose, that this is some malicious behavior on client side and i want hold this number of unneeded connections low as possible on my router.
fw1.PNG
I have to start from quite a faraway point. One would expect that a firewall would only allow a TCP connection to establish when it receives a SYN packet from a client towards the server. But this requires a bit more resources to analyse the packets, so the connection tracking module allows to reduce the load by reducing the requirements, through setting /ip ipsec connection tracking loose-tcp-tracking to yes. And this setting is a default one, and for years it even was not available as a configuration item. With this setting, any TCP packet, even a mid-connection one, will create a tracked connection in the firewall.

Now you can see in your table that the source port is 443 (https), which is not the usual way how TCP connections are established - normally, the TCP server listens at a given port (443), and TCP clients use arbitrary ports above 10000 (depends on OS and settings), so the first packet's source port is, say, 54321 and destination port is 443.

But if someone else sends a SYN packet to 2.18.76.90:443 from the public IP address of your client, the server responds to the source address and port of that request, so the response lands at your client's WAN interface. With loose-tcp-tracking set to no, such packets will be dropped (because they are SYN,ACK ones); with loose-tcp-tracking set to the (default)yes, it creates a connection with a 5-minute lifetime.

So one explanation is that someone is running a SYN-flood attack to 2.18.76.90:443, spoofing the address of your client, and your firewall creates tracked connections due to this.

Setting loose-tcp-tracking to no should hide this issue from you. Even if your client doesn't need to connect to 2.18.76.90:443, the worst that can happen is that the admin of 2.18.76.90:443 will send your client's address to some list of spammers so the client will be denied access to other sites as well.

Another explanation is that your client actually does connect to 2.18.76.90:443, but the server doesn't respond, so the connection times out on both the client PC and the firewall, and when the response comes from the server, the client PC already ignores it.
 
msatter
Forum Guru
Forum Guru
Posts: 2912
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: High number of established connections for one address

Wed Apr 29, 2020 2:35 pm

Hello, i would like to extend this topic further, i have similar situation where lots of connections are established toward my client with 0/0 orig rate and bytes. I see that these connections are established backward only when client established connection to some https server. How i can filter such behavior and not brake client connection?
Any ideas why it's happening?
Personally i suppose, that this is some malicious behavior on client side and i want hold this number of unneeded connections low as possible on my router.

If nothing helps then you could use a scheduled script every one minute and one second like this to remove those stale connections:
/ip firewall connection remove [find where !seen-reply timeout>"30s" protocol=tcp src-address~":443"];
 
dmitris
Member Candidate
Member Candidate
Posts: 127
Joined: Mon Oct 09, 2017 1:08 pm

Re: High number of established connections for one address

Wed Apr 29, 2020 4:44 pm

I have to start from quite a faraway point. One would expect that a firewall would only allow a TCP connection to establish when it receives a SYN packet from a client towards the server. But this requires a bit more resources to analyse the packets, so the connection tracking module allows to reduce the load by reducing the requirements, through setting /ip ipsec connection tracking loose-tcp-tracking to yes. And this setting is a default one, and for years it even was not available as a configuration item. With this setting, any TCP packet, even a mid-connection one, will create a tracked connection in the firewall.

Now you can see in your table that the source port is 443 (https), which is not the usual way how TCP connections are established - normally, the TCP server listens at a given port (443), and TCP clients use arbitrary ports above 10000 (depends on OS and settings), so the first packet's source port is, say, 54321 and destination port is 443.

But if someone else sends a SYN packet to 2.18.76.90:443 from the public IP address of your client, the server responds to the source address and port of that request, so the response lands at your client's WAN interface. With loose-tcp-tracking set to no, such packets will be dropped (because they are SYN,ACK ones); with loose-tcp-tracking set to the (default)yes, it creates a connection with a 5-minute lifetime.

So one explanation is that someone is running a SYN-flood attack to 2.18.76.90:443, spoofing the address of your client, and your firewall creates tracked connections due to this.

Setting loose-tcp-tracking to no should hide this issue from you. Even if your client doesn't need to connect to 2.18.76.90:443, the worst that can happen is that the admin of 2.18.76.90:443 will send your client's address to some list of spammers so the client will be denied access to other sites as well.

Another explanation is that your client actually does connect to 2.18.76.90:443, but the server doesn't respond, so the connection times out on both the client PC and the firewall, and when the response comes from the server, the client PC already ignores it.
Thank you Sindy, it's very good and clear explanation, i will check what's going on with netflow records.

If nothing helps then you could use a scheduled script every one minute and one second like this to remove those stale connections:
/ip firewall connection remove [find where !seen-reply timeout>"30s" protocol=tcp src-address~":443"];
Thank you, for handy work around!
 
tippenring
Member
Member
Posts: 304
Joined: Thu Oct 02, 2014 8:54 pm
Location: St Louis MO
Contact:

Re: High number of established connections for one address

Wed Apr 29, 2020 5:14 pm

Hello, i would like to extend this topic further, i have similar situation where lots of connections are established toward my client with 0/0 orig rate and bytes. I see that these connections are established backward only when client established connection to some https server. How i can filter such behavior and not brake client connection?
Any ideas why it's happening?
Personally i suppose, that this is some malicious behavior on client side and i want hold this number of unneeded connections low as possible on my router.
fw1.PNG
Another possibility besides the spoofed packet flooding DDoS theory is asynchronous routing. If you have more than one path to the server at 2.18.76.90:443 (presumably on the internet), the client SYN packet may be following that path, and the return SYN/ACK traffic is arriving at the router where you are seeing what appears to be unsolicited SYN/ACKs.

If you only have a single path to the internet, then this isn't possible however. In that case, the source IP spoofing/DDoS theory is probably correct.
 
jo2jo
Forum Guru
Forum Guru
Posts: 1003
Joined: Fri May 26, 2006 1:25 am

Re: High number of established connections for one address

Sun Oct 11, 2020 8:16 pm

frequently when i have seen this in nat tables (from both external IPs and internal IPs/subnets being routed internally) , its often the results of a prior run, port scan (ie nmap).

FWIW- i agree with the position that as network admins, our settings/rules should attempt to do as little damage as possible to our clients, whatever their use cases may be- (ie setting a 5m or 30m TCP est. timeout , which may break long idle ssh sessions , is NOT a good setting to run). If you are running out of rOS resources (memory, in this case) than that is a different issue, that you should address by upgraded hardware first, or as a last resort decreasing the TCP established. timeout.

another (more controversial) example- severely limiting torrenting bw , vs blocking it entirely. (i prefer to limit it as much as i can, vs fully block it)
the fewer customer calls/complaints, the better.

that is my opinion on this topic.
additionally, i too *always* have 10+ ssh sessions open, many of which are idle for days/weeks (both to internally routed services, such as VMs, and external services such as colo'd servers/vms/network HW). Its a convenience thing.
tks

Who is online

Users browsing this forum: courtois, deatras and 104 guests