Community discussions

 
sstory
just joined
Topic Author
Posts: 15
Joined: Thu Jul 01, 2010 11:03 pm

High number of established connections for one address

Mon Jan 28, 2019 4:55 pm

I am seeing pages of connections on the Firewall->Connections tab from a information TV in a lobby to rssweather.com on port 80 TCP. The state remains at established and the timeout apparently 24hours. Orig./Repl. Rate 0bps/0bps and Orig./Repl. Bytes 40 B/0B They seem to last until timed out.
Is there a way to allow the device to do whatever it needs in connnecting to that address but somehow close those connections after so long? Say close those that are over 5 minutes from those two addresses? Script if fine.
There is apparently nothing going on, on them, but they hang open. The Src address and Dst address and port are fixed.
I don't want to change the Tracking TCP Established timeout value because there may be other things users are a doing that might need to run and not get cut off--thought I can't think of what off the top of my head.
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1664
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: High number of established connections for one address

Mon Jan 28, 2019 11:52 pm

You could modify the settings under "/ip firewall connection tracking". Most connections will implement a keep-alive if they need it open for a long time.
I'm using est timeout of 30m with no issues
 
User avatar
pcunite
Forum Veteran
Forum Veteran
Posts: 945
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: High number of established connections for one address

Tue Jan 29, 2019 4:43 am

You could modify the settings under "/ip firewall connection tracking". Most connections will implement a keep-alive if they need it open for a long time. I'm using est timeout of 30m with no issues

This works well. I use a 5 minute time and have no issues. There is almost nothing doing a keep-alive quicker than that.
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1664
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: High number of established connections for one address

Tue Jan 29, 2019 12:10 pm

This works well. I use a 5 minute time and have no issues. There is almost nothing doing a keep-alive quicker than that.
... slower ... ;-)
 
sstory
just joined
Topic Author
Posts: 15
Joined: Thu Jul 01, 2010 11:03 pm

Re: High number of established connections for one address

Tue Jan 29, 2019 8:03 pm

Thanks for the replies! So if I am downloading a file from the Internet, say CentOS 7 DVD or something larger that takes a few hours on slower Internet, would it kill that after 30 minutes? If not, maybe it would work.
 
Redmor
Member Candidate
Member Candidate
Posts: 248
Joined: Wed May 31, 2017 7:40 pm
Location: Italy

Re: High number of established connections for one address

Tue Jan 29, 2019 9:02 pm

Thanks for the replies! So if I am downloading a file from the Internet, say CentOS 7 DVD or something larger that takes a few hours on slower Internet, would it kill that after 30 minutes? If not, maybe it would work.
No TCP established timeout doesn't close connection every 30 minutes.
ImageImage
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1664
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: High number of established connections for one address

Wed Jan 30, 2019 12:02 am

As said I've a time out of 30m, yet I can maintain an established connection far longer, as long as there is data flowing over that connection.
It's not documented, but I think it's <timeout> from last packet seen...That is also corroborated by connection list in firewall -> see timeout field
 
sstory
just joined
Topic Author
Posts: 15
Joined: Thu Jul 01, 2010 11:03 pm

Re: High number of established connections for one address

Thu Jan 31, 2019 7:34 pm

OK. I have put 30min in place and am trying it out. Connections dropped from 2800 to 600 so that is a good thing. Only time will tell.
 
pe1chl
Forum Guru
Forum Guru
Posts: 5523
Joined: Mon Jun 08, 2015 12:09 pm

Re: High number of established connections for one address

Thu Jan 31, 2019 8:27 pm

This works well. I use a 5 minute time and have no issues. There is almost nothing doing a keep-alive quicker than that.
It depends on your use of the network.
For the typical "browsing and mailing" use of internet it is absolutely fine, but users who use SSH to get a terminal on a remote system will hate you for it.
It is possible to set a keepalive in the server config but usually it isn't done by default.
 
mkx
Forum Guru
Forum Guru
Posts: 2570
Joined: Thu Mar 03, 2016 10:23 pm

Re: High number of established connections for one address

Fri Feb 01, 2019 8:23 am

... but users who use SSH to get a terminal on a remote system will hate you for it.
It is possible to set a keepalive in the server config but usually it isn't done by default.

Ah, we ssh users hate us router/FW admins already :wink:

ssh -o ServerAliveInterval=30
(without any change on server side) will send keep-alive packet every 30 seconds. Keeps even the most zealous firewalls from breaking idle connection.
N.b.: the above command works with OpenSSH, I'm not sure about other implementations.
BR,
Metod
 
pe1chl
Forum Guru
Forum Guru
Posts: 5523
Joined: Mon Jun 08, 2015 12:09 pm

Re: High number of established connections for one address

Fri Feb 01, 2019 11:16 am

ssh -o ServerAliveInterval=30
(without any change on server side) will send keep-alive packet every 30 seconds. Keeps even the most zealous firewalls from breaking idle connection.
N.b.: the above command works with OpenSSH, I'm not sure about other implementations.
I normally set ClientAliveInterval and ClientAliveCountMax in the server config so it also closes sessions when the client has become unreachable.
(so you don't get a pile of orphaned sessions when the network is not 100% reliable)

Of course not with a ridiculously low time/count like winbox uses...
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1664
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: High number of established connections for one address

Fri Feb 01, 2019 11:39 am

What is the use-case here of opening a ssh session and letting it sit for 30 minutes with NO data flowing in either direction? The established timeout is after last packet sent...

Edit: actually that would even be a security issue!
 
pe1chl
Forum Guru
Forum Guru
Posts: 5523
Joined: Mon Jun 08, 2015 12:09 pm

Re: High number of established connections for one address

Fri Feb 01, 2019 12:45 pm

AH... the "you should not want that" response. Typical BOFH sysadmin. I won't even reply to that.
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1664
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: High number of established connections for one address

Fri Feb 01, 2019 12:51 pm

You just did...

Anyway, why complain about imaginary issues if you can't substantiate them? Focusing on one specific element of answer to discredit the whole message??? That's just BOFH...

Ps: and lets keep it civil and on facts, so it's useful and productive.
 
R1CH
Forum Veteran
Forum Veteran
Posts: 876
Joined: Sun Oct 01, 2006 11:44 pm

Re: High number of established connections for one address

Fri Feb 01, 2019 2:13 pm

TCP sessions should be able to last days without a router breaking them. I personally have many active SSH connections that sometimes remain idle for days until a log event is triggered or similar. I would hate to be a user of a network where such connections are broken after just 30 minutes. This only encourages protocols to use more and more keepalive, sending useless data back and forth for no other reason than to stop routers breaking things, and the end result is still the same (you have a bunch of established connections, except now they're using extra data).

Unless you're actually running out of resources, I would not worry about this at all. RouterOS supposedly lowers the limits by itself once it is close to running out of memory in any case.
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1664
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: High number of established connections for one address

Fri Feb 01, 2019 3:21 pm

In today's world of cloud, auto balancing & fail-over, auto upgrades and deployments, Agile release cycles of only few minutes, I don't think that expecting TCP to live "forever" is realistic.
Ssh is still a different beast, but we also do much more patching today than ever.

Since the infrastructure we operate on changes and the processes to control change are different, the usage scenarios are bound to change too.
 
User avatar
pcunite
Forum Veteran
Forum Veteran
Posts: 945
Joined: Sat May 25, 2013 5:13 am
Location: USA

Re: High number of established connections for one address

Fri Feb 01, 2019 3:51 pm

TCP sessions should be able to last days without a router breaking them.

There is no way I'm going to hold a connection open for that long. Sorry. If you're alive, prove it. A tiny packet every 4m 30secs is not going to break the internet.
 
R1CH
Forum Veteran
Forum Veteran
Posts: 876
Joined: Sun Oct 01, 2006 11:44 pm

Re: High number of established connections for one address

Fri Feb 01, 2019 4:57 pm

TCP session state is based on the endpoints, as long as you pass packets back and forth correctly the session will be fine, there's no state necessary on the router. If you actively break this process by introducing NAT then you should accept that it's your responsibility not to break things for the endpoints. I've seen far too many broken NAT setups in the wild, and bad advice like reducing the timeout to 5 minutes (!) isn't helping things. Again, unless you're actually running into resource exhaustion, you shouldn't need to mess with these timeouts. A big list of connections in your winbox does not indicate a problem.

(On a side note, the current RouterOS implementation of TCP NAT is bugged and also requires the unacked timeout to be increased for proper operation of long-lived sessions)
 
pe1chl
Forum Guru
Forum Guru
Posts: 5523
Joined: Mon Jun 08, 2015 12:09 pm

Re: High number of established connections for one address

Fri Feb 01, 2019 6:23 pm

It is not strictly related to NAT, the use of a connection-tracking firewall is enough to cause those issues, even in a network without NAT.
For example, when you make a "fault tolerant" network by installing several routers with different redundant connections, e.g. using BGP or OSPF to autoroute around outages, and you have connection-tracking firewalls on them, the existing connections will fail due to mismatch with the tracked connections.
(or there even will be problems establishing connections when the routes are not symmetrical)
 
nostromog
Member Candidate
Member Candidate
Posts: 142
Joined: Wed Jul 18, 2018 3:39 pm

Re: High number of established connections for one address

Sat Feb 02, 2019 4:44 am

What is the use-case here of opening a ssh session and letting it sit for 30 minutes with NO data flowing in either direction? The established timeout is after last packet sent...

Edit: actually that would even be a security issue!
My use cases for this are:
  • the boss interrupting me for half an hour in the middle of a long task. By far the most common.
  • monitoring of a log that has little activity. So I do a less xxx ->F or a tail -F xxx and wait for something nasty to happen, looking into it from time to time in the hope to react on time to something bad.
  • Slow ssh session establishment due to GSS API timeouts or other flukes... and I have a standby session to be ready for intervention. Not so common.
My problem typically is not so much TCP connection tracking as VPN idle timeout, that cuts the connection and then gives me a new IP when I reconnect, thus killing all my TCP sessions. Against this byobu/screen/tmux are usually your friends...
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1664
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: High number of established connections for one address

Sat Feb 02, 2019 3:11 pm

you can work around (some of) these issues with "screen" https://www.rackaid.com/blog/linux-scre ... nd-how-to/

(and I don't mean the boss-issue ;-) )

Who is online

Users browsing this forum: Bing [Bot] and 73 guests