Community discussions

MikroTik App
 
nexlevel
just joined
Topic Author
Posts: 5
Joined: Thu Jul 22, 2021 11:51 pm

TCP session reset

Mon Dec 27, 2021 6:29 pm

I have a tcp connection that tears itself down and reconnects constantly and I can't quite comprehend the pcaps enough to fully understand where the disconnect is. It is sip and the sip helper is turned off. The bizarre thing is that if I double nat the connection with another router behind the MT, it works fine. I did try and use srcnat instead of masquerade with no change. Here is a sample from the pcap. Everything is fine until that Encrypted Alert and then both server and client fin, ack and then the server sends the resets. Then they reconnect fine and this process repeats. Any tcp masters out there that can decipher this?

1524 250.686444 209.217.207.40 10.170.1.113 TCP 54 5061 → 12498 [ACK] Seq=33316 Ack=27702 Win=110672 Len=0
1525 250.691125 209.217.207.40 10.170.1.113 TLSv1.2 602 Application Data
1526 250.694008 10.170.1.113 209.217.207.40 TLSv1.2 972 Application Data
1527 250.694032 10.170.1.113 209.217.207.40 TCP 972 [TCP Retransmission] 12498 → 5061 [PSH, ACK] Seq=27702 Ack=33864 Win=118040 Len=918
1528 250.754276 209.217.207.40 10.170.1.113 TLSv1.2 486 Application Data
1529 250.796033 10.170.1.113 209.217.207.40 TCP 60 12498 → 5061 [ACK] Seq=28620 Ack=34296 Win=118040 Len=0
1530 250.796049 10.170.1.113 209.217.207.40 TCP 60 [TCP Dup ACK 1529#1] 12498 → 5061 [ACK] Seq=28620 Ack=34296 Win=118040 Len=0
1531 251.151564 10.170.1.113 209.217.207.40 TLSv1.2 85 Encrypted Alert
1532 251.151579 10.170.1.113 209.217.207.40 TCP 85 [TCP Retransmission] 12498 → 5061 [PSH, ACK] Seq=28620 Ack=34296 Win=118040 Len=31
1533 251.153517 209.217.207.40 10.170.1.113 TCP 54 5061 → 12498 [FIN, ACK] Seq=34296 Ack=28620 Win=112508 Len=0
1534 251.154151 10.170.1.113 209.217.207.40 TCP 60 12498 → 5061 [ACK] Seq=28651 Ack=34297 Win=118040 Len=0
1535 251.154166 10.170.1.113 209.217.207.40 TCP 60 [TCP Dup ACK 1534#1] 12498 → 5061 [ACK] Seq=28651 Ack=34297 Win=118040 Len=0
1536 251.154253 10.170.1.113 209.217.207.40 TCP 60 12498 → 5061 [FIN, ACK] Seq=28651 Ack=34297 Win=118040 Len=0
1537 251.154261 10.170.1.113 209.217.207.40 TCP 60 [TCP Out-Of-Order] 12498 → 5061 [FIN, ACK] Seq=28651 Ack=34297 Win=118040 Len=0
1538 251.203992 209.217.207.40 10.170.1.113 TCP 54 5061 → 12498 [RST] Seq=34297 Win=0 Len=0
1539 251.204369 209.217.207.40 10.170.1.113 TCP 54 5061 → 12498 [RST] Seq=34297 Win=0 Len=0
1540 252.134627 10.170.1.113 8.8.8.8 DNS 90 Standard query 0x9ee4 A sbc-sip-tls-cust.northland.net
1541 252.134642 10.170.1.113 8.8.8.8 DNS 90 Standard query 0x9ee4 A sbc-sip-tls-cust.northland.net
1542 252.202503 8.8.8.8 10.170.1.113 DNS 106 Standard query response 0x9ee4 A sbc-sip-tls-cust.northland.net A 209.217.207.40
1543 252.203832 10.170.1.113 209.217.207.40 TCP 66 12207 → 5061 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=8
1544 252.203851 10.170.1.113 209.217.207.40 TCP 66 [TCP Out-Of-Order] [TCP Port numbers reused] 12207 → 5061 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=8
1545 252.250501 209.217.207.40 10.170.1.113 TCP 66 5061 → 12207 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=4
1546 252.251172 10.170.1.113 209.217.207.40 TCP 60 12207 → 5061 [ACK] Seq=1 Ack=1 Win=29200 Len=0

CRS109 - 6.49.2
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: TCP session reset

Mon Dec 27, 2021 11:04 pm

You don't need a TCP master - the TCP does what the applications ask it for. And the application here is the TLS - the server dislikes something in the initial exchange with your client, and sends you the FIN. But maybe the reason is the Encrypted Alert your client sends to the server.

The double NAT affecting that behaviour is interesting. Is 10.170.1.113 the WAN IP of the Mikrotik or the IP of the phone/PBX?

Replacing action=masquerade by action=src-nat effectively doesn't change anything except that you can control what the new IP address will be (masquerade chooses it automatically) and that the NAT connection is automatically removed if the reply-dst address of that connection disappears or changes, which is not your case.
 
nexlevel
just joined
Topic Author
Posts: 5
Joined: Thu Jul 22, 2021 11:51 pm

Re: TCP session reset

Mon Dec 27, 2021 11:18 pm

Hi sindy. Thank you for the reply. The encrypted alert may have some more information. I'll see if I can get to the logs on the client 10.170.1.113 to see if I can uncover that alert. The client 10.170.1.113 is plugged directly into a MT bridge port. I then have another router (non MT) plugged into another bridge port and more clients behind that. Those clients all connect to the server 209.217.207.40 just fine. Is there any sort of services or features on the MT that would make it manipulate or change any behavior for TLS applications?
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: TCP session reset

Tue Dec 28, 2021 12:32 am

First, compare apples to apples. You cannot say that it works with double NAT if you don't try with the same client behind the second NAT.

The encrypted alert is encrypted, so if you can see it's actual contents somewhere, it is the log of the client, not the pcap. I'd expect the client to have a problem to recognize the certificate of the server.

If there was a bug in Mikrotik, it could malform the contents of the TCP exchange, but not modify the payload in such a way that it would still make enough sense for both the client and the server to proceed with the TLS negotiation to some extent. And there is no intentional tampering with TLS contents, Mikrotik is not an application layer firewall like Fortigate and alike.
 
nexlevel
just joined
Topic Author
Posts: 5
Joined: Thu Jul 22, 2021 11:51 pm

Re: TCP session reset

Wed Dec 29, 2021 6:36 pm

Hey sindy,
If I put the client behind the nat'd router that's behind the MT it works as expected. If I move it back to the MT it continues this behavior. I found a thread that you participated in (viewtopic.php?p=810594&hilit=yealink#p810594) last year and tried all of those steps this morning (also changed the client to 1.5 to get it below .10 in case length played a roll). Looks like the OP fixed it by double nat'ing it. I would really like to figure this thing out so I don't have to do that. I did change the handset to tcp with no encryption as well as straight udp and did some captures. Same symptoms both protocols. I am going to work with the server side vendor today to hopefully understand why that fin get's sent by the server. The TCP pcap below in case you see something I missed. The sip (remove 1 binding) is odd. Almost reads like the phone is unregistering itself? I'm not super familiar with the protocol.

TCP Pcap
230 289.343483 10.170.1.5 255.255.255.255 DHCP 590 DHCP Inform - Transaction ID 0x4f1af76a
231 289.343509 10.170.1.5 255.255.255.255 DHCP 590 DHCP Inform - Transaction ID 0x4f1af76a
232 289.343552 10.170.1.5 255.255.255.255 DHCP 590 DHCP Inform - Transaction ID 0x4f1af76a
233 289.345454 10.170.1.2 10.170.1.5 DHCP 342 DHCP ACK - Transaction ID 0x4f1af76a
234 289.560905 10.170.1.5 209.217.207.40 SIP 694 Request: REGISTER sip:syr-sip-cust.northland.net:5060 (1 binding) |
235 289.560922 10.170.1.5 209.217.207.40 TCP 694 [TCP Retransmission] 12635 → 5061 [PSH, ACK] Seq=1 Ack=1 Win=29200 Len=640
236 289.606723 209.217.207.40 10.170.1.5 TCP 54 5061 → 12635 [ACK] Seq=1 Ack=641 Win=30480 Len=0
237 289.608204 209.217.207.40 10.170.1.5 TCP 54 5061 → 12635 [FIN, ACK] Seq=1 Ack=641 Win=30480 Len=0
238 289.609010 10.170.1.5 209.217.207.40 TCP 60 12635 → 5061 [FIN, ACK] Seq=641 Ack=2 Win=29200 Len=0
239 289.609032 10.170.1.5 209.217.207.40 TCP 60 [TCP Out-Of-Order] 12635 → 5061 [FIN, ACK] Seq=641 Ack=2 Win=29200 Len=0
240 289.653514 209.217.207.40 10.170.1.5 TCP 54 5061 → 12635 [ACK] Seq=2 Ack=642 Win=30480 Len=0
241 300.387392 10.170.1.5 8.8.8.8 DNS 90 Standard query 0xce7a A sbc-sip-tls-cust.northland.net
242 300.387406 10.170.1.5 8.8.8.8 DNS 90 Standard query 0xce7a A sbc-sip-tls-cust.northland.net
243 300.688173 8.8.8.8 10.170.1.5 DNS 106 Standard query response 0xce7a A sbc-sip-tls-cust.northland.net A 209.217.207.40
244 300.689406 10.170.1.5 209.217.207.40 TCP 66 11940 → 5061 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=8
245 300.689424 10.170.1.5 209.217.207.40 TCP 66 [TCP Out-Of-Order] [TCP Port numbers reused] 11940 → 5061 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=8
246 300.981473 209.217.207.40 10.170.1.5 TCP 66 5061 → 11940 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=4
247 300.982098 10.170.1.5 209.217.207.40 TCP 60 11940 → 5061 [ACK] Seq=1 Ack=1 Win=29200 Len=0
248 300.982115 10.170.1.5 209.217.207.40 TCP 60 [TCP Dup ACK 247#1] 11940 → 5061 [ACK] Seq=1 Ack=1 Win=29200 Len=0
249 301.202906 10.170.1.5 209.217.207.40 SIP 691 Request: REGISTER sip:syr-sip-cust.northland.net:5060 (remove 1 binding) |
250 301.202921 10.170.1.5 209.217.207.40 TCP 691 [TCP Retransmission] 11940 → 5061 [PSH, ACK] Seq=1 Ack=1 Win=29200 Len=637
251 301.250173 209.217.207.40 10.170.1.5 TCP 54 5061 → 11940 [ACK] Seq=1 Ack=638 Win=30476 Len=0
252 301.250962 209.217.207.40 10.170.1.5 TCP 54 5061 → 11940 [FIN, ACK] Seq=1 Ack=638 Win=30476 Len=0
253 301.251808 10.170.1.5 209.217.207.40 TCP 60 11940 → 5061 [FIN, ACK] Seq=638 Ack=2 Win=29200 Len=0
254 301.251815 10.170.1.5 209.217.207.40 TCP 60 [TCP Out-Of-Order] 11940 → 5061 [FIN, ACK] Seq=638 Ack=2 Win=29200 Len=0
255 301.296872 209.217.207.40 10.170.1.5 TCP 54 5061 → 11940 [ACK] Seq=2 Ack=639 Win=30476 Len=0
256 316.513454 10.170.1.5 8.8.8.8 DNS 90 Standard query 0x979c A sbc-sip-tls-cust.northland.net
257 316.513461 10.170.1.5 8.8.8.8 DNS 90 Standard query 0x979c A sbc-sip-tls-cust.northland.net
258 316.728476 8.8.8.8 10.170.1.5 DNS 106 Standard query response 0x979c A sbc-sip-tls-cust.northland.net A 209.217.207.40
259 316.729842 10.170.1.5 209.217.207.40 TCP 66 12197 → 5061 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=8
260 316.729849 10.170.1.5 209.217.207.40 TCP 66 [TCP Out-Of-Order] [TCP Port numbers reused] 12197 → 5061 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=8
261 316.961109 209.217.207.40 10.170.1.5 TCP 66 5061 → 12197 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=4
262 316.961742 10.170.1.5 209.217.207.40 TCP 60 12197 → 5061 [ACK] Seq=1 Ack=1 Win=29200 Len=0
263 316.961759 10.170.1.5 209.217.207.40 TCP 60 [TCP Dup ACK 262#1] 12197 → 5061 [ACK] Seq=1 Ack=1 Win=29200 Len=0
264 317.243188 10.170.1.5 209.217.207.40 SIP 691 Request: REGISTER sip:syr-sip-cust.northland.net:5060 (remove 1 binding) |
265 317.243203 10.170.1.5 209.217.207.40 TCP 691 [TCP Retransmission] 12197 → 5061 [PSH, ACK] Seq=1 Ack=1 Win=29200 Len=637
266 317.287259 209.217.207.40 10.170.1.5 TCP 54 5061 → 12197 [ACK] Seq=1 Ack=638 Win=30476 Len=0
267 317.288754 209.217.207.40 10.170.1.5 TCP 54 5061 → 12197 [FIN, ACK] Seq=1 Ack=638 Win=30476 Len=0
268 317.289625 10.170.1.5 209.217.207.40 TCP 60 12197 → 5061 [FIN, ACK] Seq=638 Ack=2 Win=29200 Len=0
269 317.289642 10.170.1.5 209.217.207.40 TCP 60 [TCP Out-Of-Order] 12197 → 5061 [FIN, ACK] Seq=638 Ack=2 Win=29200 Len=0
270 317.336862 209.217.207.40 10.170.1.5 TCP 54 5061 → 12197 [ACK] Seq=2 Ack=639 Win=30476 Len=0
271 348.771699 10.170.1.5 8.8.8.8 DNS 90 Standard query 0x539c A sbc-sip-tls-cust.northland.net
272 348.771712 10.170.1.5 8.8.8.8 DNS 90 Standard query 0x539c A sbc-sip-tls-cust.northland.net
273 349.023244 8.8.8.8 10.170.1.5 DNS 106 Standard query response 0x539c A sbc-sip-tls-cust.northland.net A 209.217.207.40
274 349.024514 10.170.1.5 209.217.207.40 TCP 66 12198 → 5061 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=8
275 349.024529 10.170.1.5 209.217.207.40 TCP 66 [TCP Out-Of-Order] [TCP Port numbers reused] 12198 → 5061 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=8
276 349.262957 209.217.207.40 10.170.1.5 TCP 66 5061 → 12198 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=4
277 349.263597 10.170.1.5 209.217.207.40 TCP 60 12198 → 5061 [ACK] Seq=1 Ack=1 Win=29200 Len=0
278 349.263616 10.170.1.5 209.217.207.40 TCP 60 [TCP Dup ACK 277#1] 12198 → 5061 [ACK] Seq=1 Ack=1 Win=29200 Len=0
279 349.537939 10.170.1.5 209.217.207.40 SIP 694 Request: REGISTER sip:syr-sip-cust.northland.net:5060 (1 binding) |
280 349.537955 10.170.1.5 209.217.207.40 TCP 694 [TCP Retransmission] 12198 → 5061 [PSH, ACK] Seq=1 Ack=1 Win=29200 Len=640
281 349.582191 209.217.207.40 10.170.1.5 TCP 54 5061 → 12198 [ACK] Seq=1 Ack=641 Win=30480 Len=0
282 349.582731 209.217.207.40 10.170.1.5 TCP 54 5061 → 12198 [FIN, ACK] Seq=1 Ack=641 Win=30480 Len=0
283 349.583461 10.170.1.5 209.217.207.40 TCP 60 12198 → 5061 [ACK] Seq=641 Ack=2 Win=29200 Len=0
284 349.583469 10.170.1.5 209.217.207.40 TCP 60 [TCP Dup ACK 283#1] 12198 → 5061 [ACK] Seq=641 Ack=2 Win=29200 Len=0
285 349.583546 10.170.1.5 209.217.207.40 TCP 60 12198 → 5061 [FIN, ACK] Seq=641 Ack=2 Win=29200 Len=0
286 349.583552 10.170.1.5 209.217.207.40 TCP 60 [TCP Out-Of-Order] 12198 → 5061 [FIN, ACK] Seq=641 Ack=2 Win=29200 Len=0
287 349.629492 209.217.207.40 10.170.1.5 TCP 54 5061 → 12198 [ACK] Seq=2 Ack=642 Win=30480 Len=0
288 360.373130 10.170.1.5 8.8.8.8 DNS 90 Standard query 0x8577 A sbc-sip-tls-cust.northland.net
289 360.373144 10.170.1.5 8.8.8.8 DNS 90 Standard query 0x8577 A sbc-sip-tls-cust.northland.net
290 360.405933 8.8.8.8 10.170.1.5 DNS 106 Standard query response 0x8577 A sbc-sip-tls-cust.northland.net A 209.217.207.40
291 360.407194 10.170.1.5 209.217.207.40 TCP 66 12539 → 5061 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=8
292 360.407205 10.170.1.5 209.217.207.40 TCP 66 [TCP Out-Of-Order] [TCP Port numbers reused] 12539 → 5061 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=8
293 360.456745 209.217.207.40 10.170.1.5 TCP 66 5061 → 12539 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=4
294 360.457444 10.170.1.5 209.217.207.40 TCP 60 12539 → 5061 [ACK] Seq=1 Ack=1 Win=29200 Len=0
295 360.457461 10.170.1.5 209.217.207.40 TCP 60 [TCP Dup ACK 294#1] 12539 → 5061 [ACK] Seq=1 Ack=1 Win=29200 Len=0
296 360.920836 10.170.1.5 209.217.207.40 SIP 691 Request: REGISTER sip:syr-sip-cust.northland.net:5060 (remove 1 binding) |
297 360.920853 10.170.1.5 209.217.207.40 TCP 691 [TCP Retransmission] 12539 → 5061 [PSH, ACK] Seq=1 Ack=1 Win=29200 Len=637
298 361.015055 209.217.207.40 10.170.1.5 TCP 54 5061 → 12539 [ACK] Seq=1 Ack=638 Win=30476 Len=0
299 361.016355 209.217.207.40 10.170.1.5 TCP 54 5061 → 12539 [FIN, ACK] Seq=1 Ack=638 Win=30476 Len=0
300 361.017167 10.170.1.5 209.217.207.40 TCP 60 12539 → 5061 [FIN, ACK] Seq=638 Ack=2 Win=29200 Len=0
301 361.017184 10.170.1.5 209.217.207.40 TCP 60 [TCP Out-Of-Order] 12539 → 5061 [FIN, ACK] Seq=638 Ack=2 Win=29200 Len=0
302 361.065372 209.217.207.40 10.170.1.5 TCP 54 5061 → 12539 [ACK] Seq=2 Ack=639 Win=30476 Len=0
303 376.501469 10.170.1.5 8.8.8.8 DNS 90 Standard query 0x3e4e A sbc-sip-tls-cust.northland.net
304 376.501482 10.170.1.5 8.8.8.8 DNS 90 Standard query 0x3e4e A sbc-sip-tls-cust.northland.net
305 376.561961 8.8.8.8 10.170.1.5 DNS 106 Standard query response 0x3e4e A sbc-sip-tls-cust.northland.net A 209.217.207.40
 
sindy
Forum Guru
Forum Guru
Posts: 10206
Joined: Mon Dec 04, 2017 9:19 pm

Re: TCP session reset

Wed Dec 29, 2021 9:15 pm

Well, that's two different cans of worms. With plaintext SIP (no matter whether TCP or UDP), if the SIP helper (ALG) in the firewall is enabled, the length issues could play a role when handling a payload (SDP) whose length is explicitly stated in the packet, so there may be a mismatch of the declared length and the actual one. But since it breaks already when handling a REGISTER which carries no SDP, it cannot be this.

"Remove 1 binding" means that the phone sends a specific contact and asks to register it for 0 more seconds, which is the way to unregister that single contact selectively. Leaving aside the case when the phone is shutting down or the user explicitly asks it to unregister, this may happen when the phone is too picky and the registrar returns a different contents of the Contact header in its 200 response than the one the phone has sent. But here, the pcap shows no 200 response to the REGISTER at all, the server sends an ACK to the packet carrying the initial REGISTER and then sends a FIN, so it doesn't like something about the contents of the REGISTER.

To tell you more, I'd have to see the very same REGISTER as it enters the router from the phone and as it leaves the router towards the registrar, i.e. you'd have to capture at both the LAN and WAN interface of the Mikrotik and filter on the registrar IP address, and I'd have to see the complete contents of the packets, not the single-row excerpts, to be able to diff them. I know you say the SIP helper is off, but it may not be off enough (seen that elsewhere, not on Mikrotik in particular).

But once the SIP exchange is encrypted, the SIP helper (ALG) has no way to manipulate anything inside the SIP message contents even if it was on, so it again suggests something related to the LAN address being sent inside the SIP message. Just an idea, can you create a dedicated subnet inside 192.168.0.0/16 or 172.16.0.0/12 (i.e. a private one but outside 10.0.0.0/8) on the Mikrotik, move the phone to this subnet, and see the outcome? I.e. do a single NAT but from a very different LAN address, ideally similar to the one used at the LAN side of the second router?
 
nexlevel
just joined
Topic Author
Posts: 5
Joined: Thu Jul 22, 2021 11:51 pm

Re: TCP session reset

Fri Dec 31, 2021 10:36 pm

Hey sindy, Happy New Year and thank you for your help! I don't want to get to far ahead of myself but I setup a new bridge and created a standard class c address space. 192.168.5.0/24. Added the test phone to the bridge and it connected and I haven't seen it dump the session like it was before. I'll still need to do some testing to confirm but it's the most progress I've had in weeks! We were using a non standard address space with the first octet 10 and mask of /24. Maybe that call server doesn't appreciate it. I will dig into that further to confirm since we have many of these deployments on MT's with similiar address schemes. I'll have an update next week. Thanks!

Who is online

Users browsing this forum: aferreira and 167 guests