VoIP call causes extreme lag

kosztyua · Tue Nov 29, 2016 4:20 pm

Hi,

I have encountered an extreme situation while configuring a new voip server. The voip server is now moved from local hosting to cloud, as such all phones internally go through boundary and NAT. Using two test phones, plugged in locally and connected to the remote server, when I call one from the other the connection is established and there is fine two-way communication, so no NAT issue. However, after about 20-30 seconds the whole network starts lagging. As I ping 8.8.8.8, the latency (measured from other devices connected to the network, or even from the router itself) goes up from 8ms to 2000ms+ and only resumes normal operation when I disconnect the phone call.
The two phones are connected to a CRS (6.37), connected to a Hex (6.36.3) as router (UPDATE both now updated to 6.37.3). Both have firewall service-port sip disabled (tested with enabled too). Both show normal resources and no major change in pps or throughput.

So my question is basically, how can a SIP call effectively cause denial-of-service? And how do I fix this?

Regards,
Andras

marrold · Tue Nov 29, 2016 4:47 pm

Do you have a packet capture?

quackyo · Tue Nov 29, 2016 4:48 pm

Post firewall on HEX, and if you have any queues - post them too.

kosztyua · Tue Nov 29, 2016 5:02 pm

Do you have a packet capture?

I dont have packet capture - what should I log?

Post firewall on HEX, and if you have any queues - post them too.

I don't have any queues, and as for firewall it worked fine for the past many months - only the SIP caused this issue, and only externally when NATing, as previously and even now internal server (going through ipsec) works fine.

/ip firewall filter
add action=fasttrack-connection chain=input comment="fasttrack related, established input" connection-state=established,related
add action=accept chain=input comment="accept related, established input" connection-state=established,related
add action=drop chain=input comment=blacklist dst-address-list=blacklist
add action=fasttrack-connection chain=output comment="fasttrack related, established output" connection-state=established,related
add action=drop chain=output comment="drop smb to internet" dst-address=0.0.0.0/0 dst-port=445 out-interface=WAN-UPC protocol=tcp
add action=drop chain=output comment=blacklist src-address-list=blacklist
add action=accept chain=forward comment="accept mail.XXXHOST.hu ssh forward" dst-address=192.168.1.6 dst-port=22 protocol=tcp src-address=XXXIP
add action=fasttrack-connection chain=forward comment="fasttrack related, established forward but exclude ipsec" connection-mark=!ipsec connection-state=established,related
add action=accept chain=forward comment="accept related,established forward" connection-state=established,related dst-address=0.0.0.0/0 src-address=0.0.0.0/0
add action=drop chain=forward src-address=192.168.1.100
add action=drop chain=forward comment=blacklist src-address-list=blacklist
add action=drop chain=forward comment=blacklist dst-address-list=blacklist
add action=drop chain=forward comment="drop invalid forward" connection-state=invalid
add action=accept chain=forward comment="accept XXXHOST3 vpn users to unraid forward" dst-address=192.168.1.4 src-address=192.168.11.0/24
add action=drop chain=forward comment="drop XXXHOST3 vpn users to all forward" src-address=192.168.11.0/24
add action=accept chain=forward comment="accept DMZ outgoing forward" out-interface=WAN-UPC src-address=192.168.10.0/24
add action=drop chain=forward comment="drop DMZ to all forward" src-address=192.168.10.0/24
add action=drop chain=input comment="drop invalid input" connection-state=invalid
add action=accept chain=input comment="accept pptp/tcp1723 input" dst-port=1723 protocol=tcp src-address-list=hunlist
add action=accept chain=input comment="accept openvpn tcp/12443 input" dst-port=12443 protocol=tcp src-address-list=hunlist
add action=accept chain=input comment="accept openvpn tcp/12443 input from mail.XXXHOST.hu @ digitalocean" dst-port=12443 protocol=tcp src-address=XXXIP
add action=accept chain=input comment="accept sstp/tcp8443 input" dst-port=8443 protocol=tcp
add action=accept chain=input comment="accept winbox main" dst-port=8291 protocol=tcp src-address=192.168.1.0/24
add action=accept chain=input comment="accept winbox pptp" dst-port=8291 protocol=tcp src-address=192.168.100.0/24
add action=accept chain=input comment="accept icmp input" protocol=icmp
add action=accept chain=input comment="accept established input" connection-state=established
add action=accept chain=input comment="accept related input" connection-state=related
add action=accept chain=input comment="accept pptp/gre input" protocol=gre src-address-list=hunlist
add action=accept chain=input comment="ipsec (udp/500)" dst-port=500 protocol=udp
add action=accept chain=forward comment="accept KRISZTI ip pool forward" src-address=192.168.106.0/24
add action=accept chain=forward comment="accept MAIN ip pool to ALL forward" dst-address=0.0.0.0/0 src-address=192.168.1.0/24
add action=accept chain=forward comment="accept roadwarrior VPN ip pool forward" dst-address=0.0.0.0/0 src-address=192.168.100.0/24
add action=accept chain=forward comment="accept ANCSA ip pool forward" dst-address=0.0.0.0/0 src-address=192.168.104.0/24
add action=accept chain=forward comment="accept XXXHOST incoming forward" dst-address=192.168.1.6-192.168.1.10 src-address=0.0.0.0/0
add action=accept chain=forward comment="accept XXXHOST3 torrent forward" dst-address=192.168.1.4 dst-port=51413 protocol=tcp src-address=0.0.0.0/0
add action=accept chain=forward comment="accept XXXHOST3 torrent forward" dst-address=192.168.1.4 dst-port=51413 protocol=udp src-address=0.0.0.0/0
add action=accept chain=forward comment="accept ts/game forward" dst-address=192.168.1.25 src-address=0.0.0.0/0
add action=accept chain=forward comment="accept mail.XXXHOST.hu webmail forward" dst-address=192.168.1.6 dst-port=443,80,25 protocol=tcp src-address=0.0.0.0/0 src-address-list=hunlist
add action=accept chain=forward comment="accept git.XXXHOST2.hu https forward" dst-address=192.168.1.28 dst-port=443 protocol=tcp src-address=0.0.0.0/0
add action=accept chain=forward comment="accept XXXHOST3 ALL to plex webui forward" dst-address=192.168.1.4 dst-port=32400 protocol=tcp src-address=0.0.0.0/0
add action=drop chain=forward comment="drop all forward" log=yes log-prefix=fordward_drop
add action=accept chain=input port=1701,500,4500 protocol=udp
add action=accept chain=input protocol=ipsec-esp
add action=accept chain=input port=1701,500,4500 protocol=udp
add action=accept chain=input protocol=ipsec-esp
add action=drop chain=input comment="drop all input" log-prefix=input_drop

/ip firewall nat
add action=accept chain=srcnat dst-address=192.168.100.0/24 src-address=192.168.1.0/24
add action=accept chain=srcnat disabled=yes dst-address=192.168.102.0/24 src-address=192.168.1.0/24
add action=accept chain=srcnat dst-address=192.168.104.0/24 src-address=192.168.1.0/24
add action=accept chain=srcnat dst-address=192.168.105.0/24 src-address=192.168.1.0/24
add action=accept chain=srcnat dst-address=192.168.106.0/24 src-address=192.168.1.0/24
add action=dst-nat chain=dstnat comment="XXXHOST3 transmission client" dst-port=51413 protocol=tcp to-addresses=192.168.1.4
add action=dst-nat chain=dstnat comment="XXXHOST3 transmission client" dst-port=51413 protocol=udp to-addresses=192.168.1.4
add action=dst-nat chain=dstnat comment="teamspeak udp" dst-port=9987 in-interface=WAN-UPC protocol=udp to-addresses=192.168.1.25
add action=dst-nat chain=dstnat comment="teamspeak tcp" dst-port=30033 in-interface=WAN-UPC protocol=tcp to-addresses=192.168.1.25
add action=dst-nat chain=dstnat comment="mail.XXXHOST.hu webmail" dst-port=443 in-interface=WAN-UPC protocol=tcp to-addresses=192.168.1.6 to-ports=443
add action=dst-nat chain=dstnat comment="mail.XXXHOST.hu webmail" dst-port=80 in-interface=WAN-UPC protocol=tcp to-addresses=192.168.1.6 to-ports=80
add action=dst-nat chain=dstnat comment="mail.XXXHOST.hu smtp" dst-port=25 in-interface=WAN-UPC protocol=tcp to-addresses=192.168.1.6 to-ports=25
add action=dst-nat chain=dstnat comment="mail.XXXHOST.hu ssh" dst-port=22 in-interface=WAN-UPC protocol=tcp to-addresses=192.168.1.6 to-ports=22
add action=log chain=srcnat disabled=yes log-prefix=srcnat
add action=masquerade chain=srcnat out-interface=WAN-UPC
add action=dst-nat chain=dstnat disabled=yes dst-port=3389 in-interface=WAN-UPC protocol=tcp to-addresses=192.168.10.100 to-ports=3389
add action=dst-nat chain=dstnat comment="NAT XXXHOST3 plex webui" dst-port=60025 in-interface=WAN-UPC protocol=tcp to-addresses=192.168.1.4 to-ports=32400

/ip firewall mangle
add action=log chain=forward connection-state=new disabled=yes out-interface=WAN-UPC
add action=change-mss chain=forward disabled=yes dst-address=192.168.105.0/24 new-mss=1360 passthrough=yes protocol=tcp src-address=192.168.1.0/24 tcp-flags=syn tcp-mss=!0-1360
add action=change-mss chain=forward disabled=yes dst-address=192.168.105.0/24 new-mss=1360 passthrough=yes protocol=tcp src-address=192.168.1.0/24 tcp-flags=syn tcp-mss=!0-1360
add action=mark-connection chain=forward comment="Mark IPsec" ipsec-policy=in,ipsec new-connection-mark=ipsec passthrough=no
add action=mark-connection chain=forward comment="Mark IPsec" ipsec-policy=out,ipsec new-connection-mark=ipsec passthrough=no

/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes

kosztyua · Wed Nov 30, 2016 7:51 pm

tslytsly · Thu Dec 01, 2016 4:49 pm

Hi,
Really need a capture of your WAN interface to diagnose this.

For instance, it could be that not all the SIP traffic is successfully traversing the firewall, causing many re-transmissions which are hitting the firewall.
This is unlikely, but possible depending on your SIP server setup.

kosztyua · Thu Dec 01, 2016 7:51 pm

Hi,
Really need a capture of your WAN interface to diagnose this.

For instance, it could be that not all the SIP traffic is successfully traversing the firewall, causing many re-transmissions which are hitting the firewall.
This is unlikely, but possible depending on your SIP server setup.

Are you saying that my SIP server may be DoSing me? That makes perfect sense, I'll create a rule to see that. What would you recommend? Netflow and wireshark? Or just log the firewall drops?

tslytsly · Fri Dec 02, 2016 9:49 am

Are you saying that my SIP server may be DoSing me? That makes perfect sense, I'll create a rule to see that. What would you recommend? Netflow and wireshark? Or just log the firewall drops?

I would start with a packet sniffer capture of the WAN interface when you get the slow down.
This will show if the server is causing the problem or not.

kosztyua · Fri Dec 02, 2016 6:42 pm

OK I've checked it, packet sniffer to wireshark, and seemingly there is nothing out of order. I can see the call being established, SRTP packets with sequence rising both ways throughout the call, even when the ping to external server is already times out. Only thing around the time when the lag begins (20s into the call) is a CLASSIC-STUN Binding request, but guess thats normal. I'd save and upload, but not sure if that would be of any help, and also for some reason wireshark has save grayed out, never tried streaming before.

tslytsly · Fri Dec 02, 2016 6:46 pm

OK I've checked it, packet sniffer to wireshark, and seemingly there is nothing out of order. I can see the call being established, SRTP packets with sequence rising both ways throughout the call, even when the ping to external server is already times out. Only thing around the time when the lag begins (20s into the call) is a CLASSIC-STUN Binding request, but guess thats normal. I'd save and upload, but not sure if that would be of any help, and also for some reason wireshark has save grayed out, never tried streaming before.

OK, it would be useful so see if the ICMP packets get delayed on the way out or in.

For example, if you can see ICMP with immediate replies back but on your PC you get large delays then it's the router's firewall introducing it.
But if you see the same lengthy delays in replies on the WAN interface then it's something upstream that is causing the problem.

savage · Fri Dec 02, 2016 8:54 pm

Just as a point of interest - are you running flow control?

I've seen cases (not MT specific, but networking in general) where flow-control sometimes really messes about with VOIP for some reason.

kosztyua · Sun Dec 04, 2016 5:20 pm

@tslytsly
I was pinging from the router itself to 8.8.8.8, and the delay was same way measurable on WAN port. Upstream then? My ISP? What I could try is to connect out on VPN and see if the issue persists over that.

@savage
Flow control is set to off on all interfaces.

tslytsly · Mon Dec 05, 2016 10:21 am

@tslytsly
I was pinging from the router itself to 8.8.8.8, and the delay was same way measurable on WAN port. Upstream then? My ISP? What I could try is to connect out on VPN and see if the issue persists over that.

If, in your Wireshark capture, the pings show as arriving very delayed, then yes the problem must be upstream.

VoIP call causes extreme lag

VoIP call causes extreme lag

Re: VoIP call causes extreme lag (wtf?)

Re: VoIP call causes extreme lag (wtf?)

Re: VoIP call causes extreme lag (wtf?)

Re: VoIP call causes extreme lag (wtf?)

Re: VoIP call causes extreme lag (wtf?)

Re: VoIP call causes extreme lag (wtf?)

Re: VoIP call causes extreme lag (wtf?)

Re: VoIP call causes extreme lag

Re: VoIP call causes extreme lag

Re: VoIP call causes extreme lag

Re: VoIP call causes extreme lag

Re: VoIP call causes extreme lag

Who is online