I know I'm bumping a very old post. But reporting in that this is still an issue.
We are running the suggested script every minute to make this run.
Thankfully it works, however it would be nice to have a long term fix.

All these years I´m still using the script, there has not been a real solution for this problem.

Having the same issue when upgraded to the latest version. 

PPPOE session gets bumped from our ISP and phones cannot re-register. Need to clear out connections and it works. Disabled SIP in Firewall Ports etc.

We do not see this issue on a /32 /30 IP address. 

The only solution I found is to use tcp transport for sip signaling. Worked immediately without any further issues. I don't understand why Mikrotik is not taking care about this.

Still got this issue in 6.41.

I've tried deleting connections but this doesnt always work. In this case the only thing to do is to reboot it. But it gals me to do that - this is about the only thing I have to reboot Mikrotik for. How sad thats its something so basic.

Actually, I have a theory.

I've seen the connections timeout period count upwards which shows they are getting in some weird condition. Maybe the delete doesn't fully delete connections in this state and it gets orphaned and hidden from the list. I theorised that loosing the WAN might do some blanket clear-out of connections associated with the WAN. This is just a theory I came up with after several days observing this issue and being unwilling to reboot as the solution.

Dropping the WAN worked in my case.

Still got this issue in 6.41.

I've tried deleting connections but this doesnt always work. In this case the only thing to do is to reboot it. But it gals me to do that - this is about the only thing I have to reboot Mikrotik for. How sad thats its something so basic.

Still in 6.42.3

I thought I was going crazy today, until I came across this thread. How can something so basic still be an issue for so long?
Though to be fair I cant see any reference in this thread to anybody presenting diagnostics to Mikrotik to get them to resolve it....

At the site that I experienced this at; I have put a script in place to drop connections as a bandaid, and I will change the WAN from PPPoE to a /30 once I can get some IP's allocated. If it was not a live site I would be the one to properly document it and report, but alas it wont be.

Since I see people have still been posting in this thead, and some of the more recent responses have mentioned that they are using PPPoE, I thought I'd stop by to let people here know that as of RouterOS 6.33, if you are using PPPoE on your WAN and suffering from this problem, you can use the "on-up" PPP event to trigger automatic, surgical removal of only the problematic NAT conntrack entry whenever your PPPoE connection bounces. No more manual clearing out the entry (or all entries, if you're in a hurry) or rebooting, or scheduling a script to run every few seconds to check if it needs to be done. See this post.

-- Nathan

I can confirm that from version 4.x till now 6.4x same thing if PPP interface is in use so I use this one as a script and no more reports from customer:
/ip firewall connection remove [/ip firewall connection find where connection-type=sip and assured=no]

Good afternoon gurus and gentleman,

Ive followed this thread till attaining a successful result with failover operational via dual LTE connections and the script running every minute to clear out the NAT connections so thank you to all those who have ventured through frustrating circumstances so that others may learn. This configuration works perfectly with my sip phones communicating directly to the voip server on port 5060 but ive now implemented cloud pbx solutions to my clients which is currently outsourced and the port numbers to the cloud pbx (FreePBX) are nowhere near 5060. So im dealing with a new IP range which changes from client to client and different ports eg 3333 which works to an extent whereby the phone is able to make a call after switching gateways but doesnt receive a call even though its registered. Id appreciate any advice on the matter with humble thanks

Have you tried adding your custom ports to SIP Helper?

Have you tried adding your custom ports to SIP Helper?

HI,

Please excuse the late reply as this exercise has my neurons in a twist. Ive tried the SIP helper by adding the port 3333, also running scripts to delete UDP connections matching the Cloud PBX server and many other permutations found on the net with no success all relating to clearing out the old NAT rules. I found that a reboot on the router OR de-registering the sip phone and registering it again brings the extension back online after failover has taken place onto the new connection.

Any suggestions?

These are the scripts ive tried:

/ip firewall connection remove [find where dst-address~"Cloud PBX IP"]
/ip firewall connection remove [find where connection-type=sip or connection-type=sip-2 or connection-type=sip-1]

I found that a reboot on the router OR de-registering the sip phone and registering it again brings the extension back online after failover has taken place onto the new connection.

Seems i was incorrect for the above statement. Rebooting the router does not work, only re-registering the extension does. So this may not be a problem with NAT?

Reboot of router removes any and all tracked connections, so any packet coming in from the WAN side, except those matching dst-nat or accept rules, is dropped as it does not match any existing tracked connection. Only by re-registering the phone you create a tracked connection so further packets coming from the WAN side are matched to that connection and forwarded in.

All the scripts removing existing connections are there only to prevent LAN->WAN packets from being src-nated to the IP address of the WAN through which their connections have been previously established, so that new registration or outgoing call could establish a new connection which gets be src-nated to the IP address of the WAN active at that moment. So even if you remove those connections with inadequate src-nat address, no incoming calls can get in until the phone re-registers as a) the telephony exchange needs to learn the new public IP to which it should send incoming INVITEs and b) the connection tracker in Mikrotik's firewall has to expect the incoming INVITE as part of the UDP (or SIP if the helper is active) connection established by the REGISTER.

A plain UDP tracked connection survives 3 minutes since the last packet seen, but tracked connections with protocol~"sip" survive 60 minutes following a successful registration no matter whether any packets are seen or not. So if the registration expires before those 60 minutes and the phone attempts to re-register, it still hits an existing connection with a wrong src-nat address, and by reattempting to register over and over again it keeps it up although a failure to register shrinks the connection life back to 3 minutes. Telephony exchanges which can deal with customer side NAT send keepalive packets (either valid SIP packets or just UDP packets with the proper source and destination socket addresses) to keep the 3-minute plain UDP connections up, so they don't need the SIP helper to be active. They also ignore the RTP destination socket which the CPE sends using SDP and learn it from incoming RTP.

Hi Sindy,

Ive been following your response on another thread and trying to figure out a solutions since ive been sitting with this issue for over 3 weeks now and im now closer to resolving. Would u mind taking a look at my export config to tell point out possibly where im going wrong?

It's not only a matter of the Mirotik configuration but also of how the exchange behaves and of the network topology (which part of the voice solution is on the public side of the firewall and which is on the private one). This should tell whether you actually need the SIP helper in Mikrotik to be active or you can do without it, or, in other cases (i.e. other than yours because for you it does work unless WAN failover happens), whether the SIP helper is even able to help.

So I don't mind looking at the config, but I'd like to know also the topology and you should check the behaviour of the exchange, maybe it does send the keepalives which would inidicate that it is also capable of dealing with the NAT without the SIP helper.

HI,

As discussed.



Config
# model = RouterBOARD 931-2nD r2

/interface bridge
add fast-forward=no name=bridge1
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n disabled=no frequency=auto mode=ap-bridge ssid=********
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] authentication-types=w****** *******="*****" mode=****** supplicant-identity=*******
/ip pool
add name=dhcp_pool0 ranges=192.168.137.2-192.168.137.254
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=bridge1 name=dhcp1
/interface bridge port
add bridge=bridge1 interface=wlan1
add bridge=bridge1 interface=ether3
/interface bridge settings
set allow-fast-path=no
/ip firewall connection tracking
set enabled=yes
/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=WAN
add interface=bridge1 list=LAN
/ip address
add address=192.168.137.1/24 interface=bridge1 network=192.168.137.0
/ip dhcp-client
add add-default-route=no dhcp-options=hostname,clientid disabled=no interface=ether1 use-peer-dns=no
add add-default-route=no dhcp-options=hostname,clientid disabled=no interface=ether2 use-peer-dns=no
/ip dhcp-server network
add address=192.168.137.0/24 gateway=192.168.137.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4,10.0.0.254
/ip firewall filter
add action=accept chain=input connection-state=established,related
add action=accept chain=input protocol=icmp
add action=drop chain=input in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN src-address=192.168.137.0/24
/ip firewall service-port
set sip ports=5060,5161 sip-timeout=10s
/ip route
add check-gateway=ping distance=1 gateway=8.8.8.8
add check-gateway=ping distance=2 gateway=8.8.4.4
add distance=1 dst-address=8.8.4.4/32 gateway=10.0.0.254 scope=10
add distance=1 dst-address=8.8.8.8/32 gateway=192.168.0.1 scope=10
/system clock
set time-zone-name=Africa/Johannesburg
/system routerboard settings
set silent-boot=no
/system script
add dont-require-permissions=no name=script1 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
source="/ip firewall connection remove [find where connection-type=sip]"

The setup communicates with a Cloud PBX running on port 3333

The i-thing to the right of the Yealink phone symbolizes what? A softphone running on a PC, a generic PC not related to the voice service but using the same internet connectivity?

OK, so I'm reading this thread and I have yet to see any real SIP debugs or information that is proving the MT's are the actual cause here. It looks like a bunch of assumptions and guessing.

I am 100% a ITSP (Internet Telephone Service Provider) and I use MT's 100% for my end users. I have a wide variety of setups from a simple legacy PBX (FXS/PRI/T1/E1) that have one or more gateways that convert SIP to their handoff interface. I have IP-PBX's on their own, IP-PBX+IP phones connected to my hosted platform and locations with purely hosted solutions (just IP phones). In all these cases I have MT's sitting there as the core router.

I have a hotel right now with 100+ rooms that is completely _hosted in the cloud_. That is over 100+ devices registering and making calls over the Internet. Not to mention their Front Desk and other phones around the hotel. So it's more like 110+ devices.

I have offices that have 15+ users and each of their phones have BLF monitoring of the other 14 users plus Voicemail boxes, 9 Park Orbit Slots and various other things. That is basically 25+ accounts per phone that is sending SUBCRIBEs or REGISERs and getting MWI and other BLF NOTIFY messages.

So again, I'm seeing the blame being laid on MT as the core of the issue for people but I have yet to experience any of this. With amount of SIP devices/endpoints that I have out there sitting behind MTs I'm pretty sure I would have seen this and have people calling about how their voice is just not working right. I'm not.

So for the people actually having this issue, forget the MT part of it (telling me how it's this and that in the MT) and just tell me what the _actual SIP issues are_. Like most things SIP can present the same symptoms for various problems and not understanding what those problems are and how they share symptoms can lead someone to the wrong troubleshooting path and end up blaming the router when it's not (or vice versa).

Samot, WHOH there tiger! I swear if someone doesn't contact you from that little bit of subtle advertising then this world is gone all crazy....Nobody blamed the MK, its a basic understanding of how the NAT and UDP protocols apply in this scenario which till now has differed from all most other listed solutions deploying the failover which is also for my own understanding and learning (Caution: may slightly defer from yours) , We all begin somewhere so consider this my entrance into the realm.

Sindy, yes those are standard Windows PC users on the same network.

I realised my description is a little vague. My deployments currently consist of sip devices and pc users on the same network mask running off two gateways on different IP ranges. The SIP devices are set to static gateway along with the most reliable LTE connection which serves as the dedicated connection this way. Now with the dire need for failover, this was the most viable solution since ive been using Mikrotik in basic setups for some time.

In your overall topology, it makes no difference whether the SIP helper is active on the 931 because the LTE router also does NAT and there is little chance that it would include a SIP ALG, and the LTE operator is quite likely to do NAT as well. So if your cloud exchange successfully handles clients connected via LTE, it definitely doesn't need the SIP helper to be running at the 931.

Looking at your configuration, I cannot see anything unusual there, except that there are no signs of automatic triggering of the connection clenaup script at the failover event.

You've chosen the scriptless failover method monitoring external reference IP addresses, which is very convenient but as it requires no script to track the state of the uplink path, that non-existent script cannot be extended with the connection cleanup part.

So what you need is to schedule a script for a periodical run (once in 10 seconds is enough as that's how often the check-gateway process pings the reference addresses) which will check the primary path availability and if it changes as compared to the previous state (in any direction, i.e. both from available to unavailable and vice versa), it cleans up the SIP connections, but rather than choosing them up to protocol~"sip", it should choose them up to dst-address~"^the.cloud.pbx.ip.address:3333\$". If the cloud PBX address is eventually specified as FQDN and it sometimes changes or multiple IP numbers are returned in the DNS response, you need to let the script deal with that because the connection's dst-address is always numeric.

But even if you automate the connection cleanup this way, you have to bear in mind that removing the old connections makes only one half of the task. The other half is that the phone has to re-register itself because the exchange must learn the new public IP through which the phone can be reached. So unless you can actively force the phones to re-register, which is unlikely, you have to set them to register as frequently as the exchange permits so that the time from failover to re-registration was as short as possible.

If the unavailability periods of up to minutes between WAN change and phone re-registration are unacceptable for your clients, you may consider a more advanced redundancy scheme. For that one, you'd have to run a virtualized Mikrotik (CHR or x86) somewhere in a data center, and create two VPN tunnels (such as GRE over IPsec) to it from the 931, each forced via one of the WANs, and route the VoIP traffic between the CHR and the 931 through these tunnels without NAT. So the failover from one of the VPN tunnels to the other one would not change the public IP address of the phone from the point of view of the cloud exchange, as it would always see the CHR's public address. Which means that the service interruptions associated to the failover events would be much shorter and even active calls wouldn't drop, there would just be a short period of silence in them. Of course, the CHR is another Single Point of Failure in the topology, the first one being the 931, but as you've seen the reliability of the connections is much lower than the reliability of the matchbox 931, so if you choose a decent data center, you should be fine too.

There are some limitations regarding the number of tunnels per CHR, so you might have to deploy more than one depending on the number of client sites. Also, you might need to NAT the LANs of the client sites when connecting via the VPN tunnels to individual addresses from some public subnet unrelated to the VoIP service to permit overlapping client subnets (although two clients use the same subnet on their LANs, the CHR can see each of them as another individual IP address from 4.4.0.0/16, but this address is accessible for it via both VPN tunnels so there is no need to remove any connections at either the 931 or the CHR if one of the tunnels goes down).

I think we're forgetting that deleting the connections manually or by a script only works some if the time. If the connections get in that wierd state where the timeout starts counting up even deleting them doesn't work - even though they are gone from the connection list I think the that fact they have a timeout value means they still in effect somewhere. It's only thing that makes sense.. Only a reboot or disabling the wan interface clears them out properly.

If the connections get in that wierd state where the timeout starts counting up even deleting them doesn't work - even though they are gone from the connection list I think the that fact they have a timeout value means they still in effect somewhere. ... Only a reboot or disabling the wan interface clears them out properly.

Good point. However, I've obtained an impression that this "zombie" behaviour is specific to connections handled by the SIP helper, and what I've pointed out was that in @Shane77's specific case, the SIP helper can be disabled completely with no impact on the service.

HI all,

Sindy, the explanation is very detailed and thorough so i respect your time taken out for this and went through the options carefully. As Ferrogragh point out, even manually deleting the connections in this case doesnt work. Only rebooting the SIP phone (or even logging into the phone setup menu and clicking the confirm button to accept a blank setting works too) brings the extension back online. May i ask one more thing of you and thats to point out a similar setup on the client side which i can follow in detail to check if i have accommodated for all that i can possible from the modem side right down to the MK. Failing to do which i will have to contract someone to complete this task and broaden my knowledge. Ive never failed at an MK task before and it seems searching and reading the web solidly over three long weeks and nights has failed.

May i ask one more thing of you and thats to point out a similar setup on the client side which i can follow in detail to check if i have accommodated for all that i can possible from the modem side right down to the MK.

In all my setups where SIP phones are behind a NAT, one of which is on Mikrotik, all the phones are registered to phone exchanges which deal with client side NAT themselves, which means that the SIP helpers of any kind are just disabled. And unfortunately none of my setups uses some type of failover setup and hosts SIP phones at the same time, it's either-or.

Given that the unremoveable connections seem to be an issue specific for connections controlled by the SIP helper, and as your overall architecture clearly shows that the phone exchange in the cloud can deal with the NAT issue without the SIP helper, I've recommended you to disable the SIP helper at the 931 completely as the only change as compared to the configuration you've posted. To my best knowledge it should be enough for you to get rid of the problem of unremovable connections, and sorry if I haven't stated that clearly enough in the previous post. Just a note, some people say you have to reboot the router to make the SIP helper really off after disabling it, I don't remember whether it was necessary in my case as I did that years ago when the SIP helper was dropping messages it was unable to parse. I have never used it anywhere ever since as its disadvantages prevail in simple cases and it is incapable to deal with complex cases.

As the next step, what @Samot has written applies - sniff the SIP communication between the phone and the exchange while you power up the phone, and then sniff when the issue actually happens. Because either the phone stops trying to re-register automatically and you have to manually kick it to do so if it fails to register because the old connection is still alive and makes the registration fail (and I've seen a lot of unusual behaviour with CPEs), which would indicate a bug in the phone's firmware, or the phone may be configured for re-registration every 5 minutes or even more often but the phone exchange tells it that it doesn't accept such a short registration time and that the minimum registration time is, say, 2 hours. So if the failover takes place and the last registration thus becomes useless (because the registered contact uri indicates the now-unavailable public IP address), and the phone doesn't re-register within some minutes, you may simply lose patience and kick it before it decides to re-register on its own.

So sniffing at phone boot should show you how the initial registration goes (whether you get the "423 registration lifetime too brief" response to REGISTER or not and if you do, what minimum registration period the exchange requires), and sniffing while it happens should show you whether the phone keeps trying to re-register after the first re-registration attempt after the failover fails.

If you actually ask me for a script which will automatically clear the connections once a failover takes place, say so clearly.

I've tried extensively WITH and WITHOUT the SIP helper. You still get zombie connections. Common though is dest-port 5060/5061.

Perhaps there is something in the firewall/NAT that recognises 5060/1 in addition to what the SIP helper does?

@Ferrograph, are you saying you have a sequence of manipulation steps which is guaranteed to create the unremovable connections? If so, would you mind sending a description of that sequence to support@mikrotik.com along with your supout.rif? What I felt so far was that it happens at random and therefore Mikrotik is unable to identify the root cause.

@Shane77, given that @Ferrograph says disabling the interface is guaranteed to clear the zombie connections (which requires that the reply-dst-address of the connection was created using action=masquerade), there are two more things to try:

• instead of using /ip firewall connection remove in the script, use /interface disable followed by /interface enable. Given that the only reason for the failover is that the path to the internet through the interface breaks down, shutting the interface down for a moment will not do any harm (which may not be that simple in case of the fallback where the secondary WAN may be used as the only one for some specific traffic)
• instead of using a common chain=srcnat rule with action=masquerade, use one action=src-nat rule per each WAN and set to-addresses to the address of that WAN manually (which may require another script to copy the currently assigned IP address into the rule's to-addresses parameter if the LTE modem doesn't accept a static address on your WAN looking towards it and you have to use a DHCP client there). The rationale behind this suggestion is that if the zombie connections only occur if the reply-dst-address is assigned using action=masquerade, assigning it using action=src-nat may be a way to avoid them.

Regarding hiring someone to solve it for you - as it seems that there is an issue with removing the zombie connections, the only people able to resolve it are Mikrotik's developers. I'm not sure it is possible to hire them for a field task like this

I'm glad i read this thread from 2009, because it's the same issue i'm having in 2019, often with sip box's (ATA's) but also with an asterisk box (freepbx) working as a proxy but suffering less often.
And thinking about this also a problem at another site with mikrotik router and asterisk, maybe once every couple of months.

At least now i have a possible solution to my issue.

I have had this problem for more than 1 year, I have 4 certifications in Mikrotik, and I can't find a solution for this problem, the next thing I am going to do is move to ubiquiti in its edgerouter range since this does not happen with them and voip is my main business.

We need a better connection tracker more eficient when the connexion is down.

Guys,

Can your SIP application work without the Linux SIP ALG/NAT helper, and if so, have you tried just turning it off...

Code: Select all

[admin@MikroTik] > /ip firewall service-port disable sip

...to see if that fixes the problem for you?

-- Nathan

Disabled the SIP port under IP -> Firewall - > Service Ports (ROS 6.47)
Been working perfectly for a week now, no need to reboot the MT

udp-stream-timeout=1h
loose-tracking=NO (standart tracking)

limit drop-invalid rule; incoming from = WAN interface-list

if you use fast-tracking exclude TCP SYN,FIN,RST packets;
move them before (above) FASTTRACK rule.

works fine for me.