Time Sync with SNTP client and IP Cloud Not Working

tuan · May 2, 2020, 9:49am

I have NTP clients configurated to sync time with external servers as follows:

[admin@MikroTik] > /system ntp client print
           enabled: yes
       primary-ntp: 0.0.0.0
     secondary-ntp: 0.0.0.0
  server-dns-names: time.google.com,3.asia.pool.ntp.org,1.asia.pool.ntp.org
              mode: unicast
     poll-interval: 16s
     active-server: 211.233.84.186

[admin@MikroTik] > /ip cloud print
          ddns-enabled: yes
  ddns-update-interval: none
           update-time: yes
        public-address: censored
              dns-name: censored.mynetname.net
                status: updated

Looking at SNTP client, I can see NTP server is resolved correctly but other fields such as Last Update, Last Update From are left blank. The clock never gets synced.

Fasttrack is disabled, all active fasttracked conns are terminated.
Try running Torch on my WAN interface, I can see an outbound NTP packet destinated to SNTP active server IP and a reply packet from NTP server (UDP with both source and destination port 123).
I suspect firewall might be the culprit so I try to log NTP packets. I can log outbound NTP packet in OUTPUT chain but I cannot log the inbound packet no matter what (it is the first rule on ip firewall tables)

Flags: X - disabled, I - invalid, D - dynamic 
 0    ;;; NTP DEBUG 
      chain=input action=log protocol=udp src-port=123 log=yes log-prefix="NTP Debug"
...

[admin@MikroTik] > /ip firewall mangle print
Flags: X - disabled, I - invalid, D - dynamic 
 0    ;;; DEBUG
      chain=input action=log protocol=udp dst-port=123 log=yes log-prefix="NTP Debug" 
...
[admin@MikroTik] > /ip firewall nat print      
Flags: X - disabled, I - invalid, D - dynamic 
 0    chain=srcnat action=masquerade out-interface-list=WAN 

 1 X  ;;; masq. vpn traffic
      chain=srcnat action=masquerade src-address=192.168.89.0/24 log=no log-prefix="" 

 2    ;;; droid port-forwarding
      chain=dstnat action=dst-nat to-addresses=192.168.88.6 protocol=tcp 
      in-interface-list=WAN dst-port=22 log=no log-prefix=""

There is no rules on ip firewall raw tables.

I don’t see anything suspicious. Could this be a bug? Can somebody please suggest how to troubleshoot this issue further?

tuan · May 4, 2020, 6:13am

please, anyone?

Jotne · May 4, 2020, 6:25am

SNTP is not the same as NTP
Do you have the NTP package installed? Look at:
System → Packages
There you should see under name a NTP package.

Not sure if SNTP can respond to other NTP request, it may only be a NTP client and not a server.

macsrwe · May 4, 2020, 6:57am

All ROS can run SNTP client.

To run NTP server you must include ntp package. Then if you also run client, you are running NTP client; which I believe is not the same code as SNTP client, which is unavailable when ntp package is present.

tuan · May 4, 2020, 12:57pm

Thanks jotne & macsrwe for your responses. Much appreciated.

Maybe later I will run NTP server on the router but for now I just want to get the NTP client working. For this purpose, SNTP should do the job just fine, no?

Besides, I have a HAP AC and I don’t think NTP package is available for this router.

[admin@MikroTik] > /system package print
Flags: X - disabled 
 #   NAME                          VERSION                          SCHEDULED              
 0   routeros-mipsbe               6.45.8                                                  
 1   system                        6.45.8                                                  
 2 X ipv6                          6.45.8                                                  
 3   wireless                      6.45.8                                                  
 4   hotspot                       6.45.8                                                  
 5   mpls                          6.45.8                                                  
 6   routing                       6.45.8                                                  
 7   ppp                           6.45.8                                                  
 8   dhcp                          6.45.8                                                  
 9   security                      6.45.8                                                  
10   advanced-tools                6.45.8

Jotne · May 4, 2020, 1:25pm

My bad.

When NTP is not installed, it uses a simple NTP or SNTP, not sure.

You set it up from CLI, did not find any info in WinBox

/system ntp client set enabled=yes
/system ntp client set primary-ntp=1.1.1.1
/system ntp client print

tuan · May 4, 2020, 1:33pm

Yes, it uses SNTP client when NTP package is not installed.
Changing server DNS name to IP doesn’t help either.

[admin@MikroTik] > /system ntp client print
           enabled: yes
       primary-ntp: 1.1.1.1
     secondary-ntp: 0.0.0.0
  server-dns-names: 
              mode: unicast
     poll-interval: 16s
     active-server: 1.1.1.1

For NTP to work the reply NTP packet should hit firewall INPUT chain but for some reason it doesn’t.

sindy · May 4, 2020, 3:37pm

The NTP peer is a separate package which is not part of the basic bundle - to install it, you have to download the archive with the individual modules, unzip it, upload the ntp peer .npk to the device, and reboot the device.

But the actual issue is different - why your SNTP client doesn’t get the response from the server.

I know the following reasons why a packet which can be sniffed on the in-interface may “disappear” before reaching /ip firewall nat filter:

the WAN is a bridge and there is an /interface bridge filter rule which drops the packet (might be your case, you haven’t shown the complete configuration)
the packet gets “un-src-nated” so the dst-address and/or dst-port in filter are different from those seen on the interface (not the case here as the SNTP client sends its queries from UDP port 123 so the responses arrive to that port)
an IPsec policy reverse-matches the packet’s src and dst address, which means that the packet should have come via an IPsec SA (not the case here because if it was, the outgoing query packets would have to go via that hypothetical SA as well, so you wouldn’t see them in plaintext on the interface)
an action=drop rule in /ip firewall raw drops them (not your case either as you’ve expressly stated that /ip firewall raw is empty
a queue drops it because it has exceeded the bandwidth limit (might be your case, you haven’t shown the complete configuration)

I also had cases in the past when packets from virtual interfaces (IPIP tunnels) were disappearing mysteriously, and after reboot everything was OK. I’m unable to tell whether it was related to some RouterOS release or whether I simply haven’t changed the configuration on the affected machines for so long that I haven’t hit that issue again ever since.

So I’d try a reboot as the first step.

Jotne · May 4, 2020, 4:11pm

1.1.1.1 is not an NTP server.

Find a server from this pool.
https://www.pool.ntp.org

macsrwe · May 4, 2020, 7:06pm

One very common reason for NTP replies to fail is that good administration practice requires you to block NTP requests coming from the WAN interfaces. However, if done Incorrectly, this will also block all replies to your own NTP client. The proper security blocking rule includes connection state indicator, allowing related traffic through but blocking new traffic. I cannot remember at the moment whether you block new or block !related, but one of those does the proper thing.

tuan · May 5, 2020, 2:21am

Thank you for your very informative answer. I’ve learned a lot since reading it. After reboot, the problem still persists.

Nothing in the bridge filter appears to be the culprit (wlan3 and wlan4 are guest network AP created by Home AP Dual default config):

[admin@MikroTik] > /interface bridge filter print
Flags: X - disabled, I - invalid, D - dynamic 
 0 I ;;; wlan3 not ready
     ;;; in/out-bridge-port matcher not possible when interface (wlan3) is not slave
     chain=forward action=drop in-interface=wlan3 

 1 I ;;; wlan3 not ready
     ;;; in/out-bridge-port matcher not possible when interface (wlan3) is not slave
     chain=forward action=drop out-interface=wlan3 

 2   chain=forward action=drop in-interface=wlan4 

 3   chain=forward action=drop out-interface=wlan4

Now talking about it, IPSec policy could be the problem. NTP client was working before and I’ve just add some IPSec settings to connect to my VPN provider via IKEv2. I connect and disconnect simply by toggling a specific IPSec peer.

[admin@MikroTik] > /ip ipsec export
/ip ipsec mode-config
add name=windscribe responder=no src-address-list=localsubnet
/ip ipsec policy group
add name=windscribe
/ip ipsec profile
add dh-group=ecp384 enc-algorithm=aes-256 hash-algorithm=sha256 name=windscribe
/ip ipsec peer
add address=sg.windscribe.com disabled=yes exchange-mode=ike2 name=windscribe-sg profile=\
    windscribe
/ip ipsec proposal
add auth-algorithms=sha256 enc-algorithms=aes-256-gcm name=windscribe pfs-group=ecp384
/ip ipsec identity
add auth-method=eap certificate=lets-encrypt-x3-cross-signed.pem.txt_0 eap-methods=\
    eap-mschapv2 generate-policy=port-strict mode-config=windscribe password=Censored peer=\
    windscribe-sg policy-template-group=windscribe remote-id=fqdn:sg.windscribe.com username=\
    Censored
/ip ipsec policy
add dst-address=0.0.0.0/0 group=windscribe proposal=windscribe src-address=0.0.0.0/0 \
    template=yes

Yes I have a Queue but it’s unlikely to be the problem because the traffic is well under limits and I have this problem for a while now.

tuan · May 5, 2020, 2:32am

I have put UDP/123 log rule on top of INPUT chain before ACCEPT related,established,untracked. And I also made sure fasttrack was disabled.

LOL but you suggested it in your previous answer. Doesn’t matter, I have proper NTP servers using DNS name but it didn’t work anyway.
It might be important to point out I have tested all these NTP servers from clients behind the router and they work fine (reply NTP packets hit FORWARD chain as expected).

macsrwe · May 5, 2020, 3:14am

Where it is is only half the question. What it is is the other half. Could you please export your firewall rules and post them here?

Jotne · May 5, 2020, 5:30am

Even better, export all.

/export hide-sensitive

tuan · January 30, 2021, 3:59am

Sorry for taking so long to respond. I didn’t get the notification and I had completely forgotten about this.
The issue remains, however. I even tried to reset the router to default config, only set up a working PPPoE connection for the internet, and tried SNTP. It didn’t work.

I can see the replying UDP packet (UDP dport 123), but it never hits INPUT chain. It works fine with other devices behind the router.

Even /ip cloud ‘update time’ option doesn’t work either. Each time a power outage happens, I have to set up the time manually or SSL dependent services will cease to work. This is very irritating.

macsrwe · January 30, 2021, 4:04am

Advice was to send us a configuration export so we could help you. Still has not been done.

tuan · January 30, 2021, 4:39am

I have since configured it to be a Dual AP using quick set.
Here you go:

[admin@MikroTik] > export hide-sensitive 
# jan/29/2021 14:59:44 by RouterOS 6.48
# software id = XXXXXXX
#
# model = RouterBOARD 962UiGS-5HacT2HnT
# serial number = XXXXX
/interface bridge
add admin-mac=E4:8D:8C:BD:EF:30 auto-mac=no comment=defconf name=bridge
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n channel-width=20/40mhz-XX \
    country="viet nam" disabled=no distance=indoors frequency=auto \
    installation=indoor mode=ap-bridge ssid=Home wireless-protocol=802.11
set [ find default-name=wlan2 ] band=5ghz-a/n/ac channel-width=20/40/80mhz-XXXX \
    country="viet nam" disabled=no distance=indoors installation=indoor mode=\
    ap-bridge ssid=Home_5G wireless-protocol=802.11
/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 name=pppoe-out1 \
    service-name=vnpt use-peer-dns=yes user=nho976ngt
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface wireless security-profiles
set [ find default=yes ] authentication-types=wpa-psk,wpa2-psk mode=\
    dynamic-keys supplicant-identity=MikroTik
add name=profile supplicant-identity=MikroTik
/interface wireless
add disabled=no mac-address=E6:8D:8C:BD:EF:35 master-interface=wlan2 name=wlan3 \
    security-profile=profile ssid=Guest
add disabled=no mac-address=E6:8D:8C:BD:EF:36 master-interface=wlan1 name=wlan4 \
    security-profile=profile ssid=Guest
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip pool
add name=dhcp ranges=192.168.88.10-192.168.88.254
add name=vpn ranges=192.168.89.2-192.168.89.255
/ip dhcp-server
add address-pool=dhcp disabled=no interface=bridge name=defconf
/ppp profile
set *FFFFFFFE local-address=192.168.89.1 remote-address=vpn
/interface bridge filter
add action=drop chain=forward in-interface=wlan3
add action=drop chain=forward out-interface=wlan3
add action=drop chain=forward in-interface=wlan4
add action=drop chain=forward out-interface=wlan4
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=sfp1
add bridge=bridge comment=defconf interface=wlan1
add bridge=bridge comment=defconf interface=wlan2
add bridge=bridge interface=wlan3
add bridge=bridge interface=wlan4
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface l2tp-server server
set enabled=yes use-ipsec=yes
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=pppoe-out1 list=WAN
/interface pptp-server server
set enabled=yes
/interface sstp-server server
set default-profile=default-encryption enabled=yes
/interface wireless access-list
add ap-tx-limit=10000000 interface=wlan4
add ap-tx-limit=10000000 interface=wlan3
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
    192.168.88.0
/ip cloud
set ddns-enabled=yes
/ip dhcp-client
add comment=defconf interface=ether1
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=accept chain=input comment="allow IPsec NAT" dst-port=4500 protocol=\
    udp
add action=accept chain=input comment="allow IKE" dst-port=500 protocol=udp
add action=accept chain=input comment="allow l2tp" dst-port=1701 protocol=udp
add action=accept chain=input comment="allow pptp" dst-port=1723 protocol=tcp
add action=accept chain=input comment="allow sstp" dst-port=443 protocol=tcp
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=\
    invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" \
    connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=\
    out,none out-interface-list=WAN
add action=masquerade chain=srcnat comment="masq. vpn traffic" src-address=\
    192.168.89.0/24
/ip upnp
set enabled=yes
/ip upnp interfaces
add interface=bridge type=internal
add interface=pppoe-out1 type=external
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
add address=::224.0.0.0/100 comment="defconf: other" list=bad_ipv6
add address=::127.0.0.0/104 comment="defconf: other" list=bad_ipv6
add address=::/104 comment="defconf: other" list=bad_ipv6
add address=::255.0.0.0/104 comment="defconf: other" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=\
    33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=\
    invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" \
    src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" \
    dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=!LAN
/ppp secret
add name=vpn
/system clock
set time-zone-name=Asia/Bangkok
/system ntp client
set enabled=yes primary-ntp=129.6.15.28 secondary-ntp=132.163.96.5
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
[admin@MikroTik] > /log print follow where topics~"ntp"
11:15:17 ntp,debug,packet     VN=4 
11:15:17 ntp,debug,packet     Mode=3 (Client) 
11:15:17 ntp,debug,packet    TransmitTimestamp=e3bf5bd5665247cb 
11:15:17 ntp,debug Wait for 16 seconds before sending next message 
11:15:33 ntp,debug Wait for 16 seconds before sending next message 
11:15:49 ntp,debug,packet sending to 129.6.15.28 NTP packet (48 bytes) 
11:15:49 ntp,debug,packet     VN=4 
11:15:49 ntp,debug,packet     Mode=3 (Client) 
11:15:49 ntp,debug,packet    TransmitTimestamp=e3bf5bf56586ec17 
11:15:49 ntp,debug Wait for 16 seconds before sending next message 
11:16:05 ntp,debug Wait for 16 seconds before sending next message 
11:16:21 ntp,debug,packet sending to 132.163.96.5 NTP packet (48 bytes) 
11:16:21 ntp,debug,packet     VN=4 
11:16:21 ntp,debug,packet     Mode=3 (Client) 
11:16:21 ntp,debug,packet    TransmitTimestamp=e3bf5c1565794a6e 
11:16:21 ntp,debug Wait for 16 seconds before sending next message 
11:16:37 ntp,debug Wait for 16 seconds before sending next message 
11:16:53 ntp,debug,packet sending to 129.6.15.28 NTP packet (48 bytes) 
11:16:53 ntp,debug,packet     VN=4 
11:16:53 ntp,debug,packet     Mode=3 (Client) 
11:16:53 ntp,debug,packet    TransmitTimestamp=e3bf5c3567cdcca7 
11:16:53 ntp,debug Wait for 16 seconds before sending next message 
11:17:09 ntp,debug Wait for 16 seconds before sending next message

[admin@MikroTik]/log> /tool torch pppoe-out1 ip-protocol=udp port=123 src-address=0.0.0.0/0
[admin@MikroTik]
MAC-PROTOCOL    IP-PROTOCOL SRC-ADDRESS                                                   SRC-PORT                      DST-PORT                              TX         RX TX-PACKETS RX-PACKETS
ip              udp         132.163.96.5                                                  123 (ntp)                     123 (ntp)                         608bps       0bps          1          0
                                                                                                                                                          608bps       0bps          1          0

tuan · April 12, 2021, 5:32am

After some researching, here are my conclusions:

SNTP client is broken out of the box on HAP AC.
DoH stops working on HAP AC after power loss (and possibly on other devices without a battery too) because DoH replies on HTTPS, and HTTPS doesn’t function without the correct system time. Without a working DNS, IP Cloud also stops working which breaks the Cloud Timesync.

I had to disable DoH and SNTP client to get IP Cloud Timesync working again. This is a big issue.

mkx · April 12, 2021, 5:39am

Actually cloud timesync is broken. I’ve read explanation by Mikrotik that cloud timesync is very approximate and only useful for setting approximate time for logs. For everything else disable cloud timesync and use (S)NTP client. In fact you should only use single time sync method as multiple fight against each other.
And yes, after reboot it can take some time (less than a minute though) for (S)NTP client to set precise date and time.

One more thing: does DHCP client, run on ether1, also set default route? As you’re using PPPoE as WAN connection, routes provided by DHCP client might break things (e.g. outbound connections started before pppoe-out1 establishes).

tuan · April 12, 2021, 11:14am

Thanks for your input.
From my experience, cloud timesync is good enough for TLS which is good enough for me. I don’t really have other options because SNTP client doesn’t work (see previous posts). NTP package is not available for HAP AC.
This is not about SNTP not working right after boot, but rather SNTP not working at all.

DHCP client running on eth1 is disabled, enabling it changes nothing. I have an ISP router act as an optical media converter. PPPoE connection is handled by HAP AC.