Hi all, first of all I need to apologize for my poor grammar and lack of networking skills. After a lot of tries I’ve managed to have a GRE tunnel linking two sites, lets call them site A and B, to share resources (disk and voip server) located at site A to the new site B. The link was working almost flawlessly for several months (in rare occasions we’ve lost connectivity, just disabling and enabling the tunnel used to be enough to restore functionality). A couple of weeks ago we were victims of RouterOS vulnerabilities in both sites, but during the attack and after fixing the problem, the tunnel was running. A couple of days ago there was a power outage, and as far as I know, those were the only issues with the environment.
Now, after restarting the routers several times and performing the disable/enable tunnel trick, we can’t get packages going through the tunnel. Connection seems to be established, but all traffic is either being rejected or ignored.
Context:
Site A: public IP 190.111.232.222
Local IP 10.10.1.1
Tunnel address: 172.16.3.1
Site B: public IP 190.104.232.136
Local IP 10.11.1.1
Tunnel address 172.16.3.2
Status: Site A shows at Interfaces | GRE Tunnel 0bps Tx and Rx
Site B shows bytes being sent, nothing received.
Using Tool | Ping from site A, setting source address 10.10.1.1 and destination 10.11.1.1 packets are rejected. From site B changing source and destination accordingly, I’m having 100% timeouts.
From the log it seems to me that both routers are seeing each other (copying just one log, but it is the same at the other side):
Nov/15/2018 12:36:49 ipsec,debug ===== received 92 bytes from 190.111.232.222[500] to 190.104.232.136[500]
Nov/15/2018 12:36:49 ipsec,debug,packet 7b381d2c 19cc1581 c3418b3c 16fdbb35 08100501 9a61aec7 0000005c ef8fe31d
Nov/15/2018 12:36:49 ipsec,debug,packet 254546a5 6e323f03 b94f14ad c68d09c1 0478a555 e4bcf21a 09ed2bae ce03b464
Nov/15/2018 12:36:49 ipsec,debug,packet 9873d0bb a8623d09 60dd6434 8706c686 2cc27282 6a874ce4 ed243e99
Nov/15/2018 12:36:49 ipsec,debug receive Information.
Nov/15/2018 12:36:49 ipsec,debug compute IV for phase2
Nov/15/2018 12:36:49 ipsec,debug phase1 last IV:
Nov/15/2018 12:36:49 ipsec,debug 87d1027c 8d202aa2 a8890c39 dd8e6792 9a61aec7
Nov/15/2018 12:36:49 ipsec,debug hash(sha1)
Nov/15/2018 12:36:49 ipsec,debug encryption(aes)
Nov/15/2018 12:36:49 ipsec,debug phase2 IV computed:
Nov/15/2018 12:36:49 ipsec,debug 4da220e8 914d662e 37b75ce8 dee822fd
Nov/15/2018 12:36:49 ipsec,debug encryption(aes)
Nov/15/2018 12:36:49 ipsec,debug IV was saved for next processing:
Nov/15/2018 12:36:49 ipsec,debug 8706c686 2cc27282 6a874ce4 ed243e99
Nov/15/2018 12:36:49 ipsec,debug encryption(aes)
Nov/15/2018 12:36:49 ipsec,debug with key:
Nov/15/2018 12:36:49 ipsec,debug a06aed9f 84b3017f 9660d275 622d3abe
Nov/15/2018 12:36:49 ipsec,debug decrypted payload by IV:
Nov/15/2018 12:36:49 ipsec,debug 4da220e8 914d662e 37b75ce8 dee822fd
Nov/15/2018 12:36:49 ipsec,debug decrypted payload, but not trimed.
Nov/15/2018 12:36:49 ipsec,debug 0b000018 7c922ed4 c19652da fbb6d1ce e45ca060 919f9c49 00000020 00000001
Nov/15/2018 12:36:49 ipsec,debug 01108d28 7b381d2c 19cc1581 c3418b3c 16fdbb35 000008a4 299f3b3f d3ce5507
Nov/15/2018 12:36:49 ipsec,debug padding len=8
Nov/15/2018 12:36:49 ipsec,debug skip to trim padding.
Nov/15/2018 12:36:49 ipsec,debug decrypted.
Nov/15/2018 12:36:49 ipsec,debug 7b381d2c 19cc1581 c3418b3c 16fdbb35 08100501 9a61aec7 0000005c 0b000018
Nov/15/2018 12:36:49 ipsec,debug 7c922ed4 c19652da fbb6d1ce e45ca060 919f9c49 00000020 00000001 01108d28
Nov/15/2018 12:36:49 ipsec,debug 7b381d2c 19cc1581 c3418b3c 16fdbb35 000008a4 299f3b3f d3ce5507
Nov/15/2018 12:36:49 ipsec,debug HASH with:
Nov/15/2018 12:36:49 ipsec,debug 9a61aec7 00000020 00000001 01108d28 7b381d2c 19cc1581 c3418b3c 16fdbb35
Nov/15/2018 12:36:49 ipsec,debug 000008a4
Nov/15/2018 12:36:49 ipsec,debug hmac(hmac_sha1)
Nov/15/2018 12:36:49 ipsec,debug HASH computed:
Nov/15/2018 12:36:49 ipsec,debug 7c922ed4 c19652da fbb6d1ce e45ca060 919f9c49
Nov/15/2018 12:36:49 ipsec,debug hash validated.
Nov/15/2018 12:36:49 ipsec,debug begin.
Nov/15/2018 12:36:49 ipsec,debug seen nptype=8(hash) len=24
Nov/15/2018 12:36:49 ipsec,debug seen nptype=11(notify) len=32
Nov/15/2018 12:36:49 ipsec,debug succeed.
Nov/15/2018 12:36:49 ipsec,debug 190.111.232.222 notify: R_U_THERE
Nov/15/2018 12:36:49 ipsec,debug 190.111.232.222 DPD R-U-There received
Nov/15/2018 12:36:49 ipsec,debug compute IV for phase2
Nov/15/2018 12:36:49 ipsec,debug phase1 last IV:
Nov/15/2018 12:36:49 ipsec,debug 87d1027c 8d202aa2 a8890c39 dd8e6792 cdbee060
Nov/15/2018 12:36:49 ipsec,debug hash(sha1)
Nov/15/2018 12:36:49 ipsec,debug encryption(aes)
Nov/15/2018 12:36:49 ipsec,debug phase2 IV computed:
Nov/15/2018 12:36:49 ipsec,debug 34be13d3 6b1194a0 9138c3a2 6e15324d
Nov/15/2018 12:36:49 ipsec,debug HASH with:
Nov/15/2018 12:36:49 ipsec,debug cdbee060 00000020 00000001 01108d29 7b381d2c 19cc1581 c3418b3c 16fdbb35
Nov/15/2018 12:36:49 ipsec,debug 000008a4
Nov/15/2018 12:36:49 ipsec,debug hmac(hmac_sha1)
Nov/15/2018 12:36:49 ipsec,debug HASH computed:
Nov/15/2018 12:36:49 ipsec,debug 498a213d 8e377ee1 4d43c1af 1f8dce5c 42b27f27
Nov/15/2018 12:36:49 ipsec,debug begin encryption.
Nov/15/2018 12:36:49 ipsec,debug encryption(aes)
Nov/15/2018 12:36:49 ipsec,debug pad length = 8
Nov/15/2018 12:36:49 ipsec,debug 0b000018 498a213d 8e377ee1 4d43c1af 1f8dce5c 42b27f27 00000020 00000001
Nov/15/2018 12:36:49 ipsec,debug 01108d29 7b381d2c 19cc1581 c3418b3c 16fdbb35 000008a4 c234ad79 7dbac407
Nov/15/2018 12:36:49 ipsec,debug encryption(aes)
Nov/15/2018 12:36:49 ipsec,debug with key:
Nov/15/2018 12:36:49 ipsec,debug a06aed9f 84b3017f 9660d275 622d3abe
Nov/15/2018 12:36:49 ipsec,debug encrypted payload by IV:
Nov/15/2018 12:36:49 ipsec,debug 34be13d3 6b1194a0 9138c3a2 6e15324d
Nov/15/2018 12:36:49 ipsec,debug save IV for next:
Nov/15/2018 12:36:49 ipsec,debug 0635291e ea0b2200 f186781a 720a2fa8
Nov/15/2018 12:36:49 ipsec,debug encrypted.
Nov/15/2018 12:36:49 ipsec,debug 92 bytes from 190.104.232.136[500] to 190.111.232.222[500]
Nov/15/2018 12:36:49 ipsec,debug 1 times of 92 bytes message will be sent to 190.111.232.222[500]
Nov/15/2018 12:36:49 ipsec,debug,packet 7b381d2c 19cc1581 c3418b3c 16fdbb35 08100501 cdbee060 0000005c 194e4c0a
Nov/15/2018 12:36:49 ipsec,debug,packet a4462ff5 bbe58a38 f6f9c325 e1df64b4 462211d4 88173699 d408310f 9562c22f
Nov/15/2018 12:36:49 ipsec,debug,packet dfd8ee19 246230f6 70fd32f8 0635291e ea0b2200 f186781a 720a2fa8
Nov/15/2018 12:36:49 ipsec,debug sendto Information notify.
Nov/15/2018 12:36:49 ipsec,debug received a valid R-U-THERE, ACK sent
What would be the best way to troubleshoot this? Would help if I add the configuration for the routers?
Thanks a lot in advance…