I had a situation today and want to understand why this will happen (Dont have much experience with PPPoE, etc)

The environment is FTTh where customers connect to relevant ISP across Vlan & PPPoE, OLTs connects to CRS on Isolated Trunk (Vlan) Ports, then branches out to the relevant ISPs based on Vlan.ID. From CRS there is another Trunk (Vlan) going to the CPE Management Device.

I noticed on the CPE Management device, that it is receiving PPPoE Session packets on ether1 for one of the ISP vlans 501. did some packet capturing and noticed these all belong to downloads of a certain customer of that ISP in the FTTh network. As per torch screen, it seems these PPPioE Session packets were sent to 0.0.0.0 which seems to be broadcasting this persons downloads out on this vlan.

Reported this to ISP and seems to be resolved now, but did not get any communication back from them on what the issue was and would like to understand what will cause something like this, can anyone please explain to me what / how this is caused?

Anyone? It is happening again, same ISP but different customer in the FTTh network.

Reported to ISP again, but does not seem they know where the problem is, almost like vlan 501 leaking over to native vlan from their side?

Have pcap file if that will help, but I cant see anything funny in it?

Where ether1 connects to ?

Ether1 is connected to crs switch.

I think what is happening is the device not storing end user device MAC address and broadcasting this PPPoE session packets on all ports?

PPP frames inside ethernet providing unique layer 2 tunnel based on unicast frames on session level. Why torch should show you destination IP, when PPP tunnel operates only with mac address?

PPP frames inside ethernet providing unique layer 2 tunnel based on unicast frames on session level. Why torch should show you destination IP, when PPP tunnel operates only with mac address?

Not sure I understand your post, is your question directed at me?

PPP frames inside ethernet providing unique layer 2 tunnel based on unicast frames on session level. Why torch should show you destination IP, when PPP tunnel operates only with mac address?

Not sure I understand your post, is your question directed at me?

Well yeah. I thought you didn't get why dst ip is 0.0.0.0 and you thought it's mean it's broadcast frame.

PPP frames inside ethernet providing unique layer 2 tunnel based on unicast frames on session level. Why torch should show you destination IP, when PPP tunnel operates only with mac address?

Not sure I understand your post, is your question directed at me?

Well yeah. I thought you didn't get why dst ip is 0.0.0.0 and you thought it's mean it's broadcast frame.

Thx, no, maybe my explanation / description was not clear. The problem is I am seeing the 8864 session packets on a device totally unrelated to the client device, so instead of these packets going to specific MAC address, it seems to be broadcasted on the specific VLAN and I can see these packets on multiple devices, if that makes more sense now?

PPP frames inside ethernet providing unique layer 2 tunnel based on unicast frames on session level. Why torch should show you destination IP, when PPP tunnel operates only with mac address?

Not sure I understand your post, is your question directed at me?

Well yeah. I thought you didn't get why dst ip is 0.0.0.0 and you thought it's mean it's broadcast frame.

Thx, no, maybe my explanation / description was not clear. The problem is I am seeing the 8864 session packets on a device totally unrelated to the client device, so instead of these packets going to specific IP / MAC address, it seems to be broadcasted on the specific VLAN and I can see these packets on multiple devices, if that makes more sense now?

Now I think I get it. I think the only way it's possible in ISP network is mac address learning of legit client on your ether1 port. Somehow.

or

it's a bug in ROS that allows you to see PADI frames with 8863 ethernet protocol numbers like 8864. Few months ago I saw a bug that prevent to watch data with torch in PPPoE interface. Try to report that possible bug, maybe Tiks will answer something.

Even it is a bug, then that ISP managed his network wrong, because clients port must be isolated from each others on layer 2 without any vlans. Just isolated by software of a switch.

Thank you @Anumrak,

I will dig a bit further and chat again to ISP....

My two cents:

• the target PPPoE client device doesn't send anything in its uplink direction so the ISP gear starts to broadcast frames for it after the record for that MAC in its forwarding table expires (this normally takes minutes after it has seen the last frame with client's MAC as source), whereas the PPPoE payload is a stream which doesn't require any backward confirmations so it doesn't stop in the absence of an uplink traffic,
• the forwarding table size is so limited at some of the ISP boxes that it starts broadcasting because newer records squeeze out older ones; once any of the boxes between that resource-limited one and you receives such a frame, it has no choice but to broadcast it as it can never see a frame in the opposite direction,
• the ISP gear is merely broken.

I can agree with @Anumrak on almost nothing:

• isolation of client-facing ports on the ISP gear from one another cannot help as it's not that you'd be getting an upload traffic of a foreign client, you get its download traffic, so it doesn't ingress through one client-facing port and egress through another one,
• I cannot imagine 7.3 Mbps of PPPoE-discovery traffic towards a MAC address of a single client (unless the client would be frenetically sending PADI at similar pace)
• if the ISP gear was receiving frames with source MAC address of the client from your gear, you would be stealing part of the stream (and the ISP gear should report MAC flapping if this was happening)

As you are seeing misdirected unicast from a port on your CRS the issue likely lies with the switch forwarding database therein.

I had the same issue with some old Mikrotiks based on AR7240 switch chips where some client MAC addresses on different ports appeared to be hashed to the same value so only one could exist in the FDB causing unicast traffic to flood out of all ports, hopefully the switches in CRS don't suffer from a similar problem.

When the problem arises check if the client MAC address exists in the FDB. Sniffing switch traffic to see if the client traffic is unidirectional (thus causing the FDB entry to age out) would require traffic to be mirrored to the CPU which may affect the switch chip behaviour.

Thx all for the feedback.

@sindy, believe me when I say, your feedback carry way more weight than 2c's

Can you sniff the PPPoE communication of a client which legitimately passes through your gear? My point 1 (stream from PPPoE server being sent although no responses come from the client) is only possible if the server doesn't send any keepalives to check the connection state, or if it does but ignores the absence of keepalive responses (like someone else complains here).

My two cents:

• the target PPPoE client device doesn't send anything in its uplink direction so the ISP gear starts to broadcast frames for it after the record for that MAC in its forwarding table expires (this normally takes minutes after it has seen the last frame with client's MAC as source), whereas the PPPoE payload is a stream which doesn't require any backward confirmations so it doesn't stop in the absence of an uplink traffic,
• the forwarding table size is so limited at some of the ISP boxes that it starts broadcasting because newer records squeeze out older ones; once any of the boxes between that resource-limited one and you receives such a frame, it has no choice but to broadcast it as it can never see a frame in the opposite direction,
• the ISP gear is merely broken.

I can agree with @Anumrak on almost nothing:

• isolation of client-facing ports on the ISP gear from one another cannot help as it's not that you'd be getting an upload traffic of a foreign client, you get its download traffic, so it doesn't ingress through one client-facing port and egress through another one,
• I cannot imagine 7.3 Mbps of PPPoE-discovery traffic towards a MAC address of a single client (unless the client would be frenetically sending PADI at similar pace)
• if the ISP gear was receiving frames with source MAC address of the client from your gear, you would be stealing part of the stream (and the ISP gear should report MAC flapping if this was happening)

About your second cent:

1) It will help alot, especially if both clients in the same broadcast domain. They could interact with one another directly. It's not about direction of traffic. It's about misconfiguration of topic starter and abusing the "network hole" by someone in same vlan.
2) I can generate whatever traffic you want with any ether type number.
3) While mac flapping happening, you can receive that bursted 7 mb/s sometimes.

I dont think that ISP switch is broken, because common access switch can keep 8k macs. Even with hash collision up to 3k.(bad switch, but enough)

I bet that ISP misconfigured port mirroring from some client to their chosen one with another vlan, and picked a port of topic starter. After he called them, they configured mirroring correctly. In this scenario you could see that traffic on a legit port.

1) It will help alot, especially if both clients in the same broadcast domain. They could interact with one another directly. It's not about direction of traffic. It's about misconfiguration of topic starter and abusing the "network hole" by someone in same vlan.

I'm not sure we talk about the same? @CZFan (the topic starter) cannot affect by his own configuration what the upstream ISP is sending to him, except if he was actively sending frames with the affected customer's MAC address as source and thus stealing the traffic by making the ISP switch learn that MAC on @CZFan's port. I would expect the ISP's switch to have client-facing ports isolated from each other (i.e. no forwarding from one client-facing port to another), but that doesn't prevent frames sent by the PPPoE server from being broadcast via all client-facing ports as long as the dst-mac is unknown.

If you start thinking about spoofed traffic, then of course isolation of client-facing ports can prevent frames spoofed by one client from being forwarded to other client-facing ports at the ISP.

2) I can generate whatever traffic you want with any ether type number.

No doubt about this As stated above, this is what port isolation can help against.

3) While mac flapping happening, you can receive that bursted 7 mb/s sometimes.

Sure, but to happen, this requires that the legit recipient doesn't send anything during that burst and that something between @CZFan's uplink port and @CZFan's clients (including both extremities) sends frames with legit recipient's MAC as source.

I bet that ISP misconfigured port mirroring from some client to their chosen one with another vlan, and picked a port of topic starter. After he called them, they configured mirroring correctly. In this scenario you could see that traffic on a legit port.

This is the most straightforward explanation, but as it happened twice for different clients, the ISP's staff must suffer from dysgraphia to make the same typo twice for two different clients to be monitored.

1) It will help alot, especially if both clients in the same broadcast domain. They could interact with one another directly. It's not about direction of traffic. It's about misconfiguration of topic starter and abusing the "network hole" by someone in same vlan.

I'm not sure we talk about the same? @CZFan (the topic starter) cannot affect by his own configuration what the upstream ISP is sending to him, except if he was actively sending frames with the affected customer's MAC address as source and thus stealing the traffic by making the ISP switch learn that MAC on @CZFan's port. I would expect the ISP's switch to have client-facing ports isolated from each other (i.e. no forwarding from one client-facing port to another), but that doesn't prevent frames sent by the PPPoE server from being broadcast via all client-facing ports as long as the dst-mac is unknown.

If you start thinking about spoofed traffic, then of course isolation of client-facing ports can prevent frames spoofed by one client from being forwarded to other client-facing ports at the ISP.

2) I can generate whatever traffic you want with any ether type number.

No doubt about this As stated above, this is what port isolation can help against.

3) While mac flapping happening, you can receive that bursted 7 mb/s sometimes.

Sure, but to happen, this requires that the legit recipient doesn't send anything during that burst and that something between @CZFan's uplink port and @CZFan's clients (including both extremities) sends frames with legit recipient's MAC as source.

I bet that ISP misconfigured port mirroring from some client to their chosen one with another vlan, and picked a port of topic starter. After he called them, they configured mirroring correctly. In this scenario you could see that traffic on a legit port.

This is the most straightforward explanation, but as it happened twice for different clients, the ISP's staff must suffer from dysgraphia to make the same typo twice for two different clients to be monitored.

1) Can affect by his own hands (Why you think that it's a session of a legit ISP client?(mac spoffing)) Also, all clients could know all macs in their service if there is no l2 isolation.
2) Yeah. Better be
3) 7 mb/s is not a big deal for tcp "window" expanding between flapping as a udp stream.

aaaaand not realy. We all do mistakes cause of human factor

@CZFan, @Anumrak's point of view made me review the whole thread and I've noticed I may be misunderstanding some points all the time.

So
1) are the two clients whose traffic you could see to arrive to the "CPE management router" connected via your own OLTs or their MAC addresses are unrelated to your part of the network at all? I.e. can we be sure that the "misforwarding" happens already at the ISP or can it happen as late as at your CRS (because the CRS gets it legitimately)?
2) what exactly did you have in mind when saying "almost like vlan 501 leaking over to native vlan from their side"? The torch shows that those frames do come to the "CPE management router" tagged with VID 501.

@tdw's suggestion regarding collision of hashes of MAC addresses cannot be excluded (neither at your CRS nor at the ISP gear), it is just a bit complex to prove that - to do so, you'd need to store the complete list of MAC addresses known to the switch while the issue happens, and then try to spoof frames with each of these MAC addresses as source, one by one, to another CRS and see which one of them purges (actually, shadows) the MAC address of that "unrelated client" from the FDB.

I am still experiencing the problem, and seeing these packets on ALL devices inside the FTTh network.

Below screenshot from another customer device, seeing all these packets not meant for this device.

I had something strange happen today, an not sure if I can replicate it again, but was trying to MAC Telnet to a MAC address, struggled to get in as it kept saying incorrect password, bet then all of a sudden it accepted it and logged in, but it was not the device I was trying to log into, the device that I managed to log into, had an "Admin MAC" on the bridge interface same as what I was trying to MAC telnet to. Seems MTs are leaking bridge MACs onto WAN ports, etc

@CZFan, @Anumrak's point of view made me review the whole thread and I've noticed I may be misunderstanding some points all the time.

So
1) are the two clients whose traffic you could see to arrive to the "CPE management router" connected via your own OLTs or their MAC addresses are unrelated to your part of the network at all? I.e. can we be sure that the "misforwarding" happens already at the ISP or can it happen as late as at your CRS (because the CRS gets it legitimately)?
2) what exactly did you have in mind when saying "almost like vlan 501 leaking over to native vlan from their side"? The torch shows that those frames do come to the "CPE management router" tagged with VID 501.

@tdw's suggestion regarding collision of hashes of MAC addresses cannot be excluded (neither at your CRS nor at the ISP gear), it is just a bit complex to prove that - to do so, you'd need to store the complete list of MAC addresses known to the switch while the issue happens, and then try to spoof frames with each of these MAC addresses as source, one by one, to another CRS and see which one of them purges (actually, shadows) the MAC address of that "unrelated client" from the FDB.

Hi @sindy, below a screenshot from my CRS of the interface connected to ISP. I do not think that I should be seeing 8864 packets coming in on that port? Hence it looks like it is coming in like that.

Either you didn't understand my question or I misunderstand your topology even deeper than I've expected.

So first - where is the PPPoE server for your clients? Is it your "CPE management router" or are you providing L2 connectivity from your OLT clients to PPPoE servers at ISP1, ISP2, ISP3?

If the clients should talk using PPPoE to a server running at your "CPE management router", it is a pure nonsense that you'd be receiving ethertype 8864 (or any other) frames with dst-mac matching one of your own clients from that ISP. However, it would not be so much of a nonsense if your CRS would be sending such frames to the ISP because it would be broadcasting them for the lack of response or due to the hypothetical hashing issue. And the issue with /tool torch is that it often uses Rx and Tx from the perspective of the remote device, not the local one,and the rules for this are so counter-intuitive that I am never sure which one is right. That's why I prefer to use/tool sniffer quick instead.

Regardless the above, my question remains - does the dst-address of those frames belong to one of the PPPoE clients connected via your OLTs or is it a complete alien in your ecosystem?

To the other issue, the MAC addresses of bridge ports do not "leak" to WAN; it's simply that the mac-server responds to requests coming to MAC of any interface of that Routerboard to which an IP address is attached and which is on the interface list configured for mac-server to listen at, regardless through which port the connection request actually came in. The responses are sent, however, from the MAC of the interface through which the requests came in, so there is no MAC address leakage as such.

But there's another surprising issue, that you've landed at a different machine than you've planned to get to. Are you using locally administered MAC addresses on the bridges, or did Mikrotik deliver two boxes with same vendor-assigned MAC addresses?

Apologies,

I am providing L2 connectivity from clients behind OLTs to ISP's, the PPPoE AC is at the ISP.

Yes, the dst MAC addresses belongs to customers behind the OLTs, traffic that I see, dst MACs changes every now and then, so it is not a specific customer behind OLTs whose traffic I see, but is all from this specific ISP customers. The CRS seems to be receiving info, then it sends it to all Vlans internally.

below a sniffer screenshot on CRS, sfp interface is directly connected to ISP equipment

The team that deployed the CPE devices seemed to have used a script with a Admin MAC set on the Bridge (LAN) interface and used the same script for multiple devices as they deployed the devices, so not fault of Mikrotik.

I am providing L2 connectivity from clients behind OLTs to ISP's, the PPPoE AC is at the ISP.

Yes, the dst MAC addresses belongs to customers behind the OLTs, traffic that I see, dst MACs changes every now and then, so it is not a specific customer behind OLTs whose traffic I see, but is all from this specific ISP customers. The CRS seems to be receiving info, then it sends it to all Vlans internally.

Okay. So the 8864 traffic for that MAC address is coming from the ISP to the CRS legitimately, and the "only" issue is that your "CPE management router" and all the other clients downstream from the CRS receive a copy of traffic of a random victim. I'd say you are lucky ZA is not subject to the famous GDPR

You haven't stated your CRS model, but the Wiki declares 16000+ rows in the unicast forwarding table for both the CRS families (1xx/2xx and 3xx).

Does the number of all PPPoE clients behind the OLTs approach 16000? Even if it doesn't, it may still be an FDB exhaustion issue if one of the ISPs is massively leaking some other traffic to you, but if you have never seen an 8864 traffic on the CPE management router in any other VLAN than 501, I'd stick with my theory that that ISP doesn't use PPPoE keepalives and if the client dies, the PPPoE server keeps on sending because it doesn't notice the client to be down, and when the association of that MAC address to the proper OLT-facing port of the switch expires in the FDB of the switch chip because no frames from the client's MAC address come in through any port, the switch chip starts broadcasting the traffic.

You can easily test that if the ISP in question provides public IP addresses to at least part of the clients. Use a test PPPoE client to establish a connection and get a public IP address from the ISP, start pinging that public IP from outside, and see that the client responds. Sniff at the CPE management router for that MAC address (/tool sniffer quick interface=ether1 mac-address=test:client:address/ff:ff:ff:ff:ff:ff), you should see nothing to come in. Now keep the sniffer runinng, re-start the ping to run forever, and then disconnect the fiber from the test client without powering it down before. If my assumption is correct, in a few minutes you should see 8864 frames in the sniff once per second.

I cannot recommend sniffing on the uplink ports of the CRS - while using /tool torch or /tool sniffer, you force the traffic through the software bridge although I suppose you normally use the "hardware accelerated" forwading (i.e. switch chip forwarding), so sniffing this way very likely throttles the bandwidth of all clients whose traffic crosses the sniffed port, as the CPUs of CRSes are not extremely powerful ones.

The CRS is a 326, max devices / MAC addresses inside the FTTh network is +- 2000.

The concern I have is I have seen on the CPE management device these packets received sometimes reaches 80 - 90 Mb/s, this going all over the internal network not good, and it effectively becomes like a DDOS to customers.

Interesting point you mentioned re Keep Alive Timeout, instead of testing the way you suggested, as I will need to have physical access to client device, I did some sniffing at a client whose data I was seeing, and the client is getting PPP LCP Echo requests and send back echo replies, but strangely it seems to come in randomly, i.e. will be every 10 seconds, and then the next 2 will be every 20 seconds, client responds immediately, 0,0005s but I can't confirm if the PPPoE AC always receives these replies.

What will the benefit be for ISP, etc to disable the "Keep Alive Timeout" settings?

What will the benefit be for ISP, etc to disable the "Keep Alive Timeout" settings?

Who said benefit, I think about bugs, or, more precisely, I'm trying to find a logical explanation why this should happen only with clients of that single ISP, given that the broadcasting itself happens at your CRS which should keep learning the right egress port from the client's traffic in upload direction.

So did you see the keepalive responses while the issue was present for the client sending the keepalives? Or you've seen them to be sent by an affected client but not while its download traffic was being broadcast?

I saw the keep alive responses being sent by the client device at the client device while the issue were present, i.e. I could see PPPoE session traffic to this client on other devices.

What else I noticed short while ago, I looked at the MAC address of this client in host table in bridge, and noticed the age from this MAC in this specific vlan sometimes went to 1 minute before it "resets", indicating that CRS has not received any frames from this MAC on that vlan for quite a while.

So above might be inline with your theory. This ISP has 2 VLAN's, one data and one voice, I am not seeing the long age on the voice vlan, so this indicates physical network / devices are not faulty.

I saw the keep alive responses being sent by the client device at the client device while the issue were present, i.e. I could see PPPoE session traffic to this client on other devices.

I cannot read this differently than that the CRS has a problem.

What else I noticed short while ago, I looked at the MAC address of this client in host table in bridge, and noticed the age from this MAC in this specific vlan sometimes went to 1 minute before it "resets", indicating that CRS has not received any frames from this MAC on that vlan for quite a while.

This might indicate some issue on the path between the client and the CRS (like insufficient bandwidth between the OLT and the CRS to accommodate upload traffic of all clients or mere bit errors causing some frames to be dropped at receiving side on FCS error), but as such it does not explain the broadcast - as long as the record in the FDB exists, even if it is an old one, the frames matching that record should not be broadcast. I've seen dumb switches which didn't update existing records, only create new ones for "unknown yet" MAC addresses, so every now and then a single frame was broadcast in the window between expiration of the old FDB record and creation of a new one, but I have never seen (maybe yet?) a switch which would ignore existing FDB records when forwarding "because they are too old".

So above might be inline with your theory. This ISP has 2 VLAN's, one data and one voice, I am not seeing the long age on the voice vlan, so this indicates physical network / devices are not faulty.

I'm afraid I don't have enough information to agree or disagree here. The voice VLAN traffic may be prioritized in the OLT so the limited bandwidth towards the CRS may not affect it.

Just a question which doesn't actually solve anything, do you use the ISP's data VLAN to connect to the client devices from the CPE MGMT router using mac-telnet or you use tagless frames? It's just that as the CRS uses per-VLAN independent learing, each MAC address occupies one row in the FDB per each VLAN used, including the native one. Also, is PPPoE used also inside the VoIP VLAN or it's a pure L2 so the clients' phones use plain IP in that VLAN and get an address from the ISP by DHCP? In the latter case there may be much more MAC addresses in use, as the phones may theoretically be sending RTP to each other directly, so the CRS might be getting ARP requests from phones behind the ISP uplink. What does /interface bridge host print count-only show?

Host count shows just over 4200.

Not sure if this will help narrowing down where the problem is, but the interface directly connected to the ISP FP Rx count (Fast-Path?) shows exact same amount of traffic I see being "broadcast" to all devices.

Post the CRS config, maybe I can see something there (don't expect too much, though). And the output of /interface bridge port print.

Will do that, but before I do that, my concern is not really why it is being sent to all clients from my CRS as ports going to client devices are trunk ports and contains all VLAN's of all ISPs, so it will make sense why from the CRS.

But for some reason, these frames come in on the wire strange way, i.e. If I sniff interface directly connected to ISP, I can see data for 2 customers inside FTTh network, keep in mind this specific ISP has 700+ customers inside the FTTh network

Post the CRS config, maybe I can see something there (don't expect too much, though). And the output of /interface bridge port print.

I will be very surprised if there is something in the IT world that you can't resolve

But for some reason, these frames come in on the wire strange way, i.e. If I sniff interface directly connected to ISP, I can see data for 2 customers inside FTTh network, keep in mind this specific ISP has 700+ customers inside the FTTh network

It becomes so complicated that I'm close to proposing a Skype call, however do you have in mind you can see only data of two customers out of expected 700 on the ISP upink interface? If so, that's most likely because the MAC addresses of those two customers aren't found in the FDB of the switch chip, so the switch chip broadcasts them not only to all Ethernet ports but also to the CPU port, which is what makes it possible for the CPU to sniff them. If you disabled the "hardware acceleration" on this uplink port, you'd see the traffic of all 700 in the sniff, but you'd throttle the bandwidth for the clients severely as each frame would go etherA -> CPU -> etherB and even if the CPU lane is a 10 Gbit/s one so effectively 5 Gbit/s in such a loopback operation, I doubt the CPU can handle that volume of traffic, given that a quad-core ARM on about the same clock (hAP ac²) glows red at 1 Gbit/s per direction.

I have no clue how the FDB table is partitioned in the switch chip, so if it is not fully dynamic but there is a limit of rows which a single VLAN can use (just a speculation!), there may be an overflow for VLAN 501 whereas others are unaffected. Does the ISP using VLAN 501 happen to have the highest number of clients? You've indicated three ISPs on the initial drawing so it seems it is not the case or it is but just by a narrow margin (2000 = 700 + 2 × 650), but I currently have no better idea.

@sindy, I don't know how to thank you, but again your posts have been extremely helpful and I learned a lot again!

What you are saying about the CPU port makes a lot of sense and answered why I was seeing these frames (I keep forgetting the CPU is also seen as a "port")

Yes, this ISP does have the biggest customer base in FTTh, have 6 ISPs coming in

i will get in touch with Mikrotik support,

i will get in touch with Mikrotik support

I'd neverteless like to see the config

Oh, and one more point - may I have the first byte of the two MAC addresses affected by the issue?

No problem, will post it tomorrow

i will get in touch with Mikrotik support

I'd neverteless like to see the config

As requested, see attached, info is a bit anonymized, so hope it makes sense

Nothing surprising or suspicious in your configuration.

When asking for the first bytes of the MAC addresses of the two clients you could sniff on the sfp-sfpplus2 without disabling hardware acceleration, I had in mind that if someone was tampering with clients' MAC addresses, he could have configured a multicast address by mistake, which would explain why these frames get broadcast to all ports. Because in addition to the dead client and ignored absence of keepalives, this would be the only reason which would explain the switch chip behaviour without using the words "bug" and "FDB exhaustion". And these two reasons can exist independent from each other, you may have some clients using multicast MAC addresses and other clients dying silently.

I'm lacking the knowledge on how the FDBs of the bridge and of the switch chip are linked together (if they are); so purely theoretically there is still a chance that there are more records in the switch FDB than you can see in the bridge FDB.

As for the ignored absence of keepalives, I've come across this related topic recently, otherwise the idea would probably haven't even come to my mind; it became even more interesting once I've noticed that the "501/700 ISP" actually uses a Mikrotik as a PPPoE AC.

@sindy,
If you connect with me on Skype (ID under my profile), I can send the the packet capture file done on sfpplus2 to see if you can see anything strange

This problem has been resolved, caused by a loop in the network

Thx to extremely knowledgeable forum member @sindy.

Hello there
I am facing this problem right now how can I trace or find a loop in the network ?