Multicast across switches does not work (CRS3xx)

subway · Wed Jan 31, 2024 1:38 pm

Hi,

I am trying to implement this example:

https://help.mikrotik.com/docs/pages/vi ... nwithVLANs

only difference is the multicast VLAN ID (201 instead of 10), and that I have no PCs (so no VLAN 20), just the single multicast VID (201 plus the management VID (99). Everything is configured exactly as the example states, yet multicast does not work on the second switch. On the first switch everything works as expected, plus the unicast part does work on both switches. It seems like the problem is between the two switches (CRS326 and CRS328) both of them are on ROS 7.13

I rechecked the config multiple times based on the example, but I cant seem to find the issue...

Any and all help is appreciated.

Fri Feb 02, 2024 5:13 pm

multicast does not work on the second switch

You're going to have to get a lot more specific than that if you want a solution. There are a bunch of common multicast protocols, and they don't all work the same way when it comes to how you configure your switches and routers to accommodate them. Which protocol? Where are the clients and the servers? If the client is on the first switch and the server on the second, that matters. If there is no server but a continuous multicast stream from outside both switches, that matters, too.

This isn't going to go well if you make me pry each key detail out of you one by one.

It's better to tell too much than too little when asking others to fill voids in your knowledge. If you knew what the "key details" were, you'd likely already have this solved, right?

If I'm reduced to wild guessing, I'd say the IGMP group join messages aren't getting from one switch to the other, so the MDB isn't being updated to send streams over the VLAN trunk between the switches. Part of "multicast flood = no" is that IGMP doesn't get across that trunk link all by itself, either. An IGMP message is itself multicast! No IGMP, no multicast except by flooding.

Furthermore, you have the warning on that page about the basic IGMP querier in ROS 7 being unable to handle VLAN tags. You might be able to set up an IGMP repeater to get the messages across, even write a blind switch rule that mirrors them across. The effectiveness of that will depend on the protocols involved, which once again you have not provided.

And if your needs are more complex, you might be forced to implement PIM-SM, which is overkill for a single VLAN, but by its nature does not have the limitation regarding VLAN tags.

subway · Fri Feb 16, 2024 6:31 pm

Hi Tangent,

First of all, thanks for taking the time.

I attached an image about the architecture (please ignore the top right corner).

This is IGMPv2, and not IPTV at all, but digitized radio data (IQ). The multicast server spits out two different types of multicast and unicast traffic untagged, that needs to be delivered to the IGMP clients in a way that one of them is untagged (PVID=vlan46) the other one is tagged (vlan48). The VLANs dont need to be able to talk to each other in any way (nor multicast nor unicast), and this is all L2 from the switches point of view. Each IGMP client will join multiple groups on the two VLANs (PVID and tagged). And there can be many more IGMP clients on both switches, same exact VLAN layout though. There is not a single IP configured on any of the switches, except for the mgmt vlan (99).

Interestingly: with only one switch everything works as expected even with "unknown multicast flood = no", the problem is clearly the inter-switch link.

Manual IGMP group config is not an option, as based on traffic and mobility, the IGMP clients will join and de-join certain groups, we cannot tell upfront which groups will be used across the switches and which groups wont be used.

Furthermore, you have the warning on that page about the basic IGMP querier in ROS 7 being unable to handle VLAN tags.

I think this is at least partially the issue. I tried to set up PIM between the two switches but the ROS7 example page is really does not tell a lot of details about how to do that properly. It kinda worked, but every hour joined groups were dropped for a few seconds or so. It is unclear for example if the PIM interface needs to be added for each VLAN, and they need to be able to talk to each other in a unicast fashion (they need interface IPs to do that), it is unclear that with PIM set up do I need to disable the multicast querier on the first switch, or do I need to disable snopping on any of the switches?

Hope this is enough details.

Thanks again for helping me.

Sat Feb 17, 2024 8:20 am

this is all L2 from the switches point of view.

Kinda true, but technically not. This is one of the many areas where the OSI model breaks down. Multicast addresses like 239.255.1.2 are clearly L3, but part of what IGMP snooping does is remember what it's seen at that level and maintain the MDB, which is based on (L2) Ethernet multicast addresses, not IP multicast addresses. Atop all that, there isn't a 1:1 mapping between the two.

I do believe I get what you're saying, but hear what I am saying in return: don't get caught up in L2 vs L3, and don't be needlessly exclusionary about it. The real world isn't nearly as simple as the ivory tower people designing X.25 would have wanted, had their gotten their way.

Interestingly: with only one switch everything works as expected even with "unknown multicast flood = no", the problem is clearly the inter-switch link.

It is possible that you have discovered a bug in the multicast implementation. A reply from an MT support request I put in recently about the multicast docs was put off on the grounds that they didn't want to do it until multicast was "fixed and fully functional in v7." So implicitly, it's currently broken and incomplete? Sad if true.

If you can boil your test case down to something a support engineer can try in a few minutes, reporting it might yield a solution faster than trying to hack around it with the help of forum people like me, who are not MT support.

PIM…kinda worked, but every hour joined groups were dropped for a few seconds or so.

That's encouraging. It tells me you're on the right track, and you merely need to find and fix the cause of that last dropout.

More, I suspect you know what the culprit is…

do I need to disable the multicast querier on the first switch

Your symptom does sound a lot like a misconfigured querier. Its whole job is to pinch off unwanted multicast streams. My only uncertainty comes from the fact that your reported "every hour" is much longer than typical querier timeouts. It's more typically down in the 1-15 minutes range, although I will concede that retries might multiply that by a factor of 3-ish. I can therefore believe 45 minutes, but 1 hour…? Only if you increased the initial querier timeouts into the 20-30 minutes range.

One expedient option you might have is to increase the timeouts further. If it only has to work for one broadcast day, a dropout of a few seconds every night in the small hours might not matter.

Or, you could disable the querier entirely. That does mean that over time, streams will flood ports that haven't been interested in the stream for days, months, or years, but a periodic reboot of the switches would clear that out. With unknown multicast flood off, ports will start out quiescent and slowly get noisier until it's time to reset things again.

do I need to disable snopping on any of the switches?

No. To return to your flawed conception above, snooping is L2, PIM is L3.

As a rule, you want one querier per network, and one only. I think that means one on each side of the PIM boundary, but I confess that I'm more of a multicast user than a multicast engineer. I know how it's supposed to work when properly configured, but I have never managed a large multicast network myself, RouterOS or no.

subway · Sat Feb 17, 2024 5:42 pm

There is an interesting phenomena happening (no PIM):

As before, all ports are set to "unknown multicast flood = no" to make sure multicast really works and we are not broadcasting.

On the second switch (where the issue is), from time to time IGMP clients can join multicast groups on both the PVID and the tagged VLAN, MDB has the correct entries, but after a while they got dropped. I checked and both switches have the default IGMP snooping parameters, no difference there.

I attached the bridge status page, can you please check if the multicast querier is set correctly? (the config is based on the official MikroTik example btw). That 0.0.0.0 address is a bit worrying. Does the querier needs some IPs set on the two bridges so they can talk to each other unicast? In the example, only the MGMT vlan has IP, nothing else...

Theoretically, this setup would not need PIM as I dont want to router multicast traffic in any way, even the two VLANs dont need to talk to each other, but that unclear statement on the IGMP MLD exmaple ("Bridge IGMP querier implementation can only send untagged IGMP queries"),

Sat Feb 17, 2024 5:54 pm

Both IGMP and PIM-SM are IP-based protocols, so yes, I do expect that having no IP on the second switch is a problem.

subway · Sat Feb 17, 2024 6:13 pm

Even if I set IPs on the two switches (they can ping each other), the second switch still shows "0.0.0.0" as the multicast querier.

MOD:

OK, I set up PIM this way:

Both switches has 1-1 bridge, both are PVID46, unicast IPs are set: 10.10.10.1 and 10.10.10.2
The PIM instances has the bridge as the IF template in it, PIM priority is set in a way that Switch1 is the designated router (DR).

The multicast querier shows the correct IF and IP of switch2.

The second switch PIM UIB-G and UIB SG shows the correct two IGMP groups for PVID46.

As I did not added the VLAN48 (tagged) traffic to the PIM interface template, those are not tracked by PIM, yet they actually work correctly, they are present in the MDB as "dynamically learned". So apparently ROS can forward and learn properly on VLAN tagged traffic, but not on untagged (PVID) traffic...

Display bug in winbox: PIM-SM --> Neighbor --> Timeout value keeps decreasing until 0, then it starts counting forward. Closing and opening the neighbor does not help, you need to switch tabs then go back and then it displays correctly. This timeout value should reset every time a hello is received from the neighbor.

subway · Sat Feb 17, 2024 9:17 pm

Well, every 1-2 hours the IGMP traffic is still dropped, now I have an extended log:

21:10:16 route,pim,packet instance { 0 IP } interface { bridge } GMP: Sending general query
 21:10:16 route,pim,packet instance { 0 IP } interface { bridge } GMP: Received query from 10.10.10.2
 21:10:16 route,pim,packet instance { 0 IP } interface { bridge } GMP: Received query from 10.10.10.1
 21:10:16 route,pim,packet instance { 0 IP } interface { bridge } GMP: Received v3 membership report from 10.10.10.2
 21:10:16 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 224.0.0.22 with sources { }
 21:10:16 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 224.0.0.13 with sources { }
 21:10:16 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 224.0.0.106 with sources { }
 21:10:16 route,pim,packet instance { 0 IP } interface { bridge } neighbor { 10.10.10.1 } hello received
 21:10:17 route,pim,packet instance { 0 IP } interface { bridge } GMP: Received v3 membership report from 10.10.10.1
 21:10:17 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 224.0.0.22 with sources { }
 21:10:17 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 224.0.0.13 with sources { }
 21:10:17 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 224.0.0.106 with sources { }
 21:10:17 route,pim,packet instance { 0 IP } interface { bridge } GMP: Received v3 membership report from 10.223.1.24
 21:10:17 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 239.0.1.0 with sources { }
 21:10:19 route,pim,packet instance { 0 IP } interface { bridge } GMP: Received v3 membership report from 10.223.1.21
 21:10:19 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 239.0.1.0 with sources { }
 21:10:21 route,pim,packet instance { 0 IP } interface { bridge } GMP: Received v3 membership report from 10.223.1.25
 21:10:21 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 239.0.1.0 with sources { }
 21:10:34 route,pim,packet instance { 0 IP } interface { bridge } sending Hello
 21:10:46 route,pim,packet instance { 0 IP } interface { bridge } neighbor { 10.10.10.1 } hello received
 21:11:04 route,pim,packet instance { 0 IP } interface { bridge } sending Hello
 21:11:16 route,pim,packet instance { 0 IP } interface { bridge } neighbor { 10.10.10.1 } hello received
 21:11:34 route,pim,packet instance { 0 IP } interface { bridge } sending Hello
 21:11:46 route,pim,packet instance { 0 IP } interface { bridge } neighbor { 10.10.10.1 } hello received
 21:12:04 route,pim,packet instance { 0 IP } interface { bridge } sending Hello
 21:12:16 route,pim,packet instance { 0 IP } interface { bridge } neighbor { 10.10.10.1 } hello received
 21:12:34 route,pim,packet instance { 0 IP } interface { bridge } sending Hello
 21:12:46 route,pim,packet instance { 0 IP } interface { bridge } neighbor { 10.10.10.1 } hello received
 21:13:04 route,pim,packet instance { 0 IP } interface { bridge } sending Hello
 21:13:16 route,pim,packet instance { 0 IP } interface { bridge } neighbor { 10.10.10.1 } hello received
 21:13:34 route,pim,packet instance { 0 IP } interface { bridge } sending Hello
 21:13:46 route,pim,packet instance { 0 IP } interface { bridge } neighbor { 10.10.10.1 } hello received
 21:14:04 route,pim,packet instance { 0 IP } interface { bridge } sending Hello
 21:14:16 route,pim,packet instance { 0 IP } interface { bridge } neighbor { 10.10.10.1 } hello received
 21:14:31 route,pim,packet instance { 0 IP } interface { bridge } GMP: Received query from 10.10.10.1
 21:14:31 route,pim,packet instance { 0 IP } interface { bridge } GMP: Received v3 membership report from 10.10.10.2
 21:14:31 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 224.0.0.22 with sources { }
 21:14:31 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 224.0.0.13 with sources { }
 21:14:31 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 224.0.0.106 with sources { }
 21:14:34 route,pim,packet instance { 0 IP } interface { bridge } sending Hello
 21:14:35 route,pim,packet instance { 0 IP } interface { bridge } GMP: Received v3 membership report from 10.10.10.1
 21:14:35 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 224.0.0.22 with sources { }
 21:14:35 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 224.0.0.13 with sources { }
 21:14:35 route,pim,packet instance { 0 IP } interface { bridge } GMP: Change IS EXCLUDE for 224.0.0.106 with sources { }
 21:14:41 route,pim,debug instance { 0 IP } uib { grp: 239.0.1.0 } update
 21:14:41 route,pim,info instance { 0 IP } uib { grp: 239.0.1.0 } destroyed
 21:14:46 route,pim,packet instance { 0 IP } interface { bridge } neighbor { 10.10.10.1 } hello received
 21:15:04 route,pim,packet instance { 0 IP } interface { bridge } sending Hello
 21:15:16 route,pim,packet instance { 0 IP } interface { bridge } neighbor { 10.10.10.1 } hello received
 21:15:34 route,pim,packet instance { 0 IP } interface { bridge } sending Hello
 21:15:46 route,pim,packet instance { 0 IP } interface { bridge } neighbor { 10.10.10.1 } hello received

Who sees where the failure is?

21:10:16 - GMP: Sending general query
21:14:31 - we should see the next "GMP: Sending general query" but that does not happen (missing)
21:14:41 - uib { grp: 239.0.1.0 } destroyed <-- this is the reason of the multicast traffic dropping, but not the cause. The cause is that there is a missing general query, therefor the multicast clients are not polled, therefor grp: 239.0.1.0 times out and got removed.

And looking at the full log, this happens occasionally, but not consistently every x minutes or y hours.

Is this a bug in the PIM-SM or IGMP implementation?

subway · Sun Feb 18, 2024 11:43 am

I let the system run for the night, this is the result:

Line 1002: 00:00:42 route,pim,info instance { 0 IP } uib { grp: 239.0.1.0 } destroyed 
Line 1407: 00:51:43 route,pim,info instance { 0 IP } uib { grp: 239.0.1.0 } destroyed 
Line 1675: 01:25:43 route,pim,info instance { 0 IP } uib { grp: 239.0.1.0 } destroyed 
Line 1807: 01:42:42 route,pim,info instance { 0 IP } uib { grp: 239.0.1.0 } destroyed 
Line 1939: 01:59:41 route,pim,info instance { 0 IP } uib { grp: 239.0.1.0 } destroyed 
Line 2003: 02:08:14 route,pim,info instance { 0 IP } uib { grp: 239.0.1.0 } destroyed 
Line 2068: 02:16:43 route,pim,info instance { 0 IP } uib { grp: 239.0.1.0 } destroyed 
Line 4989: 08:22:09 route,pim,info instance { 0 IP } uib { grp: 239.0.1.0 } destroyed 
Line 5257: 08:56:10 route,pim,info instance { 0 IP } uib { grp: 239.0.1.0 } destroyed 
Line 5354: 09:08:59 route,pim,info instance { 0 IP } uib { grp: 239.0.1.0 } destroyed

Each and every time there was a missing "GMP: Sending general query" when the grp: 239.0.1.0 got destroyed, so it is not a IGMP client malfunction that it did not responded to general query.

This particular group (grp: 239.0.1.0) is also a good candidate for testing, as it provides sync to the IGMP clients therefor the clients never leave this group.

The question is still the same: why there are occasionally missing "GMP: Sending general query" messages not sent out by the PIM instance?

Sun Feb 18, 2024 1:18 pm

It sounds like you’ve got enough evidence to take to MT support.

subway · Tue Feb 20, 2024 4:55 pm

A ticket is open, if we have some conclusions, I will share it here as well.

subway · Wed Mar 27, 2024 5:15 pm

SUP-144274 is created on February 19th, no response ever since "waiting for support".

I updated the ticket that the problem is exactly the same on ROS 7.14.1.

Multicast across switches does not work (CRS3xx)

Multicast across switches does not work (CRS3xx)

Re: Multicast across switches does not work (CRS3xx)

Re: Multicast across switches does not work (CRS3xx)

Re: Multicast across switches does not work (CRS3xx)

Re: Multicast across switches does not work (CRS3xx)

Re: Multicast across switches does not work (CRS3xx)

Re: Multicast across switches does not work (CRS3xx)

Re: Multicast across switches does not work (CRS3xx)

Re: Multicast across switches does not work (CRS3xx)

Re: Multicast across switches does not work (CRS3xx)

Re: Multicast across switches does not work (CRS3xx)

Re: Multicast across switches does not work (CRS3xx)

Who is online