After digging on the forum and other websites/blogs, I found that EoIP w/ IPSEC and PPTP are the only ones that passed multicast traffic. GRE w/IPSEC, IP Tunnel w/IPSEC and LAN to LAN IPSEC setups work fine
to be able to run multicast routing you need an interface. IPSec tunnel mode or transport mode does not provide that. not even proper routing, but policies.
any kind of IP based peer-to-peer tunnel (GRE, IP-IP, EoIP, etc) will work just fine with PIM as long they are addressed (have valid IPv4 addresses).
client-server tunnels (PPTP, L2TP, SSTP) can also work but usually they need "instantiated" static interfaces, e.g. usernames are mapped to static interfaces
as the "<xxxx>" formatted dynamic interfaces may not be added to any routing protocol as they will disappear upon session disconnect.
L2 tunnels (EoIP) can work with IGMP since they are bridging ethernet.
as long you have your "routed" or "bridged" tunnels, throwing on IPSec as transport will not break anything:
just create a policy that uses transport mode esp for the tunnelling protocol of your choice (i.e. GRE, IP protocol 47) between the
tunnel local and remote address, and that was it. you can add this afterwards, first just use your vanilla tunnel w/o encryption and make
multicast working on that.
multicast traffic requires RPF-check to pass, so the destinations need to know where to find the source.
say your MC source has an IP address of 220.127.116.11 and sends packets to multicast address 18.104.22.168
all the destination devices shall be able to reach 22.214.171.124 or they will not be part of the multicast distribution tree.
e.g. an entry for 126.96.36.199 (or a less specific route) must be in the routing table.
in case you have multiple parallel links, all of the must be configured for PIM and all of them must be able to forward traffic towards the
MC source address.
do it step by step:
1 build tunnels (w/o encryption)
2 adjust routing
3 configure PIM and IGMP
4 test multicast (do a ping from MC source to the MC destination group address) check for packets to arrive at the receivers
5 configure encryption for your tunnels as needed.