Hi all,
Let me first say thank you for this great forum. It has taught me a lot. I come from an ISP network background and need some with a Mikrotik setup for large estate (4000 units) that has a Ubiquity GPON network for customer internet connectivity. My equipment is as follows, 2x CCR 2004, 2x CRS 326, 4x Ubiquity GPON OLTs and all apartments have ONUs and a router(Ubiquity LOCOs) and managed via SPLYNX for customer connectivity management. I am a Cisco and Arista engineer and i am battling with the Mikrotiks to be honest. I have two CCR routers 2004, these will be setup for redundancy with VRRP between the routers and i will setup the failover for the two routers once this is working. These routers feed the two CRS 326 switches also setup as a bridge with some ports dedicated as access ports and some as trunks to feed to the CCRs and the Ubiquity OLTs. Each Vlan has an IP assigned on the interface (SVI). What i am trying to achieve is by setting the native vlan on the ONU it will pass the traffic to the correct FNO/ISP. This is being done now to allow additional ISPs to offer services to the customers and i am unable to get this simple setup to work correctly. I can provide the configs as needed. Any help would be great as “I am dumb when it comes to Mikrotik”.
My Problem:
The estate has its own internet (ASN and IP range) fed by BGP to the upstream provider (not the issue). I have bridge setup on the CCR’s encompassing all the VLANs and each vlan has an IP. These then feed the Layer 2 CRS’s who have 2 ports setup feeding from the CCR trunk. Some ports on the CRS is setup as access ports for servers VM hosts, ISCSI etc. and when plugging in a customer router there, it gets an IP and it works on all vlans. I have another port setup as a trunk feeding the Ubiquity GPON OLT. on the Ubiquity, on default vlan the OLT gets an IP and is manageable, i have added the vlans 10 management vlan, 15 OLT vlan (POP), VM Vlan (Iscsi Network for servers), vlan 150 internet vlan, 400 - 404 as new FNO vlans and made these tagged on the Ubiquity. Vlan 10 is untagged and is the default vlan , I get my IP from splynx which runs on the vlan 10. My Servers all get the correct IP from the correct vlan and they use access ports on the CRS set to the specific pvid id of the vlan i want and those access ports only allow untagged or priority tagged.
I can see my ONUs if i make them vlan 1 and i can get an IP from the SPLYNX dhcp via PPPOE if i specify the local address in PPP as the vlan 15 IP address and can browse the www. Now where i am battling is getting the FNO vlans to work as well, I change the untagged default vlan on the ONU to a fno vlan and i have added a address on that vlan for testing with a dhcp server via a DHCP server (Mikrotik DHCP). I do not get an IP on that vlan. My trunk ports are setup as “admit only vlan tagged” with a PVID id of 1.
I fully believe i have the whole vlan setup wrong on the Mikrotiks and i am a bit confused by the way it works, not the same as my Cisco and Aruba setups. To better understand it all i will break it down here:
Setup and what is to be achieved a bit better discribed.
2x Mikrotik CCR2004 (redundancy)
2x Mikrotik CRS 236 (Redundancy)
6x Ubiquity GPON OLT (8 port)
many ONUs (one for each customer 4000 odd)
Here is what i would like to achieve with this setup.
The Core routers are the 2 CCRs and they are setup with a Bridge and VLANs added to the Bridge as per switch (all ports excluding port 1 and 2 as they are used for the BGP session with Upstream provider) the trunk is set as allow only tagged and ingress is ticked on ports 11 trunk feeds the one CRS and the other feeds the second CCR. The CRS326 switches configured as bridge with access ports for specific devices (VM Hosts, ISCSI etc with allow untagged only and the specific PVID of the vlan it must access) and to feed the Ubiquity OLTs i have a trunk with PVID 1 and allow only tagged with ingress ticked). All VLAN interfaces have an IP (VRRP)
The CCRs have all vlans needed, 10 for management, 15 for ISCSI, 100 for the OLTs (POPs) and 150 for internet and additional fno vlan 400 - 404 to accept other ISPs on the network. These work correctly between the CCR’s and CRS’s and on the access ports on the CRS all devices get thier respective IP from PVID set on the access port and they are setup as allow untagged or priority tagged and ingress filtering is ticked.
A trunk from one CRS setup with PVID 1 as per the same setup between the CCR and CRS with admit only tagged and priority taged and ingress filter is ticked connects the Ubiquity OLT which in turn feeds the ONUs. I have added the vlans to the OLTs, 1 as untagged, 10, 15, 100, 150 and 400-404 as tagged on the OLT. This should then correctly move data in the correct vlan to the correct destiation. On the ONU, changing the vlan for the respective ISP should move traffic to the correct destination but this is where it all falls apart. Vlan 10 will become the default (native) vlan, i see Mikrotik say keep it as PVID 1 for the trunks.
If i have the ONU on the same vlan as the OLT, i can get the customer router to get an IP from splynx and it works fine, the issue comes when trying to get the second ISP connecting to a access port on the CRS to give thier customer and IP or any service. This is achieved by changing the ONU default VLAN to that of the FNO which should send the traffic to the correct ISP. To test this, i setup a dhcp server for VLAN 400 on the router (normaly, i would not carry those to the router as the layer 3 portion is handled by the 3rd party ISP (i give them a vlan on the layer 2 side). I dont get connectivity when i change the default vlan to the 3rd party ISP (no ip or any traffic) and i can only boil this down to the VLAN/TRUNK setup. I can get the same working on a Cisco setup so i am a bit stuck.
I would like the following to be achieved in this setup
CCRs handle my BGP and SPLYNX management of customers on the VLANS i provide, The 3rd party ISP get thier connectivity via layer 2 (vlan) connected to our CRS switchs. The servers get thier connectivity on the CRS via access ports set to the vlan they access, the OLTs get thier own management network on VLAN 100 with a dhcp server on the MIK providing the IPs, By changing the native vlan on the ONU will force a customer to get thier IP etc from the respective vlan assigned.
What am i doing wrong in this senario. PS, i always find it difficult to write this down, it makes sense in my head though.