Community discussions

 
Zaesch
newbie
Topic Author
Posts: 49
Joined: Thu Apr 16, 2009 12:43 pm

Strange bonding issue

Fri Nov 23, 2012 3:24 pm

Hi!

I have a RB1100 X2, that is connected to a stack of two cisco 2960S-24TS-L via bonding (bond1 = ether1+ether2, using LACP). It works so far, but:
Step 1) Disconnecting ether1. Everything works fine - no networks disruption
Step 2) Disconnecting ether2 result in a network failure. Ok, that was expected because there are no more cables left to have a connection to the stack
Step 3) Reconnecting ether1. The link is coming up. The switch recognizes that there is a connection for the etherchannel. Looks well, but no packets are coming through.
Step 4) Reconnecting ether2. The link is established again and packets are going through.

Did the same steps with ether2 at first and then ether1. The same result: If i reconnect only the link, that was disconnected first, it isn't working :-(
Very strange. I waited a time because i thought "maybe they need some time to bring the link up" but after 6 minutes there was still no connection. The LED's are flashing, but no packet comes through.

here is the export of the bond device:
arp=enabled arp-interval=100ms arp-ip-targets="" disabled=no down-delay=\
    0ms lacp-rate=30secs link-monitoring=mii-type1 mii-interval=100ms mode=\
    802.3ad mtu=1500 name=bond1 primary=none slaves=ether1,ether2 \
    transmit-hash-policy=layer-2-and-3 up-delay=0ms
and here the cisco configuration:
interface Port-Channel1
 description -- Port-Channel --
 switchport mode access
 switchport access vlan 11
 switchport nonegotiate

interface GigabitEthernet1/0/1
 description -- To ether1 --
 switchport mode access
 switchport access vlan 11
 channel-group 1 mode active
 channel-protocol lacp
 switchport nonegotiate

interface GigabitEthernet2/0/1
 description -- To ether2 --
 switchport mode access
 switchport access vlan 11
 channel-group 1 mode active
 channel-protocol lacp
 switchport nonegotiate
Any suggestions?
 
vadyalex
newbie
Posts: 47
Joined: Tue Jan 17, 2012 11:03 pm

Re: Strange bonding issue

Fri Nov 23, 2012 10:14 pm

Interesting, could you try to unplug the cable instead of shutting down interfaces, will be any difference?
 
User avatar
wulfgard
Frequent Visitor
Frequent Visitor
Posts: 86
Joined: Wed Oct 17, 2012 1:06 pm
Location: France
Contact:

Re: Strange bonding issue

Sat Nov 24, 2012 12:47 am

Hi

please check L2MTU on all ports and both sides

a+
Thierry
 
Zaesch
newbie
Topic Author
Posts: 49
Joined: Thu Apr 16, 2009 12:43 pm

Re: Strange bonding issue

Tue Nov 27, 2012 3:55 pm

In my original post "disconnect" means to unplug the cable, but i tried to shutdown the interfaces as well - very interesting: There is no problem and everything works.

Did some more testing and there is something more to say:
Did all the steps again and at the end one of the interfaces was up, but no traffic was able to run through. Then i changed something in the bond interface (for example the link monitoring from MII type 1 to type 2) and applied the change --> everything work again.
Then i plugged all cables in and make another run: in the end one link is up, but no traffic... changed the link monitoring back to the last settings. Applied it --> it works again :shock:
It seams that the bond interface did not recognize that there is a connection... But the interfaces show the proper values (S if unplugged, RS if is plugged in).

L2MTU: As far as i know Cisco IOS calculates the L2MTU internal and it is 1514 for an access port and 1518 for a trunk port (both with L3 MTU=1500 bytes). Correct me when i'm wrong.
The RB has a L2MTU of 9014 bytes (etherX) and no L2MTU for the bond interface.
 
Zaesch
newbie
Topic Author
Posts: 49
Joined: Thu Apr 16, 2009 12:43 pm

Re: Strange bonding issue

Tue Nov 27, 2012 5:54 pm

Ok, this may be a bigger problem than i thought. I build a very, very simple setup:

Image

What do we have here?
There's a RB 750GL with ether1 and ether2 bond together (Mode: 802.3ad). ether5 is only for management purpose (configuration, monitoring, etc.). Softwareversion 5.21. Complete reset to defaults before configurating bond and management ports.
At the other end there is a Cisco 2960G 8TC-L Switch. It is reset to defaults before configuration. I only configured the etherchannel interface po1, the two gigabit interfaces 0/1, 0/2 and put an ip adress on vlan 1 - so we can ping the device.

Let's show the bonding issue
---------------------------
Step 1) Start ping from RB750 to 2960G (e.g. ping 10.0.0.1 for this configuration)
Step 2) Disconnect (unplug) cable from ether1 (or ge0/1) - ping run along
Step 3) Disconnect (unplug) cable from ether2 (or ge0/2) - ping run into timeout
Everythings ok by now... nothing is unexpected
Step 4) Reconnect (plug) cable to ether1 (or ge0/1) --> sometimes the ping comes back, but most of the time it will not recover :-(

* In my configuration you have to wait for ~50 Sek. at step 4, because spanning-tree is enabled on vlan 1. It make no difference whether spanning-tree is enabled or disabled. The links are coming up, everything looks normal, but no traffic is going through...


CONFIGURATION
-------------------------------------------------

compact export - RB 750GL:
# jan/02/1970 00:13:20 by RouterOS 5.21
# software id = XXXX-XXXX
#
/interface bonding
add mode=802.3ad name=bond1 slaves=ether1,ether2 transmit-hash-policy=\
    layer-2-and-3
/ip hotspot user profile
set [ find default=yes ] idle-timeout=none keepalive-timeout=2m
/tool user-manager customer
add backup-allowed=yes disabled=no login=admin password="" \
    paypal-accept-pending=no paypal-allowed=no paypal-secure-response=no \
    permissions=owner signup-allowed=no time-zone=-00:00
/ip address
add address=172.31.1.99/24 interface=ether5
add address=10.0.0.2/24 interface=bond1
/ip dns
set max-udp-packet-size=512
/ip neighbor discovery
set bond1 disabled=yes
/system lcd
set contrast=0 enabled=no port=parallel type=24x4
/system lcd page
set time disabled=yes display-time=5s
set resources disabled=yes display-time=5s
set uptime disabled=yes display-time=5s
set packets disabled=yes display-time=5s
set bits disabled=yes display-time=5s
set version disabled=yes display-time=5s
set identity disabled=yes display-time=5s
set ether5 disabled=yes display-time=5s
set ether4 disabled=yes display-time=5s
set ether3 disabled=yes display-time=5s
set ether2 disabled=yes display-time=5s
set ether1 disabled=yes display-time=5s
set bond1 disabled=yes display-time=5s
/tool mac-server
add disabled=no
complete configuration - 2960G 8TC-L
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname Switch
!
boot-start-marker
boot-end-marker
!
!
no aaa new-model
system mtu routing 1500
ip subnet-zero
!
!
!
!
!
!
!
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
interface Port-channel1
 description -- portchannel --
 switchport mode access
!
interface GigabitEthernet0/1
 description -- portchannel --
 switchport mode access
 channel-protocol lacp
 channel-group 1 mode active
!
interface GigabitEthernet0/2
 description -- portchannel --
 switchport mode access
 channel-protocol lacp
 channel-group 1 mode active
!
interface GigabitEthernet0/3
!
interface GigabitEthernet0/4
!
interface GigabitEthernet0/5
!
interface GigabitEthernet0/6
!
interface GigabitEthernet0/7
!
interface GigabitEthernet0/8
!
interface Vlan1
 ip address 10.0.0.1 255.255.255.0
 no ip route-cache
!
ip http server
ip http secure-server
!
control-plane
!
!
line con 0
line vty 5 15
!
end
I hope this information will help to solve the problem.
Any ideas?
You do not have the required permissions to view the files attached to this post.
 
User avatar
wulfgard
Frequent Visitor
Frequent Visitor
Posts: 86
Joined: Wed Oct 17, 2012 1:06 pm
Location: France
Contact:

Re: Strange bonding issue

Tue Nov 27, 2012 6:51 pm

why do you have

set bond1 disable=yes

in your bonding device add

mii type 1
mii interval 100 ms

can you add a result from the cisco with command
"show lacp neighbor"
and
"show lacp internal"

a+
Thierry
System and Network Engineer
Mikrotik Trainer - MTCNA MTCRE
Official French Mikrotik Distributor
 
Zaesch
newbie
Topic Author
Posts: 49
Joined: Thu Apr 16, 2009 12:43 pm

Re: Strange bonding issue

Tue Nov 27, 2012 7:08 pm

why do you have

set bond1 disable=yes
hmm, it is for the lcd. Don't know why it shows up in the compact export - looks like a default... never saw that before ;-)
in your bonding device add

mii type 1
mii interval 100 ms
ok, will try
can you add a result from the cisco with command
"show lacp neighbor"
and
"show lacp internal"
of course. Here they are:
Both cables are connected:
Switch#sh lacp neighbor
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

                  LACP port                        Admin  Oper   Port    Port
Port      Flags   Priority  Dev ID          Age    key    Key    Number  State
Gi0/1     SA      255       d4ca.6d25.60fa  17s    0x0    0x11   0x1     0x3D
Gi0/2     SA      255       d4ca.6d25.60fa   6s    0x0    0x11   0x2     0x3D
Switch#
Switch#sh lacp internal
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode

Channel group 1
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Gi0/1     SA      bndl      32768         0x1       0x1     0x1         0x3D
Gi0/2     SA      bndl      32768         0x1       0x1     0x2         0x3D
Switch#
ether1 is diconnected:
Switch#sh lacp neighbor
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode

Channel group 1 neighbors

Partner's information:

                  LACP port                        Admin  Oper   Port    Port
Port      Flags   Priority  Dev ID          Age    key    Key    Number  State
Gi0/1     SP      255       d4ca.6d25.60fa  61s    0x0    0x11   0x1     0x34
Gi0/2     SA      255       d4ca.6d25.60fa   0s    0x0    0x11   0x2     0x3D
Switch#
Switch#sh lacp internal
Flags:  S - Device is requesting Slow LACPDUs
        F - Device is requesting Fast LACPDUs
        A - Device is in Active mode       P - Device is in Passive mode

Channel group 1
                            LACP port     Admin     Oper    Port        Port
Port      Flags   State     Priority      Key       Key     Number      State
Gi0/1     SA      down      32768         0x1       0x1     0x1         0x5
Gi0/2     SA      bndl      32768         0x1       0x1     0x2         0x3D
Switch#
 
Zaesch
newbie
Topic Author
Posts: 49
Joined: Thu Apr 16, 2009 12:43 pm

Re: Strange bonding issue

Tue Nov 27, 2012 7:27 pm

tried mii type 1 / 100ms
it worked once. 5 sec. after spanning-tree brought the port online, the ping works again. Tried it again and there is still no traffic after the port comes up. I waited for 5 minutes - nothing :-/

I tried mii type 2 as well as arp - negative :-(

Should the bond interface go down, when the physical links fail? It does not...
 
User avatar
wulfgard
Frequent Visitor
Frequent Visitor
Posts: 86
Joined: Wed Oct 17, 2012 1:06 pm
Location: France
Contact:

Re: Strange bonding issue

Tue Nov 27, 2012 7:31 pm

"please add"

"switchport nonegotiate" to all interface and also to port-channel on cisco

do "shut" "no shu"t off port-channel on cisco

also ensure that duplex is full because you can experience problems between cisco and mikrotik on Gbps ports
a+
Thierry
System and Network Engineer
Mikrotik Trainer - MTCNA MTCRE
Official French Mikrotik Distributor
 
Zaesch
newbie
Topic Author
Posts: 49
Joined: Thu Apr 16, 2009 12:43 pm

Re: Strange bonding issue

Tue Nov 27, 2012 8:06 pm

ok.

i added "switchport nonegotiate" and shut/no shut the interfaces:
...
interface Port-channel1
 description -- portchannel --
 switchport mode access
 switchport nonegotiate
!
interface GigabitEthernet0/1
 description -- portchannel --
 switchport mode access
 switchport nonegotiate
 channel-protocol lacp
 channel-group 1 mode active
!
interface GigabitEthernet0/2
 description -- portchannel --
 switchport mode access
 switchport nonegotiate
 channel-protocol lacp
 channel-group 1 mode active
!
...
All cisco interfaces are 1000/full duplex:
Switch#sh int status

Port      Name               Status       Vlan       Duplex  Speed Type
Gi0/1     -- portchannel --  connected    1          a-full a-1000 10/100/1000BaseTX
Gi0/2     -- portchannel --  connected    1          a-full a-1000 10/100/1000BaseTX
Gi0/3                        notconnect   1            auto   auto 10/100/1000BaseTX
Gi0/4                        notconnect   1            auto   auto 10/100/1000BaseTX
Gi0/5                        notconnect   1            auto   auto 10/100/1000BaseTX
Gi0/6                        notconnect   1            auto   auto 10/100/1000BaseTX
Gi0/7                        notconnect   1            auto   auto 10/100/1000BaseTX
Gi0/8                        notconnect   1            auto   auto Not Present
Po1       -- portchannel --  connected    1          a-full a-1000
Switch#
Hey, that's weird: The RB says "Auto negotiation: done" and "Rate: 1Gbps" in winbox. At the commandline it says: "speed=100Mbps".
Ok, fixed the speed of ether1 and ether2 to 1 Gbps. The trunk come up. Everythings fine...disconnecting ether1... ping fails - ether2 is still connected. Wtf?!?
No, problem. Disabled ether1 and ether2, an reenabled it again. Both links come up, disconnection of ether1 works now and the ping run along. Disconnected ether2 - ping fails. Reconnecting ether1 - waited for 3 minutes - nothing. ping still fails with "timeout"...
All interfaces are at 1Gbps/fd. I think that brought no improvement :-(


The thing is: If i connect ether2 instead of ether1, it works instandly! Disconnecting ether2 at first bring the same problem. Everytime the interface, that was first disconnected oder first went down, is restored alone, the etherchannel will not work.
 
User avatar
wulfgard
Frequent Visitor
Frequent Visitor
Posts: 86
Joined: Wed Oct 17, 2012 1:06 pm
Location: France
Contact:

Re: Strange bonding issue

Wed Nov 28, 2012 12:09 am

when working on port channel on cisco side you MUST apply commands to the port channel also
so you nedd to enforce speed also on port channel
all commands MUST be in the Gig and the port channel

more over there is a lot of problems regarding cisco switch and RB1xxx ports

most of the time you have to enforce 100 Mbps full on both side
also on RB1100AHx2 port 1 to 10 does not work same as port 11 and 12
System and Network Engineer
Mikrotik Trainer - MTCNA MTCRE
Official French Mikrotik Distributor
 
Zaesch
newbie
Topic Author
Posts: 49
Joined: Thu Apr 16, 2009 12:43 pm

Re: Strange bonding issue

Wed Nov 28, 2012 2:09 pm

I almost forgot: Thanks for the help!

I think it is no problem with speed and duplex. A workmate has an interesting idea and he was right:
Go through all steps --> ether1 is connected and the ping fails. Then connect ether2 to a switch of your choice or anonther device with ethernet port. It is only important that the interface come up! In the moment ether2 come up, the bond interface transmits packets again. I don't know what this is :?
 
Zaesch
newbie
Topic Author
Posts: 49
Joined: Thu Apr 16, 2009 12:43 pm

Re: Strange bonding issue

Wed Nov 28, 2012 8:14 pm

The ultimate test: MT vs. MT 8)

I replaced the Cisco 2960G 8TC-L with a RB750GL, used the same configuration as the other RB750GL and the result ist the same as with Cisco --> the link never come back with one cable connected :-(
It is exactly the same behavior as with Cisco. I think i will tell MT support about that, maybe they can help...
 
Zaesch
newbie
Topic Author
Posts: 49
Joined: Thu Apr 16, 2009 12:43 pm

Re: Strange bonding issue

Thu Dec 06, 2012 12:29 pm

Hi folks!

MT support and i have "solved" the issue. With Version 6.0rc5 it's gone and everything work like a charm.

Who is online

Users browsing this forum: No registered users and 88 guests