I have a setup where I am bonding 3 GRE tunnels from one box to another.

On the head box I have assigned 3 IP addresses in the same subnet.
On the tail box I have assigned 3 IP address in 3 subnets.

On the tail I have inserted route statements to ensure that head IP 1 gets routed via subnet 1 on the tail, head IP 2 gets routed via subnet 2 and head IP 3 via subnet 3. In this way I can balance the traffic by destination IP address to ensure the correct egress interface is used on the tail.

I then add EoIP across the link specifying IP1-1, IP2-2 and IP3-3. The tunnels come up fine.

I have then added a bonding interface and enslvaed the three EoIP tunnels. I then added a /30 subnet as a route target. Finally I have enabled ARP link monitoring specifying the /30 as the target for ARP.

Everything seems to work fine when all links are up. If I block traffic between the head and tail to simulate a link failure the link does not detect the failure. The strange thing is this was working.

I have done some diagnostic and have found something that is odd.

Each router in the link has 2 GRE connections per EoIP tunnel. One is from head to tail and one is from tail to head. I would have expected the connection to send/reply to the same connection but this appears related to the fact that each end opens a connection to the other end and thus two connections open.

I have tested an IPIP tunnel and all traffic flows over one tunnel. Any time I use EoIP to establish a tunnel I get two connections. I am convinced this is affecting the bonding and link detection but I cannot see how to ensure connection tracking.

Does anyone have any ideas how this could be solved? Or any suggestions on why the bonding does not detect the link failure?

Any help would be most welcome.

I have done some further tests and cannot seem to shed light on this.

With the EoIP tunnel setup there are two GRE connections showing on intermediate router. One from left to right and one from right to left - effectively they are asymmetric. I am not sure if this is confusing the link monitoring.

When I add arp monitoring I can see 1 packet in each direction per second (I have set arp delat to 1000ms). When I run torch on the router and monitor bonding, eoip or the ether interfaces I cannot see any arp packets. Similarily for packet sniffer. I had expected to be able to see the raw traffic to debug. When I look at the intermediate router I can see all packets are encap in GRE as expected. There is a steady flow of packets.

If I stop the flow of packets at intermediate router bonding does not detect the link failure and still tries to use the eoip link for balancing. I had assumed the lack of arp replies would have prevented this. It got me thinking that there is something strange going on with the asymmetric nature of the connection but I cannot pin it down.

The odd thing is I had this working. The fact that I did not change anything and it has stopped working and there is limited diagnostic availability concerns me about the suitability for RB in this project.

Does anyone have any ideas? Also I have looked but cannot find protocol descriptions for EoIP. As this is a proprietary protocol has it been documented?

Any help would be most welcome ...

The EoIP tunnel is a simple beast. It simply encapsulates a packet and sends it to the destination address. On the receive side it simply receives a packet and unencapsulates it. There is no actual session with the EoIP tunnel. In fact, the EoIP interfaces are always "up" just because of that. There is no detection that an EoIP interface is down, at least by the EoIP code itself. You see two "connections" because of all this -- each side is simply sending packets to the other with no sessions, handshakes, etc. Do not expect the EoIP interfaces to ever be down.

I have looked in detail at the protocol and packet flow and have indeed seen the stateless and uni-directional flow of GRE as you suggest. This all makes sense.

I have tracked down my problem on the strangeness. I need to be able to policy route traffic to leave on a particular interface from one router. I can do this easily by mangle and route mark based on destination IP address. However the router does not select the correct source interface. I have tried pref-src in the policy route statement and the packet continues to use the source address of the default gateway.

I may try static routes in the main table as opposed to policy routes as I suspect this may solve the issue. Any thoughts on why the policy route would not use the source IP address I have selected as preferred?

Any thoughts on why the policy route would not use the source IP address I have selected as preferred?

I used to use policy routing on v2.9 to select source addresses for various NAT'ed traffic but I moved away from that method in v3 due to policy routing problems. I haven't had time to sit down and figure out how policy routing is working or not working for traffic originated on the router.

I don't think your issue is due to the "two" GRE connections. I believe that the bonding will never see a down EoIP interface because that's not a state it tracks. You would need to have some netwatch disable an EoIP if the netwatch determined the other side is unreachable. This, of course, only takes care of one side. You need to watch from both sides as if one side takes down an EoIP interface, the other side happily still thinks it's up.

Alternately, you can use a stateful interface such as PPPoE, PPTP, or L2TP, with or without an EoIP in the middle, to achieve statefulness.