Kubernetes Egress IP with Cilium Through Mikrotik

RouterOS Beginner Basics
1

I am new to mikrotik, and I am trying to wire up a CiliumEgressGatewayPolicy.

The one thing to note from their documentation is The egress IP must be assigned to a network device on the node.

I have a subnet of addresses added in /ip/addresses:

10.11.1.1/24 10.11.1.0 bridge1 main

On a kubernetes host, I have run these commands:

sudo nmcli connection add type ethernet ifname enp41s0 con-name primary ipv4.method auto

sudo nmcli connection modify primary +ipv4.addresses 10.11.1.3/32

sudo nmcli connection modify primary +ipv4.routes "0.0.0.0/0 10.11.1.1 table=1013"

sudo nmcli connection modify primary +ipv4.routing-rules "priority 5 from 10.11.1.0/24 table 1013"

sudo nmcli connection up primary

Both ping -I 10.11.1.3 8.8.8.8 and traceroute 8.8.8.8 -s 10.11.1.3 succeed and do actually use the correct outbound IP.

Running curl --interface 10.11.1.3 ifconfig.me fails, presumably due to asymmetric routing.

Do I need to do something else with nmcli/NetworkManager?

2

I’m not sure why you left dhcp enabled and used an additional /32 address, but I have no experience with Kubernetes. Instead of:

I would have used:

sudo nmcli connection add type ethernet ifname enp41s0 con-name primary ipv4.method manual ipv4.addresses 10.11.1.3/24

sudo nmcli connection modify primary +ipv4.routes "0.0.0.0/0 10.11.1.1 table=1013"

sudo nmcli connection modify primary +ipv4.routing-rules "priority 5 from 10.11.1.0/24 table 1013"

sudo nmcli connection up primary
3

Thank you so much for the response!

I tried that, but the curl still fails.

In essence, I want DHCP to assign a 192.168.1.xxx address via a static lease.

I want to add an additional IP address to the same NetworkManager connection, like 10.11.1.3.

I left the DHCP enabled, because I have a DHCP server configured in mikrotik:

/ip/dhcp-server/network> print
Columns: ADDRESS, GATEWAY, DNS-SERVER
# ADDRESS         GATEWAY      DNS-SERVER
0 192.168.1.0/24  192.168.1.1  10.10.1.5

I know that I specified the gateway for DHCP as 192.168.1.1, but I do not have a DHCP server/gateway for the 10.11.1.0/24 IP address block.

Do I need to do something like add a second address pool to DHCP with the gateway set as 10.11.1.1?

4

I cannot help with Kubernetes or Cillium, but couple of things worth checking:

Is the 10.11.1.1 address list when you do the command

ip address print

Can you ping and traceroute ifconfig.me

# Ping
ping -I 10.11.1.3 ifconfig.me 
# traceroute
traceroute ifconfig.me -s 10.11.1.3

Maybe you need a broader mask for the 1.3 address

sudo nmcli connection modify primary +ipv4.addresses 10.11.1.3/24

That’s about all I can suggest. Others may have better ideas.

5

In the interest of reducing variables to debug, I went with a straight static IP and ran these commands:

sudo nmcli connection add type ethernet ifname enp41s0 con-name testing ipv4.method manual ipv4.addresses 10.11.1.3/24
sudo nmcli connection modify testing ipv4.gateway 10.11.1.1
sudo nmcli connection up testing

Both flavours of ping succeed:

ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=111 time=20.7 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=111 time=20.6 ms

and

ping -I 10.11.1.3 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.11.1.3 : 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=111 time=20.6 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=111 time=19.8 ms

This just hangs:

traceroute 8.8.8.8 -s 10.11.1.3
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  _gateway (10.11.1.1)  0.144 ms  0.091 ms  0.101 ms

The IP address does show up running ip a:

inet 10.11.1.3/24 brd 10.11.1.255 scope global noprefixroute enp41s0

Running dig @8.8.8.8 google.com just hangs.

The only NAT masquerade rule I have is this:

38    ;;; WAN Outbound
      chain=srcnat action=masquerade out-interface=sfp28-1 log=no 
      log-prefix=""

These are my filter rules:

Flags: X - DISABLED, I - INVALID; D - DYNAMIC 
 0    chain=forward action=accept connection-state=established,related 

 1    chain=forward action=accept in-interface=wg-coretransit log=no 
      log-prefix="" 

 2    chain=forward action=accept in-interface=bridge1 
      out-interface=wg-coretransit log=no log-prefix="" 

 3    ;;; accept established,related,untracked
      chain=input action=accept 
      connection-state=established,related,untracked 

 4    ;;; drop invalid
      chain=input action=drop connection-state=invalid 

 5    ;;; accept ICMP
      chain=input action=accept protocol=icmp in-interface=sfp28-1 

 6    ;;; drop webfig
      chain=input action=drop protocol=tcp dst-port=80,443 log=no 
      log-prefix="" 

 7    ;;; block everything else
      chain=input action=drop in-interface=sfp28-1

Why would manually/statically assigning an IP address fail to make it out to the internet when DHCP works correctly?

6

Sorry, I was asking if 10.11.1.1 is configured on the MikroTik device. I think it is. I assume you have 2 ip addresses assigned to interface “bridge”.

192.168.1.1/24 and 10.11.1.1/24