Hello,
I am asking for help with analyzing an elusive performance issue in our lab, where requests to a web-service, when made through the router-component of the CRS-112-8G-4S device, produce erratic and sometimes unexpectedly high response times (several seconds instead of 400ms). The request/response sizes in this scenario are small (under 10K request+response), there is no load on the network, and the router CPU utilization never exceeds 10% when I run my tests. Therefore I suspect that a misconfiguration on the router is the culprit, and not the weakness of the CRS-112 CPU.
Lab setup
CRS-112 RouterOS config has a single bridge "bridge" with members ports 2..8 are and also switch1-cpu. Ports 2..5 each are connected to Linux hosts (CentOS7) called nodeX, so that port2 is connected to host node2, port3 to host node3, etc. The webservice (OpenStack keystone) is installed on node3 (192.168.10.3). node3 has the additional IP address 192.168.10.254, which is a VIP managed by HAProxy.
The webservice endpoint against which I test is https://192.168.10.254:5000.
Ports 2..5 are set up as trunk ports, transporting 5 VLANs which are made available on the Linux hosts through 5 VNICs. The service is exposed on vlan-10.
My workstation has connectivity to nodes 2..5 like this:
- Workstation receives IP 192.168.1.199 from our TP-Link Wifi router.
- TP-Link is connected to the office router (192.168.0.1) by cable.
- CRS-112 (192.168.0.177) is also connected to the office router by cable.
- TP-link defines a static route for the network 192.168.10.0/24 with gateway 192.168.0.177 (i.e. the CRS-112).
Description of performance issue
The service under test is OpenStack keystone at https://192.168.10.254:5000. The other Openstack services installed on this node show the same performance problems as keystone, so I am using keystone here only as an example.
A curl-test to the webservice from any of node2, node3, node4 and node5 produces consistent response times of 400-450ms. When I call the service from my workstation, I receive response times that are anywhere between 400ms and 20-30 seconds.
Description of test call#!/bin/bash
echo "getting $1 tokens"
for i in $(eval echo "{1..$1}")
do
curl -w "@curl-format.conf" -k --silent --output /dev/null
-H "Content-Type: application/json"
-d '
{ "auth": {
"identity": {
"methods": ["password"],
"password": {
"user": {
"name": "admin",
"domain": { "id": "default" },
"password": "password"
}
}
},
"scope": {
"project": {
"name": "admin",
"domain": { "id": "default" }
}
}
}
}'
"https://192.168.10.254:5000/v3/auth/tokens" ; echo
done
Output of test executed from node5 (not going through CRS-112 switch1-cpu)> ./token_issue.sh 10 | ts | grep starttr
Feb 24 09:12:03 time_starttransfer: 0.451
Feb 24 09:12:03 time_starttransfer: 0.417
Feb 24 09:12:04 time_starttransfer: 0.409
Feb 24 09:12:04 time_starttransfer: 0.404
Feb 24 09:12:05 time_starttransfer: 0.410
Feb 24 09:12:05 time_starttransfer: 0.407
Feb 24 09:12:05 time_starttransfer: 0.409
Feb 24 09:12:06 time_starttransfer: 0.404
Feb 24 09:12:06 time_starttransfer: 0.409
Feb 24 09:12:07 time_starttransfer: 0.410
Output of test executed from workstation (does go through CRS-112 switch1-cpu)> ./token_issue.sh 10 | ts | grep starttr
Feb 24 09:18:42 time_starttransfer: 0.603967
Feb 24 09:18:43 time_starttransfer: 0.908884
Feb 24 09:18:44 time_starttransfer: 0.814727
Feb 24 09:18:45 time_starttransfer: 0.717794
Feb 24 09:18:46 time_starttransfer: 0.817754
Feb 24 09:18:49 time_starttransfer: 3.134013
Feb 24 09:18:50 time_starttransfer: 0.924051
Feb 24 09:18:51 time_starttransfer: 0.904291
Feb 24 09:18:52 time_starttransfer: 0.909170
Feb 24 09:18:54 time_starttransfer: 2.128209
Analysis
The responses to curl calls made from my workstation are usually 50%-100% slower compared to responses to curl calls made from hosts that are directly attached to the switch.
But every now and then, the workstation calls return only after several seconds. This never ever happens when I call from the switch-attached nodes. I don't understand why this is happening 
The only difference that I can see, is that traffic from my workstation goes through the CRS-112 CPU. Therefore, I suspect a misconfiguration of the router.
I do not think that the weak router-CPU is the issue here, because RouterOS reports it at maximum 10% load during these tests.
Below is my RouterOS configuration export.
Thank you very much to anybody who would help me understand this behavior 
RouterOS config[admin@MikroTik] > /export hide-sensitive
feb/24/2019 09:31:08 by RouterOS 6.43.8
software id = E54T-4FHN
model = CRS112-8G-4S
serial number = 94DB074A5535
/interface bridge
add name=bridge
/interface vlan
add interface=bridge name=vlan-10 vlan-id=10
add interface=bridge name=vlan-20 vlan-id=20
add interface=bridge name=vlan-30 vlan-id=30
/interface ethernet switch
set drop-if-invalid-or-src-port-not-member-of-vlan-on-ports=ether7,ether8
/interface list
add comment="office LAN" name=officeLAN
add comment="linux public LAN" name=linuxPublic
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/ip pool
add name=vpn-pool ranges=192.168.100.100-192.168.100.200
/ppp profile
add bridge=bridge local-address=vpn-pool name=vpn-profile remote-address=vpn-pool use-encryption=yes
/interface bridge port
add bridge=bridge interface=ether2
add bridge=bridge interface=ether3
add bridge=bridge interface=ether4
add bridge=bridge interface=ether5
add bridge=bridge interface=ether6
add bridge=bridge interface=ether7
add bridge=bridge interface=ether8
add bridge=bridge interface=sfp9
add bridge=bridge interface=sfp10
add bridge=bridge interface=sfp11
add bridge=bridge interface=sfp12
/interface ethernet switch egress-vlan-tag
add tagged-ports=switch1-cpu,ether2,ether3,ether4,ether5,ether6,ether7,ether8 vlan-id=20
add tagged-ports=switch1-cpu,ether2,ether3,ether4,ether5,ether6,ether7,ether8 vlan-id=10
add tagged-ports=switch1-cpu,ether2,ether3,ether4,ether5,ether6,ether7,ether8 vlan-id=30
add tagged-ports=ether2,ether3,ether4,ether5,ether6,ether7,ether8 vlan-id=40
add tagged-ports=ether2,ether3,ether4,ether5,ether6,ether7,ether8 vlan-id=50
/interface ethernet switch vlan
add ports=switch1-cpu,ether2,ether3,ether4,ether5,ether6,ether7,ether8 vlan-id=10
add ports=switch1-cpu,ether2,ether3,ether4,ether5,ether6,ether7,ether8 vlan-id=20
add ports=switch1-cpu,ether2,ether3,ether4,ether5,ether6,ether7,ether8 vlan-id=30
add ports=ether2,ether3,ether4,ether5,ether6,ether7,ether8 vlan-id=40
add ports=ether2,ether3,ether4,ether5,ether6,ether7,ether8 vlan-id=50
/interface list member
add interface=ether1 list=officeLAN
add list=linuxPublic
/interface ovpn-server server
set auth=sha1 certificate=server-certificate cipher=aes128,aes192,aes256 default-profile=vpn-profile enabled=yes keepalive-timeout=disabled require-client-certificate=yes
/ip address
add address=192.168.0.171/24 interface=ether1 network=192.168.0.0
add address=192.168.10.1/24 interface=vlan-10 network=192.168.10.0
add address=192.168.20.1/24 interface=vlan-20 network=192.168.20.0
add address=192.168.30.1/24 interface=vlan-30 network=192.168.30.0
/ip firewall filter
add action=accept chain=input comment="Allow OpenVPN" dst-port=1194 protocol=tcp
add action=accept chain=forward comment="accept all regular traffic" connection-state=established,related
add action=fasttrack-connection chain=forward comment="fast-track all regular traffic" connection-state=established,related
add action=drop chain=forward comment="drop invalid" connection-state=invalid
/ip firewall nat
add action=dst-nat chain=dstnat comment="From officeLAN to zahl:22 (ssh)" dst-port=17222 in-interface-list=officeLAN protocol=tcp to-addresses=192.168.10.2 to-ports=22
add action=dst-nat chain=dstnat comment="From OfficeLAN to node3:22" dst-port=17322 in-interface-list=officeLAN protocol=tcp to-addresses=192.168.10.3 to-ports=22
add action=dst-nat chain=dstnat comment="From officeLAN to node4:22" dst-port=17422 in-interface-list=officeLAN protocol=tcp to-addresses=192.168.10.4 to-ports=22
add action=dst-nat chain=dstnat comment="From officeLAN to mikrotik:22" dst-port=17122 in-interface-list=officeLAN protocol=tcp to-addresses=192.168.10.1 to-ports=22
add action=src-nat chain=srcnat comment="NAT outgoing traffic" out-interface-list=officeLAN to-addresses=192.168.0.171
/ip route
add distance=1 gateway=192.168.0.1
/ppp secret
add name=gerret profile=vpn-profile
add name=mpo profile=vpn-profile
/snmp
set enabled=yes
/system clock
set time-zone-name=CET
/system ntp client
set enabled=yes primary-ntp=216.239.35.0