But if I connected to the ssid using a mobile first, then the esp32 can connect to the ssid right away. I also performed the test on a virtual interface and got the same result.
Try to set
disable-running-check=yes on wifi interface(s).
Reasoning: when there are no stations connected to AP, wifi interface status changes to "not running", which in turn makes associated bridge port status to "inactive". When first station connects, wifi interface transitions to "running" which makes bridge port transition into "active". And that triggers xSTP (if enabled, by default it is) to check for any loops. This takes a few seconds (much longer if bridge mode is set to STP) and during that time, wireless station can not communicate ... e.g. it can't get DHCP lease. Some stations are short tempered and may disconnect before xSTP magic can finally enable the bridge port.
The setting I mentioned disables status transitions of wifi interface so bridge port is always active. It is possible to create a loop using wifi link, but much harder than using a patch cable between two ports of same switch. So disabling running check is pretty safe IMO.
And, BTW, your ESP32 device is way too close to hAP ac2, signal strengths higher than -35dBm or -40dBm tend to overwhelm receivers which in turn sense distorted signal and thus drop in performance.