Community discussions

MikroTik App
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

"lost connection, no beacons received"

Wed Aug 24, 2022 12:21 pm

I have a longstanding issue with a link where sometimes it logs "lost connection, no beacons received" losing the connection, which is then re-established and the same problem repeats for some time (e.g. 5 minutes) and then the issue goes away for many hours (connection is stable).
I think the loss of beacons is caused by local interference, a local AP transmitting beacons at the same rate and the same time.
After some time these timeslots drift and the problem is gone.
Is there some way to tweak the timeout on this condition, as it seems very short (a couple of seconds only), or to "dither" the transmission of beacons?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11582
Joined: Thu Mar 03, 2016 10:23 pm

Re: "lost connection, no beacons received"

Wed Aug 24, 2022 12:28 pm

I don't know if it's possible to tweak beacon timing. However, if the beacon interference is really the problem, then this would mean whole channel would be in state of high interference (during periods without beacon interference, payload interference would be high). If the problematic link doesn't show high interference (low SINR), then I think this is not the actual problem. IMO more likely is an unrelated interference (e.g. microwave oven operating sporadically). But that's hard to determine.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Wed Aug 24, 2022 1:13 pm

The AP I connect to sends 4 different SSIDs. When I monitor the situation, I see that one of them is temporarily not available while the other 3 are. Which one it is, slowly varies over time (most of the time all 4 are visible).
This leads me to the theory that there is a synchronized interference that precisely blanks the beacon transmissions and not the other traffic on the channel, and given that I think it is most likely to be another AP that is local (thus much stronger than the remote of the link that is 8.5km away), not some random carrier etc.
All this is on 5 GHz (LHG5) so not a microwave either.
The displayed SNR of the link is 38-42dB but that likely is determined mainly from data transmissions.

I think this issue could be resolved by using a dithering beacon interval, or even by a settable interval (so I could set it e.g. to 210ms instead of 200ms).
But MikroTik has no DTIM setting (that has been discussed before).
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11582
Joined: Thu Mar 03, 2016 10:23 pm

Re: "lost connection, no beacons received"

Wed Aug 24, 2022 2:34 pm

I think it is most likely to be another AP that is local (thus much stronger than the remote of the link that is 8.5km away)
You might be correct about it. But I imagine you should be able to detect such a local WiFi interferer using plain WiFi gear. But then I also imagine that there might be equipment transmitting at WiFi frequency in certain intervals but completely different modulation etc., so it might be undetectable by normal WiFi gear but would show on specialized equipment (e.g. frequency scanner scope). BTW, transmitter transmitting NV2 protocol is not detected by 802.11-compliant wifi gear so it presents a big uncorrelated interference to them. If it's about some non-wifi gear interfering it might not trigger DFS as the transmission signature doesn't match radars even remotely. Or you're operating on non-DFS frequencies and AP doesn't check it at all (knowing you my bet is on this possibility :wink:)

If the timing of the interferer is not driven by very precise clock, then the solution to your problem might be a few weeks away (drop in ambient temperature might change Tx period of transmitter enough to stop breaking your link).
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Wed Aug 24, 2022 4:54 pm

I don't have the required equipment to precisely diagnose the situation, especially not mounted at the point where it occurs.
And as the LHG5 is an "old" N-type, the actual interference from a typical modern ISP-supplied router cannot be detected on it anyway (being AC).
But I am not that interested in what it exactly is, as there will be no way to get it removed anyway.
I'm most interested in workarounds. The problem has been present for a long time, I just decided to post about it again.
In the past it was no issue because then we had BFD and the BGP would just quickly drop the link and fall back to another in such cases.
Now with RouterOS v7 under test here at home I don't have BFD (despite having whined about it for over a year now...) and it is really irritating because e.g. open winbox sessions drop because BGP does not switch quickly enough when that link is occasionally failing.
 
User avatar
Ca6ko
Long time Member
Long time Member
Posts: 500
Joined: Wed May 04, 2022 10:59 pm
Location: Kharkiv, Ukraine

Re: "lost connection, no beacons received"

Wed Aug 24, 2022 6:06 pm

How secure is the LHG reinforced/installed?
I had a similar problem on a unit whose mast was wobbling in the wind.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Wed Aug 24, 2022 7:34 pm

It is very secure, no problems due to that!
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11582
Joined: Thu Mar 03, 2016 10:23 pm

Re: "lost connection, no beacons received"

Wed Aug 24, 2022 9:10 pm

But I am not that interested in what it exactly is, as there will be no way to get it removed anyway.

If you know your enemy, you can fight it better. Without knowing it you are fighting blindly.

I feel your pain regarding inability to properly diagnose the problem.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Wed Aug 24, 2022 9:23 pm

Remember it is an unlicensed band. And ever more users that are within 10-100m from the antenna will have installed a shiny new 2/5 GHz WiFi router
from their ISP (you know, the ones that have supported AC wave2, 802.11k/r/v, MU-MIMO for a couple of years now) and this has increased the general
noise level on the band quite a lot. This link is running on 5520 MHz (10 MHz bandwidth) which is kind of less used by indoor routers, but still used a lot.
There is NOTHING I can do against that, even when I now who is doing it and where they are. They are legal users of the band on a secondary basis, like me.
And if it is something else (radar etc), there is nothing I can do either because these are the primary users of the band. I have to make room for it (DFS).

Of course a bandscope would be nice, but hey it is not Ubiquiti, it is MikroTik. We are waiting for it.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Thu Aug 25, 2022 12:41 am

Just some thoughts on this.

- MT and most vendor DTIM value is 102.4ms (102400µs) . If they all start at the same time (power restored to the building?) , they are all probably sending their beacon at the same time (every 102.4ms again).
- Not exactly the same time but drifting. It is well known in machine noise and vibration that machines running on synchronous motor, create sudden intense peaks in noise and vibration, because they just came all in full sync. The time between these burst varies, depending on the speed variation of the machines, and goes from minutes to hours. The better the sync, the longer time between these outbreaks.
- CAPsmAN triggers the freq reset to all CAPS , I presume... and all at the same moment? Then they all transmit their beacon at the same time.
- Is this in rel notes 7.5rc1 related? "*) capsman - added randomized range option for "reselect-interval" parameter (CLI only);"
- Beacons at 1Mbps, take 3% each of all airtime per transmitting SSID in that channel. 33 SSID beacons and all the channel airtime is 'full'.
- Beacons at 6 Mbps take 0.5% each of all airtime per transmitting SSID in that channel.
- In MT there is always "Wireless Sniffer" in WLAN interfaces. This does capture unicast for the WLAN and all broadcasts (beacons) in that channel, and these packets can be analysed with Wireshark.
- Don't forget to look at "Freq usage" and "Snooper" is a wonderfull tool !
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Thu Aug 25, 2022 1:08 am

Note that the link is between MikroTik dishes (or actually a LHG5 dish and a RB912UAG-5HPnD AP with sector antenna) but the suspected interfering routers are not MikroTik and are not powered by the same source.
However, such frequencies are derived from crystal oscillators, which in such equipment will not be trimmed to the correct frequency, and will be in the order of 20-50ppm of accuracy.
So they will drift relative to eachother and there will be periods where they are on top of eachother followed by long periods where they are not. Exactly what I am observing.

At the moment I am working on config changes to reduce the number of SSIDs. It is possible, but to complete it requires cooperation from others.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Thu Aug 25, 2022 9:40 am

But still, beacon broadcasts do follow the "clear channel assesment" rules, and will wait when the channel is busy.
So 2 beacons at the same time, will be transmitted after each other , and next time 'should' not be presented at the same time.

But with many (30) SSID's broadcasting at 1 Mbps the channel has no free time slots. Engenius AP, then told me "channel busy"
It is surprising how often AP's transmit on 1 Mbps in 2.4GHz even with basic rates and supported basic rates set higher.
5 GHz should use 6 Mbps or higher.

All transmitters know how busy the channel is based on their CCA experienced.
Only Draytek gave me a nice plot of the past channel load in all their AP

Minimise # SSID, indeed best practice. But with MT wifiwave2 we now need 1 SSID per VLAN.
And there is no dPSK/ePSK yet, to be able to use only 3 SSID for all. (3 = One per Security type: none/PSK/EAP) . "Access List" is a dPSK workaround.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Thu Aug 25, 2022 10:15 am

Indeed this system was configured before I understood the possibilities of assigning VLAN via RADIUS (these AP are using WPA2-EAP with RADIUS)...
So there is a separate SSID per VLAN even though this is not on wifiwave2. I am now migrating it but I need contact with all 3 users of "alternate SSIDs" to get rid of them.
That should improve the channel loading but I think it does not change the problem and it removes my way of diagnosing it.
Remember that the local APs do not see the beacons from the AP I am connecting to! It is 8.5 km away with my dish pointing it, while these APs are in the living rooms and metering closets of homes next to me. The wellknown "hidden terminal" problem that defeats attempts at "clear channel assesment" rules.

As an example, here is the pattern of the log messages about no beacons:

aug/24 14:28:23 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 14:29:03 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 14:29:08 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 14:29:19 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 14:29:24 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 14:29:51 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 14:30:01 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 14:30:20 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 14:30:41 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 14:30:48 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 14:30:53 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 14:31:22 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 15:58:14 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 15:58:41 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 16:17:18 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 16:17:40 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 20:14:12 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 20:14:40 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
aug/24 22:38:38 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
02:07:03 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
02:07:19 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
02:07:33 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
02:07:43 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
02:08:13 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
02:08:33 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
02:50:32 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
02:53:31 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
02:53:41 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
08:18:49 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
08:19:09 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
08:19:14 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
08:20:08 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
08:20:18 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
08:20:32 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
08:20:38 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
08:21:07 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
08:21:40 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
08:21:59 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
08:22:25 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
09:00:47 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received
09:01:54 wireless,info 4E:5E:0C:82:BF:F8@wlan1: lost connection, no beacons received

You can easily see the "clustering". Between those moments, there is a solid connection.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Thu Aug 25, 2022 11:21 am

Would "CTS to self" on AP or "RTS/CTS" be of any help here?

Some older wifi radio's in MT had "noise-floor-threshold" parameter in wifi. Your LHG5 is a very sensitive receiver., as AP it will hesitate to send a beacon.
I moved all my PtP and PtMP to protocol NV2, defeating neighbor AP's, with success.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Thu Aug 25, 2022 11:41 pm

I cannot use NV2 because the AP has a mix of MikroTik and Ubiquiti clients, and both of them do not implement the other's protocol.
So it has to operate in plain 802.11 mode to be compatible.
But remember: the AP has no way of seeing the interfering devices, and the interfering devices have no way of seeing the AP.
That is the "hidden terminal problem". As the devices also do not connect eachother, no protocol will fix that. Something like NV2 works only when having an AP and some clients that connect to it, on an otherwise clean channel.

I have now reorganized the configs to use only a single SSID on each AP (there are four APs on this site, it is the first site in the "photos of towers and masts" topic).
Let's see if there is any difference in behavior...
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Fri Aug 26, 2022 12:17 am

OK. You mean they are really not seeing each other ... then there is normally no "same channel (CCA detected) interference, except disruptive style like "adjacent channel interference) which requires some signal strength to be disruptive. Only the signal at the receiver is troubled by interference, not at the transmitter.

"Wireless Sniffer" of your WLAN interface might reveal something. But CCA wait states do happen, even if packets are not decodeable, and as such not stored in "Wireless Sniffer packets".
5GHz should be fairly clean (No 2.4GHz microwave oven, but USB 3 to be checked.)

Again sad discovery today: "Snooper" and "Wireless Sniffer" are missing in wifiwave2 driver. And "Scan" there is not showing radio names.

Wifiwave2 is not Mikrotik, as it is now.

Reminds me 1990 where DEC (Digital Equipment Corp) added TCP/IP to their flagship VAX/VMS, by adding the horrible UCX package.
Commands didn't even follow the very strict VMS-DCL syntax, defined and monitored by a syntax/semantics team for 20 years. UCX is not VMS.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Fri Aug 26, 2022 12:54 am

OK. You mean they are really not seeing each other ... then there is normally no "same channel (CCA detected) interference, except disruptive style like "adjacent channel interference) which requires some signal strength to be disruptive.
Please go read about the "hidden terminal problem" first.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 3005
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: "lost connection, no beacons received"

Fri Aug 26, 2022 1:27 am

wireless interface advanced winbox TAB provides some interesting parameters that maybe can help you in this

Distance
Hw Retries
Frame Lifetime
Disconnect Timeout
On Fail Retry Time

and others

i hope that can help
 
Wyz4k
Member Candidate
Member Candidate
Posts: 240
Joined: Fri Jul 10, 2009 10:23 am

Re: "lost connection, no beacons received"

Fri Aug 26, 2022 8:34 am

Please see viewtopic.php?p=953696#p953696 for how I solved my issue
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Fri Aug 26, 2022 10:49 am

wireless interface advanced winbox TAB provides some interesting parameters that maybe can help you in this
I had already experimented with those before but there was no difference. there seems to be no parameter that sets the timeout for "no beacons received".
Also disabling station roaming made no difference (tried earlier this week) and I am not running any scanning on it.

Going back to 1 SSID (from 4) also has not changed anything...
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Fri Aug 26, 2022 11:31 am

wireless interface advanced winbox TAB provides some interesting parameters that maybe can help you in this

Distance
Hw Retries
Frame Lifetime
Disconnect Timeout
On Fail Retry Time

and others

i hope that can help
There are no retries with multicast/broadcast/beacon. One shot and it should work or it fails. (Disconnect timeout is also about retries, not used for beacons)
Distance? Isn't that about ACK timing? Not used here.
Other question is: how many beacons can be missed, before "no beacons received" is activated.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Fri Aug 26, 2022 11:39 am

"Wireless Sniffer" can be set to also give the errornous received packets. This will include the corrupted beacons if sniffing done at the station.
This will at least demonstrate if there are corrupted beacons, or just some missing.
(Wifi link is not operational when "Wineless Sniffer" is active. Some problem if you want to Sniff with the other side of a PtP)
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Fri Aug 26, 2022 11:44 am

OK. You mean they are really not seeing each other ... then there is normally no "same channel (CCA detected) interference, except disruptive style like "adjacent channel interference) which requires some signal strength to be disruptive.
Please go read about the "hidden terminal problem" first.
Hidden terminal problem, is for the terminal speaking to the AP, who is busy listening to another terminal.
Don't see a direct relation with "no beacons received". The AP should see them all, and the station should not be busy listening to other stations.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Fri Aug 26, 2022 12:38 pm

A radio channel can carry only a single uncorrupted packet at a time. So when a terminal (does not matter if it is an AP or station) transmits at the same time as the AP I want to receive, but it is hidden (the terminal does not receive the AP and the AP does not receive the terminal, but my station receives the station stronger than the AP I am connected to) the "hidden terminal problem" occurs.
It is not possible to solve this in an uncoordinated situation. Only when there is a single AP connected to multiple stations on an otherwise clear channel you can solve it using solutions like NV2, but that is not the situation.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Fri Aug 26, 2022 1:22 pm

Still would be interesting to know if it is the AP that is not finding a slot to transmit the beacon, or if the station receives a corrupted beacon.
Your setup probably does not allow for the "AP-station" swap with the LHG5 (you mention other stations that join this link.)
You might still be looking at the wrong end of this link for the issue.
Last edited by bpwl on Sat Aug 27, 2022 12:53 pm, edited 1 time in total.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Fri Aug 26, 2022 2:06 pm

You have a basic misunderstanding about the hidden terminal problem. When you are not prepared to go off and read some theory about it, I give up on coming back to it again and again.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Fri Aug 26, 2022 2:29 pm

You don't have a standard hidden node problem, AFAIK. https://en.wikipedia.org/wiki/Hidden_node_problem. At least not according to the definition of "hidden node problem/ hidden terminal problem" here on Wikipedia. There could be other definitions around. What's in a name?

RTS/CTS could solve a hidden terminal problem, not in your case. It looks like this is just destructive interference from stations that do not take part in this LGH5 AP-station network, and cannot do proper clear-channel-availability checks to co-exist with that LHG5 network, based on the announced transmission times in the transmission headers. If AP's can receive each others CTS, they will only use and assign free timeslots to their clients which use RTS/CTS.

It's not exposed node problem either: https://en.wikipedia.org/wiki/Exposed_node_problem

PS: Could be because they do not all hear the other, but another possible cause is, they cannot decode the others headers. I could not find this info in the 802.11 standards: if 10MHz wide channels and 20MHz wide channels can co-exist [wait the duration of the announced transmission time from the other transmitter]? (e.g. reading each others headers) and then there is only CCA-ED (energy based CCA evaluation), potentially causing more destructive interference, but not as bad as full adjacent channel interference, that misses CCA completely.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Mon Aug 29, 2022 7:55 pm

This link is running on 5520 MHz (10 MHz bandwidth) which is kind of less used by indoor routers, but still used a lot.
...
PS: Could be because they do not all hear the other, but another possible cause is, they cannot decode the others headers. I could not find this info in the 802.11 standards: if 10MHz wide channels and 20MHz wide channels can co-exist [wait the duration of the announced transmission time from the other transmitter]? (e.g. reading each others headers) and then there is only CCA-ED (energy based CCA evaluation), potentially causing more destructive interference, but not as bad as full adjacent channel interference, that misses CCA completely.
Found something ... Have done some reading ...
Klembord-2.jpg
https://www.microsoft.com/en-us/researc ... lWidth.pdf
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Mon Aug 29, 2022 9:13 pm

Again, any concerns about incompatible channel widths (and the problem that they do not mutually detect eachothers presence) is not relevant in this case because there is no way that the AP I am connecting to is going to receive the APs that cause the local interference, and vice-versa.
I can receive the remote AP only because I have a directional antenna towards it.

This is the common situation for anyone that sets up a Pt(m)P link over larger distances in an environment where the same band is used for indoor networks.
The indoor networks are invisible to the remote AP and the indoor networks do not receive the remote AP.
The indoor APs just key up to send their beacons and they are not away they are trampling the reception of the beacons from my remote AP.

It could be solved by "dithering" the remote AP beacons (sending them at irregular intervals e.g. using a 10ms random offset from the regular 100ms interval), or by setting a nonstandard interval that is not a multiple of the standard rate. But it does not look like RouterOS provides the parameter required for that.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Mon Aug 29, 2022 10:19 pm

I do follow and understand your assesment of the problem.

However remember that a directional antenna is NOT sending a stronger beam (the signal is actual weaker than from a simple omnidirectional antenna, even at its peak direction, because of EIRP legislation), the major action of a directional antenna (with high gain) is the amplification of the receive sensitivity in the peak direction.

Having co-existence would require the remote LHG to see the indoor AP signal, and wait with it's beacon until the air is clear.
Of course having an 5GHz signal pass walls is not obvious/possible. It could be that the remote LHG is not seeing the indoor AP's at all. (Indoor AP's will certainly not see the remote LHG)
Having 10MHz channel width in a crowded 20MHz environment will certainly not help for this already very difficult co-existence. (40 and 80MHz are 20MHz compatible)

So changing channel and redirecting the local LHG some degrees (antenna pattern is a complex diagram, and the back side is of the LHG is not the weakest direction) are indeed your only options left.

NV2 is no option you said, and would even not solve much as it is just a "signal level fight" with 802.11 . NV2 does not care about the presence of a 802.11 signal, does not do CCA, it just transmits when it is time to do so.

I had to remove all omnidirection AP's near the path of my PtP links (even with only SXTsq as directional antenna), and find a direction for the SXTsq so that the reception of those AP's was weak enough.

Just a last tought, not sure if it is correct. I have seen in 802.11ac that the basic rate (multicast, beacon) is only set for one chain, VHT (MCS 0-7, 0) while unicasts are dual chain VHT (MCS 0-9, MCS 0-9). I have no idea what 802.11n (HT MCS) is doing. If true, then turning the LHG around it's peak axis might change the received signal for beacons.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11582
Joined: Thu Mar 03, 2016 10:23 pm

Re: "lost connection, no beacons received"

Tue Aug 30, 2022 8:19 am

... the signal is actual weaker than from a simple omnidirectional antenna, even at its peak direction, because of EIRP legislation ...

This is not true. If antenna gain (= gain over theoretical monopole antenna) is properly taken into account and transmitter properly implements regulatory limits, then electrical field strength (measured in Volts per meter) of transmitted signal will be exactly the same in both cases in direction of nominal antenna gain ... and EIRP is an indication of such measurements.

It is true that most of benefit of using highly directional antennae (in case of limited EIRP) comes from improvement of Rx signal. But this comes with cost of increasing interference levels as well if interferer lies within antenna beam. Things get worse if interferer doesn't use directional antenna and thus does not benefit the increased Rx sensitivity and thus misses co-channel transmitter.
What @bpwl suggests (slight change in antenna direction) might help if interferer is actually not in direction of main beam but rather in direction of a side beam (all antennae have sone, highly directional antenna tend to have more of them) with lower gain and by changing direction it might be possible to get interferer out of that side beam. If the interferer is in the main beam, then I'm afraid nothing can be done apart from moving link to another frequency channel.

As to problem by @pe1chl: I don't think changing beacon periodicity would (semi)permanently cure the problem. Beacons are interfered also by payload frames, the amount of interference then depends on airtime utilization of interfering device ... could be either AP or station of another wireless network (if interference is due to beacons of foreign network, then interfeter can only be AP in which case flow control of foreign AP might do it's job of avoiding transmissions during beacon airtime of PtP link).
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Tue Aug 30, 2022 11:36 am

Note that using "the direction of the interferer" is not practical in my (very typical) case because I have a single wanted signal in one direction combined with a "cloud" of local transmitters in many different directions, some of them the same as the direction to the wanted signal.
The relative strength of the interfering signal is determined by the distance to my antenna and the gain of the antenna in that direction.
So even when the antenna has a 30dB lower gain in that direction, that can be offset by the interferer being so much closer that its local field is 30dB stronger.
It is nothing special, it is the typical situation when trying to setup a long PtP link (8.5km in my case, so not extreme) from a residential area with many users.
When I do a scan on my INDOOR devices (a hAP ac2 and a 4011) I get ~40 APs in the scan, about 8 of them covering my frequency of 5520 MHz.
(note that this is not only APs operating on exactly that frequency, but all of the APs with some frequency in 5500-5580 in some layout like Ceee eCee ... eeeC)
And that is only indoor. It is not feasible to do this scan on my outdoor link because it is N-type and of course all APs by now are AC.

Of course actual use of airtime would also cause interference, but when looking at the time-of-day of the interference I do not suspect it is the main contributor.
(In that case I would expect a correlation with typical internet peak usage, say 1800-2300 hours, while in fact interference bursts also occur in the middle of the night at the same chance as other times, see the log above)
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Tue Aug 30, 2022 11:51 am

will be exactly the same
I knew that :-), I do know the EIRP regulation and calculations, just stated this to get your attention ?! Or try to hint on what you might experience in some cases.

My message: " it is not the intuitively expected gain."

For indoor the high gain will not work well, because the path of the signal is indirect and not known, and is expected to be in all directions, with some reflections as well There the low gain will win, and that is what I experience here in my house. High gain antenna give a low TX power radio, and that is sometimes giving a lower received signal.

But yes, the signal in the strongest direction is exactly the same, But when indirect paths do help , the lower signal strength in he other directions will not be an advantage for the high gain antenna.
And for MT with the classic drivers without TPC, the allowed EIRP is 3dB lower than for others.
And 802.11ac and 802.11n do co-exist on CCA headers., and will wait for each other. The non-coexistence for 10 MHz-width in the Microsoft paper, is just interesting new knowledge to me.


Note: Forum moderators are strict lately? One just bluntly edited my text and toke away my edit access.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Tue Aug 30, 2022 11:59 am

Experiment: A 802.11n device does see the ac-only mode 802.11ac AP's here, with a scan from the 802.11n device.
Klembord-2.jpg
... AP's on indoor-only freq are missing in the scan from a SXTsq5HP that only operates with "installation=outdoor"

But indeed avoiding a cloud of AP's is not possible (only some backward directions, like -30° have no sidelope).
There is almost no escape here. 6GHz device? 60GHz, but range then can be a problem.
You do not have the required permissions to view the files attached to this post.
 
User avatar
Ca6ko
Long time Member
Long time Member
Posts: 500
Joined: Wed May 04, 2022 10:59 pm
Location: Kharkiv, Ukraine

Re: "lost connection, no beacons received"

Tue Aug 30, 2022 4:06 pm

I have always believed that "loss of beacons" occurs on the radio air. I observed a loss of beacons between two devices at a distance of 200 m. Part of the Fresnel zone was blocked by trees and I thought they were the cause. Wyz4k pointed out that there is another reason. AP may not transmit beacons, but in case of point-to-point mode then all clients would register the error, as in our case only one client registers it means the problem is only on his radiotrace. Or the client may not receive beacons because his radio track is busy scanning the air. This can easily be fixed by turning it off or setting a short scan list.
This option should definitely be tried to exclude it.

In wiki.mikrotik it is stated that this error is caused by: "Most likely weak signal, remote turned off, strong interference, some other RF related issue that makes communication impossible".
So immediately discard all the service frames, etc. This problem is purely physical.
Analysis of the logs shows that the problem occurs briefly at different times for 5-10 seconds for 1-4 minutes. It is very similar to a swinging obstacle on the radio band (usually a tree top). Only a strong enough gust of wind causes the connection to break.
In any case, standard measures to improve radio channel quality should help.
And here you need to start by analyzing the trace, building a vertical profile of the trace and comparing the available installation options for signal enhancement and interference reduction.
Also change the place of installation so as to protect the device by natural obstacles from interference. And practice shows that using only one way is not always effective, you need to use the complex.
The main way to increase the signal to interference ratio is to use an antenna with high gain. The effect that is achieved by the antenna is not achieved by anything else. If you can not replace the antenna, you can use other ways. Move the antenna and use an additional shield from interference..
The cheapness of Microtik wifi devices is due to the very low shielding from interference. Therefore, those who have encountered such problems use additional shielding. Buy ready-made or make their own.
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Tue Aug 30, 2022 7:42 pm

In wiki.mikrotik it is stated that this error is caused by: "Most likely weak signal, remote turned off, strong interference, some other RF related issue that makes communication impossible".
So immediately discard all the service frames, etc. This problem is purely physical.
Analysis of the logs shows that the problem occurs briefly at different times for 5-10 seconds for 1-4 minutes. It is very similar to a swinging obstacle on the radio band (usually a tree top). Only a strong enough gust of wind causes the connection to break.
I know this in my case is not the cause. The signal is OK at about -72dBm and the S/N is around 40dB.
Also, before this week I had 4 SSIDs running on a single AP. I observed that when I saw "no beacons received" and I immediately switched to scanner mode, I saw the other 3 SSIDs but not my own. This has lead me to the theory that it is interference with local beacon signals from APs that send the beacons at nearly the same rate.
After some time the beacon becomes visible again and another one might disappear.
However, in an attempt to improve channel efficiency I have now reduced the config to 1 SSID and of course I can no longer do this test.
(it has not changed anything to the problem, by the way)
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Tue Aug 30, 2022 7:58 pm

Experiment: A 802.11n device does see the ac-only mode 802.11ac AP's here, with a scan from the 802.11n device.
Interesting, I did not know that... as the LHG is set to 10 MHz I normally do not see those, but now that I have set it to 20 MHz indeed I do see the AC mode APs.
When scanning the entire band I see 116 APs with signals up to -60dBm, so 12dB stronger than my wanted signal.
Limiting the scan to 5520 MHz 20 MHz BW I see 18 APs with signal up to -64dBm.

Maybe as an experiment I'll change the actual BW to 20 MHz to see how the connection is (that is something on the TODO list anyway).
 
User avatar
Ca6ko
Long time Member
Long time Member
Posts: 500
Joined: Wed May 04, 2022 10:59 pm
Location: Kharkiv, Ukraine

Re: "lost connection, no beacons received"

Tue Aug 30, 2022 10:33 pm

Once you have determined that "strong interference, some other RF related issue that makes communication impossible" is your case, then use standard methods to combat interference.
You have a very low level of useful signal. The standard signal level is -67 in the 10 MHz band and -64 in the 20 MHz band. It turns out that it would be optimal to use mANT30 + Sleeve30 kit for mANT30.
If there is no possibility to use other equipment, you will have to modify the LHG.
The main problem with the LHG is that it has very high sidelobe levels, almost to 0 dB.
Screenshot_3.jpg
Therefore, the place of installation is important. It is necessary to hide the antenna behind some obstacles (walls, roofs, corners of buildings, etc.).
You also have to use a anti-noise shield
For experiments you can use a large kitchen pan of suitable size (40cm) :lol: . Or cardboard tape and aluminum foil. This is to make sure you get an acceptable result.
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Tue Aug 30, 2022 11:59 pm

No amount of directivity will completely solve the problem. I see APs that I know are directly in the direction of the beam (some people put their address in the SSID...) as well as very close ones like my neighbors "behind" the dish. I am familiar with that kind of patterns (I am a radio amateur) and I know that it is possible to improve it a little but not to completely get rid of the interference.
In our network we also use the "ISO" type of dishes from UBNT and they indeed are better. But I also have requirements on the physical size and look of the dish, to avoid problems with placement permission. My LHG is a LHG5 XL which is about the maximum I can place here.
 
User avatar
Ca6ko
Long time Member
Long time Member
Posts: 500
Joined: Wed May 04, 2022 10:59 pm
Location: Kharkiv, Ukraine

Re: "lost connection, no beacons received"

Wed Aug 31, 2022 9:12 am

Then the most I can advise, try to make an additional screen out of the grid, paint it white. it will not change the appearance. There is a similar option in the above photos.
Be sure to check the condition of the antenna on the LHG board itself often the element can be crooked. sometimes you can get +2 dB. It is better to use HP version of the board which has additional shielding/heatsink as on the photo. The board should also be additionally shielded.
Screenshot_8.jpg
Is the LHG mounted on the wall or on a mast?

On the side of the access point you have a sector antenna, they have a fairly narrow diagram in the vertical plane. Usually they are mounted with a few degrees slanted down toward the ground. Are you the farthest customer (8.5 km) of this antenna or not? Check the angle of the antenna and see how much you can change it. I tend to think that the oscillation of this antenna is the main cause of the problem. Since you write that the problem occurred a long time ago.

PS The message is displayed in the log at the loss of signal after 5 seconds or 50 beacons
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Wed Aug 31, 2022 12:05 pm

Everything is mounted very sturdy. The antenna on my side is on a sturdy metal pole (6cm at the bottom, tapering to 4.5cm where the LHG is. It is the HP model, the is no LHG5 XL N-type that is not HP.
It does not move unless there are very high winds, and the issue does not correlate with high winds. Also, that would not explain 5-second interruptions over several minutes intervals and then normal communication again.
The AP side is as photographed in the first reply to the "Photos of towers and masts" topic and while it vibrates at little in high winds it normally is rock solid.
The downtilt angle has been carefully adjusted. I know that is very critical.
 
User avatar
Ca6ko
Long time Member
Long time Member
Posts: 500
Joined: Wed May 04, 2022 10:59 pm
Location: Kharkiv, Ukraine

Re: "lost connection, no beacons received"

Wed Aug 31, 2022 4:45 pm

Now I realize that I'm telling you the plain truth...
And yet, sometimes it's exactly what we forget.
I would try to change the angle of inclination of the sector antenna by 0.5-1 degrees while observing the signal level at the client, if there is a clear increase in the level set a new value.
When installing the antenna at a height of 220 m, the tilt angle should be around 1.3-1.5 degrees for a distance of 8-9 km. To tilt the LHG to this angle you need a plate with a thickness of 2.5-3 mm. It is placed under the lower mount between the pipe and the antenna. I use a piece of polyethylene water pipe. Sometimes I reduce the interference in this way by putting 2-3 pcs, the interference level drops faster than the useful signal.
When adjusting the antenna, it is also necessary to move it up and down over a distance of about 1 m, selecting the position with the maximum signal. At the end try to move the antenna a little to the left to the right, observing the level of interference and signal level, choosing the optimum position. After all, for LHG XL deviation from the maximum 3,5 degrees gives a decrease in signal level by 3dB, and the next 3.5 degrees almost 15-17dB

PS I have never seen a loss of beacons at the devices mounted on the wall, only if the mast, hence I assume that is related to the fluctuations of the antennas.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Wed Aug 31, 2022 5:51 pm

This has already been done long ago... and let me assure you: there is no movement like that in this tower. When there is 5+ bft winds there is vibration, but not such that it would permanently tilt over long intervals.

I have now done some experiment that aligns with my own observations: remember before we had 4 SSIDs (for different users to connect to different VLANs) and I had replaced this with a single SSID just because that is more efficient. But when the 4 SSIDs were present, I had observed that one could drop out while the others did not.

Now that the VLAN assignment is done via RADIUS (this is for 3 users, there is also a "default") I had brought back the 4 SSIDs to one.
But now I have again added a second SSID and configured the two wlan interfaces in the same bridge. So it does not matter which one I connect, it provides the same function.
In my client LHG I have configured a "connect list" with these two SSIDs.

Now I am going to wait to see if at some point the main SSID gives "no beacons received" and the connection switches over to the secondary SSID (second in the connect list)...
(as usual, the connection now did not yet fail. but that should happen within half a day or so)
 
User avatar
Ca6ko
Long time Member
Long time Member
Posts: 500
Joined: Wed May 04, 2022 10:59 pm
Location: Kharkiv, Ukraine

Re: "lost connection, no beacons received"

Wed Aug 31, 2022 6:45 pm

The problem of the lack of beacons is related to physical problems, Without changing the physics you will not get a change.
The problem involves the transmitting device, the receiving device, and foreign devices causing interference. The problem only occurs when all three factors coincide. That is why I said that the solution must be complex.
You have to try to reconfigure the antennas...

Let's calculate your experiment.
The beacon is missing, the connection is broken, the registration of the device on the new SSID started, which takes 20 seconds. - 2 minutes and half of your failures will not be detected, you just will not see them in the log.
I can assume that the problem occurs in the following way: signals from another access point, working on the same channel, in a small period of time begin to exceed the level of beacons from your access point. When such exceedance occurs for less than 5 seconds you do not know about it, in those rare moments when the interference exceeds the useful signal for more than 5 seconds there is a message in the log. The useful signal and interference are periodic with close parameters.
To analyze the problem it is better to set the signal level monitoring, for example through the dude.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Wed Aug 31, 2022 7:16 pm

Notwithstanding your stubborness I can now report that my experiment works!
A few minutes ago I got the "no beacons received" error and the LHG automatically connected to my second SSID.
It happened just before I was viewing the log window. I started background scan and indeed only the second SSID is visible. Connection is stable.
Signal is around -71dBm, SNR 44dB. In my experience that is enough at 10 MHz bandwitdh, connection are appearing below -80dBm.
A couple of minutes later the first SSID again erratically appears in the scan window. Later it is stable again.

This confirms my hypothesis of systematic interference with the beacons, likely caused by a local AP (hidden to the remote) transmitting beacons at the same rate.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11582
Joined: Thu Mar 03, 2016 10:23 pm

Re: "lost connection, no beacons received"

Wed Aug 31, 2022 10:55 pm

I guess that by setting bandwidth to (not so widely supported) 10MHz, you're effectively preventing adjacent APs from properly detecting your PtP link as strong co-channel interference worth avoiding. Since other APs don't avoid it (neither during channel reselection nor during checking for free Tx window), you may be harming your link way more than that 3dB higher signal strength due to narrower bandwidth used. I think it would be worth to try is using 20MHz changes the situation (either for better or worse).
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Wed Aug 31, 2022 11:09 pm

Yes, that is another experiment I want to try. Also because the newer (AC-type) dishes for whatever unclear reason do not support 10 MHz bandwidth, so we need to tell people all the time to buy the N type, and that way we preclude any migration to AC. Which at some time has to be done.
Unfortunately it is already difficult to allocate frequencies to the 4 APs at that tower that are not used by weather radar with this 10 MHz width. Increasing the width makes that only more difficult.
 
User avatar
Ca6ko
Long time Member
Long time Member
Posts: 500
Joined: Wed May 04, 2022 10:59 pm
Location: Kharkiv, Ukraine

Re: "lost connection, no beacons received"

Thu Sep 01, 2022 8:51 am

What settings are you working with?
Standard
Guard interval
Basic rate
Signal strengths Tx/Rx on channels Ch0 Ch1
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Thu Sep 01, 2022 10:58 am

"hypothesis of systematic interference with the beacons"
I think you can verify this hypothesis with the "Wireless Sniffer" of RouterOS. (In the LHG or some other ROS device.)
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Thu Sep 01, 2022 11:01 am

"hypothesis of systematic interference with the beacons"
I think you can verify this hypothesis with the "Wireless Sniffer" of RouterOS. (In the LHG or some other ROS device.)
The scanner should do the same thing, right? When I do a scan I see the SSIDs of visible AP which is the beacon transmission.
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2993
Joined: Mon Apr 08, 2019 1:16 am

Re: "lost connection, no beacons received"

Thu Sep 01, 2022 11:11 am

Yes but Wireless sniffer shows you each beacon and it's time of arrival.
And with 'receive errornous packets', you can mostly see which ones had interference.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Thu Sep 01, 2022 11:18 am

Ok when I happen to see a reconnect I will do a wireless sniff to see if I can derive something...
When doing it in regular condition, I see both beacons, unfortunately a millisecond after eachother (it would be better when they were evenly spaced...).
 
volkirik
Member Candidate
Member Candidate
Posts: 208
Joined: Sat Jul 23, 2016 2:03 pm

Re: "lost connection, no beacons received"

Sun Oct 08, 2023 10:58 am

I found a link from Huawei, https://support.huawei.com/enterprise/e ... -threshold

They do not allow positive values for noise-floor-threshold..

Setting noise-floor-threshold may help stability but offers less speed. IMO, setting very high values screw up the link.. While Dynamic (default) causes lots of disconnects..

I dont know why mikrotik's dynamic (auto) setting can not decrease speed (sacrifice performance) for stability.. And dunno why AC cards lack this feature..!

I am Open to discuss this parameter. What do you think?

Regards
Last edited by volkirik on Sun Oct 08, 2023 5:26 pm, edited 4 times in total.
 
pe1chl
Forum Guru
Forum Guru
Topic Author
Posts: 10215
Joined: Mon Jun 08, 2015 12:09 pm

Re: "lost connection, no beacons received"

Sun Oct 08, 2023 11:51 am

Please don't pollute the topic with unrelated things!

Who is online

Users browsing this forum: Ahrefs [Bot] and 12 guests