Over the years my home network has got more complicated and there have been multiple cases where I have changed a rule and broken something because it takes time to test everything. I will add a list below of the sort of stuff I am talking about. The 4G backup adds a layer of complexity as everything needs to be tested with that enabled and disabled. I was thinking of setting up some test script, but just writing stuff in python or java seemed kind of hacky. Is there some open source software that people use?
Some of the things I want to test
subnet that connects via wireguard to a mikrotik running in Britain. Can machines on this subnet connect to the internet via the UK? Are they excluded from connecting to anything else?
4G backup works
I can connect to a remote mikrotik that runs on a property but it can’t connect back to me
I can connect to my parent’s house but they cannot connect back to me
My travel mikrotik can connect into home via wireguard
Guest wifi users can connect to internet but not my network
My phone can connect into my home network via wireguard
I take it that you are not in the UK? Not that it matters, the internet is not arranged according to country borders, so something which will work to the next town will work to the other side of the globe.
It looks like your tests are initiated from different machines, so there will not be a single solution for your requirement. And it also looks like many of your tests are initiated from your phone, so I can see why you say that writing scripts looks hacky, but if you can find a way, don't let that put you off. I think it is either that or develop a full app yourself.
Thanks for the reply. I’m not in the UK but BBC iPlayer doesn’t work in other countries, hence the reason for the CHR in the UK. I also use it for downloading legal linux ISOs via torrent. The tests do need different devices but I can SSH into different devices from a central location to run some tests. Most are a simple command that could be run in a single line, eg
ssh tester@mypi.lan “wget x.x.x.x”
What is killing me at the moment is the multiplications effect, eg I have 4 different wireguard connections, but from 3 different devices, and with regular internet vs 4G, plus some other multipliers, phone connected to travel router and travel router at home vs away. So the number of permutations gets up around 50 or more.
With regards the phone, I was thinking to setup similar connectivity on a separate device. The wireguard connections on the phone can be turned on/off so I could just turn them all off on the phone when running the tests. Then another device could just initiate the same wireguard connections with the same IPs. This would at least test the router config, but not the settings on the phone itself. That I could do manually pretty easily as it’s just 6 tests.
But the question is what software could I use? I have used software at work for router testing but it was expensive. I have used java and it was pretty good (green tick/red cross shown for each test in the IDE)
OK, if there is one machine you can orchestrate [all or most of] the tests from, then a script on that machine with the tests is probably the best you can do. If you can catch the return codes from wget or whatever, you can probably automate determining pass or fail.
This is testing software that adds a lot of functionality, it will give a nice GUI of pass fail, it will make writing test simple, it will allow running some tests in parallel and allows setting some config before/after some tests. I think I just need to google. I thought maybe there was something that works well with MT.
I found one, which is Cisco pyATS. You can write tests in python, it will run them and display results in a GUI. I’ve managed to get it to login to a device and run a couple of SSH commands. It doesn’t have MT support, just trying to get it to SSH to an MT device now. This is an example of a test
I have got something functioning at least. It hasn’t been smooth sailing. The syntax isn’t intuitive, the documentation is terrible and chatgpt has lead me down the wrong path a lot. But I have got it checking all the LAN and WAN IPs for a range of subnets. I wrote a basic addin to get it to work with MT, so I can run linux and MT commands and parse the results. It’s really just a matter now of going though the grunt work to write all the tests. It has already found one issue with the network
Can you explain what "Some of the things I want to test" really means?
Do you want to have monitoring 24/7 tool or do you look for a tool that would check if all listed funcionalities work and then you won't use it anymore?
Read the top post again. He wants to test if his config is basically still OK after he made changes. That is not monitoring.
It is what MikroTik should have available: run a number of tests on a router when a new release has been compiled and before it is put on the website, so we do not have to find that some function that worked before no longer works.
A very valid request I would say. I have no answer but I am certainly interested in what comes up.
For me it's still "monitoring" state of the network. If you have 4 devices or 40k+ ... it's the same job to be done.
The requirements of the OP are of the type that I call "only/just". I want to test just .... or only..... (fill in what you want and check the P.S.).
Seems to be trivial but needs a HUGE tool. Usually the expensive tool just because of these tiny only requirements.
How do you want to test if external devices still are able to connect if you do not control them.
How can you be sure that eg. phone can't connect because of change in the MT's configuration or just because it was upgraded?
How can you test if travel MT still connects or not if you changed it's configuration, not the main router's one.
A lot of checks could be scripted with ROS but checking all possible combinations is non trivial. Simple example: one side of tunnel dies and then you have to restart both ends to be sure that session would be "fresh". Therefore, how to instrument the phone to reconnect after that service restart? There should be a program that starts reconnection when some "trigger" is set by testing tool. That needs external access to some testing API.
P.S.
It's like the situation when you hear: Honey, I asked you to repaint only the ground floor of the apartment.
Seems to bi trivial but you know that there are 3 bedrooms, kitchen, 2 bathrooms and a huge garage
The required test are defined by the user of the device, or in this case the opener of the topic.
When it would be a generally available tool with pre-defined tests, we could compile a list of useful tests for a majority of devices, but in the case at hand that is not relevant.
