Community discussions

 
User avatar
amt
Member
Member
Topic Author
Posts: 380
Joined: Fri Jan 16, 2015 2:05 pm

Lost connection to multiple LHG units

Mon Mar 19, 2018 7:24 pm

Hi all,

Today all our customers who has puplic ip with Mikrotik device stop working. When we check for understand what happen, we see that the ethernet interfaces of these devices are not working , and some SXT reseted.
So I wonder to ask if is that related with any security vulnerabilities ?

Thanks.
Last edited by amt on Tue Mar 20, 2018 5:06 pm, edited 1 time in total.
 
anav
Forum Veteran
Forum Veteran
Posts: 706
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada

Re: URGENT Help !! any security vulnerabilitie ?

Tue Mar 20, 2018 2:54 am

Power outages?
Didnt pay internet bills?
ISP folded?
What is the reason for the failures?
Not enough information.
 
BRMateus2
newbie
Posts: 38
Joined: Thu Oct 26, 2017 11:18 pm

Re: URGENT Help !! any security vulnerabilitie ?

Tue Mar 20, 2018 4:40 am

Version?
... details, formulate an relatory.
 
mistry7
Forum Veteran
Forum Veteran
Posts: 728
Joined: Tue Oct 13, 2009 11:57 am
Location: Germany

Re: URGENT Help !! any security vulnerabilitie ?

Tue Mar 20, 2018 6:54 am

Using to easy Passwords on public available not firewalled devices?
 
User avatar
amt
Member
Member
Topic Author
Posts: 380
Joined: Fri Jan 16, 2015 2:05 pm

Re: URGENT Help !! any security vulnerabilitie ?

Tue Mar 20, 2018 8:26 am

Power outages?
Didnt pay internet bills?
ISP folded?
What is the reason for the failures?
Not enough information.
Hi anav,
Power outages?
these devices at in different places and there were no Power outages at their place...
Didnt pay internet bills?
devices not working, ether ports not work.
What is the reason for the failures?
I dont know. thats why Im sharing this problem in here. all other cpe's are working. but who has puplic ip it's not work. all of them has same problem.

Thanks
 
User avatar
amt
Member
Member
Topic Author
Posts: 380
Joined: Fri Jan 16, 2015 2:05 pm

Re: URGENT Help !! any security vulnerabilitie ?

Tue Mar 20, 2018 8:28 am

Version?
... details, formulate an relatory.
Hi BRMateus2,

devices not working so I could not check what version they were. I will try to netinstall them today. nearly 50 device down.
 
User avatar
amt
Member
Member
Topic Author
Posts: 380
Joined: Fri Jan 16, 2015 2:05 pm

Re: URGENT Help !! any security vulnerabilitie ?

Tue Mar 20, 2018 8:29 am

Using to easy Passwords on public available not firewalled devices?
Hi mistry7,

password was not easy and also user name was not admin. winbox port and ssh, telnet, api closed to outside.

Thanks.
 
yakula
just joined
Posts: 8
Joined: Fri Jun 16, 2017 9:57 pm

Re: URGENT Help !! any security vulnerabilitie ?

Tue Mar 20, 2018 8:46 am

I have same problem. There are 10 LHG devices ether down and netinstal not working. Any suggestion how to repair them?
 
User avatar
amt
Member
Member
Topic Author
Posts: 380
Joined: Fri Jan 16, 2015 2:05 pm

Re: URGENT Help !! any security vulnerabilitie ?

Tue Mar 20, 2018 1:10 pm

I sent mail to support yesterday but not answerd me yet.
 
anav
Forum Veteran
Forum Veteran
Posts: 706
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada

Re: URGENT Help !! any security vulnerabilitie ?

Tue Mar 20, 2018 3:54 pm

Two possibilities come to mind
a. ISPs changed their setup
b. routers changed their setup - DiD you implement firmware updates at the time of failures?
 
User avatar
amt
Member
Member
Topic Author
Posts: 380
Joined: Fri Jan 16, 2015 2:05 pm

Re: URGENT Help !! any security vulnerabilitie ?

Tue Mar 20, 2018 4:20 pm

hi anav,

we are supplying to internet to these customers and we did not change any setup, we did not implement any firmware. there are more than thousand cpe. and only puplic ip assigned mikrotik's affected.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 23248
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: Lost connection to multiple SXT units

Tue Mar 20, 2018 4:22 pm

You say Ethernet interface not working. Can you access the device from the WiFI interface?
No answer to your question? How to write posts
 
User avatar
amt
Member
Member
Topic Author
Posts: 380
Joined: Fri Jan 16, 2015 2:05 pm

Re: Lost connection to multiple SXT units

Tue Mar 20, 2018 4:45 pm

You say Ethernet interface not working. Can you access the device from the WiFI interface?
Hi Normis,

there is no any wifi signal also, most of them LHG and all of them same problem... bios seems lost. we copy one of LHG bios and transfer it to broken one and it worked. but now the licence has problem,device worked and telling there is no licence. and another problem all mac same with copied one.

Thanks
Last edited by amt on Tue Mar 20, 2018 7:05 pm, edited 1 time in total.
 
tippenring
Frequent Visitor
Frequent Visitor
Posts: 92
Joined: Thu Oct 02, 2014 8:54 pm

Re: Lost connection to multiple SXT units

Tue Mar 20, 2018 5:09 pm


there is no any wifi signal also, most of them LHG and all of them same problem... bios seems lost. we copy one of LHG bios and transfer it to broken one and it worked. but now the licence has problem,device worked and telling there is no licence. and another problem all mac same with copied one.
Just a thought, but perhaps one of your customers gained access and corrupted them all. If you use the same credentials on all of your customer devices, it would not be difficult for someone to do this.
 
User avatar
amt
Member
Member
Topic Author
Posts: 380
Joined: Fri Jan 16, 2015 2:05 pm

Re: Lost connection to multiple SXT units

Tue Mar 20, 2018 7:11 pm


there is no any wifi signal also, most of them LHG and all of them same problem... bios seems lost. we copy one of LHG bios and transfer it to broken one and it worked. but now the licence has problem,device worked and telling there is no licence. and another problem all mac same with copied one.
Just a thought, but perhaps one of your customers gained access and corrupted them all. If you use the same credentials on all of your customer devices, it would not be difficult for someone to do this.
we are denied access to some ports like 22,23,8291,8728,8729..
what advice can you have for like such problems? and what was wrong with these devices, how they delete their bios firmware ?

Thanks
 
tippenring
Frequent Visitor
Frequent Visitor
Posts: 92
Joined: Thu Oct 02, 2014 8:54 pm

Re: Lost connection to multiple LHG units

Wed Mar 21, 2018 4:53 pm

Most people that think they have a "secure" network do not. I see this all the time.

As far as the question of how did this happen, it will be easier to determine once you have done some investigation. Right now, how it happened has many answers. Do you have remote management of the devices at the customer sites? Do you use common passwords across multiple devices? Do you permit remote management from a management subnet? These are just a few questions.

Is it the firmware, or is it the router software (RouterOS) that is removed? I would not expect you to be able to recover if the firmware is missing, but I could be wrong.
 
User avatar
amt
Member
Member
Topic Author
Posts: 380
Joined: Fri Jan 16, 2015 2:05 pm

Re: Lost connection to multiple LHG units

Wed Mar 21, 2018 5:08 pm

Most people that think they have a "secure" network do not. I see this all the time.
yes you are right.. plus for this inexperienced :)
Do you have remote management of the devices at the customer sites?

no
Do you use common passwords across multiple devices?

unfortunately yes same password for multiple device.
Do you permit remote management from a management subnet?

yes for management we permit for some special ip's to accesses
Is it the firmware, or is it the router software (RouterOS) that is removed? I would not expect you to be able to recover if the firmware is missing, but I could be wrong.

firmware(bios file)
 
tippenring
Frequent Visitor
Frequent Visitor
Posts: 92
Joined: Thu Oct 02, 2014 8:54 pm

Re: Lost connection to multiple LHG units

Thu Mar 22, 2018 12:22 am

I don't know how to erase firmware, so I can't begin to guess what happened.

Perhaps one of your management hosts is/was compromised. Another possibility would be an as-yet-undiscovered vulnerability since it only occurred on routers with public IPs. I have perhaps 100 MT routers with public IPs and haven't seen any go offline yet.

Please let us know when you discover the cause.
 
User avatar
amt
Member
Member
Topic Author
Posts: 380
Joined: Fri Jan 16, 2015 2:05 pm

Re: Lost connection to multiple LHG units

Fri Mar 23, 2018 12:00 pm

I don't know how to erase firmware, so I can't begin to guess what happened.

Perhaps one of your management hosts is/was compromised. Another possibility would be an as-yet-undiscovered vulnerability since it only occurred on routers with public IPs. I have perhaps 100 MT routers with public IPs and haven't seen any go offline yet.

Please let us know when you discover the cause.
I could not discover the cause but I took help from here and create some firewall rules. and I will set it up to Cpe's which are they using public IPs.
you may have a look to topic
viewtopic.php?f=2&t=132224&p=649875#p649875
 
tippenring
Frequent Visitor
Frequent Visitor
Posts: 92
Joined: Thu Oct 02, 2014 8:54 pm

Re: Lost connection to multiple LHG units

Fri Mar 23, 2018 5:15 pm

I would be curious to see your previous firewall rules to see if there is any obvious weakness.
 
User avatar
amt
Member
Member
Topic Author
Posts: 380
Joined: Fri Jan 16, 2015 2:05 pm

Re: Lost connection to multiple LHG units

Fri Mar 23, 2018 5:28 pm

I would be curious to see your previous firewall rules to see if there is any obvious weakness.
There were no any fw rules at customer side :(
Only drop rules for input and foward chain at pppoe_servers for port 22,23,8291,8728
 
tippenring
Frequent Visitor
Frequent Visitor
Posts: 92
Joined: Thu Oct 02, 2014 8:54 pm

Re: Lost connection to multiple LHG units

Fri Mar 23, 2018 5:37 pm

It sounds like your customers devices were completely exposed to the internet then. Someone probably just brute-forced their way in would be my guess.

I'd suggest you consider retaining connection logs at your border device. It gives you something to review during root cause analysis.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5539
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Lost connection to multiple LHG units

Fri Mar 23, 2018 5:47 pm

There were no any fw rules at customer side :(
Bad Idea, even worse if router has direct access from internet.
 
User avatar
amt
Member
Member
Topic Author
Posts: 380
Joined: Fri Jan 16, 2015 2:05 pm

Re: Lost connection to multiple LHG units

Fri Mar 23, 2018 8:52 pm

It sounds like your customers devices were completely exposed to the internet then. Someone probably just brute-forced their way in would be my guess.

I'd suggest you consider retaining connection logs at your border device. It gives you something to review during root cause analysis.
Thanks for your advice, keeping logs good idea I will do it asap. Thanks again...
 
User avatar
amt
Member
Member
Topic Author
Posts: 380
Joined: Fri Jan 16, 2015 2:05 pm

Re: Lost connection to multiple LHG units

Fri Mar 23, 2018 8:55 pm

There were no any fw rules at customer side :(
Bad Idea, even worse if router has direct access from internet.
I certainly agree with you, it was a very bad mistake. I hope it will be useful for my experience.
 
User avatar
amt
Member
Member
Topic Author
Posts: 380
Joined: Fri Jan 16, 2015 2:05 pm

Re: Lost connection to multiple LHG units  [SOLVED]

Fri Apr 27, 2018 2:10 pm

today I solve the problem and I would like to share with you may help another person in this forum... the attacker change th ereformat-hold-button value and you should keep pressing to reset button untill 5 minute to put device netinstall :=)

Thanks for all help.

Who is online

Users browsing this forum: AnRkey, georgirizov, Renfrew and 21 guests