CCR1009-8G-1S-1S+ keeps crashing

Sn1p3r · April 5, 2018, 8:13am

Hello,

Recently I have encountered problems on some of mine CCR1009-8G-1S-1S+.
I have 9 hotels, in each of them there are 3-4 CCR1009-8G-1S-1S+

It started maybe month ago, without any reason 3 devices, on 3 different locations just stops working, at the same time. It takes around 7-10 days between two halts.
When I come to location on little touch screen I see “it is safe to power off…” after I unplug and plug router back everything works without problem for next 7-10 days.
It just happens on this 3 locations.

It is not directly exposed to internet
There are no opened ports, it’s used for internal management.
Schedulars - empty
Scripts - empty
Log - doesn’t give any informations about halt

Thanks

pe1chl · April 5, 2018, 8:31am

Show the config and the network layout (what is on those networks?).
Maybe there is some worm operating on e.g. network video recorder or cameras.
If possible add some input firewall rule that logs new incoming connections to an external syslog server.

Sn1p3r · April 5, 2018, 8:38am

apr/05/2018 10:35:21 by RouterOS 6.30.2

software id = G870-3RWC

/interface bridge
add name=BrGosti protocol-mode=none
add arp=proxy-arp name=BrOmorika protocol-mode=none
add name=BrTrunk priority=0xA000
/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 max-mru=1480 max-mtu=
1480 mrru=1600 name=Iskon password=xxxx user=xxxx
/ip neighbor discovery
set ether4 discover=no
set BrGosti discover=no
/interface vlan
add interface=BrTrunk l2mtu=1576 name=VlGosti vlan-id=100
add interface=BrTrunk l2mtu=1576 name=VlOmorika vlan-id=13
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=PoOmorika ranges=10.13.0.101-10.13.0.200
add name=PoGosti ranges=172.16.0.31-172.16.3.254
add name=PoPPTP ranges=10.13.0.245-10.13.0.254
/ip dhcp-server
add address-pool=PoOmorika disabled=no interface=BrOmorika lease-time=1w1d
name=DhOmorika
add address-pool=PoGosti disabled=no interface=BrGosti lease-time=1h name=
DhGosti
/ppp profile
set *FFFFFFFE dns-server=10.13.0.1 local-address=10.13.0.1 remote-address=
PoPPTP
/interface bridge port
add bridge=BrOmorika interface=ether2
add bridge=BrOmorika interface=ether3
add bridge=BrOmorika interface=ether4
add bridge=BrGosti interface=VlGosti
add bridge=BrOmorika interface=VlOmorika
add bridge=BrTrunk interface=ether8
/interface pptp-server server
set enabled=yes
/ip address
add address=10.13.0.1/24 interface=BrOmorika network=10.13.0.0
add address=172.16.0.1/22 interface=BrGosti network=172.16.0.0
add address=10.10.103.220/24 interface=ether7 network=10.10.103.0
/ip dhcp-server network
add address=10.13.0.0/24 dns-server=10.13.0.1 domain=hotel.Omorika gateway=
10.13.0.1
add address=172.16.0.0/22 dns-server=172.16.0.1 domain=guests.Omorika
gateway=172.16.0.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip dns static
add address=10.13.0.2 name=CISCO-CAPWAP-CONTROLLER.hotel.Omorika
add address=10.13.0.1 name=RoOmorika.hotel.Omorika
/ip firewall address-list
add address=10.0.0.0/8 list=AdminLAN
add address=172.16.0.0/16 list=GostLAN
add address=172.16.0.2-172.16.3.254 list=GostKlijenti
/ip firewall filter
add action=drop chain=input comment=
“Zabrani ping iz Gost mreze na Admin mrezu” dst-address-list=AdminLAN
protocol=icmp src-address-list=GostLAN
add chain=input comment=“default configuration” protocol=icmp
add chain=input comment=“default configuration” connection-state=
established,related
add action=drop chain=input comment=
“Zabrani konekcije na router sa Interneta” dst-port=!1723 in-interface=
Iskon protocol=tcp
add action=drop chain=input comment=
“Dozvoli konfiguraciju routera samo iz Admin mreze” dst-port=
21,22,23,8291,80,8080 log=yes log-prefix=“=== " protocol=tcp
src-address-list=GostLAN
add action=drop chain=input comment=
“Zabrani konekcije na router sa Interneta” in-interface=Iskon protocol=
udp
add action=drop chain=forward comment=
“Zabrani komunikaciju izmedju klijenata u Gost mrezi” dst-address-list=
GostKlijenti src-address-list=GostKlijenti
add action=drop chain=forward comment=
“Zabrani konekcije iz Gost mreze prema Admin mrezi” dst-address-list=
AdminLAN src-address-list=GostLAN
add chain=forward comment=“Dozvoli promet u Admin mrezi” dst-address-list=
AdminLAN src-address-list=AdminLAN
add chain=forward comment=“Dozvoli Uspostavljene i povezane konekcije”
connection-state=established,related
add chain=forward comment=“Dozvoli ping za sve mreze” protocol=icmp
add action=drop chain=forward comment=“Zabrani SMTP za spamere” dst-port=25
protocol=tcp src-address-list=spammer
add action=drop chain=forward comment=“Zabrani DNS upite sa klijenata”
dst-port=53 protocol=udp
add action=drop chain=forward comment=“Zabrani DNS upite sa klijenata”
dst-port=53 protocol=tcp
/ip firewall nat
add action=masquerade chain=srcnat out-interface=Iskon
/ip route
add distance=1 dst-address=10.0.0.0/8 gateway=10.10.103.1
add distance=1 dst-address=192.168.20.0/24 gateway=10.10.103.1
/ppp secret
add name=xxxx password=xxx profile=default-encryption
add name=xxxxx password=xxxxx profile=default-encryption
/system clock
set time-zone-name=Europe/Zagreb
/system identity
set name=RoOmorika
/system ntp client
set enabled=yes primary-ntp=161.53.30.170 secondary-ntp=161.53.30.50
/system routerboard settings
set cpu-frequency=1200MHz memory-frequency=1066DDR protected-routerboot=
disabled
/system scheduler
add disabled=yes interval=2m name=ScUpdDNS on-event=UpdDNS policy=
ftp,reboot,read,write,policy,test,password,sniff,sensitive start-date=
jan/01/1970 start-time=00:03:03
/system script
add name=UpdDNS owner=admin policy=
ftp,reboot,read,write,policy,test,password,sniff,sensitive source=”#Korisn
icki podaci za DYNDNS\r
\n:local username "xx"\r
\n:local password "xx"\r
\n:local hostname "xxa.dyndns.org"\r
\n\r
\n \r
\n# Ako nema nijednog odgovora na ping azuriraj adresu\r
\nif ([/ping $hostname count=3] = 0) do={\r
\n# Preuzmi IP adresu sa Interneta (U slucaju visestrukog NAT-a)\r
\n\t/tool fetch mode=http address="checkip.dyndns.org" src-path="/" ds
t-path="/dyndns.checkip.html"\r
\n\t:local result [/file get dyndns.checkip.html contents]\r
\n\r
\n# Izdvoji IP adresu iz dobivenog rezultata\r
\n\t:local resultLen [:len $result]\r
\n\t:local startLoc [:find $result ": " -1]\r
\n\t:set startLoc ($startLoc + 2)\r
\n\t:local endLoc [:find $result "" -1]\r
\n\t:local currentIP [:pick $result $startLoc $endLoc]\r
\n\r
\n /tool fetch user=$username password=$password mode=http address="
members.dyndns.org" src-path="/nic/update?hostname=$hostname&myip=$cu
rrentIP" dst-path="/dyndns.txt"\r
\n :local result [/file get dyndns.txt contents]\r
\n :log info ("DynDNS - Host je azuriran IP : $currentIP")\r
\n :put ("Dyndns Update Result: ".$result)\r
\n} else={\r
\n :log info ("DynDNS: Nije potrebno azuriranje")\r
\n}"
/tool graphing interface
add interface=ether1
add interface=BrGosti
/tool graphing resource
add
/tool romon port
add

On this network I have ONLY Cisco controller and Cisco APs, nothing else.

pe1chl · April 5, 2018, 9:03am

Your RouterOS version is very old!
First, update it to “bugfix” or “current” and also update the firmware, 2 reboots required.
Then test again.

We are using the same router on a network with Cisco AP’s and controller without any issue.
Of course what is important too is what kind of users you have on the network…
We are running it in a company for the employees and visitors, and that is usually a less harsh
environment than a hotel.

Sn1p3r · April 5, 2018, 9:16am

In one of the hotels I have everything at newest version 6.41.x
But yet it happand again today

pe1chl · April 5, 2018, 9:31am

What I would do:

install a local computer (can even be raspberry pi)
setup syslog
connect serial port to the computer, if required via USB->RS232 cable
log the serial output using a terminal program running in screen
setup input rule on the router logging all “new” traffic

Then wait until it halts and inspect the logs.

tippenring · April 5, 2018, 2:59pm

Firewall rule 1 permits ICMP from address list GostLAN to AdminLAN.

Firewall rule 2 permits all ICMP. Therefore, rule 1 is not necessary unless you want a separate counter for that traffic.

In the end, there are a few input rules limiting some traffic to the router (address list GostLAN drops connections to several admin ports except 443, UDP packets received on interface lskon are dropped) then everything else is permitted.

I’d say the first thing you need to do is overhaul your firewall rules. Your routers are very exposed.