CCR2216-1G-12XS-2XQ - All Interfaces and IP Addresses disappeared

Model: CCR2216-1G-12XS-2XQ

ROS/Firmware: 7.19.4

First time I've ever experienced this issue, and of course on our Core Router. I was moving a vlan to a different interface, and winbox seemed to freeze. Now that I got back in, I have no interfaces or IP addresses, although I can tell by looking at downstream routers(and lack of phone calls) that it's still passing traffic and OSPF routes are still active and working. I can see the different IPs in neighbors from downstream routers as well. Has anyone experienced this issue before? Luckily backup script was working and it has a backup from yesterday still. Should an early morning reboot suffice, or am I in "start throwing and hope something sticks" territory?

That's a no fun kind of situation. :face_with_peeking_eye:

If it were me, I'd plan on a full power cycle during a maintenance window and be ready with a plan B just in case. I would not do anything to rock the boat until I was fully prepared.

I would guess this is most likely some kind of memory corruption. Probably a reboot would suffice, but I have seen a few situations over the years where the issue was electrical and only a power cycle would resolve it. I can't say I've seen exactly the type of failure you're seeing though. That's why I would want to be prepared for the worst.

I did have a scenario on a Ryzen server a few of years ago where the CPU failed. It remained running but was giving odd errors on certain things. When we tried to reboot. it wouldn't come back up and we were down until we could get spare parts to put in. Ended up getting a new motherboard, CPU, and RAM because we couldn't source the precise parts at the time due to covid supply chain issues.

We did end up getting a replacement CPU from AMD and were able to get the server back up once it came in weeks later. It was a bit of an ordeal. :slight_smile:

Power cycle was the right move. Not fun on a router pushing over 7-8Gbps :frowning: Interesting enough, there might have been some early signs. It turns out the backup script stopped working morning of and threw errors, although I can't remember what they were. Pretty sure it was complaining about interfaces.

Aaand it seems after trying to move a VLAN to a different interface, the issue is back.