LtAP, Verizon, Quectel EC-25AF no worky

ds37577 · Tue Nov 05, 2024 2:27 am

Hi all,

I'm trying to get an LtAP configured to work in a vehicle for Verizon (well, Visible) LTE. The LtAP came with an FG621-EA modem in it, which I was disappointed to find out was not really for use in the US. So, some googling told me that the Quectel EC-25AF was "the" modem for use in the US with Verizon, so I bought one and have swapped it out. The LtAP sees it as lte1 in MBIM mode, showing the proper vendor/model of the modem, etc. Putting a T-Mobile SIM in results in a quick attach to their LTE network and an IP address. Not so much with the Verizon one, which is what I really need to work.

The Verizon SIM *does* "just work" with the FG621-EA modem, albeit with very limited coverage due to the lack of band coverage I guess.

Anyway, the modem just refuses to ever get an IP from Verizon. Modem config:

0 default-name="lte1" name="lte1" mtu=1500 apn-profiles=verizon
allow-roaming=no sms-read=no sms-protocol=mbim network-mode=lte

APN config:

2 name="verizon" apn="VZWINTERNET" use-peer-dns=yes use-network-apn=no add-default-route=yes default-route-distance=2
ip-type=auto authentication=none

I have tried all kinds of things like "use network APN", IPv4 only, IPv6 only. I am unable to get any AT commands to the modem using at-chat, being always told the modem "is not ready" or "is too busy". The serial terminal doesn't work either - my usb3 device only has two channels available, neither of which respond *at all*, much less to things like "ATI" which should barf out something. I've even tried putting things in the modem-init string on the lte interface, but I never see them get sent to the modem with lte tracing turned on, even though I see it send other AT commands and get responses during setup. I've seen mention of needing to "enable IPV4V6 mode" to work with verizon, but if I can't get AT commands to the modem I can't even look to see if that's being done.

I'm at a bit of a loss here and pretty frustrated. I see other reports of EC25-AF working fine with Verizon (modulo disconnection problems due to invalid src-addresses, and such, but I never get that far). The SIM works fine in a Cradlepoint device I have, as well as an old Pixel 5. With lte tracing in the logs, I see something like this slowly repeat forever:

22:03:20 lte,debug lte1 mbim: retrying registration
22:03:20 lte,debug lte1 mbim: request register, operator: [auto]
22:03:20 lte,debug lte1 mbim: <<< S #14 - connect: register state
22:03:20 lte,packet,raw lte1 mbim: wdm <<< send #14
22:03:20 lte,packet,raw lte1 mbim: 03000000 40000000 0e000000 01000000
22:03:20 lte,packet,raw lte1 mbim: 00000000 a289cc33 bcbb8b4f b6b0133e
22:03:20 lte,packet,raw lte1 mbim: c2aae6df 09000000 01000000 10000000
22:03:20 lte,packet,raw lte1 mbim: 00000000 00000000 00000000 20000000
22:04:08 lte,packet,raw lte1 mbim: wdm >>> recv #14
22:04:08 lte,packet,raw lte1 mbim: 03000080 60000000 0e000000 01000000
22:04:08 lte,packet,raw lte1 mbim: 00000000 a289cc33 bcbb8b4f b6b0133e
22:04:08 lte,packet,raw lte1 mbim: c2aae6df 09000000 00000000 30000000
22:04:08 lte,packet,raw lte1 mbim: 00000000 01000000 01000000 00000000
22:04:08 lte,packet,raw lte1 mbim: 01000000 00000000 00000000 00000000
22:04:08 lte,packet,raw lte1 mbim: 00000000 00000000 00000000 02000000
22:04:08 lte,debug lte1 mbim: >>> D #14 - connect: register state
22:04:08 lte,packet lte1 mbim: status data:
22:04:08 lte,packet lte1 mbim: 00000000 01000000 01000000 00000000
22:04:08 lte,packet lte1 mbim: 01000000 00000000 00000000 00000000
22:04:08 lte,packet lte1 mbim: 00000000 00000000 00000000 02000000
22:04:08 lte,debug lte1 mbim: register state: deregistered
22:04:08 lte,debug lte1 mbim: state 12=>11

Can anyone help me make some progress here? Why can't I send AT commands to the modem? Any other tips?

Amm0 · Tue Nov 05, 2024 5:52 am

Verizon is strict on everything. The modules must be whitelisted by a device manufactured and go through Verizon-specific certification. Modules are certified too, but BOTH module and devices require certification. So this means, unless the module came from a previous Verizon device, you will not be able to activate it with Verizon (or Visible). But... if you have a SIM that was activated on another device, that should work.

I suspect the issue is Verizon implements a strict filter on any malformed packets that escape. Various things on RouterOS, especially when modem is middle of starting up, can cause packet that is not NAT'ed & that Verizon will drop the connection.

I'd try adding a firewall filter for "invalid" going out to lte1 (verizon)

/ip/firewall/filter/add add action=drop chain=output connection-state=invalid out-interface=lte1

I'm not the expert on the Quectels, but they should not need anything for Verizon AFAIK. To access more ports, you can set the mode=serial under modem in LTE, and you should be able to use /system/serial-terminal if you do want to enter LTE commands. But I'm guess it packets escaping and Verizon-end dropping on a bad packet.

ds37577 · Tue Nov 05, 2024 7:17 am

Yeah, my SIMs are already active, so I figured having the Verizon-blessed Quectel model should be okay.

I added the filter rule and it seems to be doing the same exact thing. Also zero packets hit that rule. I had already disabled cloud and neighbor discovery based on reading about this issue, so maybe that's why.

I did set it to serial, reboot and get all the serial channels so I could issue AT commands, thanks. Seems like I have to set mode=serial, then reboot, issue commands, and then change the mode back from serial to something else and reboot again? My /system reboot seems to shut down but never reboot, so I have to physically power cycle the device, which is super annoying. Am I missing something?

I tried setting it to ECM mode as some people said that worked better and was the only mode supported on ROS6. I did that and rebooted and it seems like as soon as the router came up, it issued a slew of AT commands to the modem to interrogate it, and then set it back to MBIM mode. The modem immediately disappeared from the bus and I rebooted to find it in MBIM mode again. In serial mode, lte1 is gone and I can't make any changes against it.

Many of the posts I've read are from (or involve you), but many were from ROS6 days. Any other ideas? Can I run it in PPP mode from mode=serial or something? I see reference to that. I don't see anyone explaining what the setup is there though...

Thanks for your help so far!

Amm0 · Tue Nov 05, 2024 7:45 am

In V7, Mikrotik use MBIM with their Quectel models, so I suspect that be best. But if T-Mobile is working...I'm not thinking it's MBIM vs ECM.

So yeah switch the LTE modem back to mode=auto may be best plan. Also try lowercase "vzwinternet" and disable "Use Network APN". You may also want to IPv4 as the address mode, NOT auto or IPv4v6. Additionally make sure RouterOS is at stable, and the /system/routerboard has been updated to latest firmware too.

With all that set, I would try one (or two) hard power-cycles (i.e. not /system/reboot, un-plug for 30 seconds), with Verizon SIM and APN set to vzwinternet. The USB power reset does kill the modem, but I've found sometimes a hard power reset is needed for IMEI to update.

It's possible there is some carrier firmware switch in the Quectel for Verizon, but the older ones never need anything. But I'm a bit limited since I've never used the EC25s, only the EP06 a while back in testing.

And it's also possible Visible is more strict on the IMEI matching, IDK maybe Visible service is even more picky about moving SIMs... IDK and these things change. Verizon is always a PITA is the only constant.

Re PPP...
Theoretically, you can use PPP mode. And you'd would use mode=serial for that. But it will be limited to 30Mb/s or thereabouts (or at least that was the limit at some point). But for testing, it should show up with ppp-out1 interface when you put it into serial. You'll need to go into the Advanced PPP settings to set the data-channel= and info-channel= to AT ports. I don't know the port layout for these, especially in serial mode. But if there are two port that work for AT command use the channel= from /terminal/serial-terminal in the ppp-out1's data-channel=. If there another port that works for AT, than use that for the info-channel=. If you only have one AT work, then use that both for the data and info channels. And in PPP client ppp-out1 make sure disable "Dial on Demand", set APN to vzwinternet, and then enable the interface.

ds37577 · Tue Nov 05, 2024 5:05 pm

In V7, Mikrotik use MBIM with their Quectel models, so I suspect that be best. But if T-Mobile is working...I'm not thinking it's MBIM vs ECM.

So yeah switch the LTE modem back to mode=auto may be best plan. Also try lowercase "vzwinternet" and disable "Use Network APN". You may also want to IPv4 as the address mode, NOT auto or IPv4v6.

Okay, this is how I had it originally set up (I had tried auto, IPv4, IPv6).

Additionally make sure RouterOS is at stable, and the /system/routerboard has been updated to latest firmware too.

I had no idea that there was separate board firmware from ROS. I upgraded this and it seems to now work in MBIM mode without issue. Seems hard to believe, but ... does that seem legit?

With all that set, I would try one (or two) hard power-cycles (i.e. not /system/reboot, un-plug for 30 seconds), with Verizon SIM and APN set to vzwinternet. The USB power reset does kill the modem, but I've found sometimes a hard power reset is needed for IMEI to update.

USB power reset does not seem to do anything for me. I've done a 5s reset on all the available busses and I never see anything at all in the logs. Although maybe with this new firmware...

It's possible there is some carrier firmware switch in the Quectel for Verizon, but the older ones never need anything. But I'm a bit limited since I've never used the EC25s, only the EP06 a while back in testing.

And it's also possible Visible is more strict on the IMEI matching, IDK maybe Visible service is even more picky about moving SIMs... IDK and these things change. Verizon is always a PITA is the only constant.

From what I've been able to tell, since mid-2023 Visible is basically just Verizon now in terms of all that stuff. They stopped doing the cloud-core routing, using a separate APN, etc. I have a Verizon SIM that yields the exact same config as the Visible one(s) in all the devices I've got.

Anyway, thanks so much for your help, I seem to be good. But, one last question: I seem to have no signal/band information at all in the status views of the modem. There are lots of AT commands available to pull that info out, but I don't seem to be able to see any of those in MBIM mode. Is there some trick or do I just have to fly blind?

Amm0 · Tue Nov 05, 2024 7:55 pm

Additionally make sure RouterOS is at stable, and the /system/routerboard has been updated to latest firmware too.
I had no idea that there was separate board firmware from ROS. I upgraded this and it seems to now work in MBIM mode without issue. Seems hard to believe, but ... does that seem legit?

Yeah I've found they need to align - otherwise exactly same weird issues in initializing*... Often enough that I enable the "auto-update" in /system/routerboard so that it gets updated, and I generally don't have problems. But it does mean you do a "2nd reboot" after any upgrading RouterOS version since RouterBOOT only gets applied at startup and not at same time as OS.

Not sure exactly why it effects LTE. i.e. it's like GRUB so once RouterOS boots, I'd think it go away once boot. But it may reset something or use a different parameter to start RouterOS. I'd imagine in most cases it applies the same firmware code, but there is not a seperate changelog so you really don't know if there any update to the firmware.

* It going to running, then disconnecting a few seconds later is typically the symptom of [unintentionally] "snoof'ed" packets escaping problem. But it failing in middle of MBIM setup... that's where you want to make sure all versions align (boot, routeros, latest modem firmware).

From what I've been able to tell, since mid-2023 Visible is basically just Verizon now in terms of all that stuff. They stopped doing the cloud-core routing, using a separate APN, etc. I have a Verizon SIM that yields the exact same config as the Visible one(s) in all the devices I've got.

I'd believe that. Visible did not support watches etc for a while...but they do now. Visible adding more layers over Verizon was my worry. VZW is annoying enough, without adding more things.

But, one last question: I seem to have no signal/band information at all in the status views of the modem. There are lots of AT commands available to pull that info out, but I don't seem to be able to see any of those in MBIM mode. Is there some trick or do I just have to fly blind?

There may be more tricks here. All MBIM should show at least "RSSI" (which may or may not be actually RSSI since MBIM spec calls it signal strength and modem provide it via their MBIM implementation). RouterOS typically does read AT to get CA, RSRQ, RSRP, CQI, and SINR. Since it's Quectel, and they do read all those for other Quectel... it should actually work - which be the benefit of Quectel but modems like Sierra ones, RouterOS has no built-in things to parse it AT responses for signal.

Did T-Mobile show RSRQ and RSRP? Since it could be the carrier firmware on modem (so yes the modem has two versions, one for the base and then version of the "carrier firmware"). And sometimes the Verizon firmware does act differently since Verizon has a lot of specs on what modems should do. And Mikrotik likely only test the "generic" carrier firmware (which is typically also T-Mobile since there not bully like AT&T and Verizon that do mandate specific things on modems).

mkx · Tue Nov 05, 2024 8:11 pm

... i.e. it's like GRUB so once RouterOS boots, I'd think it go away once boot.

It's not like grub, it's like BIOS or UEFI ... it initializes all hardware and can put it into some weird state which can't be remedied by drivers.

ds37577 · Tue Nov 05, 2024 8:29 pm

Nope, no signal stats at all, even RSSI from either SIM. When I had the original modem I got lights on the outside, bar graph on the quick set page, and lots of (RSSI plus others) stats somewhere in the LTE interface page. Unless there's some other way to see it, I'm not getting anything. I don't really care too much about complex stats, I'd just like at least one number so I can aim antennas, compare locations, etc.

I've heard there are firmware update possibilities from Quectel, but that getting and applying them are not super easy. If that was likely to fix it, I'd consider going down that path...

ds37577 · Tue Nov 05, 2024 9:11 pm

Actually, it looks like I might already be on the latest rev (EC25AFFAR07A14M4G) according to some posts from July. So maybe it's too new

Amm0 · Tue Nov 05, 2024 9:32 pm

Actually, it looks like I might already be on the latest rev (EC25AFFAR07A14M4G) according to some posts from July. So maybe it's too new

It should show "RSSI". If you can try T-Mobile again, that be worth a shot to know if it's a Verizon carrier firmware issue.

You might want to open a ticket at help.mikrotik.com on this one. Mikrotik generally does fix these things. If you file a ticket make sure to enable LTE debug logging,

/system/logging/add topic=lte action=memory

...
Then reboot the device... let it run a minute... than run the "Make supout.rif" from winbox, and include that in your ticket with perhaps a screenshot of the LTE cellular tab. The supout.rif will have the logs and system info for them. But that does depend on the log level, why you want to added the above additional logging. You can remove the added logging after doing the supout.rif.

You can also try the 7.17beta (again update everything). And if you do, and then need to downgrade you'd need to use /system/packages, pick "stable" as channel, and then "upgrade" - do not use the "Downgrade" button, that won't do what you think. Sometimes the newer version does fix things, why the general recommendation to get to EVERYTHING to stable

. A support ticket will take a few days, and if you know 7.17 works that be good info.

... i.e. it's like GRUB so once RouterOS boots, I'd think it go away once boot.
It's not like grub, it's like BIOS or UEFI ... it initializes all hardware and can put it into some weird state which can't be remedied by drivers.

Fair enough, GRUB is likely not good analogy. Main point was RouterBOOT sometime does effect LTE modems... whether it just resetting something IDK.

I had no idea that there was separate board firmware from ROS. I upgraded this and it seems to now work in MBIM mode without issue. Seems hard to believe, but ... does that seem legit?

I did write up the various CLI to update everything for LTE in this post (and longer discussion about Mikrotik + LTE in US):
viewtopic.php?t=199087&hilit=66#p1025119

It's Verizon's Band 13 boondoggle* that adds the complexity here. The Verizon carrier firmware often acts differently even on the same modem type. And Verizon "drop on 'spoof' src IP" rule is one way to find out if any packet leak out of the firewall... but adds complexity to multiwan since masquerade transitions can cause a wrote src IP.

* FCC created Band 13, but required users like Verizon to allow "any" device on the network that meet some fixed requirements... but then allowed Verizon to write those requirements...

ds37577 · Tue Nov 05, 2024 10:12 pm

Sorry, what I meant by "both SIMs" was that neither Visible nor T-Mobile show RSSI (or any other of those signal-related stats).

I've already got the logging enabled, so I'll get a fresh boot on T-Mobile and a couple minutes of logs and then submit, thanks. I'm also willing to try the next beta if you think it's worth it.

And yes, I know that life would just be a lot easier if I didn't use Verizon. However, in the western US in the remote places I need this to work, Verizon is by far the best option. The T-Mobile SIM helps me in a few places where Visible can't roam to one of the other networks when Verizon has less coverage.

Thanks!

Amm0 · Tue Nov 05, 2024 10:31 pm

've already got the logging enabled, so I'll get a fresh boot on T-Mobile and a couple minutes of logs and then submit, thanks. I'm also willing to try the next beta if you think it's worth it.

It be worth a ticket. They may just need some ID specific for the EC-25AF, but only MT would know. There also could be some init string that get the right reporting too.

Adding you tried v7.17 might skip a step in the resolution process. And basically the version is the only knob to tweak on the RSSI report issues, and may be some change in the version that does fix the RSSI reporting, especially since you have newer Quectel firmware. It's 5-10 minutes and 2 reboots to try, and it will take a couple days for Mikrotik to respond to a ticket....

But I'd think it would report more stats, since the Quectel AT command don't vary that much AFAIK. But the RSSI should for sure be there from MBIM.

And yes, I know that life would just be a lot easier if I didn't use Verizon. However, in the western US in the remote places I need this to work, Verizon is by far the best option. The T-Mobile SIM helps me in a few places where Visible can't roam to one of the other networks when Verizon has less coverage.

Yup, there is a reason why I'm familiar with the issues here. But does mean they are not a a PITA. I think most international readers on the forum would be surprised at how improvised the US cellular networks are.

ds37577 · Tue Nov 05, 2024 11:19 pm

I filed the ticket with the supout from stable. Then I upgraded to the beta, and then installed the newer firmware. Now I'm in a crashing boot loop

2024-11-04 23:07:00 system,info router rebooted
2024-11-04 23:07:01 system,error,critical router rebooted because some critical program crashed
2024-11-04 23:07:02 interface,info lo link up
2024-11-04 23:07:13 wireless,info 50:1F:C6:F1:A1:60@wlan1: connected, signal strength -35
2024-11-04 23:07:19 system,info,account user admin logged in from 192.168.105.248 via ssh
2024-11-04 23:07:22 lte,info lte1 mbim: network advertises lower mtu: 1428
2024-11-04 23:07:22 lte,info lte1 IPv4: 100.89.98.227, DNS: 198.224.167.135,198.224.166.135, MTU: 1428
2024-11-04 23:07:22 interface,info lte1 link up
Received disconnect from 192.168.105.1 port 22:11: shutdown/reboot at bottom.

I've never had to recover from an upgrade like this because I always stay on stable on all my mikrotik devices. Is there some way I can interrupt startup and roll back without losing all my config? I should have grabbed a config dump before I updated...

Amm0 · Tue Nov 05, 2024 11:54 pm

Sorry about that, normally the beta do work... And I have 7.17 running on a couple wAPac, but no LtAPs....

You can use the serial port to see what's going on. And perhaps the backup boot loader help, dunno.

But doing a reset to defaults may be quicker.

ds37577 · Wed Nov 06, 2024 12:09 am

Okay, so I reset to defaults with 7.17. After that it was stable _until_ I reconfigured all the LTE stuff and then it started crashlooping again (no RSSI before it crashed either). So, I reset to defaults, gave it a wired default gateway and had it reinstall 7.16, go back to 7.16 firmware, and then reset defaults there again.

By the way, it's doing a thing it did when I was first messing with it, which is as soon as it tries to connect with the verizon SIM, it gets stuck in this "waiting for provisioning (up to 20 minutes)" state where I can't do anything to it because "interface is busy". Last time I just let it sit and it finally freed up. Is this something verizon related, FOTA or something? Once I left it alone long enough to get past that the first time, I hadn't seen that again until this re-reset action.

Either way, assuming that frees up and I can get it back to working 7.16, I'll get all my settings put back in, grab a backup, and be ready for Mikrotik to respond to my ticket

Thanks!

ds37577 · Wed Nov 06, 2024 4:55 pm

Mikrotik responded with basically "if the modem doesn't do it we can't report RSSI" but no indication that they looked at the dump and confirmed that it's missing. Seems kinda weird that anyone would implement MBIM without basic RSSI indication.

It looks like the EP06 is a newer/better Cat6 device, which is maybe what I should have gotten. EP06-A supports Band 13. Do you (or anyone) have experience with that device in MBIM with verizon and RSSI? Or some better other option?

Amm0 · Fri Nov 08, 2024 3:22 am

I'd make sure they know it crashed under 7.17beta - so not just missing RSSI, they should want to take a look at that. It's a bug if something causes a boot loop. I'm pretty the modem is providing RSSI via MBIM since it's requirement and it taking a minute to get to running... kinda does indicate some protocol/timing problem.

I've used the Sierra MC7354 (cat4) and MC7455 (cat6) in past, and Telit LM960A18 (cat18) currently -all support B13. So I know all those work (Sierras show RSSI, while Telit support RSRP/RSRQ/etc). But they generally have good support for Quectel (since some MT devices use Quectel variants)... but you'd have search the forum for more details on EP06. I/other have written about Sierra MC7354/7455 in other posts with the specific on things. On LtAP, the MC7455 and LM960A18 need to use the "bottom" modem slot since that's USB 3.0 (or they could use the top one, but that require a switch to USB 2.0 mode via AT commands).

ds37577 · Wed Nov 13, 2024 1:36 am

So, I snagged an EP06-A. Definitely what I should have gotten in the first place. Put it in, the Mikrotik upgraded the firmware for me. Connected straight away and gives me *all* the stats. Curious if the beta will crash with this or not, but I gotta put this thing in its home so I'll have to let someone else figure that out. I did report it on the beta thread on this forum at least.

I'm pretty happy now, thanks for the help!

LtAP, Verizon, Quectel EC-25AF no worky

LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Re: LtAP, Verizon, Quectel EC-25AF no worky

Who is online