Cisco DHCPDISCOVER packet ignored by ROSv6.49.19, v7.12.1 & v7.20.4 DHCP server

ROSv6.49.19 DHCP server is completely ignoring a DHCPDISCOVER packet from a Cisco WS-C3550-48-EMI switch running IOS 12.2(35)SE5. The packet is reaching the local bridge interface but there is no response packet sent. All other client devices have no issue with DHCP.

I have spent hours trying to figure out if any setting is wrong and trying and testing many variations while the result remains the same. The key point is that this is the ONLY client that has ever had an issue.

Is this a RouterOS bug?

The DHCPDISCOVER packet capture from the Cisco is;

0000: ff ff ff ff ff ff 00 13 7f 48 76 00 08 00 45 00 ........ .Hv...E.
0010: 02 5c 08 5e 00 00 ff 11 b1 33 00 00 00 00 ff ff ..^.... .3......
0020: ff ff 00 44 00 43 02 48 00 00 01 01 06 00 00 00 ...D.C.H ........
0030: 18 5c 00 00 80 00 00 00 00 00 00 00 00 00 00 00 ....... ........
0040: 00 00 00 00 00 00 00 13 7f 48 76 00 00 00 00 00 ........ .Hv.....
0050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
00a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
00b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
00c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
00d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
00e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
00f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0110: 00 00 00 00 00 00 63 82 53 63 35 01 01 39 02 04 ......c. Sc5..9..
0120: 80 3d 19 00 63 69 73 63 6f 2d 30 30 31 33 2e 37 .=..cisc o-0013.7
0130: 66 34 38 2e 37 36 30 30 2d 56 6c 31 0c 06 53 77 f48.7600 -Vl1..Sw
0140: 69 74 63 68 37 08 01 06 0f 2c 03 21 96 2b 34 01 itch7... .,.!.+4.
0150: 03 ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
01a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
01b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
01c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
01d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
01e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
01f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0260: 00 00 00 00 00 00 00 00 00 00

And the decoded version is;

ff ff ff ff ff ff - destination MAC
00 13 7f 48 76 00 - source MAC
08 00 - EtherType (IPv4)

IP Header

45 - Version + Length (20 Bytes) - IPv4 Header Start
00 - ToS (Type of Service)
02 5c - Length of Field (Header + Data) - 604
02 1f - Datagram ID
00 00 - First 3-bits = Flags Field + 13-bits = First Part of Fragment Offset

ff - TTL
11 - Protocol (17/UDP)
b7 72 - Header Checksum
00 00 00 00 - Source IP Address - 0.0.0.0
ff ff ff ff - Destination IP Address - 255.255.255.255

UDP Header

00 44 - Source Port (68)
00 43 - Destination Port (67)
02 48 - Length - 584
00 00 - Checksum - 0

DHCP Payload

01 - OP - Request
01 - HTYPE - Ethernet
06 - HLEN - Hardware Address Length
00 - HOPS - Number of DHCP Relay Agents
00 00 16 47 - XID - Transaction ID - 5703
00 00 - SECS - Seconds elapsed in Lease
80 00 - FLAGS (0x8000 - Broadcast Requested)
00 00 00 00 - CIADDR - Client IP Address - 0.0.0.0
00 00 00 00 - YIADDR - Your IP Address - 0.0.0.0
00 00 00 00 - SIADDR - Server IP Address - 0.0.0.0
00 00 00 00 - GIADDR - Gateway IP Address - 0.0.0.0
00 13 7f 48 76 00 00 00 00 00 00 00 00 00 00 00 - CHADDR - Client Hardware Address

00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - Sname (Server Name - 64 Octets)
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - Boot File - 128 Octets
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

63 82 53 63 - Magic Cookie

DHCP Options

35 01 01 - DHCP Option 53 - DHCP Message Type - DHCPDISCOVER
39 02 04 80 - DHCP Option 57 - Maximum DHCP Message Size - 1152
3d 19 00 63 69 73 63 6f 2d 30 30 31 33 2e 37 66 34 38 2e 37 36 30 30 2d 56 6c 31 - DHCP Option 61 - Client Identifier - cisco-0013.7f48.7600-Vl1
0c 06 53 77 69 74 63 68 - DHCP Option 12 - Host Name - Switch
37 08 01 06 0f 2c 03 21 96 2b - DHCP Option 55 - Parameter Request List - [01 - Subnet Mask] [06 - DNS Name Server] [0F(15) - Domain Name] [2c(44) - NetBIOS over TCP/IP Name Server] [03 - Router Address] [21(33) - Static Route] [96(150) - Etherboot] [2b(43) - Vendor ID, Vendor Specific]
34 01 03 - DHCP Option 52 - Overload - 3
ff

00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
01a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
01b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
01c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
01d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
01e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
01f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0260: 00 00 00 00 00 00 00 00 00 00 ........

Can you compare it to the working packets? Maybe it is treted as malformed one? Does any other DHCP server answers correctly to that request?

Yes, I did compare it to a working packet from another device and although there will always be differences I didn't see anything that would break the protocol rules unless someone else can spot something.

Your suggestion to try another DHCP server is a good one. I think the only reason I didn't do that sooner was too much focus on trying to analyze what was in front of me. So I did grab another router (LinkSys) and the Cisco switch gets a lease immediately.

That would seem to confirm that the issue is with the MikroTik. (RB750Gr3).

I am not expecting that it can be solved by any configuration changes in the MikroTik, but I wouldn't complain if I was proven wrong :slight_smile:

I used the wrong reply button but did post a reply.

For me the DHCPDISCOVER packet is very OVERSIZED. Option 52 adds a lot of empty data. What for? Can you remove option 52 from the dhcp client? Do other devices send so big requests?
https://www.cisco.com/c/en/us/td/docs/ios/12_2sb/12_2sba/feature/guide/sbhcpcf.html
https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipaddr_dhcp/configuration/xe-16-10/dhcp-xe-16-10-book/config-dhcp-client-xe.html
Reading these manuals you can spot that there was a lot of changes among DHCP client versions.
I do not know if Mikrotik implemented DHCP client RFCs properly or maybe too strictly and therefore ignores "malformed" requests.
LinkSys is the gear from the CISCO stable so I presume that there would be "silent agreement" among brands to support each other.

The articles you linked to are interesting but I don't think IOS 12.2(35)SE5 has configurable DHCP.

Most DHCP clients I have looked at have a DHCPREQUEST packet size of 342 Bytes (or close to that) but I do have an APC AP9319 Environmental Monitor with a DHCPREQUEST packet size of 590 Bytes. It would take more work to trigger DHCPDISCOVER packets from other devices.

I don't think it is specifically Option 52 that pads the packet size. It is followed by "ff" that signals the end. It is possible to have many DHCP Options specified and if more are added, the packet size should be able to get even larger. The packet size is specified in the IP Header near the beginning of the packet.

This page - https://support.huawei.com/enterprise/en/doc/EDOC1100174721/2b689419/dhcp - shows just how many DHCP Options are possible.

This is the other page that I used to decode the packet - https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol

It should even be possible to add DHCP Option data into the MikroTik dhcp-server options area to be able to return requested data to the client. I would expect the MikroTik to just simply ignore requests for Option data that it didn't have.

I can't even know yet what part of the packet is causing the router to ignore it. Either MikroTik Support would need to look at it, or I will try to find a DHCP testing program that will allow manipulation of the DHCP packet to see if it can be found what is breaking it. I am sure there are some available but it might take some time to go through and try some of them.

I also plan on testing with more different routers when time permits. I also have some Cisco WS-C3548-XL-EN switches that I will test in the near future to see what their behavior is.

Also, there are many IOS releases found here - https://archive.org/details/cIOS-firmware-images - but there are no release notes so caution is needed in identifying the correct files. Even if a change in IOS version could get rid of the problem, I think a better understanding of what is happening would be the preferred solution.

Is that RB750Gr3 your only MikroTik device? When you have others, try it on one that runs RouterOS v7.xx.

The RB750Gr3 can also be upgraded to RouterOS v7 but you need to be careful and be prepared for some work, when you have a complex configuration working on v6. It would be best to netinstall it with v7.19.6 and then configure it again from defaults.

And, you may lose some performance as usual when upgrading old hardware to new software.

I do have a bunch more MikroTik routers and some other brands too. This issue isn't something that I have to get fixed as a priority, but I got really interested in trying to find out what the problem is. It might take a few days to do some of that testing.

The other thing that has got my interest is trying some of the DHCP testing programs and seeing how configurable the packets are with the different programs available. This approach is probably my best option and could result in finding a really useful troubleshooting tool.

I keep forgetting to use the correct reply button :frowning:

I only wanted to point out that you are looking for a possible bug in 6.49.19, which will have no priority at all to get fixed, and it may be already fixed in 7.20. A possible reason is already given above.

Well my curiosity made me keep looking :slight_smile:

The first DHCP test program I looked at was CyberShadow's dhcptest program;
https://github.com/CyberShadow/dhcptest
https://files.cy.md/dhcptest/

This command works;

dhcptest-0.9-win64.exe --query --mac 00:13:7f:48:76:00 --option "53[hex]=01" --option "57=1152" --option "61[hex]=00636973636f2d303031332e376634382e373630302d566c31" --option "12[hex]=537769746368" --request 1 --request 6 --request 15 --request 44 --request 3 request 33 --request 150 --request 43

But adding Option 52 makes it fail;

dhcptest-0.9-win64.exe --query --mac 00:13:7f:48:76:00 --option "53[hex]=01" --option "57=1152" --option "61[hex]=00636973636f2d303031332e376634382e373630302d566c31" --option "12[hex]=537769746368" --request 1 --request 6 --request 15 --request 44 --request 3 request 33 --request 150 --request 43 --option "52[hex]=03"

Some notes about Option 52;

DHCP option 52 is known as "Option Overload" and is used to indicate that the DHCP 'sname' or 'file' fields are being used to carry additional DHCP options, typically when the standard option space is insufficient.
This option allows the server to overload the 'sname' (server name) or 'file' (bootfile name) fields with DHCP options, which the client must interpret after processing the standard options.
The code for this option is 52, and its length is 1 octet, with legal values being 1 (indicating the 'file' field is used for options), 2 (indicating the 'sname' field is used for options), or 3 (indicating both fields are used for options).

It is important to note that the DHCP server only sends option 52 if the client explicitly requests it, typically through a parameter request list or a specific configuration like also request dhcp-option-overload; in ISC dhclient.
Furthermore, option 52 is specific to DHCP and should not be used in BOOTP replies, as it signals that the 'sname' and 'file' fields are being overloaded with DHCP options, which is not standard in BOOTP.

It appears that Option 52 is not meant to modify the client DHCPDISCOVER packet because if I look at the decoded packet, then Sname and Boot File sections are empty and the act of requesting a larger Option 55 Parameter Request List doesn't use much space. If the DHCP server had all of the requested Option info, the response packet could be overloaded and may have to use some of the Sname and Boot File sections.

By default, the MikroTik router likely doesn't have enough Option information to need the Option Overload function. But now I am wondering what would happen if enough Options and data were added to 'dhcp-server options' to reach that limit? Since I am thinking that this is definitely a bug in the router, I suspect the results wouldn't be pretty!

Now that I think that I have the answer (being proven wrong is not impossible), the only point in testing more routers/DHCP servers is to see if any others have this issue. I will have to check my RB5009UG+S+IN to see if it is running v7. I hope this issue hasn't continued in the latest version.

A little bit of additional info about Option 52 is that it is rarely sent by clients and it is up to the DHCP server to decide if Option 52 is needed to specify that enough data is returned to require Overloading in the sname and boot file fields.

Since there is no option to control the inclusion of Option 52 in Cisco IOS 12.2(35)SE5, it seems that it would be a simple enough fix to have the RouterOS DHCP server simply ignore Option 52 in the DHCPDISCOVER packet, instead of choking on it, as unlikely as it is to be fixed in v6.

At some point I will still have to test this behavior in v7.

Do commands change anything or do they are absent from 12.2(35)?

ip dhcp client option 52 ascii test52

or

no ip dhcp client option 52

According to some online sources (not recorded) that ability doesn't exist in 12.2(35)SE, but I should be thinking 'not guaranteed to be reliable', so the next time I have that switch powered up, I will try that command. IIRC entering 'ip dhcp client ?' didn't show any appropriate items.

I also remember reading that those type of items can only be executed before the interface is configured. I will try that too.

I don't have much time to play with this tonight but I did manage to fire up the RB5009 and confirm that the same issue exists in RouterOS v7.0.5

Odd that I don't see v7.0.5 on the MikroTik Archive.

7.0.5? Did it come with that and you never upgraded it?

I would not consider < 7.8 a representative and workable v7 release…

Try upgrading it to 7.19.6 and try again.

I can now confirm that the same issue exists in RouterOS v7.12.1 and v7.20.4

As far as removing DHCP Option 52 from the Cisco DHCPDISCOVER packet in IOS 12.2(35)SE5, it just isn't possible. You can remove any of the requested Options (except Option 1) from inside Option 55, but Option 12 (not an issue) and Option 52 are outside of the Option 55 Parameter Request List.

It is probably not that cool that Cisco has included this unnecessary Option in the DHCPDISCOVER packet but at least Linksys (and probably other router brands) have no issue with it.

I suspect it is not a difficult fix for MikroTik and now the best option is probably to do a support ticket.

Indeed that is the best option for you… with the research you have already done, it is probably a simple change for them.

Did some tests - frankly one :slight_smile:

My old 892 is loaded with 15.3 and when I try to change the dhcp client options, the only ones available are:

Router(config-if)#ip dhcp client request vendor-specific ?
  classless-static-route       Classless static route (121)
  dns-nameserver               DNS nameserver (6)
  domain-name                  Domain name (15)
  netbios-nameserver           NETBIOS nameserver (44)
  router                       Default router option (3)
  sip-server-address           SIP server address (120)
  static-route                 Static route option (33)
  tftp-server-address          TFTP server address (150)
  vendor-identifying-specific  Vendor identifying specific info (125)
  <cr>

There is no option 52 available at all.
Reading this: https://www.cisco.com/c/en/us/td/docs/ios/12_2sb/12_2sba/feature/guide/sbhcpcf.html#wp1050352 I spotted that 12.2(27) was the first version where some dhcp client options were integrated into IOS. Looking for option 52 I have noticed that it is mainly for servers that respond with option 52 to indicate that there is some extended data spaning over some possibly unsed fields.
IMHO that was a bug in 12.2 version that allowed to include option 52 in the dhcp reqest, or to better say: option 52 in the dhcp request was allocating the space in the packet as if it was an answer, not a request. What the space was allocated for? The answer was the totally new packet, so there was no need to allocate the space. Dhcp requet should just include the request for data prvided by the server for iption 52, so it is enough to specify that number as one byte in the data part of a request.

This whole thing is probably a result of the code starting out as BOOTP and later changing into DHCP or even still being compatible with it.

In BOOTP the request packet has empty spaces where the server fills in the desired data and sends it back as a reply to the client.