Follow-up to my earlier post #11 in this thread, which got the wire format
partly right and partly wrong. This post supersedes it. Headline:
We now have a complete spec of the flashfigucis wire format and a
byte-identical Linux encoder.
TL;DR:
- The payload is NOT raw LZMA1 with a 4-byte magic prefix. The LZMA
encoder that FlashFig uses primes its dictionary with a 106,952-byte
corpus of RADIUS-attribute and RouterOS-config boilerplate that is
embedded (pre-compressed) in flashfig.exe itself. The bytes that go on
the wire are what the encoder emits while processing the user script,
continuing from the state the primer left the encoder in.
- Corollary: you cannot replicate FlashFig with stock LZMA1 tooling
(xz, liblzma, Python's lzma module). Every stock encoder produces
output that won't decompress on the RouterBOARD bootloader because the
bootloader's decoder starts with the primer already loaded.
- MikroTik has been using LZMA SDK 16.04 for this for a while.
Versions 19+ of the SDK have a match-finder change that produces ~20
extra bytes on the ~107 KB primer — we have to use 16.04 to match.
Full spec, build instructions, POC encoder, and how to verify/reproduce
against a new FlashFig version are below.
Correction to post #11
Post #11 claimed the payload layout was:
0-3 "123 "
4-5 ff fe
6 LZMA properties byte (0x04)
7-10 dict size, LE (00 00 08 00)
11-18 uncompressed size, LE
19+ LZMA compressed stream
and that the repro command was:
lzma --format=alone --lzma1=lc=4,lp=0,pb=0,dict=524288 < config.rsc > config.lzma
printf '123 \xff\xfe' | cat - config.lzma > config.flashfig
Both are wrong.
- There is no 13-byte LZMA "alone" header between the magic and the
compressed data. The observed bytes at offsets 7-10 of
sjthomason's captured flashfigucis.bin are 00 01 5A 6F, not
00 00 08 00; they are part of the 6-byte slot header described
below.
writeEndMark = 0 — the wire stream has no end-of-stream marker.- The LZMA encoder is not called with the user script as direct input;
it is primed first. Raw lzma --format=alone output will not decode
on a RouterBOARD.
Correct wire format
bytes 0–3 ASCII "123 " (0x31 0x32 0x33 0x20)
bytes 4–9 6-byte slot header
bytes 10+ LZMA1 continuation stream
6-byte slot header
| offset |
size |
field |
| 0-1 |
16-bit BE word |
Type indicator. High bit, if set, flags the word as encoded as ~type; otherwise it is type directly. Type 1 = RouterOS script, type 2 = RouterOS config. FlashFig's wire always uses the flag-encoded form (FF FE = ~1). |
| 2 |
low 6 bits |
Flag/version byte. Observed 0x00 on the embedded primer blob, 0x04 on the user wire slot. This is not the LZMA properties byte (lc/lp/pb are fixed in the encoder, not carried in the header). |
| 3-5 |
24-bit BE |
Uncompressed size (must be ≤ 0x80000). |
LZMA1 continuation stream
The trailing bytes are not a fresh LZMA1 stream. They start mid-range-coder
state as a continuation of the dictionary priming described below. A stock
LZMA1 decoder starting from an empty dictionary will not decode them.
In the reference PCAP capture (config.rsc 346 B CRLF):
0000 31 32 33 20 FF FE 04 00 01 5A 6F 12 61 EF 15 87 …
0010 F2 E2 5C C5 6B BB 6D A9 E6 C9 5B FC 4C E0 AF 52 …
│——magic——│—type—│ │ │—size—│ │—LZMA1 continuation—
│ └── flag byte (0x04)
└── "low-6-bits" byte (also 0x04 here; coincidence)
0x01 5A = 346, matching the 346-byte config.rsc input.
Dictionary priming
The key mechanism post #11 missed. flashfig.exe embeds a 17,650-byte
pre-compressed LZMA1 blob in its .rdata section. In v7.22.1 it lives
at VA 0x4555c0. The blob has the same wire-format structure (6-byte
slot header + LZMA1 stream) and decompresses to exactly 106,952 bytes
of RADIUS attribute dictionary and RouterOS configuration templates:
,"User-Name","User-Password","CHAP-Password","NAS-IP-Address","NAS-Port",
"Service-Type","Framed-Protocol","Framed-IP-Address","Framed-IP-Netmask",
"Framed-Routing","Filter-Id","Framed-MTU","Framed-Compression",…
/ip firewall address-list add list=test address=1.2.3.4
/routing bgp instance set default client-to-client-reflection=yes
…
At encode time FlashFig creates a single LZMA encoder instance, feeds
the decompressed primer to it first, then feeds the user's .rsc. The
output the encoder produced during primer processing is discarded. Only
the bytes emitted while processing the user script are written on the
wire. Because both runs use the same encoder instance, its match-finder
dictionary is populated with primer tokens by the time the user script
is encoded — so common MikroTik syntax (/system, set, address=,
RADIUS attributes, etc.) compresses to short match references.
The RouterBOARD bootloader does the mirror of this: it has the same
primer in ROM (otherwise nothing would decode), feeds it through its
LZMA decoder first, then continues decoding the wire stream to recover
the user script.
This is why the compression ratio on real configs is so much better than
stock LZMA1: config.rsc (346 B) → stock LZMA1 produces ~285 B; FlashFig
produces 126 B of stream (+7 B of header = 133 B on the wire).
LZMA parameters
From disassembly of flashfig.exe v7.22.1 at VA 0x43d5d4 (the dual-mode
encode/decode function, encode branch at 0x43d605+):
| param |
value |
notes |
level |
9 |
LZMA SDK preset level |
dictSize |
0x80000 |
524,288 bytes |
lc |
4 |
literal context bits |
lp |
0 |
literal position bits |
pb |
0 |
position bits |
fb |
273 |
num-fast-bytes (0x111) |
numHashBytes |
3 |
combined with default btMode=1 → BT3 match finder |
mc |
0x80000 |
matchFinderCycles (very aggressive) |
writeEndMark |
0 |
no end-of-stream marker |
numThreads |
1 |
|
Everything else is the LZMA SDK's default after LzmaEncProps_Normalize.
Byte-identical Linux encoder
A CC0 (public-domain-dedicated) reference encoder in C, using the vendored
reference LZMA SDK 16.04, is at:
https://gitlab.com/ggiesen/mtffencoder
Verified byte-identical against sjthomason's PCAP capture:
$ ./mtffencoder ./flashfig.exe ./config.rsc /tmp/out.flashfigucis
primer: 17650 bytes embedded -> 106952 bytes text
wrote: 133 bytes -> /tmp/out.flashfigucis
$ sha256sum /tmp/out.flashfigucis ./flashfigucis.bin
53c32f29… /tmp/out.flashfigucis
53c32f29… ./flashfigucis.bin
Two implementation gotchas
-
LZMA SDK version. Every SDK from 19.00 through 24.09 produces 20
extra bytes on the primer compared to what MikroTik ships. SDK 16.04
matches. If MikroTik later upgrades their SDK, this encoder will need
a matching-version LZMA SDK update.
-
The primer-to-user boundary in the output. You cannot produce
user_wire = combined_output[primer_alone_size :]. The last ~4 bytes
of primer-alone encoding are range-coder flush bytes that are replaced
in combined encoding by actual user-encoded bytes. The correct way to
find the boundary: encode primer alone, encode primer+user, find the
first byte position where the two encodings diverge — that is where
the user portion starts.
Reproducing this for a new FlashFig version
If MikroTik releases a new flashfig.exe with a different primer or
changes anything, here is how to find out.
1. Locate the primer
The repo includes find_primer.py which scans any flashfig.exe's
.rdata section for a 6-byte slot header whose trailing LZMA1 stream
decompresses to exactly the size the header claims:
$ python3 find_primer.py flashfig.exe
.rdata: VA 0x453000, size 41240 bytes, file offset 0x51a00
✓ primer found:
VA 0x4555c0
file offset 0x53fc0
blob size 17650 bytes
decompressed size 106952 bytes
Update mtffencoder.c with:
#define PRIMER_VA 0x4555c0u
#define PRIMER_BLOB_SIZE 17650u
Paste those two values into mtffencoder.c and rebuild.
2. Verify parameters haven't changed
Capture FlashFig's wire output for a known .rsc (use Wireshark on a
Windows or Wine session; or set up a local TFTP server and save what
FlashFig sends it). Run the encoder against the same input. If
byte-identical, you are done. If not, the codec parameters have
changed; you'll need to re-extract them by disassembling around the
encoder call site (search for calls to LzmaEnc_SetProps).
3. If the byte count diverges mid-stream
Most likely cause: MikroTik upgraded their LZMA SDK version. SDK 16.04
matches v7.22.1; SDKs 19.00+ produce 20 extra bytes on the primer. Try
each SDK version until you find one that matches.
Files
mtffencoder.c — CC0 encoder (~400 LoC) using LZMA SDK
LzmaEnc.c/h, … — vendored LZMA SDK 16.04 (public domain)
find_primer.py — locates primer in any flashfig.exe (CC0)
Makefile — `make` / `make test`
README.md — detailed docs including build + repro
LICENSE — CC0 1.0 Universal dedication
Build:
$ make
cc -O2 -Wall -Wextra -D_7ZIP_ST -Wno-dangling-pointer -o mtffencoder \
mtffencoder.c LzmaEnc.c LzFind.c Alloc.c LzmaDec.c CpuArch.c
Runs on Linux, macOS, any POSIX. Single 65-70 KB static binary. No
runtime dependencies beyond libc.
Credits
rcy (2018) — original thread starter; solved this via
capture-and-replay from FlashFig.sjthomason — committed a reference PCAP, config.rsc, and the
flashfigucis wire capture that made byte-level verification possible.
The Rust encoder in that repo should be updated to drop the
alone-format header and add primer priming; happy to help with that
if useful.normis — declined to help; respected.- Lots of help from an LLM in the decoding and POC work