Skip to content

FirmwareStager: support v4.2+ MSI (native libmspack) + user-selected installer#2169

Merged
ten9876 merged 6 commits intomainfrom
aether/firmware-msi-support
Apr 30, 2026
Merged

FirmwareStager: support v4.2+ MSI (native libmspack) + user-selected installer#2169
ten9876 merged 6 commits intomainfrom
aether/firmware-msi-support

Conversation

@ten9876
Copy link
Copy Markdown
Owner

@ten9876 ten9876 commented Apr 29, 2026

Summary

Three coupled changes that make the firmware-update path work for SmartSDR v4.2+ without runtime dependencies:

  1. Native MSI extraction — vendored libmspack (LGPL-2.1, ~6k lines C) plus a small in-tree OLE Compound File reader. Pure in-tree, no 7z, no QProcess, no temp files, no external runtime deps. Works in any sandboxing model that disallows process spawning.
  2. User-selected installer — replaces auto-download with a "Select Installer..." file picker. The user downloads the SmartSDR installer themselves from flexradio.com, then points us at it. No URL guesswork.
  3. PII redactor + diagnostic-logging cleanup — fixes a regex regression (software_ver=4.2.18.41174 was being mangled to *.*.*. 41174 in logs) and removes one-shot diagnostic instrumentation that was added for the first flash attempt.

Why native instead of 7z

Earlier prototypes shelled out to 7z for MSI extraction. That approach worked on Linux but would have created silent regressions for Windows users (overwhelmingly don't have 7z on PATH) and broken in sandboxed environments. The native pipeline:

  • Zero new runtime dependencies anywhere
  • Works in App Sandbox / Snap / Flatpak
  • Returns structured errors we can present in the UI
  • ~3.75 days of work tracked in #2172

Pipeline

SmartSDR_v<x.y.z>_x64.msi
  ↓ OleCompoundFile.readMsiStreamsByPrefixSuffix("cab", ".cab")
cab1.cab, cab2.cab, …  (in memory, MSI-decoded names)
  ↓ CabExtractor.extractFirstMatchingMagic(bytes, "Salted__")
.ssdr blobs  (in memory)
  ↓ sort by size desc, pick by m_modelFamily
chosen blob  →  write to staging dir
Format Detection Extraction
InnoSetup .exe (v4.1.x and earlier) MZ (PE/COFF) header Existing byte-pattern scanner
WiX MSI (v4.2+) OLE magic D0 CF 11 E0 A1 B1 1A E1 New native pipeline
Pre-extracted .ssdr filename suffix Direct copy + header validation

What landed

Vendored library

New AetherSDR sources

  • src/core/OleCompoundFile.{h,cpp} — read-only [MS-CFB] reader. Parses header, FAT chain, directory; reads streams by name. MSI-aware accessor decodes Microsoft's 5-bit-packed name encoding. ~290 lines total.
  • src/core/CabExtractor.{h,cpp} — in-memory mspack_system adapter. Reads cab bytes from a QByteArray and writes decompressed payloads to QByteArrays. ~190 lines.

Changed AetherSDR sources

  • src/core/FirmwareStager.{h,cpp}extractFromMsi() body fully rewritten (~130 → ~80 lines); findExtractionTool() deleted; new stageFromLocalFile() for the file-selection entry point; URL handling switches between _x64.msi (v4.2+) and _Installer.exe (older).
  • src/gui/RadioSetupDialog.cpp — "Browse .ssdr..." renamed to "Select Installer..."; file dialog accepts .msi / .exe / .ssdr; "Check for Update" no longer auto-downloads, just reports the latest version.
  • src/main.cppredactPii regex extended to skip _ver= prefixed and trailing-digit cases.
  • src/core/FirmwareUploader.cpp — diagnostic logging from the one-shot-flash session removed (the actual flash succeeded; the scaffolding isn't needed anymore).

Tests

  • tests/ole_compound_file_test.cpp — 17 assertions covering OLE CFB stream extraction, MSI name decoding, libmspack CAB decompression, and the end-to-end firmware-blob match.
  • Skips with exit 77 if the licensed MSI fixture isn't available (looks at $AETHERSDR_TEST_MSI then ~/build/reference/SmartSDR_v4.2.18_x64.msi). CI doesn't have the file, so the test no-ops there.

Verification

End-to-end test against SmartSDR_v4.2.18_x64.msi:

Check Result
OLE CFB opens
6 cab streams extracted ✅ byte-identical to 7z
Cab MSCF magic 4D 53 43 46
FLEX-6x00 firmware extracted ✅ 386,289,360 bytes, Salted__ header
Final MD5 9e8888dc0558ee420ed82f370f805025 — same bytes that successfully flashed a real FLEX-8600 earlier in this branch
FLEX-9600 firmware extracted ✅ 63,813,240 bytes, Salted__ header

The tracking issue is #2172 — can be closed on merge.

Acceptance criteria (all met)

  • libmspack vendored with LGPL-2.1 text and version-tracking README
  • OleCompoundFile reads streams from real MSI
  • extractFromMsi() no longer calls QProcess or references 7-Zip
  • findExtractionTool() deleted
  • Integration test extracts a .ssdr matching the known-good MD5
  • Manual smoke test in the UI: Select Installer → MSI → staged file matches MD5
  • Build clean on Linux x86_64; tests pass

Test plan

  • Build passes locally
  • ole_compound_file_test passes (17 assertions)
  • Manual UI test: staged file MD5 matches known-good
  • CI green (Linux + macOS + Windows)
  • On-air test: someone running this build flashes their radio with v4.2 firmware

🤖 Generated with Claude Code

FlexRadio switched the SmartSDR installer format in v4.2 from a
self-extracting InnoSetup .exe to a WiX 6 MSI (OLE Compound File with
embedded LZX-compressed CABs). The old byte-pattern Salted__ scanner
fails on the new format — only one Salted__ marker is visible in the
raw MSI, and that marker is incidental MSI structure, not a firmware
boundary.

Changes:

- Add `versionUsesMsi(version)` — returns true for v4.2+ versions.
  Picks the right URL pattern (`SmartSDR_v<v>_x64.msi`) and saves
  with the right extension. Older versions still use the
  `SmartSDR_v<v>_Installer.exe` URL and InnoSetup extraction.

- Detect installer format at extraction time by reading the first 8
  bytes (OLE magic D0 CF 11 E0 A1 B1 1A E1 vs InnoSetup PE/COFF
  "MZ"). Dispatch to the matching extractor.

- Refactor the existing byte-scan logic into `extractFromInnoSetup()`
  unchanged.

- New `extractFromMsi()`: shells out to 7-Zip via QProcess to extract
  cab streams from the MSI, decompress each cab's LZX-packed payload,
  scan the decompressed files for Salted__-prefixed blobs, sort by
  size descending, and pick by family (consumer 6x00 multi-platform
  firmware is much larger than the 9600 government build, giving us a
  stable size-based mapping without needing to parse the MSI File
  table).

- New `findExtractionTool()` helper probes for `7z` / `7zz` / `7za`
  in PATH, plus common Homebrew locations on macOS. Returns empty
  string if none found, in which case extraction emits a clear error
  with installation instructions per platform.

Tested manually against `SmartSDR_v4.2.18_x64.msi`: the algorithm
extracts both firmware blobs (FLEX-6x00 386 MB, FLEX-9600 64 MB) with
intact Salted__ headers. Follow-up work to vendor a CAB+LZX decoder
in-tree (libmspack or similar) is tracked separately so we can drop
the 7-Zip runtime dependency.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@ten9876 ten9876 requested a review from jensenpat as a code owner April 29, 2026 17:58
@ten9876 ten9876 enabled auto-merge (squash) April 29, 2026 17:58
Replaces the auto-download flow with user-driven file selection: the
user downloads the SmartSDR installer themselves from flexradio.com,
then picks it via "Select Installer..." in Radio Setup.

Why:
- No URL guesswork (we don't know what FlexRadio's v4.2 download paths
  are without testing each one).
- No CDN cache concerns or stale MD5-hash-file URLs.
- User already trusts what they downloaded; we just unpack it.
- Works for any installer the user can obtain — including beta/test
  builds FlexRadio hasn't published broadly.

Changes:

- FirmwareStager: new stageFromLocalFile(installerPath, modelFamily).
  Detects format from filename suffix (.msi / .exe / .ssdr), parses
  version from filename, and dispatches:
    - .ssdr  → validate header, copy into staging dir, stageComplete.
    - .msi   → verifyAndExtract() with MSI extractor (already added in
               this branch).
    - .exe   → verifyAndExtract() with InnoSetup byte-scanner.

- RadioSetupDialog: "Browse .ssdr..." button renamed to
  "Select Installer...". File dialog accepts *.msi *.exe *.ssdr (with
  per-type filters). Successful selection kicks off the stager — the
  existing stageProgress/stageComplete/stageFailed slots already wire
  the UI, so progress and success messaging just work.

- "Check for Update" no longer re-wires itself to a download button on
  v-mismatch. It now reports "Update available: vX.Y.Z — download
  from flexradio.com, then click 'Select Installer...' to stage it."

The downloadAndStage() public method stays in the API (no current UI
callers, but the logic still works for any future opt-in flow).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@ten9876 ten9876 changed the title FirmwareStager: support v4.2+ WiX MSI installers FirmwareStager: support v4.2+ MSI + user-selected installer Apr 29, 2026
ten9876 and others added 2 commits April 29, 2026 12:05
Firmware flashing is irreversible. If something goes wrong we want
maximum forensic information from the single attempt.

Adds at QtInfoMsg level (visible without needing debug-level filter):

- Pre-upload state dump: file path, filename, size, MD5, magic bytes,
  radio model, serial, address, current firmware version. All in one
  delimited block so it's easy to grep.

- TX command echoes: "file filename" and "file upload <size> update".

- RX response capture: the full body returned by the radio for the
  file-upload command, including non-zero error codes (we previously
  only logged the code as hex; the body may contain diagnostic info
  the radio attaches).

- Socket state transitions: every QTcpSocket state change goes to the
  log so we can reconstruct exactly when/where any disconnect or
  error occurred.

- Socket error details: error enum value, error string, bytes
  transferred at error time, port in use.

No behavioral changes — all additions are log statements.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…egex

The flash worked. The diagnostic spam I added in 9507002 was scaffolding
for the one-shot-and-pray run; with end-to-end success confirmed in the
log we don't need to keep all of that in production. Reverts the
QtInfoMsg pre-upload dump, the per-state socket logging, and the
inline TX/RX command echoes — the existing protocol-channel logger
already covers the actually-useful parts.

Also widens the PII redactor's IPv4 regex to handle a case the post-flash
log surfaced: protocol-side `software_ver=4.2.18.41174` was being
redacted to `*.*.*. 41174` because the regex matched the first three
digits of `41174` as an "octet" with the trailing `74` left dangling.
Two extensions:
  (?<!ver=)  — negative lookbehind for `ver=` catches `software_ver=`,
              `firmware_ver=`, `version=`, etc. without enumerating each.
  (?![\d"])  — negative lookahead rejects matches where the 4th "octet"
              is a prefix of a longer number (e.g. 41174).
Quoted (`"0.9.2.1"`) and v-prefixed (`v0.9.2.1`) cases stay handled.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@ten9876 ten9876 disabled auto-merge April 29, 2026 20:57
Replaces the QProcess-to-7z shell-out in extractFromMsi() with an
in-tree pipeline. No external runtime dependencies, no temp files,
works in any sandboxing model that disallows process spawning.

Pipeline:

    OleCompoundFile  →  cab*.cab streams (in memory, MSI-decoded names)
    CabExtractor     →  files inside each cab (libmspack, in memory)
    Filter Salted__  →  firmware blobs only
    Sort by size     →  pick by family (consumer vs 9600)
    Write to staging

Components:

  third_party/libmspack/  — vendored subset of libmspack 0.11alpha
    (LGPL-2.1, GPL-3 compatible). 11 source files (~6k lines C);
    only CAB+LZX+MSZIP+Quantum decompression, no encoders, no CHM/
    HLP/KWAJ/LIT/OAB/SZDD. Built as a static lib with -w to keep
    upstream warning style out of our build output. README in the
    directory documents the version + license + update procedure.

  OleCompoundFile  — read-only [MS-CFB] reader. Walks the FAT chain,
    parses the directory, locates streams by name. MSI-aware
    accessor decodes Microsoft's 5-bit-packed name encoding (each
    Unicode code point in 0x3800-0x4840 represents two ASCII chars
    from a 64-entry alphabet). ~290 lines.

  CabExtractor   — in-memory mspack_system adapter for libmspack.
    Reads cab bytes from a QByteArray and writes decompressed
    payloads to QByteArrays — no temp files. extractAll() returns
    every file in the cab; extractFirstMatchingMagic() finds the
    one that starts with a given magic (used to grab the .ssdr
    blob and ignore sibling driver/installer files in the same
    cab). ~190 lines.

  FirmwareStager::extractFromMsi() — new body, ~80 lines (was 130).
    No QProcess, no findExtractionTool(), no temp directory.
    findExtractionTool() and the QProcess include are removed.

Verified end-to-end against SmartSDR_v4.2.18_x64.msi:

  • Extracted FLEX-6x00 firmware MD5 = 9e8888dc0558ee420ed82f370f805025
    matches the bytes that successfully flashed a real radio earlier
    in this branch.
  • All 6 cab streams extracted byte-identical to standalone 7z.
  • cab1 has 14 files (drivers + FLEX-9600 firmware); the magic-match
    filter correctly skips the drivers and picks the firmware blob.

New test: tests/ole_compound_file_test.cpp covers OLE CFB stream
extraction, MSI name decoding, libmspack CAB decompression, and the
end-to-end firmware-blob match against the known-good MD5.  Skips
with code 77 if the licensed MSI fixture isn't on disk (looks at
$AETHERSDR_TEST_MSI then ~/build/reference/SmartSDR_v4.2.18_x64.msi).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@ten9876 ten9876 requested a review from AetherClaude as a code owner April 29, 2026 21:57
@ten9876 ten9876 changed the title FirmwareStager: support v4.2+ MSI + user-selected installer FirmwareStager: support v4.2+ MSI (native libmspack) + user-selected installer Apr 29, 2026
system.h gates its inline memcmp/memset/strlen reimplementations behind
#ifndef MSPACK_NO_DEFAULT_SYSTEM, so even MSPACK_NO_DEFAULT_SYSTEM=0
makes the compiler take the inline branch. On MSVC those collide with
the compiler's built-in intrinsics:

  error C2169: 'memcmp': intrinsic function, cannot be defined
  error C2169: 'memset': intrinsic function, cannot be defined
  error C2169: 'strlen': intrinsic function, cannot be defined

Removing the define entirely lets system.h take the default
#include <string.h> branch, which is what we wanted in the first place.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@ten9876 ten9876 merged commit 812e70d into main Apr 30, 2026
5 checks passed
@ten9876 ten9876 deleted the aether/firmware-msi-support branch April 30, 2026 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant