Skip to content

formats/fs_prodos.cpp: Treat zero master index entries in tree files as sparse - #16103

Merged
ajrhacker merged 1 commit into
mamedev:masterfrom
Dev-next-gen:fix/prodos-sparse-tree-index
Sep 14, 2026
Merged

ajrhacker merged 1 commit into
mamedev:masterfrom
Dev-next-gen:fix/prodos-sparse-tree-index

Conversation

@Dev-next-gen

Copy link
Copy Markdown
Contributor

While reading 47891bc, which pads zero pointers inside an index block with NULs instead of reading the boot block, I noticed the tree file branch of any_read still takes each master index entry at face value. A zero there is the same situation one level up. ProDOS doesn't allocate an index block when none of its 256 data blocks are allocated, and the whole 128K range reads back as zeros (ProDOS 8 Technical Reference Manual, B.3.6). MAME fetches block 0 instead and uses the boot code as a list of data block pointers. any_blocks has the same loop, so it also records block 0 as allocated and walks the boot block.

To check this I built floptool and generated a raw ProDOS volume with a tree file laid out the way ProDOS writes a sparse one: master index entry 0 points to an index block whose first data block is filled with 'F', entry 1 is zero, entry 2 points to an index block whose first data block is filled with 'A', EOF 256.5K. Next to it is an ordinary tree file with every block allocated, as a control.

Without this change:

  • on an 800K volume, floptool hdread prodos vol.po SPARSE out fails with Block number overflow: requiring block 37121 on device of size 1600. 37121 is 0x9101, i.e. boot[0x000] | boot[0x100] << 8.
  • on a 32M volume the read succeeds, but the hole is wrong from offset 144384 onwards: those bytes are block 132, which is a data block of the other file.
  • floptool hdblocks reports the file as allocation blocks 10-11,0,12 and data blocks 13,37121,24632,51376,..., a few hundred entries long.

With it, the file reads back as 512 'F' bytes, zeros up to 256K, then 512 'A' bytes on both volume sizes. hdblocks reports 10-12 and 13-14. The control file reads and enumerates the same before and after. Seedling and sapling files take other branches and aren't affected, and the ProDOS code in floptool is read-only, so nothing on the write side depends on the old behaviour.

Script used to build the volume and check floptool's output

Run as python3 sparse_tree.py <mame checkout with floptool built> [block count].

#!/usr/bin/env python3
# Build a raw ProDOS volume holding tree files, then read them back with floptool.
#   SPARSE: EOF = 128K + 512. Master index entry 0 is 0 (the whole first 128K is a hole),
#           entry 1 -> index block -> one data block of 0x41.
#   DENSE : same EOF, every block allocated (control: must pass before and after).
import os, re, subprocess, sys

MAME = sys.argv[1]
nblocks = int(sys.argv[2]) if len(sys.argv) > 2 else 1600
work = os.path.dirname(os.path.abspath(__file__))
img_path = os.path.join(work, f"vol{nblocks}.po")

src = open(os.path.join(MAME, "src/lib/formats/fs_prodos.cpp")).read()
boot = bytes(int(x, 16) for x in re.findall(r"0x([0-9a-f]{2})", src.split("prodos_impl::boot[512] = {")[1].split("};")[0]))
assert len(boot) == 512

img = bytearray(512 * nblocks)
def blk(n): return memoryview(img)[512*n:512*(n+1)]
def w16(b, o, v): b[o] = v & 0xff; b[o+1] = v >> 8
def idx_set(b, i, v): b[i] = v & 0xff; b[i+256] = v >> 8

blk(0)[:] = boot
for n in range(2, 6):
    w16(blk(n), 0, n-1 if n > 2 else 0); w16(blk(n), 2, n+1 if n < 5 else 0)
k = blk(2)
name = b"TEST"
k[4] = 0xf0 | len(name); k[5:5+len(name)] = name
k[0x20] = 5; k[0x22] = 0xc3; k[0x23] = 0x27; k[0x24] = 0x0d
w16(k, 0x25, 2); w16(k, 0x27, 6); w16(k, 0x29, nblocks)

def entry(slot, fname, key, used, eof):
    e = memoryview(k)[4 + 39*slot: 4 + 39*(slot+1)]
    e[0] = 0x30 | len(fname); e[1:1+len(fname)] = fname
    e[0x10] = 0x06; w16(e, 0x11, key); w16(e, 0x13, used)
    e[0x15] = eof & 0xff; e[0x16] = (eof >> 8) & 0xff; e[0x17] = eof >> 16
    e[0x1c] = 5; e[0x1e] = 0xc3; w16(e, 0x25, 2)

# SPARSE, as ProDOS itself lays it out (TechRef B.3.6: the first data block is always
# allocated, an index block whose 256 data blocks are all unallocated is not allocated):
#   master 10: [0] -> index 11, [1] = 0 (128K hole), [2] -> index 12
#   index 11: [0] -> data 13 ("F"), index 12: [0] -> data 14 ("A")
idx_set(blk(10), 0, 11); idx_set(blk(10), 1, 0); idx_set(blk(10), 2, 12)
idx_set(blk(11), 0, 13); blk(13)[:] = b"F" * 512
idx_set(blk(12), 0, 14); blk(14)[:] = b"A" * 512
entry(1, b"SPARSE", 10, 5, 2*256*512 + 512)
# DENSE: master 15, index 16 -> data 20..275, index 17 -> data 276
idx_set(blk(15), 0, 16); idx_set(blk(15), 1, 17)
for i in range(256):
    idx_set(blk(16), i, 20 + i); blk(20 + i)[:] = bytes([i]) * 512
idx_set(blk(17), 0, 276); blk(276)[:] = b"B" * 512
entry(2, b"DENSE", 15, 259, 256*512 + 512)
open(img_path, "wb").write(img)

expect = {
    "SPARSE": b"F"*512 + bytes(2*256*512 - 512) + b"A"*512,
    "DENSE": b"".join(bytes([i])*512 for i in range(256)) + b"B"*512,
}
floptool = os.path.join(MAME, "floptool")
ok = True
for fname, want in expect.items():
    out = os.path.join(work, fname + ".out")
    if os.path.exists(out):
        os.remove(out)
    r = subprocess.run([floptool, "hdread", "prodos", img_path, fname, out], capture_output=True, text=True)
    got = open(out, "rb").read() if os.path.exists(out) else None
    good = r.returncode == 0 and got == want
    if r.stderr.strip():
        detail = r.stderr.strip()
    elif got is None:
        detail = "no output"
    else:
        diff = next((i for i, (a, b) in enumerate(zip(got, want)) if a != b), None)
        detail = f"{len(got)} bytes, first wrong byte at offset {diff}"
    print(f"hdread   {fname:6}: {'OK  ' if good else 'FAIL'} {detail}")
    ok &= good

# hdblocks: the sparse file owns master 10, index 11-12 and data 13-14, nothing else
r = subprocess.run([floptool, "hdblocks", "prodos", img_path], capture_output=True, text=True)
lines = [l.split() for l in (r.stdout + r.stderr).splitlines() if "SPARSE" in l]
good = r.returncode == 0 and lines == [["file", "SPARSE", "10-12", "13-14"]]
print(f"hdblocks SPARSE: {'OK  ' if good else 'FAIL'} rc={r.returncode} {' '.join(lines[0])[:120] if lines else r.stderr.strip()}")
ok &= good
dense = [l.split() for l in r.stdout.splitlines() if "DENSE" in l]
good = dense == [["file", "DENSE", "15-17", "20-276"]]
print(f"hdblocks DENSE : {'OK  ' if good else 'FAIL'} {' '.join(dense[0]) if dense else ''}")
ok &= good
sys.exit(0 if ok else 1)

AI tools used

…as sparse

A tree file whose master index block has a zero entry has no index block
for that 128K range: ProDOS allocates neither the index block nor its data
blocks, and reads the range back as zeros (ProDOS 8 Technical Reference
Manual, B.3.6). any_read and any_blocks fetched block 0, the boot block,
and used its bytes as data block pointers, so reading such a file either
failed with a block number overflow or returned blocks belonging to other
files, and flopblocks/hdblocks listed block 0 and a few hundred unrelated
blocks as belonging to the file.

This is the same case 47891bc handled for zero entries inside an
index block, one level up.
@ajrhacker
ajrhacker merged commit 492adeb into mamedev:master Sep 14, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants