EC: lift the 65535 children-per-tag wire-format ceiling (#199) - #570

Summary

The EC wire format encoded the number of children inside any tag as a uint16, silently truncating when a tag carried more than 65535 children. The most visible consequence was EC_OP_SHARED_FILES: amuled adds one child tag per shared file, so any library above 65535 files surfaced both as a counter capped at 65535 in amulegui and as silent EC corruption that broke the downloads-list refresh and other follow-on traffic. Reported by Stoatwblr in #199.

Fix

Two layered pieces, with capability negotiation for safe rollout:

1. Sentinel-extended wire format. When the negotiated EC flag EC_FLAG_LARGE_TAG_COUNT is in effect, a count >= 0xFFFF is encoded as 0xFFFF followed by a uint32 carrying the real count. Smaller counts still emit the historical uint16 — wire-byte-identical for every existing nested tag in the codebase. CECTag::WriteChildren / ReadChildren branch on socket.m_tx_flags / m_rx_flags & EC_FLAG_LARGE_TAG_COUNT, and CECTag::GetTagLen takes a new useLargeCount parameter so writer and reader compute identical wire-size estimates and m_dataLen subtraction stays consistent.

2. Capability negotiation through the existing auth handshake. New client sends EC_TAG_CAN_LARGE_TAG_COUNT in EC_OP_AUTH_REQ; new server records the capability and echoes the tag in EC_OP_AUTH_OK. ECSocket::WritePacket adds EC_FLAG_LARGE_TAG_COUNT to outgoing flags unconditionally — auto-stripped by flags &= m_my_flags when not negotiated. This is what keeps mixed-version EC safe: without negotiation, both sides stay on the historical wire format, and WriteChildren caps outgoing counts at 0xFFFE both to fit in uint16 and to avoid emitting 0xFFFF (which a fix peer without the negotiated flag would still treat as plain count, matching old behaviour).

Why sentinel-extended instead of always-uint32

Once EC_FLAG_LARGE_TAG_COUNT is negotiated both sides know the format, so the count field could in principle just always be a uint32. The sentinel choice optimises for the common case:

• Wire compactness. Most tags in EC have a handful of children (5–20 typically). Sentinel emits uint16(count) for counts under 0xFFFF — 2 bytes — and only the rare large-count tag pays the extra 4 bytes (0xFFFF marker + uint32 real count, 6 bytes total). Always-uint32 would add 2 bytes to every tag-with-children. For a typical EC stats response with ~50 tags-that-have-children, that's +100 bytes per packet at no real benefit.
• Byte-identical wire for typical packets. With sentinel, a packet whose tags all have <0xFFFF children produces wire bytes byte-for-byte identical to what an old peer would emit. Packet captures, debugging tools, third-party EC clients reading bytes manually keep working unchanged for the typical case; only when a tag actually needs >65535 children does the wire diverge.
• Single branch in the hot path. The reader's first read is unchanged from old code (uint16); the only added cost is one comparison against 0xFFFF.

Always-uint32 would simplify the code at the cost of fatter wire on every packet. Sentinel was the chosen trade-off.

Compatibility matrix

End-to-end verified on Ubuntu 26.04 with a synthetic 70001-file shared library:

Mixed-version EC is functional in both directions with no protocol breaks; the worst-case behaviour on huge libraries is the same effective ceiling old code always had.

Caveats

External EC client implementations (e.g. amuleweb, third-party Node / web bindings) keep working unchanged — they don't advertise EC_TAG_CAN_LARGE_TAG_COUNT in auth, so the daemon falls back to the historical wire format for them. To opt in, an external client adds the empty EC_TAG_CAN_LARGE_TAG_COUNT (0x0011) tag to its EC_OP_AUTH_REQ, watches for the same tag in EC_OP_AUTH_OK, and on a hit handles the sentinel branch in its TAGCOUNT reader (~10–15 lines).