Skip to content

kad: replace sort+resize filename cap with O(N) compare-and-skip insertion - #795

Merged
mrjimenez merged 1 commit into
amule-project:masterfrom
got3nks:fix/kad-filename-cap-skip-sort
May 31, 2026
Merged

kad: replace sort+resize filename cap with O(N) compare-and-skip insertion#795
mrjimenez merged 1 commit into
amule-project:masterfrom
got3nks:fix/kad-filename-cap-skip-sort

Conversation

@got3nks

@got3nks got3nks commented May 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Refactor of the CKeyEntry::m_filenames cap landed in #314 (c95e002), per @irwir's review comment on that commit.

The original implementation push_back'd each non-duplicate name and then ran std::list::sort + resize(MAX_FILENAMES) whenever the size exceeded the cap. Two cleanups, both noted in the review:

  1. Push-then-evict: when a freshly-added entry's popularity happened to be the lowest, the sort + resize would immediately discard it. Wasted work and an awkward shape.
  2. list::sort is O(N log N), but the actual job is "find one (or a few) weakest entries and drop them." std::min_element is O(N) and proportionate.

This PR replaces both with a pushBounded() lambda used at the two existing insertion sites:

auto pushBounded = [&](const sFileNameEntry & candidate) {
    if (m_filenames.size() < MAX_FILENAMES) {
        m_filenames.push_back(candidate);
        return;
    }
    auto weakest = std::min_element(m_filenames.begin(), m_filenames.end(),
        [](const sFileNameEntry & a, const sFileNameEntry & b) {
            return a.m_popularityIndex < b.m_popularityIndex;
        });
    if (candidate.m_popularityIndex > weakest->m_popularityIndex) {
        *weakest = candidate;
    }
    // else: candidate's popularity is no better than the weakest
    // already-kept entry; drop the candidate.
};

Behavioural notes

  • Eviction policy unchanged — "highest popularity wins each slot." The set of survivors after a merge is identical to what the previous sort+resize produced.
  • No more push-then-immediately-evict.
  • No global sort. Cost is O(N) per inserted candidate at the cap (vs O(N log N) per cap-fire previously).
  • Cap still kicks in at the same MAX_FILENAMES = 100 boundary.

Test plan

  • Builds clean on macOS (amule target).
  • CI exercises Kademlia code paths via unittests/CUInt128Test (no direct test for CKeyEntry::MergeIPsAndFilenames; the refactor is behaviour-preserving by construction).

Refs #314.

Per @irwir's review of c95e002 (amule-project#314), the original cap landed each
non-duplicate name with `push_back` and then ran `list::sort` +
`resize` whenever the size went over MAX_FILENAMES.  Two issues:

  1. A new entry whose popularity index happened to be the lowest
     would be pushed and then immediately evicted on the next sort —
     wasted work and a slightly awkward "add then conditionally take
     away" pattern.
  2. `list::sort` is O(N log N) and reorders the entire list, but the
     job at hand is "find one (or a few) weakest entries and drop
     them."  `std::min_element` is O(N) and proportionate to the work.

Replace both with a `pushBounded()` helper: under the cap → push;
at the cap → only replace the weakest existing entry, and only if
the candidate's popularity beats it; otherwise drop the candidate.

Functional outcome unchanged — same "highest popularity wins each
slot" eviction policy — but no global sort, no push-then-evict, and
the policy is explicit at the insertion site.
got3nks referenced this pull request May 31, 2026
CKeyEntry::MergeIPsAndFilenames merges the incoming entry's filename
list into the stored entry's, one sFileNameEntry per distinct
publisher-chosen name. The publishing-IP list right next to it caps
at 100 (Entry.cpp:429-432); the filename list never got the matching
cap. A popular file accumulates one entry per distinct filename
variant -- language renames, mirror prefixes, transliterations,
case changes, trailing-paren copies -- and the list grows
monotonically for the lifetime of the entry.

ngosang's 20-min heaptrack of amuled (after 6 h warm-up, 7 TB
shareset) showed ~2.36 MB leaked, with the leak headers
overwhelmingly traced to:

  Kademlia::CEntry::sFileNameEntry(sFileNameEntry const&)
    std::list<sFileNameEntry>::push_front
    CEntry::SetFileName (Entry.cpp:118)
    CKademliaUDPListener::Process2PublishKeyRequest

Extrapolated rate ~170 MB/day, matching the multi-day RSS creep the
issue originally reported.

Cap m_filenames at 100 entries (matching the m_publishingIPs cap
below), popularity-ordered: when the list grows past the cap, sort
by m_popularityIndex descending and resize. GetCommonFileName's
"most popular wins" semantics are preserved since the survivors are
exactly the entries with the highest popularity counts.

The CIndexed keyword-to-file map is untouched. Search queries land
there via keyword hash, not via per-file filename lists, and the
KADEMLIAMAXINDEX = 60000 cap that protects it is unrelated.
m_filenames inside each CKeyEntry only ever feeds GetCommonFileName
for the displayed name in search-result rendering -- capping it
doesn't change which files appear in results, only which name shows
next to a file when multiple publishers used multiple names.

eMule's reference code caps the equivalent list on the same shape;
amule never ported that half when the 100-IP cap was added.
@irwir

irwir commented May 31, 2026

Copy link
Copy Markdown

Candidate comes with the lowest popularity possible, might be never added with this code.

@mrjimenez
mrjimenez merged commit 4de8acb into amule-project:master May 31, 2026
7 checks passed
@got3nks

got3nks commented May 31, 2026

Copy link
Copy Markdown
Contributor Author

Fair catch, and you're right — once m_filenames saturates and every survivor has popularity ≥ 2, a fresh popularity-1 candidate can't break in. The original push-then-sort-and-resize from #314 had the same semantic (push, then drop in the resize), but pushBounded makes it explicit.

Context for the cap itself: it landed in #314 because without it m_filenames grows monotonically for the lifetime of the CKeyEntry, which on a busy publishing node added up to hundreds of MB / GB of RAM across thousands of indexed hashes. So the cap is a RAM lever — we can't drop it.

The rotation problem is independent though. Plan is to add a popularity decay on the merge path: decrement every entry's m_popularityIndex by 1 (floored at 0) every K merges to this CKeyEntry, and let any candidate replace a zero-popularity slot. Stagnant incumbents age out as soon as they stop being republished; genuinely-popular names hold position because they get bumped faster than they decay. No change to the cap, no UI impact (the index is never exposed — only GetCommonFileName()'s winner ever escapes the class).

Will send as a follow-up PR.

got3nks added a commit to got3nks/amule that referenced this pull request Jun 1, 2026
…mule-project#795)

irwir's review on amule-project#795 caught that once CKeyEntry::m_filenames hits
the 100-entry cap and every survivor has popularity >= 2, a fresh
popularity-1 candidate can never break in. The cap stays -- it's the
RAM lever -- but rotation needs an independent mechanism.

Two pieces:

1. Popularity decay. Per-CKeyEntry m_mergeCounter; on every Nth
   real merge, every entry's popularity is decremented by 1
   (floored at 0). pushBounded gains a zero-popularity escape
   hatch: if the weakest slot has decayed to 0, any candidate
   (including popularity-1) can replace it. Stagnant incumbents
   age out; popular names hold position because they get bumped
   by matching publishes faster than they decay.

2. Merge-rate instrumentation. CKeyEntry::MaybeDumpMergeStats is
   called from CKademlia::Process and emits a 5-minute summary on
   logKadEntryTracking with merges in the interval, current
   keyword-key count, derived merges/min and merges/min/key, and
   the current decay-tick value. Lets us tune the constant from
   observed traffic on a real node rather than guessing.

Starting MERGES_PER_DECAY_TICK = 20. On-disk format unchanged
(m_mergeCounter not serialised; decay phase resets per key on
restart, which is fine -- the first decay tick on any active key
fires within minutes).
got3nks added a commit to got3nks/amule that referenced this pull request Jun 1, 2026
…mule-project#795)

irwir's review on amule-project#795 caught that once CKeyEntry::m_filenames hits
the 100-entry cap and every survivor has popularity >= 2, a fresh
popularity-1 candidate can never break in. The cap stays -- it's the
RAM lever -- but rotation needs an independent mechanism.

Two pieces:

1. Popularity decay. Per-CKeyEntry m_mergeCounter; on every Nth
   real merge, every entry's popularity is decremented by 1
   (floored at 0). pushBounded gains a zero-popularity escape
   hatch: if the weakest slot has decayed to 0, any candidate
   (including popularity-1) can replace it. Stagnant incumbents
   age out; popular names hold position because they get bumped
   by matching publishes faster than they decay.

2. Merge-rate instrumentation. CKeyEntry::MaybeDumpMergeStats is
   called from CKademlia::Process and emits a 5-minute summary on
   logKadEntryTracking with merges in the interval, current
   keyword-key count, derived merges/min and merges/min/key, and
   the current decay-tick value. Lets us tune the constant from
   observed traffic on a real node rather than guessing.

Starting MERGES_PER_DECAY_TICK = 20. On-disk format unchanged
(m_mergeCounter not serialised; decay phase resets per key on
restart, which is fine -- the first decay tick on any active key
fires within minutes).
got3nks added a commit to got3nks/amule that referenced this pull request Jun 1, 2026
irwir's review on amule-project#795 caught that once CKeyEntry::m_filenames hits
the 100-entry cap and every survivor has popularity >= 2, a fresh
popularity-1 candidate can never break in. The cap stays -- it's
the RAM lever that bounds the keyword index on busy publishing
nodes (amule-project#314) -- but rotation needs an independent mechanism.

Three pieces:

1. Per-CKeyEntry m_mergeCounter. Incremented on every real merge
   (fromEntry != NULL; the IP-init no-op path does not count).

2. Decay tick. Every MERGES_PER_DECAY_TICK real merges, decrement
   every entry's m_popularityIndex by 1 (floored at 0). Gated on
   m_filenames.size() >= MAX_FILENAMES: under the cap there's
   nothing to rotate, and decay would only drag popularity numbers
   down before the list is even full. Once saturated the list
   never shrinks below the cap (pushBounded only ever replaces),
   so the gate is "off until first saturation, then always on".

3. Zero-popularity escape hatch in pushBounded. If the weakest
   slot has decayed to popularity 0, any candidate (including a
   fresh popularity-1 entry) replaces it. Stagnant incumbents age
   out as their popularity decays away; genuinely-popular names
   hold position because matching publishes bump them faster than
   decay drops them.

Plus a robustness fix in CEntry::GetCommonFileName.  Before this
commit, the loop initialised the running max with 0 and used a
strict `>` comparison, so an all-zero m_filenames (which decay can
produce on diffusely-published keys with no clear winner) would
leave the result iterator at end() and return an empty string.
That would silently break callers downstream: GetTagCount drops
TAG_FILENAME from the count, WriteTagListInc omits it on the wire,
SearchTermsMatch returns false for every term, and
CIndexed::AddKeyword rejects the entry via the
GetCommonFileName().IsEmpty() guard.  Seeding the running max
from the first entry (and switching to a "pick the first if all
tied" fallback) preserves the protocol invariant
"non-empty m_filenames yields a non-empty common name" without
changing behaviour for any case where a real winner exists.

Starting MERGES_PER_DECAY_TICK = 20 is a reasonable guess; tunable
later if rotation half-lives in production turn out to need it.
On-disk format unchanged -- m_mergeCounter is not serialised; the
decay phase resets per key on restart, which just means a one-time
grace period before decay resumes (first decay tick on any active
key still fires within minutes of restart).

Refs amule-project#765, amule-project#795.
@got3nks

got3nks commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

Decay follow-up is up at #799 — gates on saturation as discussed, plus a GetCommonFileName robustness fix for the all-zero degenerate case.

mrjimenez pushed a commit that referenced this pull request Jun 1, 2026
irwir's review on #795 caught that once CKeyEntry::m_filenames hits
the 100-entry cap and every survivor has popularity >= 2, a fresh
popularity-1 candidate can never break in. The cap stays -- it's
the RAM lever that bounds the keyword index on busy publishing
nodes (#314) -- but rotation needs an independent mechanism.

Three pieces:

1. Per-CKeyEntry m_mergeCounter. Incremented on every real merge
   (fromEntry != NULL; the IP-init no-op path does not count).

2. Decay tick. Every MERGES_PER_DECAY_TICK real merges, decrement
   every entry's m_popularityIndex by 1 (floored at 0). Gated on
   m_filenames.size() >= MAX_FILENAMES: under the cap there's
   nothing to rotate, and decay would only drag popularity numbers
   down before the list is even full. Once saturated the list
   never shrinks below the cap (pushBounded only ever replaces),
   so the gate is "off until first saturation, then always on".

3. Zero-popularity escape hatch in pushBounded. If the weakest
   slot has decayed to popularity 0, any candidate (including a
   fresh popularity-1 entry) replaces it. Stagnant incumbents age
   out as their popularity decays away; genuinely-popular names
   hold position because matching publishes bump them faster than
   decay drops them.

Plus a robustness fix in CEntry::GetCommonFileName.  Before this
commit, the loop initialised the running max with 0 and used a
strict `>` comparison, so an all-zero m_filenames (which decay can
produce on diffusely-published keys with no clear winner) would
leave the result iterator at end() and return an empty string.
That would silently break callers downstream: GetTagCount drops
TAG_FILENAME from the count, WriteTagListInc omits it on the wire,
SearchTermsMatch returns false for every term, and
CIndexed::AddKeyword rejects the entry via the
GetCommonFileName().IsEmpty() guard.  Seeding the running max
from the first entry (and switching to a "pick the first if all
tied" fallback) preserves the protocol invariant
"non-empty m_filenames yields a non-empty common name" without
changing behaviour for any case where a real winner exists.

Starting MERGES_PER_DECAY_TICK = 20 is a reasonable guess; tunable
later if rotation half-lives in production turn out to need it.
On-disk format unchanged -- m_mergeCounter is not serialised; the
decay phase resets per key on restart, which just means a one-time
grace period before decay resumes (first decay tick on any active
key still fires within minutes of restart).

Refs #765, #795.
@irwir

irwir commented Jun 1, 2026

Copy link
Copy Markdown

added up to hundreds of MB / GB of RAM across thousands of indexed hashes.

Very dramatic.
For eMule, typical size is well below 200 MB in Task Manager after a week-long session; never seen anything near or above half a gigabyte.
Key entries have limited lifetime, publishing rate is moderate, RAM usage should not grow too much.

In my opinion, setting a limit would be reasonable to prevent attacks and misuse.
Though, popularity decay seems like a voluntary and overcomplicated way to deal with multiple file names.
Maybe someone should try to log maximum list size for a single entry?
The limit could have been chosen more appropriately with this data.

@got3nks

got3nks commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

Fair pushback and you've got the empirical numbers on your side — let me walk back the framing.

The "hundreds of MB / GB" line in #314's commit message was bleed-over from a longer leak hunt we'd been doing on a busy production node, where overall amuled RSS was climbing past expected envelopes and the CKeyEntry filename list was one of several suspects in the queue. The actual dominant offender turned out to be the EC notification path leaking CECPacket trees (the fix landed in #797 with confirmed 41.8% of inuse retention disappearing), not CKeyEntry growth. So the original cap motivation oversold a problem the index itself probably wasn't the main cause of.

That said, capping at 100 still feels like reasonable defensive design — cheap insurance against pathological / adversarial publishes, and it costs essentially nothing in normal operation. And the decay piece in #799 is gated on saturation: under the cap it never runs, so on a healthy node it's effectively dead code. The case where it does matter is exactly the case where someone manages to fill the list — adversarial flood, weird mass-republish, etc. — and at that point rotation needs an unstuck path, which decay provides.

We've actually been running the instrumented version on a live HighID production node to see if/when saturation ever happens in practice. Across ~4 h of observation so far (~4000 merges across ~1800 indexed keys, ~15 merges/min sustained), the saturated-key count has stayed at 0 the entire time — matches your eMule observation. If it ever does flip non-zero we'll share the data; if not, the decay just stays dormant, which is a fine outcome.

@irwir

irwir commented Jun 1, 2026

Copy link
Copy Markdown

Good to know that absense of limit was not guilty.
In real life, too many names for the same file might signal a fake (there is guide "How to avoid fakes" in the Docs of eMule's site).
Hence, "decay" code is practically useless, and has obvious negative side effects.
It would still be interesting to get the highest number of file names in one entry - a static variable, log the value every time it increased; then search up from the bottom of log. Preferably, from multiple mules.
My guess - 100 is way too many.

@got3nks

got3nks commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

Could you spell out which side effects you're seeing? Want to make sure I'm addressing the actual concern rather than guessing.

Concretely the decay only does this:

  • Fires only when m_filenames.size() == MAX_FILENAMES (which our production instrumentation hasn't seen happen yet in normal traffic).
  • At each tick, decrements every entry's popularity by 1, floored at 0.
  • The only entries that ever get replaced by a new candidate are the ones that have already decayed all the way to 0 — i.e., names that no publisher has reinforced for many publish cycles in a row. Active popular names hold position because matching publishes bump them faster than decay drops them.

So the "removed" entries are by construction stale and abandoned in the popularity-vote sense, not active popular ones. If there's a specific failure mode you've got in mind beyond that I'd like to fix it; I just can't see it from the algorithm alone.

On the fakes angle — fair point in general, but it's out of scope for what we were optimising here. The original cap motivation (and the decay follow-up) was purely RAM containment; anti-fake heuristics would be a separate piece of work and we haven't been factoring them in.

@irwir

irwir commented Jun 1, 2026

Copy link
Copy Markdown

Initial optimization appeared to be premature (and the limit did not fix RAM growth).
Decay rate is a guess, and there is no evidently best value.
Absense of recent publishing, because a node went offline, does not invalidate the entry.
Then, after a while all values go down to zero, which means the newcomer may override the one with previously high popularity index.
This directly contradicts to the original intention to keep the highest values.
Decay idea was instroduced for the sake of inseting a new name. An artificial construction, and too much code that hopefully would never work.

@got3nks

got3nks commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

Your point about "publisher absence ≠ entry invalidation" is the right framing, and it makes me think decay was using the wrong clock. Wall-clock time (when did a publisher last touch this entry?) is what actually maps to "is this name still alive in the network." Decay was measuring publish tempo on the key instead, which is a different thing.

The cleaner answer to the same underlying need (let saturated lists rotate when genuinely stale) is a per-entry TTL rather than a popularity decrement:

struct sFileNameEntry {
    wxString m_filename;
    uint32_t m_popularityIndex;
    time_t   m_lastSeen;     // set on create / refreshed on every match
};
  • On creation in pushBounded and on every match-bump in MergeIPsAndFilenames, both m_popularityIndex++ and m_lastSeen = time(NULL). Each matching publish is independent evidence the entry is still alive, so it refreshes both fields.
  • At the top of MergeIPsAndFilenames, sweep entries older than 2 * KADEMLIAREPUBLISHTIMEK (~48 h) — one missed republish cycle of grace, then the entry's publisher set has genuinely abandoned it.
  • pushBounded reverts to the original strict > comparison.

Properties:

  • "Highest popularity wins" semantic preserved exactly — popularity never decreases.
  • Multiple publishers voting the same name → both higher popularity and fresher TTL, so popular names stay popular and immune to eviction simultaneously.
  • Single publisher going offline → popularity preserved at whatever it had reached (votes don't disappear), but m_lastSeen ages and the slot opens for reclamation after the protocol's grace window.
  • List freezes only if all 100 entries are being actively refreshed within the TTL — i.e. the saturated state is honest (the list really does carry that many live publishers).
  • Disk format unchanged: read entries get m_lastSeen = now, costing them one extra TTL of grace on restart, which seems harmless.

Memory cost is 4–8 bytes per filename × 100 ≈ 400–800 B per saturated key, which doesn't move any RAM needles.

If you think this is closer to right I'll send it as a follow-up that reverts the decay piece in favour of TTL.

@got3nks

got3nks commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

Apologies — we should have checked the existing cleanup paths properly before any of this. After a proper read of CKeyEntry::CleanUpTrackedPublishers and CIndexed::Clean, the picture is now clear and you were right from the start.

Per-IP cleanup already happens at KADEMLIAREPUBLISHTIMEK (24 h) on m_publishingIPs, and whole-entry expiry already happens at 24 h on m_tLifeTime. So "publisher goes offline" is already handled — their slot ages out at both granularities. The only thing not cleaned is a single name variant within an active CKeyEntry, which is exclusively a saturated-list concern, and the production data shows saturation just doesn't happen in realistic traffic.

Given that, both decay and the TTL I proposed are addressing a regime that doesn't exist on healthy nodes — neither is justified. Going to revert the decay piece from #799 outright; "freeze at the cap" is the correct behaviour for the saturation case if it ever does occur, and the existing cleanup machinery handles everything else.

Thanks for the patience walking us through this.

got3nks added a commit to got3nks/amule that referenced this pull request Jun 1, 2026
Reverts the decay tick + zero-popularity escape hatch landed in
df10aa8 ("kad: add popularity decay so saturated m_filenames can
still rotate") and addressed in PR amule-project#799.

Discussion on amule-project#795 with @irwir after amule-project#799 landed surfaced the
existing entry-lifetime cleanup that already handles the
"publisher offline => entry invalidation" case the decay was
targeting:

  - CKeyEntry::CleanUpTrackedPublishers ages out individual
    publisher IPs from m_publishingIPs after
    KADEMLIAREPUBLISHTIMEK (24 h) of silence.
  - CIndexed::Clean deletes the whole CKeyEntry when its
    m_tLifeTime expires (same 24 h horizon, extended on every
    publisher refresh).

So "publisher went offline" is handled at both the per-IP and
per-entry granularities.  The only thing left uncleaned is a
single name variant within an active CKeyEntry, which is
exclusively a saturated-m_filenames concern.  Empirical data from
a live HighID production node (~4000 merges over ~4 h across
~1800 indexed keys) shows the saturated-key count never lifts
off zero -- so the regime decay was designed for doesn't appear
in practice on a healthy node.  At the rare saturated cases the
"freeze at the cap" behaviour the decay was overriding is now
the correct behaviour: bounded RAM, no spurious rotation of
legitimate vote winners when traffic dips.

The 100-entry cap from amule-project#795 / amule-project#314 stays.  It protects against a
different case from the whole-entry TTL: an *active* CKeyEntry
that adversarial or pathological publishers feed unbounded
variant names under the same hash.  As @irwir noted in this
thread, "setting a limit would be reasonable to prevent attacks
and misuse."  The cap is a single integer comparison on insert
and lines up with the m_publishingIPs cap right next to it.

Concretely reverted:

  - CKeyEntry::MERGES_PER_DECAY_TICK static + m_mergeCounter
    member.
  - The decay tick block at the end of MergeIPsAndFilenames.
  - The "weakest->m_popularityIndex == 0" escape hatch in
    pushBounded -- back to the original strict `>` comparison
    from amule-project#795.

Deliberately kept: the GetCommonFileName robustness fix from
df10aa8.  Our own code no longer produces popularity-0 entries
after this revert, but two paths we don't control can still land
them in m_filenames: on-disk data from any node that ran the
decay build (it could have written 0-popularity entries to
known2.met before this revert lands), and malformed/adversarial
publishes that send popularity = 0 over the wire.  The picker's
seed-from-first / "pick first on tie" fallback closes a sharp
edge in a function whose output is on the wire (TAG_FILENAME,
SearchTermsMatch, CIndexed::AddKeyword reject path) for
essentially no cost.

Refs amule-project#795.
@irwir

irwir commented Jun 1, 2026

Copy link
Copy Markdown

For me, both discarding new entry or replacing the first with the lowest popularity index - might be considered as acceptable strategies. Both have simple implementation.

On the subject of the limit itself.
If both IPs and names were limited to 100, then what - each node sent a different name? Highly improbable.
Hence repeating: it would be interesting to get from the net highest count of file names seen in one entry.

@got3nks

got3nks commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

Fair point — will add max-list-size logging to the instrumented build we've got running on a production node and share the numbers once we have a usable window. For now the revert PR (#805) is up for review.

got3nks added a commit to got3nks/amule that referenced this pull request Jun 1, 2026
…mule-project#795)

irwir's review on amule-project#795 caught that once CKeyEntry::m_filenames hits
the 100-entry cap and every survivor has popularity >= 2, a fresh
popularity-1 candidate can never break in. The cap stays -- it's the
RAM lever -- but rotation needs an independent mechanism.

Two pieces:

1. Popularity decay. Per-CKeyEntry m_mergeCounter; on every Nth
   real merge, every entry's popularity is decremented by 1
   (floored at 0). pushBounded gains a zero-popularity escape
   hatch: if the weakest slot has decayed to 0, any candidate
   (including popularity-1) can replace it. Stagnant incumbents
   age out; popular names hold position because they get bumped
   by matching publishes faster than they decay.

2. Merge-rate instrumentation. CKeyEntry::MaybeDumpMergeStats is
   called from CKademlia::Process and emits a 5-minute summary on
   logKadEntryTracking with merges in the interval, current
   keyword-key count, derived merges/min and merges/min/key, and
   the current decay-tick value. Lets us tune the constant from
   observed traffic on a real node rather than guessing.

Starting MERGES_PER_DECAY_TICK = 20. On-disk format unchanged
(m_mergeCounter not serialised; decay phase resets per key on
restart, which is fine -- the first decay tick on any active key
fires within minutes).
@irwir

irwir commented Jun 1, 2026

Copy link
Copy Markdown

@got3nks, would be interesting to get the number.

By the way, do you see on GitHub my comment in the commit #c95e002?
Used to see such comments, but now none is shown, and your comment would be visible only after pressing Retry in error message.

mrjimenez pushed a commit that referenced this pull request Jun 1, 2026
Reverts the decay tick + zero-popularity escape hatch landed in
df10aa8 ("kad: add popularity decay so saturated m_filenames can
still rotate") and addressed in PR #799.

Discussion on #795 with @irwir after #799 landed surfaced the
existing entry-lifetime cleanup that already handles the
"publisher offline => entry invalidation" case the decay was
targeting:

  - CKeyEntry::CleanUpTrackedPublishers ages out individual
    publisher IPs from m_publishingIPs after
    KADEMLIAREPUBLISHTIMEK (24 h) of silence.
  - CIndexed::Clean deletes the whole CKeyEntry when its
    m_tLifeTime expires (same 24 h horizon, extended on every
    publisher refresh).

So "publisher went offline" is handled at both the per-IP and
per-entry granularities.  The only thing left uncleaned is a
single name variant within an active CKeyEntry, which is
exclusively a saturated-m_filenames concern.  Empirical data from
a live HighID production node (~4000 merges over ~4 h across
~1800 indexed keys) shows the saturated-key count never lifts
off zero -- so the regime decay was designed for doesn't appear
in practice on a healthy node.  At the rare saturated cases the
"freeze at the cap" behaviour the decay was overriding is now
the correct behaviour: bounded RAM, no spurious rotation of
legitimate vote winners when traffic dips.

The 100-entry cap from #795 / #314 stays.  It protects against a
different case from the whole-entry TTL: an *active* CKeyEntry
that adversarial or pathological publishers feed unbounded
variant names under the same hash.  As @irwir noted in this
thread, "setting a limit would be reasonable to prevent attacks
and misuse."  The cap is a single integer comparison on insert
and lines up with the m_publishingIPs cap right next to it.

Concretely reverted:

  - CKeyEntry::MERGES_PER_DECAY_TICK static + m_mergeCounter
    member.
  - The decay tick block at the end of MergeIPsAndFilenames.
  - The "weakest->m_popularityIndex == 0" escape hatch in
    pushBounded -- back to the original strict `>` comparison
    from #795.

Deliberately kept: the GetCommonFileName robustness fix from
df10aa8.  Our own code no longer produces popularity-0 entries
after this revert, but two paths we don't control can still land
them in m_filenames: on-disk data from any node that ran the
decay build (it could have written 0-popularity entries to
known2.met before this revert lands), and malformed/adversarial
publishes that send popularity = 0 over the wire.  The picker's
seed-from-first / "pick first on tie" fallback closes a sharp
edge in a function whose output is on the wire (TAG_FILENAME,
SearchTermsMatch, CIndexed::AddKeyword reject path) for
essentially no cost.

Refs #795.
@mrjimenez

Copy link
Copy Markdown
Contributor

By the way, do you see on GitHub my comment in the commit #c95e002? Used to see such comments, but now none is shown, and your comment would be visible only after pressing Retry in error message.

I received your comment through e-mail, but had some difficulties to find where it came from. I don't know what happened.

@got3nks

got3nks commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

Here's the full output from the instrumented build over a 12+ hour window on a production amule instance:

2026-06-01 20:53:43: Kad m_filenames high-water rose to 1 (was 0, cap = 100)
2026-06-01 20:53:43: Kad m_filenames high-water rose to 2 (was 1, cap = 100)
2026-06-01 20:54:10: Kad m_filenames high-water rose to 3 (was 2, cap = 100)
2026-06-01 20:54:17: Kad m_filenames high-water rose to 4 (was 3, cap = 100)
2026-06-01 20:54:32: Kad m_filenames high-water rose to 22 (was 4, cap = 100)
2026-06-01 21:03:21: Kad m_filenames high-water rose to 26 (was 22, cap = 100)
2026-06-01 21:38:52: Kad m_filenames high-water rose to 28 (was 26, cap = 100)
2026-06-01 21:40:06: Kad m_filenames high-water rose to 30 (was 28, cap = 100)

Peak: 30 out of MAX_FILENAMES = 100, i.e. 30 % of the cap. The node has been running 12+ hours since the last line emitted and no new high-water lines have appeared — that's the steady-state ceiling under realistic Kad load on this host, not a snapshot mid-warm-up. Most of the rise happens in the first minute after the routing table populates; from there it creeps up by single digits over tens of minutes and then plateaus.

With the decay reverted in #805, the cap is the only mechanism left upstream — and from this data, MAX_FILENAMES = 100 sits comfortably above the live ceiling. Plenty of headroom before the truncation behavior would kick in on a typical node.

On the commit-comment thread: same on my side — I can't reliably load your comment on c95e002 in the GitHub UI either, just intermittently see it referenced from notifications. Almost certainly a GH glitch rather than anything wrong on our end.

@irwir

irwir commented Jun 2, 2026

Copy link
Copy Markdown

Thanks for the information.
Made small searching on the subject of commit comments; GitHub made changes in 2022. These comments could be enabled in organization, otherwise the reliable way would be commenting in issues and PRs.

After looking deeper into the original code and comments in Entry.cpp.
Maximum number of usable file names and IPs is 255. File names require relatively little processing compared to IPs.
Taking into account decades of life without any limit, and preference to simplify things rather than complicate, in my code only one line would be changed:
if (!bDuplicate && m_listFileNames.GetCount() < 255).
That is, keeping the current behaviour with additional safety.

If you still would like to go for replacements, popularity index is not a real count, and for received names should start as 0 and not 1, because external data cannot be trusted.
And, for refreshing the list, delete+append might be better than replace.

Finally, thanks for the invitation.
I will accept it, provided that you are fine with the fact that my contribution might be tiny - there are some toys of my own to play with.

@mrjimenez

Copy link
Copy Markdown
Contributor

Finally, thanks for the invitation. I will accept it, provided that you are fine with the fact that my contribution might be tiny - there are some toys of my own to play with.

Already not a tiny contribution. Size is not the same as impact or relevance.

Welcome!

@got3nks
got3nks deleted the fix/kad-filename-cap-skip-sort branch June 3, 2026 14:16
got3nks added a commit to got3nks/amule that referenced this pull request Jun 4, 2026
…ndex

Adds 55+ merged PRs to the 3.0.0 changelog since the last update
(amule-project#747, 2026-05-27). Narrative additions cover:

- Packaging: expanded the top list to include the macOS per-arch .app
  bundles and the Windows NSIS installer alongside the existing
  AppImage / Flatpak / .dmg / .zip entries. New bullets for amule-project#785
  (alc/alcc/cas/wxcas everywhere + Windows amuleweb), amule-project#794 (.dmg
  amuleweb path), amule-project#789 (<OS>-<arch> artifact naming), amule-project#780 / amule-project#796
  (Windows DPI + comctl32 manifest), amule-project#784 (FHS share/amule paths).

- Bug Fixes & Stability: post-amule-project#744 fixes including EC notification
  leak (amule-project#797), big-library scaling (amule-project#736, amule-project#840 superseding amule-project#728),
  amulegui ghost entries (amule-project#810, amule-project#819, amule-project#841, amule-project#824, amule-project#830, amule-project#760),
  PartFile early hash (amule-project#762), server protocol fixes (amule-project#835, amule-project#788,
  amule-project#721, amule-project#787), crypto stream UB (amule-project#779), UAF prevention (amule-project#756),
  Kad rotation (amule-project#795, amule-project#799/amule-project#805), GTK warning silencing (amule-project#833,
  amule-project#826/amule-project#836), and the clang-tidy worklist (amule-project#770, amule-project#772-amule-project#774).

- Translations: late-cycle wave covering French/Turkish manpages
  (amule-project#753/amule-project#754/amule-project#776), Galician (amule-project#763), Slovenian (amule-project#771), pt-BR
  (amule-project#768/amule-project#775/amule-project#812), French (amule-project#811), plus man-page tooling for
  date+version drift (amule-project#802).

- Contributors: added ngosang for UX feedback on the late-3.0
  cycle (amule-project#817/amule-project#818/amule-project#821/amule-project#828/amule-project#844) and ongoing work on the
  user-facing manual at amule-org.github.io.

- Merged PRs flat index: extended with amule-project#746-amule-project#845 + amule-project#841.
got3nks added a commit to got3nks/amule that referenced this pull request Jun 4, 2026
…ndex

Adds 55+ merged PRs to the 3.0.0 changelog since the last update
(amule-project#747, 2026-05-27). Narrative additions cover:

- Packaging: expanded the top list to include the macOS per-arch .app
  bundles and the Windows NSIS installer alongside the existing
  AppImage / Flatpak / .dmg / .zip entries. New bullets for amule-project#785
  (alc/alcc/cas/wxcas everywhere + Windows amuleweb), amule-project#794 (.dmg
  amuleweb path), amule-project#789 (<OS>-<arch> artifact naming), amule-project#780 / amule-project#796
  (Windows DPI + comctl32 manifest), amule-project#784 (FHS share/amule paths).

- Bug Fixes & Stability: post-amule-project#744 fixes including EC notification
  leak (amule-project#797), big-library scaling (amule-project#736, amule-project#840 superseding amule-project#728),
  amulegui ghost entries (amule-project#810, amule-project#819, amule-project#841, amule-project#824, amule-project#830, amule-project#760),
  PartFile early hash (amule-project#762), server protocol fixes (amule-project#835, amule-project#788,
  amule-project#721, amule-project#787), crypto stream UB (amule-project#779), UAF prevention (amule-project#756),
  Kad rotation (amule-project#795, amule-project#799/amule-project#805), GTK warning silencing (amule-project#833,
  amule-project#826/amule-project#836), and the clang-tidy worklist (amule-project#770, amule-project#772-amule-project#774).

- Translations: late-cycle wave covering French/Turkish manpages
  (amule-project#753/amule-project#754/amule-project#776), Galician (amule-project#763), Slovenian (amule-project#771), pt-BR
  (amule-project#768/amule-project#775/amule-project#812), French (amule-project#811), plus man-page tooling for
  date+version drift (amule-project#802).

- Contributors: added ngosang for UX feedback on the late-3.0
  cycle (amule-project#817/amule-project#818/amule-project#821/amule-project#828/amule-project#844) and ongoing work on the
  user-facing manual at amule-org.github.io.

- Merged PRs flat index: extended with amule-project#746-amule-project#845 + amule-project#841.
mrjimenez pushed a commit that referenced this pull request Jun 4, 2026
Adds 55+ merged PRs to the 3.0.0 changelog since the last update
(#747, 2026-05-27). Narrative additions cover:

- Packaging: expanded the top list to include the macOS per-arch .app
  bundles and the Windows NSIS installer alongside the existing
  AppImage / Flatpak / .dmg / .zip entries. New bullets for #785
  (alc/alcc/cas/wxcas everywhere + Windows amuleweb), #794 (.dmg
  amuleweb path), #789 (<OS>-<arch> artifact naming), #780 / #796
  (Windows DPI + comctl32 manifest), #784 (FHS share/amule paths).

- Bug Fixes & Stability: post-#744 fixes including EC notification
  leak (#797), big-library scaling (#736, #840 superseding #728),
  amulegui ghost entries (#810, #819, #841, #824, #830, #760),
  PartFile early hash (#762), server protocol fixes (#835, #788,
  #721, #787), crypto stream UB (#779), UAF prevention (#756),
  Kad rotation (#795, #799/#805), GTK warning silencing (#833,
  #826/#836), and the clang-tidy worklist (#770, #772-#774).

- Translations: late-cycle wave covering French/Turkish manpages
  (#753/#754/#776), Galician (#763), Slovenian (#771), pt-BR
  (#768/#775/#812), French (#811), plus man-page tooling for
  date+version drift (#802).

- Contributors: added ngosang for UX feedback on the late-3.0
  cycle (#817/#818/#821/#828/#844) and ongoing work on the
  user-facing manual at amule-org.github.io.

- Merged PRs flat index: extended with #746-#845 + #841.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants