Skip to content

PartFile: gate periodic SavePartFile on in-memory dirty flag - #671

Merged
mrjimenez merged 2 commits into
amule-project:masterfrom
got3nks:feat/partfile-dirty-flag
May 22, 2026
Merged

PartFile: gate periodic SavePartFile on in-memory dirty flag#671
mrjimenez merged 2 commits into
amule-project:masterfrom
got3nks:feat/partfile-dirty-flag

Conversation

@got3nks

@got3nks got3nks commented May 21, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #670 (issue #669). #670 cut the per-save cost ~3x by switching SavePartFile from copy-rewrite-copy to atomic-rename. This PR closes the remaining hole: the periodic 60-second save in FlushBuffer still fired for every partfile regardless of whether anything had actually changed since the previous write.

What

A sharer with N partfiles produces N .part.met writes per minute even when nothing has changed (no new chunks, no new sources, no state transitions). For an idle seeder that is wasted disk work: every cycle writes byte-identical content. On SSDs and COW filesystems (ZFS, btrfs) this is meaningfully more expensive than reads, and compounds with the volume.

How

Two-tier dirty model on CPartFile:

Hard state — m_metDirty. Flipped on every mutation of a field whose change should land on disk promptly:

  • Gap list: AddGap / FillGap (both overloads). CPartFileWriteThread's chunk-write completion goes through FillGap, so .part data-file modtime changes are covered transitively.
  • Status / paused flag: SetStatus, PauseFile, ResumeFile.
  • Priorities: SetDownPriority, CKnownFile::SetUpPriority, SetAutoDownPriority, CKnownFile::SetAutoUpPriority (the last two moved out of the header so the partfile case can call MarkMetDirty from the base class).
  • Category: SetCategory.
  • Last-seen-complete timestamp.
  • Filename: SetFileName.
  • Corrupted-part list and AICH-error transitions are already paired with explicit SavePartFile() calls in the ICH/AICH recovery paths.

FlushBuffer's tail-end save fires at the next 60-s tick when m_metDirty is set.

Soft stats — m_statsDirty. Flipped on every upload-counter increment (CFileStatistic::AddRequest / AddAccepted / AddTransferred). These increment on every served chunk so they're deliberately excluded from m_metDirty — otherwise any popular file would re-dirty its partfile constantly and the optimisation collapses. Instead they're persisted on a 10-minute heartbeat measured from the last successful save. Because every successful save resets m_lastMetSaveTick, a regular dirty save automatically defers the next stats save by 10 more minutes — no extra writes when the two coincide.

A successful SavePartFile() clears both dirty bits and stamps m_lastMetSaveTick. LoadPartFile clears both at each successful return and seeds the tick from load time, so a freshly-loaded partfile (whose in-memory state matches the just-read .met) doesn't rewrite identical content on the first tick. The destructor's explicit save persists final state on graceful shutdown regardless of either flag.

Cadence

The 60-second BUFFER_TIME_LIMIT gate before FlushBuffer is untouched. The flags only change whether the periodic save fires, not when:

Partfile state Before After
Idle seeder, no activity 1 save / 60 s 0 saves
Idle seeder, active uploads (chunks served) 1 save / 60 s 1 save / 10 min (stats heartbeat)
Idle seeder, occasional source-complete update 1 save / 60 s 1 save when dirty, then quiet
Active downloader 1+ save / 60 s same (FillGap flips m_metDirty); a coinciding stats heartbeat is absorbed
Priority / category / pause change immediate save immediate save (direct caller, unchanged)

Worst case is bounded to one save per partfile per 60 s. Best case (truly idle, no uploads) is zero writes until shutdown. Stat counters lose at most 10 min on crash, vs the previous 60 s — accepted trade for ~83% fewer writes on a busy sharer.

Status

Draft — awaiting OP confirmation on #669 before promoting. Builds clean on macOS (monolithic + remotegui + daemon).

Follow-up to the amule-project#669 thread (Stoatwblr).  The atomic-rename in
SavePartFile cuts the per-save cost ~3x, but the periodic save
still fired every 60s for every dirty partfile regardless of
whether anything had actually changed since the previous write --
an idle sharer with N partfiles produces N byte-identical .part.met
writes per minute forever.

FlushBuffer's tail-end SavePartFile() now runs only when m_metDirty
is set.  The flag flips when a field that is serialized to the
.part.met mutates:

  - Gap list (AddGap / FillGap) -- the dominant trigger during
    active downloads.  CPartFileWriteThread's chunk-write
    completion goes through FillGap, so .part data-file modtime
    changes are covered transitively.
  - Status / paused flag (SetStatus, PauseFile, ResumeFile).
  - Priorities (SetDownPriority, CKnownFile::SetUpPriority,
    SetAutoDownPriority, CKnownFile::SetAutoUpPriority -- the
    last two moved out of the header so the partfile case can
    call MarkMetDirty from the base class).
  - Category (SetCategory).
  - Last-seen-complete timestamp.
  - Filename (SetFileName).
  - Corrupted-part list and AICH-error transitions are persisted
    via the existing explicit SavePartFile() callers in the
    ICH/AICH recovery paths; those clear the flag.

A successful SavePartFile() clears m_metDirty.  LoadPartFile clears
it at each successful return, since the in-memory state matches
the freshly-read .met and setter calls during tag parsing should
not cause a spurious rewrite on the first tick after load.  The
destructor's explicit save persists final state on graceful
shutdown regardless of the flag, so stat counters (transferred,
AllTimeRequests, etc.) -- which deliberately do not mark dirty --
still survive normal shutdowns; they fall behind only on crash,
same as the existing behavior for many other in-RAM stats.

The 60s BUFFER_TIME_LIMIT gate before FlushBuffer is unchanged --
the flag only changes whether the periodic save fires, not when.
Worst case is bounded to one save per partfile per 60s.  Best
case (truly idle seeder) is zero writes for as long as nothing
changes.  Direct SavePartFile() callers from state-change paths
(priority change, category change, etc.) continue to save
immediately.
@got3nks got3nks mentioned this pull request May 21, 2026
Earlier commit on this branch gated the periodic SavePartFile on
m_metDirty.  That correctly cuts byte-identical writes on idle
seeders, but undersaves upload-stat counters
(AllTimeRequests / AllTimeAccepts / AllTimeTransferred) on a
seeder with active uploads: stats increment on every served
chunk but were intentionally excluded from m_metDirty (otherwise
the dirty-flag optimisation collapses for any popular file), so
they would persist only at graceful shutdown -- losing the whole
session on crash.

Two-tier model:

  - m_metDirty -- hard state (gap list, status, priority, ...)
    flips this.  Saved at the next FlushBuffer tick (~60 s),
    same as before.
  - m_statsDirty -- soft stats only.  Saved only when the
    STATS_HEARTBEAT_MS (10 min) elapses since the last save,
    AND nothing else has forced a save in the meantime.  A
    regular dirty save automatically resets the heartbeat
    because m_lastMetSaveTick is updated on every successful
    save.

Tagged in CFileStatistic::AddRequest / AddAccepted /
AddTransferred (the three increment paths) with an IsPartFile()
guard so completed files in known.met are unaffected.

Net effect: an idle seeder with no activity writes nothing.  A
pure seeder with active uploads writes its .met at most once
per 10 min instead of once per 60 s -- 6x fewer writes for the
same accuracy bound.  An active downloader's cadence is
unchanged.
@got3nks
got3nks marked this pull request as ready for review May 21, 2026 22:28
@mrjimenez
mrjimenez merged commit e8363bf into amule-project:master May 22, 2026
12 checks passed
@got3nks
got3nks deleted the feat/partfile-dirty-flag branch May 22, 2026 13:43
ngosang pushed a commit to ngosang/amule that referenced this pull request Jul 28, 2026
…ect#180 phase 1) (amule-project#671)

* feat(gui): port the Preferences sidebar to wxDataViewCtrl (amule-project#180 phase 1)

Phase 1 of the accessibility work in amule-project#180: the Preferences sidebar was
a plain wxListCtrl, invisible/unusable to screen readers on macOS
(VoiceOver) and Linux (Orca) -- wxDataViewCtrl is native on both those
platforms (wxHAS_NATIVE_DATAVIEWCTRL for __WXGTK__/__WXOSX__), falling
back to the generic wx-drawn control only on Windows, where the answer
was already "works the same as before" per got3nks's review.

m_PrefsIcons is now a wxDataViewListCtrl (single icon+text column, no
tree mode -- the sidebar is a flat list today, per the earlier
correction to the amule-project#180 thread; tree mode is future work for
subcategories, not part of this port).

Along the way, fixes the positional-index bug flagged in the original
amule-project#180 phasing proposal: OnPrefsPageChange used to derive both the
current wxPanel* and the pages[] lookup from the row's live *position*
in the sidebar (event.GetIndex()), which drifts whenever the Server or
IP2Country row is hidden/re-shown -- a comment already noted this had
bitten the reset-button-visibility check once. Each row's item data is
now the page's stable pages[] array index instead, set once at
insertion and never derived from position; m_pageWidgets (indexed by
that same stable index) replaces the old SetItemPtrData(widget)
lookup. EnableServerTab's insert-at-m_IndexServerTab positional logic
is otherwise unchanged -- it was already correct for that purpose,
just fragile for the *different* purpose OnPrefsPageChange was
(mis)reusing it for.

Column width is computed by measuring actual (translated) label text
extents rather than relying on wxDataViewColumn's own auto-size timing,
which isn't reliably immediate across native vs generic backends.

Verified: built and ran both amule and amuleGUI on macOS. Visually
confirmed (by a human, not just me, since I can't reliably screenshot
this myself) that every page label renders in full (an early version
clipped the longest ones -- fixed by widening the padding budget),
page switching works, and the Server tab correctly disappears/
reappears at the right position when toggling the ED2K network on and
off. Windows and Linux screenshots still needed before this is
merge-ready, per got3nks's stated requirement -- see PR description.

* fix(gui): guard invalid dataview item and fix sidebar label measurement

EVT_DATAVIEW_SELECTION_CHANGED also fires when the selection is
cleared, unlike the old EVT_LIST_ITEM_SELECTED, and GetItemData() on
an invalid item crashes outright. Also measure the sidebar label width
with m_PrefsIcons's own font instead of the dialog's, since the two
can differ on native backends.

Addresses review feedback from got3nks on amule-project#671.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

---------

Co-authored-by: Claude Sonnet 5 <[email protected]>
ngosang pushed a commit to ngosang/amule that referenced this pull request Jul 28, 2026
…-project#663)

* feat(gui): remove the global Connect/Disconnect toolbar button

Closes amule-project#402. Per-network controls already exist (Kad pane's Start/Stop,
Servers pane's ED2K connect/disconnect), the connection state is
already shown in the status bar, and after discussing relocating it
into the Networks view the agreed call (got3nks + ngosang) was simply
to remove it -- mirrors what amule-project#585 already did on the Web UI side.

Removed the toolbar tool, its three skin icons (Toolbar_Connect/
Disconnect/Connecting) and their bitmap wiring, the button-update block
in ShowConnectionState(), and the two EnableTool(ID_BUTTONCONNECT, ...)
call sites.

CamuleDlg::OnBnConnect() itself stays -- CMuleTrayIcon::DoConnectDisconnect()
still calls it for the tray icon's own connect/disconnect action, which
is unaffected by this change.

ShowConnectionState()'s skinChanged parameter was only ever read by the
now-removed block, so it's gone too, along with the one call site that
passed true and the forceUpdate plumbing that reached it through
GuiEvents.cpp's ShowConnState() free function.

Left ID_BUTTONCONNECT and muuli_wdr.cpp's muleToolbar() (which still
calls it) alone -- that function predates Apply_Toolbar_Skin, is never
called anywhere, and touching pre-existing dead code is out of scope
for this fix.

Verified: built and ran both amule and amulegui cleanly, full unit
test suite passing (26/26). Confirmed via the po/ diff that "Connect"/
"Disconnect"/"Cancel" remain in the catalog (still used by other
per-network buttons) -- only their now-orphaned toolbar-specific
tooltip strings were dropped. Could not get a real on-screen visual
confirmation of the toolbar layout -- same macOS Accessibility
automation limitations as amule-project#180 got in the way of screenshotting the
actual app window.

* i18n: regenerate po/ catalogs after rebasing onto current master

Mechanical rebase to resolve conflicts against master's amule-project#642/amule-project#643 --
no source changes here. Resolved a real conflict in amuleDlg.cpp's
event table: amule-project#642 added the Alt+<letter> EVT_MENU block right where
this branch removed the EVT_TOOL(ID_BUTTONCONNECT, ...) line; kept
amule-project#642's block, dropped the connect-button line as intended.

Regenerated again via scripts/update-po.sh after later rebasing onto
master's amule-project#671 (Preferences sidebar) and amule-project#665/amule-project#662 (amuleapi
credentials, sparse part-file setting), which had added strings this
branch's catalogs didn't have yet.

* feat(gui): turn the per-network Disconnect buttons into Connect/Cancel/Disconnect toggles

Relocates the removed global toolbar button's functionality into the
ED2K (IDC_ED2KDISCONNECT) and Kad (ID_KADDISCONNECT) panes instead of
just dropping it, per got3nks's proposal on amule-project#663: each button now
mirrors ed2kState/kadState (off/connecting/connected) using the
existing connButImg() bitmaps, and can independently connect or
cancel/disconnect its own network -- something the old OR-toggled
global button never allowed. The ED2K pane also gains a Connect
button it never had (previously Disconnect-only).

CServerWnd::UpdateED2KConnectButton() / CKadDlg::UpdateConnectButton()
are called from CamuleDlg::ShowConnectionState() alongside the
existing UpdateED2KInfo()/UpdateKadInfo() calls, and once from each
pane's own ctor/Init() for the initial paint. OnBnClickedED2KDisconnect
and OnBnClickedDisconnectKad gained the missing "connect when off"
branch; OnBnConnect and the tray icon are untouched.

Verified interactively on Windows: both buttons show the correct
label/icon for their live state, ED2K disconnect/reconnect works from
its own button, and disabling ED2K in Preferences while Kad stays
connected confirms the two toggle independently.

po/ regenerated via scripts/update-po.sh to match the new button
labels ("Disconnect Kad" is now unused and moves to the obsolete
section; no new translatable strings).

* fix(gui): drop the per-network connect-button state cache

got3nks flagged two bugs in review (PR amule-project#663): the static cache's
early return also skipped Enable(), so the button could get stuck
disabled once IsReady()/network-pref state changed without the
Connect/Cancel/Disconnect state itself changing (master had two now-
deleted escape hatches -- the skinChanged force-path and an
unconditional EnableTool call -- that used to paper over this). Worse,
the cache was function-scope static, surviving widget recreation, so
a fresh pane's Init() call could early-return and leave the wxDesigner
default label ("Disconnect Kad") on screen.

Dropping the cache removes both: SetLabel/SetBitmap/Enable are cheap
on a plain wxButton (unlike the old toolbar tool, which had real
native-image-list reasons to memoize, see amuleDlg.cpp:1054-ish), and
master's own second call site already did an unconditional EnableTool
every tick without issue.

Second review round (macOS + Ubuntu ARM64, monolithic + amuleGUI)
confirmed the toggle works functionally but flagged three cosmetic
issues, all fixed here:

- No gap between the connect-button icon and its label. The obvious
  fix, wxButton::SetBitmapMargins(), is NOT portable (wxOSX overrides
  it, wxGTK inherits the base no-op) -- prefixing the label with a
  literal space outside the _() call behaves identically on both and
  introduces no new msgid.
- The Kad button was stretched full-width by its sizer's .Expand()
  flag, while the ED2K one sizes naturally; GTK and macOS then filled
  the slack differently (centered vs. icon-left). Dropped .Expand()
  so the two buttons size the same way -- trades away lining up with
  the "Bootstrap from known clients" button above it, but the two
  connect buttons reading as the same control matters more.
- CServerWnd::UpdateED2KConnectButton() and CKadDlg::UpdateConnectButton()
  were near-identical (same 3-state enum, same switch, same three
  bitmaps). Factored into a shared SetConnectButtonState(button, state,
  enabled) helper in muuli_wdr.cpp/.h, next to connButImg() which it
  reuses -- so the icon-spacing fix above lives in one place instead
  of two.

Verified interactively on Windows: re-tested full disconnect/reconnect
cycle on both panes, toggling ED2K off and back on via Preferences
(existing behavior: needs an app restart to re-enable) with correct
state on the fresh post-restart paint, and confirmed by screenshot
that both buttons now show a visible icon/label gap and size the same
way (Kad no longer stretched). Also confirmed amulegui (CLIENT_GUI)
builds clean with these changes; did not perform full interactive
remote-daemon testing.

* fix(gui): first-paint layout bug + connect-button placement, per review

Two more issues from got3nks's Linux/GTK pass on amule-project#663.

1. Bug: the ED2K connect button painted with the icon overlapping the
   label on first show (Kad was fine, only by luck of timing).
   CServerWnd::UpdateED2KConnectButton() runs after sizer->Show(this,
   TRUE) in the ctor, so SetBitmap() grows the button's best size
   after the row was already measured against the text-only label,
   and nothing re-lays it out until a window resize. Fixed by calling
   Layout() on the button's parent at the end of the shared
   SetConnectButtonState() helper -- covers both panes, and stops Kad
   from relying on being laid out late (its notebook page is only
   measured once shown, after Init() already set its bitmap).

2. Layout: moved both connect/disconnect toggles to lead their pane
   instead of sitting among secondary controls, per got3nks's request:
   - ED2K (serverListDlgUp): toggle now on its own top row, right-
     aligned via a stretch spacer, directly above the server list.
     The "Add server manually" form (name/IP/port/Add) moved below
     the table instead of sharing a row with the toggle; the vertical
     separator that used to sit between them is gone (no longer
     needed once they're not adjacent).
   - Kad (KadDlg): toggle moved out of the "Bootstrap" static box
     (where it sat under "Bootstrap from known clients") to its own
     top row above the box, right-aligned the same way as ED2K's.
     item0 changed from a 1x1 FlexGridSizer (which only ever held the
     two-column area) to a plain vertical wxBoxSizer so it can hold
     the new top row plus the existing two-column layout.

Both panes now read "primary control on top, secondary/manual
controls below" -- matching shape on both tabs.

Verified interactively on Windows: screenshotted both panes, confirmed
no icon/label overlap on first paint (no window resize needed), and
re-ran the disconnect/reconnect toggle cycle on both after the layout
change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants