Skip to content

sharedfiles: auto-rescan via wxFileSystemWatcher - #591

Merged
mrjimenez merged 2 commits into
amule-project:masterfrom
got3nks:feature/auto-rescan-shared
May 13, 2026
Merged

sharedfiles: auto-rescan via wxFileSystemWatcher#591
mrjimenez merged 2 commits into
amule-project:masterfrom
got3nks:feature/auto-rescan-shared

Conversation

@got3nks

@got3nks got3nks commented May 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds automatic rescan of shared directories. aMule currently only rescans shared folders on explicit user action ("Reload shared files") or at daemon start; dropping a file into a shared folder leaves it un-shared until the next manual reload — a long-standing gap for anyone who curates a library outside aMule (rsync, Finder drag-and-drop, torrent client move-on-complete, etc.).

A new CSharedDirWatcher observes every path in shareddir_list via wxFileSystemWatcher (inotify on Linux, ReadDirectoryChangesW on Windows, FSEvents on macOS, kqueue on BSD) and triggers a debounced CSharedFileList::Reload 5 s after the last event. The window is long enough to coalesce the bursts produced by tar -x / rsync, short enough that a single drag-and-drop feels responsive.

New subdirectories created under any watched parent are auto-appended to shareddir_list and persisted via a new CPreferences::SaveSharedFolders() that writes only shareddir.dat, so the change survives a restart without forcing a full preferences.dat rewrite on every mkdir event. That mirrors what users already expect from the recursive-share button.

The feature is exposed as a new pref AutoRescanSharedDirs (default ON) with a checkbox in the Directories preferences panel. A new EC tag EC_TAG_DIRECTORIES_AUTO_RESCAN (0x1A05) carries the pref over the EC channel so amulegui can flip it on a remote amuled, with the daemon-side Apply path enabling/disabling the watcher live (no restart needed).

Platform-specific details

Linux (inotify) and Windows (ReadDirectoryChangesW). Per-directory Add(). shareddir_list already enumerates every subdirectory individually when a user uses the recursive-share button, so the inotify watch count tracks shareddir_list.size() rather than total subtree depth. Per-path Add() failures (e.g. hitting /proc/sys/fs/inotify/max_user_watches) are logged and skipped; partial coverage is the graceful-degradation path rather than a hard error.

macOS (FSEvents). Routes through AddTree() instead. wx 3.3.2's bare Add() falls through to the kqueue base and returns false on otherwise-openable directories — verified against wx upstream and reproduced locally. FSEvents is also the API Apple actually recommends for directory monitoring. AddTree() is recursive, so the registration loop skips shareddir_list entries whose ancestor is also in the list to avoid overlapping streams (wx silently returns false on the second).

Deferred watcher construction. wx's inotify backend hard-requires an active wx event loop at wxFileSystemWatcher::Init() — without one, m_service stays null and every Add() returns false. CamuleApp::OnInit() runs before the loop starts, so the first Enable() from there queues itself via wxEvtHandler::CallAfter and re-runs on the next loop iteration. Calls from the prefs dialog or EC apply path already run inside an active loop and take the immediate branch.

macOS amuled — CFRunLoop pump (second commit). wx's FSEvents wrapper schedules its stream on the calling thread's CFRunLoop. Under aMule.app the Cocoa main loop spins that runloop, so callbacks deliver normally. Under amuled (wxAppConsole) the loop never spins, so events queue forever. A 200 ms wxTimer calls CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0, true) — a non-blocking drain — so queued FSEvents callbacks dispatch into wx's event queue and reach our handler. Gated by !wxTheApp->IsGUI() so aMule.app is unaffected (its Cocoa main loop already pumps).

End-to-end verification

Platform Binary Mechanism Result
Linux Ubuntu ARM64 amuled inotify drop file → 5 s → Reload → new file in shared list
Windows ARM64 amuled.exe ReadDirectoryChangesW drop file → 5 s → Reload → new file in shared list
macOS amuled FSEvents + CFRunLoop pump drop file → 5 s → Reload → new file in shared list
macOS aMule.app FSEvents via Cocoa main loop drop file → 5 s → Reload → new file in shared list

All four targets (amule, amuled, amulecmd, amulegui) compile and link clean on macOS, Linux ARM64, and Windows ARM64.

got3nks added 2 commits May 13, 2026 10:53
aMule today only re-scans the on-disk content of shareddir_list on
explicit user action ("Reload shared files") or at daemon start.
Dropping a file into a shared folder leaves it un-shared until the
next manual reload — a long-standing user-experience gap for anyone
who curates a library outside aMule (rsync, Finder drag-and-drop,
torrent client move-on-complete, etc.).

Add a directory watcher that observes every path in shareddir_list
via wxFileSystemWatcher (inotify on Linux, FSEvents on macOS,
ReadDirectoryChangesW on Windows, kqueue on BSD) and triggers a
debounced CSharedFileList::Reload when something changes. The
debounce window is 5 s, long enough to coalesce the bursts that
bulk operations (tar -x, rsync) produce into one Reload, short
enough that a single drag-and-drop feels responsive.

Auto-share new subdirectories of any watched parent. When a user
creates Music/2026/Album under an already-shared Music tree, the
new subdir is appended to shareddir_list and registered with the
watcher, mirroring what users expect from the existing recursive-
share flow. The change is persisted via a new
CPreferences::SaveSharedFolders that writes only shareddir.dat so
we don't force a full preferences.dat rewrite on every mkdir event.

New CSharedDirWatcher class (src/SharedDirWatcher.{h,cpp}) owns the
wxFileSystemWatcher and the debounce wxTimer. CSharedFileList owns
the watcher lifetime via a new EnableDirectoryWatcher(bool) entry
point, called once at daemon start (gated on the new pref) and again
when the user toggles the pref live from PrefsUnifiedDlg. Reload
calls watcher->Refresh after rebuilding the list, so dirs added or
removed since the previous Reload are picked up without having to
diff the path set ourselves.

Deferred watcher construction. wx's inotify backend (Linux) requires
an active wx event loop at wxFileSystemWatcher::Init() time —
without one, m_service stays null and every Add() silently returns
false. CamuleApp::OnInit() runs before the loop starts, so the
first Enable() call from there queues itself via wxEvtHandler's
CallAfter and re-runs at the next loop iteration. Calls from the
prefs dialog or EC apply path already run inside an active loop and
take the immediate branch. Queueing on the CSharedDirWatcher itself
(rather than wxTheApp) lets wx purge the pending event if the
watcher is destroyed before the loop drains.

Two registration strategies. Linux (inotify), Windows
(ReadDirectoryChangesW), and BSD (kqueue) use per-directory Add() —
shareddir_list already enumerates every subdirectory individually
when a user uses the recursive-share button, so inotify watch
count tracks shareddir_list.size() rather than total subtree
depth. macOS routes through AddTree() instead, because (a) wx
3.3.2's bare Add() falls through to the kqueue base and returns
false on otherwise-openable directories — verified against wx
upstream and reproduced locally — and (b) FSEvents is the API
Apple actually recommends for directory monitoring. Since
AddTree() is recursive on macOS, the registration loop skips
shareddir_list entries whose ancestor is also in the list so we
don't try to open overlapping FSEvents streams (wx silently
returns false on the second). The corresponding RegisterNew-
Subdirectory path is a no-op on the watcher under macOS — the
parent stream already delivers events for new descendants —
while other platforms still need an explicit Add() for the new
subdir's own contents.

The feature is exposed as a new pref AutoRescanSharedDirs, defaulted
ON so the behaviour is visible without opt-in. A checkbox is added
to the Directories preferences panel
(IDC_AUTO_RESCAN_SHARED = 10333, picked above the wxDesigner-
generated ID range so a future muuli_wdr.* regeneration doesn't
collide). Users on Linux hosts hitting /proc/sys/fs/inotify/
max_user_watches can disable it; the watcher already logs and
continues on per-path Add() failure so partial coverage is the
graceful-degradation path rather than a hard error.

A new EC tag EC_TAG_DIRECTORIES_AUTO_RESCAN (0x1A05) carries the
pref over the EC channel so amulegui can flip it on a remote
amuled. The amuled-side Apply path calls EnableDirectoryWatcher
immediately so the change takes effect without a daemon restart.
amule-remote-gui provides a no-op EnableDirectoryWatcher on the
GUI side — the watcher only ever lives on the file-owning daemon.

Build: amule + amuled + amulecmd + amulegui compile and link clean
on macOS Apple Silicon (Ninja, monolithic + daemon + remotegui +
amulecmd targets). End-to-end verified on macOS: dropping a file
into a watched directory triggers FSEvents → 5 s debounce →
Reload → file appears in the shared-files view.
amuled on macOS registers wxFsEventsFileSystemWatcher streams just
fine, but never receives any of their callbacks: the stream is
scheduled on the calling thread's CFRunLoop, and wxAppConsole's
event loop on macOS does not spin that runloop. Under aMule.app a
real Cocoa main loop drives it, so the GUI path has always worked;
under amuled the events queue forever and the auto-rescan feature
is silently inert.

Add a 200 ms wxTimer in CSharedDirWatcher that calls
CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0, true) on each tick.
That is a non-blocking drain: it dispatches any FSEvents callbacks
the kernel has queued on this thread's CFRunLoop and returns
immediately if there are none. wx's wrapper then translates those
callbacks into wxFileSystemWatcherEvent and posts them through the
normal event queue, where our existing OnFileSystemEvent handler
picks them up and runs the 5 s debounce as on every other platform.

The pump only starts when wxTheApp->IsGUI() is false, so aMule.app
(whose Cocoa main loop already pumps the runloop) is unaffected.

Cost in the amuled case is ~5 non-blocking wakeups per second on
the main thread; CFRunLoopRunInMode with a zero timeout returns
immediately when no sources are ready, so the steady-state overhead
is negligible relative to amuled's existing per-second timers. The
pump is started in Enable() after the watcher is up and stopped in
Disable() before the watcher is torn down, so toggling the pref
from amulegui via EC reflects on the runloop drain as well.

Build: amule + amuled compile clean on macOS Apple Silicon.
End-to-end verified on macOS amuled: dropping a file into a watched
directory triggers FSEvents → CFRunLoop pump dispatches → 5 s
debounce → Reload → file appears in the shared-files view, matching
the Linux/Windows behaviour already verified end-to-end.
@mrjimenez
mrjimenez merged commit 975f60f into amule-project:master May 13, 2026
12 checks passed
@got3nks
got3nks deleted the feature/auto-rescan-shared branch May 14, 2026 13:33
got3nks added a commit to got3nks/amule that referenced this pull request May 14, 2026
The previous commit's cold-discovery (and the existing amule-project#591 watcher
RegisterNewSubdirectory) auto-added new subdirectories to every
shared dir indiscriminately. There was no way to express "share
/Music but NOT its sensitive nested subfolders" -- the moment a new
subdir appeared under any shared root it became part of the share.
Desktop users with sensitive nested folders inside a shared root
were exposed.

This commit introduces a per-root recursive flag, persisted in a
new shareddir-recursive.dat file. The watcher and cold-discovery
now gate their auto-add behaviour on "ancestor is recursive": only
roots the user has explicitly marked recursive via the UI's right-
click "share subdirectories recursively" automatically include
new/existing subdirectories. Non-recursive shares stay strict.

Three on-disk files going forward:

* shareddir-recursive.dat (new): roots the user marked recursive.
* shareddir-explicit.dat (new): roots the user added explicitly,
  without recursion.
* shareddir.dat (existing): regenerated as the runtime union for
  backwards compatibility -- older binaries and external scripts
  that read or write this file see the same effective list as
  before.

`CPreferences::shareddir_list` (the in-memory list the share scan,
watcher, etc. read as authoritative) is recomputed at every
ReloadSharedFolders as `shareddir_explicit_list ∪ expand(shareddir_
recursive_list)`. The expansion walks each recursive root's subtree
(CDirIterator::Dir) and adds every existing descendant directory.

Migration: a pre-existing shareddir.dat with no
shareddir-{explicit,recursive}.dat is loaded entirely into the
explicit list. Existing users keep their path set but stop silently
auto-recursing -- they opt back in per-root via the UI. Safe
default for desktop users.

Reconciliation runs on every Reload (startup, EC reload, UI
reload, watcher-debounced reload), not just startup. Diffs the
on-disk shareddir.dat against the expected union; entries written
externally (Docker entrypoints, sysadmin edits, old-binary writes)
are imported into shareddir_explicit_list, and entries the
external writer removed are dropped from explicit (entries that
came from the recursive expansion are left alone -- the user's
recursive intent overrides single-entry external edits).

UI side: PrefsUnifiedDlg::CommitSharedDirsWithProgress already
extracted explicit vs recursive intents from the DirectoryTreeCtrl
(GetSharedDirectories vs GetRecursiveSharedDirectories). The
commit path now assigns each to its dedicated Preferences list
instead of conflating them into shareddir_list, and the cancel
rollback restores all three. Initial dialog state loads from the
two intent lists rather than the runtime union (which would
otherwise render auto-discovered subdirs as user-selected items).

Watcher side: CSharedDirWatcher::RegisterNewSubdirectory and
::ColdDiscoverSubdirs both check
CPreferences::IsRecursiveAncestor() before adding. Explicit shares
no longer accumulate subdirs at runtime; recursive shares behave
as before.

Verified on macOS with a three-mode smoke test:

* `recursive-root` (in shareddir-recursive.dat) with pre-existing
  subA/subB: both auto-expand into shareddir.dat at startup;
  recursive-root/rec-hot created at runtime is auto-added.
* `explicit-root` (in shareddir-explicit.dat) with pre-existing
  subC: subC NOT in shareddir.dat; explicit-root/exp-hot created
  at runtime NOT added.
* `script-added-dir` written to shareddir.dat externally
  (Docker-entrypoint pattern): imported into
  shareddir-explicit.dat on next boot, survives subsequent
  shareddir.dat regenerations.

No file format change to shareddir.dat (still one path per line);
the new files use the same format. Older binaries that ignore
the new files continue to operate against shareddir.dat as the
union, with the caveat that recursive intent is lost when an
older binary round-trips shareddir.dat (the markers in
shareddir-recursive.dat survive untouched because the old binary
doesn't read or write that file).
mrjimenez pushed a commit that referenced this pull request May 14, 2026
The previous commit's cold-discovery (and the existing #591 watcher
RegisterNewSubdirectory) auto-added new subdirectories to every
shared dir indiscriminately. There was no way to express "share
/Music but NOT its sensitive nested subfolders" -- the moment a new
subdir appeared under any shared root it became part of the share.
Desktop users with sensitive nested folders inside a shared root
were exposed.

This commit introduces a per-root recursive flag, persisted in a
new shareddir-recursive.dat file. The watcher and cold-discovery
now gate their auto-add behaviour on "ancestor is recursive": only
roots the user has explicitly marked recursive via the UI's right-
click "share subdirectories recursively" automatically include
new/existing subdirectories. Non-recursive shares stay strict.

Three on-disk files going forward:

* shareddir-recursive.dat (new): roots the user marked recursive.
* shareddir-explicit.dat (new): roots the user added explicitly,
  without recursion.
* shareddir.dat (existing): regenerated as the runtime union for
  backwards compatibility -- older binaries and external scripts
  that read or write this file see the same effective list as
  before.

`CPreferences::shareddir_list` (the in-memory list the share scan,
watcher, etc. read as authoritative) is recomputed at every
ReloadSharedFolders as `shareddir_explicit_list ∪ expand(shareddir_
recursive_list)`. The expansion walks each recursive root's subtree
(CDirIterator::Dir) and adds every existing descendant directory.

Migration: a pre-existing shareddir.dat with no
shareddir-{explicit,recursive}.dat is loaded entirely into the
explicit list. Existing users keep their path set but stop silently
auto-recursing -- they opt back in per-root via the UI. Safe
default for desktop users.

Reconciliation runs on every Reload (startup, EC reload, UI
reload, watcher-debounced reload), not just startup. Diffs the
on-disk shareddir.dat against the expected union; entries written
externally (Docker entrypoints, sysadmin edits, old-binary writes)
are imported into shareddir_explicit_list, and entries the
external writer removed are dropped from explicit (entries that
came from the recursive expansion are left alone -- the user's
recursive intent overrides single-entry external edits).

UI side: PrefsUnifiedDlg::CommitSharedDirsWithProgress already
extracted explicit vs recursive intents from the DirectoryTreeCtrl
(GetSharedDirectories vs GetRecursiveSharedDirectories). The
commit path now assigns each to its dedicated Preferences list
instead of conflating them into shareddir_list, and the cancel
rollback restores all three. Initial dialog state loads from the
two intent lists rather than the runtime union (which would
otherwise render auto-discovered subdirs as user-selected items).

Watcher side: CSharedDirWatcher::RegisterNewSubdirectory and
::ColdDiscoverSubdirs both check
CPreferences::IsRecursiveAncestor() before adding. Explicit shares
no longer accumulate subdirs at runtime; recursive shares behave
as before.

Verified on macOS with a three-mode smoke test:

* `recursive-root` (in shareddir-recursive.dat) with pre-existing
  subA/subB: both auto-expand into shareddir.dat at startup;
  recursive-root/rec-hot created at runtime is auto-added.
* `explicit-root` (in shareddir-explicit.dat) with pre-existing
  subC: subC NOT in shareddir.dat; explicit-root/exp-hot created
  at runtime NOT added.
* `script-added-dir` written to shareddir.dat externally
  (Docker-entrypoint pattern): imported into
  shareddir-explicit.dat on next boot, survives subsequent
  shareddir.dat regenerations.

No file format change to shareddir.dat (still one path per line);
the new files use the same format. Older binaries that ignore
the new files continue to operate against shareddir.dat as the
union, with the caveat that recursive intent is lost when an
older binary round-trips shareddir.dat (the markers in
shareddir-recursive.dat survive untouched because the old binary
doesn't read or write that file).
mrjimenez pushed a commit that referenced this pull request May 27, 2026
CSharedDirWatcher::RegisterAllPaths() previously subscribed only the
explicit shareddir_list to the wxFileSystemWatcher backend. But
CSharedFileList::Reload() treats three sources as shared (around
SharedFileList.cpp:370):

  1) the global Incoming dir   (thePrefs::GetIncomingDir())
  2) each category's Incoming  (theApp->glob_prefs->GetCatPath(i),
                                i = 1 .. GetCatCount()-1)
  3) the explicit shareddir_list

So the auto-rescan introduced in #591 was missing (1) and (2):

* A file dropped into ~/.aMule/Incoming did not trigger the watcher
  -> the file never appeared in Shared until some unrelated CREATE
  fired elsewhere in the watched tree and incidentally rescanned.
* Same story for any category-specific incoming dir.

Reproducer from #741:

  echo 123456 > ~/.aMule/Incoming/file_in_Incoming
  # wait — never auto-shared
  echo 654321 > ~/.aMule/extra/file_in_extra   # (extra is shared)
  # both files appear in Shared within seconds

Completed downloads aren't affected: CPartFile signals the scanner
directly when a file transitions to Incoming, so end-of-transfer
sharing keeps working. The bug only bites manual / external drops
into Incoming, which is exactly the case #741 walks through.

Fix: build the effective watch list by mirroring Reload()'s three
sources (Incoming + cat paths + shareddir_list), deduplicating
by GetRaw() so a user who happened to also list Incoming as an
explicit shared dir doesn't get a duplicate watch registration.
The dedup matters on Linux/BSD/Windows (per-dir Add()) and on
macOS (FSEvents rejects overlapping AddTree streams; an explicit
ancestor + Incoming-as-descendant still gets pruned by the existing
covered_by_ancestor logic further down).

Reported by @danim7 (#741).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants