Skip to content

Implement "Verify Local Data" function in Shared window (monolithic) - #381

Merged
got3nks merged 3 commits into
amule-org:masterfrom
danim7:verify-local-data
Jul 13, 2026
Merged

Implement "Verify Local Data" function in Shared window (monolithic)#381
got3nks merged 3 commits into
amule-org:masterfrom
danim7:verify-local-data

Conversation

@danim7

@danim7 danim7 commented Jul 9, 2026

Copy link
Copy Markdown

Opening as draft, work in progress

Intro

To share a file in ed2k, the file is hashed with MD4 and SHA1 (AICH) algorithms to uniquely identify it and help fix corruption during downloads. These hashes are stored in the known.met and known2_64.met files.

This PR introduces a new functionality to make use of the .met files as checksum files. This allows users to check the integrity of known files against the stored hashes in the .met files. It is accesible via right-click on the Shared window (monolithic-only in this PR), and the result of the check is printed to the log.

Tests

  • Create a file with fallocate -l 1g ~/.aMule/Incoming/1g, let amule discover and hash it for the first time, then re-check it via the new option menu, file is OK in the log:

Verify Local Data (MD4 & AICH): Result OK for /home/test/.aMule/Incoming/1g

  • Close amule. Backup the timestamp of the file to a separate file, create corruption in the shared file, restore the timestamp, re-open amule (which shall not re-hash it, because the timestamp is the same), ask to verify the file, the corrupted blocks are reported in the log:

 touch -r ~/.aMule/Incoming/1g /tmp/save_timestamp
 printf '\xFF' | dd of=~/.aMule/Incoming/1g bs=1 seek=123000000 conv=notrunc
 printf '\xFF' | dd of=~/.aMule/Incoming/1g bs=1 seek=124000000 conv=notrunc
 printf '\xFF' | dd of=~/.aMule/Incoming/1g bs=1 seek=987654321 conv=notrunc
 touch -r /tmp/save_timestamp ~/.aMule/Incoming/1g 

Verify Local Data (MD4 & AICH): ERRORS FOUND! /home/test/.aMule/Incoming/1g Failed blocks: MD4: 12,101. AICH: 12: (33,39), 101: (27)

  • Launch checks on multiple files at the same time, and multiple checks on the same file before it finishes: the checks are queued by the thread scheduler, and executed one at a time. The duplicated checks on the same file are discarded and only one runs per file.

  • Create a new shared file and let amule hash it. Close amule. Now, we will corrupt the known.met file. When we launch amule, it detects the .met file is corrupt, prints a log warning and self-heals! When we check the file, the result is OK, if we re-open amule, it no longer show the known file list as corrupt

printf '\xFF' | dd of=known.met bs=1 seek=100 conv=notrunc

Failed to load entry in known file list, file may be corrupt

  • Create a new shared file and let amule hash it. Close amule. Now, we will corrupt the known2_64.met file. When we ask amule to verify it, LoadHashSet() detects the AICH tree is corrupt and performs only the MD4 check.
    printf '\xFF' | dd of=known2_64.met bs=1 seek=117000 conv=notrunc
SHAHashSet: Failed to load HashSet: Calculated Masterhash differs from given Masterhash - hashset corrupt!
Verify Local Data (MD4): Result OK for /home/test/.aMule/Incoming/1g
  • Test different file sizes: 50 KB, 7 MB, 1 GB

Pending for this PR

  • Finish all test cases
  • Update .po files
  • Tier-2 clang errors??

Not in this PR

I prefer to make multiple small-step PRs, instead of one single big-step PR.
This PR creates the first check available for users, and also prepares the infrastucture for potential, future developments (no guarantees about (if/when) would do them):

  • Allow verification of Part files, not only Completed files
  • Wire this feature in EC, and make it available in amulecmd/amulegui/amuleweb/amuleapi
  • Create a workflow for files when corruption is detected: flag as corrupted, don't share corrupted parts, re-download those parts, etc...
  • This PR also facilitates creating the background periodic check discussed in feat: Hash parts on upload to detect silent bit-rot #166, the original request from there needs more work though

@got3nks

got3nks commented Jul 10, 2026

Copy link
Copy Markdown

Thanks for this, @danim7 — really nice contribution, and I agree it's a strong foundation for the follow-ups you list (part-file verification, EC wiring, a corruption-recovery workflow, the periodic check from #166). The threading care stands out: copying the AICH hashset into a local storedFile to avoid racing a peer's OP_AICHREQUEST is exactly the right call, and the #ifndef CLIENT_GUI gating is correct here — SharedFilesCtrl.cpp is compiled per-executable, so both the menu option and the CVerifyLocalDataTask symbol are genuinely excluded from amulegui. The disk-side handling is also solid: the !file.Open guard plus the CIOFailureException/CSafeIOException catches mean a file deleted or truncated after the check is triggered degrades to a log line rather than a crash (on POSIX the open fd keeps reading the original inode anyway).

A few things from a first pass:

One blocker — files smaller than PARTSIZE (9.28 MiB). The MD4 section falls through for a single-part file that verifies OK:

if (knownFile->GetHashCount() == 0 && md4Hash != m_fileID) {
    m_corruptedMD4.push_back(part);
    break;
}
if (knownFile->GetPartHash(part) != md4Hash)   // reached even when the file is OK
    m_corruptedMD4.push_back(part);

GetPartHash is wxASSERT(part < m_hashlist.size()); return m_hashlist[part];, and m_hashlist is empty for single-part files — so an OK small file hits wxASSERT(0 < 0) in debug and an out-of-bounds read (UB) in release. A continue after the single-part case fixes it. Worth adding a sub-9.28 MiB file to your test list, since the 1 GB file is multi-part and doesn't exercise this path.

One smaller thing:

  • The wxFAIL on an invalid AICH block will abort a debug build if that state is ever reachable (e.g. a last-partial-part edge). If it can happen, a debug-log-and-skip is safer than an assert; if it truly can't, a one-line comment on why would help.

Two thoughts on where this goes next:

  • It'd be great to surface progress in the Shared window rather than only the log — the verify can take a while on a large file and there's no on-screen feedback today. Reusing the existing hashing/status indication (or a new state) would make it feel first-class.
  • Since many people run aMule as amuled + amulegui (or amulecmd/amuleweb/amuleapi), the feature really wants to work over that path too — the EC wiring you already have on your "not in this PR" list. Whenever that lands it'll need testing end-to-end on a real remote amuled ↔ amulegui setup, not just the monolithic build.

The .po/translation and tidy items are already on your own list, so I won't pile on. Thanks again — ping me once the small-file case is handled and I'll take another pass.

@danim7

danim7 commented Jul 10, 2026

Copy link
Copy Markdown
Author

Hi @got3nks thanks for the review, I will implement the changes you asked for, and nice catch on the small file case, I refactored my code too fast :D

If you have a minute, would you mind checking why the clang tidy Tier2 shows an error about undeclared wxEvent and other wx stuff in ThreadTasks.h ? I can fix it by including the <wx/event> header, but I don't understand why it fails in this job and not in the build jobs? Furthermore, the code I added in this file doesn't use those objects (not directly at least)... Maybe the Tier2 needs some tuning?

@got3nks

got3nks commented Jul 10, 2026

Copy link
Copy Markdown

The wx errors are a red herring — they're tagged [clang-diagnostic-error], and the Tier-2 job deliberately filters those out (grep -vE 'clang-diagnostic-' in clang-tidy.yml), since the build job is the real compile gate. They don't fail the check; they're just log noise. So no need to add <wx/event.h> (though it wouldn't hurt as hygiene).

What's actually failing the gate are two maintainability findings on your new lines. Both are quick:

1. [performance-unnecessary-value-param]PrintReport takes the CPath by value. Change just that parameter to a const reference, in both the declaration (ThreadTasks.h) and the definition (ThreadTasks.cpp):

// before
void PrintReport(const CPath fullPath, const bool checkedAICH);
// after
void PrintReport(const CPath &fullPath, const bool checkedAICH);

2. [modernize-use-emplace] — construct the pair in place instead of push_back(make_pair(...)):

// before
m_corruptedAICH.push_back(std::make_pair(part, corruptedAICHinThisPart));
// after
m_corruptedAICH.emplace_back(part, corruptedAICHinThisPart);

That's it — Tier-2 goes green after those two.

As for why the wx noise shows up only here: Tier-2 is diff-based, and your change touches the header ThreadTasks.h. Headers aren't in compile_commands.json, so when the diff tool parses the header to check your changed lines it lacks the full -I/include context a real .cpp TU has — so the header's pre-existing wxEvent/wxEventType uses (from CHashingEvent/CMediaProbeEvent, not your code) come back undeclared and it bails with "too many errors." The normal build only ever compiles that header inside ThreadTasks.cpp, where the transitive wx includes are present — which is why the build and Tier-1 stay quiet. Nothing to tune on the CI side.

@danim7
danim7 force-pushed the verify-local-data branch from f9affaa to 258466b Compare July 10, 2026 23:52
@danim7

danim7 commented Jul 10, 2026

Copy link
Copy Markdown
Author

Oh, thanks, I see. The output of the job is a little bit misleading: i was focusing on the lines tagged as "error" and just ignoring the "warning" lines on the first pass. Next time i will also check the "warning" from the start.

/home/runner/work/amule/amule/src/ThreadTasks.cpp:514:52: warning: the const qualified parameter 'fullPath' is copied for each invocation; consider making it a reference [performance-unnecessary-value-param]
  514 | void CVerifyLocalDataTask::PrintReport(const CPath fullPath, const bool checkedAICH)
      |                                                    ^
      |                                                   &
/home/runner/work/amule/amule/src/ThreadTasks.cpp:662:22: warning: use emplace_back instead of push_back [modernize-use-emplace]
  662 |                                         m_corruptedAICH.push_back(
      |                                                         ^~~~~~~~~~
      |                                                         emplace_back
  663 |                                                 std::make_pair(part, corruptedAICHinThisPart));
      |                                                 ~~~~~~~~~~~~~~                              ~

602 warnings generated.
1638 warnings generated.

error: too many errors emitted, stopping now [clang-diagnostic-error]
/home/runner/work/amule/amule/src/ThreadTasks.h:235:30: error: expected class name [clang-diagnostic-error]
  235 | class CHashingEvent : public wxEvent

Concerning the PR, I implemented your changes, but i will run a couple of extra tests during the weekend before marking it ready. Feel free to comment anything you see.

@got3nks

got3nks commented Jul 12, 2026

Copy link
Copy Markdown

Nice, all four are handled cleanly:

  • ✅ Small-file case — if (GetHashCount() == 0) { … break; } no longer falls through to GetPartHash.
  • PrintReport(const CPath &fullPath, …).
  • emplace_back(part, corruptedAICHinThisPart).
  • ✅ the wxFAIL is now a AddDebugLogLineN(logVerifyLocalData, …) + continue — exactly right, that state can be hit on a partial AICH tree and shouldn't abort a debug build.

Three tiny things for whenever you finalize (none blocking, and take them or leave them):

  • The // for testing: printf '\xFF' | dd … scaffolding comment near the top of CVerifyLocalDataTask can come out.
  • CVerifyLocalDataTask(const CMD4Hash&) could be explicit.
  • When you do the .po pass: the user-facing result lines ("Verify Local Data (%s): Result OK…", "ERRORS FOUND!…") want _() so they land in the catalog; the AddDebugLogLineN(logVerifyLocalData, …) debug lines are fine left untranslated.

Looks good otherwise.

@danim7
danim7 force-pushed the verify-local-data branch 2 times, most recently from 10ecacd to 7b2938f Compare July 12, 2026 21:18
@danim7
danim7 marked this pull request as ready for review July 12, 2026 21:44
@danim7

danim7 commented Jul 12, 2026

Copy link
Copy Markdown
Author

Thanks for the review, I implemented your requested changes and I did some extra tests on my side, it shall be good now.

@got3nks

got3nks commented Jul 12, 2026

Copy link
Copy Markdown

One thing we missed on the earlier passes, sorry — I think it's worth fixing before merge.

Entry() holds the raw CKnownFile * from FindKnownFileByID() and dereferences it throughout the hash (GetPartCount, GetPartSize, CreateHashFromFile, GetPartHash, GetAICHHashset…), which can run for minutes on a large file. This is the in-memory object, separate from the disk-file handling we already covered. Unsharing alone is safe — CSharedFileList::RemoveFile keeps the CKnownFile alive in knownfiles — but CKnownFileList::PruneDuplicates (which runs inside Save(), e.g. when another file finishes hashing) can delete it. It skips anything still in sharedfiles/downloadqueue, so a file that stays shared is safe; the gap is when the file is unshared during the verify and it's a size-duplicate and a known.met save fires in that window → the worker's pointer dangles (use-after-free). Narrow, but real.

Cleanest fix is the ownership discipline CHashingTask already uses: grab everything the loop needs in one up-front block, then never touch the shared object again. Handily CKnownFile::CreateHashFromFile() is static, and storedFile.SetFileSize() already gives you the part geometry — so on top of what you already copy for AICH, the loop just needs a snapshot of the per-part MD4 list:

CPath   fullPath;
uint64  fileSize = 0;
std::vector<CMD4Hash> partHashes;   // per-part MD4; empty for single-part files
CAICHHash aichMaster;
// + snapshot the AICH status too
{
    // (ideally under whatever guards knownfiles; either way the window
    //  is now just this small block instead of the whole hash)
    CKnownFile *kf = theApp->knownfiles->FindKnownFileByID(m_fileID);
    if (kf == nullptr || kf->IsPartFile()) return;   // log as you do now
    fullPath = kf->GetFilePath().JoinPaths(kf->GetFileName());
    fileSize = kf->GetFileSize();
    partHashes.reserve(kf->GetHashCount());
    for (size_t i = 0; i < kf->GetHashCount(); ++i)
        partHashes.push_back(kf->GetPartHash(i));
    aichMaster = kf->GetAICHHashset()->GetMasterHash();
}
// no kf below this point

CKnownFile storedFile;
storedFile.SetFileSize(fileSize);                       // -> GetPartCount()/GetPartSize()
storedFile.GetAICHHashset()->SetMasterHash(aichMaster, /*status*/);
// ...
for (uint16 part = 0; part < storedFile.GetPartCount() && !TestDestroy(); ++part) {
    const uint64 partLength = storedFile.GetPartSize(part);
    // ...
    CKnownFile::CreateHashFromFile(file, offset, partLength, &md4Hash, aichHash);  // static
    // ...
    if (partHashes.empty()) {                           // was GetHashCount() == 0
        if (md4Hash != m_fileID) m_corruptedMD4.push_back(part);
        break;
    }
    if (partHashes[part] != md4Hash) m_corruptedMD4.push_back(part);
}

Minor while you're in there: the "…not supported on PartFile" line uses CFormat("…") without _(), while the result lines are translated — worth making consistent.

Everything else still looks good — these are the only things holding it.

@danim7
danim7 marked this pull request as draft July 12, 2026 23:24
@danim7
danim7 force-pushed the verify-local-data branch from 7b2938f to 6da33e2 Compare July 13, 2026 17:18
@danim7
danim7 force-pushed the verify-local-data branch from 6da33e2 to 41e1490 Compare July 13, 2026 17:22
@danim7
danim7 marked this pull request as ready for review July 13, 2026 17:42
@danim7

danim7 commented Jul 13, 2026

Copy link
Copy Markdown
Author

You are absolutely right on the possibility of a dangling pointer for the known file in case of deletion. May you please check if it is ok now?

@got3nks
got3nks merged commit 3e6d468 into amule-org:master Jul 13, 2026
12 checks passed
@got3nks

got3nks commented Jul 13, 2026

Copy link
Copy Markdown

Everything looks good now — merged. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants