Implement "Verify Local Data" function in Shared window (monolithic) - #381
Conversation
|
Thanks for this, @danim7 — really nice contribution, and I agree it's a strong foundation for the follow-ups you list (part-file verification, EC wiring, a corruption-recovery workflow, the periodic check from #166). The threading care stands out: copying the AICH hashset into a local A few things from a first pass: One blocker — files smaller than PARTSIZE (9.28 MiB). The MD4 section falls through for a single-part file that verifies OK: if (knownFile->GetHashCount() == 0 && md4Hash != m_fileID) {
m_corruptedMD4.push_back(part);
break;
}
if (knownFile->GetPartHash(part) != md4Hash) // reached even when the file is OK
m_corruptedMD4.push_back(part);
One smaller thing:
Two thoughts on where this goes next:
The |
|
Hi @got3nks thanks for the review, I will implement the changes you asked for, and nice catch on the small file case, I refactored my code too fast :D If you have a minute, would you mind checking why the clang tidy Tier2 shows an error about undeclared wxEvent and other wx stuff in ThreadTasks.h ? I can fix it by including the <wx/event> header, but I don't understand why it fails in this job and not in the build jobs? Furthermore, the code I added in this file doesn't use those objects (not directly at least)... Maybe the Tier2 needs some tuning? |
|
The wx errors are a red herring — they're tagged What's actually failing the gate are two maintainability findings on your new lines. Both are quick: 1. // before
void PrintReport(const CPath fullPath, const bool checkedAICH);
// after
void PrintReport(const CPath &fullPath, const bool checkedAICH);2. // before
m_corruptedAICH.push_back(std::make_pair(part, corruptedAICHinThisPart));
// after
m_corruptedAICH.emplace_back(part, corruptedAICHinThisPart);That's it — Tier-2 goes green after those two. As for why the wx noise shows up only here: Tier-2 is diff-based, and your change touches the header |
f9affaa to
258466b
Compare
|
Oh, thanks, I see. The output of the job is a little bit misleading: i was focusing on the lines tagged as "error" and just ignoring the "warning" lines on the first pass. Next time i will also check the "warning" from the start. Concerning the PR, I implemented your changes, but i will run a couple of extra tests during the weekend before marking it ready. Feel free to comment anything you see. |
|
Nice, all four are handled cleanly:
Three tiny things for whenever you finalize (none blocking, and take them or leave them):
Looks good otherwise. |
10ecacd to
7b2938f
Compare
|
Thanks for the review, I implemented your requested changes and I did some extra tests on my side, it shall be good now. |
|
One thing we missed on the earlier passes, sorry — I think it's worth fixing before merge.
Cleanest fix is the ownership discipline CPath fullPath;
uint64 fileSize = 0;
std::vector<CMD4Hash> partHashes; // per-part MD4; empty for single-part files
CAICHHash aichMaster;
// + snapshot the AICH status too
{
// (ideally under whatever guards knownfiles; either way the window
// is now just this small block instead of the whole hash)
CKnownFile *kf = theApp->knownfiles->FindKnownFileByID(m_fileID);
if (kf == nullptr || kf->IsPartFile()) return; // log as you do now
fullPath = kf->GetFilePath().JoinPaths(kf->GetFileName());
fileSize = kf->GetFileSize();
partHashes.reserve(kf->GetHashCount());
for (size_t i = 0; i < kf->GetHashCount(); ++i)
partHashes.push_back(kf->GetPartHash(i));
aichMaster = kf->GetAICHHashset()->GetMasterHash();
}
// no kf below this point
CKnownFile storedFile;
storedFile.SetFileSize(fileSize); // -> GetPartCount()/GetPartSize()
storedFile.GetAICHHashset()->SetMasterHash(aichMaster, /*status*/);
// ...
for (uint16 part = 0; part < storedFile.GetPartCount() && !TestDestroy(); ++part) {
const uint64 partLength = storedFile.GetPartSize(part);
// ...
CKnownFile::CreateHashFromFile(file, offset, partLength, &md4Hash, aichHash); // static
// ...
if (partHashes.empty()) { // was GetHashCount() == 0
if (md4Hash != m_fileID) m_corruptedMD4.push_back(part);
break;
}
if (partHashes[part] != md4Hash) m_corruptedMD4.push_back(part);
}Minor while you're in there: the "…not supported on PartFile" line uses Everything else still looks good — these are the only things holding it. |
7b2938f to
6da33e2
Compare
6da33e2 to
41e1490
Compare
|
You are absolutely right on the possibility of a dangling pointer for the known file in case of deletion. May you please check if it is ok now? |
|
Everything looks good now — merged. Thanks! |
Opening as draft, work in progress
Intro
To share a file in ed2k, the file is hashed with MD4 and SHA1 (AICH) algorithms to uniquely identify it and help fix corruption during downloads. These hashes are stored in the known.met and known2_64.met files.
This PR introduces a new functionality to make use of the .met files as checksum files. This allows users to check the integrity of known files against the stored hashes in the .met files. It is accesible via right-click on the Shared window (monolithic-only in this PR), and the result of the check is printed to the log.
Tests
fallocate -l 1g ~/.aMule/Incoming/1g, let amule discover and hash it for the first time, then re-check it via the new option menu, file is OK in the log:Verify Local Data (MD4 & AICH): Result OK for /home/test/.aMule/Incoming/1gVerify Local Data (MD4 & AICH): ERRORS FOUND! /home/test/.aMule/Incoming/1g Failed blocks: MD4: 12,101. AICH: 12: (33,39), 101: (27)Launch checks on multiple files at the same time, and multiple checks on the same file before it finishes: the checks are queued by the thread scheduler, and executed one at a time. The duplicated checks on the same file are discarded and only one runs per file.
Create a new shared file and let amule hash it. Close amule. Now, we will corrupt the known.met file. When we launch amule, it detects the .met file is corrupt, prints a log warning and self-heals! When we check the file, the result is OK, if we re-open amule, it no longer show the known file list as corrupt
printf '\xFF' | dd of=known.met bs=1 seek=100 conv=notruncFailed to load entry in known file list, file may be corruptprintf '\xFF' | dd of=known2_64.met bs=1 seek=117000 conv=notruncPending for this PR
Not in this PR
I prefer to make multiple small-step PRs, instead of one single big-step PR.
This PR creates the first check available for users, and also prepares the infrastucture for potential, future developments (no guarantees about (if/when) would do them):