Skip to content

fix: update the data file hash when a parallel data file is saved again - #2339

Merged
nedbat merged 1 commit into
coveragepy:mainfrom
ericdelorefice:fix/rehash-data-file-on-save
Oct 21, 2026
Merged

nedbat merged 1 commit into
coveragepy:mainfrom
ericdelorefice:fix/rehash-data-file-on-save

Conversation

@ericdelorefice

Copy link
Copy Markdown
Contributor

Fixes #2338. Thanks @nedbat for the quick go-ahead, and @DSeaStar for reproducing it on Windows 11 / CPython 3.14 and checking the approach (strip the old hash before adding the new one, and rename only when the name would change).

Problem

Since 7.13.1 a parallel data file carries a hash of its data in its name (.H<hash>h), which lets coverage combine skip duplicate files quickly. CoverageData.write() adds that hash only on the first write. If the process saves again after recording more data, the new data goes into the same file and the name keeps the first save's hash.

Two processes that had the same data at their first save then end up with the same hash in their names, even though their later data differs. combine_parallel_data() treats the second one as a duplicate, skips it, and deletes it, so the data it recorded after the first save is lost. Nothing warns; the combine message says "Combined 1 file, skipped 1".

save() can be called more than once in a process (test_start_save_nostop saves in the middle of a run), so a parallel run that saves at a checkpoint and then keeps measuring can hit this.

Reproduction

Two separate processes, each Coverage(data_suffix=True, source=["."]): start, import startup.py (the same in both), save(), then import work_a.py in one and work_b.py in the other, stop, save(). Then coverage combine and coverage report:

data files before combine:
.coverage.<host>.pid77877.XBHz66Jx.HUNoVFydUGSh
.coverage.<host>.pid77878.XHViNTox.HUNoVFydUGSh
Combined 1 file, skipped 1

Name         Stmts   Miss  Cover   Missing
------------------------------------------
startup.py       2      0   100%
work_a.py        3      0   100%
work_b.py        3      3     0%   1-3

work_b.py ran but is reported as never run, and its data file is gone. Same result on 7.16.2 from PyPI and on main (99024b5). With 7.13.0, before the hash went into the name, both files are combined and work_b.py is 100%. Nothing is mocked or injected: real processes, real data files, the real combine command.

Change

In CoverageData.write(), when a parallel data file already has a hash in its name and the hash of its data has changed since, rename it again with the new hash and update the hash row in meta (INSERT OR REPLACE instead of INSERT OR IGNORE). The first write is unchanged, a write with no new data does not rename, and files with the same data still get the same hash, so duplicates are still skipped. A forked child, which gets a new file name without a hash, is left as it was.

Intended behaviour change: a parallel data file can be renamed on each save() whose data changed, not only on the first.

Testing

  • test_true_suffix_written_twice (test_data.py): two files share their first write, then differ; both must be combined. Fails on main, passes with the fix.
  • test_combining_after_saving_twice (test_api.py): the same through the Coverage API, saving while still collecting. Fails on main, passes with the fix.
  • Controls that pass on main and with the fix: test_true_suffix_written_twice_same_data (identical files written twice are still skipped as duplicates) and test_true_suffix_written_again_without_new_data (no rename without new data).
  • Full suite with each core, main vs this branch: ctrace 1659 -> 1663 passed, pytrace 1623 -> 1627, sysmon 1604 -> 1608 (the four new tests), no failures. ruff format --check, pylint and mypy --strict are clean.
  • Added a CHANGES.rst entry under Unreleased.

A parallel data file got the hash of its data in its name on the first
save, and later saves added data without changing the name. Combining could
then skip the file as a duplicate of another one that had the same data at
the first save, and delete it.
@nedbat

nedbat commented Oct 21, 2026

Copy link
Copy Markdown
Member

And thanks for the quick fix!

@nedbat
nedbat merged commit 6033660 into coveragepy:main Oct 21, 2026
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Combining parallel data files loses a process's data when it saved more than once

2 participants