feat(#10794): adds replication failure log by dianabarsan · Pull Request #10823 · medic/cht-core

dianabarsan · 2026-04-07T12:59:14Z

Description

Moves all replication related api-services into a new replication dedicated folder. The list became huge!
Add replication failure logging to /api/v1/replication/get-ids, matching the existing setup on the initial-replication route
Each failure records timestamp, status code, duration, request ID, roles, and subjects count (if available)
Failures are stored per-user per-month in the logs DB, capped at 50 detailed entries with a running total count
Captures both server errors and client cancellations
Add e2e tests for failure logging, accumulation, monthly separation, per-user separation, and the 50-entry cap
Adds api endpoint to get replication failures for a given month. Defaults to the current month.

closes #10794

AI Disclosure

This is developed with the help of Claude. I have made all design decisions, reviewed and corrected all code.

Code review checklist

UI/UX backwards compatible: Test it works for the new design (enabled by default). And test it works in the old design, enable can_view_old_navigation permission to see the old design. Test it has appropriate design for RTL languages.
Readable: Concise, well named, follows the style guide
Documented: Configuration and user documentation on cht-docs
Tested: Unit and/or e2e where appropriate
Internationalised: All user facing text
Backwards compatible: Works with existing data and configuration or includes a migration. Any breaking changes documented in the release notes.
AI disclosure: Please disclose use of AI per the guidelines.

Compose URLs

If Build CI hasn't passed, these may 404:

License

The software is provided under AGPL-3.0. Contributions to this project are accepted under the same license.

Signed-off-by: Diana Barsan <[email protected]>

# Conflicts: # api/tests/mocha/controllers/replication-limit-log.spec.js # api/tests/mocha/controllers/users.spec.js

Signed-off-by: Diana Barsan <[email protected]>

… log controller Signed-off-by: Diana Barsan <[email protected]>

Signed-off-by: Diana Barsan <[email protected]>

# Conflicts: # api/src/middleware/authorization.js # api/src/services/replication/bulk-docs.js # api/tests/mocha/controllers/users.spec.js # api/tests/mocha/middleware/authorization.spec.js # tests/integration/api/controllers/replication.spec.js

Signed-off-by: Diana Barsan <[email protected]>

… for replication failure log service Signed-off-by: Diana Barsan <[email protected]>

# Conflicts: # tests/integration/api/controllers/replication.spec.js

…ails

Signed-off-by: Diana Barsan <[email protected]>

…ctionality Signed-off-by: Diana Barsan <[email protected]>

… comments for nouveau behavior in k3d Signed-off-by: Diana Barsan <[email protected]>

Signed-off-by: Diana Barsan <[email protected]>

dianabarsan · 2026-04-22T16:39:09Z

I know your review queue is long, but i have to ask for another one. I appreciate the time you spend on this review.

jkuester

Very nice! I left a bunch of random thoughts inline.

Have you thought at all about if we should report a total value of these failures via the /monitoring api? Maybe even just a count of the distinct users that have replication failures.

mrjones-plip · 2026-04-23T23:18:11Z

if we should report a total value of these failures via the /monitoring api

a million times yes!! But also, I can wait 'til another iteration if it delays this PR/is too much feature creep.

Signed-off-by: Diana Barsan <[email protected]>

…ails

Signed-off-by: Diana Barsan <[email protected]>

…ails

…d enhance pagination with cursor validation Signed-off-by: Diana Barsan <[email protected]>

Signed-off-by: Diana Barsan <[email protected]>

dianabarsan · 2026-05-06T12:52:35Z

Hi @jkuester

A bunch of changes occured, mostly as part of your review feedback:

API
- month -> reporting_period
- cht-datasource-like cursor based pagination and response style (to keep PR small i didn't make changes to datasource, but added a small lib into api)
- user is now a query param and I improved performance for getting all user failed periods
Doc format
- timestamp -> date to match connected user logs
- added docs_count and unpurged_docs_count along subjects_count, to show how far the request progressed. all these default to unknown.
Routing
- I finally understood the confusion about endpoints, because there were two :D. initial-replication/get-ids was added in 4.2 when we did the initial replication refactor, and in 4.4 became deprecated with replication/get-ids added with the Nairobi Protocol. We left initial-replication/get-ids in case older clients tried to replicate and then in 5.0 we forgot to remove the endpoint. Because updating from 4.2 to 5.x requires a step to 4.5 (?), it is now completely safe to be removed, so I did.

I hope I didn't miss anything. Review at your discretion. Appreciate your time and thoughts!

jkuester

Couple minor suggestions/thoughts, but this is good to go! 👍 LGTM

jkuester · 2026-05-06T18:55:29Z

-app.get(
-  '/api/v1/initial-replication/get-ids',
-  authorization.handleAuthErrors,
-  authorization.onlineUserPassThrough,
-  replication.getDocIds,
-);


jkuester · 2026-05-06T19:07:02Z

+   *     summary: Get replication failure logs
+   *     operationId: v1ReplicationFailureLogsGet
+   *     description: >
+   *       Returns a paginated page of full replication failure log documents. Optionally filter by `user`


Suggested change

* Returns a paginated page of full replication failure log documents. Optionally filter by `user`

* Returns a page of replication failure log documents. Optionally filter by `user`

jkuester · 2026-05-06T19:36:08Z

+  // Counts are only set on userCtx as each phase of the request completes. When a count is missing
+  // we record 'unknown' instead of omitting the key, so a stable shape on the failure entry tells you
+  // (by which counters are 'unknown') how far the request progressed before failing.


Don't have to change anything here, but setting unknown in these cases does not seem to make it any easier for consumer code to parse the data from the doc when compared to setting:

"",

null

undefined

I kind of understand wanting to keep the object shape "stable", but I am not really sure how much value there is in that when we don't have any meaningful default value to use... 🤷

jkuester · 2026-05-06T19:51:40Z

+  const oldestPeriod = await findOldestReportingPeriod();
+  if (!oldestPeriod) {
+    return { data: [], cursor: null };
+  }
+
+  // Username is the doc-id suffix, preventing prefix-scan; bulk-fetch the user's candidate keys (max 60).
+  const candidateKeys = enumerateReportingPeriods(oldestPeriod).map(period => getDocId(user, period));
+  const result = await db.medicLogs.allDocs({ keys: candidateKeys, include_docs: true });


Is there a strong reason we cannot just index these docs by user? This is kind of a crazy amount of logic to have/maintain (even if the performance is okay). It is fancy that all of this actually works, but at the same time it really smells like a premature optimization.

jkuester · 2026-05-06T20:03:22Z

+  const getUserFailureLog = async (username) => {
+    const response = await getFailureLogs({ user: username });
+    const currentPeriodId = getFailureLogId(username);
+    return response.data.find(log => log._id === currentPeriodId) || response.data[0] || null;


Minor, but the currentPeriodId logic here feels like surprizing behavior that might be confusing to debug later. (When we call this function are we expecting a doc for the current period or are we expecting to just get back the first one? How can someone reading the code know?) Would rather just have two dedicated functions...

dianabarsan added 17 commits April 1, 2026 17:27

move stuff

51c3cb6

Signed-off-by: Diana Barsan <[email protected]>

log replication failures for users without medic user-settings docs

75b7133

Signed-off-by: Diana Barsan <[email protected]>

Merge branch 'master' into 10794-replication-fails

3d57bfb

# Conflicts: # api/tests/mocha/controllers/replication-limit-log.spec.js # api/tests/mocha/controllers/users.spec.js

improve testing by stopping nouveau

400ee08

Signed-off-by: Diana Barsan <[email protected]>

move replication service tests to dedicated directory and add failure…

012113d

… log controller Signed-off-by: Diana Barsan <[email protected]>

add tests for capturing replication failures in authorization middleware

a75e13d

Signed-off-by: Diana Barsan <[email protected]>

return purge log as part of wait for purge

dcfba11

Signed-off-by: Diana Barsan <[email protected]>

fix bad merge

ac2b567

Signed-off-by: Diana Barsan <[email protected]>

add global chai.expect, simplify async test assertions, and add tests…

f238a5c

… for replication failure log service Signed-off-by: Diana Barsan <[email protected]>

Merge branch 'master' into 10794-replication-fails

95dcd21

# Conflicts: # tests/integration/api/controllers/replication.spec.js

Merge remote-tracking branch 'origin/master' into 10794-replication-f…

45eab05

…ails

rename replication failure logs route and add openapi documentation

ad3e133

Signed-off-by: Diana Barsan <[email protected]>

add replication failure logs tests and enhance service/controller fun…

7b3702b

…ctionality Signed-off-by: Diana Barsan <[email protected]>

add tests for additional replication failure log scenarios and update…

85761b6

… comments for nouveau behavior in k3d Signed-off-by: Diana Barsan <[email protected]>

add user-specific replication failure logs endpoint and update tests

f5b211f

Signed-off-by: Diana Barsan <[email protected]>

make calls at the same time

31f2612

Signed-off-by: Diana Barsan <[email protected]>

dianabarsan requested a review from jkuester April 21, 2026 10:05

jkuester requested changes Apr 23, 2026

View reviewed changes

dianabarsan added 9 commits April 24, 2026 16:15

make calls at the same time

f37152a

Signed-off-by: Diana Barsan <[email protected]>

Merge remote-tracking branch 'origin/master' into 10794-replication-f…

cf65a50

…ails

code review feedback

9e54020

Signed-off-by: Diana Barsan <[email protected]>

add cleanup. global cleanup only works in wdio :(

4525a34

Signed-off-by: Diana Barsan <[email protected]>

add cleanup. global cleanup only works in wdio :(

2773868

Signed-off-by: Diana Barsan <[email protected]>

Merge remote-tracking branch 'origin/master' into 10794-replication-f…

0e97be7

…ails

update replication failure log tests to use consistent user naming an…

93a7e53

…d enhance pagination with cursor validation Signed-off-by: Diana Barsan <[email protected]>

don't use initial-replication/get-ids path in tests

4e9d48f

Signed-off-by: Diana Barsan <[email protected]>

appease the sonar gods

14f44d9

Signed-off-by: Diana Barsan <[email protected]>

dianabarsan requested a review from jkuester May 6, 2026 12:52

jkuester approved these changes May 6, 2026

View reviewed changes

	* Returns a paginated page of full replication failure log documents. Optionally filter by `user`
	* Returns a page of replication failure log documents. Optionally filter by `user`

Uh oh!

Conversation

dianabarsan commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

AI Disclosure

Code review checklist

Compose URLs

License

Uh oh!

dianabarsan commented Apr 22, 2026

Uh oh!

jkuester left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mrjones-plip commented Apr 23, 2026

Uh oh!

dianabarsan commented May 6, 2026

Uh oh!

jkuester left a comment

Choose a reason for hiding this comment

Uh oh!

jkuester May 6, 2026

Choose a reason for hiding this comment

Uh oh!

jkuester May 6, 2026

Choose a reason for hiding this comment

Uh oh!

jkuester May 6, 2026

Choose a reason for hiding this comment

Uh oh!

jkuester May 6, 2026

Choose a reason for hiding this comment

Uh oh!

jkuester May 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dianabarsan commented Apr 7, 2026 •

edited

Loading