Skip to content

fix(jats): keep the license and funding statements of the article - #4866

Merged
ceberam merged 3 commits into
docling-project:mainfrom
Thanatos9404:fix-jats-license-funding
Nov 8, 2026
Merged

ceberam merged 3 commits into
docling-project:mainfrom
Thanatos9404:fix-jats-license-funding

Conversation

@Thanatos9404

@Thanatos9404 Thanatos9404 commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

From <article-meta>, the JATS backend reads only the title, the authors with their affiliations, and the abstract (_parse_metadata). The copyright and license statements in <permissions> and the funding statements in <funding-group> are never visited, so their text and their links are missing from the document.

Issue resolved by this Pull Request:
Resolves #4794

Where the statements go

The issue asks where these statements belong. This PR puts them in the furniture content layer:

  • docling-core defines the body layer as the main content and the furniture layer for content around it, such as page headers and footers. A license or a funding statement describes the article, but it is not part of its reading text.
  • Other backends already use furniture for this kind of content. The HTML backend adds the page <title> as furniture "since it is part of the document metadata", and the content of <footer>, where sites put copyright and license notices, also goes to furniture. The Word and Pages backends put headers, footers and footnotes there, and the ODF backend its footnotes; the iWork backend gives the reason: "available to callers that ask for it but stay out of the reading order by default".
  • Improve JATS parsing with nested lists, inline formulas, ordered lists, content layers #1067 lists "distribute content in body or furniture content layers" as a goal for this backend.
  • The default Markdown, text and chunking output of an article does not change. The statements are in export_to_dict(), in doc.texts and in any export that includes ContentLayer.FURNITURE.

If you prefer them in the body, for example after the abstract under their own headings, that is a small change and I can update the PR.

Change

  • _parse_statements collects permissions/copyright-statement, permissions/license/license-p and funding-group/funding-statement from the article metadata in source order. Like abstract paragraphs, they are kept as nodes and walked at emit time.
  • _add_statements walks each statement with _walk_linear, so <ext-link> targets and styling are kept as in body paragraphs, and emits it under the title with _emit_inline in the furniture layer. A statement with several runs gets an inline group, also in the furniture layer.
  • _emit_inline takes an optional content_layer. The existing callers do not pass it and do not change.
  • The statements are added after the back matter and the sub-articles. As furniture, their position matters only when a caller includes that layer, and then they follow the article as article information. Adding them last also keeps every existing item at its index, so the reference data only grows.

Scope: I included <copyright-statement> because it is the other statement in <permissions> and a license is attributed with it. The structured funding data (<award-group>, <funding-source>, <award-id>), the xlink:href of <license> and <ali:license_ref> are not converted. They are identifiers rather than statements, and DoclingDocument has no document-level metadata field for them. In all four fixtures with a license, the license URL is also the target of an <ext-link> in the license text, so the URL is kept. One known limit: block children of <license-p> (a list, formula, figure or table) are still emitted as body content; none of the fixtures has one.

Behavior on the input from the issue:

  • Before (main): CHARLIE-LICENCE, CHARLIE-FUNDING and the license URL are not in export_to_dict().
  • After: doc.texts gets CHARLIE-LICENCE terms of the, licence (hyperlink https://example.org/c3-licence), . and CHARLIE-FUNDING statement., all in the furniture layer. export_to_markdown() is unchanged. With included_content_layers={ContentLayer.BODY, ContentLayer.FURNITURE}, the two statements follow the body.

Reference data

Only the .json files change, and only by additions (503 lines added, none removed). The .md and .itxt files do not change, which confirms that the body is the same.

  • elife-56337: the license (with its CC0 link) and the funding statement.
  • pntd.0008301: the license (CC0 link) and the funding statement (link to usaid.gov).
  • pone.0234687: the license (CC0 link) and the funding statement (links to the three funders).
  • ptag100: the copyright statement and the two license paragraphs (CC BY link, and SCOAP<sup>3</sup> as a superscript run). Its <funding-group> has only an <award-group>, so no funding statement.
  • pmc2231364: no change (no <permissions> or <funding-group>).

This PR and #4813 regenerate the same JSON reference files (pntd.0008301, pone.0234687, ptag100). Whichever merges second needs a rebase and regenerated reference data. I will do that.

Checklist:

  • Documentation has been updated, if necessary.
  • Examples have been added, if necessary.
  • Tests have been added, if necessary.

Review evidence:

New tests in tests/test_backend_jats_metadata.py. tests/test_backend_jats.py has 1494 lines and the max-lines hook allows 1500, so the tests are in a new module that reuses get_converter, as tests/test_backend_pptx_charts.py does.

  • test_jats_license_and_funding_statements_are_furniture: the input from the issue. Checks the furniture runs and the license link, that the default Markdown has the abstract but not the statements, and that the statements follow the body when furniture is included.
  • test_jats_copyright_and_each_license_paragraph_are_kept_in_order: a copyright statement, two license paragraphs (one with <sup>) and two funding statements, each kept as its own item in source order; <copyright-year> and <award-group> are not converted.

Both fail on main (f690a52) and pass with this change.

Commands and results (Windows 11, Python 3.10.20, docling-core 2.100.0, environment from uv sync --frozen --group dev --all-packages --no-group docs --no-group examples --extra format-xml-jats):

Command Result
uv run pytest tests/test_backend_jats_metadata.py with the backend of main 2 failed
uv run pytest tests/test_backend_jats_metadata.py tests/test_backend_jats.py tests/test_input_doc.py tests/test_backend_optional_dependencies.py tests/test_data_gen_flag.py 155 passed
DOCLING_GEN_TEST_DATA=1 uv run pytest tests/test_backend_jats.py -k e2e reference data regenerated, reviewed as described above
uv run prek run --files <changed files> (what make validate runs) all hooks passed: trailing whitespace, end of files, merge conflicts, large files, license header, Ruff, Ruff format, ty, Tach, Tach module coverage, max lines
uv run ty check docling/backend/xml/jats_backend.py 0 errors; the same 5 warnings as on main

Not run: the full test suite and the ML suites. On Windows, the hooks that call python3 ran through a python3 alias of the project interpreter.

@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @Thanatos9404, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 All 2 merge protections satisfied — ready to merge.

Show 2 satisfied protections

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

🟢 Require two reviewer for test updates

When test data is updated, we require two reviewers

  • #approved-reviews-by >= 2

@github-actions github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) bug Something isn't working xml issue related to supported schema-specific XML formats labels Oct 16, 2026
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

AI triage (advisory)

Checked commit 5d49f203ea. The analysis read the diff only. It did not run code from this pull request.

The JATS backend now reads copyright, license and funding statements from the article metadata. It adds them to the document in the furniture content layer, with their links.

CI safety: no concern found. A maintainer must still approve CI.

2 related pull request(s) in the same code
PR State Signal Assessment
#5216 open changes the same code related: This PR also changes the JATS metadata code. It keeps author-notes as footnotes. This PR keeps license and funding statements. The two PRs fix different problems and edit nearby lines.
#4813 merged referenced as #4813 related: This PR is merged and fixes links in mixed-citation. It does not handle license or funding statements. It also has the same author and uses the same inline-run helpers.

Reference data changes

Change Files
structure 4
  • structure tests/data/jats/groundtruth/elife-56337.nxml.json: text: 216 → 220
  • structure tests/data/jats/groundtruth/pntd.0008301.nxml.json: text: 168 → 174
  • structure tests/data/jats/groundtruth/pone.0234687.nxml.json: text: 644 → 654
  • structure tests/data/jats/groundtruth/ptag100.xml.json: text: 241 → 247

Assessment: expected. Four JATS reference files gain 4 to 10 text items each. This matches the new furniture items for license, copyright and funding statements. The PR text does not describe other changes.

The ai:* labels are advisory. They do not approve CI and they do not add tests:full. A new push runs this check again.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI first review (advisory): no blocker found

This review covers the full diff at 8c60271a48. It read the code and did not run it.

I found no defect in the change. The new statements are walked with _walk_linear and emitted with ContentLayer.FURNITURE. They do not enter the default Markdown or text export, and the new tests check this. The reviewer should confirm that furniture is the layer the maintainers want for license and funding text, because the PR author offers to move it to the body.

A maintainer makes the merge decision. This review never approves the pull request.

@github-actions github-actions Bot added the ai:review-lgtm AI first review found no blocker or major issue (advisory) label Oct 16, 2026
@codecov

codecov Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.61538% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
docling/backend/xml/jats_backend.py 84.61% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@github-actions github-actions Bot removed ai:ci-safe AI triage found no CI-safety concern (advisory) ai:review-lgtm AI first review found no blocker or major issue (advisory) labels Oct 19, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI first review (advisory): no blocker found

This review covers the changes since 8c60271a48 at 01cacfd6d4. It read the code and did not run it.

The new changes are a merge of the base branch. They do not touch the JATS backend, its tests, or its reference data, so the earlier review of the JATS code still holds.

A maintainer makes the merge decision. This review never approves the pull request.

@github-actions github-actions Bot added the ai:review-lgtm AI first review found no blocker or major issue (advisory) label Oct 19, 2026
dolfim-ibm
dolfim-ibm previously approved these changes Oct 21, 2026

@dolfim-ibm dolfim-ibm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

ceberam
ceberam previously approved these changes Oct 23, 2026

@ceberam ceberam left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Thanatos9404 for this improvement to JATS documents.

I left a non-blocking design question.

Comment thread docling/backend/xml/jats_backend.py Resolved
@Thanatos9404
Thanatos9404 dismissed stale reviews from ceberam and dolfim-ibm via 3baef17 October 29, 2026 12:46
@github-actions github-actions Bot added ai:ci-needs-care AI triage: check CI-safety findings before you approve CI and removed ai:review-lgtm AI first review found no blocker or major issue (advisory) labels Oct 29, 2026
dolfim-ibm
dolfim-ibm previously approved these changes Oct 29, 2026
@github-actions github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) and removed ai:ci-needs-care AI triage: check CI-safety findings before you approve CI labels Nov 5, 2026
@ceberam

ceberam commented Nov 6, 2026

Copy link
Copy Markdown
Member

@Thanatos9404 We've done quite a few significant PRs lately. Could you please resolve the conflicts again?

@github-actions github-actions Bot mentioned this pull request Nov 6, 2026
3 tasks done
@PeterStaar-IBM PeterStaar-IBM added the jats issue related to JATS XML backend label Nov 6, 2026
Thanatos9404 and others added 2 commits November 6, 2026 21:50
The backend read only the title, the authors and the abstract from
<article-meta>. The copyright and license statements in <permissions>
and the funding statements in <funding-group> were dropped, together
with their links.

Walk each <copyright-statement>, <license-p> and <funding-statement>
with _walk_linear, so styling and ext-link targets are kept, and add it
to the furniture layer. Other backends put content that is outside the
reading order there, such as page headers and footers, so the default
Markdown and text exports do not change. The statements are added after
the back matter, so the existing items keep their positions.

Regenerate the JSON ground truth. The Markdown and indented text ground
truth does not change.

Resolves docling-project#4794

Signed-off-by: Yashvardhan Thanvi <[email protected]>
Signed-off-by: Cesar Berrospi Ramis <[email protected]>
Co-Authored-By: Cesar Berrospi Ramis <[email protected]>
Main now keeps the boundary whitespace of JATS inline runs (the
InlineGroup whitespace contract), so the text run before the ext-link
in the licence statement ends with a space.

Signed-off-by: Yashvardhan Thanvi <[email protected]>
…_jats

Merge the two tests from test_backend_jats_metadata.py into
test_backend_jats.py and delete the old file. Reuse
convert_jats_article_meta, which now takes an optional body, instead
of a separate helper.

Signed-off-by: Cesar Berrospi Ramis <[email protected]>
@ceberam
ceberam force-pushed the fix-jats-license-funding branch from 4a9afd6 to 5d49f20 Compare November 6, 2026 21:14
@github-actions github-actions Bot added ai:ci-safe AI triage found no CI-safety concern (advisory) and removed ai:ci-safe AI triage found no CI-safety concern (advisory) labels Nov 6, 2026

@ceberam ceberam left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Thanatos9404 Since another PR #5216 depends on this one, I moved ahead and rebased and resolved the conflicts.
I have also moved the two tests into the main test file for JATS backend, since a new file was not necessary. This allows us to reuse an existing helper.
Thanks for your contribution 🚀

@PeterStaar-IBM PeterStaar-IBM left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@ceberam
ceberam merged commit 896f2fd into docling-project:main Nov 8, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai:ci-safe AI triage found no CI-safety concern (advisory) bug Something isn't working jats issue related to JATS XML backend xml issue related to supported schema-specific XML formats

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JATS: licence and funding statements in <article-meta> are not converted

4 participants