Repository navigation
[BUG] Docstring pre-clean breaks numpydoc integration due to removal of \n\n #6904
Description
Describe the bug
Let me introduce the issue by sharing that I am not entirely sure this should be a bug on Cython, on numpydoc or whether the docstrings should be adapted for the new reality, but I just wanted to raise a regression issue that was caused due to:
This was found on the Apache Arrow repository on the Pyarrow implementation:
When updating to Cython>=3.1 we found that our numpydocs integration is failing. Given a function like:
def equals(self, ReadOptions other):
"""
Parameters
----------
other : pyarrow.csv.ReadOptions
Returns
-------
bool
"""We got the following docstring with Cython<3.1:
PyDoc_STRVAR(__pyx_doc_7pyarrow_4_csv_11ReadOptions_6equals, "ReadOptions.equals(self, ReadOptions other)\n\n Parameters\n ----------\n other : pyarrow.csv.ReadOptions\n\n Returns\n -------\n bool\n ");
With newer Cython versions:
PyDoc_STRVAR(__pyx_doc_7pyarrow_4_csv_11ReadOptions_6equals, "ReadOptions.equals(self, ReadOptions other)\nParameters\n----------\nother : pyarrow.csv.ReadOptions\n\nReturns\n-------\nbool");
This is currently causing numpydoc to not interpret parameters correctly and interpret those as part of the summary because it's expecting a double \n:
(Pdb) self.doc['Parameters']
[]
(Pdb) self.doc['Summary']
['ReadOptions.equals(self, ReadOptions other)', 'Parameters', '----------', 'other : pyarrow.csv.ReadOptions']Hence breaking our pydoc checks.
Shouldn't the initial double \n\n between signature and parameters be respected?
Code to reproduce the behaviour:
# example codeExpected behaviour
No response
OS
Linux
Python version
3.9-3.13
Cython version
3.1
Additional context
No response
Activity
da-woods commentedon May 21, 2025
I think we probably should add in a new line after the signature.
If I understand correctly in the past, most people would have written their functions in the style:
which would end up "naturally" adding a blank line after the signature. Obviously that wouldn't apply to people who did
(although I doubt that was many people). We now can't distinguish the two cases. But I suspect a new line is right most of the time.
scoder commentedon May 22, 2025
Starting the docstring in the first line (without initial line break) is common and recommended:
https://peps.python.org/pep-0008/#documentation-strings
https://peps.python.org/pep-0257/#handling-docstring-indentation
We should make sure there are two line breaks after our auto-embedded signature.
Always add two blank lines after embedded signatures.
scoder commentedon May 23, 2025
After creating a PR, I started wondering if the reason for a single line break was to keep the signature and the original start of the documentation (the highly visible first line) close together. There's probably value in that, although tools are unlikely to interpret a mix of signature and text as intended. If users chose to let Cython generate a signature line, I think it's reasonable to show that line in the short docs and keep the text description apart, as the long version.
BTW, let me drop a note here about the different signature formats that the docstring embedding supports since Cython 3.0. You can now configure it to use Python annotation format (
embedsignature.format=python, orcorclinic), which might allow more tools to process it.Always add two blank lines after embedded signatures (GH-6910)
Always add two blank lines after embedded signatures (GH-6910)
GH-46516: [CI][Python] Force Cython>3.1.1 for docs builds (#46770)
apacheGH-46516: [CI][Python] Force Cython>3.1.1 for docs builds (apac…