Repository navigation
Meaning and clarification of PyBuffer_Release() #83652
Description
| BPO | 39471 |
|---|---|
| Nosy | @mdickinson, @seberg |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2020-04-22.17:40:06.563>
created_at = <Date 2020-01-27.22:14:19.077>
labels = ['expert-C-API', '3.8', 'type-bug', '3.7', '3.9']
title = 'Meaning and clarification of PyBuffer_Release()'
updated_at = <Date 2020-04-22.17:52:40.233>
user = 'https://github.com/seberg'bugs.python.org fields:
activity = <Date 2020-04-22.17:52:40.233>
actor = 'mark.dickinson'
assignee = 'none'
closed = True
closed_date = <Date 2020-04-22.17:40:06.563>
closer = 'seberg'
components = ['C API']
creation = <Date 2020-01-27.22:14:19.077>
creator = 'seberg'
dependencies = []
files = []
hgrepos = []
issue_num = 39471
keywords = []
message_count = 4.0
messages = ['360809', '360814', '360818', '367032']
nosy_count = 3.0
nosy_names = ['mark.dickinson', 'seberg', 'Eric Wieser']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39471'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']
Activity
seberg commentedon Jan 27, 2020
The current documentation of
PyBuffer_Release()and the PEP is a bit fuzzy about what the function can and cannot do.When an object exposes the buffer interface, I believe it should always return a
view(in NumPy speak) of its own data, i.e. the data exposed by the object is also owned by it directly.On the other hand the buffer view itself has fields such as
strides, etc. which may need allocating.In other words, I think
PyBuffer_Release()should be documented to deallocate/invalidate thePy_buffer. But, it must not invalidate the actual memory it points to. If I copy all information out of thePy_bufferand then free it, the copy must still be valid.I think this is the intention, but it is not spelled out clear enough, it is also the reason for the behaviour of the "#s", etc. keyword argument parsers failing due to the code:
which in turn currently means NumPy decides to _not_ implement bf_releasebuffer at all (leading to very ugly work arounds).
I am happy to make a PR, if we can get to a point where everyone is absolutely certain that the above interpretation was always correct, we could clean up a lot of code inside NumPy as well!
seberg commentedon Jan 27, 2020
Hmmm, it seems I had missed this chunk of PEP-3118 before:
Which reads like the opposite of what I would like to see, unfortunately. Which I guess means that the parsing issue should be addressed differently. Opinions still welcome, though.
seberg commentedon Jan 27, 2020
I went through Python,
arrayseems to not break the logic. pickling has a comment which specifically wants to run into the argument parsing corner case above (I am not sure that it is really important). However,[Modules/_testbuffer.c](https://github.com/python/cpython/blob/main/Modules/_testbuffer.c)(which is just test) actually does allocate new memory for the buffer!I can be convinced that this is how it is supposed to be, but right now I am still not quite. The main idea is efficient no-copy data sharing. It is always possible to create a
to_memoryview()method if a realloc was desired/necessary...seberg commentedon Apr 22, 2020
Ok, I will just close it. It is painfully clear that e.g.
mmapuses it this way to prohibit closing, and alsomemoryviewhas all the machinery necessary to do counting of how many exports, etc. exists.I admit, this still rubs me the wrong way, and I think it means that we may need to check
bf_releasebuffer != NULLalso in NumPy.We still have the issue of not being able to use
releasebuffereasily in NumPy. But there is nothing to be done, unless we can consider limiting the"#s", etc. type argparsing, which is likely not an option.