Repository navigation
Open
Labels
Description
Proposed new feature or change:
A number of points arose in discussion of #25347 that were deemed better done in follow-up, to not distract from the meat of that (very large) PR. This issue is to remind us of those.
- The doc for
StringDTypehas a size argument that was needed in development but not for actual release. It can be removed. This should be done before the NumPy 2.0 release because it is public API. (Though might it be useful for a short version that only uses arena? see below. Probably can put it back if needed...) - The
addufunc needs a promoter so addition with a python string works. - Add a cython interface for the stringdtype C API.
- It is likely better to use a flag for strings "long strings" (stored outside of the numpy array proper) instead of one for short ones (stored inside), so that an all-zero entry correctly implies a zero-length string (see API: Introduce stringdtype [NEP 55] #25347 (comment))
- Refactor the flags in the vstring implementation to use bitfields. This will improve clarity and eliminate complicated bitflag math.
- Possibly, the arena should have more recoverable flags/size information (see API: Introduce stringdtype [NEP 55] #25347 (comment))
- Investigate refactoring
new_stringdtype_instanceintotp_init - Replace the long
#defineincasts.cwith templating (or.c.src) - Replace ufunc wrappers with passing functions into
*auxdata(see here, here, here, and here) [e.g.,minimumandmaximum; the various comparison functions; the templated unary functions;find,rfindand maybecount;startswithandendswith;lstrip,rstripandstrip, pluswhitespaceversions]. Attempt in MAINT: combine string ufuncs by passing on auxilliary data #25796 - Check
array2stringformatter overrides. - Adjust error messages referring to "object array" (e.g.,
a.view('2u8')currently errors with"Cannot change data-type for object array."). - Have some helper functions that make it easy for
StringDTypeufuncs to useoutarguments, also for in-place operations. - See whether null-handling code in ufunc loops and casts can be consolidated into a helper function to reduce code duplication.
- Add checks for very long strings, see MAINT: Ensure correct handling for very large unicode strings #27875
Things possibly for longer term
- Support in structured arrays (perhaps not super useful, but could be seen as similar to
object). - Expose more of the currently private
NpyStringAPI. This will depend on feedback from users. - Fix
longdoubleto string, which is listed as broken in aTODOitem incasts.c. isn'tdragonable to deal with it? - Add a DType API callback that triggers when the initial filling of a newly created array (e.g. after
PyArray_FromAnyfinishes filling a new array). We could use this to trim the arena buffer to the exact size needed by the array. Currently we over-allocate because the buffer grows exponentially with a growth factor of 1.25. - Might it make sense on 64bit systems, where normally the size is 16 bytes, to have a 8-byte version (short strings up to 7, only arena allocations for long ones; might use the
sizeargument...). - In principle,
.view(StringDType())could be possible in some cases (e.g., to change the NA behaviour). Would need to share the allocator (and thus have reference counts for that...). - Dealing with array scalars vs
strscalars - see also more general discussion about array scalars in ENH: No longer auto-convert array scalars to numpy scalars in ufuncs (and elsewhere?) #24897. ENH: add a StringDType scalar type that wraps a UTF-8 string #28165
Small warts, possibly not solvable
StringDTypeis added toadd_dtype_helperlate in the initialization ofmultiarraymodule; can this be easier?- Can the cases of having and not having gil be factored out, so that one doesn't get the kind of hybrid stuff in
load_non_nullable_stringwith itshas_gilargument. - to have
dtype.hasobjectbe true is logical but not quite the right name.
Reacted by Sebastian Berg
Activity
ngoldbaum commentedon Jan 26, 2024
Added a few things I had on my personal list. Thanks for opening this and for all the review this week!
[-]Follow-up things for stringdtype[/-][+]TSK: Follow-up things for stringdtype[/+]ngoldbaum commentedon Feb 6, 2024
@asmeurer pointed out to me that we should add a proper scalar type. Right now stringdtype's scalar is
str, which doesn't inherit fromnp.generic, so stringdtype ends up as an oddball in the API from the perspective of the type hierarchy of the scalar types.We can fix this by defining a proper scalar type that wraps a single packed string entry and exposes an API that is duck-compatible with
str, making use of the ufunc implementations we're adding.This may also lead to performance improvements, since scalar indexing won't need to copy to python's internal string representation.
I don't think this needs to happen for 2.0 since this is something we can improve later and I doubt it will have major user-facing impacts, since object string arrays also effectively have a
strscalar.seberg commentedon Feb 6, 2024
I personally don't see this as a big issue. Although it might be friendly to not convret to scalar as often as we currently do if it is a string scalar (bad timeline too though).
mhvk commentedon Feb 6, 2024
I think the main problem may be that people expect generally that they can treat
array[something]as array-ish, with a.shape, etc. In that sense we probably do need a scalar type (or, perhaps preferably, just not drop to a scalar in the first place...).If we're going to be "not quite right" for 2.0, should we perhaps err on the side of not creating a
strobject?ngoldbaum commentedon Feb 6, 2024
Are you saying that we should create a 0D array instead?
ngoldbaum commentedon Feb 6, 2024
If so, I think it's much more important for the scalar to be duck-compatible with
strthan withndarray. Especially if the goal is replace object string arrays in downstream packages.mhvk commentedon Feb 6, 2024
Yes, I was, I really dislike array scalars and wish everything was 0-D arrays instead. But you make a good point that we want to make sure that object arrays can easily be replaced...
EDIT: because I dislike how the type that comes out of
__getitem__or.sum()depends on the arguments. Why shouldaxis=Nonegive me a different instance thanaxis=-1? And it is just a hassle if one subclassesndarray...asmeurer commentedon Feb 7, 2024
I also completely agree that scalars are terrible for a dozen different reasons, and wish numpy just had 0-d arrays. But I guess this was too ambitious for NumPy 2.0. As it stands, scalars exist. Nathan makes a good point that object scalars are already kind of specially broken because they aren't even numpy types, and this new dtype replaces object
strarrays for various use-cases (tbf, it also replacesnp.str_arrays for many use-cases too). But at least if you are working with object arrays you probably (or at least hopefully) are doing it on purpose and can be careful about this. For every other dtype, the scalar is at least array-like in most respects. And this is documented too https://numpy.org/doc/stable/reference/arrays.scalars.html#numpy.object_:Making stringdtype scalars subclass from both
strandgenericseems like the best option from a usability point of view, although if actually making it a realstrsubclass is not good for performance, just making it duck type and defining__instancecheck__is probably good enough.seberg commentedon Feb 7, 2024
Even subclassing from generic gives you crazy things, like indexing must be string indexing and not array indexing! In other words: IMO, part of the problem with scalars is that they pretend to be array likes, even though it is currently vital for much code that they do (because we create them too often).
Plus, the ABI is likely odd, so care has to be taken (numpy strings pull it off, so not sure if it is easy or not).
I envisioned we could create a new
np.NumPyScalar, fully abstract, class you can register with. But not sure it matters here.In short, I don't care about the hierarchy at all, but I agree there are two things:
np.array(scalar)andnp.array(str_array[0])without adtypebecause we do not pick the new dtype for those for BC reasons.objectdtype, the fact that we don't preserve 0-D arrays is more harmful for this DType compared to numerical dtypes where the scalars somewhat duck-type as arrays.I suggest moving discussion about point 2 to gh-24897. I could imagine making my try-0d-preservation opt-in more agressively for all but current dtypes (minus object maybe), even if we don't dare change it otherwise without 2.0.
And yes, maybe we should never convert to scalars and make that something users always would have to do explicitly with
.item()or so, but not sure I think that is as realistic.MarkusSintonen commentedon May 23, 2024
I'm exited about
StringDTypebut noticed that its not working with structure arrays which is a bummer. We are using structured arrays for efficiently identifying "good values" from a boolean mask. This boolean mask is carried together with the actual data to efficiently detect which are good values in the array. It looks something like this:numpy.dtype([("values", some_type), ("is_valid", numpy.dtype("?"))])But this wouldn't now work with
some_type=StringDType()😞 I see there isStringDType(na_object=None)way to define something like this but it wont work for efficiently detecting positions of good/bad values compared to being able to carry it with a mask in a struct.I also did some basic performance tests in
v2.0.0rc2for the experimentalStringDTypebut it seems to be much slower than"O"or"U"dtypes in certain cases, are you aware of these issues:ngoldbaum commentedon May 23, 2024
Supporting structured dtypes is possible in principle but we punted on it for the initial version because it would add a bunch of complexity. The structured dtype code inside numpy (let alone in external libraries) makes strong assumptions about dtype instances being interchangeable, which aren't valid for stringdtype.
One path forward would be adding a mode to stringdtype that always heap-allocates. This isn't great for performance or cache locality, but does mean that the packed strings always store heap pointers and the dtype instance does not store any sidecar data. I'm not sure how hard that would be - the structured dtype code is quite complicated and I'm not sure how easy it is to determine whether a stringdtype value lives inside a structured dtype vs a stand-alone stringdtype instance.
Another thing I have in mind is to eventually add a fixed-width string mode to stringdtype, which would we could also straightforwardly support in structured dtypes, although if you need variable-width strings that doesn't help much.
Thanks for the report about unique being slow, I opened #26510 to track that. Please also open followup issues with reports about performance if you have others, it's very appreciated, especially with a script to reproduce what you're seeing.
MarkusSintonen commentedon May 23, 2024
Thanks, and thanks for your great work!
Variable-width strings are the thing we are interested in :) Essentially improving performance with some columnar type of data processing that may have free form strings. There is the
"O"type for this, but it has some issues. Eg the above shows hownumpy.unique(for str-O) is quite slow compared to Pythons ownset. Potentially the more native StringDType could help there.seberg commentedon May 23, 2024
FWIW, I think we should allow putting stringdtype into structs.
I think the main issue is that string dtype requires a new dtype instance to be created in some places and that property must be "inherited" by the structured dtype.
That might be as easy as implementing the hook, but may also need a new dtype flag and auditing some other paths (I am not sure, things tend to turn out harder than you think).
But I am not sure if anyone prioritize working on this soon, if anyone is interested, I think we will definitely help out to get started.
uniquein NumPy doesn't use hashes (for the time being at least), but a custom sorting would presumably get it to comparable to the old strings (dunno if faster or slower). But comparing to Python has to be done with cares: Python strings cache their hash and can "cheat" because you have identical objects which are known to be equal without even looking at the string itself!31 remaining items
ngoldbaum commentedon Apr 9, 2025
I think I agree with that. Maybe we’ll find other arguments for the trailing nulls but this may not be compelling enough.
That said, I also think we can probably avoid copies, particularly for many short strings, if we provide the
gatherAPI for h5py to consume. There may also be a way to encode the size prefix for arena strings in such a way that the first byte is almost always null, so arena strings happen to nearly always be null-terminated for free.vadimkantorov commentedon Apr 15, 2025
Am I correct that currently NumPy does not support representing a list of varlen strings where the strings are stored concatenated in the array's buffer (e.g. represented as UTF-8 bytes)?
I read through https://numpy.org/devdocs/user/basics.strings.html, and it seems that varlen strings are only supported via np.object (btw np.object not mentioned in https://numpy.org/devdocs/user/basics.strings.html) or via StringDType, but both just store pointers to some string objects stored on Python object heap or in some string heap?
But wouldn't it make sense to also support some packed buffer formats for readonly string arrays? Or does stringdtype does it already?
Then the NumPy could offer zero-copy conversions from Arrow, and even mmap directly a super-large text-file (and e.g. find the string lengths via scanning for newlines; and do no decoding if the user asserts it's already utf-8 encoded, or let the user provide the string lengths separately if they have this info already), and also make a better memory layout for sequential vectorized processing... This also makes it simpler for serializing/deserializing such an array, and for sharing it between processes, etc
ngoldbaum commentedon Apr 15, 2025
Fair point, we could probably document that np.object is still a thing, but we really wanted to point people to StringDType rather than object string arrays. Our hope is that StringDType provides a superset of functionality of object strings.
Very much agree. If StringDType had a mode where the array is immutable and it exposed exactly the arrow format that would be really cool. This can’t be the default though because then StringDType can’t replace object string arrays.
vadimkantorov commentedon Apr 15, 2025
I stumbled on np.object as representation from strings when doing sth like
pyarrow.parquet.read_table("some.parquet")["string_col"].to_numpy().dtype, although I wish it did not create tons of Python string objects (which itself is maybe not nice if dealing with millions or billions of strings)Was looking of a modern NumPy way of representing a large dataset with three string columns which plays nicely with serialization (so three independent columns of packed strings, or somehow an array of varlen records, where each record packs three strings)
I hoped there was a zero-copy way of representing a readonly column of strings, obtained from Arrow or HDF5 disk formats (and maybe of some other raw formats) to be representable zero-copy by NumPy - or at least to not leave millions of Python string objects around
ngoldbaum commentedon Apr 15, 2025
Yes unfortunately we can’t switch the default inferred representation until NumPy 3.0 probably ☹ . Yet another reason that page should definitely mention object strings. When I was updating those docs to talk about StringDType I wasn’t thinking about it from that perspective.
PyArrow might certainly be able to do something smarter on newer NumPy versions that support StringDType, I haven’t followed PyArrow development recently or tried to push that.
vadimkantorov commentedon Apr 15, 2025
I also discussed these matters in PyTorch (and various useful internal representations):
detokenize(...)and arrays of strings). NestedTensor could become a representation for arrays-of-strings pytorch/pytorch#135704Basically, there are currently problems with sharing large readonly string arrays across processes - and packed memory layouts are much more beneficial than sharing tons of objects (and it appears there still are problems even with NumPy arrays in some cases):
So if NumPy proposes new modern designs here, it's more likely that PyTorch would follow and implement some support for string readonly arrays and some GPU-accelerated string processing.
Also, np.object as strings appear in Triton Inference Server Python bindings:
curltriton-inference-server/server#6337 (comment)mhvk commentedon Apr 15, 2025
@vadimkantorov - I think the new
StringDTypecould in principle come close - although how numpy operates on arrays means that there always will have to be pointers (that elements are separated by a fixed stride is non-negotiable in the numpy design), but at least those pointers can in principle be to within a previously allocated block of memory (though that needs new transfer code to be written).vadimkantorov commentedon Apr 15, 2025
Regarding the "default" internal representation, I think there could be several supported internal representations concurrently - for various usecases, and the default might be switched later.
E.g. it can make sense to support both utf-8 and utf-32 (for fixed-size char and indexing simplicity), and support zero-byte termination or "termination" with newline (or another custom char). At least for readonly ways of representing an existing dataset without copies and indexing into it - these can be useful. Some more discussion in the first pytorch issue I linked.
Another nice usecase would be parallelized transcoding of the whole array of strings into another encoding (also, NumPy should just copy the buffer of Python string object if it's already encoded in the correct encoding)
For PyTorch, there is now in core a concept of NestedTensor which is a tuple of a data storage and an indexes tensor, so I thought maybe this structure could represent varlen strings (downsides are that currently indexes is always int64 which sometimes is wasteful for small strings), e.g. int8 could be interpreted by stringproc functions as ascii, uint8 as utf-8 and uint32/int32 as utf-32)...
ngoldbaum commentedon Apr 15, 2025
We intentionally made the packed string an opaque pointer so the heap allocation can be changed in the future.
I’m also planning to add support for encodings besides UTF-8 and a fixed-width mode to replace the fixed -width legacy string dtypes, but am focusing on other projects for at least the short to medium term. If you’re interested in working on this I’m happy help.
vadimkantorov commentedon Apr 16, 2025
Also, maybe studying some of arrow binary formats for string columns might be useful, as then zero-copy conversion from arrow can be supported
https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout
seems quite tricky and has some optimizations (and even paddings) for short strings
vadimkantorov commentedon Apr 16, 2025
Also, currently these
object-typed arrays (sourced from parquet/arrow or triton-server) cannot benp.load-ed without pickle (althoughnp.savesomehow passes okay). I would suggest that thenp.saveshould also then requireallow_pickle=True(to make it less unexpected fornp.load)Basically I was looking for a way to load a bunch of string columns from parquet and save them to a plain
.npyfile. Seems not so easy :) What would be the recommended way of doing so (preferably withoutallow_pickle=True- for safety)?vadimkantorov commentedon Apr 16, 2025
import numpy as np; np.save('a.npy', np.array(['abc', 'def'], dtype = object).astype(np.dtypes.StringDType()))warns with:I would suggest, maybe both saving/loading an array of str objects and saving/loading an StringDType-d array should not require pickling for saving/loading - similar to how PyTorch disabled now pickle by default for saving/loading...
Is there currently a way of saving/loading an array of varlen strings without requiring pickle?
ngoldbaum commentedon Apr 16, 2025
Unfortunately that requires updating the npz format, since npz doesn’t have any support for sidecar files or non-strided data. There’s earlier discussion about this on the issue tracker.
I’d love to fix that, it just takes time and effort to get it right, since any update to the npz format can’t break existing readers of the existing npz format.
vadimkantorov commentedon Apr 16, 2025
Hopefully packing metadata + packed string data could circumvent any npz buffer format restrictions? (e.g.
number of strings + lens/indices/extra metadata | raw packed string data- at least for serialization to disk... or maybe it could directly adopt some existing serialization formats like arrow for non-compressed / parquet for compressed)ngoldbaum commentedon Apr 16, 2025
See #24110 (comment).
FWIW @seberg and I are working on a PEP to make StringDType work with the buffer protocol by allowing libraries to define their own formats. That might help with this sort of thing, or at least codifying something that a new npz format could use. See https://discuss.python.org/t/buffer-protocol-and-arbitrary-data-types/26256/13.
If you’re interested in working on any of this I’d be happy to provide feedback or help. As I said upthread I’m focusing on other things (ecosystem free-threaded support mostly) for the short to medium term.