Skip to content

TSK: Follow-up things for stringdtype #25693

Open

Description

@mhvk

Proposed new feature or change:

A number of points arose in discussion of #25347 that were deemed better done in follow-up, to not distract from the meat of that (very large) PR. This issue is to remind us of those.

  1. The doc for StringDType has a size argument that was needed in development but not for actual release. It can be removed. This should be done before the NumPy 2.0 release because it is public API. (Though might it be useful for a short version that only uses arena? see below. Probably can put it back if needed...)
  2. The add ufunc needs a promoter so addition with a python string works.
  3. Add a cython interface for the stringdtype C API.
  4. It is likely better to use a flag for strings "long strings" (stored outside of the numpy array proper) instead of one for short ones (stored inside), so that an all-zero entry correctly implies a zero-length string (see API: Introduce stringdtype [NEP 55] #25347 (comment))
  5. Refactor the flags in the vstring implementation to use bitfields. This will improve clarity and eliminate complicated bitflag math.
  6. Possibly, the arena should have more recoverable flags/size information (see API: Introduce stringdtype [NEP 55] #25347 (comment))
  7. Investigate refactoring new_stringdtype_instance into tp_init
  8. Replace the long #define in casts.c with templating (or .c.src)
  9. Replace ufunc wrappers with passing functions into *auxdata (see here, here, here, and here) [e.g., minimum and maximum; the various comparison functions; the templated unary functions; find, rfind and maybe count; startswith and endswith; lstrip, rstrip and strip, plus whitespace versions]. Attempt in MAINT: combine string ufuncs by passing on auxilliary data #25796
  10. Check array2string formatter overrides.
  11. Adjust error messages referring to "object array" (e.g., a.view('2u8') currently errors with "Cannot change data-type for object array.").
  12. Have some helper functions that make it easy for StringDType ufuncs to use out arguments, also for in-place operations.
  13. See whether null-handling code in ufunc loops and casts can be consolidated into a helper function to reduce code duplication.
  14. Add checks for very long strings, see MAINT: Ensure correct handling for very large unicode strings #27875

Things possibly for longer term

  • Support in structured arrays (perhaps not super useful, but could be seen as similar to object).
  • Expose more of the currently private NpyString API. This will depend on feedback from users.
  • Fix longdouble to string, which is listed as broken in a TODO item in casts.c. isn't dragon able to deal with it?
  • Add a DType API callback that triggers when the initial filling of a newly created array (e.g. after PyArray_FromAny finishes filling a new array). We could use this to trim the arena buffer to the exact size needed by the array. Currently we over-allocate because the buffer grows exponentially with a growth factor of 1.25.
  • Might it make sense on 64bit systems, where normally the size is 16 bytes, to have a 8-byte version (short strings up to 7, only arena allocations for long ones; might use the size argument...).
  • In principle, .view(StringDType()) could be possible in some cases (e.g., to change the NA behaviour). Would need to share the allocator (and thus have reference counts for that...).
  • Dealing with array scalars vs str scalars - see also more general discussion about array scalars in ENH: No longer auto-convert array scalars to numpy scalars in ufuncs (and elsewhere?) #24897. ENH: add a StringDType scalar type that wraps a UTF-8 string #28165

Small warts, possibly not solvable

  • StringDType is added to add_dtype_helper late in the initialization of multiarraymodule; can this be easier?
  • Can the cases of having and not having gil be factored out, so that one doesn't get the kind of hybrid stuff in load_non_nullable_string with its has_gil argument.
  • to have dtype.hasobject be true is logical but not quite the right name.

Activity

  1. ngoldbaum commented on Jan 26, 2024

    @ngoldbaum
    Member

    Added a few things I had on my personal list. Thanks for opening this and for all the review this week!

  2. changed the title [-]Follow-up things for stringdtype[/-] [+]TSK: Follow-up things for stringdtype[/+] on Jan 31, 2024
  3. ngoldbaum commented on Feb 6, 2024

    @ngoldbaum
    Member

    @asmeurer pointed out to me that we should add a proper scalar type. Right now stringdtype's scalar is str, which doesn't inherit from np.generic, so stringdtype ends up as an oddball in the API from the perspective of the type hierarchy of the scalar types.

    We can fix this by defining a proper scalar type that wraps a single packed string entry and exposes an API that is duck-compatible with str, making use of the ufunc implementations we're adding.

    This may also lead to performance improvements, since scalar indexing won't need to copy to python's internal string representation.

    I don't think this needs to happen for 2.0 since this is something we can improve later and I doubt it will have major user-facing impacts, since object string arrays also effectively have a str scalar.

  4. seberg commented on Feb 6, 2024

    @seberg
    Member

    so stringdtype ends up as an oddball in the API from the perspective of the type hierarchy of the scalar types.

    I personally don't see this as a big issue. Although it might be friendly to not convret to scalar as often as we currently do if it is a string scalar (bad timeline too though).

  5. mhvk commented on Feb 6, 2024

    @mhvk
    ContributorAuthor

    I think the main problem may be that people expect generally that they can treat array[something] as array-ish, with a .shape, etc. In that sense we probably do need a scalar type (or, perhaps preferably, just not drop to a scalar in the first place...).

    If we're going to be "not quite right" for 2.0, should we perhaps err on the side of not creating a str object?

  6. ngoldbaum commented on Feb 6, 2024

    @ngoldbaum
    Member

    Are you saying that we should create a 0D array instead?

  7. ngoldbaum commented on Feb 6, 2024

    @ngoldbaum
    Member

    If so, I think it's much more important for the scalar to be duck-compatible with str than with ndarray. Especially if the goal is replace object string arrays in downstream packages.

  8. mhvk commented on Feb 6, 2024

    @mhvk
    ContributorAuthor

    Yes, I was, I really dislike array scalars and wish everything was 0-D arrays instead. But you make a good point that we want to make sure that object arrays can easily be replaced...

    EDIT: because I dislike how the type that comes out of __getitem__ or .sum() depends on the arguments. Why should axis=None give me a different instance than axis=-1? And it is just a hassle if one subclasses ndarray...

  9. asmeurer commented on Feb 7, 2024

    @asmeurer
    Member

    I also completely agree that scalars are terrible for a dozen different reasons, and wish numpy just had 0-d arrays. But I guess this was too ambitious for NumPy 2.0. As it stands, scalars exist. Nathan makes a good point that object scalars are already kind of specially broken because they aren't even numpy types, and this new dtype replaces object str arrays for various use-cases (tbf, it also replaces np.str_ arrays for many use-cases too). But at least if you are working with object arrays you probably (or at least hopefully) are doing it on purpose and can be careful about this. For every other dtype, the scalar is at least array-like in most respects. And this is documented too https://numpy.org/doc/stable/reference/arrays.scalars.html#numpy.object_:

    The object type is also special because an array containing object_ items does not return an object_ object on item access, but instead returns the actual object that the array item refers to.

    Making stringdtype scalars subclass from both str and generic seems like the best option from a usability point of view, although if actually making it a real str subclass is not good for performance, just making it duck type and defining __instancecheck__ is probably good enough.

  10. seberg commented on Feb 7, 2024

    @seberg
    Member

    Even subclassing from generic gives you crazy things, like indexing must be string indexing and not array indexing! In other words: IMO, part of the problem with scalars is that they pretend to be array likes, even though it is currently vital for much code that they do (because we create them too often).
    Plus, the ABI is likely odd, so care has to be taken (numpy strings pull it off, so not sure if it is easy or not).

    I envisioned we could create a new np.NumPyScalar, fully abstract, class you can register with. But not sure it matters here.

    In short, I don't care about the hierarchy at all, but I agree there are two things:

    1. You cannot write np.array(scalar) and np.array(str_array[0]) without a dtype because we do not pick the new dtype for those for BC reasons.
    2. Just like for object dtype, the fact that we don't preserve 0-D arrays is more harmful for this DType compared to numerical dtypes where the scalars somewhat duck-type as arrays.

    I suggest moving discussion about point 2 to gh-24897. I could imagine making my try-0d-preservation opt-in more agressively for all but current dtypes (minus object maybe), even if we don't dare change it otherwise without 2.0.
    And yes, maybe we should never convert to scalars and make that something users always would have to do explicitly with .item() or so, but not sure I think that is as realistic.

  11. MarkusSintonen commented on May 23, 2024

    @MarkusSintonen

    Support in structured arrays (perhaps not super useful, but could be seen as similar to object)

    I'm exited about StringDType but noticed that its not working with structure arrays which is a bummer. We are using structured arrays for efficiently identifying "good values" from a boolean mask. This boolean mask is carried together with the actual data to efficiently detect which are good values in the array. It looks something like this:
    numpy.dtype([("values", some_type), ("is_valid", numpy.dtype("?"))])

    But this wouldn't now work with some_type=StringDType() 😞 I see there is StringDType(na_object=None) way to define something like this but it wont work for efficiently detecting positions of good/bad values compared to being able to carry it with a mask in a struct.

    I also did some basic performance tests in v2.0.0rc2 for the experimental StringDType but it seems to be much slower than "O" or "U" dtypes in certain cases, are you aware of these issues:

    print(numpy.__version__)  # '2.0.0rc2'
    
    options = ["a", "bb", "ccc", "dddd"]
    lst = random.choices(options, k=1000)
    arr_s = numpy.fromiter(lst, dtype=StringDType(), count=len(lst))
    arr_o = numpy.fromiter(lst, dtype="O", count=len(lst))
    arr_u = numpy.fromiter(lst, dtype="U5", count=len(lst))
    
    print(timeit.timeit(lambda: numpy.unique(arr_s), number=10000))  # 4.270 <- why so much slower?
    print(timeit.timeit(lambda: numpy.unique(arr_o), number=10000))  # 2.357
    print(timeit.timeit(lambda: numpy.unique(arr_u), number=10000))  # 0.502
    print(timeit.timeit(lambda: sorted(set(lst)), number=10000))  #  0.078
  12. ngoldbaum commented on May 23, 2024

    @ngoldbaum
    Member

    Supporting structured dtypes is possible in principle but we punted on it for the initial version because it would add a bunch of complexity. The structured dtype code inside numpy (let alone in external libraries) makes strong assumptions about dtype instances being interchangeable, which aren't valid for stringdtype.

    One path forward would be adding a mode to stringdtype that always heap-allocates. This isn't great for performance or cache locality, but does mean that the packed strings always store heap pointers and the dtype instance does not store any sidecar data. I'm not sure how hard that would be - the structured dtype code is quite complicated and I'm not sure how easy it is to determine whether a stringdtype value lives inside a structured dtype vs a stand-alone stringdtype instance.

    Another thing I have in mind is to eventually add a fixed-width string mode to stringdtype, which would we could also straightforwardly support in structured dtypes, although if you need variable-width strings that doesn't help much.

    Thanks for the report about unique being slow, I opened #26510 to track that. Please also open followup issues with reports about performance if you have others, it's very appreciated, especially with a script to reproduce what you're seeing.

  13. MarkusSintonen commented on May 23, 2024

    @MarkusSintonen

    Thanks, and thanks for your great work!

    Another thing I have in mind is to eventually add a fixed-width string mode to stringdtype, which would we could also straightforwardly support in structured dtypes, although if you need variable-width strings that doesn't help much.

    Variable-width strings are the thing we are interested in :) Essentially improving performance with some columnar type of data processing that may have free form strings. There is the "O" type for this, but it has some issues. Eg the above shows how numpy.unique (for str-O) is quite slow compared to Pythons own set. Potentially the more native StringDType could help there.

  14. seberg commented on May 23, 2024

    @seberg
    Member

    FWIW, I think we should allow putting stringdtype into structs.

    I think the main issue is that string dtype requires a new dtype instance to be created in some places and that property must be "inherited" by the structured dtype.
    That might be as easy as implementing the hook, but may also need a new dtype flag and auditing some other paths (I am not sure, things tend to turn out harder than you think).

    But I am not sure if anyone prioritize working on this soon, if anyone is interested, I think we will definitely help out to get started.

    unique in NumPy doesn't use hashes (for the time being at least), but a custom sorting would presumably get it to comparable to the old strings (dunno if faster or slower). But comparing to Python has to be done with cares: Python strings cache their hash and can "cheat" because you have identical objects which are known to be equal without even looking at the string itself!

  15. 31 remaining items

  16. ngoldbaum commented on Apr 9, 2025

    @ngoldbaum
    Member

    I think I agree with that. Maybe we’ll find other arguments for the trailing nulls but this may not be compelling enough.

    That said, I also think we can probably avoid copies, particularly for many short strings, if we provide the gather API for h5py to consume. There may also be a way to encode the size prefix for arena strings in such a way that the first byte is almost always null, so arena strings happen to nearly always be null-terminated for free.

  17. vadimkantorov commented on Apr 15, 2025

    @vadimkantorov

    Am I correct that currently NumPy does not support representing a list of varlen strings where the strings are stored concatenated in the array's buffer (e.g. represented as UTF-8 bytes)?

    I read through https://numpy.org/devdocs/user/basics.strings.html, and it seems that varlen strings are only supported via np.object (btw np.object not mentioned in https://numpy.org/devdocs/user/basics.strings.html) or via StringDType, but both just store pointers to some string objects stored on Python object heap or in some string heap?

    But wouldn't it make sense to also support some packed buffer formats for readonly string arrays? Or does stringdtype does it already?

    Then the NumPy could offer zero-copy conversions from Arrow, and even mmap directly a super-large text-file (and e.g. find the string lengths via scanning for newlines; and do no decoding if the user asserts it's already utf-8 encoded, or let the user provide the string lengths separately if they have this info already), and also make a better memory layout for sequential vectorized processing... This also makes it simpler for serializing/deserializing such an array, and for sharing it between processes, etc

  18. ngoldbaum commented on Apr 15, 2025

    @ngoldbaum
    Member

    btw np.object not mentioned in https://numpy.org/devdocs/user/basics.strings.htm

    Fair point, we could probably document that np.object is still a thing, but we really wanted to point people to StringDType rather than object string arrays. Our hope is that StringDType provides a superset of functionality of object strings.

    But wouldn't it make sense to also support some packed buffer formats for readonly string arrays?

    Very much agree. If StringDType had a mode where the array is immutable and it exposed exactly the arrow format that would be really cool. This can’t be the default though because then StringDType can’t replace object string arrays.

  19. vadimkantorov commented on Apr 15, 2025

    @vadimkantorov

    but we really wanted to point people to StringDType rather than object string arrays

    I stumbled on np.object as representation from strings when doing sth like pyarrow.parquet.read_table("some.parquet")["string_col"].to_numpy().dtype, although I wish it did not create tons of Python string objects (which itself is maybe not nice if dealing with millions or billions of strings)

    Was looking of a modern NumPy way of representing a large dataset with three string columns which plays nicely with serialization (so three independent columns of packed strings, or somehow an array of varlen records, where each record packs three strings)

    I hoped there was a zero-copy way of representing a readonly column of strings, obtained from Arrow or HDF5 disk formats (and maybe of some other raw formats) to be representable zero-copy by NumPy - or at least to not leave millions of Python string objects around

  20. ngoldbaum commented on Apr 15, 2025

    @ngoldbaum
    Member

    Yes unfortunately we can’t switch the default inferred representation until NumPy 3.0 probably ☹ . Yet another reason that page should definitely mention object strings. When I was updating those docs to talk about StringDType I wasn’t thinking about it from that perspective.

    PyArrow might certainly be able to do something smarter on newer NumPy versions that support StringDType, I haven’t followed PyArrow development recently or tried to push that.

  21. vadimkantorov commented on Apr 15, 2025

    @vadimkantorov

    I also discussed these matters in PyTorch (and various useful internal representations):

    Basically, there are currently problems with sharing large readonly string arrays across processes - and packed memory layouts are much more beneficial than sharing tons of objects (and it appears there still are problems even with NumPy arrays in some cases):

    So if NumPy proposes new modern designs here, it's more likely that PyTorch would follow and implement some support for string readonly arrays and some GPU-accelerated string processing.


    Also, np.object as strings appear in Triton Inference Server Python bindings:

  22. mhvk commented on Apr 15, 2025

    @mhvk
    ContributorAuthor

    @vadimkantorov - I think the new StringDType could in principle come close - although how numpy operates on arrays means that there always will have to be pointers (that elements are separated by a fixed stride is non-negotiable in the numpy design), but at least those pointers can in principle be to within a previously allocated block of memory (though that needs new transfer code to be written).

  23. vadimkantorov commented on Apr 15, 2025

    @vadimkantorov

    Regarding the "default" internal representation, I think there could be several supported internal representations concurrently - for various usecases, and the default might be switched later.

    E.g. it can make sense to support both utf-8 and utf-32 (for fixed-size char and indexing simplicity), and support zero-byte termination or "termination" with newline (or another custom char). At least for readonly ways of representing an existing dataset without copies and indexing into it - these can be useful. Some more discussion in the first pytorch issue I linked.

    Another nice usecase would be parallelized transcoding of the whole array of strings into another encoding (also, NumPy should just copy the buffer of Python string object if it's already encoded in the correct encoding)

    For PyTorch, there is now in core a concept of NestedTensor which is a tuple of a data storage and an indexes tensor, so I thought maybe this structure could represent varlen strings (downsides are that currently indexes is always int64 which sometimes is wasteful for small strings), e.g. int8 could be interpreted by stringproc functions as ascii, uint8 as utf-8 and uint32/int32 as utf-32)...

  24. ngoldbaum commented on Apr 15, 2025

    @ngoldbaum
    Member

    We intentionally made the packed string an opaque pointer so the heap allocation can be changed in the future.

    I’m also planning to add support for encodings besides UTF-8 and a fixed-width mode to replace the fixed -width legacy string dtypes, but am focusing on other projects for at least the short to medium term. If you’re interested in working on this I’m happy help.

  25. vadimkantorov commented on Apr 16, 2025

    @vadimkantorov

    Also, maybe studying some of arrow binary formats for string columns might be useful, as then zero-copy conversion from arrow can be supported

    https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-view-layout

    seems quite tricky and has some optimizations (and even paddings) for short strings

  26. vadimkantorov commented on Apr 16, 2025

    @vadimkantorov

    Also, currently these object-typed arrays (sourced from parquet/arrow or triton-server) cannot be np.load-ed without pickle (although np.save somehow passes okay). I would suggest that the np.save should also then require allow_pickle=True (to make it less unexpected for np.load)

    import numpy as np
    
    a = b = np.array(['abc', 'def'], dtype = object)
    np.save('a.npy', a)
    _a = np.load('a.npy')
    
    #    np.load('a.npy')
    #  File "/home/inferencer/.local/lib/python3.10/site-packages/numpy/lib/_npyio_impl.py", line 488, in load
    #    return format.read_array(fid, allow_pickle=allow_pickle,
    #  File "/home/inferencer/.local/lib/python3.10/site-packages/numpy/lib/format.py", line 822, in read_array
    #    raise ValueError("Object arrays cannot be loaded when "
    #ValueError: Object arrays cannot be loaded when allow_pickle=False
    
    #res = np.rec.fromarrays([a, b], names = ['a', 'b'])
    #np.save('res.npy', res)
    #_ab = np.load('res.npy')

    Basically I was looking for a way to load a bunch of string columns from parquet and save them to a plain .npy file. Seems not so easy :) What would be the recommended way of doing so (preferably without allow_pickle=True - for safety)?

  27. vadimkantorov commented on Apr 16, 2025

    @vadimkantorov

    import numpy as np; np.save('a.npy', np.array(['abc', 'def'], dtype = object).astype(np.dtypes.StringDType())) warns with:

    /home/inferencer/.local/lib/python3.10/site-packages/numpy/lib/format.py:382: UserWarning: Custom dtypes are saved as python objects using the pickle protocol. Loading this file requires allow_pickle=True to be set.
      d['descr'] = dtype_to_descr(array.dtype)
    

    I would suggest, maybe both saving/loading an array of str objects and saving/loading an StringDType-d array should not require pickling for saving/loading - similar to how PyTorch disabled now pickle by default for saving/loading...

    Is there currently a way of saving/loading an array of varlen strings without requiring pickle?

  28. ngoldbaum commented on Apr 16, 2025

    @ngoldbaum
    Member

    Unfortunately that requires updating the npz format, since npz doesn’t have any support for sidecar files or non-strided data. There’s earlier discussion about this on the issue tracker.

    I’d love to fix that, it just takes time and effort to get it right, since any update to the npz format can’t break existing readers of the existing npz format.

  29. vadimkantorov commented on Apr 16, 2025

    @vadimkantorov

    Hopefully packing metadata + packed string data could circumvent any npz buffer format restrictions? (e.g. number of strings + lens/indices/extra metadata | raw packed string data - at least for serialization to disk... or maybe it could directly adopt some existing serialization formats like arrow for non-compressed / parquet for compressed)

  30. ngoldbaum commented on Apr 16, 2025

    @ngoldbaum
    Member

    See #24110 (comment).

    FWIW @seberg and I are working on a PEP to make StringDType work with the buffer protocol by allowing libraries to define their own formats. That might help with this sort of thing, or at least codifying something that a new npz format could use. See https://discuss.python.org/t/buffer-protocol-and-arbitrary-data-types/26256/13.

    If you’re interested in working on any of this I’d be happy to provide feedback or help. As I said upthread I’m focusing on other things (ecosystem free-threaded support mostly) for the short to medium term.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions