Specify how non-finite numbers are encoded in attributes - #398

This adds a short section to the v3 core specification that says how NaN, Infinity and -Infinity are written in attributes. It addresses #141, whose last comment said a PR adding a representation would be welcome.

The Problem

The specification defines metadata documents with the JSON type system of RFC 8259, which has no representation for a number that is not finite. It gives fill_value a rule for this (the strings "NaN", "Infinity" and "-Infinity"), but says nothing for attributes. Implementations have each filled the gap differently, and the results do not interoperate.

zarr-python writes the bare tokens NaN and Infinity, which are not valid JSON. Most other implementations parse zarr.json with a strict JSON parser and cannot open a group or array that has one such attribute. We ran into this from two directions: zarrita.js could not open NWB files that hold "resolution": NaN, and Icechunk refuses to store the same metadata.

The table below summarizes a survey of what implementations do today. The details, with versions and sources, are at the end.

Two changes outside this repository make the question timely. zarr-developers/zarr-python#4400 adds a setting that will eventually make zarr-python refuse non-finite attributes, and it does not yet say what a writer should produce instead. Faster JSON libraries for Python (orjson and msgspec) reject the bare tokens on reading and write null in their place, so a library that changes its JSON backend stops reading the files that zarr-python writes today.

The Proposal

A non-finite number in attributes is written as one of the three strings that the specification already defines for fill values: "NaN", "Infinity" or "-Infinity". This applies to every value within attributes, at any depth.

A fill value has the array's data type to say that the string denotes a number. An attribute has nothing, so a rule is needed for a string that has the same characters. Such a string is written with the prefix _str_, and a reader removes one prefix:

Only a string that is zero or more _str_ prefixes followed by exactly one of the three reserved strings is affected, so no other string changes. An affected string always gains exactly one prefix when it is written and loses one when it is read, however many it already has. The encoding is reversible for every value. A reference implementation of about twenty lines is here:

import math, re

_AFFECTED = re.compile(r"(?:_str_)*(?:NaN|Infinity|-Infinity)")


def encode(value):
    if isinstance(value, float) and not math.isfinite(value):
        return "NaN" if math.isnan(value) else ("Infinity" if value > 0 else "-Infinity")
    if isinstance(value, str):
        return "_str_" + value if _AFFECTED.fullmatch(value) else value
    if isinstance(value, list):
        return [encode(v) for v in value]
    if isinstance(value, dict):
        return {k: encode(v) for k, v in value.items()}
    return value


def decode(value):
    if isinstance(value, str) and _AFFECTED.fullmatch(value):
        return value[len("_str_"):] if value.startswith("_str_") else float(value)
    if isinstance(value, list):
        return [decode(v) for v in value]
    if isinstance(value, dict):
        return {k: decode(v) for k, v in value.items()}
    return value

The section also says that a reader should accept the bare tokens within attributes, so that data zarr-python has already written stays readable.

Why Strings and Not a Tagged Object

The discussion in #141 leaned toward an object such as {"@type": ...}, and the objection to plain strings there and in zarr-developers/zarr-python#412 was that a number could not be told apart from a string. I think strings with an escape rule are the better choice now, for these reasons.

• Four implementations already write these strings for a non-finite attribute, and the specification already uses them for fill values. Protocol Buffers' JSON mapping uses the same three strings. An object form would be a second encoding for the same values within one document, and those implementations would have to change what they write.
• The escape rule removes the ambiguity that the object form was meant to remove. zarrita.js shows the ambiguity as it stands: it writes {"a": NaN, "d": "NaN"} as {"a": "NaN", "d": "NaN"}.
• A reader that does not know the rule still gets something it can use. float("NaN") in Python and Number("NaN") in JavaScript convert the string, where an object would need handling written for it.
• The object form was attractive partly because it could carry a data type. For the three values in question there is no precision to preserve, and a finite float written in decimal survives a round trip without one.

null is not a workable alternative. It loses the value, and of the readers we tested, hdmf rejects it, netCDF-C returns the text null, and GDAL returns an empty string.

Compatibility

• Data that holds the bare tokens is not changed by this, and readers are asked to keep accepting it.
• Data written by zarrita.js, zarr-java, JZarr or netCDF-C, where "NaN" stands for a number, starts to be read as a number by every implementation that follows this section.
• An attribute that holds one of the three strings as text is read as a number once a reader follows this section. I do not know how common that is. I would expect it to be rare, and the text nearly always means the number, but it is a change in meaning for existing data and is the main cost of this proposal.
• No current reader converts the string "NaN" in an attribute to a number, apart from netCDF-C with its own type annotations. Every reader therefore needs a small change whichever encoding is chosen.

Open Questions

1. The spelling of the prefix. _str_ describes itself to someone who meets it unprepared. A single _ is shorter and matches the leading underscore the ecosystem already treats as reserved. I do not have a strong preference.
2. Core specification or convention. I have written this as a core rule because the current behavior produces documents that are not valid JSON, and because a file that does not declare a convention would stay ambiguous. If this ought to be a ZEP or a registered convention instead, I am glad to move it.
3. Scope. The rule covers attributes only. A non-finite number could in principle appear in the configuration of a codec or another extension, and one rule for the whole document outside fill_value may be simpler.
4. The change log. I added the entry under an "Unreleased" heading because I was not sure which version this would belong to.

Survey Details

Rows marked "tested" were run with the version given. Rows marked "source" come from reading the implementation's source and were not run.

Related

• Support for inf, nan, binary data in attributes #141, the earlier discussion of this question
• generalize the JSON serialization of scalars in data type definitions #351, on generalizing the JSON encoding of scalars
• Special treatment of NaN in JSON user attributes zarr-python#412 and WIP: Add custom json encoder and decoder zarr-python#533, an earlier attempt at plain strings
• np.float32(np.nan) is not a valid attribute value zarr-python#3478 and feat(attributes): warn when writing attributes that are not valid JSON zarr-python#4400, the current work in zarr-python
• support for np.nan and math.nan in attributes earth-mover/icechunk#1238