inline_threshold not encoding time value?

In the example below I was expecting that time would get encoded because inline_threshold=400 and time is only 8 bytes long.

Below we see that depth is encoded (it's 332 bytes long), but time is not.

Is this expected behavior?

It seems inline is only called for normal, non-record arrays. Another thing to fix! Obviously, not too much use of netCDF3 has been seen.

Well, it's not a high priority for me -- if truth be told, I was really just trying to figure out how to inject a known time value into that reference.

You can replace the value with a binary, if you want. But also, there already is a function that does exactly this process for any reference set, so it just needs to be invoked.

You can replace the value with a binary, if you want. But also, there already is a function that does exactly this process for any reference set, so it just needs to be invoked.

Can you point to the function that can inject a known time value or how to replace the value with a binary?

@kmsampson

This may be fixed in #466 , if you would care to try.

@kmsampson , the spec says:

the str format of a reference value may be:
a string starting “base64:”, which will be decoded to binary
any other string, interpreted as ascii data

so set the key's value in the JSON accordingly. If still in memory, you can also directly assign the binary you want it to have. You could also use the filesystem interface, if you already made a filesystem, fs.pipe("time/0", b"\x00\x00..."); this modification can be outputted again with fs.save_json, or a .flush on the parquet/lazy storage, if you are using that. Too many options?

Yes, this is fixed in #466:

I still can't figure out how to assign a specific value though:

(perhaps I should ask this in discussions?)

I would do

d["refs"]["time/0"] = data_bytes

and kerchunk.utils._encode_for_JSON or consolidate can do the encoding for you.

You can also make a filesystem and interact with it

fs = fsspec.filesystem("reference", fo=d, ...)
fs.cat("time/0", data_bytes)
fs.save_json(filename) OR grab fs.references

I'm feeling kind of dumb here, but I still don't get it:

Oh sorry, the function works on the inner reference dict, d["refs"]

Navigation Menu

Repository navigation

inline_threshold not encoding time value? #468

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

inline_threshold not encoding time value? #468

Description

Activity

martindurant commented on Jun 23, 2024

rsignell commented on Jun 23, 2024

martindurant commented on Jun 23, 2024

kmsampson commented on Jun 23, 2024

martindurant commented on Jun 24, 2024

rsignell commented on Jun 24, 2024

martindurant commented on Jun 24, 2024

rsignell commented on Jun 24, 2024

martindurant commented on Jun 24, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions