Repository navigation
Commit 32c9ee5
authored
fix: register object-dtype and out-of-ns-range coordinates (xqlsystems#223) (xqlsystems#224)
## Overview
`from_dataset` could not register several common NetCDF/xarray datasets:
non-cftime object-dtype variables and coordinates (strings) raised
`ArrowNotImplementedError`, and datetime coordinates outside the
`datetime64[ns]` range raised `OverflowError`. This fixes both in
`_parse_schema` and `_block_metadata`, and revives cftime partition
pruning that an object-dtype skip had silently disabled. Fixes xqlsystems#223.
## Before
```python
ds = xr.Dataset({"label": (["x"], np.array(["a", "b"], dtype=object))}, coords={"x": [1, 2]})
XarrayContext().from_dataset("t", ds, chunks={"x": 2})
# pyarrow.lib.ArrowNotImplementedError: Unsupported numpy type 17
times = xr.date_range("0001-01-01", periods=3, freq="100YS", use_cftime=True).to_datetimeindex(time_unit="us")
ds = xr.Dataset({"v": (["time"], np.arange(3.0))}, coords={"time": times})
XarrayContext().from_dataset("t", ds, chunks={"time": 2})
# OverflowError: Cannot convert Timestamp to nanoseconds without overflow
```
## After
```python
# both register and query cleanly
c = XarrayContext(); c.from_dataset("t", ds, chunks={"x": 2})
c.sql("SELECT label FROM t ORDER BY x").to_pandas()["label"].tolist()
# ['a', 'b']
# string coordinates, cftime calendars, and pre-1678 / post-2262 dates all register;
# out-of-int64 datetimes skip pruning instead of raising, so registration always succeeds.
```1 parent 6720f89 commit 32c9ee5
4 files changed
Lines changed: 209 additions & 17 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
145 | 145 | | |
146 | 146 | | |
147 | 147 | | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
148 | 156 | | |
149 | 157 | | |
150 | 158 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
| |||
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
| 20 | + | |
19 | 21 | | |
20 | 22 | | |
21 | 23 | | |
| |||
532 | 534 | | |
533 | 535 | | |
534 | 536 | | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
172 | 172 | | |
173 | 173 | | |
174 | 174 | | |
175 | | - | |
| 175 | + | |
176 | 176 | | |
177 | 177 | | |
178 | 178 | | |
179 | 179 | | |
180 | 180 | | |
181 | 181 | | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
182 | 187 | | |
183 | 188 | | |
184 | 189 | | |
185 | 190 | | |
186 | | - | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
187 | 196 | | |
188 | 197 | | |
189 | 198 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
386 | 386 | | |
387 | 387 | | |
388 | 388 | | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
389 | 403 | | |
390 | 404 | | |
391 | 405 | | |
| |||
414 | 428 | | |
415 | 429 | | |
416 | 430 | | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
417 | 435 | | |
418 | 436 | | |
419 | 437 | | |
420 | 438 | | |
421 | 439 | | |
422 | | - | |
423 | | - | |
424 | | - | |
425 | | - | |
426 | | - | |
427 | | - | |
428 | | - | |
429 | | - | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
430 | 454 | | |
431 | 455 | | |
432 | 456 | | |
| |||
466 | 490 | | |
467 | 491 | | |
468 | 492 | | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
469 | 502 | | |
470 | 503 | | |
471 | 504 | | |
472 | 505 | | |
473 | 506 | | |
474 | 507 | | |
475 | | - | |
476 | | - | |
477 | | - | |
478 | | - | |
479 | 508 | | |
480 | 509 | | |
481 | 510 | | |
482 | 511 | | |
483 | 512 | | |
484 | 513 | | |
485 | 514 | | |
486 | | - | |
487 | | - | |
488 | | - | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
489 | 527 | | |
490 | 528 | | |
491 | 529 | | |
| |||
0 commit comments