Repository navigation
Commit bdd253c
committed
fix(bench): read Spark's NULL partition directories as NULL
Spark writes rows whose partition key is NULL to a
`<column>=__HIVE_DEFAULT_PARTITION__/` directory, and TPC-DS fact tables
have rows with a NULL date key. DataFusion casts each directory's value to
the partition column's type, so with `ss_sold_date_sk` declared `Int32`
every query that scanned such a table failed with a cast error
(apache/datafusion#18083).
When a partition column has that directory, register the files as
`<table>_raw` with the column as a string, and register `<table>` as a
view that maps the directory to NULL and casts the other values back.
Filters on the column still skip directories. Tables without that
directory are registered as before.1 parent adaff3c commit bdd253c
3 files changed
Lines changed: 199 additions & 20 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
411 | 411 | | |
412 | 412 | | |
413 | 413 | | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
414 | 423 | | |
415 | 424 | | |
416 | 425 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
533 | 533 | | |
534 | 534 | | |
535 | 535 | | |
536 | | - | |
537 | | - | |
538 | | - | |
539 | | - | |
540 | | - | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
541 | 544 | | |
542 | 545 | | |
543 | 546 | | |
544 | 547 | | |
545 | 548 | | |
546 | | - | |
547 | | - | |
548 | | - | |
| 549 | + | |
| 550 | + | |
549 | 551 | | |
550 | | - | |
| 552 | + | |
551 | 553 | | |
552 | 554 | | |
553 | 555 | | |
| |||
562 | 564 | | |
563 | 565 | | |
564 | 566 | | |
565 | | - | |
| 567 | + | |
566 | 568 | | |
567 | 569 | | |
568 | 570 | | |
| |||
571 | 573 | | |
572 | 574 | | |
573 | 575 | | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
574 | 596 | | |
575 | 597 | | |
576 | 598 | | |
| |||
581 | 603 | | |
582 | 604 | | |
583 | 605 | | |
584 | | - | |
585 | | - | |
586 | | - | |
587 | | - | |
588 | | - | |
589 | | - | |
590 | | - | |
| 606 | + | |
591 | 607 | | |
592 | 608 | | |
593 | 609 | | |
| |||
602 | 618 | | |
603 | 619 | | |
604 | 620 | | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
605 | 680 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
40 | 43 | | |
| 44 | + | |
41 | 45 | | |
42 | 46 | | |
43 | 47 | | |
| |||
259 | 263 | | |
260 | 264 | | |
261 | 265 | | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
262 | 270 | | |
263 | 271 | | |
264 | 272 | | |
| |||
267 | 275 | | |
268 | 276 | | |
269 | 277 | | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
270 | 281 | | |
271 | 282 | | |
272 | 283 | | |
| |||
294 | 305 | | |
295 | 306 | | |
296 | 307 | | |
| 308 | + | |
| 309 | + | |
297 | 310 | | |
298 | 311 | | |
299 | 312 | | |
300 | 313 | | |
301 | 314 | | |
| 315 | + | |
302 | 316 | | |
303 | 317 | | |
304 | 318 | | |
305 | 319 | | |
306 | 320 | | |
307 | 321 | | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
308 | 330 | | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
309 | 370 | | |
310 | 371 | | |
311 | | - | |
312 | | - | |
| 372 | + | |
| 373 | + | |
313 | 374 | | |
314 | 375 | | |
315 | 376 | | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
316 | 411 | | |
317 | 412 | | |
318 | 413 | | |
| |||
0 commit comments