Repository navigation
Commit 62dc7ac
authored
Support INSERT OVERWRITE for MemTable (apache#24969)
## Which issue does this PR close?
- Part of apache#19617.
## Rationale for this change
DataFusion parses and plans `INSERT OVERWRITE`, but `MemTable` rejects
every insert operation except append. Users therefore cannot replace the
contents of an in-memory table with the standard overwrite operation.
## What changes are included in this PR?
- allow `InsertOp::Overwrite` in `MemTable::insert_into`
- let `MemSink` replace each target partition after the input stream
completes successfully
- continue rejecting the distinct `InsertOp::Replace` operation
## What is the testing strategy for this PR?
Three unit tests verify that overwrite replaces existing rows,
distributes multiple replacement batches across multiple target
partitions, and clears the table for empty input. A separate test
confirms that `InsertOp::Replace` remains unsupported.
Validated with:
- `cargo test -p datafusion --lib test_insert_overwrite -- --nocapture`
- `cargo clippy -p datafusion -p datafusion-datasource -p
datafusion-catalog --all-targets --all-features -- -D warnings`
- `cargo fmt --all -- --check`
## Are there any user-facing changes?
Yes. `INSERT OVERWRITE` now replaces all existing data in a `MemTable`.
There are no breaking public API changes.1 parent 18cbb87 commit 62dc7ac
3 files changed
Lines changed: 151 additions & 8 deletions
File tree
- datafusion
- catalog/src/memory
- core/src/datasource
- datasource/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
349 | 349 | | |
350 | 350 | | |
351 | 351 | | |
352 | | - | |
| 352 | + | |
353 | 353 | | |
354 | 354 | | |
355 | | - | |
| 355 | + | |
| 356 | + | |
356 | 357 | | |
357 | 358 | | |
358 | 359 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| |||
318 | 318 | | |
319 | 319 | | |
320 | 320 | | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
321 | 331 | | |
322 | 332 | | |
323 | 333 | | |
| |||
339 | 349 | | |
340 | 350 | | |
341 | 351 | | |
342 | | - | |
| 352 | + | |
343 | 353 | | |
344 | 354 | | |
345 | 355 | | |
| |||
479 | 489 | | |
480 | 490 | | |
481 | 491 | | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
482 | 608 | | |
483 | 609 | | |
484 | 610 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
920 | 920 | | |
921 | 921 | | |
922 | 922 | | |
| 923 | + | |
923 | 924 | | |
924 | 925 | | |
925 | 926 | | |
| |||
953 | 954 | | |
954 | 955 | | |
955 | 956 | | |
956 | | - | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
957 | 968 | | |
958 | 969 | | |
959 | 970 | | |
| |||
981 | 992 | | |
982 | 993 | | |
983 | 994 | | |
984 | | - | |
| 995 | + | |
985 | 996 | | |
986 | | - | |
987 | | - | |
| 997 | + | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
988 | 1004 | | |
989 | 1005 | | |
990 | 1006 | | |
| |||
0 commit comments