Repository navigation
Commit 9c622e6
authored
Python: fix(core): fold streamed text runs in one pass instead of repeated += (microsoft#8953)
* Python: fix(core): fold streamed text runs in one pass instead of repeated +=
_coalesce_text_content folded chunks with first_new_content += content, and
every Content.__add__ rebuilds the text string, re-merges
additional_properties, and re-concatenates raw_representation lists, so
aggregating n chunks cost O(n^2) (microsoft#8908).
Collect each mergeable run and fold it in one pass in _merge_content_run:
one join per segment, a single props merge (the earliest chunk still wins
per key), one flattened raw-representation list. The fold reproduces the old
semantics exactly, including the run head's raw representation being dropped
(the old path deep-copied the head, and deepcopy discards raw_representation
via _SHALLOW_COPY_FIELDS) and the text_reasoning id / reasoning-text split
boundaries.
16000 chunks: 0.302s -> 0.011s on this machine.
Signed-off-by: Yufeng He <[email protected]>
* Python: fix(core): match the repeated-+= id fallback and detach the run head in the one-pass fold
Two exactness gaps against the old fold, both found by fuzzing old vs new:
- id for a text_reasoning run fell back to None when no chunk carried a
truthy id, but repeated `self.id or other.id` yields the last chunk's
value there (so an all-falsy run ending in id="" kept ""). Fall back to
run[-1].id.
- the old path deep-copied the run head before folding, so the aggregate
never aliased the head's nested additional_properties values or
annotation dicts. The one-pass merge reused them. Deep-copy the head's
contributions the same way.
Also harden the equivalence tests: compare raw_representation explicitly
(Content.__eq__ skips it, and the oracle's expected side now uses shallow
copies so deepcopy does not strip raw up front), cover annotations on
head/later chunks, and add a seeded fuzz test folding random streams
through the live __add__ so future drift between the add-path and the
one-pass fold fails CI. Cross-reference _merge_content_run from
_add_text_content/_add_text_reasoning_content.
* Python: test(core): narrow the optional props before subscripting in the detach test
The four test type checkers all rejected subscripting Content's optional
additional_properties/annotations directly. Bind and assert them first.
Signed-off-by: Yufeng He <[email protected]>
---------
Signed-off-by: Yufeng He <[email protected]>1 parent 3e6db91 commit 9c622e6
2 files changed
Lines changed: 322 additions & 29 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1738 | 1738 | | |
1739 | 1739 | | |
1740 | 1740 | | |
1741 | | - | |
| 1741 | + | |
| 1742 | + | |
| 1743 | + | |
| 1744 | + | |
| 1745 | + | |
1742 | 1746 | | |
1743 | 1747 | | |
1744 | 1748 | | |
| |||
1748 | 1752 | | |
1749 | 1753 | | |
1750 | 1754 | | |
1751 | | - | |
| 1755 | + | |
| 1756 | + | |
| 1757 | + | |
| 1758 | + | |
| 1759 | + | |
| 1760 | + | |
1752 | 1761 | | |
1753 | 1762 | | |
1754 | 1763 | | |
| |||
2399 | 2408 | | |
2400 | 2409 | | |
2401 | 2410 | | |
| 2411 | + | |
| 2412 | + | |
| 2413 | + | |
| 2414 | + | |
| 2415 | + | |
| 2416 | + | |
| 2417 | + | |
| 2418 | + | |
| 2419 | + | |
| 2420 | + | |
| 2421 | + | |
| 2422 | + | |
| 2423 | + | |
| 2424 | + | |
| 2425 | + | |
| 2426 | + | |
| 2427 | + | |
| 2428 | + | |
| 2429 | + | |
| 2430 | + | |
| 2431 | + | |
| 2432 | + | |
| 2433 | + | |
| 2434 | + | |
| 2435 | + | |
| 2436 | + | |
| 2437 | + | |
| 2438 | + | |
| 2439 | + | |
| 2440 | + | |
| 2441 | + | |
| 2442 | + | |
| 2443 | + | |
| 2444 | + | |
| 2445 | + | |
| 2446 | + | |
| 2447 | + | |
| 2448 | + | |
| 2449 | + | |
| 2450 | + | |
| 2451 | + | |
| 2452 | + | |
| 2453 | + | |
| 2454 | + | |
| 2455 | + | |
| 2456 | + | |
| 2457 | + | |
| 2458 | + | |
| 2459 | + | |
| 2460 | + | |
2402 | 2461 | | |
2403 | | - | |
| 2462 | + | |
| 2463 | + | |
| 2464 | + | |
| 2465 | + | |
| 2466 | + | |
2404 | 2467 | | |
2405 | 2468 | | |
2406 | 2469 | | |
2407 | | - | |
| 2470 | + | |
| 2471 | + | |
| 2472 | + | |
| 2473 | + | |
| 2474 | + | |
| 2475 | + | |
| 2476 | + | |
| 2477 | + | |
| 2478 | + | |
| 2479 | + | |
| 2480 | + | |
| 2481 | + | |
| 2482 | + | |
2408 | 2483 | | |
2409 | | - | |
2410 | | - | |
2411 | | - | |
2412 | | - | |
2413 | | - | |
2414 | | - | |
2415 | | - | |
2416 | | - | |
2417 | | - | |
2418 | | - | |
2419 | | - | |
2420 | | - | |
2421 | | - | |
2422 | | - | |
2423 | | - | |
2424 | | - | |
2425 | | - | |
2426 | | - | |
2427 | | - | |
2428 | | - | |
2429 | | - | |
2430 | | - | |
2431 | | - | |
| 2484 | + | |
| 2485 | + | |
2432 | 2486 | | |
2433 | | - | |
2434 | | - | |
| 2487 | + | |
| 2488 | + | |
| 2489 | + | |
| 2490 | + | |
| 2491 | + | |
| 2492 | + | |
| 2493 | + | |
| 2494 | + | |
| 2495 | + | |
| 2496 | + | |
| 2497 | + | |
| 2498 | + | |
| 2499 | + | |
| 2500 | + | |
| 2501 | + | |
| 2502 | + | |
| 2503 | + | |
| 2504 | + | |
| 2505 | + | |
| 2506 | + | |
| 2507 | + | |
| 2508 | + | |
| 2509 | + | |
2435 | 2510 | | |
2436 | 2511 | | |
2437 | 2512 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
| |||
41 | 42 | | |
42 | 43 | | |
43 | 44 | | |
| 45 | + | |
44 | 46 | | |
45 | 47 | | |
46 | 48 | | |
| |||
2580 | 2582 | | |
2581 | 2583 | | |
2582 | 2584 | | |
| 2585 | + | |
| 2586 | + | |
| 2587 | + | |
| 2588 | + | |
| 2589 | + | |
| 2590 | + | |
| 2591 | + | |
| 2592 | + | |
| 2593 | + | |
| 2594 | + | |
| 2595 | + | |
| 2596 | + | |
| 2597 | + | |
| 2598 | + | |
| 2599 | + | |
| 2600 | + | |
| 2601 | + | |
| 2602 | + | |
| 2603 | + | |
| 2604 | + | |
| 2605 | + | |
| 2606 | + | |
| 2607 | + | |
| 2608 | + | |
| 2609 | + | |
| 2610 | + | |
| 2611 | + | |
| 2612 | + | |
| 2613 | + | |
| 2614 | + | |
| 2615 | + | |
| 2616 | + | |
| 2617 | + | |
| 2618 | + | |
| 2619 | + | |
| 2620 | + | |
| 2621 | + | |
| 2622 | + | |
| 2623 | + | |
| 2624 | + | |
| 2625 | + | |
| 2626 | + | |
| 2627 | + | |
| 2628 | + | |
| 2629 | + | |
| 2630 | + | |
| 2631 | + | |
| 2632 | + | |
| 2633 | + | |
| 2634 | + | |
| 2635 | + | |
| 2636 | + | |
| 2637 | + | |
| 2638 | + | |
| 2639 | + | |
| 2640 | + | |
| 2641 | + | |
| 2642 | + | |
| 2643 | + | |
| 2644 | + | |
| 2645 | + | |
| 2646 | + | |
| 2647 | + | |
| 2648 | + | |
| 2649 | + | |
| 2650 | + | |
| 2651 | + | |
| 2652 | + | |
| 2653 | + | |
| 2654 | + | |
| 2655 | + | |
| 2656 | + | |
| 2657 | + | |
| 2658 | + | |
| 2659 | + | |
| 2660 | + | |
| 2661 | + | |
| 2662 | + | |
| 2663 | + | |
| 2664 | + | |
| 2665 | + | |
| 2666 | + | |
| 2667 | + | |
| 2668 | + | |
| 2669 | + | |
| 2670 | + | |
| 2671 | + | |
| 2672 | + | |
| 2673 | + | |
| 2674 | + | |
| 2675 | + | |
| 2676 | + | |
| 2677 | + | |
| 2678 | + | |
| 2679 | + | |
| 2680 | + | |
| 2681 | + | |
| 2682 | + | |
| 2683 | + | |
| 2684 | + | |
| 2685 | + | |
| 2686 | + | |
| 2687 | + | |
| 2688 | + | |
| 2689 | + | |
| 2690 | + | |
| 2691 | + | |
| 2692 | + | |
| 2693 | + | |
| 2694 | + | |
| 2695 | + | |
| 2696 | + | |
| 2697 | + | |
| 2698 | + | |
| 2699 | + | |
| 2700 | + | |
| 2701 | + | |
| 2702 | + | |
| 2703 | + | |
| 2704 | + | |
| 2705 | + | |
| 2706 | + | |
| 2707 | + | |
| 2708 | + | |
| 2709 | + | |
| 2710 | + | |
| 2711 | + | |
| 2712 | + | |
| 2713 | + | |
| 2714 | + | |
| 2715 | + | |
| 2716 | + | |
| 2717 | + | |
| 2718 | + | |
| 2719 | + | |
| 2720 | + | |
| 2721 | + | |
| 2722 | + | |
| 2723 | + | |
| 2724 | + | |
| 2725 | + | |
| 2726 | + | |
| 2727 | + | |
| 2728 | + | |
| 2729 | + | |
| 2730 | + | |
| 2731 | + | |
| 2732 | + | |
| 2733 | + | |
| 2734 | + | |
| 2735 | + | |
| 2736 | + | |
| 2737 | + | |
| 2738 | + | |
| 2739 | + | |
| 2740 | + | |
| 2741 | + | |
| 2742 | + | |
| 2743 | + | |
| 2744 | + | |
| 2745 | + | |
| 2746 | + | |
| 2747 | + | |
| 2748 | + | |
| 2749 | + | |
| 2750 | + | |
| 2751 | + | |
| 2752 | + | |
| 2753 | + | |
| 2754 | + | |
| 2755 | + | |
| 2756 | + | |
| 2757 | + | |
| 2758 | + | |
| 2759 | + | |
| 2760 | + | |
| 2761 | + | |
| 2762 | + | |
| 2763 | + | |
| 2764 | + | |
| 2765 | + | |
| 2766 | + | |
| 2767 | + | |
| 2768 | + | |
| 2769 | + | |
| 2770 | + | |
| 2771 | + | |
| 2772 | + | |
| 2773 | + | |
| 2774 | + | |
| 2775 | + | |
| 2776 | + | |
| 2777 | + | |
| 2778 | + | |
| 2779 | + | |
| 2780 | + | |
| 2781 | + | |
| 2782 | + | |
| 2783 | + | |
| 2784 | + | |
| 2785 | + | |
| 2786 | + | |
| 2787 | + | |
| 2788 | + | |
| 2789 | + | |
| 2790 | + | |
| 2791 | + | |
| 2792 | + | |
| 2793 | + | |
| 2794 | + | |
| 2795 | + | |
| 2796 | + | |
| 2797 | + | |
| 2798 | + | |
| 2799 | + | |
| 2800 | + | |
2583 | 2801 | | |
2584 | 2802 | | |
2585 | 2803 | | |
| |||
0 commit comments