Repository navigation
Commit 470a346
fix(api): make WatchSandbox loss-aware and resumable (NVIDIA#3209)
* fix(api): emit warning on WatchSandbox broadcast lag instead of terminating
Broadcast lag on the status, log, and platform receivers was converted to a RESOURCE_EXHAUSTED status that terminated the whole watch stream. Lag is recoverable: the receiver resumes at the oldest surviving message. Emit a SandboxStreamWarning and continue streaming instead; keep terminating on Closed. Add helpers and unit tests covering the warning payload and receiver recovery after lag.
Partially addresses NVIDIA#3055 (cursor/resume follow up separately).
Signed-off-by: Artem Lytvyn <[email protected]>
* refactor(server): group per-sandbox log bus state and stamp sequence numbers
Signed-off-by: Artem Lytvyn <[email protected]>
* feat(proto): add resume cursor fields to sandbox watch API
Signed-off-by: Artem Lytvyn <[email protected]>
* feat(server): stamp watch cursors from a shared per-sandbox sequence
Allocate cursors from a single SeqAllocator shared by the log and
platform event buses, so a sandbox's merged watch stream carries
unique, strictly increasing cursors. A single resume_after_cursor can
then unambiguously locate a client's position across both sources.
Rewrite both publish paths to allocate the sequence, stamp
event.cursor, send, and append to the tail under one lock. This
removes the previous get_mut().expect() TOCTOU race where a concurrent
remove() between the two lock sections could panic.
Signed-off-by: Artem Lytvyn <[email protected]>
* feat(server): serve WatchSandbox resume from cursor with gap detection
Add tail_after() to the log and platform event buses, returning every
buffered event newer than a client's resume cursor. Each PerSandbox now
tracks last_trimmed_seq (the highest seq it has evicted) so a resume is
reported as an unrecoverable ResumeGap only when this bus dropped an
event the client still needs.
Judging gaps by evictions, not by the tail's oldest seq, is required
under the shared cursor space: each bus's tail is non-contiguous in the
global sequence because the other bus owns the missing seqs, so
comparing against tail.front() would flag false gaps.
Signed-off-by: Artem Lytvyn <[email protected]>
* feat(server): resume WatchSandbox from cursor across log and platform buses
Wire resume_after_cursor into the watch producer. On a non-zero cursor,
replay events strictly after it from both the log and platform buses,
merge by shared cursor, and emit in order before entering the live loop.
A trimmed range on either bus is an unrecoverable gap and terminates the
stream with OUT_OF_RANGE carrying the requested and earliest-available
cursors, distinct from recoverable lag which warns and continues.
Signed-off-by: Artem Lytvyn <[email protected]>
* test(server): cover WatchSandbox cursor resume paths
Add handler-level tests for the resumable watch stream: replay strictly
after the client cursor, merge log and platform events in shared-cursor
order, suppress duplicates when resuming at the latest cursor, and
terminate with OUT_OF_RANGE when the requested cursor has been trimmed.
Signed-off-by: Artem Lytvyn <[email protected]>
* docs(api): document WatchSandbox loss-awareness and resume
Signed-off-by: Artem Lytvyn <[email protected]>
* fix(server): deliver watch events once and harden cursor teardown
Signed-off-by: Artem Lytvyn <[email protected]>
* feat(sdk): add loss-aware resumable watch_logs to Rust SDK client
Signed-off-by: Artem Lytvyn <[email protected]>
* fix(server): keep watch cursors monotonic across teardown and restart
Signed-off-by: Artem Lytvyn <[email protected]>
* fix(server): merge live watch sources by cursor before emission
Signed-off-by: Artem Lytvyn <[email protected]>
* fix(api): bind watch cursors to a cursor space and merge tail sources
Signed-off-by: Artem Lytvyn <[email protected]>
* fix(server): revalidate the watch cursor space after collecting replay
Signed-off-by: Artem Lytvyn <[email protected]>
* test(server): update the public RPC schema fingerprint for the string cursor
Signed-off-by: Artem Lytvyn <[email protected]>
* fix(server): hold watch events above the publication watermark and emit the watch lag warning before its batch
Signed-off-by: Artem Lytvyn <[email protected]>
* test(server): synchronize the watch live-order test with the end of initialization
Signed-off-by: Artem Lytvyn <[email protected]>
* fix(sdk): use canonical sandbox name in watch_logs
Signed-off-by: Artem Lytvyn <[email protected]>
* test(sdk): guard canonical-name addressing in watch_logs
Signed-off-by: Artem Lytvyn <[email protected]>
* fix(server): fix public rpc schema
Signed-off-by: Artem Lytvyn <[email protected]>
* fix(api): reconcile watch resume rebase
Signed-off-by: John Myers <[email protected]>
* fix(server): bound interactive relay cleanup
Signed-off-by: John Myers <[email protected]>
---------
Signed-off-by: Artem Lytvyn <[email protected]>
Signed-off-by: John Myers <[email protected]>
Co-authored-by: John Myers <[email protected]>1 parent 8835777 commit 470a346
21 files changed
Lines changed: 3180 additions & 158 deletions
File tree
- architecture
- crates
- openshell-cli
- src
- tests
- openshell-sdk
- src
- tests
- openshell-server/src
- compute
- grpc
- docs/observability
- proto
- sdk/go/proto/openshellv1
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
440 | 440 | | |
441 | 441 | | |
442 | 442 | | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
443 | 519 | | |
444 | 520 | | |
445 | 521 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
809 | 809 | | |
810 | 810 | | |
811 | 811 | | |
| 812 | + | |
812 | 813 | | |
813 | 814 | | |
814 | 815 | | |
| |||
3613 | 3614 | | |
3614 | 3615 | | |
3615 | 3616 | | |
| 3617 | + | |
3616 | 3618 | | |
3617 | 3619 | | |
3618 | 3620 | | |
| |||
5847 | 5849 | | |
5848 | 5850 | | |
5849 | 5851 | | |
| 5852 | + | |
5850 | 5853 | | |
5851 | 5854 | | |
5852 | 5855 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
743 | 743 | | |
744 | 744 | | |
745 | 745 | | |
| 746 | + | |
746 | 747 | | |
747 | 748 | | |
748 | 749 | | |
| |||
753 | 754 | | |
754 | 755 | | |
755 | 756 | | |
| 757 | + | |
756 | 758 | | |
757 | 759 | | |
758 | 760 | | |
| |||
764 | 766 | | |
765 | 767 | | |
766 | 768 | | |
| 769 | + | |
767 | 770 | | |
768 | 771 | | |
769 | 772 | | |
| |||
777 | 780 | | |
778 | 781 | | |
779 | 782 | | |
| 783 | + | |
780 | 784 | | |
781 | 785 | | |
782 | 786 | | |
783 | 787 | | |
784 | 788 | | |
| 789 | + | |
785 | 790 | | |
786 | 791 | | |
787 | 792 | | |
| |||
801 | 806 | | |
802 | 807 | | |
803 | 808 | | |
| 809 | + | |
804 | 810 | | |
805 | 811 | | |
806 | 812 | | |
807 | 813 | | |
808 | 814 | | |
809 | 815 | | |
| 816 | + | |
810 | 817 | | |
811 | 818 | | |
812 | 819 | | |
| |||
815 | 822 | | |
816 | 823 | | |
817 | 824 | | |
| 825 | + | |
818 | 826 | | |
819 | 827 | | |
820 | 828 | | |
| |||
829 | 837 | | |
830 | 838 | | |
831 | 839 | | |
| 840 | + | |
832 | 841 | | |
833 | 842 | | |
834 | 843 | | |
| |||
840 | 849 | | |
841 | 850 | | |
842 | 851 | | |
| 852 | + | |
843 | 853 | | |
844 | 854 | | |
845 | 855 | | |
846 | 856 | | |
847 | 857 | | |
848 | 858 | | |
| 859 | + | |
849 | 860 | | |
850 | 861 | | |
851 | 862 | | |
| |||
857 | 868 | | |
858 | 869 | | |
859 | 870 | | |
| 871 | + | |
860 | 872 | | |
861 | 873 | | |
862 | 874 | | |
863 | 875 | | |
864 | 876 | | |
| 877 | + | |
865 | 878 | | |
866 | 879 | | |
867 | 880 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| |||
0 commit comments