Skip to content

Commit 2900b06

Browse files
rhussletv1nnn
andauthored
feat(go-sdk): WatchLogs (#10)
* feat(go-sdk): WatchLogs Signed-off-by: Artem Lytvyn <[email protected]> * fix(go-sdk): wait the WatchLogs reconnect backoff once and keep it cancellable Signed-off-by: letv1nnn <[email protected]> * fix(go-sdk): reset the WatchLogs backoff only after a delivered item Signed-off-by: letv1nnn <[email protected]> * fix(sandbox): move WatchLogs to an optional SandboxLogsWatchInterface Signed-off-by: Artem Lytvyn <[email protected]> * refactor(sandbox): inject the WatchLogs backoff wait and tighten its tests Signed-off-by: Artem Lytvyn <[email protected]> * docs(sandbox): document WatchLogs ordering, retry and OutOfRange semantics Signed-off-by: Artem Lytvyn <[email protected]> * test(fake): cover the WatchLogs stub Signed-off-by: Artem Lytvyn <[email protected]> --------- Signed-off-by: Artem Lytvyn <[email protected]> Signed-off-by: letv1nnn <[email protected]> Signed-off-by: Artem Lytvyn <[email protected]> Co-authored-by: Artem Lytvyn <[email protected]> Co-authored-by: letv1nnn <[email protected]>
1 parent 27895b9 commit 2900b06

24 files changed

Lines changed: 1384 additions & 3 deletions

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -321,6 +321,7 @@ stream completion.
321321
| File transfer API (transport capability-gated) | `FileInterface` | [Files](https://ai-openshell.github.io/sdk-go/api/files.html) |
322322
| Policy management (draft review, approve, reject, merge, global policy) | `PolicyInterface` | [Policy](https://ai-openshell.github.io/sdk-go/api/policy.html) |
323323
| Sandbox logs (streaming retrieval) | `SandboxInterface` | [Sandboxes](https://ai-openshell.github.io/sdk-go/api/sandboxes.html) |
324+
| Resumable log and platform event watch (loss-aware reconnect) | `SandboxLogsWatchInterface` | [Sandboxes](https://ai-openshell.github.io/sdk-go/api/sandboxes.html) |
324325
| Workspace management (create, get, list, delete, members) | `WorkspaceInterface` | [Workspaces](https://ai-openshell.github.io/sdk-go/api/workspaces.html) |
325326
| Sandbox provider attachment (attach, detach, list) | `SandboxInterface` | [Sandboxes](https://ai-openshell.github.io/sdk-go/api/sandboxes.html) |
326327
| Gateway info and current user identity | `HealthInterface` | [Health](https://ai-openshell.github.io/sdk-go/api/health.html) |

‎docs/src/api/sandboxes.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -264,6 +264,83 @@ for event := range watcher.ResultChan() {
264264
// Channel closes after Ready or Error
265265
```
266266

267+
## WatchLogs
268+
269+
Streams a sandbox's log lines and platform events with loss-aware resume. Returns a `WatchInterface[*WatchLogEvent]`.
270+
271+
`WatchLogs` lives on the optional `SandboxLogsWatchInterface`, reached through `client.SandboxLogsWatch()`, so `SandboxInterface` stays unchanged for existing mocks and wrappers.
272+
273+
Each delivered item carries an opaque `Cursor`. The watcher tracks the highest cursor it has seen, reconnects transparently when the stream drops, and asks the gateway to replay only what follows that cursor, so a reconnect neither loses nor duplicates events. The gateway delivers live log and platform events in cursor order across both sources, which is what makes the highest cursor a safe resume point.
274+
275+
Only gRPC `Unavailable` (a dropped connection or gateway restart) is retried, with exponential backoff from 100ms capped at 2s, and it is retried indefinitely. Nothing is emitted while the watcher reconnects, so a gateway that stays down looks the same as a quiet sandbox. Bound the watch with a context deadline, or call `Stop`, when that matters. This differs from `Watch`, which reports `Unavailable` as an `EventError` immediately. Every other error ends the stream with an `EventError`.
276+
277+
Every `Event[*WatchLogEvent]` has `Type` `EventAdded`, except a terminal failure, which arrives as `EventError` with `Err` set and no `Object`. The `Object.Kind` field selects the populated payload:
278+
279+
| Kind | Payload | Meaning |
280+
|------|---------|---------|
281+
| `WatchLogKindLog` | `Log *LogLine` | A log line |
282+
| `WatchLogKindEvent` | `Event *PlatformEvent` | A runtime event from the compute backend |
283+
| `WatchLogKindWarning` | `Warning string` | Recoverable loss — the server skipped ahead after a lag and the stream continues |
284+
285+
**Available options** (`WatchLogsOptions`):
286+
287+
| Field | Description |
288+
|-------|-------------|
289+
| `FollowLogs bool` | Stream log lines. Defaults to true when neither follow flag is set |
290+
| `FollowEvents bool` | Stream platform events |
291+
| `LogSources []string` | Filter log lines by source (e.g. `"gateway"`, `"sandbox"`) |
292+
| `LogMinLevel string` | Minimum log level (e.g. `"WARN"`) |
293+
| `ResumeAfterCursor string` | Replay only what follows a cursor from an earlier watch |
294+
| `LogTailLines uint32` | Bound the initial log backfill. Ignored when resuming from a cursor |
295+
| `EventTail uint32` | Bound the initial event backfill. Ignored when resuming from a cursor |
296+
297+
```go
298+
watcher, err := client.SandboxLogsWatch().WatchLogs(ctx, "default", "my-sandbox", v1.WatchLogsOptions{
299+
FollowLogs: true,
300+
FollowEvents: true,
301+
LogTailLines: 100,
302+
})
303+
if err != nil {
304+
log.Fatal(err)
305+
}
306+
defer watcher.Stop()
307+
308+
var cursor string
309+
for event := range watcher.ResultChan() {
310+
if event.Type == v1.EventError {
311+
if v1.IsOutOfRange(event.Err) {
312+
// Unrecoverable gap: events after the resume point are gone.
313+
fmt.Println("log history was trimmed; some events were lost")
314+
}
315+
break
316+
}
317+
switch item := event.Object; item.Kind {
318+
case v1.WatchLogKindLog:
319+
fmt.Printf("[%s] %s\n", item.Log.Level, item.Log.Message)
320+
case v1.WatchLogKindEvent:
321+
fmt.Printf("event %s: %s\n", item.Event.Reason, item.Event.Message)
322+
case v1.WatchLogKindWarning:
323+
fmt.Printf("warning: %s\n", item.Warning)
324+
}
325+
if item := event.Object; item.Cursor > cursor {
326+
cursor = item.Cursor
327+
}
328+
}
329+
```
330+
331+
Loss is never silent. A recoverable gap arrives as a `WatchLogKindWarning` item and the stream continues. An unrecoverable gap ends the stream with an `EventError` carrying `ErrorOutOfRange`: the resume point is gone, so the events after it cannot be replayed. The SDK does not retry that error, because restarting from the tail would hide the loss. To continue anyway, start a new watch with an empty `ResumeAfterCursor`; retrying the same cursor fails identically.
332+
333+
Persist the highest cursor to resume across process restarts:
334+
335+
```go
336+
watcher, err := client.SandboxLogsWatch().WatchLogs(ctx, "default", "my-sandbox", v1.WatchLogsOptions{
337+
FollowLogs: true,
338+
ResumeAfterCursor: savedCursor,
339+
})
340+
```
341+
342+
Cursors are opaque. Do not parse or construct one. The only supported operation is comparing two cursors observed on the same stream and keeping the greater one.
343+
267344
## GetLogs
268345

269346
Retrieves log entries from a sandbox. The sandbox is looked up by name (the SDK resolves the name to an internal ID automatically). Use functional options to filter results.

‎docs/src/error-handling.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,9 @@ Use these top-level functions to check error types. They work with wrapped error
4242
| `v1.IsInvalidArgument` | `ErrorInvalidArgument` | Invalid request parameters |
4343
| `v1.IsDeadlineExceeded` | `ErrorDeadlineExceeded` | Operation timed out |
4444
| `v1.IsCancelled` | `ErrorCancelled` | Operation was cancelled (context cancellation) |
45+
| `v1.IsOutOfRange` | `ErrorOutOfRange` | Operation attempted past a valid range (gRPC `OUT_OF_RANGE`). On `WatchLogs`, the resume cursor is gone and the events after it are unrecoverable |
46+
47+
gRPC `OUT_OF_RANGE` maps to `ErrorOutOfRange` on every RPC. Earlier releases mapped it to `ErrorInvalidArgument`, so code that matched `IsInvalidArgument` for an out-of-range result should match `IsOutOfRange` instead.
4548

4649
For `ErrorInternal` (server-side errors), no convenience predicate exists. Match it directly via the `Code` field:
4750

‎openshell/v1/client.go‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ type ClientInterface interface {
2222
SandboxTemplates() SandboxTemplateInterface
2323
CreateSandboxFromTemplate(ctx context.Context, workspace, name, templateName string, spec *SandboxSpec, opts ...CreateOption) (*Sandbox, error)
2424
SandboxSuspendResume() SandboxSuspendResumeInterface
25+
SandboxLogsWatch() SandboxLogsWatchInterface
2526
Providers() ProviderInterface
2627
Services() ServiceInterface
2728
Exec() ExecInterface
@@ -55,6 +56,7 @@ type Client struct {
5556
sandboxes SandboxInterface
5657
templateCreate SandboxTemplateCreateInterface
5758
suspendResume SandboxSuspendResumeInterface
59+
logsWatch SandboxLogsWatchInterface
5860
templates SandboxTemplateInterface
5961
providers ProviderInterface
6062
services ServiceInterface
@@ -102,6 +104,7 @@ func NewClient(cfg Config) (*Client, error) {
102104
c.sandboxes = sandboxes
103105
c.templateCreate = sandboxes
104106
c.suspendResume = sandboxes
107+
c.logsWatch = sandboxes
105108
c.templates = newSandboxTemplateClient(conn)
106109
c.providers = newProviderClient(conn)
107110
c.services = newServiceClient(conn)
@@ -132,6 +135,9 @@ func (c *Client) CreateSandboxFromTemplate(ctx context.Context, workspace, name,
132135
// SandboxSuspendResume returns the sandbox suspend and resume sub-client.
133136
func (c *Client) SandboxSuspendResume() SandboxSuspendResumeInterface { return c.suspendResume }
134137

138+
// SandboxLogsWatch returns the resumable sandbox log and event watch sub-client.
139+
func (c *Client) SandboxLogsWatch() SandboxLogsWatchInterface { return c.logsWatch }
140+
135141
// Providers returns the provider sub-client.
136142
func (c *Client) Providers() ProviderInterface { return c.providers }
137143

‎openshell/v1/errors.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ const (
2323
ErrorUnimplemented = types.ErrorUnimplemented
2424
ErrorConflict = types.ErrorConflict
2525
ErrorUnauthenticated = types.ErrorUnauthenticated
26+
ErrorOutOfRange = types.ErrorOutOfRange
2627
)
2728

2829
// StatusError is the typed error returned by all SDK operations.
@@ -64,3 +65,7 @@ func IsConflict(err error) bool { return types.IsConflict(err) }
6465

6566
// IsUnauthenticated returns true if the error indicates invalid or missing credentials.
6667
func IsUnauthenticated(err error) bool { return types.IsUnauthenticated(err) }
68+
69+
// IsOutOfRange returns true if the error indicates a resume cursor is no longer
70+
// available and the events after it are unrecoverable.
71+
func IsOutOfRange(err error) bool { return types.IsOutOfRange(err) }

‎openshell/v1/errors_test.go‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,8 @@ func TestErrorCode_String(t *testing.T) {
120120
{ErrorInternal, "Internal"},
121121
{ErrorUnimplemented, "Unimplemented"},
122122
{ErrorConflict, "Conflict"},
123+
{ErrorUnauthenticated, "Unauthenticated"},
124+
{ErrorOutOfRange, "OutOfRange"},
123125
}
124126
for _, tt := range tests {
125127
assert.Equal(t, tt.want, tt.code.String())

‎openshell/v1/fake/fake.go‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ type Client struct {
2626
sandboxes v1.SandboxInterface
2727
templateCreate v1.SandboxTemplateCreateInterface
2828
suspendResume v1.SandboxSuspendResumeInterface
29+
logsWatch v1.SandboxLogsWatchInterface
2930
templates v1.SandboxTemplateInterface
3031
providers v1.ProviderInterface
3132
services v1.ServiceInterface
@@ -86,6 +87,7 @@ func NewClient(opts ...ClientOption) *Client {
8687
fc.sandboxes = sandboxes
8788
fc.templateCreate = sandboxes
8889
fc.suspendResume = sandboxes
90+
fc.logsWatch = sandboxes
8991
fc.templates = newFakeSandboxTemplateClient(fc.templateStore, fc.isClosed)
9092
fc.providers = newFakeProviderClient(fc.providerStore, fc.isClosed)
9193
fc.services = newFakeServiceClient(fc.isClosed)
@@ -128,6 +130,11 @@ func (fc *Client) SandboxSuspendResume() v1.SandboxSuspendResumeInterface {
128130
return fc.suspendResume
129131
}
130132

133+
// SandboxLogsWatch returns the resumable sandbox log and event watch sub-client.
134+
func (fc *Client) SandboxLogsWatch() v1.SandboxLogsWatchInterface {
135+
return fc.logsWatch
136+
}
137+
131138
// Providers returns the provider sub-client.
132139
func (fc *Client) Providers() v1.ProviderInterface { return fc.providers }
133140

‎openshell/v1/fake/fake_test.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,7 @@ func TestFakeClient_SubClients(t *testing.T) {
116116
fc := NewClient()
117117

118118
assert.NotNil(t, fc.Sandboxes())
119+
assert.NotNil(t, fc.SandboxLogsWatch())
119120
assert.NotNil(t, fc.Providers())
120121
assert.NotNil(t, fc.Exec())
121122
assert.NotNil(t, fc.Files())

‎openshell/v1/fake/sandbox.go‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -293,6 +293,7 @@ type fakeSandboxClient struct {
293293
var _ v1.SandboxInterface = (*fakeSandboxClient)(nil)
294294
var _ v1.SandboxTemplateCreateInterface = (*fakeSandboxClient)(nil)
295295
var _ v1.SandboxSuspendResumeInterface = (*fakeSandboxClient)(nil)
296+
var _ v1.SandboxLogsWatchInterface = (*fakeSandboxClient)(nil)
296297

297298
// newFakeSandboxClient creates a new fakeSandboxClient.
298299
func newFakeSandboxClient(
@@ -899,6 +900,15 @@ func (c *fakeSandboxClient) GetLogs(_ context.Context, _, _ string, _ ...v1.LogO
899900
return nil, &types.StatusError{Code: types.ErrorUnimplemented, Message: "GetLogs not implemented in fake client"}
900901
}
901902

903+
// WatchLogs returns Unimplemented — the fake client has no log or platform
904+
// event stream to replay, so it cannot honor the cursor resume contract.
905+
func (c *fakeSandboxClient) WatchLogs(_ context.Context, _, _ string, _ ...v1.WatchLogsOptions) (types.WatchInterface[*types.WatchLogEvent], error) {
906+
if c.closedFunc() {
907+
return nil, &types.StatusError{Code: types.ErrorUnavailable, Message: "client is closed"}
908+
}
909+
return nil, &types.StatusError{Code: types.ErrorUnimplemented, Message: "WatchLogs not implemented in fake client"}
910+
}
911+
902912
// ListProviders returns a pager over stub Provider objects attached to the
903913
// sandbox. The returned providers contain only the Name field, since the fake
904914
// client does not maintain a full provider registry per sandbox.

‎openshell/v1/fake/sandbox_test.go‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1008,6 +1008,25 @@ func TestSandbox_GetLogs_ClosedReturnsUnavailable(t *testing.T) {
10081008
assert.True(t, types.IsUnavailable(err))
10091009
}
10101010

1011+
func TestSandbox_WatchLogs_ReturnsUnimplemented(t *testing.T) {
1012+
sc := newTestSandboxClient()
1013+
w, err := sc.WatchLogs(context.Background(), "default", "sb-1")
1014+
require.Error(t, err)
1015+
assert.Nil(t, w)
1016+
assert.True(t, types.IsUnimplemented(err))
1017+
}
1018+
1019+
func TestSandbox_WatchLogs_ClosedReturnsUnavailable(t *testing.T) {
1020+
store := newobjectStore(sandboxName, copySandbox)
1021+
templateStore := newobjectStore(sandboxWorkloadTemplate, copySandboxWorkloadTemplate)
1022+
broadcaster := newWatchBroadcaster[*types.Sandbox]()
1023+
sc := newFakeSandboxClient(store, templateStore, broadcaster, func() bool { return true })
1024+
w, err := sc.WatchLogs(context.Background(), "default", "sb-1")
1025+
require.Error(t, err)
1026+
assert.Nil(t, w)
1027+
assert.True(t, types.IsUnavailable(err))
1028+
}
1029+
10111030
func TestSandbox_SuspendAndResume(t *testing.T) {
10121031
sc := newTestSandboxClient()
10131032
ctx := context.Background()

0 commit comments

Comments
 (0)