Skip to content

Commit a093946

Browse files
test(windows): add gateway-restart scenario to the MXC e2e runner
Addresses review feedback on NVIDIA#5299: cover a graceful gateway shutdown and restart with persisted state, not only a hard kill, and record what happens to the workload and the sandbox record. Add a default process_container scenario to run-mxc-e2e.ps1. It runs the gateway on a SQLite file, creates a sandbox with a long-running marked workload, sends a real Ctrl-C, and requires the gateway to log a graceful shutdown and exit and every recorded sandbox process to end. It then restarts the gateway on the same database and requires the sandbox record to still be listed, its workload not to be running (MXC has no restart recovery: gateway_manages_lifecycle is false, so the gateway's shutdown sweep and startup recovery are skipped), and the record to be deletable with its name reusable. The scenario pins this no-recovery behavior and needs updating when recovery is added. Start-Gw takes an optional database URL (default stays in-memory) and clears an inherited "ignore Ctrl-C" setting, which a session such as ssh can pass to the gateway and which would otherwise stop it from ever seeing a graceful-stop request. gateway-kill now shares the new Wait-SandboxTree and Wait-TreeGone helpers with the new scenario. Tests: on Windows x64 the scenario passes (6 sandbox processes ended on Ctrl-C, gateway exit code 0, record persisted in phase Provisioning and not recovered, delete and re-create work). gateway-kill, fs-rw and fs-readonly still pass after the refactor. A graceful stop ends the workloads with or without the per-generation job (checked against a gateway built without it), so this scenario documents graceful behavior; the job is what covers a hard kill. The runner's existing port probe (GetActiveTcpListeners) does not see the listener on the test machine (issue NVIDIA#11), so the runs used a copy of the runner with a loopback connect probe; the probe in the repository is unchanged. Refs NVIDIA#5298 Signed-off-by: ashwind <[email protected]>
1 parent 3e19fbe commit a093946

2 files changed

Lines changed: 177 additions & 33 deletions

File tree

‎crates/openshell-driver-mxc/README.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -382,6 +382,18 @@ gateway are tracked, so other sandboxes on the host are left alone. It relies on
382382
each sandbox generation's kill-on-close job object; no workload outlives the
383383
gateway. Select it with `-Scenario gateway-kill`.
384384

385+
The default `gateway-restart` scenario covers a graceful gateway stop with
386+
persisted state. It runs the gateway on a SQLite file, creates a sandbox with a
387+
long-running workload, sends a real Ctrl-C, and requires the gateway to log a
388+
graceful shutdown and exit, every recorded sandbox process to end, and a restart
389+
on the same database to start. After the restart the sandbox record must still be
390+
listed (the observed phase is `Provisioning` and does not change), its workload
391+
must not be running (MXC has no restart recovery), and the record must be
392+
deletable with its name reusable. The scenario pins today's no-recovery behavior,
393+
so it needs updating when recovery is added. A graceful stop ends the workloads
394+
with or without the per-generation job; the job is what covers a hard kill.
395+
Select it with `-Scenario gateway-restart`.
396+
385397
The `ws-agent.yaml` and `openclaw-gateway.yaml` policies preserve the Windows
386398
branch's workload grants. `inference.yaml` is a template: replace its filesystem,
387399
curl executable, inference host, and port placeholders before use. Authorize

‎crates/openshell-driver-mxc/examples/run-mxc-e2e.ps1‎

Lines changed: 165 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,10 @@
3939
# host-service - workload reaches an HTTP service on host 127.0.0.1.
4040
# gateway-kill - hard-killing the gateway ends every sandbox process and
4141
# frees its port. processcontainer only.
42+
# gateway-restart - a graceful gateway stop (Ctrl-C) ends every sandbox process;
43+
# after a restart on the same database the sandbox record is
44+
# still there but its workload is not recovered, and the
45+
# record can be deleted and its name reused. processcontainer only.
4246
# https-ca - opt-in outbound HTTPS using the injected proxy CA.
4347

4448
[CmdletBinding()]
@@ -317,8 +321,12 @@ function Test-PortListening {
317321
return @([Net.NetworkInformation.IPGlobalProperties]::GetIPGlobalProperties().GetActiveTcpListeners() | Where-Object { $_.Port -eq $Port }).Count -gt 0
318322
}
319323

320-
function Start-Gw {
324+
function Start-Gw([string]$DbUrl = "sqlite::memory:") {
321325
Remove-Item $gwLog, $gwErrLog -Force -ErrorAction SilentlyContinue
326+
# A parent that ignores Ctrl-C (some ssh/service sessions do) passes that on to
327+
# the gateway, which would then never see a graceful-stop request. Clear it for
328+
# this process so the gateway starts with the default handling.
329+
[void][OpenShellMxcCtrlC]::EnableCtrlC()
322330
# Ephemeral in-memory DB: this is a test harness, so it must NOT write sandbox
323331
# records to the persistent default store (%LOCALAPPDATA%\openshell\gateway\
324332
# openshell.db). Without this, sandbox names persist across gateway restarts
@@ -329,7 +337,7 @@ function Start-Gw {
329337
# config path containing a space gets split and the gateway's arg parser rejects it.
330338
$env:OPENSHELL_GATEWAY_CONFIG = $toml
331339
$p = Start-Process -FilePath $gateway `
332-
-ArgumentList @("--disable-tls", "--db-url", "sqlite::memory:", "--port", $Port, "--log-level", "info") `
340+
-ArgumentList @("--disable-tls", "--db-url", $DbUrl, "--port", $Port, "--log-level", "info") `
333341
-WorkingDirectory $here -PassThru -WindowStyle Hidden `
334342
-RedirectStandardOutput $gwLog -RedirectStandardError $gwErrLog
335343
$deadline = (Get-Date).AddSeconds(30)
@@ -339,6 +347,7 @@ function Start-Gw {
339347
throw "gateway exited early (code $($p.ExitCode)). See $gwLog."
340348
}
341349
if (Test-PortListening) {
350+
$null = $p.Handle # keep the handle so ExitCode is available later
342351
return $p
343352
}
344353
Start-Sleep -Milliseconds 400
@@ -547,6 +556,90 @@ function Test-ServiceForwarding([string]$sandboxName, [string]$readyFile, [strin
547556
# Detect an agent *launch* failure (binary not found / not implemented) vs a
548557
# legitimate policy denial. Used to avoid false-passing a deny scenario when the
549558
# agent never actually ran.
559+
if (-not ("OpenShellMxcCtrlC" -as [type])) {
560+
Add-Type -TypeDefinition @'
561+
using System;
562+
using System.Runtime.InteropServices;
563+
564+
public static class OpenShellMxcCtrlC
565+
{
566+
[DllImport("kernel32.dll", SetLastError = true)]
567+
[return: MarshalAs(UnmanagedType.Bool)]
568+
static extern bool SetConsoleCtrlHandler(IntPtr handler, bool add);
569+
570+
// Restore default Ctrl-C handling for this process (and what it starts).
571+
public static bool EnableCtrlC() { return SetConsoleCtrlHandler(IntPtr.Zero, false); }
572+
}
573+
'@
574+
}
575+
576+
# Deliver a real Ctrl-C to another console process. Attaching to its console
577+
# detaches the caller from its own, so this runs in a throwaway PowerShell.
578+
function Send-CtrlC([int]$ProcessId) {
579+
$script = @"
580+
Add-Type -TypeDefinition @'
581+
using System; using System.Runtime.InteropServices;
582+
public static class SendCtrlC {
583+
[DllImport("kernel32.dll", SetLastError=true)] static extern bool FreeConsole();
584+
[DllImport("kernel32.dll", SetLastError=true)] static extern bool AttachConsole(uint pid);
585+
[DllImport("kernel32.dll", SetLastError=true)] static extern bool SetConsoleCtrlHandler(IntPtr h, bool add);
586+
[DllImport("kernel32.dll", SetLastError=true)] static extern bool GenerateConsoleCtrlEvent(uint ev, uint grp);
587+
public static string Run(uint pid) {
588+
FreeConsole();
589+
if (!AttachConsole(pid)) return "AttachConsole failed: " + Marshal.GetLastWin32Error();
590+
SetConsoleCtrlHandler(IntPtr.Zero, true);
591+
bool ok = GenerateConsoleCtrlEvent(0, 0);
592+
int err = Marshal.GetLastWin32Error();
593+
FreeConsole();
594+
return ok ? "sent" : "GenerateConsoleCtrlEvent failed: " + err;
595+
}
596+
}
597+
'@
598+
[SendCtrlC]::Run($ProcessId)
599+
"@
600+
$encoded = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($script))
601+
$result = Invoke-NativeCaptured "powershell.exe" @("-NoProfile", "-NonInteractive", "-EncodedCommand", $encoded) 30
602+
# An encoded command also emits CLIXML progress noise on stderr; the verdict is
603+
# the one line that starts with one of the helper's results.
604+
$verdict = @($result.Output | ForEach-Object { "$_".Trim() } | Where-Object { $_ -match '^(sent|AttachConsole failed|GenerateConsoleCtrlEvent failed)' })
605+
if ($verdict.Count -eq 0) { return "no result from the Ctrl-C helper (exit $($result.ExitCode))" }
606+
return $verdict[0]
607+
}
608+
609+
# Wait until a sandbox's whole process tree exists, then return it as
610+
# @{ pid = name }. Only processes under $gwProc, plus the process whose command
611+
# line carries $nonce, are included, so unrelated MXC sandboxes on the host are
612+
# never counted or touched. Throws if the tree does not come up.
613+
function Wait-SandboxTree($gwProc, [string]$nonce) {
614+
$victims = @{}
615+
$deadline = (Get-Date).AddSeconds(60)
616+
do {
617+
Start-Sleep -Milliseconds 500
618+
$victims = @{}
619+
foreach ($id in (Get-DescendantPids $gwProc.Id)) {
620+
$proc = Get-Process -Id $id -ErrorAction SilentlyContinue
621+
if ($proc) { $victims[$id] = $proc.ProcessName }
622+
}
623+
$marked = @(Get-CimInstance Win32_Process | Where-Object { $_.CommandLine -like "*$nonce*" -and $_.ProcessId -ne $PID })
624+
foreach ($proc in $marked) { $victims[[int]$proc.ProcessId] = $proc.Name }
625+
$names = @($victims.Values | ForEach-Object { $_ -replace '\.exe$', '' })
626+
$ready = ($names -contains 'wxc-exec') -and ($names -contains 'openshell-supervisor') -and ($marked.Count -gt 0)
627+
} while (-not $ready -and (Get-Date) -lt $deadline)
628+
if (-not $ready) { throw "sandbox process tree did not come up (saw: $(($victims.Values) -join ', '))" }
629+
return $victims
630+
}
631+
632+
# Processes from a recorded tree that are still alive after waiting up to
633+
# $seconds for them to end.
634+
function Wait-TreeGone($victims, [int]$seconds = 30) {
635+
$deadline = (Get-Date).AddSeconds($seconds)
636+
do {
637+
Start-Sleep -Milliseconds 500
638+
$survivors = @($victims.Keys | Where-Object { Get-Process -Id $_ -ErrorAction SilentlyContinue })
639+
} while ($survivors.Count -gt 0 -and (Get-Date) -lt $deadline)
640+
return ($survivors | ForEach-Object { "$($victims[$_])($_)" })
641+
}
642+
550643
# All descendants of a process, found by walking ParentProcessId links in one
551644
# snapshot. Orphans keep their ParentProcessId, so the walk is still correct
552645
# for a tree whose root has since died, but it must run while the root lives.
@@ -814,6 +907,11 @@ try {
814907
Name = 'gateway-kill'; PolicyFile = Join-Path $policyDir 'fs-rw.yaml'
815908
SandboxId = 'gk'; Backends = 'process_container'; Kind = 'gateway-kill'
816909
Description = 'hard-killing the gateway ends every sandbox process and frees its port'
910+
},
911+
@{
912+
Name = 'gateway-restart'; PolicyFile = Join-Path $policyDir 'fs-rw.yaml'
913+
SandboxId = 'gr'; Backends = 'process_container'; Kind = 'gateway-restart'
914+
Description = 'graceful gateway stop ends sandbox processes; the record persists but is not recovered'
817915
}
818916
)
819917

@@ -940,7 +1038,7 @@ try {
9401038
} else {
9411039
$command = @($cmdExe, "/c", "echo denied 1> $denied")
9421040
}
943-
} elseif ($sc.Kind -eq 'gateway-kill') {
1041+
} elseif ($sc.Kind -in @('gateway-kill', 'gateway-restart')) {
9441042
# A long-running workload carrying a unique marker, so its process
9451043
# can be identified without matching unrelated cmd.exe instances.
9461044
$killNonce = [guid]::NewGuid().ToString('N')
@@ -1003,7 +1101,11 @@ try {
10031101
# policyUsed already lives in the result bundle and is the exact
10041102
# rendered policy passed to OpenShell.
10051103

1006-
$gw = Start-Gw
1104+
# gateway-restart needs state that survives the gateway; every other
1105+
# scenario keeps the ephemeral in-memory database.
1106+
$persistedDb = Join-Path $resultDir "gateway.$($sc.Name).db"
1107+
$gwDbUrl = if ($sc.Kind -eq 'gateway-restart') { "sqlite:${persistedDb}?mode=rwc" } else { "sqlite::memory:" }
1108+
$gw = Start-Gw $gwDbUrl
10071109
Info "gateway pid $($gw.Id)"
10081110
Register-Cli
10091111

@@ -1019,7 +1121,7 @@ try {
10191121
"--policy", [string]$policyUsed,
10201122
"--driver-config-json", $driverConfig
10211123
)
1022-
$createArguments += if ($sc.Kind -in @('host-service', 'gateway-kill')) { '--detach' } else { '--no-tty' }
1124+
$createArguments += if ($sc.Kind -in @('host-service', 'gateway-kill', 'gateway-restart')) { '--detach' } else { '--no-tty' }
10231125
$createResult = Invoke-NativeCaptured $cli $createArguments
10241126
$createOut = $createResult.Output
10251127
$createExitCode = $createResult.ExitCode
@@ -1047,39 +1149,15 @@ try {
10471149
try {
10481150
if ($createExitCode -ne 0) { throw "sandbox creation failed: $createOutStr" }
10491151
# Wait for the whole sandbox tree to exist, then record it.
1050-
# Only processes under THIS gateway (plus the marked workload)
1051-
# are tracked, so unrelated MXC sandboxes on the host are never
1052-
# counted or touched.
1053-
$victims = @{}
1054-
$deadline = (Get-Date).AddSeconds(60)
1055-
do {
1056-
Start-Sleep -Milliseconds 500
1057-
$victims = @{}
1058-
foreach ($id in (Get-DescendantPids $gw.Id)) {
1059-
$proc = Get-Process -Id $id -ErrorAction SilentlyContinue
1060-
if ($proc) { $victims[$id] = $proc.ProcessName }
1061-
}
1062-
foreach ($proc in @(Get-CimInstance Win32_Process | Where-Object { $_.CommandLine -like "*$killNonce*" -and $_.ProcessId -ne $PID })) {
1063-
$victims[[int]$proc.ProcessId] = $proc.Name
1064-
}
1065-
$names = @($victims.Values | ForEach-Object { $_ -replace '\.exe$', '' })
1066-
$ready = ($names -contains 'wxc-exec') -and ($names -contains 'openshell-supervisor') -and (@(Get-CimInstance Win32_Process | Where-Object { $_.CommandLine -like "*$killNonce*" -and $_.ProcessId -ne $PID }).Count -gt 0)
1067-
} while (-not $ready -and (Get-Date) -lt $deadline)
1068-
if (-not $ready) { throw "sandbox process tree did not come up (saw: $(($victims.Values) -join ', '))" }
1152+
$victims = Wait-SandboxTree $gw $killNonce
10691153
Info "tracking $($victims.Count) sandbox processes: $((($victims.GetEnumerator() | Sort-Object Name | ForEach-Object { "$($_.Value)($($_.Key))" })) -join ', ')"
10701154

10711155
Info "hard-killing gateway pid $($gw.Id)"
10721156
Stop-Process -Id $gw.Id -Force
10731157
$gw.WaitForExit()
1074-
$survivors = @()
1075-
$deadline = (Get-Date).AddSeconds(30)
1076-
do {
1077-
Start-Sleep -Milliseconds 500
1078-
$survivors = @($victims.Keys | Where-Object { Get-Process -Id $_ -ErrorAction SilentlyContinue })
1079-
} while ($survivors.Count -gt 0 -and (Get-Date) -lt $deadline)
1158+
$survivors = @(Wait-TreeGone $victims 30)
10801159
if ($survivors.Count -gt 0) {
1081-
$left = ($survivors | ForEach-Object { "$($victims[$_])($_)" }) -join ', '
1082-
throw "sandbox processes survived the gateway's death: $left"
1160+
throw "sandbox processes survived the gateway's death: $($survivors -join ', ')"
10831161
}
10841162
# The port must be free again: a replacement gateway must bind it.
10851163
$gw = Start-Gw
@@ -1090,6 +1168,57 @@ try {
10901168
Bad "gateway-kill: $($_.Exception.Message)"
10911169
$results += [pscustomobject]@{ Scenario = $sc.Name; Result = 'FAIL'; Reason = $_.Exception.Message }
10921170
}
1171+
} elseif ($sc.Kind -eq 'gateway-restart') {
1172+
# Documents today's behavior (MXC has no restart recovery): a graceful
1173+
# gateway stop ends the workloads, the persisted record is left
1174+
# behind, and nothing restarts it. Revisit the "not recovered" check
1175+
# when recovery is added.
1176+
try {
1177+
if ($createExitCode -ne 0) { throw "sandbox creation failed: $createOutStr" }
1178+
$victims = Wait-SandboxTree $gw $killNonce
1179+
Info "tracking $($victims.Count) sandbox processes: $((($victims.GetEnumerator() | Sort-Object Name | ForEach-Object { "$($_.Value)($($_.Key))" })) -join ', ')"
1180+
1181+
Info "sending Ctrl-C to gateway pid $($gw.Id)"
1182+
$sent = Send-CtrlC $gw.Id
1183+
if ($sent -ne 'sent') { throw "could not send Ctrl-C to the gateway: $sent" }
1184+
if (-not $gw.WaitForExit(60000)) { throw 'gateway did not exit within 60 s of Ctrl-C (no graceful shutdown)' }
1185+
Info "gateway exited (exit code $($gw.ExitCode))"
1186+
if (-not (Select-String -Path $gwLog, $gwErrLog -Pattern 'Shutdown signal received' -Quiet -ErrorAction SilentlyContinue)) {
1187+
throw 'gateway exited without logging a graceful shutdown (it was not a graceful stop)'
1188+
}
1189+
$survivors = @(Wait-TreeGone $victims 30)
1190+
if ($survivors.Count -gt 0) {
1191+
throw "sandbox processes survived a graceful gateway stop: $($survivors -join ', ')"
1192+
}
1193+
Copy-Item $gwLog "$gwLog.first" -Force -ErrorAction SilentlyContinue
1194+
Copy-Item $gwErrLog "$gwErrLog.first" -Force -ErrorAction SilentlyContinue
1195+
1196+
# Restart on the same database: the record persists; nothing recovers it.
1197+
$gw = Start-Gw $gwDbUrl
1198+
Info "restarted gateway pid $($gw.Id) on the same database"
1199+
$listed = Invoke-NativeCaptured $cli @('sandbox', 'list')
1200+
$row = @($listed.Output | Where-Object { $_ -match [regex]::Escape($sandboxName) }) | Select-Object -First 1
1201+
if (-not $row) { throw "persisted sandbox record '$sandboxName' is missing after restart: $($listed.Output -join ' ')" }
1202+
Info "record after restart: $(($row -replace '\s+', ' ').Trim())"
1203+
Start-Sleep -Seconds 10
1204+
$running = @(Get-CimInstance Win32_Process | Where-Object { $_.CommandLine -like "*$killNonce*" -and $_.ProcessId -ne $PID })
1205+
if ($running.Count -gt 0) { throw "workload was running after restart ($($running.Count) process(es)); recovery behavior changed, update this scenario and the Windows docs" }
1206+
1207+
# The stale record must be manageable: delete it, then reuse the name.
1208+
$deleted = Invoke-NativeCaptured $cli @('sandbox', 'delete', $sandboxName)
1209+
if ($deleted.ExitCode -ne 0) { throw "deleting the stale record failed: $($deleted.Output -join ' ')" }
1210+
$listed = Invoke-NativeCaptured $cli @('sandbox', 'list')
1211+
if (@($listed.Output | Where-Object { $_ -match [regex]::Escape($sandboxName) }).Count -gt 0) { throw 'record still listed after delete' }
1212+
$again = Invoke-NativeCaptured $cli $createArguments
1213+
if ($again.ExitCode -ne 0) { throw "re-creating '$sandboxName' failed: $($again.Output -join ' ')" }
1214+
$null = Wait-SandboxTree $gw $killNonce
1215+
1216+
Ok "gateway-restart: graceful stop ended $($victims.Count) processes; record persisted, not recovered; delete and re-create work"
1217+
$results += [pscustomobject]@{ Scenario = $sc.Name; Result = 'PASS'; Reason = "graceful stop ended $($victims.Count) processes; record persisted, not recovered, name reusable" }
1218+
} catch {
1219+
Bad "gateway-restart: $($_.Exception.Message)"
1220+
$results += [pscustomobject]@{ Scenario = $sc.Name; Result = 'FAIL'; Reason = $_.Exception.Message }
1221+
}
10931222
} elseif ($sc.Kind -eq 'socket-owner') {
10941223
try {
10951224
if (-not (Wait-File $sc.ControlTarget 30)) { throw "parent cmd did not launch: $createOutStr" }
@@ -1286,7 +1415,10 @@ Host-service verifies two exact direct HTTP replies from a service bound to
12861415
host 127.0.0.1, with host-side request evidence. It checks localhost connectivity.
12871416
Gateway-kill creates a sandbox, hard-kills the gateway, and requires every process
12881417
that was under that gateway (and the marked workload) to end and the port to be
1289-
bindable by a replacement gateway.
1418+
bindable by a replacement gateway. Gateway-restart stops the gateway gracefully
1419+
(Ctrl-C) with a persisted database, requires the same, then restarts it and checks
1420+
that the sandbox record is still listed, its workload is not recovered, and the
1421+
record can be deleted and its name reused.
12901422
"@
12911423
Set-Content -Path (Join-Path $resultDir "summary.txt") -Value $summary -Encoding UTF8
12921424
Write-Host $summary -ForegroundColor ($(if ($verdict -eq "PASS") { "Green" } else { "Red" }))

0 commit comments

Comments
 (0)