sync repo to upstream#12
Merged
Merged
Conversation
… multi-step trials (#1566) Multi-step support added in PR #1234 made the trial layer call agent.run() once per step but did not update Terminus2, which stores per-trial state on the instance. Three categories of bugs result: 1. Trajectory step IDs are non-sequential. The initial-prompt Step appends with step_id=1 hardcoded, but _trajectory_steps persists across run() calls. After step 2 we get [1,2,3,1,2,3,...] which fails Pydantic validation in _dump_trajectory(): all terminus-2 multi-step trials fail. 2. Per-run state accumulators leak across steps. _api_request_times, _trajectory_steps, _subagent_metrics, _subagent_rollout_details, _summarization_count, _session_id, _pending_completion, _pending_subagent_refs, _pending_handoff_prompt, _timestamped_markers are all written but never reset. Concrete consequences: - All step_results' metadata.api_request_times_msec reference the same growing list (Python aliasing) -> per-step latency tracking unusable. - Step N's trajectory.json contains all of steps 1..N (quadratic disk usage, downstream consumers see duplicated content). - All per-step trajectory.json files share one session_id. - If summarization fires in step 1, every later step's reported n_input_tokens / cost_usd is inflated by step 1's summarization cost. 3. Trial._execute_step_agent only catches asyncio.TimeoutError and NonZeroAgentExitCodeError. Any other exception (LLM errors, network errors, validation errors, anything from a subprocess agent) bubbles to trial-level. step_result.exception_info stays None on the failing step and remaining steps are silently aborted. Fix: - Add Terminus2._reset_per_run_state(), called at the top of run(). Clears all per-trial accumulators. A user-provided session_id (kwarg) is preserved via a new _user_provided_session_id attribute. - Widen Trial._execute_step_agent's except to Exception, matching the sibling _verify_step (line 603) and the caller of _run_step_setup (line 638). The explicit abort at trial.py:673 (`if exception_info and not verifier_result: break`) still fires when needed; the trial smartly continues if the verifier still produced a result. Verified against a 2-step task: 1/1 trial, mean reward 1.0, 0 exceptions, distinct session ids per step, distinct api_request_times_msec per step. Verified against a step-1-timeout-step-2-recovers task: step 1 records TimeoutError, step 2 still runs with fully isolated state, trial reward 0.5 (mean of 0 + 1.0). Co-authored-by: Cursor <[email protected]>
…1599) (#1639) PR #1559 (docker-compose support) introduced `_write_ca_overlay`, which bind-mounted the VM's CA bundle into the `main` service and set NODE_EXTRA_CA_CERTS / SSL_CERT_FILE / REQUESTS_CA_BUNDLE. PR #1599 (merged 2 minutes earlier) had just removed the `_VM_CA_BUNDLE` constant and the equivalent `docker run -v` mount, because the redundant CA mount caused `dpkg` to fail installing `ca-certificates` inside the container — the runner image already trusts the gateway's MITM certs via its base CA store. Neither PR rebased on the other. Upstream main currently references `_VM_CA_BUNDLE` at 4 call sites inside `_write_ca_overlay` with no matching definition. The module imports (Python late-binds names in function bodies) but compose-mode tasks crash with `NameError: name '_VM_CA_BUNDLE' is not defined` the moment a sandbox starts. Fix: drop the provider-side overlay entirely. Removed: - `_write_ca_overlay` method and its caller in `_start_compose` - `_COMPOSE_CA_OVERLAY_NAME` constant - the `-f` flag for the overlay in `_compose_file_flags` - the two overlay unit tests and the overlay assertion at test_islo.py:1280 Daytona's DinD compose path (daytona.py:461) already works without any provider-side overlay — tasks declare their own locale + env in their compose/Dockerfile. Matching that contract on islo as well. Added a regression test (`TestComposeFileFlagsHasNoProviderOverlay`) that asserts no `docker-compose-islo-*` path is injected into the `-f` flags. Verified end-to-end against api.islo.dev with the oracle agent on examples/tasks/hello-mcp (compose-mode): build + compose-up + verifier complete cleanly, reward 1.0. Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
* snapshot fixes * fix * fixed
* Update GDB adapter dependency and invocation Pin the adapter to lica-gdb 0.2.1 and remove the adapter's conflicting gdb console script so generation uses the explicit module entry point. Made-with: Cursor * Update GDB registry dataset docs Made-with: Cursor * Update GDB parity review links Made-with: Cursor * Add GDB adapter CLI alias Made-with: Cursor
* Add separate verifier environments * Add separate verifier changelog and compose env compatibility * Handle verifier artifact staging collisions * minor updates. * Minor fixes. * Update skills. Add blog post.
* Add Novita environment support to Harbor - Introduced NovitaEnvironment class for integration with Novita's cloud sandbox service. - Implemented end-to-end and unit tests for NovitaEnvironment functionality. * Fix CI failures: type errors, lint, and pytest collection crash - Add type: ignore comments for novita_sandbox SDK type issues - Move sys.exit() guard into __main__ block so pytest collection doesn't crash - Add template reuse test phase to e2e integration test Co-Authored-By: Claude Opus 4.6 <[email protected]> * Fix COPY instruction parsing and timeout_sec=0 handling - Skip COPY --from=... instructions (multi-stage builds) - Filter out COPY flags (--chown, --chmod) before extracting source path - Use explicit None check for timeout_sec to allow timeout_sec=0 Co-Authored-By: Claude Opus 4.6 <[email protected]> * Address Devin review: internet flag, default timeout, multi-source COPY - Set can_disable_internet to False (not yet supported by Novita SDK) - Change default exec timeout from 60s to 0 (no timeout), matching e2b - Handle multi-source COPY instructions (COPY a.py b.py /dest/) Co-Authored-By: Claude Opus 4.6 <[email protected]> * Fix Windows path separator in upload_dir remote paths Use PurePosixPath for remote sandbox paths to ensure forward slashes on all platforms. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Change default exec timeout from 0 to 300s The novita_sandbox SDK defaults to 60s internally when 0 is passed. Use 300s (5 minutes) to avoid premature termination of long-running agent and verifier commands. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Fix build error log index and defer API base URL resolution - Use logs[-1] instead of logs[-2] for build failure error message - Move NOVITA_BASE_URL lookup from class definition to __init__, consistent with NOVITA_API_KEY handling Co-Authored-By: Claude Opus 4.6 <[email protected]> * Handle null logs in build failure error reporting Use `status.get("logs") or []` instead of `status.get("logs", [])` to handle API returning `"logs": null`. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Wrap _http_client.aclose() in try/except in stop() Prevent transport-level errors during HTTP client cleanup from propagating out of stop() and masking the trial outcome. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Preserve sandbox when delete=False for debugging When stop(delete=False) is called, skip killing the sandbox and closing the HTTP client so the sandbox remains running for debugging purposes. This aligns with how other environments (e.g. GKE) handle the delete flag. Co-Authored-By: Claude Opus 4.6 <[email protected]> * novita: use alias endpoint for template lookup and fix stale alias recovery - Replace _api_list_templates + iteration with direct GET /templates/aliases/{alias} endpoint for O(1) template lookup instead of scanning all templates - Add stale alias recovery in _api_create_template: on 403 "Alias already used", look up the stale template via alias endpoint, delete it, then retry creation - Include API key suffix in template alias to avoid cross-account conflicts - Increase build timeout from 600s to 1200s for heavy Dockerfiles - Add _MIN_MEMORY_MB_PER_CPU constant (512 MB/CPU) - Update tests to cover new alias endpoint behavior (44 tests passing) Co-Authored-By: Claude Opus 4.6 <[email protected]> * novita: auto-recover from stale cached templates on sandbox creation When _find_template_by_alias returns a template ID that no longer exists in the backend (alias registered but build failed/incomplete), AsyncSandbox would raise a SandboxException("404: template not found"). Now start() catches this case, deletes the stale template via REST API, and triggers a fresh build before retrying sandbox creation. Co-Authored-By: Claude Opus 4.6 <[email protected]> * novita: include last 5 log lines in build failure error message Previously only the last log line was shown, which was often just "Postprocessing finished. Cleaning up..." instead of the actual error. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat(novita): upload COPY files via S3 pre-signed URL to fix 413 errors * chore: update parity_summary.csv [skip ci] * Fix review issues and CI failures in Novita environment - Add _merge_env(env) call in exec() so persistent env vars (--ae flags, task [environment.env] config) are correctly forwarded to sandbox commands - Add user parameter to exec(), is_dir(), is_file() to match BaseEnvironment interface (fixes type-check invalid-method-override errors) - Close HTTP client in stop(delete=False) to prevent resource leak; update test to assert aclose is called - Fix uv.lock: missing [[package]] header before networkx entry caused TOML parse errors that broke all CI checks; regenerate lockfile cleanly Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Fix exec() to respect user parameter via _resolve_user The user parameter was accepted but never used — all commands ran as root. Now calls _resolve_user(user) to honour the orchestrator-set default_user (e.g. task agent.user / verifier.user from task.toml). Novita SDK's user parameter is Literal["root", "user"], so map any non-root resolved user to "user"; add Literal import accordingly. Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Add preflight() and chmod 777 on log dirs in Novita environment - Add preflight() classmethod to validate NOVITA_API_KEY before any trials are queued, giving immediate feedback instead of failing mid-job - chmod 777 agent/verifier log directories after creation in start() so non-root agent/verifier users can write reward files and logs - Update start() test mocks to handle both foreground (healthcheck) and background (exec) sandbox.commands.run call patterns Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * style: ruff format test_novita.py Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Fix template name slash escaping and cwd quoting in exec - Replace '/' with '__' in template alias construction so org/name task names (e.g. harbor/hello-world) don't break REST API URL paths - Use shlex.quote(effective_cwd) in exec() to handle paths with spaces or shell metacharacters safely Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Use timeout=0 (no limit) as default in exec, aligning with E2B timeout_sec or 0 matches E2B and the Novita SDK docs where 0 means no connection time limit, avoiding premature 300s cutoffs on long-running agent setup or verifier scripts. Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Update src/harbor/environments/novita.py Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: deal with build conflict error and enhance Dockerfile handling in NovitaEnvironment * refactor: move novita-sandbox to optional extra, matching other cloud providers - Move `novita-sandbox` from main deps to `[novita]` optional extra - Add `dockerfile-parse` to `novita` extra (was only in `e2b`, but novita.py needs it) - Include `harbor[novita]` in the `cloud` bundle - Wrap SDK imports in try/except with `_HAS_NOVITA` flag, following the same lazy-import pattern introduced for daytona/e2b/modal in the upstream refactor - Raise `MissingExtraError` in `preflight()` when novita-sandbox is not installed - Regenerate uv.lock Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * fix: add _HAS_NOVITA guard in __init__ for clear MissingExtraError Without this guard, instantiating NovitaEnvironment when novita-sandbox is not installed raises a raw NameError (on DockerfileParser) instead of a helpful MissingExtraError with install instructions. Follows the same pattern as E2BEnvironment and RunloopEnvironment. Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Update src/harbor/environments/novita.py Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Update src/harbor/environments/novita.py Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: import EnvironmentCapabilities in Novita environment Add the missing capabilities import after migrating NovitaEnvironment to the new capabilities API so ruff and ty can resolve the type. Co-Authored-By: Claude Opus 4.7 <[email protected]> * fix: update Novita capability tests Update Novita environment tests to assert the new capabilities API after migrating away from deprecated properties. Co-Authored-By: Claude Opus 4.7 <[email protected]> * fix: fix file upload endpoint --------- Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Refactor trial execution by shape * Clean up trial helper typing * Skip Windows container hello world without Docker * Partial refactor. * Improve artifact handler. * Minor multi step fixes. * Make artifact handler paths operation scoped * Fix CI after trial flow cleanup * Keep download dir excludes explicit * Rename download dir exclusions helper * Address artifact exclusion review comments * Avoid duplicate single-step artifact recovery * Avoid double stop after cancellation --------- Co-authored-by: gabeorlanski <[email protected]>
… error messages (#1657) - _tmux_send_keys: append `--` end-of-options marker to the `tmux send-keys -t <session>` prefix so keys beginning with `-` (e.g. `-x`, `-Lfoo`) are treated as literal key arguments rather than being parsed as tmux options. - _send_blocking_keys / _send_non_blocking_keys: include `command` (truncated to 100 chars), `return_code`, `stderr`, and `stdout` in the raised RuntimeError to make intermittent send-keys failures easier to diagnose from logs. - tests: update _extract_send_keys_payload helper for the new `--` separator and add coverage for keys starting with `-` and for the enriched failure messages. Co-authored-by: Cursor <[email protected]>
* add repeatable skill inputs * Register injected skills for Cursor CLI * Use Cursor native skills directory * Simplify skill resolution * Make injected skills readable by agents * Address skill input review comments * Reject relative task skills dir for injected skills * Add skills CLI alias * Rename injected skill config to skills * Add runtime skills job example * Trim runtime skills example config
* add repeatable extra docker compose overlays * preserve modal compose build markers * preserve cloud compose file precedence * Guard extra compose by environment capability * Rename extra compose config paths * Revert "Rename extra compose config paths" This reverts commit 5c531c6. * Add extra compose job example * Address extra compose example comments * Nest extra compose job example
* Add runtime MCP config support * Use extra compose overlay for MCP proof example * Remove MCP proof example volume * Use Python base image in MCP proof task * Document MCP proof compose context * Trim MCP proof job defaults * Embed MCP proof runtime config
* feat: add support for --extra-instruction-paths * Add extra instruction path support * Fix lock equality env serialization * Fix lock equality for digest-backed paths --------- Co-authored-by: ZHAO Jin-Xiang <[email protected]>
Update based on change in #1620
* Add minimal verifier extension hook Add a small verifier factory hook that allows jobs to provide an optional custom verifier by import path while keeping the existing task verification flow as the default. This enables job-specific verification to supplement task-specific checks. For example, a job can attach generic trajectory evaluators, policy checks, or run-level scoring logic across many tasks without rebuilding, copying, or modifying those task definitions. The hook keeps task authorship and job evaluation concerns separate: tasks continue to define their normal verification, and jobs can opt into additional verifier behavior only when needed. Default behavior is unchanged when no custom verifier is configured. Signed-off-by: Anuradha Karuppiah <[email protected]> * Tighten verifier extension contract Introduce BaseVerifier and VerifierContext so custom verifiers receive a stable construction context while the built-in verifier keeps legacy kwargs compatibility. Require verifier outputs to be VerifierResult before assigning them to trial results, preserving Harbor aggregation semantics for built-in and imported verifiers. Keep legacy import-path constructors working through an adapter that enforces the return contract. Signed-off-by: Anuradha Karuppiah <[email protected]> * Reject unused verifier kwargs Fail fast when verifier kwargs are provided without a verifier import path, since the built-in verifier does not consume arbitrary extension kwargs. This makes CLI/config mistakes visible instead of silently dropping values like --verifier-kwarg foo=bar. Signed-off-by: Anuradha Karuppiah <[email protected]> * Fix verifier factory test patch Update Windows multi-step verifier tests to patch VerifierFactory.create_verifier_from_config after trial verification moved behind the factory hook. Signed-off-by: Anuradha Karuppiah <[email protected]> * Simplify verifier extension constructor * Simplify verifier factory contract * Fix skills merge example config paths --------- Signed-off-by: Anuradha Karuppiah <[email protected]> Co-authored-by: Alex Shaw <[email protected]>
* Add Novita environment support to Harbor - Introduced NovitaEnvironment class for integration with Novita's cloud sandbox service. - Implemented end-to-end and unit tests for NovitaEnvironment functionality. * Fix CI failures: type errors, lint, and pytest collection crash - Add type: ignore comments for novita_sandbox SDK type issues - Move sys.exit() guard into __main__ block so pytest collection doesn't crash - Add template reuse test phase to e2e integration test Co-Authored-By: Claude Opus 4.6 <[email protected]> * Fix COPY instruction parsing and timeout_sec=0 handling - Skip COPY --from=... instructions (multi-stage builds) - Filter out COPY flags (--chown, --chmod) before extracting source path - Use explicit None check for timeout_sec to allow timeout_sec=0 Co-Authored-By: Claude Opus 4.6 <[email protected]> * Address Devin review: internet flag, default timeout, multi-source COPY - Set can_disable_internet to False (not yet supported by Novita SDK) - Change default exec timeout from 60s to 0 (no timeout), matching e2b - Handle multi-source COPY instructions (COPY a.py b.py /dest/) Co-Authored-By: Claude Opus 4.6 <[email protected]> * Fix Windows path separator in upload_dir remote paths Use PurePosixPath for remote sandbox paths to ensure forward slashes on all platforms. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Change default exec timeout from 0 to 300s The novita_sandbox SDK defaults to 60s internally when 0 is passed. Use 300s (5 minutes) to avoid premature termination of long-running agent and verifier commands. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Fix build error log index and defer API base URL resolution - Use logs[-1] instead of logs[-2] for build failure error message - Move NOVITA_BASE_URL lookup from class definition to __init__, consistent with NOVITA_API_KEY handling Co-Authored-By: Claude Opus 4.6 <[email protected]> * Handle null logs in build failure error reporting Use `status.get("logs") or []` instead of `status.get("logs", [])` to handle API returning `"logs": null`. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Wrap _http_client.aclose() in try/except in stop() Prevent transport-level errors during HTTP client cleanup from propagating out of stop() and masking the trial outcome. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Preserve sandbox when delete=False for debugging When stop(delete=False) is called, skip killing the sandbox and closing the HTTP client so the sandbox remains running for debugging purposes. This aligns with how other environments (e.g. GKE) handle the delete flag. Co-Authored-By: Claude Opus 4.6 <[email protected]> * novita: use alias endpoint for template lookup and fix stale alias recovery - Replace _api_list_templates + iteration with direct GET /templates/aliases/{alias} endpoint for O(1) template lookup instead of scanning all templates - Add stale alias recovery in _api_create_template: on 403 "Alias already used", look up the stale template via alias endpoint, delete it, then retry creation - Include API key suffix in template alias to avoid cross-account conflicts - Increase build timeout from 600s to 1200s for heavy Dockerfiles - Add _MIN_MEMORY_MB_PER_CPU constant (512 MB/CPU) - Update tests to cover new alias endpoint behavior (44 tests passing) Co-Authored-By: Claude Opus 4.6 <[email protected]> * novita: auto-recover from stale cached templates on sandbox creation When _find_template_by_alias returns a template ID that no longer exists in the backend (alias registered but build failed/incomplete), AsyncSandbox would raise a SandboxException("404: template not found"). Now start() catches this case, deletes the stale template via REST API, and triggers a fresh build before retrying sandbox creation. Co-Authored-By: Claude Opus 4.6 <[email protected]> * novita: include last 5 log lines in build failure error message Previously only the last log line was shown, which was often just "Postprocessing finished. Cleaning up..." instead of the actual error. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat(novita): upload COPY files via S3 pre-signed URL to fix 413 errors * chore: update parity_summary.csv [skip ci] * Fix review issues and CI failures in Novita environment - Add _merge_env(env) call in exec() so persistent env vars (--ae flags, task [environment.env] config) are correctly forwarded to sandbox commands - Add user parameter to exec(), is_dir(), is_file() to match BaseEnvironment interface (fixes type-check invalid-method-override errors) - Close HTTP client in stop(delete=False) to prevent resource leak; update test to assert aclose is called - Fix uv.lock: missing [[package]] header before networkx entry caused TOML parse errors that broke all CI checks; regenerate lockfile cleanly Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Fix exec() to respect user parameter via _resolve_user The user parameter was accepted but never used — all commands ran as root. Now calls _resolve_user(user) to honour the orchestrator-set default_user (e.g. task agent.user / verifier.user from task.toml). Novita SDK's user parameter is Literal["root", "user"], so map any non-root resolved user to "user"; add Literal import accordingly. Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Add preflight() and chmod 777 on log dirs in Novita environment - Add preflight() classmethod to validate NOVITA_API_KEY before any trials are queued, giving immediate feedback instead of failing mid-job - chmod 777 agent/verifier log directories after creation in start() so non-root agent/verifier users can write reward files and logs - Update start() test mocks to handle both foreground (healthcheck) and background (exec) sandbox.commands.run call patterns Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * style: ruff format test_novita.py Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Fix template name slash escaping and cwd quoting in exec - Replace '/' with '__' in template alias construction so org/name task names (e.g. harbor/hello-world) don't break REST API URL paths - Use shlex.quote(effective_cwd) in exec() to handle paths with spaces or shell metacharacters safely Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Use timeout=0 (no limit) as default in exec, aligning with E2B timeout_sec or 0 matches E2B and the Novita SDK docs where 0 means no connection time limit, avoiding premature 300s cutoffs on long-running agent setup or verifier scripts. Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Update src/harbor/environments/novita.py Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: deal with build conflict error and enhance Dockerfile handling in NovitaEnvironment * refactor: move novita-sandbox to optional extra, matching other cloud providers - Move `novita-sandbox` from main deps to `[novita]` optional extra - Add `dockerfile-parse` to `novita` extra (was only in `e2b`, but novita.py needs it) - Include `harbor[novita]` in the `cloud` bundle - Wrap SDK imports in try/except with `_HAS_NOVITA` flag, following the same lazy-import pattern introduced for daytona/e2b/modal in the upstream refactor - Raise `MissingExtraError` in `preflight()` when novita-sandbox is not installed - Regenerate uv.lock Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * fix: add _HAS_NOVITA guard in __init__ for clear MissingExtraError Without this guard, instantiating NovitaEnvironment when novita-sandbox is not installed raises a raw NameError (on DockerfileParser) instead of a helpful MissingExtraError with install instructions. Follows the same pattern as E2BEnvironment and RunloopEnvironment. Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Update src/harbor/environments/novita.py Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Update src/harbor/environments/novita.py Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: import EnvironmentCapabilities in Novita environment Add the missing capabilities import after migrating NovitaEnvironment to the new capabilities API so ruff and ty can resolve the type. Co-Authored-By: Claude Opus 4.7 <[email protected]> * fix: update Novita capability tests Update Novita environment tests to assert the new capabilities API after migrating away from deprecated properties. Co-Authored-By: Claude Opus 4.7 <[email protected]> * fix: fix file upload endpoint * fix: integrate Novita SDK template builds Use the Novita SDK template builder directly while preserving Harbor's Dockerfile COPY handling, and pin the alpha SDK version without enabling global prerelease resolution. Co-Authored-By: Claude Opus 4.7 <[email protected]> * fix: pin Novita sandbox domain Use the regional Novita sandbox endpoint consistently so local domain overrides cannot route template operations to the wrong API host. Co-Authored-By: Claude Opus 4.7 <[email protected]> * fix: avoid Novita SDK import during test collection Load Novita SDK modules only when the Novita environment actually needs them so pytest can collect E2B and Novita tests in the same process without duplicate protobuf descriptor registration. Co-Authored-By: Claude Opus 4.7 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Migrate legacy memory/storage fields in a before validator instead of Field(deprecated=...) plus an after validator, and reject conflicting legacy and modern resource values. Closes #1693 Co-authored-by: Cursor <[email protected]>
Cursor CLI stream-json reports token usage on result events but not dollar cost. Parse optional totalCost when present and otherwise estimate from per-category token counts using LiteLLM pricing. Co-authored-by: Cursor <[email protected]>
* Add harbor leaderboard submit CLI command * Better testing script * Add worker dir for dynamic validation * Improve deploy and expose dynamic validation verdict * Add cost analysis and batching to analyze * Only deploy on harbor version changes * Update validation to warn on malformed trials * Calculate dynamic validation accuracy using verdict * Fetch trials after validation pass * Lint fix * Add logging to dynamic validation worker * Remove validation worker from harbor framework repo * Add validation worker version to analyze config * Update tests to check no linked jobs/trials can be deleted
* fix: experiment sessions can now be found on reruns * lints
Bump version after publishing the experiment session rerun fix to PyPI. Co-authored-by: Cursor <[email protected]>
…uns (#1794) Headless `gemini --prompt` cannot show the auth-method dialog and exits with "Invalid auth method selected" unless a method is pre-selected in settings.json. The agent only wrote security.auth.selectedType for the OAuth path, so API-key and Vertex runs relied on gemini-cli auto-detection -- which also breaks when a custom GOOGLE_GEMINI_BASE_URL (e.g. a proxy) is set. Resolve the auth method from the env credentials in use (GEMINI_API_KEY/GOOGLE_API_KEY -> gemini-api-key, GOOGLE_GENAI_USE_VERTEXAI -> vertex-ai) and pin it in settings.json. The settings builder now takes a single auth_type ("oauth-personal", "gemini-api-key", or "vertex-ai") instead of an OAuth-only boolean. Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: Boxuan Li <[email protected]>
* Update Islo SDK integration Require the latest Islo SDK so Harbor picks up the new compute-plane default, and pass through ISLO_COMPUTE_URL for custom compute endpoints. Co-authored-by: Cursor <[email protected]> * Use Islo init intent for sandbox creation Co-authored-by: Cursor <[email protected]> --------- Co-authored-by: Cursor <[email protected]>
Third-party output routed through Live's redirected streams can invert Console and Live locks during long benchmark runs. Co-authored-by: Cursor <[email protected]>
* Suppress litellm debug output unless --debug is enabled. Set litellm.suppress_debug_info by default to stop high-frequency Provider List prints on unrecognized models, and re-enable when Harbor job debug mode is on. Co-authored-by: Cursor <[email protected]> * Configure litellm debug suppression via Terminus llm_kwargs. Suppress litellm Provider List prints by default when Terminus 2 initializes the LiteLLM backend, and allow opt-in via llm_kwargs litellm_debug instead of job --debug. Co-authored-by: Cursor <[email protected]> * Fix litellm suppress_debug_info assignment for type checker. Use setattr because litellm types suppress_debug_info as Literal[False]. Co-authored-by: Cursor <[email protected]> --------- Co-authored-by: Cursor <[email protected]>
Use HTML anchor tags with target="_blank" so diff viewer links posted on PRs open externally instead of navigating away from GitHub. Co-authored-by: Cursor <[email protected]>
Adds Docker-in-Docker compose support to the GKE environment so multi-container tasks (those shipping a docker-compose.yaml) can run on GKE Standard. A single privileged dind pod runs dockerd; `docker compose` orchestrates the task's services inside it, and exec/upload/download target the `main` service via a two-hop path (k8s exec into the dind container, then `docker compose exec`/`cp`). Rebuilt on top of the current compose model (rather than the stale #1242 branch): - Reuses the shared compose templates and the `write_resources_compose_file` override (parity with the Daytona/Modal DinD strategies) instead of the removed docker-compose-base.yaml. - `capabilities()` advertises accelerators only in single-container mode; a task that ships docker-compose.yaml AND requests a GPU/TPU is rejected at preflight. No env supports TPU-in-compose, and a privileged dind pod cannot expose an accelerator into nested compose services. - Outer pod sized to the task's total budget, Burstable in AUTO mode (no fabricated daemon-overhead constant); inner `main` resources override only imposes a hard limit when the task sets ResourceMode.LIMIT. - Two-hop transfer reuses the existing python kubernetes-client exec/tar machinery (no new kubectl dependency); the single-container Direct path is untouched apart from extracting the shared `_create_pod` / pod-delete helpers. Requires GKE Standard (Autopilot blocks privileged pods). The DinD runtime path needs a kind/GKE smoke test before merge. Co-Authored-By: Claude Opus 4.7 <[email protected]> Co-authored-by: Ruiyang Wang <[email protected]>
* add use.computer environment * remove windows use computer example --------- Co-authored-by: Use.Computer <[email protected]>
* fix(pi): Reduce logging footprint by filtering message_updates * Fix typo * Run formatter
…ription (#1766) * fix: accept all Claude auth methods in harbor check / analyze, prefer the free subscription harbor check and harbor analyze run their LLM rubric through claude-agent-sdk, which forwards the host environment to the Claude Code CLI and lets the CLI resolve credentials. The preflight hard-required ANTHROPIC_API_KEY -- which both rejected valid auth (OAuth tokens, `claude /login`, cloud providers) and pushed subscribers onto a paid Console key they did not need. Replace it with a shared resolve_claude_sdk_auth() in harbor.utils.env that: - accepts the full chain the CLI honors: CLAUDE_CODE_OAUTH_TOKEN, ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN, ~/.claude/.credentials.json, the macOS login Keychain (service "Claude Code-credentials", read straight from the SDK's own constant), and Bedrock/Vertex/Foundry. Blank exports count as unset. - prefers the free subscription: when a subscription credential is present alongside ANTHROPIC_API_KEY, it drops the key for the run (the CLI would otherwise bill it ahead of the subscription) and prints a notice. HARBOR_FORCE_API_KEY=1 opts out, mirroring Codex's CODEX_FORCE_AUTH_JSON flag (parsed via the same parse_bool_env_value helper). Both commands call the one helper. Co-Authored-By: Claude Opus 4.8 <[email protected]> * fix: default Claude SDK auth to API key * fix: remove extra Claude force OAuth guard * fail loudly when CLAUDE_FORCE_OAUTH is set but no subscription credential --------- Co-authored-by: Claude Opus 4.8 <[email protected]> Co-authored-by: Kobe Chen <[email protected]>
…1838) * fix(modal): tar-based directory transfers to stop silent file drops Modal's _sdk_upload_dir/_sdk_download_dir transferred directories one file at a time via Sandbox.filesystem exec-streamed calls. On large multi-file uploads (e.g. DinD compose tasks) files were observed to be silently skipped, and the per-file path also loses exec bits, symlinks, and empty directories — unlike docker compose cp on the local Docker environment (terminal-bench-3#846). Pack the directory into a single tar.gz, transfer it with one SDK call, and extract on the other side. gzip CRC + tar exit status make partial transfers fail loudly. Falls back to the previous per-file path when tar/gzip is unavailable on the sandbox. https://claude.ai/code/session_01XmMGntgUhjovVk3LKBavzU * refactor(env): shared tar_transfer lib; full-fidelity dir transfers for Modal, Daytona, GKE Extract the tar-based directory packing/extraction introduced for Modal into harbor.environments.tar_transfer and apply it uniformly: - Modal (direct + DinD): unchanged behavior, now built on the shared lib. - Daytona (direct + DinD): pack/upload/extract as a single archive, preserving exec bits, symlinks, and empty dirs that the batched fs.upload_files staging dropped; falls back to the batched per-file path when tar/gzip is unavailable. Missing source dirs now raise instead of silently uploading nothing. - GKE (direct + DinD): already tar-streamed on the wire, but packed only regular files; now packs the whole tree so symlinks and empty dirs survive, and extraction uses the safe 'data' filter everywhere. All dir transfers now match docker compose cp fidelity on the local Docker environment. https://claude.ai/code/session_01XmMGntgUhjovVk3LKBavzU * refactor(env): require tar/gzip on sandboxes, drop per-file transfer fallbacks Modal and Daytona directory transfers are now tar-only, matching GKE, which has always required tar in the container for its exec-streamed transfers. Any image Harbor can run (it already requires a shell) ships tar and gzip, and the fallback paths silently degraded fidelity (no exec bits, symlinks, or empty dirs), which is exactly the bug class being fixed. A missing tar now fails loudly with a clear error instead of falling back. https://claude.ai/code/session_01XmMGntgUhjovVk3LKBavzU * test(env): skip POSIX-fidelity transfer tests on Windows The new dir-transfer tests create symlinks (WinError 1314 without elevation on Windows runners) and assert exec bits (chmod is a no-op on NTFS). They verify POSIX semantics of transfers into Linux sandboxes, which a Windows filesystem cannot represent, so skip them on win32. The truncated-archive test now uses a plain tree so it keeps running cross-platform. https://claude.ai/code/session_01XmMGntgUhjovVk3LKBavzU --------- Co-authored-by: Claude <[email protected]>
* Add Novita environment support to Harbor - Introduced NovitaEnvironment class for integration with Novita's cloud sandbox service. - Implemented end-to-end and unit tests for NovitaEnvironment functionality. * Fix CI failures: type errors, lint, and pytest collection crash - Add type: ignore comments for novita_sandbox SDK type issues - Move sys.exit() guard into __main__ block so pytest collection doesn't crash - Add template reuse test phase to e2e integration test Co-Authored-By: Claude Opus 4.6 <[email protected]> * Fix COPY instruction parsing and timeout_sec=0 handling - Skip COPY --from=... instructions (multi-stage builds) - Filter out COPY flags (--chown, --chmod) before extracting source path - Use explicit None check for timeout_sec to allow timeout_sec=0 Co-Authored-By: Claude Opus 4.6 <[email protected]> * Address Devin review: internet flag, default timeout, multi-source COPY - Set can_disable_internet to False (not yet supported by Novita SDK) - Change default exec timeout from 60s to 0 (no timeout), matching e2b - Handle multi-source COPY instructions (COPY a.py b.py /dest/) Co-Authored-By: Claude Opus 4.6 <[email protected]> * Fix Windows path separator in upload_dir remote paths Use PurePosixPath for remote sandbox paths to ensure forward slashes on all platforms. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Change default exec timeout from 0 to 300s The novita_sandbox SDK defaults to 60s internally when 0 is passed. Use 300s (5 minutes) to avoid premature termination of long-running agent and verifier commands. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Fix build error log index and defer API base URL resolution - Use logs[-1] instead of logs[-2] for build failure error message - Move NOVITA_BASE_URL lookup from class definition to __init__, consistent with NOVITA_API_KEY handling Co-Authored-By: Claude Opus 4.6 <[email protected]> * Handle null logs in build failure error reporting Use `status.get("logs") or []` instead of `status.get("logs", [])` to handle API returning `"logs": null`. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Wrap _http_client.aclose() in try/except in stop() Prevent transport-level errors during HTTP client cleanup from propagating out of stop() and masking the trial outcome. Co-Authored-By: Claude Opus 4.6 <[email protected]> * Preserve sandbox when delete=False for debugging When stop(delete=False) is called, skip killing the sandbox and closing the HTTP client so the sandbox remains running for debugging purposes. This aligns with how other environments (e.g. GKE) handle the delete flag. Co-Authored-By: Claude Opus 4.6 <[email protected]> * novita: use alias endpoint for template lookup and fix stale alias recovery - Replace _api_list_templates + iteration with direct GET /templates/aliases/{alias} endpoint for O(1) template lookup instead of scanning all templates - Add stale alias recovery in _api_create_template: on 403 "Alias already used", look up the stale template via alias endpoint, delete it, then retry creation - Include API key suffix in template alias to avoid cross-account conflicts - Increase build timeout from 600s to 1200s for heavy Dockerfiles - Add _MIN_MEMORY_MB_PER_CPU constant (512 MB/CPU) - Update tests to cover new alias endpoint behavior (44 tests passing) Co-Authored-By: Claude Opus 4.6 <[email protected]> * novita: auto-recover from stale cached templates on sandbox creation When _find_template_by_alias returns a template ID that no longer exists in the backend (alias registered but build failed/incomplete), AsyncSandbox would raise a SandboxException("404: template not found"). Now start() catches this case, deletes the stale template via REST API, and triggers a fresh build before retrying sandbox creation. Co-Authored-By: Claude Opus 4.6 <[email protected]> * novita: include last 5 log lines in build failure error message Previously only the last log line was shown, which was often just "Postprocessing finished. Cleaning up..." instead of the actual error. Co-Authored-By: Claude Opus 4.6 <[email protected]> * feat(novita): upload COPY files via S3 pre-signed URL to fix 413 errors * chore: update parity_summary.csv [skip ci] * Fix review issues and CI failures in Novita environment - Add _merge_env(env) call in exec() so persistent env vars (--ae flags, task [environment.env] config) are correctly forwarded to sandbox commands - Add user parameter to exec(), is_dir(), is_file() to match BaseEnvironment interface (fixes type-check invalid-method-override errors) - Close HTTP client in stop(delete=False) to prevent resource leak; update test to assert aclose is called - Fix uv.lock: missing [[package]] header before networkx entry caused TOML parse errors that broke all CI checks; regenerate lockfile cleanly Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Fix exec() to respect user parameter via _resolve_user The user parameter was accepted but never used — all commands ran as root. Now calls _resolve_user(user) to honour the orchestrator-set default_user (e.g. task agent.user / verifier.user from task.toml). Novita SDK's user parameter is Literal["root", "user"], so map any non-root resolved user to "user"; add Literal import accordingly. Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Add preflight() and chmod 777 on log dirs in Novita environment - Add preflight() classmethod to validate NOVITA_API_KEY before any trials are queued, giving immediate feedback instead of failing mid-job - chmod 777 agent/verifier log directories after creation in start() so non-root agent/verifier users can write reward files and logs - Update start() test mocks to handle both foreground (healthcheck) and background (exec) sandbox.commands.run call patterns Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * style: ruff format test_novita.py Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Fix template name slash escaping and cwd quoting in exec - Replace '/' with '__' in template alias construction so org/name task names (e.g. harbor/hello-world) don't break REST API URL paths - Use shlex.quote(effective_cwd) in exec() to handle paths with spaces or shell metacharacters safely Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Use timeout=0 (no limit) as default in exec, aligning with E2B timeout_sec or 0 matches E2B and the Novita SDK docs where 0 means no connection time limit, avoiding premature 300s cutoffs on long-running agent setup or verifier scripts. Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Update src/harbor/environments/novita.py Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: deal with build conflict error and enhance Dockerfile handling in NovitaEnvironment * refactor: move novita-sandbox to optional extra, matching other cloud providers - Move `novita-sandbox` from main deps to `[novita]` optional extra - Add `dockerfile-parse` to `novita` extra (was only in `e2b`, but novita.py needs it) - Include `harbor[novita]` in the `cloud` bundle - Wrap SDK imports in try/except with `_HAS_NOVITA` flag, following the same lazy-import pattern introduced for daytona/e2b/modal in the upstream refactor - Raise `MissingExtraError` in `preflight()` when novita-sandbox is not installed - Regenerate uv.lock Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * fix: add _HAS_NOVITA guard in __init__ for clear MissingExtraError Without this guard, instantiating NovitaEnvironment when novita-sandbox is not installed raises a raw NameError (on DockerfileParser) instead of a helpful MissingExtraError with install instructions. Follows the same pattern as E2BEnvironment and RunloopEnvironment. Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]> * Update src/harbor/environments/novita.py Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Update src/harbor/environments/novita.py Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: import EnvironmentCapabilities in Novita environment Add the missing capabilities import after migrating NovitaEnvironment to the new capabilities API so ruff and ty can resolve the type. Co-Authored-By: Claude Opus 4.7 <[email protected]> * fix: update Novita capability tests Update Novita environment tests to assert the new capabilities API after migrating away from deprecated properties. Co-Authored-By: Claude Opus 4.7 <[email protected]> * fix: fix file upload endpoint * fix: integrate Novita SDK template builds Use the Novita SDK template builder directly while preserving Harbor's Dockerfile COPY handling, and pin the alpha SDK version without enabling global prerelease resolution. Co-Authored-By: Claude Opus 4.7 <[email protected]> * fix: pin Novita sandbox domain Use the regional Novita sandbox endpoint consistently so local domain overrides cannot route template operations to the wrong API host. Co-Authored-By: Claude Opus 4.7 <[email protected]> * fix: avoid Novita SDK import during test collection Load Novita SDK modules only when the Novita environment actually needs them so pytest can collect E2B and Novita tests in the same process without duplicate protobuf descriptor registration. Co-Authored-By: Claude Opus 4.7 <[email protected]> * fix: fix novita sandbox default domain * feat: Add docker-compose support to Novita (in-sandbox docker compose) and extend unit tests for env-driven domain/API settings Signed-off-by: muskmo <[email protected]> * fix: fix dependencies * fix(novita): restore bridge DNS for DinD compose on prod kernels Signed-off-by: muskmo <[email protected]> --------- Signed-off-by: muskmo <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mac <[email protected]> Co-authored-by: muskmo <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.