Skip to content

Add new environment islo.dev#1434

Merged
alexgshaw merged 14 commits into
harbor-framework:mainfrom
islo-labs:islo-upstream-pr
Apr 21, 2026
Merged

Add new environment islo.dev#1434
alexgshaw merged 14 commits into
harbor-framework:mainfrom
islo-labs:islo-upstream-pr

Conversation

@AdamGold

@AdamGold AdamGold commented Apr 15, 2026

Copy link
Copy Markdown
Contributor

Add islo.dev as a new environment provider for Harbor, enabling agent evaluation in lightweight microVM sandboxes via the islo SDK.

Supports three modes:

  • Pre-built Docker image (docker_image config)
  • Dockerfile build via Docker-in-VM
  • Plain islo-runner sandbox (fallback)

Includes full unit test coverage for lifecycle, exec routing, Docker-in-VM build, file transfer, and attach.


Open with Devin

@vercel

vercel Bot commented Apr 15, 2026

Copy link
Copy Markdown

@assaf-benjosef is attempting to deploy a commit to the Harbor Framework Team on Vercel.

A member of the Team first needs to authorize it.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@alexgshaw

Copy link
Copy Markdown
Collaborator

Amazing excited about this. Can you address devin's comments? (You'll need to keep iterating until they are finished). Also can you run terminal-bench/terminal-bench-2 oracle and make sure you score > 90%.

AdamGold and others added 8 commits April 16, 2026 11:47
Add ISLO as a new environment provider for Harbor, enabling agent
evaluation in lightweight microVM sandboxes via the islo SDK.

Supports three modes:
- Pre-built Docker image (docker_image config)
- Dockerfile build via Docker-in-VM
- Plain islo-runner sandbox (fallback)

Includes full unit test coverage for lifecycle, exec routing,
Docker-in-VM build, file transfer, and attach.
BaseEnvironment.exec() now accepts a `user` parameter. Thread it through
IsloEnvironment.exec() → _docker_exec() (via `docker exec -u`) and accept
it in _sandbox_exec() for API consistency (bare sandbox always runs as root).

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Extract hardcoded infrastructure timeouts to named constants and increase
them to safer values, preventing spurious failures under I/O pressure or
during parallel benchmark runs.

- docker run: 30s → 120s
- docker cleanup: 30s → 60s
- docker stop: 15s → 30s

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
The transparent proxy in Islo sandboxes uses MITM certificates for TLS
interception. While the CA bundle is mounted into Docker containers,
Node.js and some other tools don't use the system CA store.

Add platform environment variables to docker run:
- NODE_EXTRA_CA_CERTS: Required for Node.js (e.g., Claude Code installer)
- SSL_CERT_FILE: Used by curl, Python requests, rustls-based tools
- LANG/LC_ALL: Unicode locale for proper output rendering

Fixes ISL-348: Claude Code install fails in swebench with certificate error

Co-Authored-By: Claude Opus 4.5 <[email protected]>
…rm env

Docker-in-VM containers need these env vars for Python requests/urllib3
and libcurl to trust the transparent proxy CA certificate.

Made-with: Cursor
curl/libcurl already respects SSL_CERT_FILE which is already set.
CURL_CA_BUNDLE is unnecessary.

Made-with: Cursor
…vels

Forward the user parameter to _sandbox_exec in direct sandbox mode and
wrap commands with su (matching Daytona/Runloop pattern).  Fix disk size
conversion to use // 1024 for consistency with other environments.
Downgrade logger.info to logger.debug for routine operational messages.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
…s without it

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@AdamGold

AdamGold commented Apr 16, 2026

Copy link
Copy Markdown
Contributor Author

@alexgshaw Ran terminal-bench:

terminal-bench • oracle
┏━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━┓
┃ Trials ┃ Exceptions ┃  Mean ┃
┡━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━┩
│     88 │          2 │ 0.921 │
└────────┴────────────┴───────┘

┏━━━━━━━━┳━━━━━━━┓
┃ Reward ┃ Count ┃
┡━━━━━━━━╇━━━━━━━┩
│ 0.0    │     6 │
│ 1.0    │    82 │
└────────┴───────┘

┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
┃ Exception         ┃ Count ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
│ AgentTimeoutError │     1 │
│ AddTestsDirError  │     1 │
└───────────────────┴───────┘

Job Info
Total runtime: 1h 2m 13s
Results written to jobs/2026-04-16__10-59-16/result.json
Inspect results by running `harbor view jobs`

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

AdamGold and others added 2 commits April 17, 2026 10:24
In DinD mode, exec() runs inside the Docker container while the islo
SDK transfers files to the sandbox filesystem. Only /logs, /tests,
/solution are bind-mounted, so uploads/downloads to other container
paths were silently invisible to exec.

Public file-transfer methods now dispatch: direct SDK when no Docker
container (or when the path is volume-mounted), two-hop via docker cp
otherwise. Matches the pattern in Daytona (_DaytonaDinD) and Modal
(_ModalDinD).

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
AdamGold and others added 2 commits April 21, 2026 14:35
islo requires an external Python SDK like modal/daytona/e2b do,
so it belongs in [project.optional-dependencies] rather than base
dependencies. Set pip_extra="islo" in the factory so users get a
helpful install hint if the import fails.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
dockerfile_parse is used at __init__ time when a Dockerfile is
present, making it a real dependency for Dockerfile-based tasks.
Matches the e2b extra which bundles it for the same reason.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@alexgshaw
alexgshaw merged commit b88e146 into harbor-framework:main Apr 21, 2026
5 of 8 checks passed
harshraj172-scale pushed a commit to harshraj172-scale/harbor that referenced this pull request May 30, 2026
* feat: add ISLO microVM sandbox environment (--env islo)

Add ISLO as a new environment provider for Harbor, enabling agent
evaluation in lightweight microVM sandboxes via the islo SDK.

Supports three modes:
- Pre-built Docker image (docker_image config)
- Dockerfile build via Docker-in-VM
- Plain islo-runner sandbox (fallback)

Includes full unit test coverage for lifecycle, exec routing,
Docker-in-VM build, file transfer, and attach.

* fix(islo): add user parameter to exec() to match updated base class API

BaseEnvironment.exec() now accepts a `user` parameter. Thread it through
IsloEnvironment.exec() → _docker_exec() (via `docker exec -u`) and accept
it in _sandbox_exec() for API consistency (bare sandbox always runs as root).

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(islo): increase Docker lifecycle timeouts

Extract hardcoded infrastructure timeouts to named constants and increase
them to safer values, preventing spurious failures under I/O pressure or
during parallel benchmark runs.

- docker run: 30s → 120s
- docker cleanup: 30s → 60s
- docker stop: 15s → 30s

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix(islo): pass platform env vars to Docker containers for TLS trust

The transparent proxy in Islo sandboxes uses MITM certificates for TLS
interception. While the CA bundle is mounted into Docker containers,
Node.js and some other tools don't use the system CA store.

Add platform environment variables to docker run:
- NODE_EXTRA_CA_CERTS: Required for Node.js (e.g., Claude Code installer)
- SSL_CERT_FILE: Used by curl, Python requests, rustls-based tools
- LANG/LC_ALL: Unicode locale for proper output rendering

Fixes ISL-348: Claude Code install fails in swebench with certificate error

Co-Authored-By: Claude Opus 4.5 <[email protected]>

* fix(islo): add REQUESTS_CA_BUNDLE and CURL_CA_BUNDLE to Docker platform env

Docker-in-VM containers need these env vars for Python requests/urllib3
and libcurl to trust the transparent proxy CA certificate.

Made-with: Cursor

* fix(islo): remove redundant CURL_CA_BUNDLE from Docker platform env

curl/libcurl already respects SSL_CERT_FILE which is already set.
CURL_CA_BUNDLE is unnecessary.

Made-with: Cursor

* fix(islo): address PR review feedback — user param, disk calc, log levels

Forward the user parameter to _sandbox_exec in direct sandbox mode and
wrap commands with su (matching Daytona/Runloop pattern).  Fix disk size
conversion to use // 1024 for consistency with other environments.
Downgrade logger.info to logger.debug for routine operational messages.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix(islo): lazy-import dockerfile_parse to avoid ImportError for users without it

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix(islo): reset _sandbox_name in stop() to clear stale state

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix(islo): honor task_env_config.workdir as cwd fallback in exec()

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix(islo): two-hop file transfer in Docker-in-VM mode

In DinD mode, exec() runs inside the Docker container while the islo
SDK transfers files to the sandbox filesystem. Only /logs, /tests,
/solution are bind-mounted, so uploads/downloads to other container
paths were silently invisible to exec.

Public file-transfer methods now dispatch: direct SDK when no Docker
container (or when the path is volume-mounted), two-hop via docker cp
otherwise. Matches the pattern in Daytona (_DaytonaDinD) and Modal
(_ModalDinD).

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

* fix(islo): move islo SDK to optional extra, add to cloud group

islo requires an external Python SDK like modal/daytona/e2b do,
so it belongs in [project.optional-dependencies] rather than base
dependencies. Set pip_extra="islo" in the factory so users get a
helpful install hint if the import fails.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(islo): add dockerfile-parse to islo extra

dockerfile_parse is used at __init__ time when a Dockerfile is
present, making it a real dependency for Dockerfile-based tasks.
Matches the e2b extra which bundles it for the same reason.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

---------

Co-authored-by: islo <[email protected]>
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
Co-authored-by: Assaf Ben Josef <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants