Skip to content

Commit e48d771

Browse files
authored
Merge pull request #1067 from NVIDIA/codex/contributor-onboarding-verification
docs: validate coding-agent contributor onboarding
2 parents 3fb77b4 + ca53178 commit e48d771

4 files changed

Lines changed: 355 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -984,6 +984,13 @@ make format-check
984984
`make test-unit` excludes tests marked `integration` or `provider`. This selection
985985
does not cover every CI job or guarantee network isolation.
986986

987+
Some unit tests start loopback HTTP servers and local multiprocessing sockets.
988+
Restricted agent sandboxes can deny these operations. Keep the failing
989+
command and traceback, and use an appropriately permitted environment for a
990+
diagnostic rerun. Report that result separately; a passing subset does not make
991+
the original full target pass. See the
992+
[contributor onboarding check](docs/validation/CONTRIBUTOR_ONBOARDING.md).
993+
987994
Before submitting a PR, [CONTRIBUTING.md](CONTRIBUTING.md#how-to-contribute)
988995
requires `make test`. It runs both `test-unit` and `test-integration`; integration
989996
tests invoke the full graph and may send test content to a configured LLM provider.

‎docs/DEVELOPMENT.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,10 @@ prerequisites, clone, virtual environment, and dependency-installation recipe.
2525
The supported Python range is defined in [pyproject.toml](../pyproject.toml).
2626
All Makefile targets assume the virtual environment is already created and activated.
2727

28+
To observe this workflow with a coding agent, use the
29+
[contributor onboarding check](validation/CONTRIBUTOR_ONBOARDING.md), which
30+
records setup and check results separately from full PR validation.
31+
2832
- **Installation**: `make install` and `make install-dev` use **uv** if available (`uv sync` / `uv sync --all-extras`), otherwise **pip** (`pip install -e .` / `pip install -e ".[dev]"`).
2933
- **Environment**: Optional `.env` in the project root. The LangGraph dev server loads it (see [langgraph.json](../langgraph.json) `"env": ".env"`). Key variables:
3034
- **`SKILLSPECTOR_PROVIDER`**: Selects the active LLM provider — `openai`, `anthropic`, `anthropic_proxy`, `bedrock`, `nv_build`, `ollama`, `azure_openai`, `openai_compatible`, `claude_cli`, `gemini_cli`, or `opencode_cli`. Defaults to `nv_build` when unset.
Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Checking contributor onboarding with a coding agent
2+
3+
Use this procedure to check whether a coding agent can discover the contributor
4+
guidance, set up a checkout, and perform the documented initial checks. Record
5+
what actually ran, including failures. A documentation checklist alone does not
6+
establish that the workflow works.
7+
8+
The [README setup](../../README.md#setup) remains the canonical installation
9+
recipe. This procedure does not change the submission requirements in
10+
[CONTRIBUTING.md](../../CONTRIBUTING.md#how-to-contribute).
11+
12+
## Prepare the observation
13+
14+
1. Pin the source commit. Start with a clean checkout, no existing virtual
15+
environment, and no project `.env` file. State whether the repository was
16+
provisioned locally or cloned by the agent over HTTPS.
17+
2. Record the OS, architecture, shell, Python, package manager, Git, and Make
18+
versions. Identify the coding agent/client and model build when available;
19+
explicitly record any unavailable version rather than inventing one.
20+
3. Run project commands with a fresh home directory and an explicit environment
21+
allowlist. Retain only the paths and settings needed for the chosen tools and
22+
temporary/cache directories. Do not inherit provider credentials, personal
23+
configuration, shell startup files, or an activated virtual environment.
24+
Record whether the package cache starts empty. Do not print secrets.
25+
4. Define a time limit before starting. Give each check its own log and exit
26+
status so a failed or timed-out test does not hide independent lint and
27+
formatting results. Package installation needs registry access. Credential
28+
removal and pytest marker filtering do not establish network isolation.
29+
5. Start an agent session without earlier project conversation. Provide the
30+
checkout location and execution/logging constraints, but let the agent find
31+
the guides and choose the setup commands.
32+
33+
Example task prompt:
34+
35+
> Discover the contributor setup guidance in this checkout, install development
36+
> dependencies in a fresh supported Python virtual environment, and run the
37+
> documented initial local checks. Report exactly what passed, failed, was
38+
> blocked, or was not run. Explain formatting versus checking based on what you
39+
> find. Do not modify tracked source, publish anything, or configure/use an LLM
40+
> provider. Integration and live-provider tests are outside this bounded run.
41+
> Do not replace the documented initial unit selection with a smoke subset.
42+
> Record timeouts as incomplete. Diagnose failures without silently changing
43+
> the checkout or claiming full PR compliance.
44+
45+
## Observe and record
46+
47+
Keep the original prompt and a transcript of the agent's guide discovery,
48+
commands, relevant output, exit codes, and elapsed times. Record any operator
49+
intervention or sandbox/network restriction separately from project failures.
50+
For the current guide, check that the agent:
51+
52+
- Finds the canonical setup through the repository documentation and uses a
53+
supported Python version in a newly created, activated virtual environment.
54+
- Installs development dependencies and records the resulting tool versions
55+
and any lockfile change.
56+
- Attempts the complete `make test-unit` selection, `make lint`, and
57+
`make format-check`, with a separate outcome for each.
58+
- Recognizes that `make format` edits files, while `make format-check` checks
59+
without applying changes. Capture final `git status`/diff to check this.
60+
- Reports unrun integration/provider checks explicitly, without treating a
61+
skipped or timed-out check as passed.
62+
63+
If a check fails, retain the exact failing command and diagnostic. A focused
64+
diagnostic rerun is additional evidence; it does not replace the original
65+
selection. Attribute missing tools, unavailable registries, collection errors,
66+
test failures, and timeouts separately. Use synthetic configuration only when
67+
needed for a recovery check, and record it. Paid provider inference is not
68+
required for this observation.
69+
70+
Some unit tests need loopback HTTP servers and local multiprocessing sockets.
71+
If a sandbox denies these operations, an authorized rerun in an appropriately
72+
permitted environment can help attribute the failure. Keep the original full
73+
target's outcome alongside any focused control; do not generalize a subset pass.
74+
75+
## Publish the result
76+
77+
Use a dated report beside this procedure. Include:
78+
79+
- Source commit, environment/tool versions, initial state, agent identity,
80+
original task, and the observation's time limits.
81+
- Guide-discovery path and exact setup/check commands, with exit codes, test
82+
counts, timings, and diagnostic excerpts. Label excerpts and any redactions.
83+
- Passed, failed, blocked, and not-run checks; actionable defects and the
84+
evidence supporting their attribution; final tracked-file status.
85+
- Limits: tested OS/Python/package-manager path, source provisioning, agent
86+
version uncertainty, network observation, and checks outside the run.
87+
88+
The initial checks are a bounded onboarding result. `make test` also includes
89+
integration tests and remains the stated contributor requirement. If that
90+
evidence is missing, record the submission limit and seek a maintainer decision
91+
about who supplies validation; do not grant a unit-only exemption or assume
92+
another pipeline will cover the PR.
93+
94+
Observed runs:
95+
96+
- [6 November 2026](contributor-onboarding-2026-11-06.md)
Lines changed: 248 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,248 @@
1+
# Contributor onboarding observation: 6 November 2026
2+
3+
This report records one coding agent following the contributor setup on
4+
[commit `3fb77b4cc9a97724481a6745157efcbfb20e2f32`](https://github.com/NVIDIA/SkillSpector/tree/3fb77b4cc9a97724481a6745157efcbfb20e2f32).
5+
It evaluates that revision's initial setup/check path. It does not establish full
6+
PR compliance or validate later code changes. See the
7+
[repeatable procedure](CONTRIBUTOR_ONBOARDING.md).
8+
9+
The agent found the guides, installed development dependencies, and passed lint
10+
and formatting checks. The full initial unit target **failed**: 20,909 passed,
11+
4 failed, and 51 had setup errors. A focused control outside the execution
12+
sandbox recovered 54 affected cases; one macOS ACL test still failed. The
13+
onboarding observation is therefore complete with an unresolved test failure,
14+
not a green initial-check result.
15+
16+
## Environment and task
17+
18+
| Item | Observed value |
19+
|------|----------------|
20+
| OS | macOS 26.7.2, build 25G332, arm64 |
21+
| Shell | Bash 3.2.57(1)-release, `--noprofile --norc` |
22+
| Python | CPython 3.12.13, fresh project `.venv` |
23+
| Package manager | uv 0.10.9 (`f675560f3`, 2026-03-06) |
24+
| Test/lint tools after install | pytest 9.1.1; Ruff 0.15.19 |
25+
| Git / Make | Git 2.54.0 (Apple Git-157); GNU Make 3.81 |
26+
| Agent | Codex coding subagent, started without the parent conversation; exact model/client build unavailable in this session |
27+
| Source provisioning | Clean local `git clone --no-hardlinks --no-checkout` of fetched public source, then detached checkout of the commit above; HTTPS cloning was not tested |
28+
| Initial state | No `.venv` or `.env`; empty temporary HOME, TMPDIR, and uv cache |
29+
| Unit-selection bound | 900 seconds, declared before execution |
30+
31+
The observed agent's repository reads and commands went through a wrapper that
32+
supplied an explicit subprocess environment instead of inheriting the caller's.
33+
It started Bash with `--noprofile --norc`, captured the exact script, combined
34+
stdout/stderr, exit status, and elapsed time, and terminated the process group
35+
at the command's timeout. The observer independently checked those receipts.
36+
37+
The environment allowlist contained `HOME`, `PATH`, `TMPDIR`, `UV_CACHE_DIR`,
38+
`UV_PYTHON_INSTALL_DIR`, `UV_PYTHON_DOWNLOADS=never`, `UV_LINK_MODE=copy`,
39+
`PYTHONNOUSERSITE=1`, `PYTHONUNBUFFERED=1`, `GIT_CONFIG_GLOBAL=/dev/null`,
40+
`GIT_CONFIG_NOSYSTEM=1`, `NO_COLOR=1`, and `LC_ALL=en_US.UTF-8`. `PATH` exposed
41+
the existing uv/Python tools and system executables; activation added `.venv/bin`.
42+
No provider credential variables were passed, no `.env` was copied, and the
43+
temporary home had no personal provider configuration. This did not isolate the
44+
filesystem or monitor/block network traffic. Package downloads used the network.
45+
46+
The task given to the agent included this instruction (excerpt):
47+
48+
> Discover the contributor setup guidance, install development dependencies in a
49+
> fresh supported Python virtual environment, run the documented initial local
50+
> checks, and report exactly what passed, failed, was blocked, or was not run.
51+
> Explain formatting versus checking based on what you find. Do not modify
52+
> tracked source, publish anything, or configure/use any LLM provider; integration
53+
> and provider tests are outside this bounded run. Do not replace the full
54+
> documented initial unit selection with a smoke subset. If it exceeds a
55+
> 900-second limit, record timeout rather than passing. A failure is a valid
56+
> observed result; diagnose it briefly without silently changing the checkout or
57+
> claiming full PR compliance.
58+
59+
The agent also received the provisioned checkout/revision, wrapper invocation,
60+
logging constraints, and permission to retry package downloads outside the
61+
restricted execution sandbox. It was not given setup commands or guide paths.
62+
Independent lint/format/version observations were allowed while tests ran.
63+
64+
## Guide discovery
65+
66+
The agent enumerated repository files, read `README.md` and `CONTRIBUTING.md`,
67+
then inspected `Makefile`, `pyproject.toml`, and `docs/DEVELOPMENT.md`. It found
68+
the canonical [README Setup section](https://github.com/NVIDIA/SkillSpector/blob/3fb77b4cc9a97724481a6745157efcbfb20e2f32/README.md#setup),
69+
the links to it from the other two guides, the supported Python range
70+
`>=3.12,<3.16`, and all three intended initial checks. Its first `rg` discovery
71+
command was unavailable on the wrapper's restricted PATH; it recovered with
72+
`find`/`grep`. That was a harness-tool limitation, not a repository prerequisite
73+
failure.
74+
75+
Discovery command excerpts from receipts 002 and 003:
76+
77+
```bash
78+
cat README.md
79+
if [ -f CONTRIBUTING.md ]; then cat CONTRIBUTING.md; fi
80+
nl -ba Makefile
81+
nl -ba pyproject.toml
82+
sed -n '1,150p' docs/DEVELOPMENT.md
83+
```
84+
85+
Reading the Makefile established that `make format` runs `ruff check --fix` and
86+
`ruff format` in place. The agent chose `make format-check`; it did not run the
87+
modifying target. The agent also identified the separate `make test` submission
88+
requirement and its integration/provider prerequisites.
89+
90+
## Check results
91+
92+
Times below are wrapper wall-clock times. Each check ran from the checkout root
93+
in a separate shell; the virtual environment was activated again for each one.
94+
Guide dumps and progress polling are omitted from this transcript. Output shown
95+
below is excerpted; temporary paths and the ACL username are normalized.
96+
97+
| Receipt | Command/step | Result | Seconds |
98+
|---------|--------------|--------|---------|
99+
| 004 | Create `.venv`, activate, `make install-dev` | Venv created; install failed (exit 2): sandbox DNS lookup for `files.pythonhosted.org` | 4.312 |
100+
| 005 | Activate, retry `make install-dev` with permitted package-network access | Passed (exit 0); 158 packages installed; initial cache was empty before the first attempt | 13.384 |
101+
| 006 | Activate, `make test-unit` | Failed (exit 2); no timeout | 588.299 |
102+
| 007 | Activate, `make lint` | Passed (exit 0), `All checks passed!` | 0.624 |
103+
| 008 | Activate, `make format-check` | Passed (exit 0), `365 files already formatted` | 0.125 |
104+
| 020 | Focused 55-case control outside the execution sandbox | 54 passed, 1 failed (exit 1); no timeout | 156.917 |
105+
| 023 | Direct stdlib ACL-write probe outside the execution sandbox | Write-only open succeeded; no SkillSpector call | 0.040 |
106+
| 024 | Final version, Git status/diff, `.env`/`.venv` checks | Passed (exit 0); tracked checkout unchanged, including `uv.lock`; `.env` absent | 0.294 |
107+
108+
The install retry changed the execution permission context, not the recipe or
109+
source. Lint and formatting ran independently while the unit selection ran.
110+
111+
Exact setup and initial-check scripts:
112+
113+
Receipt 004:
114+
115+
```bash
116+
set -e
117+
uv venv --python 3.12 .venv
118+
source .venv/bin/activate
119+
make install-dev
120+
```
121+
122+
Receipt 005:
123+
124+
```bash
125+
set -e
126+
source .venv/bin/activate
127+
make install-dev
128+
```
129+
130+
Receipt 006:
131+
132+
```bash
133+
source .venv/bin/activate
134+
make test-unit
135+
```
136+
137+
Receipt 007:
138+
139+
```bash
140+
source .venv/bin/activate
141+
make lint
142+
```
143+
144+
Receipt 008:
145+
146+
```bash
147+
source .venv/bin/activate
148+
make format-check
149+
```
150+
151+
The full unit command expanded to:
152+
153+
```text
154+
pytest -m "not integration and not provider" tests/
155+
collected 21267 items / 258 deselected / 21009 selected
156+
4 failed, 20909 passed, 41 skipped, 258 deselected, 4 xfailed,
157+
1060 warnings, 51 errors in 582.68s (0:09:42)
158+
make: *** [test-unit] Error 1
159+
```
160+
161+
The warnings included unregistered `pytest.mark.timeout` markers. This run's
162+
900-second outer limit came from the observation wrapper; the markers alone
163+
are not evidence that individual test timeouts were active.
164+
165+
## Failure attribution and control
166+
167+
All 51 setup errors in `tests/unit/test_tp4_openai_http.py` failed while binding
168+
the fixture's HTTP server to `127.0.0.1`, with
169+
`PermissionError: [Errno 1] Operation not permitted`. These fixtures use a local
170+
HTTP stub and synthetic provider configuration. They are not live-provider
171+
acceptance tests.
172+
173+
Three failures in `tests/test_batch_scan_security.py` had the same denied local
174+
socket bind in the multiprocessing manager, followed by `EOFError` or a missing
175+
manager-start assertion:
176+
177+
- `test_worker_timeout_kills_scan_and_releases_pool_capacity`
178+
- `test_process_pool_shares_slots_and_cancels_waiting_owner`
179+
- `test_api_pool_manager_stops_when_supervisor_dies`
180+
181+
The observer requested one focused control with the same source, virtual
182+
environment, allowlisted environment, and a 450-second bound, changing only the
183+
execution sandbox permission context. Exact script:
184+
185+
```bash
186+
source .venv/bin/activate
187+
pytest -q tests/test_batch_scan_security.py::test_worker_timeout_kills_scan_and_releases_pool_capacity tests/test_batch_scan_security.py::test_process_pool_shares_slots_and_cancels_waiting_owner tests/test_batch_scan_security.py::test_api_pool_manager_stops_when_supervisor_dies tests/unit/test_suppression.py::test_dump_baseline_accepts_acl_write_grant_without_mode_write_bits tests/unit/test_tp4_openai_http.py
188+
```
189+
190+
Output excerpt:
191+
192+
```text
193+
FAILED tests/unit/test_suppression.py::test_dump_baseline_accepts_acl_write_grant_without_mode_write_bits
194+
1 failed, 54 passed in 155.75s (0:02:35)
195+
```
196+
197+
All three multiprocessing cases and all 51 loopback HTTP cases recovered. This
198+
supports attributing those original outcomes to execution-sandbox restrictions.
199+
It does not turn the original full unit target into a pass; the complete target
200+
was not rerun outside the sandbox.
201+
202+
The remaining macOS case is reproducible in both runs:
203+
`tests/unit/test_suppression.py::test_dump_baseline_accepts_acl_write_grant_without_mode_write_bits`.
204+
It creates a baseline with mode `0444`, adds a user ACL write grant, then calls
205+
`dump_baseline`. The call fails at
206+
[`suppression.py:801`](https://github.com/NVIDIA/SkillSpector/blob/3fb77b4cc9a97724481a6745157efcbfb20e2f32/src/skillspector/suppression.py#L801):
207+
208+
```text
209+
descriptor = os.open(absolute.name, flags, dir_fd=directory_fd)
210+
PermissionError: [Errno 13] Permission denied: 'baseline.yaml'
211+
```
212+
213+
A separate temporary-file control applied the same mode and ACL grant, then
214+
called `os.open(output, os.O_WRONLY)` directly without SkillSpector. It succeeded:
215+
216+
```text
217+
0: user:CURRENT_USER allow write
218+
Direct stdlib os.open(O_WRONLY): SUCCESS
219+
```
220+
221+
The host can honor this ACL grant for a direct write-only open. The production
222+
path additionally uses `O_NONBLOCK`, `O_NOFOLLOW`, and a directory descriptor;
223+
those differences and the test fixture were not isolated further. Investigate
224+
the macOS baseline-write/test path before claiming the failure is a general host
225+
ACL limitation or a confirmed product root cause. No runtime fix was attempted
226+
in this observation.
227+
228+
## Remaining limits
229+
230+
- **Passed:** guide discovery, fresh uv development installation after the
231+
network retry, lint, non-modifying format check, and final tracked-file checks.
232+
- **Failed:** the original full unit target and the focused control's one
233+
persistent macOS ACL case. No check hit its declared timeout.
234+
- **Not run:** `make test`, integration/live-provider targets, coverage, Docker,
235+
Node extension checks, other CI jobs, the pip fallback, other OS/Python paths,
236+
and an HTTPS clone. No full submission or CI pass is claimed.
237+
- **Submission limit:** `CONTRIBUTING.md` still requires `make test`, including
238+
integration tests. This observation supplies no integration evidence and
239+
establishes no approved replacement/handoff policy. A maintainer must resolve
240+
missing validation; the initial checks do not authorize a unit-only exemption.
241+
- **Reproducibility limit:** the source and tool versions above are recorded,
242+
but the exact agent/client build was unavailable. This is one observed task,
243+
not a model-pinned benchmark or proof of general agent success. No network
244+
capture was performed; marker exclusions do not prove zero network access.
245+
- **Revision limit:** this observation preceded this PR's sandbox troubleshooting
246+
note and validation documents. Rerun it after setup, command, dependency, or
247+
contribution-workflow changes. Broader contribution-type discovery was outside
248+
this task, and historical release checks were not reused as candidate evidence.

0 commit comments

Comments
 (0)