[Ready for Review - fixing] Adapter: ARC-AGI-2 benchmark by Ji-Pengliang · Pull Request #220 · harbor-framework/harbor

Ji-Pengliang · 2025-12-13T07:28:02Z

This feature aims to add ARC-AGI-2 benchmark suite to the adapter list.

Source:

Oracle Test

Oracle has passed 100%:

Parity Test

Overview

Field	Value
Date	December 28, 2025
Benchmark	ARC-AGI-2
Dataset	120 puzzles (167 task-pairs) from official evaluation set
Metric	Pass@2 (at least 1 correct out of 2 attempts per task-pair)
Trials	3 independent runs per framework

Settings

The Harbor setup is aligned with the official arc-agi-benchmarking configuration across both model settings and execution environment. Specifically, both use the gpt-5.2 model from the OpenAI provider with the codex agent, medium reasoning effort, and default sampling parameters (temperature and top_p), as well as an identical timeout of 200 minutes. Execution is conducted in a Docker container in both cases, with the Codex CLI running inside the container, configuration specified via config.toml (including reasoning_effort), and outputs written to the same path (/testbed/output.json). This ensures strict equivalence between the two setups and eliminates configuration or environment discrepancies as a source of performance differences.

Results

Per-Trial Results (Pass@2)

Trial	Harbor Adapter	arc-agi-benchmarking Original
(1)	35.90%	34.17%
(2)	38.90%	38.75%
(3)	33.33%	35.42%
(4)	35.42%	35.42%
(5)	36.67%	34.58%
(6)	34.58%	37.92%

Aggregate Statistics

Framework	Mean	Std Error
Harbor Adapter	35.80%	±1.90%
arc-agi-benchmarking Original	36.04%	±1.86%

Conclusion

Mean difference is ±0.24%.
Both frameworks use identical model settings, agent configuration, and execution environment

Repositories:

Harbor Adapter fork: https://github.com/Ji-Pengliang/harbor
arc-agi-benchmarking fork: https://github.com/Ji-Pengliang/arc-agi-benchmarking
Dataset: https://github.com/arcprize/ARC-AGI-2

The experiment logs are available at the following link for reference:
https://huggingface.co/datasets/Ji-Pengliang/Harbor-Parity-Test-ARC-AGI-2

Oracle Test with harbor-dataset merged

PR merged: harbor-framework/harbor-datasets#34

Oracle Test passed:

gemini-code-assist · 2025-12-13T07:28:21Z

Summary of Changes

Hello @Ji-Pengliang, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands the capabilities of the Harbor system by adding a dedicated adapter for the ARC-AGI-2 benchmark. This integration allows for the automated conversion of complex abstract reasoning tasks into a standardized, executable format, complete with all necessary files for agent evaluation. The new adapter streamlines the process of setting up and running experiments with ARC-AGI-2, providing a robust framework for testing and comparing AI agents on these challenging grid transformation puzzles.

Highlights

New Benchmark Integration: This pull request introduces a new adapter to integrate the ARC-AGI-2 benchmark suite into the existing adapter list, enabling the conversion of its abstract reasoning tasks into Harbor-compatible formats.
Task Conversion Logic: A Python-based adapter (adapter.py) has been added to load ARC-AGI-2 tasks from HuggingFace datasets, parse them into ARCAGIRecord objects, and generate structured Harbor task directories, including handling multi-test splitting.
Comprehensive Task Structure: Each generated Harbor task includes instruction.md, task.toml, environment/Dockerfile, tests/test.sh, tests/verify.py, tests/config.json, tests/expected.json, solution/solve.sh, and solution/expected.json, providing a complete environment for agents.
Command-Line Utility: A run_adapter.py script is provided to facilitate the command-line execution of the conversion process, allowing users to specify task directories, splits (training/evaluation), individual task IDs, timeouts, and other generation options.
Detailed Documentation and Configuration: A README.md file offers extensive usage instructions, task structure details, answer format, evaluation steps, and metrics. Additionally, an arc_agi_2.yaml configuration file is included for defining Harbor job settings.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a new adapter for the ARC-AGI-2 benchmark, which is a great addition. The implementation is well-structured, including documentation, configuration, and necessary scripts. My review focuses on a critical issue in the Dockerfile that will prevent it from building, using the official data source for the benchmark, improving performance in data loading, and some smaller suggestions for code clarity and robustness.

linhaowei1

This PR is very clean! I have used codespace to check that the adapter can correctly produce the datasets. I left two questions related to the number of tasks and the config.json.
Please also upload the dataset to https://github.com/laude-institute/harbor-datasets and register this adapter to registry.json.
Please also resolve the comments from gemini-code-assist:)

linhaowei1

Perfectly done 💯! Thank you so much for the hard work.

Slimshilin

The PR looks neat and clear! Great work and thank you so much for the effort and active engagement here. I think two more things left

The original benchmark seems to have both higher means and std than the adapters, and it is relatively significant considering the dataset size. So I wonder if we should run another 1-2 round parity to get more stable scores? None of the trials seem to share the same score, so I think some more runs would be helpful
Let's resolve the registry issue.

But other than that, I think the adapter is well implemented! 💯

Ji-Pengliang · 2025-12-28T23:31:36Z

The PR looks neat and clear! Great work and thank you so much for the effort and active engagement here. I think two more things left

The original benchmark seems to have both higher means and std than the adapters, and it is relatively significant considering the dataset size. So I wonder if we should run another 1-2 round parity to get more stable scores? None of the trials seem to share the same score, so I think some more runs would be helpful

Let's resolve the registry issue.

But other than that, I think the adapter is well implemented! 💯

Thank you @Slimshilin ! I’ve attached the updated sanity test results and logs above, and I’ve pushed the fixed registry in the latest commit.

Ji-Pengliang · 2025-12-28T23:45:27Z

Looks like GitHub is still showing “This branch has conflicts that must be resolved.” which is outdated. I already pushed a fix in the latest commit. It might just need a refresh on GitHub’s side. @Slimshilin

Add Harbor adapter for ARC-AGI-2 abstract reasoning benchmark. Features: - Loads tasks from HuggingFace (Ardea/arc_agi_v2) - Generates Harbor-compatible task directories - Splits multi-test tasks into separate Harbor tasks - ASCII grid visualization in instructions - Binary reward scoring (exact grid match) - Oracle agent support for verifier testing Dataset: 1,000 training + 120 evaluation tasks Source: https://github.com/arcprize/ARC-AGI-2

- Aggregates 167 task results to 120 puzzle accuracy - Matches official ARC-AGI-2 leaderboard scoring - Puzzle passes only if ALL test pairs are correct

- Added 'score' subcommand to run_adapter.py - Moved scoring functions to utils.py - Removed standalone compute_accuracy.py - Usage: run_adapter.py score --result-path <path>

Official scoring (from arc-agi-benchmarking): - Per test pair: pass@k - if ANY attempt is correct, pair passes - Per puzzle: score = correct_pairs / total_pairs - Overall: sum(puzzle_scores) / num_puzzles Now reports: - Trial-level: raw attempt counts - Pair-level: pass@k grouped by test pair - Puzzle-level OFFICIAL: fractional score - Puzzle-level STRICT: all-or-nothing

…vars

- Update instruction.md to match official arc-agi-benchmarking prompt - Add parity_experiment.json for Harbor vs arc-agi-benchmarking comparison

- Add parity experiment results: Harbor 37.3% vs arc-agi-benchmarking 38.9% (pass@2) - Update instruction.md to match arc-agi-benchmarking prompt format

- Rewrite README following Harbor adapter template - Add comprehensive documentation with parity results - Add pyproject.toml with datasets dependency

This reverts commit afb2a29.

Slimshilin

Great work and thank you so much for the adapter! Remember to update the parity result link - but everything else lgtm now so I will merge 👍

…ework#220) * feat(adapters): add ARC-AGI-2 benchmark adapter Add Harbor adapter for ARC-AGI-2 abstract reasoning benchmark. Features: - Loads tasks from HuggingFace (Ardea/arc_agi_v2) - Generates Harbor-compatible task directories - Splits multi-test tasks into separate Harbor tasks - ASCII grid visualization in instructions - Binary reward scoring (exact grid match) - Oracle agent support for verifier testing Dataset: 1,000 training + 120 evaluation tasks Source: https://github.com/arcprize/ARC-AGI-2 * fix: update arc_agi_2.yaml dataset path to correct tasks location * feat: add compute_accuracy.py for official ARC-AGI-2 puzzle-level metric - Aggregates 167 task results to 120 puzzle accuracy - Matches official ARC-AGI-2 leaderboard scoring - Puzzle passes only if ALL test pairs are correct * refactor: integrate scoring into run_adapter.py and utils.py - Added 'score' subcommand to run_adapter.py - Moved scoring functions to utils.py - Removed standalone compute_accuracy.py - Usage: run_adapter.py score --result-path <path> * feat: add pass@k support for official ARC-AGI-2 scoring Official scoring (from arc-agi-benchmarking): - Per test pair: pass@k - if ANY attempt is correct, pair passes - Per puzzle: score = correct_pairs / total_pairs - Overall: sum(puzzle_scores) / num_puzzles Now reports: - Trial-level: raw attempt counts - Pair-level: pass@k grouped by test pair - Puzzle-level OFFICIAL: fractional score - Puzzle-level STRICT: all-or-nothing * chore: set n_attempts=2 to match official pass@2 scoring * Update ARC-AGI-2 instruction template and utils * Update arc_agi_2.yaml: terminus-2 agent, n_concurrent_trials=24, env vars * Set n_concurrent_trials to 8 * fix(arc_agi_2): set default config for pass@2 scoring * Add ARC-AGI-2 parity experiment config - Update instruction.md to match official arc-agi-benchmarking prompt - Add parity_experiment.json for Harbor vs arc-agi-benchmarking comparison * Update adapter_pr with arc-agi-benchmarking repo URL * Update ARC-AGI-2 parity experiment results and instruction template - Add parity experiment results: Harbor 37.3% vs arc-agi-benchmarking 38.9% (pass@2) - Update instruction.md to match arc-agi-benchmarking prompt format * Update ARC-AGI-2 parity experiment results * Update arc_agi_2.yaml to use agentx with gpt-5.2 * Update ARC-AGI-2 adapter README and add pyproject.toml - Rewrite README following Harbor adapter template - Add comprehensive documentation with parity results - Add pyproject.toml with datasets dependency * Remove unused config.json generation from ARC-AGI-2 adapter * Add ARC-AGI-2 adapter to registry (167 tasks) * Update ARC-AGI-2 description to high-level benchmark overview * update readme, registry and parity exp with the latest results * resolve conflict * Update registry.json * Revert "Update registry.json" This reverts commit afb2a29. * style: format arc_agi_2 adapter files with ruff * fix: remove duplicate arc_agi_2 entry and add git_commit_id to tasks * update parity exp --------- Co-authored-by: Horde <[email protected]>

* Update hello cuda. * Fix typing. * Support Azure OpenAI endpoints for LiteLLM agents (#269) * Extend API key forwarding to support Azure OpenAI endpoints for agents that use litellm * Add docs and whitespace * Remove unreachable catch-all * Use openhands model-agnostic key names * Rename variable to avoid shadowing builtin * Preserve prior behavior exactly * Preserve exception * Fix the registry. * Do not pass host env variables to Docker environment (#272) This is a security risk, and it also creates an inconsistent user experience for Docker compared to other environments Proposal: adding an explicit `--rollout-env-var` CLI arg to pass environment values. * Terminus-2: Support optional interleaved thinking (#265) * Support interleaved thinking * Add unit test and integration test * Fix ruff * Reduce duplicate code * Add initial CITATION.cff file (#260) * feat(mini-swe-agent): pass through OPENAI_API_BASE for custom endpoints (#251) * initial commit to enable agent setup timeout override (#247) * fix: add Alpine Linux support for claude-code agent (#259) * fix: add Alpine Linux support for claude-code agent Fixes #258 Changes: - Add distro detection in claude-code install script - Alpine: uses apk to install curl, bash, nodejs, npm - Debian/Ubuntu: uses apt-get + nvm (unchanged) - Add hello-alpine example task Note: Alpine-based task containers must have bash installed in their Dockerfile for Harbor to work (Harbor uses bash for command execution). * fix: add set -e and handle unsupported distributions - Add set -e for early failure on errors - Add explicit error message and exit for unsupported distros - Add comment clarifying Alpine uses Node.js 24.x Addresses Gemini review comments on PR #259 * fix: improve test.sh robustness - Add set -e for early failure on setup errors - Simplify test result checking using command as if condition - Disable set -e for pytest to properly capture exit code Addresses Gemini review comments on PR #259 * Update CompileBench adapter, fixing oracle on Daytona (#275) Update the Git commit hash of CompileBench repo, fixing oracle not passing on Daytona (previously oracle only passed when running locally with Docker). $ uv run harbor run --agent oracle -n 15 --dataset [email protected] --registry-path registry.json -e daytona --force-build 15/15 Mean: 1.000 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:02:33 0:00:00 Results written to jobs/2025-12-23__12-48-21/result.json oracle on compilebench ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓ ┃ Metric ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩ │ Agent │ oracle │ │ Dataset │ compilebench │ │ Trials │ 15 │ │ Errors │ 0 │ │ │ │ │ Mean │ 1.000 │ │ │ │ │ Reward Distribution │ │ │ reward = 1.0 │ 15 │ └─────────────────────┴──────────────┘ Note that CompileBench adapter tasks require access to internet (not enabled by default on Daytona Tier 1/2). * Small updates. * Add CI gate for ruff linter on modified files (#274) * Add CI for ruff linter * Modify a file to showcase * Run 1 file left unchanged * Enable linter auto-format in vscode * Auto formatting * Revert "Run 1 file left unchanged" This reverts commit 157723910c846daa2036546de1e3e0a84389de6a. * Merge ruff checks * style: auto-format code with ruff * Auto lint --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * pass env variables when oracle agent executes the command (#284) * [Adapter] GPQA-Diamond Adapter (#264) * init gpqa * Update GPQA Diamond adapter documentation and templates * Update parity_experiment.json and README.md for GPQA Diamond adapter with agent, model, and accuracy metrics * Update GPQA Diamond adapter results to use GPT-5.2 model and improve documentation * Add GPQA-Diamond registry * Update parity_experiment.json to enhance metrics details --------- Co-authored-by: Ubuntu <[email protected]> Co-authored-by: Slimshilin <[email protected]> * revise the parity results (#292) * Adapters: SWE-bench Pro (#249) * swebenchpro * Fix SWE-Bench Pro adapter and maximize oracle pass rate Around half the tasks currently don't pass oracle. This PR fixes maximizes oracle pass rate to 99.3%. Fixes are mostly with Docker images and test harnesses. There are 5 tasks that intrinsically cannot pass oracle (bad gold patch from original benchmark), and there are 3 long-running tasks. All documented in README. * Address Gemini code review comments * Fixes * Further improve adapter * Complete SWE-bench Pro adapter * More parity experiment trials & updated results * Reposition SWE-bench Pro entries in registry.json * Remove broken and redundant autocodebench record from registry.json * Add parity experiment per-trial details * Auto-clone original benchmark repo and copy run_scripts/ to tmp dir (later removed) * Also update parity experiment result table in README * Move copy_tasks.sh to adapters/swebenchpro * Add citation and authors information Added citation and authors section to README.md --------- Co-authored-by: Narek Maloyan <[email protected]> Co-authored-by: Slimshilin <[email protected]> * Add hr command * Fix ruff check on fork (#313) * Fix ruff format CI * make a random edit to trigger ruff linter failure * Avoid applying formatting automatically since it doesn't work on fork * . * Fix for fork * Fix for fork * Another random edit to a file * Apply formatting as suggested * Add smart port handling for `harbor view` (#308) * Add smart port handling for harbor view When port is occupied, behavior depends on --port argument: - Single port (--port 8080): fail with clear error message - Range (--port 8080-8090): try ports in order with feedback - Default: tries 8080-8089 automatically Adds proper error handling with PortInUseError and PortPermissionError exceptions, keeping sys.exit() in CLI layer only. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]> * Use Rich for harbor view output Adds colored, formatted output matching other CLI commands: - Bold header, cyan labels, clickable URL link - Yellow warnings for port scanning - Red errors with dim hints - Proper pluralization (1 job vs 2 jobs) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]> * Address review feedback: imports, port range consistency - Move 'import errno' to top (PEP 8) - Fix suggested range to match default (8080-8089, 10 ports) - Make ports required param in start_server() (DRY) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]> --------- Co-authored-by: Claude Opus 4.5 <[email protected]> * Change opencode command to output JSON format (#316) Co-authored-by: Alex Shaw <[email protected]> * Add override flags * Revert to include env vars in docker. (#318) * feat(viewer): round rewards to 4 decimal places (#319) * made newline a requirement in prompt since small LLMs were failing (#298) * made newline a requirement in prompt since small LLMs were failing * Fix formatting and clarify keystrokes instructions * updated golden tests --------- Co-authored-by: Alex Shaw <[email protected]> * Fix modal (and also fix lock). * Bump versions. * Revert modal app name. * fix: default registry URL reference (#323) * fix: default registry URL reference * fix: gemini review comment * Bump version. * Add CLAUDE.md documentation for AI assistants (#326) * Add CLAUDE.md documentation for AI assistants Add comprehensive documentation explaining the Harbor codebase structure, development workflows, key conventions, and common tasks for AI assistants working with this repository. Also remove CLAUDE.md from .gitignore so it can be tracked in version control. * Update CLAUDE.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: Claude <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Fix formatting. * [Adapter] Adding SWTBench Adapter (#96) * add swtbench adapter * Delete adapters/swtbench/uv.lock * Delete adapters/swtbench/task_artifacts.py * Delete adapters/swtbench/utils.py * edit registry, update readme, update config yaml file * add parity_experiment.json * resolve ruff checks * resolve gemini comments * add registry * remove typos * Revise citation and add contribution details Updated citation format and added contribution guidelines. --------- Co-authored-by: Slimshilin <[email protected]> * [FEATURE] Make asciinema recording optional (#276) * optional terminal recording * Update tmux_session.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * add guarded download before merge * add guarded download before merge --------- Co-authored-by: Benjamin Feuer <[email protected]> Co-authored-by: Alex Shaw <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Bump OpenHands to 1.0.0 in tests and update gold trajectories & traces (#341) * Fix the registry. * [TINY] Warn user if required model_info is left unset (#336) * Update claude_code.py to allow access to all tools (#333) * [TINY] Increase tmux history limit (#337) * Increase tmux history limit * Update src/harbor/agents/terminus_2/tmux_session.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: Benjamin Feuer <[email protected]> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * [FEATURE] Improve LiteLLM Handling of vLLM Hosted Models (#339) * litellm provider canonical name * clearer contract for vLLM hosted models in Harbor * clearer contract for vLLM hosted models in Harbor * clearer contract for vLLM hosted models in Harbor * cleanup * cleanup * fix test --------- Co-authored-by: Benjamin Feuer <[email protected]> * Add get model limit utility and fix Terminus-2 error message (#350) * Add get model limit utility and fix Terminus-2 error message * Update src/harbor/llms/lite_llm.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Restrict environment variables passed to Oracle container (#360) * Add include-standard-metadata option to tasks init (#359) * Make internet configurable from task config (#327) * feat: add support for custom environment implementations via import path (#325) * Change -a shorthand in start-env from --agent to --all (#361) * Fix AttributeError when accessing default environment type in CLI (#362) * feat(registry): add Alibaba terminal-bench-pro dataset (#376) * Fix unit test failure in lite_llm.py (#364) * Fix unit test failure in lite_llm.py It was caused by merge of two concurrent PRs * Do not repeat yourself * [Ready for Review] mmau adapter (#307) * mmau adapter * update parity exp * fix issues * update registry * update readme * update results * update readme * update readme * update readme * Add parity detail. Add parity detail. --------- Co-authored-by: Slimshilin <[email protected]> * [Ready for Review - fixing] Adapter: Humanevalfix (#257) * [Adapter] Humanevalfix adapter * pass lint * Add and test registry * Follow the README template for the doc formatting. * remove tmux and asciinema * state openhands prompts in the README * Modify parity experiment to required format * Update commit id in registry * pass lint * add parity_pr * Postgres registry (#388) * Temp state. * feat(registry): add GitHub Action to sync registry.json to Supabase Add automated sync workflow that triggers when registry.json changes on main: - Creates/updates/deletes datasets, tasks, and metrics in Supabase - Supports dry-run mode for testing - Uses service role key from GitHub secrets * feat(registry): add get_datasets method and update CLI to use registry factory - Add abstract get_datasets() method to BaseRegistryClient - Implement get_datasets() in JsonRegistryClient (returns registry datasets) - Implement get_datasets() in HarborRegistryClient using new get_all_datasets RPC - Update datasets.py CLI to use create_registry_client factory - Refactor HarborRegistryClient to use shared _parse_dataset_data helper * refactor(registry): use Supabase select DSL instead of RPC functions Replace get_dataset and get_all_datasets RPC calls with direct table queries using Supabase's select DSL with nested relations. This removes the need to create custom SQL functions in Supabase. * refactor(registry): use junction tables for dataset-task/metric relationships - Update sync script to diff entire dataset objects (including tasks/metrics) - Use dataset_task and dataset_metric junction tables instead of direct FKs - Never delete from tasks/metrics tables, only from junction tables - Update HarborRegistryClient to query through junction tables - Flatten junction table response for easier comparison in sync script * refactor(registry): simplify base class by removing wrapper method * refactor(registry): use maybe_single() for single dataset fetch * fix(registry): update to match actual Supabase schema Schema changes: - Table names: dataset, task (singular) - Composite primary keys (no id columns) - task: (git_url, git_commit_id, path) - no name column - dataset: (name, version) - dataset_task: references via dataset_name/version and task_git_url/commit_id/path - dataset_metric: has metric_name and kwargs directly (no separate metrics table) Derive task name from path basename in harbor client. * refactor(ci): use uv inline script metadata for sync script * fix(registry): include task name field in schema * refactor(registry): use RegistryClientFactory class pattern * chore: use SUPABASE_SECRET_KEY env var name * refactor: remove unnecessary comments * feat(sync): add tenacity retries to database operations * refactor(sync): use batch inserts for Supabase operations Consolidate individual insert calls into batch operations to reduce API calls. Collect all new tasks, datasets, dataset_tasks, and dataset_metrics into lists, then insert them in single batch calls. * chore: use uv add --script for dependency metadata * fix: add assertion to satisfy type checker for Supabase client * refactor: simplify schema - embed tasks directly in dataset_task - Remove separate task table, tasks now stored directly in dataset_task - Use upsert for datasets instead of separate insert/update logic - Use get_dataset RPC function for single dataset fetch - Leverage ON DELETE CASCADE for cleanup * refactor(sync): use upsert for all batch operations * fix: only clear tasks/metrics for datasets being updated * refactor(sync): true diffing for tasks and metrics Only insert/delete tasks and metrics that have actually changed: - Fetch existing tasks/metrics from Supabase - Compare with registry using set operations - Only insert new items, only delete removed items - Track unchanged datasets separately * refactor(sync): extract diff_dataset function * refactor(sync): convert to Pydantic models - Add pydantic dependency - Create models: Dataset, DatasetTask, DatasetMetric, DatasetDiff, ExistingData, SyncStats - Add DiffStatus enum for type-safe status handling - Use model_dump() for DB operations * refactor(sync): add Registry models and rename to Supabase prefix - Add RegistryTask, RegistryMetric, RegistryDataset models - Rename ExistingData -> SupabaseData - Rename Dataset -> SupabaseDataset - Rename DatasetTask -> SupabaseTask - Rename DatasetMetric -> SupabaseMetric - load_registry now returns list[RegistryDataset] - Organize code into sections with headers * refactor(sync): remove get_name helper, inline logic * refactor(sync): fetch all data in single query with nested select * refactor(sync): embed tasks/metrics in SupabaseDataset - SupabaseDataset now contains tasks and metrics sets directly - Remove SupabaseData wrapper class - fetch_supabase_datasets returns dict[key, SupabaseDataset] - diff_dataset takes just RegistryDataset and SupabaseDataset - Add task_keys() and metric_keys() methods to RegistryDataset * style: format with ruff * fix: add git_url to swtbench-verified and handle null git_commit_id - Add git_url to swtbench-verified tasks in registry.json - Use "HEAD" as default when git_commit_id is null - Add python-dotenv to load .env file for local runs * fix: add git_commit_id to swtbench-verified tasks * Update the registry versions. * fix: use model_config instead of deprecated class Config * Add sync. * Update the registry logic. * Fix the query. * Add registry tests. * fix(sync): include name in task delete filter * fix(sync): include kwargs in metric delete filter * fix: delete tasks/metrics by ID instead of composite key JSONB field comparison via .eq() in PostgREST can be unreliable. Instead of filtering by composite keys including kwargs (JSONB), we now: - Store the id field when fetching existing data - Compare kwargs in Python using json.dumps with sort_keys - Delete by integer ID using .in_("id", ids) This ensures reliable deletion regardless of JSONB comparison quirks. * refactor: load Supabase credentials from environment variables Load HARBOR_SUPABASE_URL and HARBOR_SUPABASE_PUBLISHABLE_KEY from environment variables with hardcoded defaults as fallbacks. This allows key rotation without code changes while maintaining ease of development. * feat: load dotenv in harbor config * Fix the registry. --------- Co-authored-by: Claude <[email protected]> * LFS support. * Remove --verbose flag from datasets list command (#394) Display task counts by default in the datasets list output, removing the need for the -v/--verbose flag. This simplifies the CLI and provides more useful information upfront. Co-authored-by: Claude <[email protected]> * [Ready for Review - fixing] Adapter: ARC-AGI-2 benchmark (#220) * feat(adapters): add ARC-AGI-2 benchmark adapter Add Harbor adapter for ARC-AGI-2 abstract reasoning benchmark. Features: - Loads tasks from HuggingFace (Ardea/arc_agi_v2) - Generates Harbor-compatible task directories - Splits multi-test tasks into separate Harbor tasks - ASCII grid visualization in instructions - Binary reward scoring (exact grid match) - Oracle agent support for verifier testing Dataset: 1,000 training + 120 evaluation tasks Source: https://github.com/arcprize/ARC-AGI-2 * fix: update arc_agi_2.yaml dataset path to correct tasks location * feat: add compute_accuracy.py for official ARC-AGI-2 puzzle-level metric - Aggregates 167 task results to 120 puzzle accuracy - Matches official ARC-AGI-2 leaderboard scoring - Puzzle passes only if ALL test pairs are correct * refactor: integrate scoring into run_adapter.py and utils.py - Added 'score' subcommand to run_adapter.py - Moved scoring functions to utils.py - Removed standalone compute_accuracy.py - Usage: run_adapter.py score --result-path <path> * feat: add pass@k support for official ARC-AGI-2 scoring Official scoring (from arc-agi-benchmarking): - Per test pair: pass@k - if ANY attempt is correct, pair passes - Per puzzle: score = correct_pairs / total_pairs - Overall: sum(puzzle_scores) / num_puzzles Now reports: - Trial-level: raw attempt counts - Pair-level: pass@k grouped by test pair - Puzzle-level OFFICIAL: fractional score - Puzzle-level STRICT: all-or-nothing * chore: set n_attempts=2 to match official pass@2 scoring * Update ARC-AGI-2 instruction template and utils * Update arc_agi_2.yaml: terminus-2 agent, n_concurrent_trials=24, env vars * Set n_concurrent_trials to 8 * fix(arc_agi_2): set default config for pass@2 scoring * Add ARC-AGI-2 parity experiment config - Update instruction.md to match official arc-agi-benchmarking prompt - Add parity_experiment.json for Harbor vs arc-agi-benchmarking comparison * Update adapter_pr with arc-agi-benchmarking repo URL * Update ARC-AGI-2 parity experiment results and instruction template - Add parity experiment results: Harbor 37.3% vs arc-agi-benchmarking 38.9% (pass@2) - Update instruction.md to match arc-agi-benchmarking prompt format * Update ARC-AGI-2 parity experiment results * Update arc_agi_2.yaml to use agentx with gpt-5.2 * Update ARC-AGI-2 adapter README and add pyproject.toml - Rewrite README following Harbor adapter template - Add comprehensive documentation with parity results - Add pyproject.toml with datasets dependency * Remove unused config.json generation from ARC-AGI-2 adapter * Add ARC-AGI-2 adapter to registry (167 tasks) * Update ARC-AGI-2 description to high-level benchmark overview * update readme, registry and parity exp with the latest results * resolve conflict * Update registry.json * Revert "Update registry.json" This reverts commit afb2a29ea9e645247311941d95ed35c2e4033a33. * style: format arc_agi_2 adapter files with ruff * fix: remove duplicate arc_agi_2 entry and add git_commit_id to tasks * update parity exp --------- Co-authored-by: Horde <[email protected]> * update arc-agi-2 parity test pr (#401) * feat(adapters): add ARC-AGI-2 benchmark adapter Add Harbor adapter for ARC-AGI-2 abstract reasoning benchmark. Features: - Loads tasks from HuggingFace (Ardea/arc_agi_v2) - Generates Harbor-compatible task directories - Splits multi-test tasks into separate Harbor tasks - ASCII grid visualization in instructions - Binary reward scoring (exact grid match) - Oracle agent support for verifier testing Dataset: 1,000 training + 120 evaluation tasks Source: https://github.com/arcprize/ARC-AGI-2 * fix: update arc_agi_2.yaml dataset path to correct tasks location * feat: add compute_accuracy.py for official ARC-AGI-2 puzzle-level metric - Aggregates 167 task results to 120 puzzle accuracy - Matches official ARC-AGI-2 leaderboard scoring - Puzzle passes only if ALL test pairs are correct * refactor: integrate scoring into run_adapter.py and utils.py - Added 'score' subcommand to run_adapter.py - Moved scoring functions to utils.py - Removed standalone compute_accuracy.py - Usage: run_adapter.py score --result-path <path> * feat: add pass@k support for official ARC-AGI-2 scoring Official scoring (from arc-agi-benchmarking): - Per test pair: pass@k - if ANY attempt is correct, pair passes - Per puzzle: score = correct_pairs / total_pairs - Overall: sum(puzzle_scores) / num_puzzles Now reports: - Trial-level: raw attempt counts - Pair-level: pass@k grouped by test pair - Puzzle-level OFFICIAL: fractional score - Puzzle-level STRICT: all-or-nothing * chore: set n_attempts=2 to match official pass@2 scoring * Update ARC-AGI-2 instruction template and utils * Update arc_agi_2.yaml: terminus-2 agent, n_concurrent_trials=24, env vars * Set n_concurrent_trials to 8 * fix(arc_agi_2): set default config for pass@2 scoring * Add ARC-AGI-2 parity experiment config - Update instruction.md to match official arc-agi-benchmarking prompt - Add parity_experiment.json for Harbor vs arc-agi-benchmarking comparison * Update adapter_pr with arc-agi-benchmarking repo URL * Update ARC-AGI-2 parity experiment results and instruction template - Add parity experiment results: Harbor 37.3% vs arc-agi-benchmarking 38.9% (pass@2) - Update instruction.md to match arc-agi-benchmarking prompt format * Update ARC-AGI-2 parity experiment results * Update arc_agi_2.yaml to use agentx with gpt-5.2 * Update ARC-AGI-2 adapter README and add pyproject.toml - Rewrite README following Harbor adapter template - Add comprehensive documentation with parity results - Add pyproject.toml with datasets dependency * Remove unused config.json generation from ARC-AGI-2 adapter * Add ARC-AGI-2 adapter to registry (167 tasks) * Update ARC-AGI-2 description to high-level benchmark overview * update readme, registry and parity exp with the latest results * resolve conflict * Update registry.json * Revert "Update registry.json" This reverts commit afb2a29ea9e645247311941d95ed35c2e4033a33. * style: format arc_agi_2 adapter files with ruff * fix: remove duplicate arc_agi_2 entry and add git_commit_id to tasks * update parity exp * update arc-agi-2 parity test pr --------- Co-authored-by: Horde <[email protected]> * [Ready for Review] Adapter: SATBench (#395) * init SATBench * transform to tasks in harbor * trim task.toml and fix a path bug in verify * update README and simplify mapping function * Update adapters/satbench/README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * add parity mode * update timeout * add parity experiment results * add SATBench into registry --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * [Ready for Review - fixing] Adapter: BigCodeBench-Hard (#330) * add bigcodebench adapter * Update adapters/bigcodebench_hard/adapter.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * minior fix * Update adapters/bigcodebench_hard/run_adapter.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * minior fix * minior fix * update registry * fixes * fix doc * fixes * add hf link * fixed * fix parity * fix parity * fix parity --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Slimshilin <[email protected]> * Remove repository path setup from run_adapter.py to fix ruff error (#418) * Remove repository path setup from run_adapter.py to fix ruff error * use ruff format for reformatting the file * Fix registry.json * Format the whole repo. * Fix messed up name. * Fix: Remove duplicate error_msg in OutputLengthExceededError handling (#420) * Fix: Remove duplicate error_msg in OutputLengthExceededError handling When OutputLengthExceededError is raised in terminus_2.py, the error_msg was being added to chat.messages twice: 1. Explicitly on line 968 via chat.messages.append() 2. Implicitly when chat.chat(error_msg) is called inside _query_llm() The Chat.chat() method automatically appends the prompt to messages, so the manual append was causing duplication. This fix removes the redundant manual append, ensuring error_msg appears only once in the conversation history. Fixes: https://github.com/laude-institute/harbor/issues/371 Co-authored-by: openhands <[email protected]> * Remove unit tests, keep only the fix Co-authored-by: openhands <[email protected]> --------- Co-authored-by: openhands <[email protected]> * Enhance documentation and structure in CLAUDE.md and README.md (#399) - Updated CLAUDE.md to include new agent and environment structures, detailing installed, internal, and utility agents. - Added support for additional adapters and improved organization of existing sections. - Corrected typographical errors in README.md for clarity. - Introduced new workflows in CLAUDE.md for syncing dataset registries. This commit improves the overall clarity and usability of the documentation, making it easier for users to understand the framework's capabilities. * Viewer cost estimate with LiteLLM (#367) * fix(viewer): show "No data" instead of "$0.0000" for unavailable cost When cost_usd is null/unavailable, the viewer now displays "No data" in muted gray instead of the misleading "$0.0000". A tooltip explains "Cost data not reported by agent" on hover. Changes: - models.py: Changed cost_usd to float | None (None = not reported) - data.py: Preserve None when cost not reported, accumulate only when present - viewer.html: Display "No data" when cost_usd is null, show actual value (including $0.0000 for free models) when it's a number This distinguishes between: - cost_usd = null → "No data" (agent didn't report cost) - cost_usd = 0.0 → "$0.0000" (free model) - cost_usd > 0 → "$X.XXXX" (actual cost) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]> * Add cost estimation to Harbor viewer using LiteLLM pricing - Show "No data" instead of "$0.0000" when cost unavailable - Estimate cost from tokens using LiteLLM's pricing database - Display "~$X.XXXX" with "Cost (estimate from LiteLLM)" label for estimates - Normalize model names (strip date suffixes) for LiteLLM lookup - Include cached tokens in cost calculation for accurate estimates 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]> * Improve model name normalization for cost estimation - Handle both Anthropic (YYYYMMDD) and OpenAI (YYYY-MM-DD) date formats - Anchor regex to end of string to prevent false matches - Try variants in order: exact match, provider-stripped, date-stripped 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]> * Address code review feedback - Move `import re` to top of file (PEP 8) - Log warning to stderr on cost estimation failure (consistent with file patterns) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]> --------- Co-authored-by: Claude Opus 4.5 <[email protected]> * update parity (#424) Co-authored-by: Slimshilin <[email protected]> * [Ready for Review - final discussion] Adapter: FinanceAgent (#267) * add an adapter to support financeagent benchmark * add parity experiment and refine the draft * remove unnecessary args for custom agents * add terminal version for cli agents * update git commit id * Revise README with new links and contributor details Updated links and added contributor information in README. * fix ruff error * Codex results in addition. --------- Co-authored-by: Slimshilin <[email protected]> * [Ready for review - Final Review] Adapter: LAB-Bench FigQA (#201) * Add labbench figqa code with clean branch history * Resolve conflict at registry.json * Revise authorship and contribution details in README Updated author information and contribution instructions. * Update author name in task.toml * Update canary string in Dockerfile comments * Ruff formatting --------- Co-authored-by: Slimshilin <[email protected]> * Adding swe-agent (#279) * add swe-agent * updates installation * Add SWE-agent example config to repo (#242) * Rename swe-agent-templates.yaml to swe-agent-infer-single.yaml * Update install-swe-agent.sh.j2 * Inline swe-agent inference config (#243) * Update swesmith_infer.yaml * Update swesmith_infer.yaml * test config * temp working version * Update swesmith_infer.yaml * Update swe_agent.py * Update swe-agent-infer-single.yaml * fix config * clean example configs * swe-agent-lm config * Update src/harbor/agents/installed/swe_agent.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update src/harbor/agents/installed/swe_agent.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * fix for installation comments * Update test_simple_agents.py * Update src/harbor/agents/installed/swe_agent.py Co-authored-by: Boxuan Li <[email protected]> * Update swe_agent.py Addressed comments * format --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Boxuan Li <[email protected]> * [Ready for Review - Final Fix] Adapter: bfcl (#358) * feat: Add BFCL adapter for simple_python - Implement adapter for Berkeley Function Call Leaderboard - Support simple_python category (400 tasks) - AST-based function call evaluation - Oracle verification: 400/400 tasks passed (100% reward) * fix: address code review feedback - Refactor task ID collection logic and fix --limit bug - Use template files for task generation (reduces ~140 lines) - Add JSON decode error logging - Simplify condition checks and remove dead code - Fix test.sh Docker path (/tests instead of /app/tests) * feat: BFCL adapter with all single-turn categories * fix: remove --break-system-packages for Java (old pip) * fix: upgrade pip before installing pytest for Java env * feat: increase concurrency to 150 for full run * chore: update metadata and remove backup file - Update README.md with correct task counts - Update bfcl.yaml header comment to reflect all categories - Update parity_experiment.json with oracle verification status - Remove bfcl.yaml.bak backup file * feat: update BFCL adapter with parity testing support - Update adapter.py with improved evaluation logic - Add parity sample (123 tasks) with IDs files - Add bfcl_parity.yaml for parity testing - Add generate_parity_sample.py for reproducibility - Add run_parity_harbor.sh script for batch testing * feat: update bfcl_parity config for Daytona cloud (50 concurrent) * chore: switch to gpt-4o-mini for parity testing * chore: remove unused script * feat: improve instruction clarity for irrelevance detection * Sync prompt template for parity testing * Add BFCL parity experiment results (gpt-5-mini: 82.11% vs Harbor 78.87%) * Revert to working prompt template (78.87% -> 8.1% issue fixed) * Add 'Execute a command' hint to instruction template * Align Harbor prompt with BFCL WriteFile prompt for parity * Update parity results: Harbor 81.83%±1.22% vs BFCL 82.1% (0.27% diff) * Add all 6 parity comparisons (3 handlers × 2 models) * Add adapter_pr link to all parity entries * BFCL adapter: basic cleanup, README needs update before dataset registration * Clean up BFCL adapter: remove redundant ID file, simplify parity_experiment.json * Clarify BFCL adapter scope: single-turn + live only (excludes multi-turn/agentic) * Update BFCL adapter: parity results, README, and config for Daytona * Complete local BFCL adapter modifications, pending review * Register BFCL datasets in registry.json * Update BFCL registry with correct commit ID * Complete BFCL adapter - ready for review * Update README structure and parity_experiment.json format per review * Merge upstream registry.json * Add bfcl and bfcl_parity to registry * Fix ruff linting: remove unused variable, add noqa for E402 * Fix ruff F841: remove unused functions_data variable * Apply ruff format to bfcl adapter files * Update README * Fix ruff linting issues * Remove temporary helper scripts * Add citation --------- Co-authored-by: Slimshilin <[email protected]> * [Ready for Review] Adapter: QCircuitBench (#351) * [add] qcircuitbench adapter v1 * [add] github repo clone; [fix] verification script * [add] clone github subset * [add] robust temp repo deletion on Windows * [add] support for single task generation * [add] save solution.py to verifier output * [add] configuration.yaml and task template * [add] task difficulty * [rename] run_qcircuitbench.yaml * [remove] shot-count metric * [add] README, parity experiment results, [update] registry.json * [fix] ruff format * Add unified exec and xhigh reasoning to codex. * Fix ruff check error for financeagent Adapter (#434) * add an adapter to support financeagent benchmark * add parity experiment and refine the draft * remove unnecessary args for custom agents * add terminal version for cli agents * update git commit id * Revise README with new links and contributor details Updated links and added contributor information in README. * fix ruff error * Codex results in addition. * Sync registry.json with upstream/main Replaced local registry.json with upstream version to avoid formatting conflicts and ensure consistency with the main repository. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> --------- Co-authored-by: Slimshilin <[email protected]> Co-authored-by: Claude <[email protected]> * [Ready for review -Final Fix] Quixbugs adapter (#346) * quixbugs adapter init * full oracle pass * Update adapters/quixbugs/run_adapter.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update adapters/quixbugs/run_adapter.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update adapters/quixbugs/adapter.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * align timeout * update * done * add registry * done * t-bench --> harbor * Update benchmark names and metrics in JSON * Add transitive equivalence * done * add marker --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Slimshilin <[email protected]> * [Ready for review - final fix] Adapter: BixBench (#154) * vc init * fix some of adapters * first oracle test pass * some clean up * id by capsule uuid * copy workspace * update adapter.py * vc Option 3: Wrap flda env as a custom server and interact through custom cli command * Enhance nbcli functionality and update documentation - Added logging for nbcli copy operations in adapter.py. - Updated project description in pyproject.toml. - Changed default value of USE_DOCKER in README.md to false. - Removed unnecessary USE_DOCKER environment variable setting in cli.py. - Cleaned up imports in server.py. - Modified Dockerfile to add uv to PATH and install with --system flag. * Update nbcli dependencies and improve installation instructions - Updated Python version requirement in pyproject.toml to >=3.12. - Upgraded fhda dependency to version 1.5.0 in pyproject.toml and README.md. - Changed build backend to setuptools in pyproject.toml. - Enhanced error handling for fhda imports in server.py with clear installation instructions. - Modified Dockerfile to simplify installation process by removing uv and using pip directly. * simplify * Add nbcli server daemon and update documentation - Introduced a new server daemon (`nbcli-server`) for persistent background operations. - Updated `pyproject.toml` to include the new server command. - Enhanced README.md with architecture details and server management instructions. - Modified CLI commands for improved usage clarity and added support for JSON output. - Updated Dockerfile to set up the server daemon and ensure it starts automatically. * Enhance nbcli initialization and error handling - Updated the `init` command to require both `nb_path` and `work_dir` arguments, improving usage clarity. - Modified error messages to guide users on proper command usage. - Enhanced the `initialize_environment` function to accept `nb_path` and raise an error if not provided. - Adjusted command handlers to check for environment initialization and return appropriate error messages. * Refactor nbcli response handling and update documentation - Removed the option for human-readable output in favor of JSON format for all responses in `cli.py`. - Updated the README.md to remove the `NB_JSON_OUTPUT` environment variable description, reflecting the change in output format. - Simplified the response structure in `server_daemon.py` by consolidating notebook content handling. * heuristic detect R * Refactor nbcli to use HTTP for communication and update dependencies - Replaced Unix socket communication with HTTP requests in `client.py`, enhancing compatibility and ease of use. - Updated `server_daemon.py` to handle HTTP requests instead of socket connections, improving server architecture. - Added `aiohttp` as a dependency in `pyproject.toml` to support the new HTTP server functionality. - Modified Dockerfile to set environment variables for HTTP server configuration. * fix prev. * Update BixBench adapter and instructions for answer output format - Modified `_generate_oracle_answers` method to clarify output format in `adapter.py`. - Updated instructions in `instruction.md` to specify the creation of a plaintext file for final answers. - Enhanced `solve.sh` to write answers directly to `/testbed/answer.txt`. - Refactored `test_outputs.py` to read answers from the new file location instead of log files. * cleanups * empty notebook to begin with * rename * minor cleanups * fix prev * some clean ups * fix ruff * update * rm uv.lock * Refactor BixBench adapter for capsule handling during Docker build - Removed obsolete capsule download methods from `adapter.py`. - Introduced `download_capsule.py` script to manage capsule downloads and extractions during Docker build. - Updated `Dockerfile` to include the new download script and handle capsule data extraction. - Adjusted workspace setup to create an initial empty notebook during the build process, aligning with previous behavior. * simplify prev. and update doc * mop traces of hf download * add options to use nbterm instead * prettier output from nbcli -h * Refactor BixBench adapter to support question-level handling - Introduced separate loading methods for individual questions and all questions within a capsule. - Updated BixBenchRecord to reflect question-specific attributes. - Enhanced prompt templates for multiple-choice and open-ended questions. - Modified instruction and solution templates to align with new question handling. - Updated run_adapter to accept question IDs instead of task IDs for more granular control. - Adjusted test output validation to focus on multiple-choice answers in XML format. * Update BixBench templates to standardize workspace paths and enhance notebook usage - Modified instruction and solution templates to direct outputs to `/workspace/answer.txt` instead of `/testbed/answer.txt`. - Updated the Dockerfile to set the working directory to `/workspace` and copy `nbcli` there. - Adjusted test scripts to reflect the new answer file location and ensure proper logging. - Added instructions for initializing and using the Jupyter notebook environment with `nbcli` commands. * Refactor BixBenchLoader and update documentation for improved clarity - Changed question ID handling in BixBenchLoader to directly use the question's ID from the input items. - Simplified the randomization of answer options by using the question ID for seeding. - Revised the instruction documentation to streamline the initialization and usage of the nbcli commands. - Updated the Dockerfile and pyproject.toml to specify exact versions for dependencies, ensuring consistency in the environment setup. * Refactor BixBench Dockerfile and adapter to streamline script management - Removed the copying of `download_capsule.py` and `entrypoint.sh` from local templates, now downloading them directly from GitHub in the Dockerfile. - Updated the Dockerfile to install `nbcli` from GitHub instead of copying it, reducing local dependencies. - Enhanced the Dockerfile to create directories and install dependencies in a single layer for better caching and efficiency. * Simplified the loading and parsing of questions in bix adapter * wip: adapting for llm as a judge; starting from tests and templates * wip: adapting for llm as a judge; per capsule templates * wip: adapting llm as a judge; add grader cascade * had to micro average * expose commit * minor fixes * fix * reword the instructions * wip cleanup nbcli * wip clean up judge * fix adapter for range verifier and add registry * fix ruff * openai judge (tested on bix-3) * draft bixbench custom agent * wip rewrite server * wip: simplifies server and cli (nbcli for by parity exp only and not for terminal agent) * fix serialization * fix submit_answer * wip: custom agent issues atomic tool calls across harbor env interface * wip: close the agent loop buy building the next agent obs incrementally * wip custom agent: dummy dummy_agent_get_asv * fix custom agent import * wip: debugging connection issue with custom agent * wip: health handle * wip: fix connection and read reward issues * _dummy_get_asv runs and minor fixes * wip: prepare for proper get_asv * ruff format * add parity_debug.yaml as job config * max_steps=1 seems to work with gpt-4o * prev. * use llm range verifier * upgrade custom agent to bixbench 1.5 * wip: upgrade adapter to support bixbench1.5 * fix custom agent matching tool id * wip: misc alignment from proxy * wip: prev * temp remove registry * update registry * update readme * update default agent * add parity subset * wip: refactor custom agent and server * wip: remove old sub commands and write to answer path * verified on bix-37-q4, more json loading try catch (seems to be connection error) * add parity subset question ids * wip: micro parity judge * setting OPENAI_BASE_URL doesn't work well with docker * add parity exp configs, reducing retries * clean up logging outputs * no submission should make judge error out * wip * update readme * tmp remove registry * fix prev * bring back registry * consistent results in parity json vs readme * fix * fix dataset_pr * more notes * tmp remove registry * update registry * [Ready for Review] Adapter: StrongReject (#363) * Add StrongReject adapter * Update code based on gemini suggestions * Update registry.json: move info to the middle of file * Update contribution section * Update code based on reviewers' feedback * update README * fix the import error * remove unnecessary import * update format based on ruff test * Remove canary string and update README * remove redudant comments --------- Co-authored-by: Slimshilin <[email protected]> * fix hello-world (#438) * Update StrongReject adapter with new registry (#446) * update parity_experiments.json and registry * update harbor PR link * Bump version. * [Ready for Review] Adapter: DS1000 (#432) * init * upd * Update README * update README * Update according to Gemini-code-assist's suggestions * update parity experiment * upd registry * update parity * add 1 run for original eval * fix parity_experiment.json * fix according to gemini-code-assist' suggestions * fix ruff errors & format of parity_experiment.json & registry.json * fix registry * [Ready for Review] Adapter: Ineqmath (#403) * [WIP]: add ineqmath adapter placeholder * Finalize ineqmath adapter: code, config, docs, registry * Fix typos and improve clarity in README.md * Clarify task types and reproduction steps in README Updated README to clarify task types and reproduction steps. * Enhance README with evaluation logic section Added evaluation logic details for answer validation and relation types. * Update IneqMath test_outputs * Update registry.json for IneqMath adapter dataset commit * Update IneqMath adapter: run script and submission formatter * Update Readme with official submssion instruction * Update Readme with new section order and more details for demonstration * Update registry.json for IneqMath adapter * update agent version in parity experiment json * Update IneqMath registry to latest dataset commit * update registry json --------- Co-authored-by: Slimshilin <[email protected]> * Make reasoning parameters configurable via kwargs (#444) * Make reasoning parameters configurable via kwargs - Codex: Make reasoning_effort configurable (was hardcoded to xhigh) - Claude Code: Make max_thinking_tokens configurable (was env-only) - OpenHands: Make reasoning_effort configurable (was env-only) All parameters now accept kwargs with fallback to environment variables. This allows per-job configuration without modifying agent code. Example usage: agents: - name: codex kwargs: reasoning_effort: high - name: claude-code kwargs: max_thinking_tokens: 15000 - name: openhands kwargs: reasoning_effort: medium * Fix: Store kwargs in agent __init__ to make them accessible - Added self._kwargs storage in Codex, Claude Code, and OpenHands - Changed self.kwargs references to self._kwargs - Fixes AttributeError when accessing reasoning parameters * Address review feedback: Make kwargs explicit with type hints and defaults - Codex: reasoning_effort: str | None = 'high' - Claude Code: max_thinking_tokens: int | None = None - OpenHands: reasoning_effort: str | None = 'medium' This provides better type safety, IDE autocomplete, and sensible defaults: - Codex defaults to 'high' (works with gpt-5-mini) - Claude Code defaults to None (unlimited thinking tokens) - OpenHands defaults to 'medium' (avoids 400 errors with unverified orgs) Addresses @alexgshaw review comment. * [Ready for Review] Adapter: Algotune (#440) * init algotune * add parity * ruff fix * fix ruff * fix ruff * add oracle infor and some insights in README * fix canary string & README * fix harmonic mean * update dataset pr * update commit-id --------- Co-authored-by: Slimshilin <[email protected]> Co-authored-by: linhaowei <[email protected]> * [Ready for Review] Adapter: Spider2 (#436) * Initial commit of spider2-dbt adapter with Oracle Solution Pass Rate 100% * spider agent * add registry and update parity_template * Update adapters/spider2-dbt/README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update adapters/spider2-dbt/adapter.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * update parity experiment.json * remove duplicate title * update readme * accelerate evaluation. Each time, there is no need to force build docker. * fix registry.json * fix "uvx ruff check --fix ." * running “uvx ruff format adapters/spider2-dbt/__init__.py adapters/spider2-dbt/adapter.py adapters/spider2-dbt/run_adapter.py adapters/spider2-dbt/spider_agent_dbt.py adapters/spider2-dbt/template/tests/test_dbt.py” * clarify pass tasks in readme and parity_experiment.json * remove canary string * remove canary string * fix registry * update parity for canary * Revise maintainer details and contributions section Updated maintainer information and added contribution guidelines. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Slimshilin <[email protected]> * Add custom API base URL support for claude code agent (#236) * feat: Add custom API base URL support and fix project permissions for Claude Code agent - Add support for ANTHROPIC_BASE_URL environment variable to enable custom API endpoints - Add CLAUDE_CODE_MAX_OUTPUT_TOKENS environment variable for controlling output limits - Fix permission issues on project root directory using chmod before reading JSONL files - Prioritize ANTHROPIC_MODEL environment variable over agent model_name for better flexibility - Configure subagent and haiku models to use the same model when custom base URL is set These changes enable Claude Code agent to work with custom API endpoints (e.g., proxy servers, local model servers) and resolve permission errors when accessing project files in sandboxed environments. * Update src/harbor/agents/installed/claude_code.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update src/harbor/agents/installed/claude_code.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * restore the _get_session_dir modification * restore the _get_session_dir modification * Improve Claude Code model config priority and custom API support * uv formated --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * fix(claude-code): copy skills from ~/.claude/skills when CLAUDE_CONFIG_DIR is set (#459) When Harbor sets CLAUDE_CONFIG_DIR=/logs/agent/sessions, Claude Code looks for user skills at $CLAUDE_CONFIG_DIR/skills instead of ~/.claude/skills. Since /logs/agent/ is mounted from the host as an empty volume, skills copied to ~/.claude/skills in task Dockerfiles are not found. This fix copies skills from ~/.claude/skills to $CLAUDE_CONFIG_DIR/skills before running Claude Code, ensuring skills work consistently. This matches the behavior of Codex, which has a fallback to ~/.codex/skills regardless of CODEX_HOME. * chore: pin modal sdk to `1.3.1.dev9` (#457) * [Ready for Review] Adapter: Spider2 - Experiment with Harbor-supported agents (#452) * add Experiment with Harbor-supported agents and small discussion on other splits (snow and lite) * Update adapters/spider2-dbt/README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update adapters/spider2-dbt/README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update adapters/spider2-dbt/README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * Update adapters/spider2-dbt/README.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * fix (adapter): registery (#466) * [ready for review] bixbench-cli addition (#451) * copied cli instructions * minus R and jupyternotebook * init the dockerfile-cli * working bixbench-cli oracle * mv cli * simplify cot instructions * reverse tmp * add bixbench-cli jobs config * add bixbench-cli results and update readme/registry * revert registry --------- Co-authored-by: Slimshilin <[email protected]> * Show hints. * feat: add secret and volume support to modal environment (#470) * add codex trajectory.json back (#458) * add codex trajectory.json back * add supports-atif var * fix ruff * uvx format * resolve gemini comments * add symlink to auth.json and trap rm * Apply suggestion from @li-boxuan --------- Co-authored-by: TheMikeMerrill <[email protected]> Co-authored-by: Boxuan Li <[email protected]> * Update the version. * Revert litellm hack for OpenHands (#475) * Revert litellm hack for togetherAI * Upgrade openhand to 1.1.0 in tests * Improve terminal bench mapper functionality (#476) * [FEATURE] Improve Terminal-Bench mapper - Fix timeout mapping: verifier now uses max_test_timeout_sec instead of max_agent_timeout_sec - Add EXTREMELY_HARD difficulty level to support terminal-bench tasks - Map additional metadata fields: parser_name, run_tests_in_same_shell, disable_asciinema - Improve reward logging: capture exit code reliably and preserve it for downstream tools - Add comprehensive unit tests for the mapper (16 tests) * Remove base image substitution logic from Terminal-Bench mapper The mapper now preserves original Dockerfiles without modifying base images. Terminal-Bench images should be used as-is. * Remove all metadata properties from Terminal-Bench mapper Simplified the mapper to only handle essential config: - instruction, difficulty, and timeouts - Extra fields from terminal-bench are now ignored (extra="ignore") - Removed docker metadata tracking (is_multi_container, custom_compose, etc.) - Methods now return void instead of metadata dicts * Restore terminal-bench metadata while keeping internal tracking removed Keep terminal-bench task metadata (author, category, tags, etc.) but still exclude internal tracking metadata (moved_*, is_multi_container, custom_docker_compose, etc.) that was used to track mapper changes. * Remove solution.yaml support from Terminal-Bench mapper solution.yaml is no longer supported. If a task uses solution.yaml, the mapper now throws an error suggesting to convert it to solution.sh. * Add warnings for unusual docker-compose configurations Instead of storing metadata properties, print warnings for: - Custom command (not the expected sleep infinity) - Custom user - Multi-container setups - Environment variables moved to Dockerfile - Working directory moved to Dockerfile - Entrypoint moved to Dockerfile * Simplify docker-compose mapper - Remove _move_compose_fields_to_dockerfile (mixing runtime/build concerns) - Remove _process_dockerfile and _has_env_var_reference (unused) - Simplify _process_docker_compose: just convert services and copy Dockerfile - Remove tests for deleted methods * Prefer collapsing docker-compose to Dockerfile for cloud compatibility When docker-compose can be represented as just a Dockerfile (single container, no networks/volumes/depends_on), move env vars, working_dir, expose, and entrypoint to the Dockerfile instead of writing a docker-compose.yaml file. This improves compatibility with cloud providers that work better with pure Dockerfiles. Only write docker-compose.yaml for complex setups that require it (multi-container, networks, volumes, etc.). * Fix type check error in test_ignores_unknown_fields * Refactor terminal-bench mapper for better modularity Structure the mapper into clear sections: - Constants: HARBOR_MAIN_SERVICE, REWARD_LOGGING_SCRIPT, COMPOSE_ONLY_KEYS - Models: TerminalBenchTaskConfig with to_harbor_config() method - DockerComposeProcessor: Dedicated class for compose handling - File utilities: Standalone functions for file operations - TerminalBenchMapper: Orchestrates the mapping process This makes the code easier to read, test, and maintain. * Allow default terminal-bench compose to collapse to Dockerfile Template-only volumes (${T_BENCH_...}) are now ignored when checking if a compose can collapse, since Harbor provides its own volume mounts. This means the default terminal-bench docker-compose will collapse to just a Dockerfile, improving cloud compatibility. * Remove unnecessary comments from terminal-bench mapper * Use whitelist approach for docker-compose collapse detection Instead of hardcoding specific fields that prevent collapsing, use a whitelist of allowed fields: - DOCKERFILE_FIELDS: Can be moved to Dockerfile (environment, working_dir, expose, entrypoint) - IGNORED_FIELDS: Safely ignored (build, image, container_name, command, template-only volumes) Any unrecognized field (networks, depends_on, ports, user, healthcheck, privileged, etc.) will prevent collapsing. This is safer than trying to enumerate all compose fields that require docker-compose. * Check for exact terminal-bench template variables Instead of checking for any ${...} pattern, now checks for the exact default terminal-bench volumes and environment variables: - TBENCH_DEFAULT_VOLUMES: The two log path mounts - TBENCH_DEFAULT_ENV: TEST_DIR=${T_BENCH_TEST_DIR} Custom template variables or different volume mounts will not collapse. * Use subset check for terminal-bench default volumes/env Real terminal-bench tasks often use only one default volume mount (${T_BENCH_TASK_LOGS_PATH}:${T_BENCH_CONTAINER_LOGS_PATH}) rather than both. Changed from exact set match to subset check so tasks can collapse to Dockerfile when they use any subset of the allowed default mounts. Tested against terminal-bench-core==0.1.1 dataset: - Before: All 61 tasks got docker-compose.yaml (none collapsed) - After: Only 6 tasks need docker-compose.yaml (multi-container or runtime features like tmpfs) * Report both success and failure counts in task migration The migrate command now displays both successfully migrated and failed task counts at the end of the migration process. Changes: - Added MapResult class to return both mapped paths and failures - Updated CLI to show "✓ Successfully migrated X task(s)" and "✗ Failed to migrate Y task(s)" messages * Preserve non-standard compose fields in Harbor docker-compose When a task can't collapse to Dockerfile and needs docker-compose.yaml, we now preserve all fields from the original main service except those that Harbor explicitly replaces (build, image, container_name, command, environment, volumes, deploy). This fixes issues where fields like tmpfs, networks, depends_on, ports, healthcheck, etc. were being dropped from the mapped docker-compose. * Check for unsupported solution.yaml before creating directories Moved the solution.yaml check to the beginning of _map_task so that failed tasks don't leave partial directories in the output. * Extract _validate_task method for pre-migration checks * Use EXIT trap for reward logging instead of appending The previous approach appended exit code capture at the end of the test script, which failed if the script had explicit exit commands or additional commands after the test. Now we insert a trap after the shebang that runs on any exit, ensuring the reward is always written regardless of how the script terminates. * Simplify reward logging to append suffix instead of trap * Replace T_BENCH_TASK_DOCKER_NAME_PREFIX with hb__ in docker-compose * Use 'hb' instead of 'hb__' for docker name prefix replacement * Fix tests detection to match 'COPY tests' without trailing slash * Add ENV TEST_DIR=/tests to Dockerfile when collapsing * [FEATURE] Improve Terminal-Bench mapper - Fix timeout mapping: verifier now uses max_test_timeout_sec instead of max_agent_timeout_sec - Add EXTREMELY_HARD difficulty level to support terminal-bench tasks - Map additional metadata fields: parser_name, run_tests_in_same_shell, disable_asciinema - Improve reward logging: capture exit code reliably and preserve it for downstream tools - Add comprehensive unit tests for the mapper (16 tests) * Remove base image substitution logic from Terminal-Bench mapper The mapper now preserves original Dockerfiles without modifying base images. Terminal-Bench images should be used as-is. * Remove all metadata properties from Terminal-Bench mapper Simplified the mapper to only handle essential config: - instruction, difficulty, and timeouts - Extra fields from terminal-bench are now ignored (extra="ignore") - Removed docker metadata tracking (is_multi_container, custom_compose, etc.) - Methods now return void instead of metadata dicts * Restore terminal-bench metadata while keeping internal tracking removed Keep terminal-bench task metadata (author, category, tags, etc.) but still exclude internal tracking metadata (moved_*, is_multi_container, custom_docker_compose, etc.) that was used to track mapper changes. * Remove solution.yaml support from Terminal-Bench mapper solution.yaml is no longer supported. If a task uses solution.yaml, the mapper now throws an error suggesting to convert it to solution.sh. * Add warnings for unusual docker-compose configurations Instead of storing metadata properties, print warnings for: - Custom command (not the expected sleep infinity) - Custom user - Multi-container setups - Environment variables moved to Dockerfile - Working directory moved to Dockerfile - Entrypoint moved to Dockerfile * Simplify docker-compose mapper - Remove _move_compose_fields_to_dockerfile (mixing runtime/build concerns) - Remove _process_dockerfile and _has_env_var_reference (unused) - Simplify _process_docker_compose: just convert services and copy Dockerfile - Remove tests for deleted methods * Prefer collapsing docker-compose to Dockerfile for cloud compatibility When docker-compose can be represented as just a Dockerfile (single container, no networks/volumes/depends_on), move env vars, working_dir, expose, and entrypoint to the Dockerfile instead of writing a docker-compose.yaml file. This improves compatibility with cloud providers that work better with pure Dockerfiles. Only write docker-compose.yaml for complex setups that require it (multi-container, networks, volumes, etc.). * Fix type check error in test_ignores_unknown_fields * Refactor terminal-bench mapper for better modularity Structure the mapper into clear sections: - Constants: HARBOR_MAIN_SERVICE, REWARD_LOGGING_SCRIPT, COMPOSE_ONLY_KEYS - Models: TerminalBenchTaskConfig with to_harbor_config() method - DockerComposeProcessor: Dedicated class for compose handling - File utilities: Standalone functions for file operations - TerminalBenchMapper: Orchestrates the mapping process This makes the code easier to read, test, and maintain. * Allow default terminal-bench compose to collapse to Dockerfile Template-only volumes (${T_BENCH_...}) are now ignored when checking if a compose can collapse, since Harbor provides its own volume mounts. This means the default …

…ework#220) * feat(adapters): add ARC-AGI-2 benchmark adapter Add Harbor adapter for ARC-AGI-2 abstract reasoning benchmark. Features: - Loads tasks from HuggingFace (Ardea/arc_agi_v2) - Generates Harbor-compatible task directories - Splits multi-test tasks into separate Harbor tasks - ASCII grid visualization in instructions - Binary reward scoring (exact grid match) - Oracle agent support for verifier testing Dataset: 1,000 training + 120 evaluation tasks Source: https://github.com/arcprize/ARC-AGI-2 * fix: update arc_agi_2.yaml dataset path to correct tasks location * feat: add compute_accuracy.py for official ARC-AGI-2 puzzle-level metric - Aggregates 167 task results to 120 puzzle accuracy - Matches official ARC-AGI-2 leaderboard scoring - Puzzle passes only if ALL test pairs are correct * refactor: integrate scoring into run_adapter.py and utils.py - Added 'score' subcommand to run_adapter.py - Moved scoring functions to utils.py - Removed standalone compute_accuracy.py - Usage: run_adapter.py score --result-path <path> * feat: add pass@k support for official ARC-AGI-2 scoring Official scoring (from arc-agi-benchmarking): - Per test pair: pass@k - if ANY attempt is correct, pair passes - Per puzzle: score = correct_pairs / total_pairs - Overall: sum(puzzle_scores) / num_puzzles Now reports: - Trial-level: raw attempt counts - Pair-level: pass@k grouped by test pair - Puzzle-level OFFICIAL: fractional score - Puzzle-level STRICT: all-or-nothing * chore: set n_attempts=2 to match official pass@2 scoring * Update ARC-AGI-2 instruction template and utils * Update arc_agi_2.yaml: terminus-2 agent, n_concurrent_trials=24, env vars * Set n_concurrent_trials to 8 * fix(arc_agi_2): set default config for pass@2 scoring * Add ARC-AGI-2 parity experiment config - Update instruction.md to match official arc-agi-benchmarking prompt - Add parity_experiment.json for Harbor vs arc-agi-benchmarking comparison * Update adapter_pr with arc-agi-benchmarking repo URL * Update ARC-AGI-2 parity experiment results and instruction template - Add parity experiment results: Harbor 37.3% vs arc-agi-benchmarking 38.9% (pass@2) - Update instruction.md to match arc-agi-benchmarking prompt format * Update ARC-AGI-2 parity experiment results * Update arc_agi_2.yaml to use agentx with gpt-5.2 * Update ARC-AGI-2 adapter README and add pyproject.toml - Rewrite README following Harbor adapter template - Add comprehensive documentation with parity results - Add pyproject.toml with datasets dependency * Remove unused config.json generation from ARC-AGI-2 adapter * Add ARC-AGI-2 adapter to registry (167 tasks) * Update ARC-AGI-2 description to high-level benchmark overview * update readme, registry and parity exp with the latest results * resolve conflict * Update registry.json * Revert "Update registry.json" This reverts commit afb2a29. * style: format arc_agi_2 adapter files with ruff * fix: remove duplicate arc_agi_2 entry and add git_commit_id to tasks * update parity exp --------- Co-authored-by: Horde <[email protected]>

- README.md: `parity-experiments/tree/main/adapters/gdb` 404s (the adapter lives on PR harbor-framework#220, not on main). Point at the HF discussion URL that the other adapters and this adapter's own `parity_experiment.json` already use. - adapter_metadata.json: `parity_matching_agents` now follows the `agent@version+model` format documented in the adapter tutorial, so `[email protected]` becomes `[email protected]+claude-sonnet-4-20250514`. No behaviour or parity-number changes. Made-with: Cursor

* [WIP] Adapter: GDB (GraphicDesignBench) Adapt GDB (GraphicDesignBench) for Harbor. GDB evaluates VLMs/LLMs on professional graphic design tasks across 7 domains: SVG, category, layout, typography, template, temporal, and Lottie. - All 39 benchmarks adapted (33,786 total benchmark samples) - Uses GDB's own Python code for prompt construction and evaluation (exact fidelity by construction) - All GDB metrics reported in reward.json; primary metric to reward.txt - Dependencies (gdb[metrics,svg-metrics]) installed at eval time in test.sh - Parity type: Scenario 2 (original is LLM-based, no agent harness) - 1/(1+error) for error metrics to preserve granularity Made-with: Cursor * Address Devin review: fix loguru import and inf sanitization - Replace loguru with stdlib logging in run_adapter.py to match all other adapters and avoid ModuleNotFoundError (loguru is not a project dependency). Fixes dual-logging mismatch with adapter.py. - Add math.isinf() check alongside math.isnan() in has_real_score and the fallback loop, preventing float('inf') from propagating as a meaningful reward value. Made-with: Cursor * Add logging.basicConfig() so progress output is visible Without basicConfig(), Python's default root logger has no handlers and silently drops INFO-level messages. The user saw no progress during task generation. Matches the pattern used by ~46 other adapters. Made-with: Cursor * Remove narrating comments, redundant docstrings, and defensive check Drop docstrings that restate the method name, decorative section dividers, narrating shell comments, and the template_dir existence check that no other adapter uses. Made-with: Cursor * Remove changelog-style note from README Replace historical note about upstream svg-6 fix with plain statement. Made-with: Cursor * Sanitize reward value for NaN/Inf in write_reward Clamp NaN/Inf to 0.0 before writing reward.txt and reward.json, matching the sanitization already applied to all_scores entries. Made-with: Cursor * Fix review issues: trap cleanup, bake deps into Docker image, harden cache - Replace unreachable fallback in test.sh with trap EXIT pattern so reward.txt is always written even when evaluate.py crashes - Move pip install from test.sh into Dockerfile (avoids repeated downloads; install CPU-only PyTorch, libcairo2-dev, bert-score) - Make _load_sample fall back to _load_samples_for_benchmark when cache is unpopulated instead of silently returning empty list - Use %-style logging in run_adapter.py (consistent with adapter.py) - Update README to reflect deps baked into Docker image Made-with: Cursor * Split optional pip install into separate RUN to avoid masking failures The || true was applying to the entire &&-chain, silently masking failures from torch and gdb installs. Separate the optional Levenshtein/bert-score install into its own RUN layer. Made-with: Cursor * Update sample cache when a larger n is loaded The old condition only stored results when the benchmark was entirely uncached. When a partial cache existed (e.g. n=2) and a larger n was requested, the newly loaded samples were discarded, leaving stale data. Made-with: Cursor * Slice return value to n on non-cached path in _load_samples_for_benchmark If bench.load_data(n=n) returns more than n samples, the non-cached path now correctly slices the result, matching the cached path behavior. Made-with: Cursor * Fill in adapter, dataset, and parity PR links Made-with: Cursor * Track full vs partial cache loads to fix n=None returning stale data Added _fully_loaded set to distinguish complete loads from partial ones. n=None now only returns from cache if the benchmark was previously loaded without a limit; otherwise it reloads all samples as documented. Made-with: Cursor * Add tesseract-ocr to Dockerfile for typography OCR evaluation GDB's typography benchmarks use pytesseract (which calls the tesseract binary) for OCR metrics. Without the system package, _run_ocr silently returns None and all OCR scores become NaN. Made-with: Cursor * Update oracle results: 73/78 after tesseract-ocr and cairo fixes svg-6/7/8 and typography-7/8 now pass with the Dockerfile dependency fixes, reducing expected failures from 13 to 5. Made-with: Cursor * Download remote GT assets at task generation time Some layout-1 samples store ground truth as GCS URLs rather than local files. _find_gt_image now downloads these to temp files so they are packaged into the task directory. Also updated upstream GDB to prefer local copies and index .mp4 files. Oracle: 78/78 tasks pass. Made-with: Cursor * Fix PRIMARY_METRICS to match GDB's actual primary metrics svg-6/7/8 should use mse (lower-is-better), not svg_validity (higher-is-better). typography-2 should use rgb_l2_distance (lower-is-better), not delta_e_below_5 (higher-is-better). layout-1 should use nima_score, not clip_score. The direction mismatch caused incorrect 1/(1+x) reward transformation for svg-6/7/8 and typography-2, producing wrong rewards for imperfect agent outputs. Made-with: Cursor * Remove _download_gt_url — GT assets should be local in the dataset Made-with: Cursor * fix: rename deprecated *_trials fields to *_runs in parity_experiment.json Addresses Devin review comment — the validation script (scripts/validate_adapter.py) expects number_of_runs, original_runs, and harbor_runs instead of the legacy *_trials names. Made-with: Cursor * fix: don't count skipped tasks as generated in run_adapter.py Pre-existing task directories that are skipped (without --overwrite) were incorrectly counted in total_generated. Add a separate total_skipped counter so the summary log accurately reflects work done. Made-with: Cursor * Update parity data with real agent-run results Replace construction-time parity estimates with actual claude-code agent runs. Original: GDB run_benchmarks.py (non-agentic). Harbor: claude-code in Docker (agentic). Both use claude-sonnet-4-20250514. All values converted to reward space for fair comparison. Made-with: Cursor * Update parity results with retry runs (37/39 full coverage) Merge results from 3 harbor runs (initial + 2 retries). All 39 benchmarks now have at least 1 sample; 37 have full 2-sample coverage (layout-2, layout-3 have 1 each due to infra timeouts). 15 exact, 6 close, 10 different, 8 N/A. Made-with: Cursor * Fix PRIMARY_METRICS for 8 benchmarks to match actual GDB evaluate output layout-8: clip_identity -> nima_score lottie-1/2: lottie_validity -> structural_similarity template-3: ari -> nmi temporal-1: percent_perfect -> pairwise_accuracy temporal-4/5/6: generation_success_rate -> lottie_validity Made-with: Cursor * Use logger.debug for per-task output in adapter.py Per-benchmark progress and final summary in run_adapter.py already provide sufficient user-facing feedback at INFO level. Made-with: Cursor * Update parity results with full 2-sample coverage for layout-2 and layout-3 Both benchmarks now have complete 2-sample runs. layout-2 remains 0.0000, layout-3 updated to 0.4899 (avg of 0.5178 and 0.4619). All 39 benchmarks now have full coverage. Made-with: Cursor * Fix parity data consistency: number_of_runs and harbor_runs array lengths - Set number_of_runs to 2 (matching 2 samples per benchmark) - Trim template-2 and template-3 harbor_runs from 3 to 2 entries (3rd entry was from a retry, inconsistent with stated 78 = 2×39) - template-2 now exact match (0.5 vs 0.5); summary updated to 16 exact, 6 close, 9 different, 8 N/A Made-with: Cursor * Update distribution name from gdb to lica-gdb The upstream package was renamed from gdb to lica-gdb in pyproject.toml. Python import name (from gdb.xxx) is unchanged. Made-with: Cursor * Address Claude adapter review feedback parity_experiment.json: - Aggregate harbor_runs to per-run means (single entry each) so that number_of_runs=1 matches array lengths consistently - Pin agent version: claude-code@latest -> [email protected] - Expand notes to explain run structure and nima_score scale adapter_metadata.json: - adapted_benchmark_size: 39 -> 33786 (total task count across 39 benchmarks) - parity_sampling_rate: 0.5 -> 0.0023 (78/33786) - registry_benchmark_size: null -> 78 (parity subset registered) - Remove Ryan Marten; update contact to [email protected] - Pin parity_matching_agents version: [email protected] README.md: - "Number of Trials" -> "Number of Runs" - Clarify nima_score uses native 1-10 scale, not 0-1 - Add Reproduction section with concrete original-side and Harbor-side commands - Justify oracle scope (parity subset vs full 33,786 tasks) - Sync rounded values to JSON (layout-1, layout-5, svg-5, svg-8) - Remove Ryan Marten from Authors section Made-with: Cursor * Refactor gdb adapter to harbor adapter init scaffold Align layout with the standardized package structure produced by `harbor adapter init` (see src/harbor/cli/template-adapter/): adapters/gdb/ ├── pyproject.toml # uv package ├── gdb.yaml # renamed from run_gdb.yaml └── src/gdb_adapter/ ├── __init__.py ├── adapter.py # class renamed: GDBAdapter -> Adapter ├── main.py # standardized CLI entry point └── task-template/ # renamed from template/ Python package is named `gdb_adapter` (not `gdb`) to avoid shadowing the `gdb` module provided by the upstream `lica-gdb` distribution. Adapter constructor now accepts the standardized flags (output_dir, limit, overwrite, task_ids) in addition to GDB-specific ones (dataset_root, benchmarks, split, n_per_benchmark), and exposes a run() method that encapsulates the task-generation loop previously living in run_adapter.py. CLI usage: uv run python -m gdb_adapter.main --dataset-root <path> \ --output-dir datasets/gdb [--split parity|--benchmarks ...|--limit N] README restructured to match the template (adds "Run Evaluation / Harness" and renames "Parity" to "Comparison with Original Benchmark (Parity)") and points users at the new CLI. Made-with: Cursor * Fix lottie oracle output when ground truth is a parsed dict/list In `_format_oracle_text` for `lottie-1`/`lottie-2`, if GDB stores `lottie_json` as an already-parsed dict/list (rather than a serialized string), returning it as-is caused `_build_oracle`'s f-string to call `str()` on it, producing Python repr (`{'v': '5.5.2'}`) instead of valid JSON (`{"v": "5.5.2"}`). The oracle then wrote syntactically invalid JSON to `answer.json`, silently failing evaluation. Now json.dumps non-string values before returning. Flagged by Devin Review on PR #1433. Made-with: Cursor * Switch gdb parity to Scenario 2 (agentic on both sides) Both the upstream GDB harness and the Harbor task now run the same agent (claude-code with claude-sonnet-4-20250514). The upstream side uses the new `claude_code` provider from the harbor-adapter branch of https://github.com/lica-world/GDB, which drives the Claude Code CLI against a temporary workspace and scores with GDB's own evaluator. - Refresh parity_experiment.json "original_runs" with values from the new upstream parity run (parity_claude_code.json), applying the same reward-space transform the Harbor verifier uses (LOWER_IS_BETTER -> 1/(1+x)). - Drop stale Scenario 1 originals for the 4 metrics upstream doesn't emit without optional deps (bertscore_f1, nima_score, ocr_accuracy); record them as N/A with an explicit note. - PRIMARY_METRICS for temporal-4/5/6: lottie_validity -> generation_success_rate to match what upstream's TaskMeta actually reports (Harbor previously relied on the first-numeric-score fallback in evaluate.py). - Update README parity narrative, reproduction steps, and summary table (19 exact / 5 close / 11 diff / 4 N/A across 39 benchmarks). - adapter_metadata.json: update parity notes to reflect the agent-on-both-sides setup. Made-with: Cursor * Fix gdb adapter parity: portable ground truth + post-hoc NIMA Resolves the remaining N/A and 0.0000 entries in the gdb parity report (layout-8 nima_score, typography-7/8 ocr_accuracy, category-2 bertscore_f1) so the Harbor-vs-upstream comparison is fully populated and reproducible on a clean host. Adapter (adapter.py): - _localize_gt_paths copies any host-absolute asset paths inside sample.ground_truth (e.g. input_image, mask, reference_asset, ground_truth_image) into <task>/workspace/gt/ and rewrites the paths to container-relative /workspace/gt/<name>. Without this, typography-7 (inpaint_reconstruction) and layout-8 (identity metrics) silently produced evaluated_samples=0 inside Docker. - Added an explicit "Image output requirements" block to the instruction prompt for layout-1/8 and typography-7/8 so the agent produces a real rasterized PNG instead of text/SVG-as-text. Filed upstream issue lica-world/GDB#4 to track a cleaner native fix (typed asset manifest / dataset_root resolution). Verifier (task-template/tests/evaluate.py): - _posthoc_nima_score computes NIMA directly on output.png via pyiqa when primary_metric is nima_score and bench.evaluate() did not return it (LayerAwareObjectInsertion and peers don't emit NIMA natively; upstream does it in a separate rescoring pass). Filed upstream issue lica-world/GDB#3 to track adding it to BaseBenchmark.evaluate(). - _disable_heavy_optional_metrics short-circuits PickScore / HPSv2 / ImageReward / Dreamsim bundles before evaluate() runs, so the verifier doesn't try to lazily download ~8 GB of secondary-metric weights per trial. Primary metrics (nima_score, ocr_accuracy, clip_score, ssim, lpips, color_delta_e) are unaffected. Environment (task-template/Dockerfile, task.toml): - Add libgl1/libglib2.0-0 (required for pyiqa via cv2 to import). - Add pyiqa/timm/lpips to the pip layer (used by post-hoc NIMA). - Raise task memory from 4096 MB to 8192 MB (the claude-code install + torch + lica-gdb metric extras OOM'd at 4 GB). Parity report (README.md, parity_experiment.json): - layout-8 nima_score: Harbor 5.1513 vs upstream 4.5225 (CLOSE). - typography-7 ocr_accuracy: Harbor 0.5970 vs upstream 0.6101 (CLOSE). - typography-8 ocr_accuracy: upstream filled in at 0.8843 (CLOSE). - category-2 bertscore_f1: upstream filled in at 0.8791 (YES). - Summary: 20 YES / 8 CLOSE / 11 DIFF / 0 N/A across 39 benchmarks. Made-with: Cursor * gdb adapter: align with harbor adapter init template review - parity_experiment.json: adapted_benchmark_size 39 -> 33786 to match adapter_metadata.json (39 is the number of benchmarks, not tasks) - gdb.yaml: flesh out to match the orchestrator/environment/agents shape used by other adapter configs so `harbor run -c adapters/gdb/gdb.yaml` is runnable without extra flags Made-with: Cursor * gdb adapter: update registry command examples to harbor/gdb-parity The dataset is registered as harbor/gdb-parity (parity subset, 78 tasks), matching the harbor/bfcl-parity convention. Update the 'Running with Datasets Registry' section to reference -d gdb-parity and note the registered dataset name. Made-with: Cursor * gdb adapter: move ground truth out of agent-visible /workspace/ Security fix from adapter review: image/video generation benchmarks (layout-1/8, typography-7/8, temporal-4/5/6) were placing ground truth answer files and localized reference assets under `environment/workspace/gt/`. The Dockerfile COPYs `workspace/` into `/workspace/`, making those files trivially readable by the agent at runtime — an agent could just `cp /workspace/gt/ground_truth.png /workspace/output.png` to get a perfect score. Ground truth assets now live in two isolated locations: - `tests/gt/` -> mounted at `/tests/gt/` ONLY at verification time. All absolute host paths in ground_truth.json are rewritten to `/tests/gt/<name>`, and `ground_truth_file` points there too. - `solution/ground_truth.<ext>` -> mounted at `/solution/` ONLY for the oracle agent. Oracle solve.sh copies from `/solution/...` to `/workspace/...`. `evaluate.py` needs no change since it reads paths from ground_truth.json (which now contains the rewritten `/tests/gt/...` paths). Made-with: Cursor * gdb adapter: infer benchmarks from --task-ids for fast filtering Previously, passing --task-ids without --benchmarks forced the adapter to enumerate every sample in all 39 benchmarks (33,786 candidates) before filtering down to the requested handful, making spot-regen painful. Now main.py parses each task ID (gdb-<benchmark>-s<index>) and passes the unique benchmark IDs through to Adapter, short-circuiting enumeration. Made-with: Cursor * gdb adapter: content-hash ground-truth asset filenames The previous scheme encoded only stem[:40] + size + suffix into the localized filename. Two different source files sharing a 40-char stem prefix, byte size, and extension collided to the same dest name: the first was copied and both ground-truth entries were silently remapped to the same container path, so the second entry pointed at the wrong bytes. Switch to a SHA-1 content digest (first 10 hex chars). Distinct-content files always get distinct names, and genuinely-identical files are naturally deduped. Made-with: Cursor * gdb adapter: ruff format Made-with: Cursor * gdb adapter: refresh README dependency list and GT layout description Made-with: Cursor * gdb adapter: review pass — pin lica-gdb, tighten verifier/overwrite edges Targeted review-feedback pass. No changes to oracle formatting, ground- truth serialization, or the LOWER_IS_BETTER transform, so existing parity_experiment.json numbers remain representative. Reproducibility - Pin lica-gdb to the uv.lock SHA in the verifier Dockerfile and pyproject so the container evaluates against the same benchmark code that produced parity. Other deps (torch, pyiqa, etc.) stay unpinned: pinning them here to versions we did not actually validate against parity would risk silently shifting NIMA/BERTScore/LPIPS scores. Correctness - adapter.generate_task: when --overwrite is used, rmtree the existing task directory before copytree-ing the template back in. The old path left stale environment/workspace/inputs/*, solution/ground_truth.*, and tests/gt/* around from prior generations. - adapter.generate_task: fail loud with RuntimeError for the four image benchmarks (layout-1, layout-8, typography-7, typography-8) when no ground-truth image is found. Previously we silently fell back to `touch`, which scored 0 and masked missing-dataset issues. - adapter._build_instruction: only list input files that were actually copied into environment/workspace/inputs/, and log a warning when a sample declares an input path that does not exist. Avoids dangling /workspace/inputs/... references in instruction.md. - evaluate.write_reward: normalise the reward through float() before the finite check. GDB returns numpy scalars for some metrics, and isinstance(np.float32(nan), float) is False, which previously let NaN/Inf slip straight through to reward.txt. - tests/test.sh: the EXIT trap now also emits a minimal reward.json ({"reward": 0.0, "error": "verifier_crashed"}) when evaluate.py crashes before writing one, so downstream analysis can distinguish a hard crash from a legitimate zero score. reward.txt behaviour is unchanged. Cleanup - main.py: move logging.basicConfig() into main() so importing the module (e.g. from a test) does not reconfigure the root logger. - main.py: broaden _TASK_ID_RE to tolerate multi-segment benchmark IDs (falls back to the full-enumeration path on no match, same as today). - adapter.py: drop the unused task_dir legacy alias property. - adapter.py: note on _list_benchmark_ids() that accessing BenchmarkRegistry._benchmarks directly is safe because lica-gdb is pinned to a specific SHA. Made-with: Cursor * gdb adapter: clarify nima_score scale exception and oracle scope Made-with: Cursor * gdb adapter: refresh parity after lottie/template/temporal rerun After re-running lottie-1/2, template-2/3/5, and temporal-1 against normalized GT data with the correct primary-metric selection, Harbor matches upstream on template-3/5 and temporal-1 (DIFF -> YES), lottie-1 (DIFF -> YES), and lottie-2 (DIFF -> CLOSE at 0.12, within n=2 noise). Two template samples failed with a claude-code harness ARG_MAX limit (template-2 s0, template-3 s1) when the instruction exceeds the Linux `bash -c` argument size; these are documented as a follow-up on the agent harness, and the surviving samples are reported with n=1. Final parity: 27 YES / 9 CLOSE / 3 DIFF across 39 benchmarks. Made-with: Cursor * claude-code agent: pipe instruction via stdin instead of argv The previous run() implementation embedded the full prompt as a `bash -c "... claude --print -- <instruction>"` argument, which fails with `exec /bin/bash: argument list too long` on Linux once the prompt exceeds ARG_MAX (~128 KB). The GDB benchmarks `template-2` and `template-3` hit this ceiling on samples that include large layout dumps, causing the trial to record a silent `no_output` / reward 0. Stage the instruction on disk (trial logs_dir), upload it to the container at /tmp/harbor-claude-instruction.txt, and invoke `claude --print < /tmp/harbor-claude-instruction.txt`. This matches the upstream GDB `claude_code` provider (which also pipes via stdin) and supports arbitrarily long instructions. Made-with: Cursor * gdb adapter: refresh parity after claude-code stdin fix Re-ran template-2-s0 and template-3-s1 (previously blocked by the claude-code ARG_MAX harness bug). With the agent now piping via stdin, both samples run to completion. template-3 is YES at n=2; template-2 s0 is a genuine agent failure (Claude returned a chat answer instead of writing /workspace/answer.txt) — upstream also fails both samples on this benchmark. Final parity: 27 YES / 9 CLOSE / 3 DIFF across 39 benchmarks, all with full 2-sample coverage. Made-with: Cursor * gdb adapter: note upstream run-to-run variance for remaining DIFFs Two independent upstream parity runs disagree with each other by 0.5 on layout-6, template-2, and typography-1 — Harbor's reported numbers fall inside upstream's own variance band, so these are n=2 agent stochasticity rather than systematic harness differences. Made-with: Cursor * gdb adapter: flatten to harbor adapter init layout, drop duplicate log Addresses the structural-drift and duplicate-log points from the Claude review on PR #1433. Layout - Move src/gdb_adapter/{adapter,main}.py and src/gdb_adapter/task-template/ up to the adapter root, matching the adapters/aider_polyglot/ (and `harbor adapter init`) convention: adapter.py + run_adapter.py + task- template/ directly under adapters/gdb/. Drop the now-empty src/ wrapper and __init__.py. - Rename main.py to run_adapter.py for consistency with the other adapters; switch `from .adapter import Adapter` to `from adapter import Adapter` since adapter.py is now a sibling module rather than a package member. - pyproject.toml: drop the hatchling wheel packaging (packages = ["src/gdb_adapter"]) and mark the project as non-packageable (tool.uv package = false) — the adapter is invoked as a script, not installed as a wheel. Regenerated uv.lock to match. Logging - run_adapter.main no longer re-emits the "Done. generated=..." summary; Adapter.run already logs that line at INFO. README - Update all `python -m gdb_adapter.main` invocations to `python run_adapter.py` so the docs match the on-disk layout. Made-with: Cursor * gdb adapter: drop changelog-style narrative from docs Rewrite the parity-differences notes in README.md, the parity_experiment.json "notes" field, and an evaluate.py docstring so they describe the adapter's current state (harness, metrics, dependencies) instead of the sequence of fixes/reruns that got us there. No behaviour changes; no parity numbers change. Made-with: Cursor * gdb adapter: rename task-template/ to template/ to match validator `scripts/validate_adapter.py` (`harbor adapter review`) requires the template directory to be named `template/` (singular); it flags both `task-template/` as missing and `templates/` (plural) as wrong. Rename accordingly and update adapter.py's template_dir resolution to match. All 28 structural validation checks now pass; `parity_costs: null` is the only remaining (accepted) warning. Made-with: Cursor * gdb adapter: fix README parity-results link and agent metadata format - README.md: `parity-experiments/tree/main/adapters/gdb` 404s (the adapter lives on PR #220, not on main). Point at the HF discussion URL that the other adapters and this adapter's own `parity_experiment.json` already use. - adapter_metadata.json: `parity_matching_agents` now follows the `agent@version+model` format documented in the adapter tutorial, so `[email protected]` becomes `[email protected]+claude-sonnet-4-20250514`. No behaviour or parity-number changes. Made-with: Cursor * gdb adapter: benchmark-level parity with 3-run mean +/- std Replace single-run parity with 3 independent runs per side (234 trials each, 468 total) and report benchmark-level mean +/- population std. Macro reward (37 of 39 benchmarks, excluding category-2 bertscore_f1 and layout-8 nima_score which only have upstream run-1 post-hoc rescoring): Harbor 0.6396 +/- 0.0390 vs upstream 0.5486 +/- 0.0209. Adapter changes: - template/task.toml + adapter.py: emit [task].name = "lica-world/gdb-<bench>-s<sample>" so generated tasks are registry-ready. - _aggregate_parity.py (new): ingest 3 harbor jobs + 3 upstream JSONs, macro with category-2/layout-8 excluded, emit parity_experiment.json + per_benchmark_parity.json, and a --export-summaries mode that writes result_{harbor,original}_run{1,2,3}.json for HF upload. - adapter_metadata.json: set parity_costs to $260 (3 runs x 78 tasks x 2 sides at claude-sonnet-4 prices). - README: rewrite "Comparison with Original Benchmark (Parity)" with the 3-run numbers, per-benchmark mean+/-std table, and updated notes (cairo infra gap for svg-6/7/8 makes upstream pin at reward 0.5 for runs 2 and 3). Made-with: Cursor * gdb adapter: full 39-benchmark parity and PR cleanup Rescore upstream category-2 (bertscore_f1) and layout-8 (nima_score) against all 3 runs so every GDB sub-benchmark participates in the macro: macro_reward (claude-code / claude-sonnet-4-20250514, 3 runs each side) original: 0.5553 +/- 0.0203 harbor: 0.6380 +/- 0.0420 match: 20 exact / 5 close / 14 different (39 total) Also drop the local parity aggregator helper (_aggregate_parity.py) from the adapter tree since no other adapter ships one and it hard-codes local paths; it stays out-of-tree for internal use. Made-with: Cursor * gdb adapter: normalize nima_score reward to [0, 1] evaluate.py passed nima_score through untouched, so reward.txt for layout-1 / layout-8 was written on pyiqa's native ~[1, 10] scale while every other metric landed in [0, 1]. Apply `value / 10.0` alongside the LOWER_IS_BETTER branch so Harbor's primary reward signal honors the adapter's documented [0, 1] contract. Parity tables / JSONs are unchanged: the aggregator reads raw nima_score from reward.json.scores and applies /10 itself, so the published 3-run macro (0.5553 +/- 0.0203 vs 0.6380 +/- 0.0420) and the 20/5/14 match split are still correct. Also fix the README's conversion note which previously read (value - 1) / 9 instead of /10. Made-with: Cursor * gdb adapter: close svg-6/7/8 cairo gap, update parity Re-ran upstream svg-6/7/8 three times with libcairo reachable so cairosvg can rasterize, then aggregated against the existing Harbor runs. Every GDB sub-benchmark now runs the same pipeline on both sides, and the two-way infra gap (cairo availability) that was pinning upstream svg-6/7/8 at mse=1.0 (reward 0.5) is closed: macro_reward (claude-code / claude-sonnet-4-20250514, 3 runs each side) original: 0.5923 +/- 0.0203 (was 0.5553 +/- 0.0203) harbor: 0.6380 +/- 0.0420 match: 23 exact / 5 close / 11 different (39 total) svg-6 upstream: 0.5000 +/- 0.0000 -> 0.9688 +/- 0.0067 (harbor 0.9705) svg-7 upstream: 0.5000 +/- 0.0000 -> 0.9879 +/- 0.0017 (harbor 0.9883) svg-8 upstream: 0.5000 +/- 0.0000 -> 0.9845 +/- 0.0035 (harbor 0.9854) Also refresh the notes in parity_experiment.json / README to describe the steady-state dependency set (bert_score, pyiqa, cairosvg + libcairo) rather than calling out a cairo-specific infra gap that no longer exists. Made-with: Cursor * Revert "claude-code agent: pipe instruction via stdin instead of argv" This reverts commit 7666367. * gdb adapter: drop stale note referencing reverted agent change The stdin-piping note in adapters/gdb/README.md pointed to src/harbor/agents/installed/claude_code.py, which is no longer part of this PR. Remove the bullet. Made-with: Cursor * gdb adapter: standardize to harbor adapters init format (PR #1289) Restructure the GDB adapter to match the standardized layout introduced in #1289. - Move adapter into src/gdb_adapter/ (_adapter suffix because the upstream lica-gdb dependency ships a top-level `gdb` package that would collide with `src/gdb/`, same pattern swebench uses). - Expose `uv run gdb` via [project.scripts] -> gdb_adapter.main:main and switch pyproject.toml to the package-based (setuptools) format. - Rename Adapter -> GDBAdapter and point template_dir at the new task-template/ directory. - Move run_adapter.py -> src/gdb_adapter/main.py with the standardized argparse layout (--output-dir, --limit, --overwrite, --task-ids), and keep the GDB-specific flags (--dataset-root, --benchmarks, --n, --split) as additions. - Upgrade task-template/task.toml to schema_version 1.0: add [task].name = "lica-world/gdb__{task_id}", [task].authors from the GDB paper BibTeX, [task].keywords (dropping the old metadata.tags), and lowercase the template placeholders. - Default gdb.yaml to the `oracle` agent with the claude-code parity agent kept as a commented-out example. - Update README commands to `uv run gdb ...` and `harbor run ...`, and align the "Running Individual Trials" heading with swebench's. Made-with: Cursor * gdb adapter: align pyproject + yaml with standardized majority Minor consistency fixes after another review pass over the standardized adapters: - Switch pyproject build backend from setuptools to hatchling to match the dominant convention (aime, algotune, arc_agi_2, aider_polyglot, autocodebench, swesmith, mlgym-bench, ...). The only reason our first pass used setuptools was to carry the task-template package-data spec forward from swebench/replicationbench; hatchling includes everything under the package dir automatically, so that spec is no longer needed. `tool.hatch.metadata.allow-direct-references = true` is required to keep the SHA-pinned `lica-gdb` git dep. - Flip gdb.yaml defaults to `force_build: true` / `delete: true` so first-time users get a clean build/teardown, matching aime, algotune, arc_agi_2, swebench. Power users can still override for iterative runs. Made-with: Cursor * gdb adapter: report parity as mean +/- sample SEM with range-overlap match Aligns parity reporting with the adapters.mdx spec: values are mean +/- sample SEM across the 3 runs per side, and the Match column uses the range-overlap criterion (max(a) >= min(b) AND max(b) >= min(a)) instead of a delta-reward threshold. Macro ranges overlap; 35/39 per-benchmark ranges overlap. Made-with: Cursor * gdb adapter: add benchmark author emails, document package name deviation task-template/task.toml now credits each GDB paper author with the email published alongside their name in the arXiv v2 manuscript ({name}@lica.world). pyproject.toml has an inline note explaining why the wheel package is `gdb_adapter` rather than the folder-derived `gdb` (the `lica-gdb` dependency installs a top-level `gdb` module, same collision `adapters/bfcl` resolves with `bfcl_adapter`). Made-with: Cursor * gdb adapter: add gdb-oracle for in-process full-source oracle coverage gdb-oracle drives the oracle answer through every GDB sample (~33,786 across 39 benchmarks) without Docker. It loads each sample, synthesizes the oracle ModelOutput exactly like solve.sh would (text copy for text/SVG/JSON, GT image/video for image/video tasks), runs bench.parse_model_output + bench.evaluate in process, and normalizes rewards the same way the containerized verifier does. Complements the 78-task parity subset: that covers adapter template plumbing under full Docker; this covers per-sample data integrity + evaluator behavior across the full source. Smoke tested on text, SVG, layout, lottie, and pixel-MSE benchmarks: all return reward 1.0. Addresses the "oracle not run on full benchmark" review finding. Made-with: Cursor * gdb adapter: drop gdb-oracle section from README (keep entrypoint only) Made-with: Cursor * gdb adapter: align with convention, drop non-standard extras Other adapters ship a single [project.scripts] entry and no companion data files; gdb had a second `gdb-oracle` entrypoint, an in-process oracle driver, and a per-benchmark parity JSON that no peer adapter ships. Removing all three keeps the reviewer's eye on the adapter, not on the extras. README oracle verification is now two lines (matching pixiu/bfcl cadence); the 33,771/33,786 full-source pass number stays, the defensive residual-tail explanation does not. Parity_experiment.json and adapter_metadata.json notes lose references to the deleted JSON. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * gdb adapter: name the 15 oracle failures in README Reviewer asked what the 15 non-passing samples are. They're all `layout-1` with `.mp4` ground-truth paths; the full-source sweep's PIL image loader rejects video files, while the Docker pipeline reads them via `solve.sh` file-copy. One factual sentence, no defensive padding. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * gdb adapter: rename gdb.yaml to run_gdb.yaml per adapters.mdx spec The adapter spec canonicalises the config filename as `run_<adapter-name>.yaml`. Renaming to match and updating the README reference + in-file usage comment. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * gdb adapter: address Ternura143 review comments - Overview expanded to the template's bulleted form (task types, dataset size, provenance, original harness, adaptation notes). The standalone "What is GDB?" stays as a short pointer to the paper/scoring spec. - Citation now precedes Authors & Contributions (template order). - Authors & Contributions adopts the template phrasing, with the adapter author attributed to the Lica team (not Harbor team). - adapter_metadata.json: parity_unmatching_agents [] -> null, matching the majority of peer adapters. - Acknowledgement section intentionally omitted: parity compute was self-funded via our own ANTHROPIC_API_KEY, not the 2077AI- sponsored proxy from adapters/parity_api_instructions.md. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * gdb adapter: README — attribute author to Harbor team, linter pass Adopts the peer-adapter convention ("from the Harbor team") used across bfcl, pixiu, bird_bench, etc. Also picks up linter-normalised blank lines after the bullet-list intros and aligned-column Markdown for the Supported Benchmarks table. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * gdb adapter: switch lica-gdb pin from git-SHA to PyPI version lica-gdb is now published on PyPI (0.1.1); the commit our SHA pin targets (4450b310) is the same commit tagged as 0.1.1. Using the version pin means uv sync and docker build can use the PyPI wheel cache instead of cloning the repo, and drops the allow-direct-references escape hatch plus the git/curl apt packages that were only there to satisfy the git install. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * gdb adapter: evaluate.py — address two Devin review findings 1. _disable_heavy_optional_metrics now calls setattr(bench, ...) instead of setattr(bench.__class__, ...). The class-level set works today in lica-gdb 0.1.1 because no instance shadows these attributes, but would silently become a no-op if upstream ever initialised them in __init__. Instance-level set is robust. 2. write_reward now normalises all_scores entries through float() before the NaN/Inf check. Previously isinstance(v, float) was False for numpy scalars, so numpy nan slipped past the check into json.dumps, which succeeded (allow_nan=True) but produced invalid JSON; on some paths it landed as the string "nan" via the str(v) fallback. All non-finite numerics — native or numpy — now become null. Native bool/int/float are preserved verbatim; numpy/Decimal are coerced to plain float so json can serialise them. Verified: native + numpy nan/inf → null, plain floats preserved, numpy.float64/int64 coerced to Python numeric, bool preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * gdb adapter: address Slimshilin review comments - add run_gdb_parity.yaml with the 78 task names and parity selection logic (deterministic first-N per benchmark from upstream load_data()) - drop the TODO paragraph from _posthoc_nima_score; the preceding paragraph already explains the rationale Made-with: Cursor --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> Co-authored-by: Slimshilin <[email protected]>

…k#1433) * [WIP] Adapter: GDB (GraphicDesignBench) Adapt GDB (GraphicDesignBench) for Harbor. GDB evaluates VLMs/LLMs on professional graphic design tasks across 7 domains: SVG, category, layout, typography, template, temporal, and Lottie. - All 39 benchmarks adapted (33,786 total benchmark samples) - Uses GDB's own Python code for prompt construction and evaluation (exact fidelity by construction) - All GDB metrics reported in reward.json; primary metric to reward.txt - Dependencies (gdb[metrics,svg-metrics]) installed at eval time in test.sh - Parity type: Scenario 2 (original is LLM-based, no agent harness) - 1/(1+error) for error metrics to preserve granularity Made-with: Cursor * Address Devin review: fix loguru import and inf sanitization - Replace loguru with stdlib logging in run_adapter.py to match all other adapters and avoid ModuleNotFoundError (loguru is not a project dependency). Fixes dual-logging mismatch with adapter.py. - Add math.isinf() check alongside math.isnan() in has_real_score and the fallback loop, preventing float('inf') from propagating as a meaningful reward value. Made-with: Cursor * Add logging.basicConfig() so progress output is visible Without basicConfig(), Python's default root logger has no handlers and silently drops INFO-level messages. The user saw no progress during task generation. Matches the pattern used by ~46 other adapters. Made-with: Cursor * Remove narrating comments, redundant docstrings, and defensive check Drop docstrings that restate the method name, decorative section dividers, narrating shell comments, and the template_dir existence check that no other adapter uses. Made-with: Cursor * Remove changelog-style note from README Replace historical note about upstream svg-6 fix with plain statement. Made-with: Cursor * Sanitize reward value for NaN/Inf in write_reward Clamp NaN/Inf to 0.0 before writing reward.txt and reward.json, matching the sanitization already applied to all_scores entries. Made-with: Cursor * Fix review issues: trap cleanup, bake deps into Docker image, harden cache - Replace unreachable fallback in test.sh with trap EXIT pattern so reward.txt is always written even when evaluate.py crashes - Move pip install from test.sh into Dockerfile (avoids repeated downloads; install CPU-only PyTorch, libcairo2-dev, bert-score) - Make _load_sample fall back to _load_samples_for_benchmark when cache is unpopulated instead of silently returning empty list - Use %-style logging in run_adapter.py (consistent with adapter.py) - Update README to reflect deps baked into Docker image Made-with: Cursor * Split optional pip install into separate RUN to avoid masking failures The || true was applying to the entire &&-chain, silently masking failures from torch and gdb installs. Separate the optional Levenshtein/bert-score install into its own RUN layer. Made-with: Cursor * Update sample cache when a larger n is loaded The old condition only stored results when the benchmark was entirely uncached. When a partial cache existed (e.g. n=2) and a larger n was requested, the newly loaded samples were discarded, leaving stale data. Made-with: Cursor * Slice return value to n on non-cached path in _load_samples_for_benchmark If bench.load_data(n=n) returns more than n samples, the non-cached path now correctly slices the result, matching the cached path behavior. Made-with: Cursor * Fill in adapter, dataset, and parity PR links Made-with: Cursor * Track full vs partial cache loads to fix n=None returning stale data Added _fully_loaded set to distinguish complete loads from partial ones. n=None now only returns from cache if the benchmark was previously loaded without a limit; otherwise it reloads all samples as documented. Made-with: Cursor * Add tesseract-ocr to Dockerfile for typography OCR evaluation GDB's typography benchmarks use pytesseract (which calls the tesseract binary) for OCR metrics. Without the system package, _run_ocr silently returns None and all OCR scores become NaN. Made-with: Cursor * Update oracle results: 73/78 after tesseract-ocr and cairo fixes svg-6/7/8 and typography-7/8 now pass with the Dockerfile dependency fixes, reducing expected failures from 13 to 5. Made-with: Cursor * Download remote GT assets at task generation time Some layout-1 samples store ground truth as GCS URLs rather than local files. _find_gt_image now downloads these to temp files so they are packaged into the task directory. Also updated upstream GDB to prefer local copies and index .mp4 files. Oracle: 78/78 tasks pass. Made-with: Cursor * Fix PRIMARY_METRICS to match GDB's actual primary metrics svg-6/7/8 should use mse (lower-is-better), not svg_validity (higher-is-better). typography-2 should use rgb_l2_distance (lower-is-better), not delta_e_below_5 (higher-is-better). layout-1 should use nima_score, not clip_score. The direction mismatch caused incorrect 1/(1+x) reward transformation for svg-6/7/8 and typography-2, producing wrong rewards for imperfect agent outputs. Made-with: Cursor * Remove _download_gt_url — GT assets should be local in the dataset Made-with: Cursor * fix: rename deprecated *_trials fields to *_runs in parity_experiment.json Addresses Devin review comment — the validation script (scripts/validate_adapter.py) expects number_of_runs, original_runs, and harbor_runs instead of the legacy *_trials names. Made-with: Cursor * fix: don't count skipped tasks as generated in run_adapter.py Pre-existing task directories that are skipped (without --overwrite) were incorrectly counted in total_generated. Add a separate total_skipped counter so the summary log accurately reflects work done. Made-with: Cursor * Update parity data with real agent-run results Replace construction-time parity estimates with actual claude-code agent runs. Original: GDB run_benchmarks.py (non-agentic). Harbor: claude-code in Docker (agentic). Both use claude-sonnet-4-20250514. All values converted to reward space for fair comparison. Made-with: Cursor * Update parity results with retry runs (37/39 full coverage) Merge results from 3 harbor runs (initial + 2 retries). All 39 benchmarks now have at least 1 sample; 37 have full 2-sample coverage (layout-2, layout-3 have 1 each due to infra timeouts). 15 exact, 6 close, 10 different, 8 N/A. Made-with: Cursor * Fix PRIMARY_METRICS for 8 benchmarks to match actual GDB evaluate output layout-8: clip_identity -> nima_score lottie-1/2: lottie_validity -> structural_similarity template-3: ari -> nmi temporal-1: percent_perfect -> pairwise_accuracy temporal-4/5/6: generation_success_rate -> lottie_validity Made-with: Cursor * Use logger.debug for per-task output in adapter.py Per-benchmark progress and final summary in run_adapter.py already provide sufficient user-facing feedback at INFO level. Made-with: Cursor * Update parity results with full 2-sample coverage for layout-2 and layout-3 Both benchmarks now have complete 2-sample runs. layout-2 remains 0.0000, layout-3 updated to 0.4899 (avg of 0.5178 and 0.4619). All 39 benchmarks now have full coverage. Made-with: Cursor * Fix parity data consistency: number_of_runs and harbor_runs array lengths - Set number_of_runs to 2 (matching 2 samples per benchmark) - Trim template-2 and template-3 harbor_runs from 3 to 2 entries (3rd entry was from a retry, inconsistent with stated 78 = 2×39) - template-2 now exact match (0.5 vs 0.5); summary updated to 16 exact, 6 close, 9 different, 8 N/A Made-with: Cursor * Update distribution name from gdb to lica-gdb The upstream package was renamed from gdb to lica-gdb in pyproject.toml. Python import name (from gdb.xxx) is unchanged. Made-with: Cursor * Address Claude adapter review feedback parity_experiment.json: - Aggregate harbor_runs to per-run means (single entry each) so that number_of_runs=1 matches array lengths consistently - Pin agent version: claude-code@latest -> [email protected] - Expand notes to explain run structure and nima_score scale adapter_metadata.json: - adapted_benchmark_size: 39 -> 33786 (total task count across 39 benchmarks) - parity_sampling_rate: 0.5 -> 0.0023 (78/33786) - registry_benchmark_size: null -> 78 (parity subset registered) - Remove Ryan Marten; update contact to [email protected] - Pin parity_matching_agents version: [email protected] README.md: - "Number of Trials" -> "Number of Runs" - Clarify nima_score uses native 1-10 scale, not 0-1 - Add Reproduction section with concrete original-side and Harbor-side commands - Justify oracle scope (parity subset vs full 33,786 tasks) - Sync rounded values to JSON (layout-1, layout-5, svg-5, svg-8) - Remove Ryan Marten from Authors section Made-with: Cursor * Refactor gdb adapter to harbor adapter init scaffold Align layout with the standardized package structure produced by `harbor adapter init` (see src/harbor/cli/template-adapter/): adapters/gdb/ ├── pyproject.toml # uv package ├── gdb.yaml # renamed from run_gdb.yaml └── src/gdb_adapter/ ├── __init__.py ├── adapter.py # class renamed: GDBAdapter -> Adapter ├── main.py # standardized CLI entry point └── task-template/ # renamed from template/ Python package is named `gdb_adapter` (not `gdb`) to avoid shadowing the `gdb` module provided by the upstream `lica-gdb` distribution. Adapter constructor now accepts the standardized flags (output_dir, limit, overwrite, task_ids) in addition to GDB-specific ones (dataset_root, benchmarks, split, n_per_benchmark), and exposes a run() method that encapsulates the task-generation loop previously living in run_adapter.py. CLI usage: uv run python -m gdb_adapter.main --dataset-root <path> \ --output-dir datasets/gdb [--split parity|--benchmarks ...|--limit N] README restructured to match the template (adds "Run Evaluation / Harness" and renames "Parity" to "Comparison with Original Benchmark (Parity)") and points users at the new CLI. Made-with: Cursor * Fix lottie oracle output when ground truth is a parsed dict/list In `_format_oracle_text` for `lottie-1`/`lottie-2`, if GDB stores `lottie_json` as an already-parsed dict/list (rather than a serialized string), returning it as-is caused `_build_oracle`'s f-string to call `str()` on it, producing Python repr (`{'v': '5.5.2'}`) instead of valid JSON (`{"v": "5.5.2"}`). The oracle then wrote syntactically invalid JSON to `answer.json`, silently failing evaluation. Now json.dumps non-string values before returning. Flagged by Devin Review on PR harbor-framework#1433. Made-with: Cursor * Switch gdb parity to Scenario 2 (agentic on both sides) Both the upstream GDB harness and the Harbor task now run the same agent (claude-code with claude-sonnet-4-20250514). The upstream side uses the new `claude_code` provider from the harbor-adapter branch of https://github.com/lica-world/GDB, which drives the Claude Code CLI against a temporary workspace and scores with GDB's own evaluator. - Refresh parity_experiment.json "original_runs" with values from the new upstream parity run (parity_claude_code.json), applying the same reward-space transform the Harbor verifier uses (LOWER_IS_BETTER -> 1/(1+x)). - Drop stale Scenario 1 originals for the 4 metrics upstream doesn't emit without optional deps (bertscore_f1, nima_score, ocr_accuracy); record them as N/A with an explicit note. - PRIMARY_METRICS for temporal-4/5/6: lottie_validity -> generation_success_rate to match what upstream's TaskMeta actually reports (Harbor previously relied on the first-numeric-score fallback in evaluate.py). - Update README parity narrative, reproduction steps, and summary table (19 exact / 5 close / 11 diff / 4 N/A across 39 benchmarks). - adapter_metadata.json: update parity notes to reflect the agent-on-both-sides setup. Made-with: Cursor * Fix gdb adapter parity: portable ground truth + post-hoc NIMA Resolves the remaining N/A and 0.0000 entries in the gdb parity report (layout-8 nima_score, typography-7/8 ocr_accuracy, category-2 bertscore_f1) so the Harbor-vs-upstream comparison is fully populated and reproducible on a clean host. Adapter (adapter.py): - _localize_gt_paths copies any host-absolute asset paths inside sample.ground_truth (e.g. input_image, mask, reference_asset, ground_truth_image) into <task>/workspace/gt/ and rewrites the paths to container-relative /workspace/gt/<name>. Without this, typography-7 (inpaint_reconstruction) and layout-8 (identity metrics) silently produced evaluated_samples=0 inside Docker. - Added an explicit "Image output requirements" block to the instruction prompt for layout-1/8 and typography-7/8 so the agent produces a real rasterized PNG instead of text/SVG-as-text. Filed upstream issue lica-world/GDB#4 to track a cleaner native fix (typed asset manifest / dataset_root resolution). Verifier (task-template/tests/evaluate.py): - _posthoc_nima_score computes NIMA directly on output.png via pyiqa when primary_metric is nima_score and bench.evaluate() did not return it (LayerAwareObjectInsertion and peers don't emit NIMA natively; upstream does it in a separate rescoring pass). Filed upstream issue lica-world/GDB#3 to track adding it to BaseBenchmark.evaluate(). - _disable_heavy_optional_metrics short-circuits PickScore / HPSv2 / ImageReward / Dreamsim bundles before evaluate() runs, so the verifier doesn't try to lazily download ~8 GB of secondary-metric weights per trial. Primary metrics (nima_score, ocr_accuracy, clip_score, ssim, lpips, color_delta_e) are unaffected. Environment (task-template/Dockerfile, task.toml): - Add libgl1/libglib2.0-0 (required for pyiqa via cv2 to import). - Add pyiqa/timm/lpips to the pip layer (used by post-hoc NIMA). - Raise task memory from 4096 MB to 8192 MB (the claude-code install + torch + lica-gdb metric extras OOM'd at 4 GB). Parity report (README.md, parity_experiment.json): - layout-8 nima_score: Harbor 5.1513 vs upstream 4.5225 (CLOSE). - typography-7 ocr_accuracy: Harbor 0.5970 vs upstream 0.6101 (CLOSE). - typography-8 ocr_accuracy: upstream filled in at 0.8843 (CLOSE). - category-2 bertscore_f1: upstream filled in at 0.8791 (YES). - Summary: 20 YES / 8 CLOSE / 11 DIFF / 0 N/A across 39 benchmarks. Made-with: Cursor * gdb adapter: align with harbor adapter init template review - parity_experiment.json: adapted_benchmark_size 39 -> 33786 to match adapter_metadata.json (39 is the number of benchmarks, not tasks) - gdb.yaml: flesh out to match the orchestrator/environment/agents shape used by other adapter configs so `harbor run -c adapters/gdb/gdb.yaml` is runnable without extra flags Made-with: Cursor * gdb adapter: update registry command examples to harbor/gdb-parity The dataset is registered as harbor/gdb-parity (parity subset, 78 tasks), matching the harbor/bfcl-parity convention. Update the 'Running with Datasets Registry' section to reference -d gdb-parity and note the registered dataset name. Made-with: Cursor * gdb adapter: move ground truth out of agent-visible /workspace/ Security fix from adapter review: image/video generation benchmarks (layout-1/8, typography-7/8, temporal-4/5/6) were placing ground truth answer files and localized reference assets under `environment/workspace/gt/`. The Dockerfile COPYs `workspace/` into `/workspace/`, making those files trivially readable by the agent at runtime — an agent could just `cp /workspace/gt/ground_truth.png /workspace/output.png` to get a perfect score. Ground truth assets now live in two isolated locations: - `tests/gt/` -> mounted at `/tests/gt/` ONLY at verification time. All absolute host paths in ground_truth.json are rewritten to `/tests/gt/<name>`, and `ground_truth_file` points there too. - `solution/ground_truth.<ext>` -> mounted at `/solution/` ONLY for the oracle agent. Oracle solve.sh copies from `/solution/...` to `/workspace/...`. `evaluate.py` needs no change since it reads paths from ground_truth.json (which now contains the rewritten `/tests/gt/...` paths). Made-with: Cursor * gdb adapter: infer benchmarks from --task-ids for fast filtering Previously, passing --task-ids without --benchmarks forced the adapter to enumerate every sample in all 39 benchmarks (33,786 candidates) before filtering down to the requested handful, making spot-regen painful. Now main.py parses each task ID (gdb-<benchmark>-s<index>) and passes the unique benchmark IDs through to Adapter, short-circuiting enumeration. Made-with: Cursor * gdb adapter: content-hash ground-truth asset filenames The previous scheme encoded only stem[:40] + size + suffix into the localized filename. Two different source files sharing a 40-char stem prefix, byte size, and extension collided to the same dest name: the first was copied and both ground-truth entries were silently remapped to the same container path, so the second entry pointed at the wrong bytes. Switch to a SHA-1 content digest (first 10 hex chars). Distinct-content files always get distinct names, and genuinely-identical files are naturally deduped. Made-with: Cursor * gdb adapter: ruff format Made-with: Cursor * gdb adapter: refresh README dependency list and GT layout description Made-with: Cursor * gdb adapter: review pass — pin lica-gdb, tighten verifier/overwrite edges Targeted review-feedback pass. No changes to oracle formatting, ground- truth serialization, or the LOWER_IS_BETTER transform, so existing parity_experiment.json numbers remain representative. Reproducibility - Pin lica-gdb to the uv.lock SHA in the verifier Dockerfile and pyproject so the container evaluates against the same benchmark code that produced parity. Other deps (torch, pyiqa, etc.) stay unpinned: pinning them here to versions we did not actually validate against parity would risk silently shifting NIMA/BERTScore/LPIPS scores. Correctness - adapter.generate_task: when --overwrite is used, rmtree the existing task directory before copytree-ing the template back in. The old path left stale environment/workspace/inputs/*, solution/ground_truth.*, and tests/gt/* around from prior generations. - adapter.generate_task: fail loud with RuntimeError for the four image benchmarks (layout-1, layout-8, typography-7, typography-8) when no ground-truth image is found. Previously we silently fell back to `touch`, which scored 0 and masked missing-dataset issues. - adapter._build_instruction: only list input files that were actually copied into environment/workspace/inputs/, and log a warning when a sample declares an input path that does not exist. Avoids dangling /workspace/inputs/... references in instruction.md. - evaluate.write_reward: normalise the reward through float() before the finite check. GDB returns numpy scalars for some metrics, and isinstance(np.float32(nan), float) is False, which previously let NaN/Inf slip straight through to reward.txt. - tests/test.sh: the EXIT trap now also emits a minimal reward.json ({"reward": 0.0, "error": "verifier_crashed"}) when evaluate.py crashes before writing one, so downstream analysis can distinguish a hard crash from a legitimate zero score. reward.txt behaviour is unchanged. Cleanup - main.py: move logging.basicConfig() into main() so importing the module (e.g. from a test) does not reconfigure the root logger. - main.py: broaden _TASK_ID_RE to tolerate multi-segment benchmark IDs (falls back to the full-enumeration path on no match, same as today). - adapter.py: drop the unused task_dir legacy alias property. - adapter.py: note on _list_benchmark_ids() that accessing BenchmarkRegistry._benchmarks directly is safe because lica-gdb is pinned to a specific SHA. Made-with: Cursor * gdb adapter: clarify nima_score scale exception and oracle scope Made-with: Cursor * gdb adapter: refresh parity after lottie/template/temporal rerun After re-running lottie-1/2, template-2/3/5, and temporal-1 against normalized GT data with the correct primary-metric selection, Harbor matches upstream on template-3/5 and temporal-1 (DIFF -> YES), lottie-1 (DIFF -> YES), and lottie-2 (DIFF -> CLOSE at 0.12, within n=2 noise). Two template samples failed with a claude-code harness ARG_MAX limit (template-2 s0, template-3 s1) when the instruction exceeds the Linux `bash -c` argument size; these are documented as a follow-up on the agent harness, and the surviving samples are reported with n=1. Final parity: 27 YES / 9 CLOSE / 3 DIFF across 39 benchmarks. Made-with: Cursor * claude-code agent: pipe instruction via stdin instead of argv The previous run() implementation embedded the full prompt as a `bash -c "... claude --print -- <instruction>"` argument, which fails with `exec /bin/bash: argument list too long` on Linux once the prompt exceeds ARG_MAX (~128 KB). The GDB benchmarks `template-2` and `template-3` hit this ceiling on samples that include large layout dumps, causing the trial to record a silent `no_output` / reward 0. Stage the instruction on disk (trial logs_dir), upload it to the container at /tmp/harbor-claude-instruction.txt, and invoke `claude --print < /tmp/harbor-claude-instruction.txt`. This matches the upstream GDB `claude_code` provider (which also pipes via stdin) and supports arbitrarily long instructions. Made-with: Cursor * gdb adapter: refresh parity after claude-code stdin fix Re-ran template-2-s0 and template-3-s1 (previously blocked by the claude-code ARG_MAX harness bug). With the agent now piping via stdin, both samples run to completion. template-3 is YES at n=2; template-2 s0 is a genuine agent failure (Claude returned a chat answer instead of writing /workspace/answer.txt) — upstream also fails both samples on this benchmark. Final parity: 27 YES / 9 CLOSE / 3 DIFF across 39 benchmarks, all with full 2-sample coverage. Made-with: Cursor * gdb adapter: note upstream run-to-run variance for remaining DIFFs Two independent upstream parity runs disagree with each other by 0.5 on layout-6, template-2, and typography-1 — Harbor's reported numbers fall inside upstream's own variance band, so these are n=2 agent stochasticity rather than systematic harness differences. Made-with: Cursor * gdb adapter: flatten to harbor adapter init layout, drop duplicate log Addresses the structural-drift and duplicate-log points from the Claude review on PR harbor-framework#1433. Layout - Move src/gdb_adapter/{adapter,main}.py and src/gdb_adapter/task-template/ up to the adapter root, matching the adapters/aider_polyglot/ (and `harbor adapter init`) convention: adapter.py + run_adapter.py + task- template/ directly under adapters/gdb/. Drop the now-empty src/ wrapper and __init__.py. - Rename main.py to run_adapter.py for consistency with the other adapters; switch `from .adapter import Adapter` to `from adapter import Adapter` since adapter.py is now a sibling module rather than a package member. - pyproject.toml: drop the hatchling wheel packaging (packages = ["src/gdb_adapter"]) and mark the project as non-packageable (tool.uv package = false) — the adapter is invoked as a script, not installed as a wheel. Regenerated uv.lock to match. Logging - run_adapter.main no longer re-emits the "Done. generated=..." summary; Adapter.run already logs that line at INFO. README - Update all `python -m gdb_adapter.main` invocations to `python run_adapter.py` so the docs match the on-disk layout. Made-with: Cursor * gdb adapter: drop changelog-style narrative from docs Rewrite the parity-differences notes in README.md, the parity_experiment.json "notes" field, and an evaluate.py docstring so they describe the adapter's current state (harness, metrics, dependencies) instead of the sequence of fixes/reruns that got us there. No behaviour changes; no parity numbers change. Made-with: Cursor * gdb adapter: rename task-template/ to template/ to match validator `scripts/validate_adapter.py` (`harbor adapter review`) requires the template directory to be named `template/` (singular); it flags both `task-template/` as missing and `templates/` (plural) as wrong. Rename accordingly and update adapter.py's template_dir resolution to match. All 28 structural validation checks now pass; `parity_costs: null` is the only remaining (accepted) warning. Made-with: Cursor * gdb adapter: fix README parity-results link and agent metadata format - README.md: `parity-experiments/tree/main/adapters/gdb` 404s (the adapter lives on PR harbor-framework#220, not on main). Point at the HF discussion URL that the other adapters and this adapter's own `parity_experiment.json` already use. - adapter_metadata.json: `parity_matching_agents` now follows the `agent@version+model` format documented in the adapter tutorial, so `[email protected]` becomes `[email protected]+claude-sonnet-4-20250514`. No behaviour or parity-number changes. Made-with: Cursor * gdb adapter: benchmark-level parity with 3-run mean +/- std Replace single-run parity with 3 independent runs per side (234 trials each, 468 total) and report benchmark-level mean +/- population std. Macro reward (37 of 39 benchmarks, excluding category-2 bertscore_f1 and layout-8 nima_score which only have upstream run-1 post-hoc rescoring): Harbor 0.6396 +/- 0.0390 vs upstream 0.5486 +/- 0.0209. Adapter changes: - template/task.toml + adapter.py: emit [task].name = "lica-world/gdb-<bench>-s<sample>" so generated tasks are registry-ready. - _aggregate_parity.py (new): ingest 3 harbor jobs + 3 upstream JSONs, macro with category-2/layout-8 excluded, emit parity_experiment.json + per_benchmark_parity.json, and a --export-summaries mode that writes result_{harbor,original}_run{1,2,3}.json for HF upload. - adapter_metadata.json: set parity_costs to $260 (3 runs x 78 tasks x 2 sides at claude-sonnet-4 prices). - README: rewrite "Comparison with Original Benchmark (Parity)" with the 3-run numbers, per-benchmark mean+/-std table, and updated notes (cairo infra gap for svg-6/7/8 makes upstream pin at reward 0.5 for runs 2 and 3). Made-with: Cursor * gdb adapter: full 39-benchmark parity and PR cleanup Rescore upstream category-2 (bertscore_f1) and layout-8 (nima_score) against all 3 runs so every GDB sub-benchmark participates in the macro: macro_reward (claude-code / claude-sonnet-4-20250514, 3 runs each side) original: 0.5553 +/- 0.0203 harbor: 0.6380 +/- 0.0420 match: 20 exact / 5 close / 14 different (39 total) Also drop the local parity aggregator helper (_aggregate_parity.py) from the adapter tree since no other adapter ships one and it hard-codes local paths; it stays out-of-tree for internal use. Made-with: Cursor * gdb adapter: normalize nima_score reward to [0, 1] evaluate.py passed nima_score through untouched, so reward.txt for layout-1 / layout-8 was written on pyiqa's native ~[1, 10] scale while every other metric landed in [0, 1]. Apply `value / 10.0` alongside the LOWER_IS_BETTER branch so Harbor's primary reward signal honors the adapter's documented [0, 1] contract. Parity tables / JSONs are unchanged: the aggregator reads raw nima_score from reward.json.scores and applies /10 itself, so the published 3-run macro (0.5553 +/- 0.0203 vs 0.6380 +/- 0.0420) and the 20/5/14 match split are still correct. Also fix the README's conversion note which previously read (value - 1) / 9 instead of /10. Made-with: Cursor * gdb adapter: close svg-6/7/8 cairo gap, update parity Re-ran upstream svg-6/7/8 three times with libcairo reachable so cairosvg can rasterize, then aggregated against the existing Harbor runs. Every GDB sub-benchmark now runs the same pipeline on both sides, and the two-way infra gap (cairo availability) that was pinning upstream svg-6/7/8 at mse=1.0 (reward 0.5) is closed: macro_reward (claude-code / claude-sonnet-4-20250514, 3 runs each side) original: 0.5923 +/- 0.0203 (was 0.5553 +/- 0.0203) harbor: 0.6380 +/- 0.0420 match: 23 exact / 5 close / 11 different (39 total) svg-6 upstream: 0.5000 +/- 0.0000 -> 0.9688 +/- 0.0067 (harbor 0.9705) svg-7 upstream: 0.5000 +/- 0.0000 -> 0.9879 +/- 0.0017 (harbor 0.9883) svg-8 upstream: 0.5000 +/- 0.0000 -> 0.9845 +/- 0.0035 (harbor 0.9854) Also refresh the notes in parity_experiment.json / README to describe the steady-state dependency set (bert_score, pyiqa, cairosvg + libcairo) rather than calling out a cairo-specific infra gap that no longer exists. Made-with: Cursor * Revert "claude-code agent: pipe instruction via stdin instead of argv" This reverts commit 7666367. * gdb adapter: drop stale note referencing reverted agent change The stdin-piping note in adapters/gdb/README.md pointed to src/harbor/agents/installed/claude_code.py, which is no longer part of this PR. Remove the bullet. Made-with: Cursor * gdb adapter: standardize to harbor adapters init format (PR harbor-framework#1289) Restructure the GDB adapter to match the standardized layout introduced in harbor-framework#1289. - Move adapter into src/gdb_adapter/ (_adapter suffix because the upstream lica-gdb dependency ships a top-level `gdb` package that would collide with `src/gdb/`, same pattern swebench uses). - Expose `uv run gdb` via [project.scripts] -> gdb_adapter.main:main and switch pyproject.toml to the package-based (setuptools) format. - Rename Adapter -> GDBAdapter and point template_dir at the new task-template/ directory. - Move run_adapter.py -> src/gdb_adapter/main.py with the standardized argparse layout (--output-dir, --limit, --overwrite, --task-ids), and keep the GDB-specific flags (--dataset-root, --benchmarks, --n, --split) as additions. - Upgrade task-template/task.toml to schema_version 1.0: add [task].name = "lica-world/gdb__{task_id}", [task].authors from the GDB paper BibTeX, [task].keywords (dropping the old metadata.tags), and lowercase the template placeholders. - Default gdb.yaml to the `oracle` agent with the claude-code parity agent kept as a commented-out example. - Update README commands to `uv run gdb ...` and `harbor run ...`, and align the "Running Individual Trials" heading with swebench's. Made-with: Cursor * gdb adapter: align pyproject + yaml with standardized majority Minor consistency fixes after another review pass over the standardized adapters: - Switch pyproject build backend from setuptools to hatchling to match the dominant convention (aime, algotune, arc_agi_2, aider_polyglot, autocodebench, swesmith, mlgym-bench, ...). The only reason our first pass used setuptools was to carry the task-template package-data spec forward from swebench/replicationbench; hatchling includes everything under the package dir automatically, so that spec is no longer needed. `tool.hatch.metadata.allow-direct-references = true` is required to keep the SHA-pinned `lica-gdb` git dep. - Flip gdb.yaml defaults to `force_build: true` / `delete: true` so first-time users get a clean build/teardown, matching aime, algotune, arc_agi_2, swebench. Power users can still override for iterative runs. Made-with: Cursor * gdb adapter: report parity as mean +/- sample SEM with range-overlap match Aligns parity reporting with the adapters.mdx spec: values are mean +/- sample SEM across the 3 runs per side, and the Match column uses the range-overlap criterion (max(a) >= min(b) AND max(b) >= min(a)) instead of a delta-reward threshold. Macro ranges overlap; 35/39 per-benchmark ranges overlap. Made-with: Cursor * gdb adapter: add benchmark author emails, document package name deviation task-template/task.toml now credits each GDB paper author with the email published alongside their name in the arXiv v2 manuscript ({name}@lica.world). pyproject.toml has an inline note explaining why the wheel package is `gdb_adapter` rather than the folder-derived `gdb` (the `lica-gdb` dependency installs a top-level `gdb` module, same collision `adapters/bfcl` resolves with `bfcl_adapter`). Made-with: Cursor * gdb adapter: add gdb-oracle for in-process full-source oracle coverage gdb-oracle drives the oracle answer through every GDB sample (~33,786 across 39 benchmarks) without Docker. It loads each sample, synthesizes the oracle ModelOutput exactly like solve.sh would (text copy for text/SVG/JSON, GT image/video for image/video tasks), runs bench.parse_model_output + bench.evaluate in process, and normalizes rewards the same way the containerized verifier does. Complements the 78-task parity subset: that covers adapter template plumbing under full Docker; this covers per-sample data integrity + evaluator behavior across the full source. Smoke tested on text, SVG, layout, lottie, and pixel-MSE benchmarks: all return reward 1.0. Addresses the "oracle not run on full benchmark" review finding. Made-with: Cursor * gdb adapter: drop gdb-oracle section from README (keep entrypoint only) Made-with: Cursor * gdb adapter: align with convention, drop non-standard extras Other adapters ship a single [project.scripts] entry and no companion data files; gdb had a second `gdb-oracle` entrypoint, an in-process oracle driver, and a per-benchmark parity JSON that no peer adapter ships. Removing all three keeps the reviewer's eye on the adapter, not on the extras. README oracle verification is now two lines (matching pixiu/bfcl cadence); the 33,771/33,786 full-source pass number stays, the defensive residual-tail explanation does not. Parity_experiment.json and adapter_metadata.json notes lose references to the deleted JSON. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * gdb adapter: name the 15 oracle failures in README Reviewer asked what the 15 non-passing samples are. They're all `layout-1` with `.mp4` ground-truth paths; the full-source sweep's PIL image loader rejects video files, while the Docker pipeline reads them via `solve.sh` file-copy. One factual sentence, no defensive padding. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * gdb adapter: rename gdb.yaml to run_gdb.yaml per adapters.mdx spec The adapter spec canonicalises the config filename as `run_<adapter-name>.yaml`. Renaming to match and updating the README reference + in-file usage comment. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * gdb adapter: address Ternura143 review comments - Overview expanded to the template's bulleted form (task types, dataset size, provenance, original harness, adaptation notes). The standalone "What is GDB?" stays as a short pointer to the paper/scoring spec. - Citation now precedes Authors & Contributions (template order). - Authors & Contributions adopts the template phrasing, with the adapter author attributed to the Lica team (not Harbor team). - adapter_metadata.json: parity_unmatching_agents [] -> null, matching the majority of peer adapters. - Acknowledgement section intentionally omitted: parity compute was self-funded via our own ANTHROPIC_API_KEY, not the 2077AI- sponsored proxy from adapters/parity_api_instructions.md. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * gdb adapter: README — attribute author to Harbor team, linter pass Adopts the peer-adapter convention ("from the Harbor team") used across bfcl, pixiu, bird_bench, etc. Also picks up linter-normalised blank lines after the bullet-list intros and aligned-column Markdown for the Supported Benchmarks table. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * gdb adapter: switch lica-gdb pin from git-SHA to PyPI version lica-gdb is now published on PyPI (0.1.1); the commit our SHA pin targets (4450b310) is the same commit tagged as 0.1.1. Using the version pin means uv sync and docker build can use the PyPI wheel cache instead of cloning the repo, and drops the allow-direct-references escape hatch plus the git/curl apt packages that were only there to satisfy the git install. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * gdb adapter: evaluate.py — address two Devin review findings 1. _disable_heavy_optional_metrics now calls setattr(bench, ...) instead of setattr(bench.__class__, ...). The class-level set works today in lica-gdb 0.1.1 because no instance shadows these attributes, but would silently become a no-op if upstream ever initialised them in __init__. Instance-level set is robust. 2. write_reward now normalises all_scores entries through float() before the NaN/Inf check. Previously isinstance(v, float) was False for numpy scalars, so numpy nan slipped past the check into json.dumps, which succeeded (allow_nan=True) but produced invalid JSON; on some paths it landed as the string "nan" via the str(v) fallback. All non-finite numerics — native or numpy — now become null. Native bool/int/float are preserved verbatim; numpy/Decimal are coerced to plain float so json can serialise them. Verified: native + numpy nan/inf → null, plain floats preserved, numpy.float64/int64 coerced to Python numeric, bool preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * gdb adapter: address Slimshilin review comments - add run_gdb_parity.yaml with the 78 task names and parity selection logic (deterministic first-N per benchmark from upstream load_data()) - drop the TODO paragraph from _posthoc_nima_score; the preceding paragraph already explains the rationale Made-with: Cursor --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> Co-authored-by: Slimshilin <[email protected]>

gemini-code-assist Bot reviewed Dec 13, 2025

View reviewed changes

linhaowei1 self-requested a review December 16, 2025 05:28

linhaowei1 self-assigned this Dec 16, 2025

linhaowei1 reviewed Dec 16, 2025

View reviewed changes

Comment thread adapters/arc_agi_2/README.md

Comment thread adapters/arc_agi_2/adapter.py Outdated

linhaowei1 approved these changes Dec 26, 2025

View reviewed changes

Slimshilin requested changes Dec 27, 2025

View reviewed changes

Slimshilin changed the title ~~Add ARC-AGI-2 benchmark to adapters~~ [Ready for Review] Adapter: ARC-AGI-2 benchmark Dec 28, 2025

Slimshilin changed the title ~~[Ready for Review] Adapter: ARC-AGI-2 benchmark~~ [Ready for Review - fixing] Adapter: ARC-AGI-2 benchmark Dec 28, 2025

Ji-Pengliang and others added 19 commits January 2, 2026 05:28

fix: update arc_agi_2.yaml dataset path to correct tasks location

6224761

feat: add compute_accuracy.py for official ARC-AGI-2 puzzle-level metric

1075f72

- Aggregates 167 task results to 120 puzzle accuracy - Matches official ARC-AGI-2 leaderboard scoring - Puzzle passes only if ALL test pairs are correct

refactor: integrate scoring into run_adapter.py and utils.py

2b714be

- Added 'score' subcommand to run_adapter.py - Moved scoring functions to utils.py - Removed standalone compute_accuracy.py - Usage: run_adapter.py score --result-path <path>

chore: set n_attempts=2 to match official pass@2 scoring

f41277b

Update ARC-AGI-2 instruction template and utils

2b9002b

Update arc_agi_2.yaml: terminus-2 agent, n_concurrent_trials=24, env …

808a051

…vars

Set n_concurrent_trials to 8

8568234

fix(arc_agi_2): set default config for pass@2 scoring

367324a

Add ARC-AGI-2 parity experiment config

00af912

- Update instruction.md to match official arc-agi-benchmarking prompt - Add parity_experiment.json for Harbor vs arc-agi-benchmarking comparison

Update adapter_pr with arc-agi-benchmarking repo URL

46ea441

Update ARC-AGI-2 parity experiment results and instruction template

a055ee3

- Add parity experiment results: Harbor 37.3% vs arc-agi-benchmarking 38.9% (pass@2) - Update instruction.md to match arc-agi-benchmarking prompt format

Update ARC-AGI-2 parity experiment results

56e0d31

Update arc_agi_2.yaml to use agentx with gpt-5.2

32eb6a4

Update ARC-AGI-2 adapter README and add pyproject.toml

292f2e3

- Rewrite README following Harbor adapter template - Add comprehensive documentation with parity results - Add pyproject.toml with datasets dependency

Remove unused config.json generation from ARC-AGI-2 adapter

eaa112d

Add ARC-AGI-2 adapter to registry (167 tasks)

0f387ef

Update ARC-AGI-2 description to high-level benchmark overview

bae6b96

Horde added 2 commits January 2, 2026 05:32

update readme, registry and parity exp with the latest results

a0bffaf

resolve conflict

96a4a36

Ji-Pengliang force-pushed the main branch from 7ee7be0 to 96a4a36 Compare January 2, 2026 05:33

Horde added 2 commits January 2, 2026 05:34

Update registry.json

afb2a29

Revert "Update registry.json"

11b8c8c

This reverts commit afb2a29.

Ji-Pengliang requested a review from Slimshilin January 2, 2026 05:36

Horde added 3 commits January 2, 2026 07:48

style: format arc_agi_2 adapter files with ruff

dd67cbe

fix: remove duplicate arc_agi_2 entry and add git_commit_id to tasks

cd85e65

update parity exp

c93fd94

Slimshilin approved these changes Jan 2, 2026

View reviewed changes

Slimshilin merged commit 53f0ab9 into harbor-framework:main Jan 2, 2026
2 of 3 checks passed

mohitgargai mentioned this pull request Apr 17, 2026

[Ready for Review] Adapter: GDB (GraphicDesignBench) #1433

Merged

claude Bot mentioned this pull request Apr 28, 2026

[Ready for Review] Update GDB adapter dependency and invocation #1527

Merged

Uh oh!

Conversation

Ji-Pengliang commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Oracle Test

Parity Test

Overview

Settings

Results

Per-Trial Results (Pass@2)

Aggregate Statistics

Conclusion

Oracle Test with harbor-dataset merged

Uh oh!

gemini-code-assist Bot commented Dec 13, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

linhaowei1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

linhaowei1 left a comment

Choose a reason for hiding this comment

Uh oh!

Slimshilin left a comment

Choose a reason for hiding this comment

Uh oh!

Ji-Pengliang commented Dec 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ji-Pengliang commented Dec 28, 2025

Uh oh!

Slimshilin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Ji-Pengliang commented Dec 13, 2025 •

edited

Loading

linhaowei1 left a comment •

edited

Loading

Ji-Pengliang commented Dec 28, 2025 •

edited

Loading