Add harbor leaderboard submit command#1695
Merged
alexgshaw merged 15 commits intoJun 1, 2026
Merged
Conversation
|
@rebekah-west is attempting to deploy a commit to the Harbor Framework Team on Vercel. A member of the Team first needs to authorize it. |
josef223maker
approved these changes
May 21, 2026
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Enjoy a better diff viewing experience by clicking one of these URLs: |
alexgshaw
approved these changes
May 30, 2026
…anges-on-leaderboard-supabase-branch
Messimeimei
added a commit
to Messimeimei/bitfun-harbor
that referenced
this pull request
Jul 7, 2026
81e9c27 Add job plugin support and refactor Harbor Hub upload (harbor-framework#1762) a870a8b Add harbor-langsmith plugin package for LangSmith integration. (harbor-framework#1702) 284456b Add harbor-langsmith publish script and PyPI package metadata. a7ea96b Fail fast on Harbor Hub auth errors when using --upload (harbor-framework#1781) 087217b claude_code: bundle one LLM turn into a single ATIF step (RFC-0001) (harbor-framework#1760) 5c7d0ea fix(openclaw): Use --retry-all-errors flag only when available (harbor-framework#1789) 67bcaea Add harbor leaderboard submit command (harbor-framework#1695) 466fb98 fix: experiment sessions can now be found on reruns (harbor-framework#1804) 298a27d Release harbor-langsmith 0.1.1 e23c4d5 ci: upgrade github actions to latest versions (harbor-framework#1819) 51f6007 v0.13.1 Co-authored-by: Cursor <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
harbor leaderboard submit, the CLI path for entering official Harbor Hub leaderboards afterharbor runandharbor upload. Submissions are validated client-side against leaderboard rules, then written to Supabase as pending entries for admin review.harbor leaderboard submit(-lleaderboard slug,-jjob UUID(s),-mmetadata.yaml, optional-sexisting submission,-oJSON report). Supports creating a submission, attaching more jobs to a pending submission, and metadata-only updates.LeaderboardDBfor authenticated Supabase reads/writes (leaderboard,leaderboard_submission, trial pagination,attach_submission_jobRPC);RegistryDBextensions for dataset version resolution and task-ref → dataset version hints.terminal-bench-2-1documented as the first supported slug).scripts/test_leaderboard_submit.shfor manual end-to-end checks against a configured Supabase project.Depends on: Supabase schema/RPC from the leaderboard Supabase branch (e.g.
leaderboard_submission,leaderboard_submission_job,attach_submission_job). This PR is the Harbor client; merge or deploy DB changes before the command works in production.Test plan
uv run pytest tests/unit/test_leaderboard_*.py tests/unit/test_registry_db_task_refs.pyuv run ruff check --fix . && uv run ruff format . && uv run ty checkharbor auth loginagainst a project with leaderboard tables seededharbor run+harbor uploadforterminal-bench-2-1with-k 5, then:harbor leaderboard submit -l terminal-bench-2-1 -j <JOB_UUID> -m ./metadata.yaml→ pending submission id printed-s <SUBMISSION_UUID> -j <JOB_UUID_2>(same dataset version)-s <SUBMISSION_UUID> -m ./metadata.yaml-owrites full JSON)./scripts/test_leaderboard_submit.shwithHARBOR_SUPABASE_URL,HARBOR_SUPABASE_PUBLISHABLE_KEY, and test job/submission UUIDs (RUN_MUTATING=1for write paths)