Skip to content

feat: skill references to work with jobs (incl server connection)#3683

Merged
dantasse merged 2 commits into
mainfrom
dantasse/job_skills
Jul 17, 2026
Merged

feat: skill references to work with jobs (incl server connection)#3683
dantasse merged 2 commits into
mainfrom
dantasse/job_skills

Conversation

@dantasse

Copy link
Copy Markdown
Contributor

Some additions to our lancedb skill to enable agents to use the jobs methods that we recently added. Eval tests (below, with and without these additions to the skill) suggest that they're helping, mostly to find the right method calls. These are a little unusual because they require REST server connection, they're not yet implemented in the SDKs.

┌─────────────────────┬───────────┬────────────┬─────────────┬──────────┬───────────┬──────────┬───────────┐
│        eval         │ grade w/o │ grade with │ improvement │ time w/o │ time with │ cost w/o │ cost with │
├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤
│ 8-list-running-jobs │ 2.5/3     │ 3/3        │ +0.5        │ 123s     │ 29s       │ $0.58    │ $0.18     │
├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤
│ 9-describe-job      │ 1/5       │ 5/5        │ +4.0        │ 159s     │ 52s       │ $0.62    │ $0.25     │
├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤
│ 10-cancel-job       │ 3/3       │ 3/3        │ +0.0        │ 99s      │ 35s       │ $0.55    │ $0.21     │
├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤
│ TOTAL               │ 6.5/11    │ 11/11      │ +4.5        │ 381s     │ 116s      │ $1.75    │ $0.65     │
└─────────────────────┴───────────┴────────────┴─────────────┴──────────┴───────────┴──────────┴───────────┘

Failure reasons are because the agent didn't know the right method to call, spent all its turns guessing REST calls, tried to inspect lancedb code, but didn't find the answer in here.

@dantasse
dantasse requested review from ChilePiquin and prrao87 July 16, 2026 21:10

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@github-actions github-actions Bot added the enhancement New feature or request label Jul 16, 2026

@prrao87 prrao87 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a significant bump in success rate. I also like that it's in its own file. Great stuff.

@dantasse
dantasse merged commit ab3041e into main Jul 17, 2026
7 checks passed
@dantasse
dantasse deleted the dantasse/job_skills branch July 17, 2026 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants