Skip to content

Python: Prevents LLMs from stripping complex function names - #4366

Merged
Alex Chao (alexchaomander) merged 3 commits into
microsoft:mainfrom
juliomenendez:issue-3237-stepwiseplan-recall-context-not-used
Dec 20, 2023
Merged

Alex Chao (alexchaomander) merged 3 commits into
microsoft:mainfrom
juliomenendez:issue-3237-stepwiseplan-recall-context-not-used

Conversation

@juliomenendez

@juliomenendez Julio Carlos Menendez (juliomenendez) commented Dec 18, 2023 •

Copy link
Copy Markdown
Contributor

Motivation and Context

When using the StepwisePlanner planner along with core skills, for example TextMemorySkill, the LLMs tend to plans with function names stripped down to the name without the skill name. This makes the plan fail with an exception like:

Error invoking action recall: (<ErrorCodes.UnknownError: -1>, "The function 'recall' was not found.", None)

This error is because the completion returned by the planner prompt does not include the fully structured skill function name. Ex.: instead of returning _GLOBAL_FUNCTIONS_.recall it returns recall or GLOBAL_FUNCTIONS.recall.

This is somehow relates #3237

Description

This change instructs the LLM to keep the skill function name unchanged in its response by adding an example that includes _ in the name. This approach mimics how the SequentialPlanner solves this same issue.

Contribution Checklist

@shawncal Shawn Callegari (shawncal) added the python Pull requests for the Python Semantic Kernel label Dec 18, 2023
@github-actions github-actions Bot changed the title Prevents LLMs from stripping complex function names Python: Prevents LLMs from stripping complex function names Dec 18, 2023
@juliomenendez
Julio Carlos Menendez (juliomenendez) force-pushed the issue-3237-stepwiseplan-recall-context-not-used branch from 6baa890 to 040120f Compare December 18, 2023 22:39
@juliomenendez
Julio Carlos Menendez (juliomenendez) force-pushed the issue-3237-stepwiseplan-recall-context-not-used branch from 040120f to 473797a Compare December 18, 2023 22:41
@juliomenendez
Julio Carlos Menendez (juliomenendez) marked this pull request as ready for review December 18, 2023 22:45

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just have a high level question.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left another comment on the prompt itself.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, adding testing would be good, but can be a separate PR.

@moonbox3 Evan Mattson (moonbox3) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much for working on this!

@moonbox3
Evan Mattson (moonbox3) added this pull request to the merge queue Dec 19, 2023
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Dec 19, 2023
@moonbox3
Evan Mattson (moonbox3) added this pull request to the merge queue Dec 19, 2023
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Dec 19, 2023
@alexchaomander
Alex Chao (alexchaomander) added this pull request to the merge queue Dec 20, 2023
Merged via the queue into microsoft:main with commit 09d850b Dec 20, 2023
@juliomenendez
Julio Carlos Menendez (juliomenendez) deleted the issue-3237-stepwiseplan-recall-context-not-used branch December 20, 2023 14:46
Bryan (Bryan-Roe) pushed a commit to Bryan-Roe-ai/semantic-kernel that referenced this pull request Oct 6, 2024
…t#4366)

### Motivation and Context

When using the `StepwisePlanner` planner along with core skills, for
example `TextMemorySkill`, the LLMs tend to plans with function names
stripped down to the name without the skill name. This makes the plan
fail with an exception like:

`Error invoking action recall: (<ErrorCodes.UnknownError: -1>, "The
function 'recall' was not found.", None)`

This error is because the completion returned by the planner prompt does
not include the fully structured skill function name. Ex.: instead of
returning `_GLOBAL_FUNCTIONS_.recall` it returns `recall` or
`GLOBAL_FUNCTIONS.recall`.

This is somehow relates microsoft#3237

### Description


This change instructs the LLM to keep the skill function name unchanged
in its response by adding an example that includes `_` in the name. This
approach mimics how the `SequentialPlanner` solves this same issue.

### Contribution Checklist

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Pull requests for the Python Semantic Kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants