Repository navigation
Description
Describe the bug
Entering a non-trivial, multi-step/multi-skill ask into the stepwise planner will often result in the planner never reaching a final answer.
To Reproduce
Steps to reproduce the behavior:
- Load Stepwise planner with a set of skills
- Submit an ask requiring use of multiple skills, summary of the findings.
- Observe the same thought repeating over and over without a final answer.
Expected behavior
The Stepwise planner should reach a final answer, especially when the thoughts look like final answers.
Screenshots
I won't be able to share screenshots due to it being organizational private data.
Platform
- OS: MacOS 14.0
- IDE: Visual Studio Code
- Language: Python
- Source: main branch of repository
Additional context
I think I have identified a solution here but I'd like to discuss with others before I submit a PR for it. I took a look at the C# semantic skill and included the final conditions that the C# skill includes in the Python version. It seems to fix the problem.
This is the addition to StepwiseStep/skprompt.txt:
Every Question should be followed by a Thought.
Every Thought should be followed by an Action or Final Answer.
Every Action should be followed by an Observation.
Every Observation should be followed by a Thought or Final Answer.
Produce Thoughts and Actions as necessary until you have a Final Answer.
But I've also experimented with just adding:
Every Thought should be followed by an Action or Final Answer.
Which also seems to be effective at reaching a final answer. I think this works because the core problem is that the model tries to follow a THOUGHT with something that doesn't lead to information discovery or conclusions. Like another thought or observation, ending the completion request before it is able to offer a next step or answer.
This is my full StepwiseStep prompt now:
[INSTRUCTION]
Answer the following questions as accurately as possible using the provided functions.
[AVAILABLE FUNCTIONS]
The function definitions below are in the following format:
<functionName>: <description>
inputs:
- <parameterName>: <parameterDescription>
- ...
{{$function_descriptions}}
[END AVAILABLE FUNCTIONS]
[USAGE INSTRUCTIONS]
To use the functions, specify a JSON blob representing an action. The JSON blob should contain an "action" key with the name of the function to use, and an "action_variables" key with a JSON object of string values to use when calling the function.
Do not call functions directly; they must be invoked through an action.
The "action_variables" value should always include an "input" key, even if the input value is empty. Additional keys in the "action_variables" value should match the defined [PARAMETERS] of the named "action" in [AVAILABLE FUNCTIONS].
Dictionary values in "action_variables" must be strings and represent the actual values to be passed to the function.
Ensure that the $JSON_BLOB contains only a SINGLE action; do NOT return multiple actions.
IMPORTANT: Use only the available functions listed in the [AVAILABLE FUNCTIONS] section. Do not attempt to use any other functions that are not specified.
Here is an example of a valid $JSON_BLOB:
{
"action": "functionName",
"action_variables": {"parameterName": "some value", ...}
}
[END USAGE INSTRUCTIONS]
[END INSTRUCTION]
[THOUGHT PROCESS]
[QUESTION]
the input question I must answer
[THOUGHT]
To solve this problem, I should carefully analyze the given question and identify the necessary steps. Any facts I discover earlier in my thought process should be repeated here to keep them readily available.
[ACTION]
{
"action": "functionName",
"action_variables": {"parameterName": "some value", ...}
}
[OBSERVATION]
The result of the action will be provided here.
... (These Thought/Action/Observation can repeat until the final answer is reached.)
Every observation should be followed by a Thought or Final Answer.
[FINAL ANSWER]
Once I have gathered all the necessary observations and performed any required actions, I can provide the final answer in a clear and human-readable format.
[END THOUGHT PROCESS]
Every Question should be followed by a Thought.
Every Thought should be followed by an Action or Final Answer.
Every Action should be followed by an Observation.
Every Observation should be followed by a Thought or Final Answer.
Produce Thoughts and Actions as necessary until you have a Final Answer.
Begin!
[QUESTION]
{{$question}}
{{$agent_scratch_pad}}
Any thoughts about submitting a PR for this? Any objections/concerns?
Activity
[-]Python: StepWise planner often repeats thoughts forever, without final answer[/-][+].Net: Python: StepWise planner often repeats thoughts forever, without final answer[/+][-].Net: Python: StepWise planner often repeats thoughts forever, without final answer[/-][+]Python: StepWise planner often repeats thoughts forever, without final answer[/+]kevin-m-kent commentedon Oct 20, 2023
Just to clarify - happy to submit the PR here for my proposed changes. But I also would like some feedback on what kinds of tests are relevant for changing something like a planner. Thank you!
madsbolaris commentedon Oct 24, 2023
Thanks for recommending this fix. The Semantic Kernel team is currently doing some research to determine if we can rewrite our planners to be more effective, so for the time being, I would hold off on creating a PR.
kevin-m-kent commentedon Oct 24, 2023
Thanks for the feedback and insight. Looking forward to what comes out of that process!
madsbolaris commentedon Nov 27, 2023
We recently released our v2 of Stepwise planner. Do you mind giving it a shot and see if it addresses the previous challenges?
kevin-m-kent commentedon Nov 29, 2023
Will give it a try this week and let you know. Thanks for the message
kevin-m-kent commentedon Dec 1, 2023
Will there be a release for Python as well? I was working with the Python library.