[FEATURE]: Prompt + file edit logging for benchmarking

Hi! Would it be possible to add logging for:

This would make it easier to replay the exact same prompt(s) and outputs for benchmarking across different models.

Optionally, we could also support an evaluation mode where an LLM acts as a judge to score/compare results for the same session.

This issue might be a duplicate of existing issues. Please check:

Feature Request: Add LLM API Call Duration Logging #9037: Feature Request: Add LLM API Call Duration Logging - Also requests logging of execution metrics for performance monitoring and debugging

Feel free to ignore if none of these address your specific case.

To stay organized issues are automatically closed after 90 days of no activity. If the issue is still relevant please open a new one.

Navigation Menu