[FEA]add profiler agent to the examples folder #120

zac-wang-nv · 2025-04-11T01:17:59Z

Description

This PR introduces a new example agent, profiler agent.
Users can use this agent to chat with the profiling traces in their AgentIQ agent application and use it as an interactive profiling tool.

Screen.Recording.2025-04-08.at.7.58.50.PM.mov

This agent requires users to configure AgentIQ with profiling and telemetry features enabled, following the documentation

By Submitting this PR I confirm:

I am familiar with the Contributing Guidelines.
We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
- Any contribution which contains commits that are not Signed-Off will not be accepted.
When the PR is ready for review, new or existing tests cover these changes.
When the PR is ready for review, the documentation is up to date with these changes.

copy-pr-bot · 2025-04-11T01:18:03Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

mdemoret-nv

Very cool example. A few questions for you.

examples/profiler_agent/pyproject.toml

dnandakumar-nv

This is a great feature! I had a few minor comments and requests for changes below. Aside from those, I think it would be wonderful to get some unit testing in for the tools in this example where appropriate, we well as the agent itself. Other examples in the repository have some tests you can reference.

Furthermore, I have a few suggestions I'm curious for your take on. It would be wonderful if this agent could also use output from the aiq profiler runs (such as latency and bottleneck analysis) to enrich results further.

The aiq profiler trace also contains a lot of rich metadata you could use that is not present in Pheonix. Using those traces could also mean this workflow can run standalone without requiring users to use Pheonix!

examples/profiler_agent/src/aiq_profiler_agent/register.py

zac-wang-nv · 2025-04-22T23:18:34Z

will add tests as well.

mdemoret-nv · 2025-05-01T01:56:21Z

@zac-wang-nv Would love to get this in for our next release. The cutoff is Friday. Do you think you could add some tests by the end of the week?

mdemoret-nv · 2025-05-01T01:57:47Z

@zac-wang-nv Can you also update your branch to adhere to our signoff policy? All commits need to be signed off in order to be merged: https://github.com/NVIDIA/AIQToolkit/pull/120/checks?check_run_id=40976234780

Signed-off-by: zac-wang-nv <[email protected]>

zac-wang-nv · 2025-05-01T15:01:06Z

@zac-wang-nv Can you also update your branch to adhere to our signoff policy? All commits need to be signed off in order to be merged: https://github.com/NVIDIA/AIQToolkit/pull/120/checks?check_run_id=40976234780

Sure.
I rebased my changes and singed them off, also two unit tests are added for tools. e2e test requires phoenix server and LLM calls, which I am not sure how to add.

examples/profiler_agent/src/aiq_profiler_agent/register.py

Signed-off-by: zac-wang-nv <[email protected]>

Copilot

Pull Request Overview

This PR introduces a new profiler agent example that integrates several tools (flow chart, token usage, Phoenix query, and response composer) to enable interactive profiling of LLM traces. Key changes include new tests, tool implementations and their registration, and updated configuration and documentation files to support the profiler agent.

Reviewed Changes

Copilot reviewed 15 out of 17 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
examples/profiler_agent/tests/test_profiler_agent.py	Added tests validating flow chart and token usage functionality.
examples/profiler_agent/src/aiq_profiler_agent/tool/*.py	New tools for token usage, flow chart, px query, and response composing.
examples/profiler_agent/src/aiq_profiler_agent/agent.py	New profiler agent implementation integrating the tools.
examples/profiler_agent/configs/config.yml	Updated configuration supporting the profiler agent.
examples/profiler_agent/README.md	Documentation updates for installation and usage instructions.

Files not reviewed (2)

examples/profiler_agent/tests/test_spans.csv: Language not supported
external/agentiq-opensource-ui: Language not supported

examples/profiler_agent/src/aiq_profiler_agent/tool/flow_chart.py

examples/profiler_agent/README.md

Co-authored-by: Copilot <[email protected]> Signed-off-by: Anuradha Karuppiah <[email protected]>

AnuradhaKaruppiah · 2025-05-01T23:08:40Z

/ok to test 6c96605

Signed-off-by: Anuradha Karuppiah <[email protected]>

AnuradhaKaruppiah · 2025-05-02T00:03:28Z

/ok to test 9a30d36

Signed-off-by: Anuradha Karuppiah <[email protected]>

AnuradhaKaruppiah · 2025-05-02T00:09:37Z

/ok to test a958860

Signed-off-by: Anuradha Karuppiah <[email protected]>

AnuradhaKaruppiah · 2025-05-02T00:13:44Z

/ok to test 9c76263

Signed-off-by: Anuradha Karuppiah <[email protected]>

AnuradhaKaruppiah · 2025-05-02T04:04:35Z

/ok to test 6bcf300

Signed-off-by: Anuradha Karuppiah <[email protected]>

Add steps for generating traces and using the UI Signed-off-by: Anuradha Karuppiah <[email protected]>

Signed-off-by: Anuradha Karuppiah <[email protected]>

Signed-off-by: zac-wang-nv <[email protected]>

…evelop

AnuradhaKaruppiah · 2025-05-05T19:46:10Z

/ok to test 0c30da3

Signed-off-by: Anuradha Karuppiah <[email protected]>

AnuradhaKaruppiah · 2025-05-05T19:51:18Z

/ok to test 0138b4c

Signed-off-by: Anuradha Karuppiah <[email protected]>

examples/profiler_agent/pyproject.toml

Signed-off-by: zac-wang-nv <[email protected]>

…evelop

AnuradhaKaruppiah · 2025-05-05T22:09:55Z

/ok to test 060b91d

AnuradhaKaruppiah · 2025-05-05T22:54:38Z

/ok to test c8e5775

AnuradhaKaruppiah · 2025-05-05T23:16:26Z

/ok to test 5c010c3

AnuradhaKaruppiah · 2025-05-05T23:35:49Z

/ok to test 3191a8d

AnuradhaKaruppiah · 2025-05-05T23:42:42Z

/merge

Reviewer unavailable

This PR introduces a new example agent, profiler agent. Users can use this agent to chat with the profiling traces in their AgentIQ agent application and use it as an interactive profiling tool. https://github.com/user-attachments/assets/318f9a1e-fb5a-47dd-a9f4-32e088aebd51 This agent requires users to configure AgentIQ with profiling and telemetry features enabled, following [the documentation](https://docs.nvidia.com/agentiq/latest/guides/profiler.html) ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Zac Wang (https://github.com/zac-wang-nv) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: NVIDIA#120 Signed-off-by: Yuchen Zhang <[email protected]>

This PR introduces a new example agent, profiler agent. Users can use this agent to chat with the profiling traces in their AgentIQ agent application and use it as an interactive profiling tool. https://github.com/user-attachments/assets/318f9a1e-fb5a-47dd-a9f4-32e088aebd51 This agent requires users to configure AgentIQ with profiling and telemetry features enabled, following [the documentation](https://docs.nvidia.com/agentiq/latest/guides/profiler.html) ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Zac Wang (https://github.com/zac-wang-nv) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: NVIDIA#120 Signed-off-by: Eric Evans <[email protected]>

This PR introduces a new example agent, profiler agent. Users can use this agent to chat with the profiling traces in their AgentIQ agent application and use it as an interactive profiling tool. https://github.com/user-attachments/assets/318f9a1e-fb5a-47dd-a9f4-32e088aebd51 This agent requires users to configure AgentIQ with profiling and telemetry features enabled, following [the documentation](https://docs.nvidia.com/agentiq/latest/guides/profiler.html) ## By Submitting this PR I confirm: - I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/AgentIQ/blob/develop/docs/source/advanced/contributing.md). - We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license. - Any contribution which contains commits that are not Signed-Off will not be accepted. - When the PR is ready for review, new or existing tests cover these changes. - When the PR is ready for review, the documentation is up to date with these changes. Authors: - Zac Wang (https://github.com/zac-wang-nv) - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) URL: NVIDIA#120

zac-wang-nv requested a review from a team as a code owner April 11, 2025 01:17

mdemoret-nv added feature request New feature or request non-breaking Non-breaking change labels Apr 15, 2025

mdemoret-nv reviewed Apr 15, 2025

View reviewed changes

examples/profiler_agent/pyproject.toml Outdated Show resolved Hide resolved

examples/profiler_agent/pyproject.toml Outdated Show resolved Hide resolved

examples/profiler_agent/pyproject.toml Show resolved Hide resolved

dnandakumar-nv previously requested changes Apr 21, 2025

View reviewed changes

examples/profiler_agent/src/aiq_profiler_agent/register.py Outdated Show resolved Hide resolved

examples/profiler_agent/src/aiq_profiler_agent/register.py Outdated Show resolved Hide resolved

examples/profiler_agent/src/aiq_profiler_agent/register.py Outdated Show resolved Hide resolved

zac-wang-nv added 2 commits May 1, 2025 09:47

add profiler agent to the examples folder

be1800b

Signed-off-by: zac-wang-nv <[email protected]>

rename to aiqtoolkit

517a073

Signed-off-by: zac-wang-nv <[email protected]>

zac-wang-nv force-pushed the develop branch 2 times, most recently from fad2ae6 to 517a073 Compare May 1, 2025 14:58

AnuradhaKaruppiah reviewed May 1, 2025

View reviewed changes

examples/profiler_agent/src/aiq_profiler_agent/register.py Outdated Show resolved Hide resolved

fix merge errors

a15a692

Signed-off-by: zac-wang-nv <[email protected]>

AnuradhaKaruppiah requested a review from Copilot May 1, 2025 23:01

Copilot AI reviewed May 1, 2025

View reviewed changes

examples/profiler_agent/src/aiq_profiler_agent/tool/flow_chart.py Outdated Show resolved Hide resolved

examples/profiler_agent/README.md Outdated Show resolved Hide resolved

AnuradhaKaruppiah and others added 2 commits May 1, 2025 16:06

Update examples/profiler_agent/README.md

b0db0fe

Co-authored-by: Copilot <[email protected]> Signed-off-by: Anuradha Karuppiah <[email protected]>

Update examples/profiler_agent/src/aiq_profiler_agent/tool/flow_chart.py

6c96605

Co-authored-by: Copilot <[email protected]> Signed-off-by: Anuradha Karuppiah <[email protected]>

AnuradhaKaruppiah added 2 commits May 1, 2025 17:00

Fixes for precommit run -a

db5f49a

Signed-off-by: Anuradha Karuppiah <[email protected]>

Add copyright header and change to editable install for consistency

9a30d36

Signed-off-by: Anuradha Karuppiah <[email protected]>

Add copyright headers

a958860

Signed-off-by: Anuradha Karuppiah <[email protected]>

One more missing CPR header

9c76263

Signed-off-by: Anuradha Karuppiah <[email protected]>

Only run example tests if the example is installed

6bcf300

Signed-off-by: Anuradha Karuppiah <[email protected]>

AnuradhaKaruppiah and others added 6 commits May 5, 2025 11:37

Check if request is present before setting attributes

d5206d3

Signed-off-by: Anuradha Karuppiah <[email protected]>

Update profiler agent example README

d3ce047

Add steps for generating traces and using the UI Signed-off-by: Anuradha Karuppiah <[email protected]>

Merge branch 'NVIDIA:develop' into develop

5f48bc1

Add aiq-profiler-agent to the parent pyproject.toml file

c77646a

Signed-off-by: Anuradha Karuppiah <[email protected]>

fix extract user query

fe560b5

Signed-off-by: zac-wang-nv <[email protected]>

Merge branch 'develop' of https://github.com/zzzacwork/AgentIQ into d…

0c30da3

…evelop

Remove old UI submodule that was introduced accidentally

0138b4c

Signed-off-by: Anuradha Karuppiah <[email protected]>

Add a sample response image to the readme

2edeab1

Signed-off-by: Anuradha Karuppiah <[email protected]>

AnuradhaKaruppiah reviewed May 5, 2025

View reviewed changes

examples/profiler_agent/pyproject.toml Show resolved Hide resolved

zac-wang-nv added 2 commits May 5, 2025 16:06

add profile image

46fce32

Signed-off-by: zac-wang-nv <[email protected]>

Merge branch 'develop' of https://github.com/zzzacwork/AgentIQ into d…

060b91d

…evelop

Merge remote-tracking branch 'aiq/develop' into develop

5c010c3

zac-wang-nv force-pushed the develop branch from c8e5775 to 5c010c3 Compare May 5, 2025 23:15

Merge remote-tracking branch 'aiq/develop' into develop

3191a8d

AnuradhaKaruppiah approved these changes May 5, 2025

View reviewed changes

rapids-bot bot merged commit 9677506 into NVIDIA:develop May 5, 2025
10 checks passed

[FEA]add profiler agent to the examples folder #120

[FEA]add profiler agent to the examples folder #120

Uh oh!

Conversation

zac-wang-nv commented Apr 11, 2025

Description

By Submitting this PR I confirm:

Uh oh!

copy-pr-bot bot commented Apr 11, 2025

Uh oh!

mdemoret-nv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dnandakumar-nv left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zac-wang-nv commented Apr 22, 2025

Uh oh!

mdemoret-nv commented May 1, 2025

Uh oh!

mdemoret-nv commented May 1, 2025

Uh oh!

zac-wang-nv commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

AnuradhaKaruppiah commented May 1, 2025

Uh oh!

AnuradhaKaruppiah commented May 2, 2025

Uh oh!

AnuradhaKaruppiah commented May 2, 2025

Uh oh!

AnuradhaKaruppiah commented May 2, 2025

Uh oh!

AnuradhaKaruppiah commented May 2, 2025

Uh oh!

AnuradhaKaruppiah commented May 5, 2025

Uh oh!

AnuradhaKaruppiah commented May 5, 2025

Uh oh!

Uh oh!

AnuradhaKaruppiah commented May 5, 2025

Uh oh!

AnuradhaKaruppiah commented May 5, 2025

Uh oh!

AnuradhaKaruppiah commented May 5, 2025

Uh oh!

AnuradhaKaruppiah commented May 5, 2025

Uh oh!

AnuradhaKaruppiah commented May 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dnandakumar-nv left a comment •

edited

Loading

zac-wang-nv commented May 1, 2025 •

edited

Loading