fix(cli): benchmark related products by bcfre · Pull Request #158 · sgl-project/rbg

bcfre · 2026-02-11T08:41:25Z

Ⅰ. Motivation

Ⅱ. Modifications

Ⅲ. Does this pull request fix one issue?

fixes #XXXX

Ⅳ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.

Ⅴ. Describe how to verify it

VI. Special notes for reviews

Checklist

Format your code make fmt.
Add unit tests or integration tests.
Update the documentation related to the change.

gemini-code-assist · 2026-02-11T08:41:40Z

Summary of Changes

Hello @bcfre, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request integrates the 'genai-bench' benchmark tool into the project's build system. It establishes the necessary infrastructure to build and manage a Docker image for 'genai-bench', enabling its use within the CLI's benchmark-related products.

Highlights

New Benchmark Tool Integration: Introduced support for a new benchmark tool, 'genai-bench', by defining its Docker image and Dockerfile path in the Makefile.
Docker Build and Push Targets: Added new Makefile targets, 'docker-build-benchtool-genai' and 'docker-push-benchtool-genai', to facilitate building and pushing the 'genai-bench' Docker image.
genai-bench Dockerfile: Created a dedicated Dockerfile ('Dockerfile-genai') for the 'genai-bench' tool, outlining the steps to build its image, including system dependencies, Python package management with 'pipx' and 'uv', and cloning/installing the 'genai-bench' source.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

Makefile
- Added BENCHMARK_BENCHTOOL_IMG and BENCHMARK_BENCHTOOL_DOCKERFILE variables.
- Introduced docker-build-benchtool-genai and docker-push-benchtool-genai targets.
cmd/cli/cmd/llm/benchmark/benchtool/Dockerfile-genai
- New file created to define the Docker image build process for genai-bench.
- Includes installation of system dependencies, pipx, uv, and cloning/installing genai-bench.

Ignored Files

Ignored by pattern: .github/workflows/** (2)
- .github/workflows/benchmark-benchtool-build.yml
- .github/workflows/release-cli.yml

Activity

No human activity has been recorded on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces support for a new benchmark tool, genai-bench, by adding a new Dockerfile and updating the Makefile to build and push the corresponding image. My review focuses on improving the new Dockerfile. I've provided a suggestion to optimize the Docker image size by restructuring the RUN commands. This change ensures that build dependencies and temporary files are cleaned up within a single layer, which is a best practice for creating smaller and more efficient Docker images.

coveralls · 2026-02-11T08:46:27Z

Pull Request Test Coverage Report for Build 22010296167

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 38.465%

Totals
Change from base Build 21977348362:	0.0%
Covered Lines:	4732
Relevant Lines:	12302

💛 - Coveralls

RongGu

/lgtm
/approve

Copilot

Pull request overview

This pull request adds infrastructure to support building and distributing the genai-bench benchmark tool and CLI binaries. It introduces a Dockerfile for containerizing genai-bench, Makefile targets for building the benchmark tool image, and GitHub Actions workflows for automated builds and releases.

Changes:

Added Dockerfile for genai-bench benchmark tool with Python 3.12 base image
Added Makefile targets for building and pushing the benchtool Docker image
Added GitHub Actions workflow for automated CLI binary releases across multiple platforms
Added GitHub Actions workflow for building and pushing the benchmark benchtool image

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File	Description
cmd/cli/cmd/llm/benchmark/benchtool/Dockerfile-genai	New Dockerfile that builds genai-bench tool image from source
Makefile	Added variables and targets for building/pushing benchtool-genai Docker image
.github/workflows/release-cli.yml	New workflow for releasing CLI binaries for multiple OS/arch combinations
.github/workflows/benchmark-benchtool-build.yml	New workflow for building and pushing benchmark benchtool Docker image

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

bcfre · 2026-02-11T12:55:46Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a new Docker image for benchtool-genai to support benchmarking, along with the necessary Makefile targets to build and push the image. My review focuses on improving the new Dockerfile for better efficiency and security. I've suggested using a multi-stage build to create a smaller final image. Additionally, it appears the new make targets (docker-build-benchtool-genai, docker-push-benchtool-genai) have not been added to the aggregate DOCKER_BUILD and DOCKER_PUSH variables in the Makefile. You might want to add them for consistency with other images in the project, so that make docker-build and make docker-push include this new image.

RongGu

/lgtm

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-12T06:39:42Z

+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Set up Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: ${{ env.GO_VERSION }}
+
+      - name: Get version info
+        id: version
+        run: |
+          if [[ "${{ github.event_name }}" == "push" && "${{ github.ref_type }}" == "tag" ]]; then
+            TAG=${GITHUB_REF#refs/tags/}
+          elif [[ -n "${{ inputs.tag }}" ]]; then
+            TAG="${{ inputs.tag }}"
+          else
+            # Fallback to latest git tag for manual dispatch without input
+            TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "dev")
+          fi


The workflow_dispatch path computes a tag but the repo is always checked out at the default ref. This can produce binaries whose embedded version/tag doesn’t match the source commit (and git_commit will be for the checked-out branch, not the requested tag). Update the checkout to use the requested tag/ref (or perform a second checkout after determining the tag) so the build inputs match the release tag.

cheyang · 2026-02-13T06:58:25Z

/gemini review

gemini-code-assist · 2026-02-13T06:58:28Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-13T07:05:10Z


+.PHONY: docker-build-benchtool-genai
+docker-build-benchtool-genai: ## Build docker image for benchmark benchtool (genai-bench)
+	$(CONTAINER_TOOL) build -f ${BENCHMARK_BENCHTOOL_DOCKERFILE} -t ${BENCHMARK_BENCHTOOL_IMG}:${TAG} .


This Docker build command is missing the $(DOCKER_BUILD_ARGS) parameter that is used in other Docker build targets (see lines 191, 195, 199, 203). While this may be intentional since the Dockerfile doesn't use GOPROXY/GOPRIVATE/GOSUMDB build args, it's inconsistent with the pattern established for other images in this Makefile. Consider either adding $(DOCKER_BUILD_ARGS) for consistency or documenting why it's not needed.

Suggested change

$(CONTAINER_TOOL) build -f ${BENCHMARK_BENCHTOOL_DOCKERFILE} -t ${BENCHMARK_BENCHTOOL_IMG}:${TAG} .

$(CONTAINER_TOOL) build -f ${BENCHMARK_BENCHTOOL_DOCKERFILE} -t ${BENCHMARK_BENCHTOOL_IMG}:${TAG} $(DOCKER_BUILD_ARGS) .

Signed-off-by: cheyang <[email protected]>

cheyang · 2026-02-14T02:41:44Z

/gemini review

gemini-code-assist · 2026-02-14T02:41:47Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-14T02:47:54Z

+          if [[ "${{ github.event_name }}" == "push" && "${{ github.ref_type }}" == "tag" ]]; then
+            TAG=${GITHUB_REF#refs/tags/}
+          elif [[ -n "${{ inputs.tag }}" ]]; then
+            TAG="${{ inputs.tag }}"
+          else
+            # Fallback to latest git tag for manual dispatch without input
+            TAG=$(git describe --tags --abbrev=0 2>/dev/null || echo "dev")
+          fi


On tag pushes, github.ref_type is not reliably set for the push event, so this condition can fail and fall back to git describe, potentially releasing with the wrong tag. A more reliable check is to test GITHUB_REF (e.g., refs/tags/*) or use ${{ startsWith(github.ref, 'refs/tags/') }} to detect tag refs.

Copilot · 2026-02-14T02:47:54Z

  workflow_dispatch: # Allow manual trigger
+    inputs:
+      tag:
+        description: 'Release tag (e.g., v0.1.0). If not specified, uses the latest git tag.'
+        required: false
+        type: string


workflow_dispatch is now supported, but the workflow still creates a GitHub Release without explicitly setting tag_name. For manual dispatch, action-gh-release will default to the branch ref (e.g., main), which can fail or create an unintended release. Consider wiring the chosen tag (input/fallback) through as an output and using it as tag_name (and optionally target_commitish) in the release step, or skipping release creation for non-tag runs.

Copilot · 2026-02-14T02:47:54Z

  workflow_dispatch: # Allow manual trigger
+    inputs:
+      tag:
+        description: 'Release tag (e.g., v0.1.0). If not specified, uses the latest git tag.'
+        required: false
+        type: string


The PR description still contains the placeholder fixes #XXXX, so it’s unclear what issue (if any) this change is meant to close. Please update the PR description to reference the real issue number or set it to NONE.

cheyang

/lgtm
/approve

Signed-off-by: cheyang <[email protected]>

bcfre requested a review from cheyang February 11, 2026 08:41

bcfre requested a review from Syspretor February 11, 2026 08:41

gemini-code-assist Bot reviewed Feb 11, 2026

View reviewed changes

Comment thread tools/benchmark/genai/Dockerfile Outdated

RongGu approved these changes Feb 11, 2026

View reviewed changes

RongGu requested review from RongGu and Copilot February 11, 2026 11:36

Copilot started reviewing on behalf of RongGu February 11, 2026 11:36 View session

Copilot AI reviewed Feb 11, 2026

View reviewed changes

cheyang reviewed Feb 11, 2026

View reviewed changes

Comment thread cmd/cli/cmd/llm/benchmark/benchtool/Dockerfile-genai Outdated

cheyang reviewed Feb 11, 2026

View reviewed changes

Comment thread .github/workflows/release-cli.yml

cheyang reviewed Feb 11, 2026

View reviewed changes

Comment thread cmd/cli/cmd/llm/benchmark/benchtool/Dockerfile-genai Outdated

gemini-code-assist Bot reviewed Feb 11, 2026

View reviewed changes

Comment thread tools/benchmark/genai/Dockerfile

bcfre requested a review from cheyang February 11, 2026 12:59

RongGu approved these changes Feb 12, 2026

View reviewed changes

RongGu requested a review from Copilot February 12, 2026 02:22

Copilot started reviewing on behalf of RongGu February 12, 2026 02:22 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

Comment thread tools/benchmark/genai/Dockerfile Outdated

Comment thread tools/benchmark/genai/Dockerfile Outdated

Comment thread .github/workflows/release-cli.yml

Comment thread Makefile

RongGu requested review from RongGu and Copilot February 12, 2026 06:35

Copilot started reviewing on behalf of RongGu February 12, 2026 06:35 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

bcfre added 2 commits February 13, 2026 14:53

fix(cli): benchmark related products

802c9c4

review

6e77449

cheyang force-pushed the bench-phase3 branch from e41bdfe to 6e77449 Compare February 13, 2026 06:55

RongGu requested a review from Copilot February 13, 2026 06:59

Copilot started reviewing on behalf of RongGu February 13, 2026 07:00 View session

Copilot AI reviewed Feb 13, 2026

View reviewed changes

optimize Dockerfile by consolidating redundant operations

32f55b9

Signed-off-by: cheyang <[email protected]>

RongGu requested a review from Copilot February 14, 2026 02:43

Copilot started reviewing on behalf of RongGu February 14, 2026 02:44 View session

Copilot AI reviewed Feb 14, 2026

View reviewed changes

cheyang approved these changes Feb 14, 2026

View reviewed changes

fix(ci): secure tag input handling in release workflow

d2cb0bf

Signed-off-by: cheyang <[email protected]>

cheyang merged commit 3078e73 into sgl-project:main Feb 14, 2026
8 checks passed

	$(CONTAINER_TOOL) build -f ${BENCHMARK_BENCHTOOL_DOCKERFILE} -t ${BENCHMARK_BENCHTOOL_IMG}:${TAG} .
	$(CONTAINER_TOOL) build -f ${BENCHMARK_BENCHTOOL_DOCKERFILE} -t ${BENCHMARK_BENCHTOOL_IMG}:${TAG} $(DOCKER_BUILD_ARGS) .

Conversation

bcfre commented Feb 11, 2026

Ⅰ. Motivation

Ⅱ. Modifications

Ⅲ. Does this pull request fix one issue?

Ⅳ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.

Ⅴ. Describe how to verify it

VI. Special notes for reviews

Checklist

Uh oh!

gemini-code-assist Bot commented Feb 11, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

coveralls commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 22010296167

Details

💛 - Coveralls

Uh oh!

RongGu left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bcfre commented Feb 11, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

RongGu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

cheyang commented Feb 13, 2026

Uh oh!

gemini-code-assist Bot commented Feb 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

coveralls commented Feb 11, 2026 •

edited

Loading

RongGu left a comment •

edited

Loading