-
Notifications
You must be signed in to change notification settings - Fork 481
feat!: Redis Object Store; Object Store Improvements #649
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat!: Redis Object Store; Object Store Improvements #649
Conversation
b80642a to
7aca8a5
Compare
…ple directions Signed-off-by: Will Killian <[email protected]>
7aca8a5 to
e1d7420
Compare
3be08ed to
9815fef
Compare
Signed-off-by: Will Killian <[email protected]>
077a855 to
3ba3ac5
Compare
Signed-off-by: Will Killian <[email protected]>
3ba3ac5 to
d88b365
Compare
Signed-off-by: Will Killian <[email protected]>
Signed-off-by: Will Killian <[email protected]>
WalkthroughAdds a Redis-backed object store and tests; refactors S3/MySQL object-store constructors to keyword-only args and config unpacking; updates docs, examples, and deployment files (MinIO/MySQL/Redis); removes bespoke upload scripts; and introduces a plugin-aware CLI exposing upload/delete and wired into the NAT entrypoint. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor U as User
participant C as NAT CLI
participant WB as WorkflowBuilder
participant F as Object Store Factory
participant S as ObjectStore Client
participant B as Backend (S3/MySQL/Redis)
U->>C: nat object-store <type> upload --dir <path>
C->>WB: init workflow (load plugin config)
WB->>F: get_object_store_client(config)
F->>S: async construct via **kwargs** (exclude "type")
S-->>F: ready (async context)
F-->>WB: yield client
loop per file
C->>S: upsert_object(key, ObjectStoreItem)
S->>B: write object
B-->>S: ack
S-->>C: ok
end
C-->>U: summary / exit
rect rgba(200,230,255,0.25)
note right of F: New behavior — keyword-only ctors + async context
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes ✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
Status, Documentation and Community
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 13
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (12)
examples/object_store/user_report/README.md (1)
240-247: Fix user ID typo in Expected Output.“678901” looks accidental; align with earlier “6789”.
-User report for 678901 with date latest added successfully +User report for 6789 with date latest added successfullypackages/nvidia_nat_redis/src/nat/plugins/redis/memory.py (1)
33-59: Add docstrings, return type, and ensure the Redis client is closed.
- Public APIs require return type hints and Google-style docstrings.
- Close the Redis client after use to avoid connection leaks in long-lived processes.
Apply:
-@register_memory(config_type=RedisMemoryClientConfig) -async def redis_memory_client(config: RedisMemoryClientConfig, builder: Builder): +@register_memory(config_type=RedisMemoryClientConfig) +async def redis_memory_client(config: RedisMemoryClientConfig, builder: Builder) -> "AsyncIterator[RedisEditor]": + """Create and yield a Redis-backed memory editor. + + Args: + config: Redis memory client configuration. + builder: NAT builder used to resolve embedders and dependencies. + + Yields: + A configured RedisEditor instance bound to the provided Redis connection. + """ @@ - redis_client = redis.Redis(host=config.host, + redis_client = redis.Redis(host=config.host, port=config.port, db=config.db, decode_responses=True, socket_timeout=5.0, socket_connect_timeout=5.0) @@ - memory_editor = RedisEditor(redis_client=redis_client, key_prefix=config.key_prefix, embedder=embedder) - - yield memory_editor + memory_editor = RedisEditor(redis_client=redis_client, key_prefix=config.key_prefix, embedder=embedder) + try: + yield memory_editor + finally: + try: + await redis_client.aclose() + except Exception: # best-effort close + passAdd the import at top of file (or use collections.abc if preferred):
+from collections.abc import AsyncIteratorpackages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py (4)
63-74: Add return type hints and brief method docstrings to satisfy public-API standards.- async def __aenter__(self): + async def __aenter__(self) -> "S3ObjectStore": + """Open an S3 client context and ensure the bucket exists.""" @@ - async def __aexit__(self, exc_type, exc_value, traceback): + async def __aexit__(self, exc_type, exc_value, traceback) -> None: + """Close the S3 client context."""
75-82: Bucket creation: handle AccessDenied and non-us-east-1 regions.
- Trying to create on 403 (AccessDenied) is misleading; re-raise instead.
- For AWS regions other than us-east-1,
CreateBucketConfigurationwithLocationConstraintis required.Apply:
- try: - await self._client.head_bucket(Bucket=self.bucket_name) - except ClientError as e: - if e.response['Error']['Code'] == '404': - await self._client.create_bucket(Bucket=self.bucket_name) - logger.info("Created bucket %s", self.bucket_name) + try: + await self._client.head_bucket(Bucket=self.bucket_name) + except ClientError as e: + code = e.response.get("Error", {}).get("Code") + if code in ("404", "NoSuchBucket"): + create_args = {"Bucket": self.bucket_name} + # For AWS S3 outside us-east-1, LocationConstraint is required + region_name = self._client_args.get("region_name") + if region_name and region_name != "us-east-1": + create_args["CreateBucketConfiguration"] = {"LocationConstraint": region_name} + await self._client.create_bucket(**create_args) + logger.info("Created bucket %s", self.bucket_name) + elif code in ("403", "AccessDenied"): + logger.error("Access denied to bucket %s; cannot create or verify.", self.bucket_name) + raise + else: + raise
110-124: Precondition mapping: also match the S3 error code name.Some providers set
Error.CodetoPreconditionFailedinstead of relying on HTTP 412. Handle both.- http_status_code = e.response.get("ResponseMetadata", {}).get("HTTPStatusCode", None) - if http_status_code == 412: + http_status_code = e.response.get("ResponseMetadata", {}).get("HTTPStatusCode") + error_code = e.response.get("Error", {}).get("Code") + if http_status_code == 412 or error_code == "PreconditionFailed": raise KeyAlreadyExistsError( key=key, additional_message=f"S3 object {self.bucket_name}/{key} already exists", ) from e # Other errors — rethrow or handle accordingly raise
156-171: Delete semantics: avoid GET, use HEAD; don’t treat DeleteMarker as an error.
- Using
get_objectto test existence is costly;head_objectis cheaper.- Raising on
DeleteMarker=Trueis incorrect for versioned buckets; it indicates a successful delete marker creation.Apply:
- try: - await self._client.get_object(Bucket=self.bucket_name, Key=key) - except ClientError as e: - if e.response['Error']['Code'] == 'NoSuchKey': - raise NoSuchKeyError(key=key, additional_message=str(e)) from e - raise - - results = await self._client.delete_object(Bucket=self.bucket_name, Key=key) - - if results.get('DeleteMarker', False): - raise NoSuchKeyError(key=key, additional_message="Object was a delete marker") + try: + await self._client.head_object(Bucket=self.bucket_name, Key=key) + except ClientError as e: + if e.response.get('Error', {}).get('Code') in ('404', 'NoSuchKey', 'NotFound'): + raise NoSuchKeyError(key=key, additional_message=str(e)) from e + raise + + await self._client.delete_object(Bucket=self.bucket_name, Key=key)examples/object_store/user_report/pyproject.toml (1)
1-1: Add missing SPDX Apache-2.0 header topyproject.tomlThe file
examples/object_store/user_report/pyproject.tomldoes not include the required SPDX header and will fail our CI policy check inci/scripts/github/checks.sh.• File to update:
• examples/object_store/user_report/pyproject.tomlPlease prepend the following standard header at the very top of that file:
# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0docs/source/store-and-retrieve/object-store.md (2)
18-18: Heading violates product naming guidance.Per docs guidelines, headings should use “NeMo Agent Toolkit” (without “NVIDIA”). Update to:
- “Object Store for NeMo Agent Toolkit”.
-# Object Store for NVIDIA NeMo Agent Toolkit +# Object Store for NeMo Agent Toolkit
145-149: metadata uses a set literal; should be a dict.This will not serialize as intended and contradicts the stated metadata type.
- await object_store.upsert_object("greeting.txt", ObjectStoreItem( - data=b"Goodbye, World!", - content_type="text/plain", - metadata={"author", "user123"} - )) + await object_store.upsert_object( + "greeting.txt", + ObjectStoreItem( + data=b"Goodbye, World!", + content_type="text/plain", + metadata={"author": "user123"}, + ), + )packages/nvidia_nat_redis/src/nat/plugins/redis/object_store.py (1)
34-41: Annotate the factory’s return type and document it (pyright treats missing annotations as errors).The factory is a public API; add a return type and a short docstring.
Apply this diff:
-@register_object_store(config_type=RedisObjectStoreClientConfig) -async def redis_object_store_client(config: RedisObjectStoreClientConfig, _builder: Builder): +@register_object_store(config_type=RedisObjectStoreClientConfig) +async def redis_object_store_client( + config: RedisObjectStoreClientConfig, + _builder: Builder, +) -> AsyncIterator[ObjectStore]: + """Create and yield a Redis-backed ObjectStore client for the given config.""" @@ - async with RedisObjectStore(**config.model_dump(exclude={"type"})) as store: + async with RedisObjectStore(**config.model_dump(exclude={"type"})) as store: yield storepackages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (1)
101-112: Type-annotate aexit with standard signature.Explicit types avoid pyright/flake warnings.
Apply this diff (add
from types import TracebackTypeto imports):- async def __aexit__(self, exc_type, exc_value, traceback): + async def __aexit__( + self, + exc_type: type[BaseException] | None, + exc_value: BaseException | None, + traceback: TracebackType | None, + ) -> None:docs/source/extend/object-store.md (1)
231-244: Add missing imports in the “Minimal Skeleton” object_store.py snippet.Readers will otherwise copy a snippet that references names that aren’t imported.
Apply this diff:
-```python -from nat.data_models.object_store import ObjectStoreBaseConfig +```python +from nat.builder.builder import Builder +from nat.cli.register_workflow import register_object_store +from nat.data_models.object_store import ObjectStoreBaseConfig @@ -@register_object_store(config_type=MyCustomObjectStoreConfig) +@register_object_store(config_type=MyCustomObjectStoreConfig) async def my_custom_object_store(config: MyCustomObjectStoreConfig, _builder: Builder): @@ from .my_custom_object_store import MyCustomObjectStore yield MyCustomObjectStore(**config.model_dump(exclude={"type"}))</blockquote></details> </blockquote></details>🧹 Nitpick comments (38)
src/nat/cli/commands/object_store/__init__.py (1)
1-15: Add a minimal module docstring (public module requirement) and optionally define explicit exports.Per coding guidelines, every public Python module should have a concise Google‑style docstring. Since this initializer intentionally has no functional code, a short docstring and empty
__all__keeps API surface clear and satisfies docs linting.Apply:
# SPDX-License-Identifier: Apache-2.0 # limitations under the License. +"""Initializer for the object-store CLI command package. + +This module exists to mark the object_store command directory as a package and +enable plugin discovery/wiring via nat's CLI entrypoints. No public API is +intentionally exported from here. +""" + +__all__: list[str] = []examples/object_store/user_report/configs/config_mysql.yml (1)
29-29: Confirm env-var interpolation semantics for default values in application config loader.
${MYSQL_ROOT_PASSWORD:-my_password}is shell-style expansion. Ensure the NAT config loader expands the:-default operator; otherwise this will be treated as a literal string and authentication will fail at runtime. If your loader only supports simple${VAR}or${env:VAR,default}patterns, adjust accordingly and mirror the docs.If needed, consider switching to a supported pattern (example):
- password: ${MYSQL_ROOT_PASSWORD:-my_password} + # Use the interpolation syntax supported by the NAT config loader: + password: ${env:MYSQL_ROOT_PASSWORD,my_password}examples/deploy/docker-compose.minio.yml (1)
31-35: Healthcheck usescurlinside the container; verify it exists or switch to a shell-basedwgetcheck.Some MinIO images don’t include
curlby default. Ifcurlis missing, the container will report unhealthy despite being up. Consider aCMD-SHELLwgetprobe or confirmcurlavailability for your tag.Option A — keep curl but use CMD-SHELL (if curl available):
- healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:9000/minio/health/live || exit 1"] interval: 30s timeout: 10s retries: 3 start_period: 30sOption B — switch to wget (often present in minimal images):
- healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://localhost:9000/minio/health/live >/dev/null 2>&1 || exit 1"] interval: 30s timeout: 10s retries: 3 start_period: 30sexamples/deploy/README.md (4)
35-39: Tighten phrasing and articles in Key Features.Minor grammar/readability tweaks:
- Add missing articles.
- Prefer “observability server” phrasing.
Apply:
-**Phoenix Observability:** Includes `docker-compose.phoenix.yml` for running Phoenix observability server to monitor and debug workflows. +**Phoenix Observability:** Includes `docker-compose.phoenix.yml` for running a Phoenix observability server to monitor and debug workflows.-**Redis Service:** Includes `docker-compose.redis.yml` for running Redis memory backend with Redis Insight for memory-based examples. +**Redis Service:** Includes `docker-compose.redis.yml` for running a Redis memory backend with Redis Insight for memory-based examples.
65-73: Optional: cue first-time users that this creates the MinIO data volume.A small hint can reduce questions about persistence:
-To start MinIO (for object store examples): +To start MinIO (for object store examples; this also creates a local data volume):
80-84: Clarify Redis usage contexts.Add “(and Redis-backed object store if selected)” to avoid confusion that Redis is only for memory:
-To start Redis (required for memory and object store examples): +To start Redis (required for memory examples and for the Redis-backed object store):
87-106: Add a note on data persistence when stopping services.New users might unintentionally expect data to be cleared. Consider:
-To stop the MinIO service: +To stop the MinIO service (data persists; use `-v` to remove volumes if you want a clean slate):Likewise for MySQL and Redis:
-docker compose -f examples/deploy/docker-compose.mysql.yml down +docker compose -f examples/deploy/docker-compose.mysql.yml down +# To also remove the MySQL data volume (destructive): +# docker compose -f examples/deploy/docker-compose.mysql.yml down -vexamples/object_store/user_report/README.md (5)
71-83: MinIO section: grammar fixes and clarity.
- Use “on your local machine”.
- Add article “a” in “a drop‑in replacement”.
- Fix “should not to be used” → “should not be used”.
Apply:
-If you want to run this example in a local setup without creating a bucket in AWS, you can set up MinIO in your local machine. MinIO is an object storage system and acts as drop-in replacement for AWS S3. +If you want to run this example in a local setup without creating a bucket in AWS, you can set up MinIO on your local machine. MinIO is an object storage system and acts as a drop-in replacement for AWS S3.-> This is not a secure configuration and should not to be used in production systems. +> This is not a secure configuration and should not be used in production systems.
84-102: MySQL section: security note grammar + tiny wording tweak.-You should first specify the `MYSQL_ROOT_PASSWORD` environment variable. +First, set the `MYSQL_ROOT_PASSWORD` environment variable.-> This is not a secure configuration and should not to be used in production systems. +> This is not a secure configuration and should not be used in production systems.
103-113: Redis section: security note grammar.-> This is not a secure configuration and should not to be used in production systems. +> This is not a secure configuration and should not be used in production systems.
118-133: Clarify “my-bucket” semantics across providers and credentials handling.
- For MySQL/Redis, “my-bucket” maps to a logical namespace/prefix rather than an actual bucket. Consider adding a one-liner to prevent confusion.
- Passing secrets via CLI can leak into shell history. Suggest env vars or interactive prompt.
Proposed additions (after the code block):
+Note: +- For MySQL and Redis, `my-bucket` is a logical namespace (key prefix), not a physical bucket. +- Avoid passing secrets on the command line when possible (they may be saved in shell history). Prefer environment variables or a prompt.
153-159: Cross-reference available configs to reduce guesswork.Add explicit pointers:
-The above command will use the S3-compatible object store. Other configuration files are available in the `configs` directory for the different object stores. +The above command uses the S3-compatible object store. Alternative configs: `configs/config_mysql.yml` and `configs/config_redis.yml`.examples/object_store/user_report/pyproject.toml (1)
12-16: Remove trailing comma in the TOML array.Some TOML parsers in tooling still choke on trailing commas in arrays. Safer to drop the trailing comma after the last dependency.
Apply:
dependencies = [ "nvidia-nat[mysql]~=1.3", "nvidia-nat[redis]~=1.3", - "nvidia-nat[s3]~=1.3", + "nvidia-nat[s3]~=1.3" ]docs/source/store-and-retrieve/object-store.md (3)
20-23: Fix first-use product name in body text.First mention in body should be “NVIDIA NeMo Agent toolkit”; subsequent mentions “NeMo Agent toolkit”.
-The NeMo Agent toolkit Object Store subsystem provides a standardized interface... +The NVIDIA NeMo Agent toolkit Object Store subsystem provides a standardized interface...
182-183: Prefer --data-binary (or -T) for raw file uploads and address MD014.-curl -d defaults to application/x-www-form-urlencoded; use --data-binary to send bytes as-is. Also drop the leading “$” to satisfy markdownlint MD014.
- $ curl -X PUT -d @data.txt http://localhost:9000/static/folder/data.txt + curl -X PUT --data-binary @data.txt http://localhost:9000/static/folder/data.txtAlso apply the same “no $ prompt” style to the other curl examples in this section.
190-191: Use --data-binary for POST and align prompt style.Same rationale as PUT; avoids content-type surprises and MD014 lints.
- $ curl -X POST -d @data_new.txt http://localhost:9000/static/folder/data.txt + curl -X POST --data-binary @data_new.txt http://localhost:9000/static/folder/data.txtsrc/nat/cli/entrypoint.py (1)
116-116: Avoid broad type: ignore if possible.If feasible, add precise typing to start_command.get_command or cast to click.Command at the call site to remove the blanket ignore.
-cli.add_command(start_command.get_command(None, "mcp"), name="mcp") # type: ignore +from typing import cast +cli.add_command(cast(click.Command, start_command.get_command(None, "mcp")), name="mcp")packages/nvidia_nat_s3/src/nat/plugins/s3/object_store.py (1)
44-49: Constructor kwargs and relative import are correct; add return type + docstring.Good switch to keyword unpacking and avoiding an unused builder. To meet API typing/docstring guidelines, add a return type and concise docstring.
-async def s3_object_store_client(config: S3ObjectStoreClientConfig, _builder: Builder): +async def s3_object_store_client( + config: S3ObjectStoreClientConfig, + _builder: Builder, +) -> "AsyncIterator[S3ObjectStore]": + """Yield an async S3ObjectStore client constructed from the provided config.""" @@ - async with S3ObjectStore(**config.model_dump(exclude={"type"})) as store: + async with S3ObjectStore(**config.model_dump(exclude={"type"})) as store: yield storeAdd these imports at the top of the file:
# at top-level (outside selected range) from __future__ import annotations from collections.abc import AsyncIteratorpackages/nvidia_nat_mysql/src/nat/plugins/mysql/object_store.py (1)
61-66: Mirror S3 improvements: add return type + docstring.Signature update to _builder and kwargs is good. Add typing and a docstring to satisfy public API requirements.
-async def mysql_object_store_client(config: MySQLObjectStoreClientConfig, _builder: Builder): +async def mysql_object_store_client( + config: MySQLObjectStoreClientConfig, + _builder: Builder, +) -> "AsyncIterator[MySQLObjectStore]": + """Yield an async MySQLObjectStore client constructed from the provided config.""" @@ - async with MySQLObjectStore(**config.model_dump(exclude={"type"})) as store: + async with MySQLObjectStore(**config.model_dump(exclude={"type"})) as store: yield storeAdd these imports at the top of the file:
# at top-level (outside selected range) from __future__ import annotations from collections.abc import AsyncIteratorpackages/nvidia_nat_redis/tests/test_redis_object_store.py (1)
32-41: Isolate test buckets to avoid key collisions across runs.Using a static bucket_name (“test”) can create cross-run interference. Prefer a unique bucket per run.
Apply this diff:
@@ -from contextlib import asynccontextmanager +from contextlib import asynccontextmanager +from uuid import uuid4 @@ - RedisObjectStoreClientConfig(bucket_name="test", host="localhost", port=6379, db=0), + RedisObjectStoreClientConfig( + bucket_name=f"test-{uuid4().hex}", + host="localhost", + port=6379, + db=0, + ),packages/nvidia_nat_redis/src/nat/plugins/redis/object_store.py (1)
16-22: Add a module docstring and missing imports for types to satisfy pyright and docs.Public modules require a concise docstring; we also need typing for the factory’s return type.
Apply this diff:
@@ -from pydantic import Field +"""Redis object store config and factory for nat plugins.""" + +from typing import AsyncIterator + +from pydantic import Field @@ -from nat.data_models.object_store import ObjectStoreBaseConfig +from nat.data_models.object_store import ObjectStoreBaseConfig +from nat.object_store.interfaces import ObjectStorepackages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (3)
113-137: Annotate return type for put_object.Interface returns None on success; add annotation to match.
Apply this diff:
- async def put_object(self, key: str, item: ObjectStoreItem): + async def put_object(self, key: str, item: ObjectStoreItem) -> None:
139-164: Annotate return type for upsert_object.Apply this diff:
- async def upsert_object(self, key: str, item: ObjectStoreItem): + async def upsert_object(self, key: str, item: ObjectStoreItem) -> None:
188-213: Annotate return type for delete_object.Apply this diff:
- async def delete_object(self, key: str): + async def delete_object(self, key: str) -> None:packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py (5)
29-35: Docstring phrasing: stored JSON, not “binary value.”You are storing JSON produced by
model_dump_json(). Adjust wording for accuracy.Apply this diff:
@@ - Each object is stored as a single binary value at key "nat:object_store:bucket:{bucket_name}:{object_key}". + Each object is stored as a single JSON-encoded value at key + "nat:object_store:bucket:{bucket_name}:{object_key}".
80-96: Annotate return type for put_object.Match interface’s None-returning contract.
Apply this diff:
- async def put_object(self, key: str, item: ObjectStoreItem): + async def put_object(self, key: str, item: ObjectStoreItem) -> None:
94-103: Annotate return type for upsert_object.Apply this diff:
- async def upsert_object(self, key: str, item: ObjectStoreItem): + async def upsert_object(self, key: str, item: ObjectStoreItem) -> None:
117-128: Annotate return type for delete_object.Apply this diff:
- async def delete_object(self, key: str): + async def delete_object(self, key: str) -> None:
16-26: Add a brief module docstring and types import.Keep modules documented and add TracebackType import to support the annotations above.
Apply this diff:
@@ -import logging +"""Redis-backed ObjectStore implementation for nat.""" +import logging @@ -import redis.asyncio as redis +import redis.asyncio as redis +from types import TracebackTypedocs/source/extend/object-store.md (1)
18-21: Fix first-use product naming per style guide.First body use should be “NVIDIA NeMo Agent toolkit”; keep headings as “NeMo Agent Toolkit”.
Apply this diff:
-This documentation presumes familiarity with the NeMo Agent toolkit plugin architecture, the concept of "function registration" using `@register_function`, and how we define tool/workflow configurations in the NeMo Agent toolkit config described in the [Creating a New Tool and Workflow](../tutorials/create-a-new-workflow.md) tutorial. +This documentation presumes familiarity with the NVIDIA NeMo Agent toolkit plugin architecture, the concept of "function registration" using `@register_function`, and how we define tool/workflow configurations in the NeMo Agent toolkit config described in the [Creating a New Tool and Workflow](../tutorials/create-a-new-workflow.md) tutorial.src/nat/cli/commands/object_store/object_store.py (6)
131-136: Normalize keys to POSIX-style paths for cross-platform consistency.On Windows, str(relative_path) uses backslashes, which can fragment keys across platforms. Use as_posix().
- key = str(file_path.relative_to(local_dir)) + key = file_path.relative_to(local_dir).as_posix()
83-91: Decorator should preserve metadata and add type hints.Use functools.wraps to preserve the original function’s metadata (useful for Click/help) and annotate types. Also, it’s safe to ensure a context object here as an extra guard.
-def object_store_command_decorator(async_func): +from collections.abc import Awaitable, Callable +from functools import wraps + +def object_store_command_decorator(async_func: Callable[..., Awaitable[int]]) -> Callable[..., int]: @@ - @click.pass_context - def wrapper(ctx: click.Context, **kwargs): + @wraps(async_func) + @click.pass_context + def wrapper(ctx: click.Context, **kwargs) -> int: - config = ctx.obj["store_config"] + ctx.ensure_object(dict) + config = ctx.obj["store_config"]
112-126: Add return type hints and align docstrings with Google style.These are public CLI entrypoints; per guidelines, add return annotations and ensure “Args:” blocks match parameter order.
-async def upload_command(store: ObjectStore, local_dir: Path, **_kwargs): +async def upload_command(store: ObjectStore, local_dir: Path, **_kwargs) -> int:
145-157: Broaden type for keys and add return type.Click passes a tuple for variadic arguments. Using Sequence[str] better reflects accepted inputs and keeps static typing happy.
-from nat.object_store.interfaces import ObjectStore +from collections.abc import Sequence +from nat.object_store.interfaces import ObjectStore @@ -async def delete_command(store: ObjectStore, keys: list[str], **_kwargs): +async def delete_command(store: ObjectStore, keys: Sequence[str], **_kwargs) -> int:
31-41: Consider validating STORE_CONFIGS lookups for better error messages.KeyErrors from unknown store_type will be cryptic. Wrap lookup with a clear ClickException message.
- config = STORE_CONFIGS[store_type] + try: + config = STORE_CONFIGS[store_type] + except KeyError as e: + raise click.ClickException(f"Unknown object store type: {store_type}") from e
213-221: Optional: log missing providers instead of silent pass.Swallowing ImportError makes it hard to discover why a provider is absent. Consider logging at debug/info.
- except ImportError: - pass + except ImportError as e: + logger.debug("Object store provider '%s' not available: %s", store_type, e)examples/object_store/user_report/configs/config_redis.yml (2)
20-21: Avoid wildcard CORS in production.allow_origins: ['*'] is fine for examples, but should be tightened in real deployments to specific origins.
If this example is used beyond local dev, consider parameterizing CORS via environment vars and documenting safe values.
69-79: Nit: minor copy edits for function descriptions.E.g., “Deletes user diagnostic report from the bucket” → “Deletes a user diagnostic report from the bucket.”
Suggested wording improvements (no functional impact):
- “Deletes a user diagnostic report from the bucket...”
- “Do not run delete_user_report without the explicit intention to delete an existing report...”
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock📒 Files selected for processing (24)
docs/source/extend/object-store.md(4 hunks)docs/source/store-and-retrieve/object-store.md(3 hunks)examples/deploy/README.md(2 hunks)examples/deploy/docker-compose.minio.yml(1 hunks)examples/deploy/docker-compose.mysql.yml(1 hunks)examples/memory/redis/configs/config.yml(1 hunks)examples/object_store/user_report/README.md(5 hunks)examples/object_store/user_report/configs/config_mysql.yml(1 hunks)examples/object_store/user_report/configs/config_redis.yml(1 hunks)examples/object_store/user_report/pyproject.toml(1 hunks)examples/object_store/user_report/upload_to_minio.sh(0 hunks)examples/object_store/user_report/upload_to_mysql.sh(0 hunks)packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py(5 hunks)packages/nvidia_nat_mysql/src/nat/plugins/mysql/object_store.py(1 hunks)packages/nvidia_nat_redis/src/nat/plugins/redis/memory.py(1 hunks)packages/nvidia_nat_redis/src/nat/plugins/redis/object_store.py(1 hunks)packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py(1 hunks)packages/nvidia_nat_redis/src/nat/plugins/redis/register.py(1 hunks)packages/nvidia_nat_redis/tests/test_redis_object_store.py(1 hunks)packages/nvidia_nat_s3/src/nat/plugins/s3/object_store.py(1 hunks)packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py(4 hunks)src/nat/cli/commands/object_store/__init__.py(1 hunks)src/nat/cli/commands/object_store/object_store.py(1 hunks)src/nat/cli/entrypoint.py(2 hunks)💤 Files with no reviewable changes (2)
- examples/object_store/user_report/upload_to_minio.sh
- examples/object_store/user_report/upload_to_mysql.sh
🧰 Additional context used
📓 Path-based instructions (4)
**/*.py
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
**/*.py: In code comments, use abbreviations: NAT (informal/comments), nat (API namespace/CLI), nvidia-nat (package). Do not use these abbreviations in documentation.
Run isort first with line_length=120, multi_line_output=3, include_trailing_comma=true, force_single_line=true.
Run yapf second with PEP8 base and column_limit=120.
Respect Python naming: snake_case for functions and variables, PascalCase for classes, UPPER_CASE for constants.
Fix flake8 (pflake8) warnings unless explicitly ignored in pyproject.toml.
All public Python APIs require type hints (Python 3.11+) on parameters and return values.
Prefer typing/collections.abc abstractions (e.g., Sequence over list).
Use typing.Annotated to express units or extra metadata when useful.
Treat pyright warnings as errors during development (configured via pyproject.toml).
Provide Google‑style docstrings for every public module, class, function, and CLI command.
Docstring first line must be a concise description ending with a period.
Do not hard‑code version numbers in Python code; versions are derived by setuptools‑scm.
Prefer httpx with SSL verification enabled by default for HTTP calls; follow OWASP Top‑10.
Use async/await for I/O‑bound work (HTTP, DB, file reads).
Cache expensive computations with functools.lru_cache or an external cache when appropriate.
Leverage NumPy vectorized operations when beneficial and feasible.Files:
packages/nvidia_nat_s3/src/nat/plugins/s3/object_store.pysrc/nat/cli/commands/object_store/__init__.pypackages/nvidia_nat_redis/src/nat/plugins/redis/object_store.pysrc/nat/cli/commands/object_store/object_store.pypackages/nvidia_nat_redis/tests/test_redis_object_store.pysrc/nat/cli/entrypoint.pypackages/nvidia_nat_mysql/src/nat/plugins/mysql/object_store.pypackages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.pypackages/nvidia_nat_redis/src/nat/plugins/redis/register.pypackages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.pypackages/nvidia_nat_redis/src/nat/plugins/redis/memory.pypackages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py{src/**/*.py,packages/*/src/**/*.py}
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
All importable Python code must live under src/ or packages//src/.
Files:
packages/nvidia_nat_s3/src/nat/plugins/s3/object_store.pysrc/nat/cli/commands/object_store/__init__.pypackages/nvidia_nat_redis/src/nat/plugins/redis/object_store.pysrc/nat/cli/commands/object_store/object_store.pysrc/nat/cli/entrypoint.pypackages/nvidia_nat_mysql/src/nat/plugins/mysql/object_store.pypackages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.pypackages/nvidia_nat_redis/src/nat/plugins/redis/register.pypackages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.pypackages/nvidia_nat_redis/src/nat/plugins/redis/memory.pypackages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py**/*
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
**/*: Every file must start with the standard SPDX Apache‑2.0 header.
All source files must include the SPDX Apache‑2.0 header template; CI verifies via ci/scripts/github/checks.sh.Files:
packages/nvidia_nat_s3/src/nat/plugins/s3/object_store.pyexamples/memory/redis/configs/config.ymlexamples/deploy/docker-compose.mysql.ymlsrc/nat/cli/commands/object_store/__init__.pypackages/nvidia_nat_redis/src/nat/plugins/redis/object_store.pysrc/nat/cli/commands/object_store/object_store.pyexamples/object_store/user_report/configs/config_mysql.ymldocs/source/store-and-retrieve/object-store.mdexamples/object_store/user_report/pyproject.tomlexamples/deploy/docker-compose.minio.ymlpackages/nvidia_nat_redis/tests/test_redis_object_store.pysrc/nat/cli/entrypoint.pypackages/nvidia_nat_mysql/src/nat/plugins/mysql/object_store.pypackages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.pypackages/nvidia_nat_redis/src/nat/plugins/redis/register.pypackages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.pypackages/nvidia_nat_redis/src/nat/plugins/redis/memory.pyexamples/deploy/README.mdexamples/object_store/user_report/README.mdexamples/object_store/user_report/configs/config_redis.ymlpackages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.pydocs/source/extend/object-store.mddocs/source/**/*.md
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
docs/source/**/*.md: Use the exact product naming in documentation: first use “NVIDIA NeMo Agent toolkit”; subsequent references “NeMo Agent toolkit”; in headings use “NeMo Agent Toolkit” (capital T).
Never use deprecated names (Agent Intelligence toolkit, aiqtoolkit, AgentIQ, AIQ/aiq) in documentation unless explicitly referring to deprecated names or compatibility layers.
Use the intended name “AIQ Blueprint” and do not change it in docs.
Documentation sources must be Markdown files under docs/source.
Surround code entities with backticks in documentation to avoid Vale false positives.
Keep documentation in sync with code; Sphinx build must be error‑free with no broken links.
Do not hard‑code project version numbers in documentation; versions are derived by setuptools‑scm.Files:
docs/source/store-and-retrieve/object-store.mddocs/source/extend/object-store.md🧬 Code graph analysis (8)
packages/nvidia_nat_s3/src/nat/plugins/s3/object_store.py (1)
packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py (1)
S3ObjectStore(30-170)packages/nvidia_nat_redis/src/nat/plugins/redis/object_store.py (1)
packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py (1)
RedisObjectStore(29-127)src/nat/cli/commands/object_store/object_store.py (3)
examples/object_store/user_report/tests/test_user_report_tool.py (2)
builder(33-49)object_store(53-55)src/nat/object_store/models.py (1)
ObjectStoreItem(21-38)packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py (2)
upsert_object(95-102)delete_object(118-127)packages/nvidia_nat_redis/tests/test_redis_object_store.py (2)
examples/object_store/user_report/tests/test_user_report_tool.py (2)
builder(33-49)object_store(53-55)packages/nvidia_nat_redis/src/nat/plugins/redis/object_store.py (1)
RedisObjectStoreClientConfig(23-31)src/nat/cli/entrypoint.py (1)
src/nat/cli/commands/object_store/object_store.py (1)
object_store_command(174-176)packages/nvidia_nat_mysql/src/nat/plugins/mysql/object_store.py (1)
packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (1)
MySQLObjectStore(30-212)packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py (2)
src/nat/object_store/models.py (1)
ObjectStoreItem(21-38)packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (2)
put_object(114-136)get_object(166-185)packages/nvidia_nat_redis/src/nat/plugins/redis/register.py (1)
examples/object_store/user_report/tests/test_user_report_tool.py (1)
object_store(53-55)🪛 markdownlint-cli2 (0.17.2)
docs/source/store-and-retrieve/object-store.md
182-182: Dollar signs used before commands without showing output
(MD014, commands-show-output)
186-186: Dollar signs used before commands without showing output
(MD014, commands-show-output)
190-190: Dollar signs used before commands without showing output
(MD014, commands-show-output)
🪛 LanguageTool
examples/deploy/README.md
[grammar] ~37-~37: There might be a mistake here.
Context: ...cludesdocker-compose.phoenix.ymlfor running Phoenix observability server to monitor...(QB_NEW_EN)
[grammar] ~37-~37: There might be a mistake here.
Context: ...y server to monitor and debug workflows. - Redis Service: Includes `docker-compos...(QB_NEW_EN)
[grammar] ~43-~43: There might be a mistake here.
Context: ... ## Available Services -minio:docker-compose.minio.yml-mysql:docker-compose.mysql.yml- **`phoeni...(QB_NEW_EN)
[grammar] ~44-~44: There might be a mistake here.
Context: ...ocker-compose.minio.yml- **mysql**:docker-compose.mysql.yml- **phoenix**:docker-compose.phoenix.yml- **redi...(QB_NEW_EN)
examples/object_store/user_report/README.md
[grammar] ~28-~28: There might be a mistake here.
Context: ...up-api-keys) - Choose an Object Store - Setting up MinIO ...(QB_NEW_EN)
[grammar] ~29-~29: There might be a mistake here.
Context: ...an-object-store) - Setting up MinIO - Setting up MySQL ...(QB_NEW_EN)
[grammar] ~30-~30: There might be a mistake here.
Context: ...etting-up-minio) - Setting up MySQL - Setting up Redis -...(QB_NEW_EN)
[grammar] ~31-~31: There might be a mistake here.
Context: ...etting-up-mysql) - Setting up Redis - Loading Mock Data -...(QB_NEW_EN)
[grammar] ~32-~32: There might be a mistake here.
Context: ...setting-up-redis) - Loading Mock Data - [NeMo Agent Toolkit File Server](#nemo-ag...(QB_NEW_EN)
[style] ~73-~73: Consider replacing this word to strengthen your wording.
Context: ...hine. MinIO is an object storage system and acts as drop-in replacement for AWS S3....(AND_THAT)
[grammar] ~73-~73: There might be a mistake here.
Context: ...IO is an object storage system and acts as drop-in replacement for AWS S3. You ca...(QB_NEW_EN)
[grammar] ~81-~81: There might be a mistake here.
Context: ...r-compose.minio.yml up -d ``` > [!NOTE] > This is not a secure configuration and...(QB_NEW_EN)
[grammar] ~100-~100: There might be a mistake here.
Context: ...r-compose.mysql.yml up -d ``` > [!NOTE] > This is not a secure configuration and...(QB_NEW_EN)
[grammar] ~111-~111: There might be a mistake here.
Context: ...r-compose.redis.yml up -d ``` > [!NOTE] > This is not a secure configuration and...(QB_NEW_EN)
[grammar] ~165-~165: There might be a mistake here.
Context: ...erver (Optional) - Download an object:curl -X GET http://<hostname>:<port>/static/{file_path} -o {filename}- Upload an object: `curl -X POST http://<...(QB_NEW_EN)
[grammar] ~166-~166: There might be a mistake here.
Context: ...ath} -o {filename}- Upload an object:curl -X POST http://:/static/{file_path} -d @{filename}- Upsert an object:curl -X PUT http://<h...(QB_NEW_EN)
[grammar] ~167-~167: There might be a mistake here.
Context: ...th} -d @{filename}- Upsert an object:curl -X PUT http://:/static/{file_path} -d @{filename}- Delete an object:curl -X DELETE http:/...(QB_NEW_EN)
[grammar] ~179-~179: There might be a mistake here.
Context: ... three options for running the workflow: 1. Using the S3-compatible object store (`c...(QB_NEW_EN)
[grammar] ~180-~180: There might be a mistake here.
Context: ...ompatible object store (config_s3.yml) 2. Using the MySQL object store (`config_my...(QB_NEW_EN)
[grammar] ~181-~181: There might be a mistake here.
Context: ... MySQL object store (config_mysql.yml) 3. Using the Redis object store (`config_re...(QB_NEW_EN)
[grammar] ~184-~184: There might be a mistake here.
Context: ...hich uses an S3-compatible object store. You can change the configuration file by...(QB_NEW_EN)
[grammar] ~185-~185: There might be a mistake here.
Context: ...config_mysql.ymlfor the MySQL server orconfig_redis.ymlfor the Redis serv...(QB_NEW_EN)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: CI Pipeline / Check
🔇 Additional comments (15)
examples/memory/redis/configs/config.yml (1)
42-43: LGTM: numeric Redis db/port align with updated config types.Changing
dbandportto integers matches the Redis config model updates and prevents accidental string-to-int coercion issues.examples/deploy/docker-compose.mysql.yml (1)
18-18: Remove invalid tag concernThe Docker Hub manifest list for
mysql:9.3confirms that the tag exists and supports Linux on both AMD64 and ARM64 platforms. The earlier warning about an invalid or unavailable tag is therefore not applicable; no change to the image reference is required.Likely an incorrect or invalid review comment.
examples/object_store/user_report/README.md (1)
179-187: Nice clarity on backend options.This list and the follow-up guidance make backend switching explicit. Looks good.
packages/nvidia_nat_redis/src/nat/plugins/redis/memory.py (2)
24-29: Config field type tightening is correct and consistent with CLI/registry.Switching to concrete types for host/port/db/key_prefix improves validation and aligns with the breaking-change note. Good change.
36-39: Lazy imports reduce optional-dependency pressure.Deferring
redis.asyncioandLLMFrameworkEnumimport avoids import-time failures and circulars. Good practice here.packages/nvidia_nat_redis/src/nat/plugins/redis/register.py (1)
22-24: Redis object-store registration and lazy imports verified ✅
- Confirmed the
@register_object_store(config_type=RedisObjectStoreClientConfig)decorator is present inpackages/nvidia_nat_redis/src/nat/plugins/redis/object_store.py(line 34), ensuring the provider registers at import time.- Verified that
object_store.pyonly performs lightweight imports at the top level and defers any Redis client imports (import redis.asyncio) to runtime within theredis_object_store_clientfunction, keeping module import cheap.No further changes required—LGTM.
examples/object_store/user_report/pyproject.toml (1)
12-16: Extras Declaration Verified – Please Confirm Packaging PublicationI’ve confirmed that the root
pyproject.tomlincludes the following under[project.optional-dependencies](lines 72–74):
mysql = ["nvidia-nat-mysql"]redis = ["nvidia-nat-redis"]s3 = ["nvidia-nat-s3"]These map exactly to the example’s
nvidia-nat[mysql],nvidia-nat[redis], andnvidia-nat[s3]extras.• Please ensure that your CI/build pipeline publishes the corresponding
nvidia-nat-mysql,nvidia-nat-redis, andnvidia-nat-s3distributions from the monorepo (so that users installing these extras will receive the correct packages).docs/source/store-and-retrieve/object-store.md (2)
74-74: Redis provider addition reads clearly.Good addition and path reference; consistent with other providers.
206-207: Link target and toctree entry validatedThe file
docs/source/extend/object-store.mdis present and already referenced in the toctree:
- File exists at
docs/source/extend/object-store.md- Included in the toctree in
docs/source/index.md:117The link to
../extend/object-store.mdindocs/source/store-and-retrieve/object-store.mdis therefore valid, and no Sphinx build breakages will occur.src/nat/cli/entrypoint.py (2)
111-111: Wiring the new object-store command into the CLI looks good.Nice, this exposes nat object-store … subcommands alongside the existing groups.
36-36: object_store_command is properly decorated as a Click groupVerified that
object_store_commandinsrc/nat/cli/commands/object_store/object_store.pyis decorated with@click.group, fulfilling the requirement for use withcli.add_command. No changes are needed.• Location:
src/nat/cli/commands/object_store/object_store.py, lines 173–174@click.group(name="object-store", invoke_without_command=False, help="Manage object store operations.") def object_store_command(**_kwargs): """Manage object store operations including uploading files and directories.""" passpackages/nvidia_nat_s3/src/nat/plugins/s3/object_store.py (1)
48-49: LGTM on context-managed lifecycle.Using async context manager to guarantee connect/teardown is consistent with other stores.
packages/nvidia_nat_mysql/src/nat/plugins/mysql/object_store.py (1)
65-66: LGTM on async context management.Consistent with the MySQLObjectStore implementation lifecycle.
packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (1)
148-155: Confirm server compatibility for row aliasing in ON DUPLICATE KEY UPDATE.The
VALUES (... ) AS new ... UPDATE size=new.sizepattern requires a server that supports row aliasing in INSERT. Please verify your minimum MySQL version in CI/prod and adjust if needed.src/nat/cli/commands/object_store/object_store.py (1)
31-41: All provider module paths and config classes verified
- Confirmed
nat.plugins.s3.object_storeexists and definesS3ObjectStoreClientConfiginpackages/nvidia_nat_s3/src/nat/plugins/s3/object_store.py:26.- Confirmed
nat.plugins.mysql.object_storeexists and definesMySQLObjectStoreClientConfiginpackages/nvidia_nat_mysql/src/nat/plugins/mysql/object_store.py:26.- Confirmed
nat.plugins.redis.object_storeexists and definesRedisObjectStoreClientConfiginpackages/nvidia_nat_redis/src/nat/plugins/redis/object_store.py:23.No mismatches detected—no changes required.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
docs/source/store-and-retrieve/object-store.md (1)
145-149: Fix Python snippet: metadata is a set, not a dict.
{"author", "user123"}creates a set; should be a dict{"author": "user123"}.- await object_store.upsert_object("greeting.txt", ObjectStoreItem( - data=b"Goodbye, World!", - content_type="text/plain", - metadata={"author", "user123"} - )) + await object_store.upsert_object("greeting.txt", ObjectStoreItem( + data=b"Goodbye, World!", + content_type="text/plain", + metadata={"author": "user123"} + ))examples/object_store/user_report/README.md (2)
18-21: Apply product naming guide in heading and fix opening sentence grammar.
- Headings must use “NeMo Agent Toolkit” (capital T) without NVIDIA prefix.
- First body mention should use “NVIDIA NeMo Agent toolkit”.
- Fix “And example” → “An example”.
-# Report Tool for NVIDIA NeMo Agent Toolkit +# Report Tool for NeMo Agent Toolkit @@ -And example tool in the NeMo Agent toolkit that makes use of an Object Store to retrieve data. +An example tool in the NVIDIA NeMo Agent toolkit that makes use of an Object Store to retrieve data.
238-243: Fix user ID typo in expected output (“678901” → “6789”).This mismatches the earlier request and can confuse readers.
-User report for 678901 with date latest added successfully +User report for 6789 with date latest added successfully
♻️ Duplicate comments (5)
docs/source/store-and-retrieve/object-store.md (2)
179-191: Clarify POST conflict behavior and fix markdownlint MD014 ($ prompts).
- Explicitly note that POST returns 409 if the object already exists.
- Remove the “$ ” prompts in fenced blocks to satisfy MD014.
-This enables HTTP endpoints for object store operations: -- **PUT** `/static/{file_path}` - Create or replace an object at the given path (upsert) - ```console - $ curl -X PUT --data-binary @data.txt http://localhost:9000/static/folder/data.txt - ``` +This enables HTTP endpoints for object store operations: +- **PUT** `/static/{file_path}` - Create or replace an object at the given path (upsert) + ```console + curl -X PUT --data-binary @data.txt http://localhost:9000/static/folder/data.txt + ``` ... -- **POST** `/static/{file_path}` - Upload a new object - ```console - $ curl -X POST --data-binary @data_new.txt http://localhost:9000/static/folder/data.txt - ``` +- **POST** `/static/{file_path}` - Create a new object only; returns 409 if the object already exists + ```console + curl -X POST --data-binary @data_new.txt http://localhost:9000/static/folder/data.txt + ```
112-121: Add required Redis key_prefix in config example (breaking change).Per PR notes, Redis requires host, port, db, and key_prefix (db/port as ints). The example is missing key_prefix; add it to prevent copy/paste failures.
object_stores: my_object_store: _type: redis host: localhost port: 6379 db: 0 + key_prefix: my-object-store bucket_name: my-bucketpackages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py (2)
35-41: Credentials handling implementation looks good.The implementation now properly handles optional credentials by conditionally building the client arguments dictionary only when credentials are provided. This allows the AWS default credential chain to work as expected.
50-57: Flexible credential handling implemented correctly.The conditional building of
_client_argsproperly supports both explicit credentials and AWS default credential chain scenarios. Only non-None values are added to the client arguments, allowing aioboto3 to handle credential resolution appropriately.packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (1)
17-17: Good fix: added regex import for identifier validation.This enables local validation without relying on DB errors.
🧹 Nitpick comments (6)
docs/source/store-and-retrieve/object-store.md (1)
18-18: Fix product naming to follow style guide (heading + first mention).
- Headings must use “NeMo Agent Toolkit” (capital T) without the NVIDIA prefix.
- First body mention should be “NVIDIA NeMo Agent toolkit”; subsequent mentions “NeMo Agent toolkit”.
- Minor grammar tweak in the second paragraph.
-# Object Store for NVIDIA NeMo Agent Toolkit +# Object Store for NeMo Agent Toolkit-The NeMo Agent toolkit Object Store subsystem provides a standardized interface for storing and retrieving binary data with associated metadata. This is particularly useful for building applications that need to manage files, documents, images, or any other binary content within these workflows. - -The object store module is extensible, which allows developers to create custom object store backends. The providers in NeMo Agent toolkit terminology supports different storage systems. +The NVIDIA NeMo Agent toolkit Object Store subsystem provides a standardized interface for storing and retrieving binary data with associated metadata. This is particularly useful for building applications that need to manage files, documents, images, or any other binary content within these workflows. + +The object store module is extensible, allowing developers to create custom object store backends. In NeMo Agent toolkit terminology, providers support different storage systems.Also applies to: 20-23
examples/object_store/user_report/README.md (1)
165-168: Add Content-Type to curl examples to avoid defaults.
--data-binarypreserves payloads, but cURL may default Content-Type. Recommend showing a header to reduce confusion; adjust type per payload (json/octet-stream).-- Upload an object: `curl -X POST http://<hostname>:<port>/static/{file_path} --data-binary @{filename}` -- Upsert an object: `curl -X PUT http://<hostname>:<port>/static/{file_path} --data-binary @{filename}` +- Upload an object: `curl -X POST http://<hostname>:<port>/static/{file_path} --data-binary @{filename} -H 'Content-Type: application/json'` +- Upsert an object: `curl -X PUT http://<hostname>:<port>/static/{file_path} --data-binary @{filename} -H 'Content-Type: application/json'`packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (4)
50-53: Quoting strategy: consider returning unquoted schema and quoting at call-sites.Storing backticks in the _schema property works, but it couples quoting to the property. Returning the bare name and applying backticks where used or via a small quote_identifier helper improves flexibility (e.g., for
{schema}.tablecompositions).
54-55: Prefer typing.Self for aenter return type.Forward-ref is fine; using Self reads cleaner and aligns with Python 3.11+ typing.
Apply this diff (and ensure imports are isort/yapf-compliant):
+from typing import Self @@ - async def __aenter__(self) -> "MySQLObjectStore": + async def __aenter__(self) -> Self:
134-134: Tighten user-facing error messages: say “bucket” (schema) rather than “table.”These messages mention “MySQL table,” but the bucket maps to a schema whose tables are implementation detail. Improves clarity for callers.
Apply this diff:
- key=key, additional_message=f"MySQL table {self._bucket_name} already has key {key}") + key=key, additional_message=f"MySQL bucket '{self._bucket_name}' already has key '{key}'") @@ - raise NoSuchKeyError(key=key, - additional_message=f"MySQL table {self._bucket_name} does not have key {key}") + raise NoSuchKeyError( + key=key, + additional_message=f"MySQL bucket '{self._bucket_name}' does not have key '{key}'", + ) @@ - key=key, additional_message=f"MySQL table {self._bucket_name} does not have key {key}") + key=key, additional_message=f"MySQL bucket '{self._bucket_name}' does not have key '{key}'")Also applies to: 189-190, 213-213
90-97: Optional: consider JSON or LONGTEXT for serialized items instead of LONGBLOB.You’re storing JSON text (
model_dump_json()) in a LONGBLOB. Using JSON (MySQL 5.7+) or LONGTEXT aligns type with content and can simplify tooling/inspection. Not a blocker.Would JSON be compatible with how S3/Redis providers represent metadata, or do you intentionally want opaque bytes across stores?
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (7)
docs/source/store-and-retrieve/object-store.md(3 hunks)examples/object_store/user_report/README.md(5 hunks)packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py(6 hunks)packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py(1 hunks)packages/nvidia_nat_redis/tests/test_redis_object_store.py(1 hunks)packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py(5 hunks)src/nat/cli/commands/object_store/object_store.py(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
- packages/nvidia_nat_redis/tests/test_redis_object_store.py
- src/nat/cli/commands/object_store/object_store.py
- packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py
🧰 Additional context used
📓 Path-based instructions (4)
**/*
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
**/*: Every file must start with the standard SPDX Apache‑2.0 header.
All source files must include the SPDX Apache‑2.0 header template; CI verifies via ci/scripts/github/checks.sh.
Files:
examples/object_store/user_report/README.mdpackages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.pypackages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.pydocs/source/store-and-retrieve/object-store.md
**/*.py
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
**/*.py: In code comments, use abbreviations: NAT (informal/comments), nat (API namespace/CLI), nvidia-nat (package). Do not use these abbreviations in documentation.
Run isort first with line_length=120, multi_line_output=3, include_trailing_comma=true, force_single_line=true.
Run yapf second with PEP8 base and column_limit=120.
Respect Python naming: snake_case for functions and variables, PascalCase for classes, UPPER_CASE for constants.
Fix flake8 (pflake8) warnings unless explicitly ignored in pyproject.toml.
All public Python APIs require type hints (Python 3.11+) on parameters and return values.
Prefer typing/collections.abc abstractions (e.g., Sequence over list).
Use typing.Annotated to express units or extra metadata when useful.
Treat pyright warnings as errors during development (configured via pyproject.toml).
Provide Google‑style docstrings for every public module, class, function, and CLI command.
Docstring first line must be a concise description ending with a period.
Do not hard‑code version numbers in Python code; versions are derived by setuptools‑scm.
Prefer httpx with SSL verification enabled by default for HTTP calls; follow OWASP Top‑10.
Use async/await for I/O‑bound work (HTTP, DB, file reads).
Cache expensive computations with functools.lru_cache or an external cache when appropriate.
Leverage NumPy vectorized operations when beneficial and feasible.
Files:
packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.pypackages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py
{src/**/*.py,packages/*/src/**/*.py}
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
All importable Python code must live under src/ or packages//src/.
Files:
packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.pypackages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py
docs/source/**/*.md
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
docs/source/**/*.md: Use the exact product naming in documentation: first use “NVIDIA NeMo Agent toolkit”; subsequent references “NeMo Agent toolkit”; in headings use “NeMo Agent Toolkit” (capital T).
Never use deprecated names (Agent Intelligence toolkit, aiqtoolkit, AgentIQ, AIQ/aiq) in documentation unless explicitly referring to deprecated names or compatibility layers.
Use the intended name “AIQ Blueprint” and do not change it in docs.
Documentation sources must be Markdown files under docs/source.
Surround code entities with backticks in documentation to avoid Vale false positives.
Keep documentation in sync with code; Sphinx build must be error‑free with no broken links.
Do not hard‑code project version numbers in documentation; versions are derived by setuptools‑scm.
Files:
docs/source/store-and-retrieve/object-store.md
🪛 LanguageTool
examples/object_store/user_report/README.md
[grammar] ~28-~28: There might be a mistake here.
Context: ...up-api-keys) - Choose an Object Store - Setting up MinIO ...
(QB_NEW_EN)
[grammar] ~29-~29: There might be a mistake here.
Context: ...an-object-store) - Setting up MinIO - Setting up MySQL ...
(QB_NEW_EN)
[grammar] ~30-~30: There might be a mistake here.
Context: ...etting-up-minio) - Setting up MySQL - Setting up Redis -...
(QB_NEW_EN)
[grammar] ~31-~31: There might be a mistake here.
Context: ...etting-up-mysql) - Setting up Redis - Loading Mock Data -...
(QB_NEW_EN)
[grammar] ~32-~32: There might be a mistake here.
Context: ...setting-up-redis) - Loading Mock Data - [NeMo Agent Toolkit File Server](#nemo-ag...
(QB_NEW_EN)
[style] ~73-~73: Consider replacing this word to strengthen your wording.
Context: ...hine. MinIO is an object storage system and acts as drop-in replacement for AWS S3....
(AND_THAT)
[grammar] ~73-~73: There might be a mistake here.
Context: ...IO is an object storage system and acts as drop-in replacement for AWS S3. You ca...
(QB_NEW_EN)
[grammar] ~81-~81: There might be a mistake here.
Context: ...r-compose.minio.yml up -d ``` > [!NOTE] > This is not a secure configuration and...
(QB_NEW_EN)
[grammar] ~100-~100: There might be a mistake here.
Context: ...r-compose.mysql.yml up -d ``` > [!NOTE] > This is not a secure configuration and...
(QB_NEW_EN)
[grammar] ~111-~111: There might be a mistake here.
Context: ...r-compose.redis.yml up -d ``` > [!NOTE] > This is not a secure configuration and...
(QB_NEW_EN)
[grammar] ~165-~165: There might be a mistake here.
Context: ...erver (Optional) - Download an object: curl -X GET http://<hostname>:<port>/static/{file_path} -o {filename} - Upload an object: `curl -X POST http://<...
(QB_NEW_EN)
[grammar] ~166-~166: There might be a mistake here.
Context: ...ath} -o {filename}- Upload an object:curl -X POST http://:/static/{file_path} --data-binary @{filename}- Upsert an object:curl -X PUT http://<h...
(QB_NEW_EN)
[grammar] ~167-~167: There might be a mistake here.
Context: ...binary @{filename}- Upsert an object:curl -X PUT http://:/static/{file_path} --data-binary @{filename}- Delete an object:curl -X DELETE http:/...
(QB_NEW_EN)
[grammar] ~179-~179: There might be a mistake here.
Context: ... three options for running the workflow: 1. Using the S3-compatible object store (`c...
(QB_NEW_EN)
[grammar] ~180-~180: There might be a mistake here.
Context: ...ompatible object store (config_s3.yml) 2. Using the MySQL object store (`config_my...
(QB_NEW_EN)
[grammar] ~181-~181: There might be a mistake here.
Context: ... MySQL object store (config_mysql.yml) 3. Using the Redis object store (`config_re...
(QB_NEW_EN)
[grammar] ~184-~184: There might be a mistake here.
Context: ...hich uses an S3-compatible object store. You can change the configuration file by...
(QB_NEW_EN)
[grammar] ~185-~185: There might be a mistake here.
Context: ... config_mysql.yml for the MySQL server or config_redis.yml for the Redis serv...
(QB_NEW_EN)
🪛 markdownlint-cli2 (0.17.2)
docs/source/store-and-retrieve/object-store.md
182-182: Dollar signs used before commands without showing output
(MD014, commands-show-output)
186-186: Dollar signs used before commands without showing output
(MD014, commands-show-output)
190-190: Dollar signs used before commands without showing output
(MD014, commands-show-output)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: CI Pipeline / Check
🔇 Additional comments (8)
examples/object_store/user_report/README.md (1)
129-133: Consider adding required key prefix for Redis CLI upload (verify CLI arg name).The schema change makes key_prefix required for Redis. If the CLI does not infer it from bucket name, expose a flag (e.g.,
--key-prefix). Please verify the exact flag name innat object-store redisand update the example accordingly.nat object-store \ - redis --host 127.0.0.1 --port 6379 --db 0 my-bucket \ + redis --host 127.0.0.1 --port 6379 --db 0 --key-prefix user-report my-bucket \ upload ./examples/object_store/user_report/data/object_store/If the CLI already derives a safe prefix from the bucket, add a short note stating that and leave the command as-is.
packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py (5)
59-59: LGTM: Proper return type annotation.The
__aenter__method now correctly returns"S3ObjectStore"type, following proper async context manager patterns.
81-81: LGTM: Proper return type annotation for async context manager.The
__aexit__method is correctly annotated to returnNone, which is the expected type for async context manager exit methods.
114-119: LGTM: Proper error handling with chained exceptions.The error handling correctly raises
KeyAlreadyExistsErrorwith proper exception chaining usingfrom e. The additional message provides helpful context about the S3 bucket and key.
149-150: LGTM: Consistent error handling pattern.The
NoSuchKeyErroris properly raised with exception chaining, maintaining consistency with the error handling pattern used input_object.
160-161: LGTM: Consistent error handling in delete operation.The error handling follows the same pattern as other methods, properly chaining exceptions and providing context.
packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (2)
73-99: Nice: suppressing sql_notes during idempotent DDL reduces noisy warnings.The try/finally that restores the setting is correct and safe for pooled connections.
158-160: Fix invalid INSERT … ON DUPLICATE KEY UPDATE syntax and timestamp columnThe current code at packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (lines 158–160):
INSERT INTO object_meta (path, size) VALUES (%s, %s) AS new ON DUPLICATE KEY UPDATE size=new.size, created_at=CURRENT_TIMESTAMPhas two issues:
- MySQL does not permit a row alias immediately after
VALUES. The alias must be declared on the target table (and that syntax is only available in MySQL 8.0.20+).- Regardless of aliasing, the duplicate-key path should update
updated_at, notcreated_at, when a row already exists.Affected location:
- packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py: lines 158–160
Apply one of the following patches based on NAT’s minimum supported MySQL version:
- MySQL 8.0.20+ (preferred)—use table aliasing:
- INSERT INTO object_meta (path, size) - VALUES (%s, %s) AS new - ON DUPLICATE KEY UPDATE size=new.size, created_at=CURRENT_TIMESTAMP + INSERT INTO object_meta AS new (path, size) + VALUES (%s, %s) + ON DUPLICATE KEY UPDATE size=new.size, updated_at=CURRENT_TIMESTAMP
- Older MySQL—use the legacy
VALUES()function:- INSERT INTO object_meta (path, size) - VALUES (%s, %s) AS new - ON DUPLICATE KEY UPDATE size=new.size, created_at=CURRENT_TIMESTAMP + INSERT INTO object_meta (path, size) + VALUES (%s, %s) + ON DUPLICATE KEY UPDATE size=VALUES(size), updated_at=CURRENT_TIMESTAMPPlease confirm the minimum MySQL version supported by NAT so we can land the correct variant.
Signed-off-by: Will Killian <[email protected]>
a43238f to
dc1519b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (5)
examples/object_store/user_report/README.md (1)
240-243: Fix user_id typo in expected output (678901 → 6789).The tool response should match the requested user_id.
Apply this diff:
-User report for 678901 with date latest added successfully +User report for 6789 with date latest added successfullypackages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (3)
119-121: Add return type hints and short Google‑style docstrings for public methods.Guidelines require return annotations for all public APIs and docstrings.
Apply this diff:
@override -async def put_object(self, key: str, item: ObjectStoreItem): +async def put_object(self, key: str, item: ObjectStoreItem) -> None: + """Create a new object; fail if the key already exists.""" @@ @override -async def upsert_object(self, key: str, item: ObjectStoreItem): +async def upsert_object(self, key: str, item: ObjectStoreItem) -> None: + """Create or replace an object at the given key.""" @@ @override -async def delete_object(self, key: str): +async def delete_object(self, key: str) -> None: + """Delete the object at the given key; raise if the key does not exist."""Additionally, consider adding a brief module-level docstring at the top of the file describing the MySQL-backed object store (per docs requirement).
Also applies to: 145-145, 194-194
129-141: AvoidINSERT IGNORE; it can mask data issues. Detect duplicates explicitly viaIntegrityError.
INSERT IGNOREmay swallow non-duplicate errors (e.g., data truncation), leading to silent corruption. Use a plainINSERTand translate duplicate-key errors toKeyAlreadyExistsError.Apply this diff:
try: - await cur.execute("START TRANSACTION;") - await cur.execute("INSERT IGNORE INTO object_meta (path, size) VALUES (%s, %s)", - (key, len(item.data))) - if cur.rowcount == 0: - raise KeyAlreadyExistsError( - key=key, additional_message=f"MySQL table {self._bucket_name} already has key {key}") + await cur.execute("START TRANSACTION;") + await cur.execute("INSERT INTO object_meta (path, size) VALUES (%s, %s)", (key, len(item.data))) await cur.execute("SELECT id FROM object_meta WHERE path=%s FOR UPDATE;", (key, )) (obj_id, ) = await cur.fetchone() blob = item.model_dump_json() await cur.execute("INSERT INTO object_data (id, data) VALUES (%s, %s)", (obj_id, blob)) await conn.commit() - except Exception: + except aiomysql.IntegrityError as e: + await conn.rollback() + # 1062 = ER_DUP_ENTRY + if e.args and e.args[0] == 1062: + raise KeyAlreadyExistsError( + key=key, additional_message=f"MySQL table {self._bucket_name} already has key {key}" + ) from e + raise + except Exception: await conn.rollback() raise
203-214: Delete byobject_metaonly; current JOIN prevents cleanup ifobject_datarow is missing.If a partial write left
object_metawithoutobject_data, the current inner-join delete won’t match and will wrongly raiseNoSuchKeyError. Deleting fromobject_metasuffices; the FK cascade cleans upobject_datawhen present.Apply this diff:
- await cur.execute( - """ - DELETE m, d - FROM object_meta m - JOIN object_data d USING(id) - WHERE m.path=%s - """, (key, )) + await cur.execute( + """ + DELETE FROM object_meta + WHERE path=%s + """, + (key,), + )packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py (1)
72-78: Mandate: Explicitly set bucket region on creation and handle us-east-1 correctlyTo avoid cross-region errors (IllegalLocationConstraintException), you must pass a CreateBucketConfiguration when the client’s region isn’t us-east-1. Please update the logic in packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py (around lines 75–78) as follows:
• Detect the configured region from
self._client_args["region_name"].
• If it’s set and not"us-east-1", callcreate_bucketwithCreateBucketConfiguration={'LocationConstraint': region}.
• Otherwise (or for us-east-1), callcreate_bucketwithout the configuration block.Suggested diff:
- await self._client.create_bucket(Bucket=self.bucket_name) + region = self._client_args.get("region_name") + if region and region != "us-east-1": + await self._client.create_bucket( + Bucket=self.bucket_name, + CreateBucketConfiguration={'LocationConstraint': region} + ) + else: + await self._client.create_bucket(Bucket=self.bucket_name)Additionally, ensure that the IAM policy attached to the client’s credentials grants both:
s3:HeadBucket(to check for bucket existence), ands3:CreateBucket(to create the bucket).
♻️ Duplicate comments (6)
docs/source/store-and-retrieve/object-store.md (2)
112-121: Add required Redis key_prefix to the example (breaking change).Per PR notes, Redis configs now require key_prefix; omitting it will cause copy/paste failures. Port and db are already shown as integers, which is correct.
Apply this diff to the Redis example:
Example configuration for Redis storage: ```yaml object_stores: my_object_store: _type: redis host: localhost port: 6379 db: 0 + key_prefix: my-object-store bucket_name: my-bucket--- `180-191`: **Align POST semantics with server behavior and fix markdownlint MD014.** Docs should state that POST is create-only and returns 409 if the object exists. Also, switch code fences to bash and drop leading “$” to satisfy MD014. Apply this diff: ```diff -This enables HTTP endpoints for object store operations: -- **PUT** `/static/{file_path}` - Create or replace an object at the given path (upsert) - ```console - $ curl -X PUT --data-binary @data.txt http://localhost:9000/static/folder/data.txt - ``` -- **GET** `/static/{file_path}` - Download an object - ```console - $ curl -X GET http://localhost:9000/static/folder/data.txt - ``` -- **POST** `/static/{file_path}` - Upload a new object - ```console - $ curl -X POST --data-binary @data_new.txt http://localhost:9000/static/folder/data.txt - ``` +This enables HTTP endpoints for object store operations: +- **PUT** `/static/{file_path}` - Create or replace an object at the given path (upsert) + ```bash + curl -X PUT --data-binary @data.txt http://localhost:9000/static/folder/data.txt + ``` +- **GET** `/static/{file_path}` - Download an object + ```bash + curl -X GET http://localhost:9000/static/folder/data.txt + ``` +- **POST** `/static/{file_path}` - Create a new object only; returns HTTP 409 if the object already exists + ```bash + curl -X POST --data-binary @data_new.txt http://localhost:9000/static/folder/data.txt + ```src/nat/cli/commands/object_store/object_store.py (2)
54-77: Good: non‑blocking file read and real wall‑clock timestamp.Thanks for addressing earlier feedback to avoid blocking the event loop and to use time.time() for upload_timestamp.
204-213: Expose required Redis key_prefix in CLI.Without key_prefix, users can’t construct valid Redis configs from the CLI.
Apply this diff:
@click.group(name="redis", invoke_without_command=False, help="Redis object store operations.") @click.argument("bucket_name", type=str, required=True) @click.option("--host", type=str, help="Redis host") @click.option("--port", type=int, help="Redis port") @click.option("--db", type=int, help="Redis db") +@click.option("--key-prefix", "key_prefix", type=str, required=True, help="Redis key prefix namespace (required).") @click.pass_context def redis(ctx: click.Context, **kwargs): ctx.ensure_object(dict) ctx.obj["store_config"] = get_object_store_config(store_type="redis", **kwargs)packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (2)
36-47: Enforce MySQL schema-name length cap forbucket_name(≤57 chars withbucket_prefix).Prevents runtime DDL failures on schema creation.
Apply this diff:
def __init__(self, *, bucket_name: str, host: str, port: int, username: str | None, password: str | None): super().__init__() if not re.fullmatch(r"[A-Za-z0-9_-]+", bucket_name): raise ValueError("bucket_name must match [A-Za-z0-9_-]+") + # MySQL schema (database) identifiers are limited to 64 chars; "bucket_" prefix is 7 chars. + if len(bucket_name) > 64 - len("bucket_"): + raise ValueError( + f"bucket_name is too long; max {64 - len('bucket_')} characters to fit within MySQL's 64-char limit." + ) self._bucket_name = bucket_name self._host = host self._port = port self._username = username self._password = password
80-89: Track last-modified separately; don’t mutatecreated_at.Add
updated_atand preservecreated_atsemantics. This also enables correct upsert bookkeeping.Apply this diff:
CREATE TABLE IF NOT EXISTS object_meta ( id INT AUTO_INCREMENT PRIMARY KEY, path VARCHAR(768) NOT NULL UNIQUE, size BIGINT NOT NULL, - created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP ) ENGINE=InnoDB;
🧹 Nitpick comments (11)
packages/nvidia_nat_redis/src/nat/plugins/redis/register.py (2)
16-17: Replace blanket flake8 disable with targeted ignores on side‑effect imports.Avoid silencing all lint checks for the entire file. Keep side‑effect imports, but scope the suppression to F401 only.
Apply this diff:
-# flake8: noqa # isort:skip_file # Import any providers which need to be automatically registered here -from . import memory -from . import object_store +from . import memory # noqa: F401 +from . import object_store # noqa: F401If import order is not semantically significant, consider removing
# isort:skip_filein a follow-up to let isort manage order automatically.Also applies to: 21-22
19-23: Add a concise module docstring to comply with docstring policy.This module is importable and should carry a brief Google‑style module docstring per guidelines.
Suggested docstring (place after the license block and before imports):
"""Registers Redis-related providers for the NeMo Agent Toolkit by importing modules that perform decorator-based registration."""docs/source/store-and-retrieve/object-store.md (1)
44-47: Use code formatting and fix phrasing for operation names.Minor doc polish: prefer backticks for code entities and clarify “upsert” wording.
Apply this diff:
-- **put_object(key, item)**: Store a new object with a unique key. Raises if the key already exists. -- **upsert_object(key, item)**: Update (or inserts) an object with the given key. -- **get_object(key)**: Retrieve an object by its key. Raises if the key doesn't exist. -- **delete_object(key)**: Remove an object from the store. Raises if the key doesn't exist. +- `put_object(key, item)`: Store a new object with a unique key. Raises if the key already exists. +- `upsert_object(key, item)`: Insert or update an object with the given key. +- `get_object(key)`: Retrieve an object by its key. Raises if the key doesn't exist. +- `delete_object(key)`: Remove an object from the store. Raises if the key doesn't exist.examples/object_store/user_report/README.md (3)
20-20: Fix article: “An example tool”, not “And example tool.”Simple grammar cleanup for the opening sentence.
Apply this diff:
-And example tool in the NeMo Agent toolkit that makes use of an Object Store to retrieve data. +An example tool in the NeMo Agent toolkit that makes use of an object store to retrieve data.
73-75: Tighten phrasing: “a drop‑in replacement.”Minor style improvement.
Apply this diff:
-If you want to run this example in a local setup without creating a bucket in AWS, you can set up MinIO in your local machine. MinIO is an object storage system and acts as drop-in replacement for AWS S3. +If you want to run this example in a local setup without creating a bucket in AWS, you can set up MinIO on your local machine. MinIO is an object storage system and acts as a drop‑in replacement for AWS S3.
165-168: Add Content-Type for POST/PUT examples to avoid misclassification.When uploading JSON (or other types), set an explicit Content-Type. Keep --data-binary to avoid newline mangling.
Apply this diff:
-- Upload an object: `curl -X POST http://<hostname>:<port>/static/{file_path} --data-binary @{filename}` -- Upsert an object: `curl -X PUT http://<hostname>:<port>/static/{file_path} --data-binary @{filename}` +- Upload an object: `curl -X POST http://<hostname>:<port>/static/{file_path} --data-binary @{filename} -H 'Content-Type: application/json'` +- Upsert an object: `curl -X PUT http://<hostname>:<port>/static/{file_path} --data-binary @{filename} -H 'Content-Type: application/json'`src/nat/cli/commands/object_store/object_store.py (3)
91-109: Guard against missing ctx.obj["store_config"] with a clear error.Defensive check improves UX when this decorator is reused elsewhere or groups change.
Apply this diff:
@click.pass_context def wrapper(ctx: click.Context, **kwargs): - config = ctx.obj["store_config"] + if not getattr(ctx, "obj", None) or "store_config" not in ctx.obj: + raise click.ClickException("Internal error: store configuration not initialized. " + "Invoke via a store group (s3/mysql/redis) and pass required options.") + config = ctx.obj["store_config"]
16-24: Add missing typing imports to support return type annotations below.Keeps public APIs fully typed per guidelines.
Apply this diff:
import time from pathlib import Path import click +from typing import Any +from typing import Callable
54-55: Add explicit return type annotations to public APIs.Complies with “All public Python APIs require type hints”.
Apply this diff:
-async def upload_file(object_store: ObjectStore, file_path: Path, key: str): +async def upload_file(object_store: ObjectStore, file_path: Path, key: str) -> None: @@ -async def upload_command(store: ObjectStore, local_dir: Path, **_kwargs): +async def upload_command(store: ObjectStore, local_dir: Path, **_kwargs) -> int: @@ -async def delete_command(store: ObjectStore, keys: list[str], **_kwargs): +async def delete_command(store: ObjectStore, keys: list[str], **_kwargs) -> int: @@ -def object_store_command(**_kwargs): +def object_store_command(**_kwargs) -> None: @@ -def register_object_store_commands(): +def register_object_store_commands() -> None:Also applies to: 112-118, 145-151, 173-176, 179-179
packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (2)
181-191: Be robust to driver return types when decoding BLOBs.Some drivers/versions may return
bytearray/memoryview. Normalize to bytes before decoding.Apply this diff:
- return ObjectStoreItem.model_validate_json(row[0].decode("utf-8")) + payload = bytes(row[0]) + return ObjectStoreItem.model_validate_json(payload.decode("utf-8"))
31-34: Docstring is concise; consider adding brief notes on transactional semantics.Optional: mention that writes are transactional per object and that schema is created lazily on first connect.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (8)
docs/source/store-and-retrieve/object-store.md(4 hunks)examples/object_store/user_report/README.md(5 hunks)packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py(6 hunks)packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py(1 hunks)packages/nvidia_nat_redis/src/nat/plugins/redis/register.py(1 hunks)packages/nvidia_nat_redis/tests/test_redis_object_store.py(1 hunks)packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py(5 hunks)src/nat/cli/commands/object_store/object_store.py(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- packages/nvidia_nat_redis/tests/test_redis_object_store.py
- packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py
🧰 Additional context used
📓 Path-based instructions (4)
**/*.py
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
**/*.py: In code comments, use abbreviations: NAT (informal/comments), nat (API namespace/CLI), nvidia-nat (package). Do not use these abbreviations in documentation.
Run isort first with line_length=120, multi_line_output=3, include_trailing_comma=true, force_single_line=true.
Run yapf second with PEP8 base and column_limit=120.
Respect Python naming: snake_case for functions and variables, PascalCase for classes, UPPER_CASE for constants.
Fix flake8 (pflake8) warnings unless explicitly ignored in pyproject.toml.
All public Python APIs require type hints (Python 3.11+) on parameters and return values.
Prefer typing/collections.abc abstractions (e.g., Sequence over list).
Use typing.Annotated to express units or extra metadata when useful.
Treat pyright warnings as errors during development (configured via pyproject.toml).
Provide Google‑style docstrings for every public module, class, function, and CLI command.
Docstring first line must be a concise description ending with a period.
Do not hard‑code version numbers in Python code; versions are derived by setuptools‑scm.
Prefer httpx with SSL verification enabled by default for HTTP calls; follow OWASP Top‑10.
Use async/await for I/O‑bound work (HTTP, DB, file reads).
Cache expensive computations with functools.lru_cache or an external cache when appropriate.
Leverage NumPy vectorized operations when beneficial and feasible.
Files:
packages/nvidia_nat_redis/src/nat/plugins/redis/register.pypackages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.pysrc/nat/cli/commands/object_store/object_store.pypackages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py
{src/**/*.py,packages/*/src/**/*.py}
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
All importable Python code must live under src/ or packages//src/.
Files:
packages/nvidia_nat_redis/src/nat/plugins/redis/register.pypackages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.pysrc/nat/cli/commands/object_store/object_store.pypackages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py
**/*
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
**/*: Every file must start with the standard SPDX Apache‑2.0 header.
All source files must include the SPDX Apache‑2.0 header template; CI verifies via ci/scripts/github/checks.sh.
Files:
packages/nvidia_nat_redis/src/nat/plugins/redis/register.pypackages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.pyexamples/object_store/user_report/README.mdsrc/nat/cli/commands/object_store/object_store.pypackages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.pydocs/source/store-and-retrieve/object-store.md
docs/source/**/*.md
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
docs/source/**/*.md: Use the exact product naming in documentation: first use “NVIDIA NeMo Agent toolkit”; subsequent references “NeMo Agent toolkit”; in headings use “NeMo Agent Toolkit” (capital T).
Never use deprecated names (Agent Intelligence toolkit, aiqtoolkit, AgentIQ, AIQ/aiq) in documentation unless explicitly referring to deprecated names or compatibility layers.
Use the intended name “AIQ Blueprint” and do not change it in docs.
Documentation sources must be Markdown files under docs/source.
Surround code entities with backticks in documentation to avoid Vale false positives.
Keep documentation in sync with code; Sphinx build must be error‑free with no broken links.
Do not hard‑code project version numbers in documentation; versions are derived by setuptools‑scm.
Files:
docs/source/store-and-retrieve/object-store.md
🧬 Code graph analysis (2)
packages/nvidia_nat_redis/src/nat/plugins/redis/register.py (1)
examples/object_store/user_report/tests/test_user_report_tool.py (1)
object_store(53-55)
src/nat/cli/commands/object_store/object_store.py (5)
examples/object_store/user_report/tests/test_user_report_tool.py (2)
builder(33-49)object_store(53-55)src/nat/object_store/models.py (1)
ObjectStoreItem(21-38)packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (2)
upsert_object(145-169)delete_object(194-218)packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py (2)
upsert_object(94-101)delete_object(117-126)packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py (2)
upsert_object(121-137)delete_object(152-166)
🪛 LanguageTool
examples/object_store/user_report/README.md
[grammar] ~28-~28: There might be a mistake here.
Context: ...up-api-keys) - Choose an Object Store - Setting up MinIO ...
(QB_NEW_EN)
[grammar] ~29-~29: There might be a mistake here.
Context: ...an-object-store) - Setting up MinIO - Setting up MySQL ...
(QB_NEW_EN)
[grammar] ~30-~30: There might be a mistake here.
Context: ...etting-up-minio) - Setting up MySQL - Setting up Redis -...
(QB_NEW_EN)
[grammar] ~31-~31: There might be a mistake here.
Context: ...etting-up-mysql) - Setting up Redis - Loading Mock Data -...
(QB_NEW_EN)
[grammar] ~32-~32: There might be a mistake here.
Context: ...setting-up-redis) - Loading Mock Data - [NeMo Agent Toolkit File Server](#nemo-ag...
(QB_NEW_EN)
[style] ~73-~73: Consider replacing this word to strengthen your wording.
Context: ...hine. MinIO is an object storage system and acts as drop-in replacement for AWS S3....
(AND_THAT)
[grammar] ~73-~73: There might be a mistake here.
Context: ...IO is an object storage system and acts as drop-in replacement for AWS S3. You ca...
(QB_NEW_EN)
[grammar] ~81-~81: There might be a mistake here.
Context: ...r-compose.minio.yml up -d ``` > [!NOTE] > This is not a secure configuration and...
(QB_NEW_EN)
[grammar] ~100-~100: There might be a mistake here.
Context: ...r-compose.mysql.yml up -d ``` > [!NOTE] > This is not a secure configuration and...
(QB_NEW_EN)
[grammar] ~111-~111: There might be a mistake here.
Context: ...r-compose.redis.yml up -d ``` > [!NOTE] > This is not a secure configuration and...
(QB_NEW_EN)
[grammar] ~165-~165: There might be a mistake here.
Context: ...erver (Optional) - Download an object: curl -X GET http://<hostname>:<port>/static/{file_path} -o {filename} - Upload an object: `curl -X POST http://<...
(QB_NEW_EN)
[grammar] ~166-~166: There might be a mistake here.
Context: ...ath} -o {filename}- Upload an object:curl -X POST http://:/static/{file_path} --data-binary @{filename}- Upsert an object:curl -X PUT http://<h...
(QB_NEW_EN)
[grammar] ~167-~167: There might be a mistake here.
Context: ...binary @{filename}- Upsert an object:curl -X PUT http://:/static/{file_path} --data-binary @{filename}- Delete an object:curl -X DELETE http:/...
(QB_NEW_EN)
[grammar] ~179-~179: There might be a mistake here.
Context: ... three options for running the workflow: 1. Using the S3-compatible object store (`c...
(QB_NEW_EN)
[grammar] ~180-~180: There might be a mistake here.
Context: ...ompatible object store (config_s3.yml) 2. Using the MySQL object store (`config_my...
(QB_NEW_EN)
[grammar] ~181-~181: There might be a mistake here.
Context: ... MySQL object store (config_mysql.yml) 3. Using the Redis object store (`config_re...
(QB_NEW_EN)
[grammar] ~184-~184: There might be a mistake here.
Context: ...hich uses an S3-compatible object store. You can change the configuration file by...
(QB_NEW_EN)
[grammar] ~185-~185: There might be a mistake here.
Context: ... config_mysql.yml for the MySQL server or config_redis.yml for the Redis serv...
(QB_NEW_EN)
🪛 markdownlint-cli2 (0.17.2)
docs/source/store-and-retrieve/object-store.md
182-182: Dollar signs used before commands without showing output
(MD014, commands-show-output)
186-186: Dollar signs used before commands without showing output
(MD014, commands-show-output)
190-190: Dollar signs used before commands without showing output
(MD014, commands-show-output)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (5)
- GitHub Check: CI Pipeline / Test (arm64, 3.11)
- GitHub Check: CI Pipeline / Build Wheels
- GitHub Check: CI Pipeline / Test (amd64, 3.11)
- GitHub Check: CI Pipeline / Test (amd64, 3.12)
- GitHub Check: CI Pipeline / Test (arm64, 3.12)
🔇 Additional comments (15)
packages/nvidia_nat_redis/src/nat/plugins/redis/register.py (2)
22-22: Side‑effect import to trigger Redis object‑store registration looks good.Importing
object_storehere is the correct pattern to ensure decorator-based registration happens on module import.
21-22: Registry and provider name consistency verified
- The factory in
object_store.pyis correctly decorated with@register_object_store(config_type=RedisObjectStoreClientConfig)on line 34.- The
RedisObjectStoreClientConfigclass extendsObjectStoreBaseConfigand advertisesname="redis"on line 23.- The
redisplugin package contains an__init__.py, ensuring it’s importable by the plugin discovery mechanism.All checks pass—no further action needed.
docs/source/store-and-retrieve/object-store.md (2)
74-74: Redis provider listing looks correct.Provider path matches the new package layout and helps discoverability.
148-149: LGTM: metadata example is now a valid mapping.Thanks for fixing the dict to use "key: value" pairs.
src/nat/cli/commands/object_store/object_store.py (1)
181-191: LGTM: group setup ensures ctx.obj exists; MySQL --db now a string.These changes prevent NoneType errors and correctly treat database names as strings.
Also applies to: 192-203
packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (5)
17-17: LGTM: addedrefor identifier validation.Import is used for validating
bucket_name; aligns with security guidance.
50-53: LGTM: centralizes quoted schema construction.Backtick-quoting via
_schemais correct and avoids identifier collisions.
54-69: LGTM: async connection lifecycle and type hint for__aenter__.Pool creation is explicit and logs are informative.
107-117: LGTM:__aexit__annotation and pool shutdown sequence.Properly closes and awaits pool closure; state reset is clear.
90-98: LGTM:object_datalayout withROW_FORMAT=DYNAMIC.Appropriate for large payloads; FK with
ON DELETE CASCADEis correct.packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py (5)
35-57: Constructor follows past feedback correctly.The constructor now properly supports AWS default credential chain by making credentials optional and only including them in
_client_argswhen provided. This addresses the previous review comment about not hard-requiring credentials.
59-59: Add explicit return type annotation.The return type annotation should be more specific to follow Python typing best practices.
- async def __aenter__(self) -> "S3ObjectStore": + async def __aenter__(self) -> "S3ObjectStore":Wait, this is already correct. The forward reference with quotes is appropriate here.
81-81: LGTM on explicit return type.The
Nonereturn type annotation is correctly added for the async context manager exit method.
114-119: Improved error handling with proper chaining.The error handling now properly uses
raise ... from eto preserve the original exception context, which is good practice for debugging.
149-150: Consistent error handling across methods.Good consistency in error handling pattern using
raise ... from efor bothget_objectanddelete_objectmethods.Also applies to: 160-161
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Will Killian <[email protected]>
|
@coderabbitai full review |
✅ Actions performedFull review triggered. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (7)
packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py (2)
71-79: Bucket creation: handle non-us-east-1 regions.AWS requires
CreateBucketConfigurationoutsideus-east-1. Without it, create may fail on AWS (MinIO ignores region, so tests can pass locally).# Ensure the bucket exists try: await self._client.head_bucket(Bucket=self.bucket_name) except ClientError as e: - if e.response['Error']['Code'] == '404': - await self._client.create_bucket(Bucket=self.bucket_name) + if e.response['Error']['Code'] in ('404', 'NoSuchBucket', 'NotFound'): + create_args = {"Bucket": self.bucket_name} + # Only required on AWS when region != us-east-1 + if (region := self._client.meta.config.region_name) and region != "us-east-1": + create_args["CreateBucketConfiguration"] = {"LocationConstraint": region} + await self._client.create_bucket(**create_args) logger.info("Created bucket %s", self.bucket_name)
152-167: Delete semantics: skip pre-fetch of object body and don’t raise on DeleteMarker.
get_objectjust to check existence is expensive; usehead_object.- Raising
NoSuchKeyErrorwhenDeleteMarkeris True is incorrect for versioned buckets—S3 returns a marker on successful deletes. Treat that as success.- async def delete_object(self, key: str) -> None: + async def delete_object(self, key: str) -> None: if self._client is None: raise RuntimeError("Connection not established") - try: - await self._client.get_object(Bucket=self.bucket_name, Key=key) - except ClientError as e: - if e.response['Error']['Code'] == 'NoSuchKey': - raise NoSuchKeyError(key=key, additional_message=str(e)) from e - raise - - results = await self._client.delete_object(Bucket=self.bucket_name, Key=key) - - if results.get('DeleteMarker', False): - raise NoSuchKeyError(key=key, additional_message="Object was a delete marker") + # Check existence cheaply + try: + await self._client.head_object(Bucket=self.bucket_name, Key=key) + except ClientError as e: + if e.response['Error']['Code'] in ('NoSuchKey', '404', 'NotFound'): + raise NoSuchKeyError(key=key, additional_message=str(e)) from e + raise + + # Proceed with deletion; treat DeleteMarker as success in versioned buckets + await self._client.delete_object(Bucket=self.bucket_name, Key=key)packages/nvidia_nat_mysql/src/nat/plugins/mysql/object_store.py (2)
41-49: Avoid import-time int coercion and fix env var placeholders in descriptions
- Import-time
int(os.environ.get(...))can raise on malformed env values at import. Preferdefault_factoryto defer evaluation.- Use f-strings so
{HOST_ENV}/{DEFAULT_HOST}are interpolated, not shown literally.Apply:
- host: str = Field( - default=os.environ.get(HOST_ENV, DEFAULT_HOST), - description="The host of the MySQL server" - " (uses {HOST_ENV} if unspecified; falls back to {DEFAULT_HOST})", - ) - port: int = Field( - default=int(os.environ.get(PORT_ENV, DEFAULT_PORT)), - description="The port of the MySQL server" - " (uses {PORT_ENV} if unspecified; falls back to {DEFAULT_PORT})", - ) + host: str = Field( + default=os.environ.get(HOST_ENV, DEFAULT_HOST), + description=f"The host of the MySQL server (uses {HOST_ENV} if unspecified; falls back to {DEFAULT_HOST})", + ) + port: int = Field( + default_factory=lambda: int(os.environ.get(PORT_ENV, DEFAULT_PORT)), + description=f"The port of the MySQL server (uses {PORT_ENV} if unspecified; falls back to {DEFAULT_PORT})", + )
50-57: Fix typos and interpolation inconsistencies in username/password descriptionsCorrect “unspecifed” → “unspecified” and interpolate
{PASSWORD_ENV}.Apply:
- username: str | None = Field( - default=os.environ.get(USERNAME_ENV), - description=f"The username used to connect to the MySQL server (uses {USERNAME_ENV} if unspecifed)", - ) - password: str | None = Field( - default=os.environ.get(PASSWORD_ENV), - description="The password used to connect to the MySQL server (uses {PASSWORD_ENV} if unspecifed)", - ) + username: str | None = Field( + default=os.environ.get(USERNAME_ENV), + description=f"The username used to connect to the MySQL server (uses {USERNAME_ENV} if unspecified)", + ) + password: str | None = Field( + default=os.environ.get(PASSWORD_ENV), + description=f"The password used to connect to the MySQL server (uses {PASSWORD_ENV} if unspecified)", + )examples/object_store/user_report/README.md (2)
20-20: Fix grammar (“And example tool” → “An example tool”).User-facing doc.
-And example tool in the NeMo Agent toolkit that makes use of an Object Store to retrieve data. +An example tool in the NeMo Agent toolkit that makes use of an Object Store to retrieve data.
238-244: Fix inconsistent user ID in example output (6789 vs 678901).This mismatch can confuse readers.
-User report for 678901 with date latest added successfully +User report for 6789 with date latest added successfullypackages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (1)
137-139: Ensure BLOB is bytes: encode JSON before insert.
model_dump_json()returns str; encode to UTF-8 to guarantee driver sends bytes to LONGBLOB anddecode()on read is valid.- blob = item.model_dump_json() + blob = item.model_dump_json().encode("utf-8") await cur.execute("INSERT INTO object_data (id, data) VALUES (%s, %s)", (obj_id, blob))Apply the same change in
upsert_objectbelow.
♻️ Duplicate comments (13)
packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py (1)
46-51: Type-annotate async context manager parameters (pyright-friendly).Add precise types for
__aenter__/__aexit__to satisfy static checkers.+from types import TracebackType @@ - async def __aenter__(self) -> "RedisObjectStore": + async def __aenter__(self) -> "RedisObjectStore": ... @@ - async def __aexit__(self, exc_type, exc_value, traceback) -> None: + async def __aexit__( + self, + exc_type: type[BaseException] | None, + exc_value: BaseException | None, + traceback: TracebackType | None, + ) -> None:Also applies to: 68-75
packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py (1)
50-57: LGTM: credential handling now respects the AWS default credential chain.Only injecting explicit keys when provided allows roles/SSO/env to work. Matches boto best practices.
docs/source/store-and-retrieve/object-store.md (2)
112-121: Redis config example is missing required key_prefix (breaking change)Per PR notes,
key_prefixis required anddb/portare ints. Addkey_prefixto prevent copy-paste failures.Apply:
object_stores: my_object_store: _type: redis host: localhost port: 6379 db: 0 + key_prefix: my-object-store bucket_name: my-bucket
179-191: Clarify POST semantics to reflect create-only with 409 on conflictDocs now state PUT is upsert (good). Also call out that POST returns 409 if the object exists, matching the server behavior.
Apply:
- **POST** `/static/{file_path}` - Upload a new object + - **POST** `/static/{file_path}` - Create a new object only; returns 409 if the object already exists ```console $ curl -X POST --data-binary @data_new.txt http://localhost:9000/static/folder/data.txt</blockquote></details> <details> <summary>examples/object_store/user_report/README.md (1)</summary><blockquote> `129-133`: **Expose required Redis key prefix in CLI example (if applicable).** Earlier discussion noted a breaking change requiring a key prefix for Redis. If that applies to the object-store CLI, include it so the command runs as-is. If that requirement is only for the “memory provider,” ignore this. ```diff nat object-store \ - redis --host 127.0.0.1 --port 6379 --db 0 my-bucket \ + redis --host 127.0.0.1 --port 6379 --db 0 --key-prefix reports: my-bucket \ upload ./examples/object_store/user_report/data/object_store/Please confirm whether the Redis Object Store requires
--key-prefix. If not, we should add a note clarifying the difference between the Redis memory provider and the Redis object store to avoid confusion.packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (2)
81-88: Track last modification time separately; don’t overwrite created_at.Add
updated_atand preservecreated_atsemantics.CREATE TABLE IF NOT EXISTS object_meta ( id INT AUTO_INCREMENT PRIMARY KEY, path VARCHAR(768) NOT NULL UNIQUE, size BIGINT NOT NULL, - created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP ) ENGINE=InnoDB;
39-43: Harden bucket_name validation: enforce MySQL identifier length.MySQL schema (database) identifiers are limited to 64 characters. With the
"bucket_"prefix, the suffix must be ≤57 chars to avoid runtime DDL failures.if not re.fullmatch(r"[A-Za-z0-9_-]+", bucket_name): raise ValueError("bucket_name must match [A-Za-z0-9_-]+") + # Enforce MySQL schema (database) identifier length (≤64). + if len("bucket_") + len(bucket_name) > 64: + raise ValueError("bucket_name is too long; max 57 characters (fits 'bucket_' + name into 64).")src/nat/cli/commands/object_store/object_store.py (5)
32-42: STORE_CONFIGS module paths are correct per plugin architecture.The mapping to nat.plugins.{s3,mysql,redis}.object_store for config classes matches the architecture (configs + registration live in object_store.py; implementations live elsewhere). Thanks for keeping this correct.
64-65: Good fix: non-blocking file reads and real timestamps.Using asyncio.to_thread for file reads and time.time() for wall time resolves the earlier concerns about blocking I/O and monotonic clocks. LGTM.
Also applies to: 72-73
131-136: POSIX key normalization is spot on.Using .as_posix() ensures forward slashes across platforms (important for S3-like semantics). Nice catch.
181-203: CLI group setup looks correct; obj initialization and MySQL --db type fixed.
- ctx.ensure_object(dict) avoids NoneType issues.
- MySQL --db switched to str, which is the correct type for schema names.
204-213: Expose required Redis key_prefix in CLI (blocking).Per the PR’s breaking-change note, Redis config requires key_prefix. The CLI currently can’t pass it, making valid Redis configs impossible to provide from the CLI.
Apply this diff:
@click.group(name="redis", invoke_without_command=False, help="Redis object store operations.") @click.argument("bucket_name", type=str, required=True) @click.option("--host", type=str, help="Redis host") @click.option("--port", type=int, help="Redis port") @click.option("--db", type=int, help="Redis db") +@click.option("--key-prefix", "key_prefix", type=str, required=True, + help="Redis key prefix namespace (required).") @click.pass_context def redis(ctx: click.Context, **kwargs): ctx.ensure_object(dict) ctx.obj["store_config"] = get_object_store_config(store_type="redis", **kwargs)Request for alignment: If host/port/db are also required in the new schema, consider marking them required=True for early CLI validation.
examples/object_store/user_report/configs/config_redis.yml (1)
23-30: Add required Redis key_prefix to match the new schema.Without key_prefix this config will fail validation or default to an unintended namespace.
Apply this diff:
object_stores: report_object_store: _type: redis host: localhost db: 0 port: 6379 bucket_name: my-bucket + key_prefix: user-report/
🧹 Nitpick comments (34)
src/nat/cli/commands/object_store/__init__.py (1)
1-14: License-only package init is fine; consider adding a module docstring if you plan future exports.No functional issues. If this package will later re-export CLI entrypoints (e.g., to aid discoverability or tooling), add explicit exports via all or a brief module docstring to clarify intent.
examples/object_store/user_report/configs/config_mysql.yml (1)
24-31: Example uses MySQL root user; acceptable for local demos, but document a non-root alternative for production.Config sets username: root. Consider a follow-up example showing a dedicated DB user with least privilege and matching docker-compose env (MYSQL_DATABASE/USER/PASSWORD).
examples/deploy/docker-compose.mysql.yml (4)
24-26: Use .env file or secrets for MYSQL_ROOT_PASSWORD to avoid accidental leaks.Compose supports a .env file and/or secrets. At minimum, document that users should override the default before exposing the port beyond localhost.
Apply to move the password to .env (compose remains the same), or use secrets:
services: mysql: image: mysql:9.3 + env_file: + - .env environment: - - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-my_password} + - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}.env (not committed):
MYSQL_ROOT_PASSWORD=my_strong_password
16-27: Add a healthcheck for readiness and optionally create an app user/database.Improves local DX by preventing race conditions and promotes least privilege for non-demo use.
Here’s a minimal enhancement while keeping your current root-based example:
services: mysql: - image: mysql:9.3 + image: mysql:9.3 volumes: - mysql-data:/var/lib/mysql container_name: mysql ports: - 3306:3306 environment: - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-my_password} restart: unless-stopped + healthcheck: + test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-uroot", "-p$$MYSQL_ROOT_PASSWORD"] + interval: 10s + timeout: 5s + retries: 5If you want a non-root example (requires updating examples/object_store/user_report/config_mysql.yml to use that user):
environment: - - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-my_password} + - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-my_password} + - MYSQL_DATABASE=nat + - MYSQL_USER=nat + - MYSQL_PASSWORD=${MYSQL_PASSWORD:-nat_password}
23-23: Expose port only on localhost for safer defaults.Prevents unintended remote access in shared dev environments.
- - 3306:3306 + - 127.0.0.1:3306:3306
18-18: Pin MySQL image to a stable GA/LTS tagConfirmed via Docker Hub API that
mysql:9.3(as well as 9.2, 9.1, 9.0 and 8.4) all exist, so the Compose run won’t break due to a missing tag. However, for production stability it’s best to target an officially GA/LTS release.• Replace the floating or interim tag with a known GA/LTS version
• e.g., change:- image: mysql:9.3 + image: mysql:8.4examples/deploy/docker-compose.minio.yml (2)
26-28: Parameterize credentials; avoid hard-coded defaults.Use env expansion to discourage committing real creds and allow overrides via
.env. Keeps examples usable while nudging safer defaults.environment: - - MINIO_ROOT_USER=minioadmin - - MINIO_ROOT_PASSWORD=minioadmin + - MINIO_ROOT_USER=${MINIO_ROOT_USER:-minioadmin} + - MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD:-minioadmin}Optional: document setting these in an
.envfile colocated with the compose file. I can add a sample.env.exampleif helpful.
30-35: Healthcheck may rely on tools not present in the image. Verify curl availability or switch to CMD-SHELL.Many minimal images don’t ship curl. If
curlis absent, the healthcheck will always fail. Prefer CMD-SHELL and a more robust command. Also use 127.0.0.1 to avoid DNS overhead.- healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://127.0.0.1:9000/minio/health/live || exit 1"] interval: 30s timeout: 10s retries: 3 start_period: 30sIf curl isn’t available, alternatives:
- Use
wget -qO-if present.- Replace with a TCP check:
bash -c '</dev/tcp/127.0.0.1/9000'(requires a shell with /dev/tcp support).- Run healthcheck from a sidecar (heavier).
I can adjust based on your target environments.examples/deploy/README.md (3)
37-39: Grammar: add article before “Phoenix observability server”.- **Phoenix Observability:** Includes `docker-compose.phoenix.yml` for running Phoenix observability server to monitor and debug workflows. + **Phoenix Observability:** Includes `docker-compose.phoenix.yml` for running a Phoenix observability server to monitor and debug workflows.
56-57: Fix spacing in command description.- - Linux: Run`sudo systemctl start docker` (ensure your user has permission to run Docker). + - Linux: Run `sudo systemctl start docker` (ensure your user has permission to run Docker).
80-84: Clarify Redis scope to avoid implying it’s required for all object-store examples.Redis isn’t needed for S3/MySQL object-store examples. Tighten wording to prevent confusion.
-To start Redis (required for memory and object store examples): +To start Redis (required for Redis memory and Redis object-store examples):packages/nvidia_nat_redis/src/nat/plugins/redis/memory.py (3)
25-28: Breaking change: types for host/db/port/key_prefix are now concrete (str/int).Good alignment, but this is a schema break (db/port → int). Ensure docs, changelog, and sample configs reflect this and call out the migration in release notes.
I can scan docs/examples and open a follow-up PR to update any lingering string values for db/port.
43-49: Consider an early connectivity check to fail fast on misconfiguration.A quick
PINGhelps surface bad host/port/db immediately instead of later during indexing.redis_client = redis.Redis(host=config.host, port=config.port, db=config.db, decode_responses=True, socket_timeout=5.0, socket_connect_timeout=5.0) + # Fail fast if Redis is unreachable + try: + await redis_client.ping() + except Exception as e: + # Optional: wrap with a clearer message + raise RuntimeError(f"Unable to connect to Redis at {config.host}:{config.port}/{config.db}") from e
56-59: Close the Redis client when the builder tears down the resource.Yielding without a finally risks leaked connections in long-lived processes and tests.
- memory_editor = RedisEditor(redis_client=redis_client, key_prefix=config.key_prefix, embedder=embedder) - - yield memory_editor + memory_editor = RedisEditor(redis_client=redis_client, key_prefix=config.key_prefix, embedder=embedder) + try: + yield memory_editor + finally: + # redis-py asyncio >=5.0 provides `aclose`; fall back if needed. + if hasattr(redis_client, "aclose"): + await redis_client.aclose() + else: + await redis_client.connection_pool.disconnect()Please confirm your redis-py version; if it’s <5.0,
aclosemay not exist and the fallback is appropriate.packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py (2)
31-34: Docstring/key-format mismatch.Docstring says
nat:object_store:bucket:{bucket_name}:{object_key}, but_make_keyuses slashes. Align to what the code actually does.- Each object is stored as a single binary value at key "nat:object_store:bucket:{bucket_name}:{object_key}". + Each object is stored as a single value at key "nat/object_store/{bucket_name}/{object_key}".
104-115: Serialization format: confirm JSON is the intended on-wire format.The PR notes indicated pickling, but the current implementation uses JSON via Pydantic. JSON is safer and more portable; if that’s the intent, great—just ensure tests/docs reflect JSON, not pickle. If you truly need pickle (caveats: security, compatibility), we’ll need to switch both write/read paths.
I can update docs/tests either way after you confirm the desired format.
packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py (3)
95-111: Precondition strategy: verify If-None-Match interoperability across AWS/MinIO.
IfNoneMatch='*'on PutObject is the right idea but has spotty support across S3-compatible endpoints. Validate against your target providers; otherwise fallback to a HEAD+PUT-without-precondition sequence or a multipart with conditional copy.If needed, I can wire a provider capability flag and adapt behavior per endpoint.
139-147: Defensive access to optional response fields.Some S3-compatible backends omit
ContentType/Metadata. Avoid KeyError by using defaults.- return ObjectStoreItem(data=data, content_type=response['ContentType'], metadata=response['Metadata']) + return ObjectStoreItem( + data=data, + content_type=response.get('ContentType'), + metadata=response.get('Metadata') or {}, + )
81-89: Add param type annotations to async context manager for consistency and static checks.+from types import TracebackType @@ - async def __aexit__(self, exc_type, exc_value, traceback) -> None: + async def __aexit__( + self, + exc_type: type[BaseException] | None, + exc_value: BaseException | None, + traceback: TracebackType | None, + ) -> None:src/nat/cli/entrypoint.py (2)
111-111: Consider a short alias for ergonomicsIf you want easier typing, add a non-hyphen alias (e.g.,
objectstore) in addition toobject-store.Apply:
cli.add_command(sizing, name="sizing") cli.add_command(object_store_command, name="object-store") +cli.add_command(object_store_command, name="objectstore")
116-116: Avoid blanket type: ignore; narrow with castPrefer a targeted cast over a file-littering
# type: ignore.Apply:
-cli.add_command(start_command.get_command(None, "mcp"), name="mcp") # type: ignore +from typing import cast # at top-level imports +cli.add_command(cast(click.Command, start_command.get_command(None, "mcp")), name="mcp")Note: keep the new import near other stdlib typing imports.
docs/source/store-and-retrieve/object-store.md (1)
182-182: Satisfy markdownlint MD014 by dropping leading '$' in command-only blocksRemove the
$prompts since these blocks don’t show output.Apply:
- $ curl -X PUT --data-binary @data.txt http://localhost:9000/static/folder/data.txt + curl -X PUT --data-binary @data.txt http://localhost:9000/static/folder/data.txt- $ curl -X GET http://localhost:9000/static/folder/data.txt + curl -X GET http://localhost:9000/static/folder/data.txt- $ curl -X POST --data-binary @data_new.txt http://localhost:9000/static/folder/data.txt + curl -X POST --data-binary @data_new.txt http://localhost:9000/static/folder/data.txtAlso applies to: 186-186, 190-190
packages/nvidia_nat_s3/src/nat/plugins/s3/object_store.py (1)
44-49: Normalize kwargs by excluding Nones on constructionIf
S3ObjectStorealready has sensible defaults, avoid passingNoneexplicitly.Apply:
- async with S3ObjectStore(**config.model_dump(exclude={"type"})) as store: + async with S3ObjectStore(**config.model_dump(exclude={"type"}, exclude_none=True)) as store:packages/nvidia_nat_mysql/src/nat/plugins/mysql/object_store.py (1)
65-65: Exclude None values when unpacking configPrevents overriding constructor defaults with explicit
Nones.Apply:
- async with MySQLObjectStore(**config.model_dump(exclude={"type"})) as store: + async with MySQLObjectStore(**config.model_dump(exclude={"type"}, exclude_none=True)) as store:packages/nvidia_nat_redis/tests/test_redis_object_store.py (1)
30-37: Make the availability check more robust and configurable.Increase the timeout slightly and allow CI to override host/port/db via env vars. Also, use a non-default DB to avoid collisions with a developer’s local DB 0 contents when tests assert “create” semantics.
Apply:
+import os @@ -def _redis_available(host: str = "localhost", port: int = 6379) -> bool: +REDIS_HOST = os.getenv("NAT_TEST_REDIS_HOST", "localhost") +REDIS_PORT = int(os.getenv("NAT_TEST_REDIS_PORT", "6379")) +REDIS_DB = int(os.getenv("NAT_TEST_REDIS_DB", "14")) # avoid clobbering db 0 + +def _redis_available(host: str = REDIS_HOST, port: int = REDIS_PORT) -> bool: with socket.socket() as s: - s.settimeout(0.25) + s.settimeout(0.5) try: s.connect((host, port)) return True except OSError: return FalseAnd wire these through:
-@pytest.mark.skipif(not _redis_available(), reason="Redis server not available") +@pytest.mark.skipif(not _redis_available(REDIS_HOST, REDIS_PORT), reason="Redis server not available") @@ - RedisObjectStoreClientConfig(bucket_name="test", host="localhost", port=6379, db=0), + RedisObjectStoreClientConfig(bucket_name="test", host=REDIS_HOST, port=REDIS_PORT, db=REDIS_DB),examples/object_store/user_report/README.md (1)
165-168: Set Content-Type for POST/PUT to avoid defaulting to form-encoded.You already switched to
--data-binary(great). AddContent-Typefor correctness with JSON payloads.-- Upload an object: `curl -X POST http://<hostname>:<port>/static/{file_path} --data-binary @{filename}` -– Upsert an object: `curl -X PUT http://<hostname>:<port>/static/{file_path} --data-binary @{filename}` +- Upload an object: `curl -X POST http://<hostname>:<port>/static/{file_path} --data-binary @{filename} -H 'Content-Type: application/json'` +- Upsert an object: `curl -X PUT http://<hostname>:<port>/static/{file_path} --data-binary @{filename} -H 'Content-Type: application/json'`If arbitrary binary files are expected here, consider noting that the header should match the file type instead.
packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (2)
164-166: Avoid REPLACE; prefer INSERT … ON DUPLICATE KEY UPDATE for data row.
REPLACEdeletes then inserts, which can have side effects (triggers, gaps). A straightforward upsert avoids that and keeps FKs stable.- blob = item.model_dump_json() - await cur.execute("REPLACE INTO object_data (id, data) VALUES (%s, %s)", (obj_id, blob)) + blob = item.model_dump_json().encode("utf-8") + await cur.execute( + "INSERT INTO object_data (id, data) VALUES (%s, %s) " + "ON DUPLICATE KEY UPDATE data=VALUES(data)", + (obj_id, blob), + )
188-191: Guard decode path against driver returning str.If the connector ever returns
strfor BLOBs,.decode()raises. A small normalization avoids brittle assumptions.- return ObjectStoreItem.model_validate_json(row[0].decode("utf-8")) + raw = row[0] + if isinstance(raw, str): + return ObjectStoreItem.model_validate_json(raw) + return ObjectStoreItem.model_validate_json(raw.decode("utf-8"))docs/source/extend/object-store.md (2)
90-104: Sample shows async aenter/aexit; annotate aexit and mention context management requirement.Small correctness/readability tweak.
async def __aenter__(self) -> "MyCustomObjectStore": # if async, set up connections to your backend here return self - - async def __aexit__(self, exc_type, exc_value, traceback): + + async def __aexit__(self, exc_type, exc_value, traceback) -> None: # if async, clean up connections to your backend here passAdd a note that registration should use
async withif the store implements async context management (see next comment).
229-244: Minimal skeleton: include missing imports and use async context if applicable.Keeps the example copy-pastable.
-from nat.data_models.object_store import ObjectStoreBaseConfig +from nat.data_models.object_store import ObjectStoreBaseConfig +from nat.builder.builder import Builder +from nat.cli.register_workflow import register_object_store @@ -@register_object_store(config_type=MyCustomObjectStoreConfig) -async def my_custom_object_store(config: MyCustomObjectStoreConfig, _builder: Builder): - - from .my_custom_object_store import MyCustomObjectStore - yield MyCustomObjectStore(**config.model_dump(exclude={"type"})) +@register_object_store(config_type=MyCustomObjectStoreConfig) +async def my_custom_object_store(config: MyCustomObjectStoreConfig, _builder: Builder): + from .my_custom_object_store import MyCustomObjectStore + # If your store implements async context management: + async with MyCustomObjectStore(**config.model_dump(exclude={"type"})) as store: + yield store + # Otherwise, for purely synchronous init: + # yield MyCustomObjectStore(**config.model_dump(exclude={"type"}))src/nat/cli/commands/object_store/object_store.py (3)
45-51: Harden get_object_store_config with a friendly error for unknown store types.Right now a bad store_type yields a KeyError/ImportError. Surface a clear Click error with the allowed values.
Apply this diff:
def get_object_store_config(**kwargs) -> ObjectStoreBaseConfig: """Process common object store arguments and return the config class""" - store_type = kwargs.pop("store_type") - config = STORE_CONFIGS[store_type] + store_type = kwargs.pop("store_type") + try: + config = STORE_CONFIGS[store_type] + except KeyError as e: + raise click.BadParameter( + f"Unknown object store type '{store_type}'. Available: {', '.join(STORE_CONFIGS.keys())}" + ) from e module = importlib.import_module(config["module"]) config_class = getattr(module, config["config_class"]) return config_class(**kwargs)
66-73: Provide a safe default content type when mimetypes lookup returns None.mimetypes.guess_type can return None; set application/octet-stream as fallback to avoid passing None where a string may be expected downstream.
Apply this diff:
- item = ObjectStoreItem(data=data, - content_type=mimetypes.guess_type(str(file_path))[0], + # Determine content type; default to octet-stream if unknown + content_type = mimetypes.guess_type(str(file_path))[0] or "application/octet-stream" + item = ObjectStoreItem(data=data, + content_type=content_type, metadata={ "original_filename": file_path.name, "file_size": str(len(data)), "file_extension": file_path.suffix, "upload_timestamp": str(int(time.time())) })
216-225: Log skipped plugins at debug level for discoverability.Silently swallowing ImportError makes it harder to see why a provider didn’t register. Emit a debug log.
Apply this diff:
- except ImportError: - pass + except ImportError as e: + logger.debug("Skipping '%s' plugin (%s): %s", store_type, config["module"], e)examples/object_store/user_report/configs/config_redis.yml (1)
20-21: Open CORS in example – OK for samples, but warn for prod.allow_origins: ['*'] is fine for demos; tighten in production to specific origins.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (24)
docs/source/extend/object-store.md(4 hunks)docs/source/store-and-retrieve/object-store.md(4 hunks)examples/deploy/README.md(2 hunks)examples/deploy/docker-compose.minio.yml(1 hunks)examples/deploy/docker-compose.mysql.yml(1 hunks)examples/memory/redis/configs/config.yml(1 hunks)examples/object_store/user_report/README.md(5 hunks)examples/object_store/user_report/configs/config_mysql.yml(1 hunks)examples/object_store/user_report/configs/config_redis.yml(1 hunks)examples/object_store/user_report/pyproject.toml(1 hunks)examples/object_store/user_report/upload_to_minio.sh(0 hunks)examples/object_store/user_report/upload_to_mysql.sh(0 hunks)packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py(6 hunks)packages/nvidia_nat_mysql/src/nat/plugins/mysql/object_store.py(1 hunks)packages/nvidia_nat_redis/src/nat/plugins/redis/memory.py(1 hunks)packages/nvidia_nat_redis/src/nat/plugins/redis/object_store.py(1 hunks)packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py(1 hunks)packages/nvidia_nat_redis/src/nat/plugins/redis/register.py(1 hunks)packages/nvidia_nat_redis/tests/test_redis_object_store.py(1 hunks)packages/nvidia_nat_s3/src/nat/plugins/s3/object_store.py(1 hunks)packages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.py(5 hunks)src/nat/cli/commands/object_store/__init__.py(1 hunks)src/nat/cli/commands/object_store/object_store.py(1 hunks)src/nat/cli/entrypoint.py(2 hunks)
💤 Files with no reviewable changes (2)
- examples/object_store/user_report/upload_to_minio.sh
- examples/object_store/user_report/upload_to_mysql.sh
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: willkill07
PR: NVIDIA/NeMo-Agent-Toolkit#649
File: src/nat/cli/commands/object_store/object_store.py:32-42
Timestamp: 2025-08-23T02:55:39.272Z
Learning: In the NAT object store plugin architecture, config classes like S3ObjectStoreClientConfig, MySQLObjectStoreClientConfig, and RedisObjectStoreClientConfig are defined in the object_store.py files (e.g., nat.plugins.s3.object_store), while the implementation classes are in separate files (e.g., nat.plugins.s3.s3_object_store). The registration functions are also in the object_store.py files.
📚 Learning: 2025-08-23T02:55:39.272Z
Learnt from: willkill07
PR: NVIDIA/NeMo-Agent-Toolkit#649
File: src/nat/cli/commands/object_store/object_store.py:32-42
Timestamp: 2025-08-23T02:55:39.272Z
Learning: In the NAT object store plugin architecture, config classes like S3ObjectStoreClientConfig, MySQLObjectStoreClientConfig, and RedisObjectStoreClientConfig are defined in the object_store.py files (e.g., nat.plugins.s3.object_store), while the implementation classes are in separate files (e.g., nat.plugins.s3.s3_object_store). The registration functions are also in the object_store.py files.
Applied to files:
packages/nvidia_nat_redis/src/nat/plugins/redis/object_store.pypackages/nvidia_nat_redis/src/nat/plugins/redis/register.pypackages/nvidia_nat_mysql/src/nat/plugins/mysql/object_store.pypackages/nvidia_nat_s3/src/nat/plugins/s3/s3_object_store.pypackages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.pypackages/nvidia_nat_s3/src/nat/plugins/s3/object_store.pypackages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.pydocs/source/store-and-retrieve/object-store.mdsrc/nat/cli/commands/object_store/object_store.pydocs/source/extend/object-store.md
🪛 LanguageTool
examples/object_store/user_report/README.md
[grammar] ~28-~28: There might be a mistake here.
Context: ...up-api-keys) - Choose an Object Store - Setting up MinIO ...
(QB_NEW_EN)
[grammar] ~29-~29: There might be a mistake here.
Context: ...an-object-store) - Setting up MinIO - Setting up MySQL ...
(QB_NEW_EN)
[grammar] ~30-~30: There might be a mistake here.
Context: ...etting-up-minio) - Setting up MySQL - Setting up Redis -...
(QB_NEW_EN)
[grammar] ~31-~31: There might be a mistake here.
Context: ...etting-up-mysql) - Setting up Redis - Loading Mock Data -...
(QB_NEW_EN)
[grammar] ~32-~32: There might be a mistake here.
Context: ...setting-up-redis) - Loading Mock Data - [NeMo Agent Toolkit File Server](#nemo-ag...
(QB_NEW_EN)
[style] ~73-~73: Consider replacing this word to strengthen your wording.
Context: ...hine. MinIO is an object storage system and acts as drop-in replacement for AWS S3....
(AND_THAT)
[grammar] ~73-~73: There might be a mistake here.
Context: ...IO is an object storage system and acts as drop-in replacement for AWS S3. You ca...
(QB_NEW_EN)
[grammar] ~81-~81: There might be a mistake here.
Context: ...r-compose.minio.yml up -d ``` > [!NOTE] > This is not a secure configuration and...
(QB_NEW_EN)
[grammar] ~100-~100: There might be a mistake here.
Context: ...r-compose.mysql.yml up -d ``` > [!NOTE] > This is not a secure configuration and...
(QB_NEW_EN)
[grammar] ~111-~111: There might be a mistake here.
Context: ...r-compose.redis.yml up -d ``` > [!NOTE] > This is not a secure configuration and...
(QB_NEW_EN)
[grammar] ~165-~165: There might be a mistake here.
Context: ...erver (Optional) - Download an object: curl -X GET http://<hostname>:<port>/static/{file_path} -o {filename} - Upload an object: `curl -X POST http://<...
(QB_NEW_EN)
[grammar] ~166-~166: There might be a mistake here.
Context: ...ath} -o {filename}- Upload an object:curl -X POST http://:/static/{file_path} --data-binary @{filename}- Upsert an object:curl -X PUT http://<h...
(QB_NEW_EN)
[grammar] ~167-~167: There might be a mistake here.
Context: ...binary @{filename}- Upsert an object:curl -X PUT http://:/static/{file_path} --data-binary @{filename}- Delete an object:curl -X DELETE http:/...
(QB_NEW_EN)
[grammar] ~179-~179: There might be a mistake here.
Context: ... three options for running the workflow: 1. Using the S3-compatible object store (`c...
(QB_NEW_EN)
[grammar] ~180-~180: There might be a mistake here.
Context: ...ompatible object store (config_s3.yml) 2. Using the MySQL object store (`config_my...
(QB_NEW_EN)
[grammar] ~181-~181: There might be a mistake here.
Context: ... MySQL object store (config_mysql.yml) 3. Using the Redis object store (`config_re...
(QB_NEW_EN)
[grammar] ~184-~184: There might be a mistake here.
Context: ...hich uses an S3-compatible object store. You can change the configuration file by...
(QB_NEW_EN)
[grammar] ~185-~185: There might be a mistake here.
Context: ... config_mysql.yml for the MySQL server or config_redis.yml for the Redis serv...
(QB_NEW_EN)
examples/deploy/README.md
[grammar] ~37-~37: There might be a mistake here.
Context: ...cludes docker-compose.phoenix.yml for running Phoenix observability server to monitor...
(QB_NEW_EN)
[grammar] ~37-~37: There might be a mistake here.
Context: ...y server to monitor and debug workflows. - Redis Service: Includes `docker-compos...
(QB_NEW_EN)
[grammar] ~43-~43: There might be a mistake here.
Context: ... ## Available Services - minio: docker-compose.minio.yml - mysql: docker-compose.mysql.yml - **`phoeni...
(QB_NEW_EN)
[grammar] ~44-~44: There might be a mistake here.
Context: ...ocker-compose.minio.yml - **mysql**: docker-compose.mysql.yml - **phoenix**: docker-compose.phoenix.yml - **redi...
(QB_NEW_EN)
🪛 markdownlint-cli2 (0.17.2)
docs/source/store-and-retrieve/object-store.md
182-182: Dollar signs used before commands without showing output
(MD014, commands-show-output)
186-186: Dollar signs used before commands without showing output
(MD014, commands-show-output)
190-190: Dollar signs used before commands without showing output
(MD014, commands-show-output)
🔇 Additional comments (16)
examples/object_store/user_report/configs/config_mysql.yml (1)
29-29: No changes needed—loader already supports Bash-style defaultsThe
yaml_load/yaml_loadsfunctions insrc/nat/utils/io/yaml_tools.pyinvoke
_interpolate_variables, which callsexpandvars.expandvarson the full YAML text
before parsing. The third-party expandvars library implements Bash-style expansions,
including the${VAR:-default}syntax. Therefore, yourpassword: ${MYSQL_ROOT_PASSWORD:-my_password}
will be correctly expanded at load time to either the environment value ormy_password.• See
yaml_tools.py(lines 18–25):_interpolate_variables→expandvars.expandvars
• expandvars supports${VAR:-default}and other Bash defaults (pypi.org)examples/memory/redis/configs/config.yml (1)
42-43: Type alignment LGTM (db/port to integers).Matches the tightened RedisMemoryClientConfig types. Please ensure the docs mention that db and port must be integers after this change to avoid breaking existing user configs.
examples/object_store/user_report/pyproject.toml (1)
12-16: Extras Verified in pyproject.tomlAll three
nvidia-natextras (mysql,redis,s3) are present inexamples/object_store/user_report/pyproject.toml(lines 13–15) and the trailing comma in the TOML array is valid. No further changes are required.• File to check: examples/object_store/user_report/pyproject.toml (lines 12–16)
• Confirmed extras match other example projects for consistency.Please remember to verify that your CI pipeline correctly picks up these extras when using the local editable source under
[tool.uv.sources].packages/nvidia_nat_redis/src/nat/plugins/redis/memory.py (1)
36-39: LGTM: lazy imports reduce import-time overhead.Moving
redis.asyncioandLLMFrameworkEnuminside the factory minimizes optional deps at import time.packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py (2)
80-93: LGTM: atomic create semantics via SET NX are correct.Using
SET ... NXensures “put” doesn’t clobber existing keys and maps cleanly toKeyAlreadyExistsError.
73-75: No change needed:.close()is correct for redis-py v4.xThe project’s
pyproject.tomlpins Redis to~=4.3.4, and in redis-py 4.x the asyncio client usesawait client.close()to disconnect the pool. In redis-py v5+,.aclose()was introduced andclose()was deprecated in favor of it—but that only applies once you upgrade past 4.x.References:
- In redis-py v4.3.5, the docs show using
await connection.close()to close the asyncio client (which also closes its pool) (redis-py.readthedocs.io).- In redis-py v5+, the docs and source mark
close()as deprecated and useawait client.aclose()instead (redis-py.readthedocs.io).Given the current dependency constraint, the existing code is correct. If you later bump to redis-py v5+, you can replace
await self._client.close()withawait self._client.aclose().Likely an incorrect or invalid review comment.
packages/nvidia_nat_redis/src/nat/plugins/redis/register.py (1)
22-22: Register-by-import for Redis object-store — LGTMImporting
.object_storehere to trigger registration side-effects is consistent with the plugin pattern used elsewhere. No action needed.src/nat/cli/entrypoint.py (1)
36-36: Wire-up of object-store CLI command — LGTMThe import correctly exposes the new plugin-aware command surface.
docs/source/store-and-retrieve/object-store.md (2)
74-74: Add Redis to included providers — LGTMThe new provider is clearly documented and linked to its implementation path.
148-148: Fixed metadata dict syntax — LGTMThe colon correction avoids a subtle example bug.
packages/nvidia_nat_mysql/src/nat/plugins/mysql/object_store.py (1)
61-61: Signature renaming to _builder — LGTMUnderscore conveys intentional non-use and keeps parity with other providers.
packages/nvidia_nat_redis/tests/test_redis_object_store.py (2)
40-42: Good: integration marker + runtime skip guard keep CI green.The import-time connectivity check with skipif is pragmatic and prevents spurious failures when Redis isn’t running locally.
44-52: Confirm ObjectStoreTests expects an asynccontextmanager named _get_store.Pattern looks right. If ObjectStoreTests awaits/uses this context, we’re good. If you see a “not awaited”/“async generator” warning, wrap the implementation in a concrete async CM class or ensure the base test calls it with “async with self._get_store()”.
Would you like a quick sanity check against nat.test.object_store_tests to confirm the expected method name and signature?
packages/nvidia_nat_redis/src/nat/plugins/redis/object_store.py (2)
23-31: Config shape is clear and aligns with registry expectations.Fields are typed (ints for db/port), and the name="redis" registration key matches the CLI/provider patterns elsewhere.
34-40: Ensure all async initialization runs by context-managing the store (already done).Good use of
async with RedisObjectStore(**config.model_dump(...))so any connection checks/initialization in__aenter__occur deterministically.packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py (1)
156-160: Ensure portable upsert syntax across MySQL ≥ 8.0.19 and MariaDBTo maintain compatibility and correct timestamp semantics, replace the MySQL-only alias form and avoid updating
created_aton conflict. Instead, use the more portableVALUES()reference and updateupdated_at.• File – packages/nvidia_nat_mysql/src/nat/plugins/mysql/mysql_object_store.py, lines 156–160
• Current snippet:INSERT INTO object_meta (path, size) VALUES (%s, %s) AS new ON DUPLICATE KEY UPDATE size=new.size, created_at=CURRENT_TIMESTAMP• Suggested change:
- INSERT INTO object_meta (path, size) - VALUES (%s, %s) AS new - ON DUPLICATE KEY UPDATE size=new.size, created_at=CURRENT_TIMESTAMP + INSERT INTO object_meta (path, size) + VALUES (%s, %s) + ON DUPLICATE KEY UPDATE + size = VALUES(size), + updated_at = CURRENT_TIMESTAMPPlease confirm which database versions we officially support in CI/runtime (MySQL ≥ 8.0.19 only vs. MariaDB compatibility)—that will determine whether the
VALUES()form is mandatory for portability.
Signed-off-by: Will Killian <[email protected]>
Signed-off-by: Will Killian <[email protected]>
Signed-off-by: Will Killian <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
♻️ Duplicate comments (1)
packages/nvidia_nat_redis/tests/test_redis_object_store.py (1)
29-46: Nice: flaky CI guard implemented via fixture + fail_missing override.This addresses the earlier suggestion to gate integration tests instead of relying on comments.
🧹 Nitpick comments (3)
packages/nvidia_nat_redis/tests/test_redis_object_store.py (3)
24-27: Trim Docker how-to comment; rely on the guarded fixture.The skip logic already communicates when Redis isn’t up. Consider shortening or moving these instructions to docs to keep tests lean.
-# NOTE: This test requires a local Redis server to be running. -# To launch a local server using docker, run the following command: -# docker run --rm -ti --name test-redis -p 6379:6379 redis:7-alpine +# Integration tests require a reachable Redis server; see docs for setup guidance.
52-61: Type-hint the async context manager and avoid cross-run key collisions.Add a return type and use a per-run bucket (env override + uuid) to avoid collisions across parallel CI runs.
Add imports (outside this hunk):
@@ -from contextlib import asynccontextmanager +from contextlib import asynccontextmanager +from typing import AsyncIterator +from uuid import uuid4 +import osUpdate the method:
- @asynccontextmanager - async def _get_store(self): + @asynccontextmanager + async def _get_store(self) -> AsyncIterator[object]: async with WorkflowBuilder() as builder: - await builder.add_object_store( - "object_store_name", - RedisObjectStoreClientConfig(bucket_name="test", host="localhost", port=6379, db=0), - ) - - yield await builder.get_object_store_client("object_store_name") + host = os.getenv("NAT_TEST_REDIS_HOST", "localhost") + port = int(os.getenv("NAT_TEST_REDIS_PORT", "6379")) + db = int(os.getenv("NAT_TEST_REDIS_DB", "0")) + bucket = os.getenv("NAT_TEST_REDIS_BUCKET", f"test-{uuid4().hex[:8]}") + await builder.add_object_store( + "object_store_name", + RedisObjectStoreClientConfig(bucket_name=bucket, host=host, port=port, db=0 if db is None else db), + ) + client = await builder.get_object_store_client("object_store_name") + yield client
29-46: Enhance Redis fixture with env-based configuration and timeoutsThe
fail_missingfixture (defined inpackages/nvidia_nat_test/src/nat/test/plugin.py) is already wired to the--fail_missingCLI flag and yields a boolean, so this change can safely leverage it. To prevent long hangs when Redis is unreachable and to allow CI to override host/port/db without touching code, update the Redis fixture inpackages/nvidia_nat_redis/tests/test_redis_object_store.pyas follows:
- Add an
import os- Read
NAT_TEST_REDIS_HOST,NAT_TEST_REDIS_PORT, andNAT_TEST_REDIS_DBwith sensible defaults- Configure
socket_connect_timeoutandsocket_timeoutfor faster failures- Clarify skip messages and include connection details
--- a/packages/nvidia_nat_redis/tests/test_redis_object_store.py +++ b/packages/nvidia_nat_redis/tests/test_redis_object_store.py @@ -from contextlib import asynccontextmanager +from contextlib import asynccontextmanager +import os @pytest.fixture(name="redis_server", scope="module") def fixture_redis_server(fail_missing: bool): """Fixture to safely skip redis based tests if redis is not running""" try: import redis - client = redis.Redis(host="localhost", port=6379, db=0) + host = os.getenv("NAT_TEST_REDIS_HOST", "localhost") + port = int(os.getenv("NAT_TEST_REDIS_PORT", "6379")) + db = int(os.getenv("NAT_TEST_REDIS_DB", "0")) + client = redis.Redis( + host=host, + port=port, + db=db, + socket_connect_timeout=0.5, + socket_timeout=0.5, + ) if not client.ping(): raise RuntimeError("Failed to connect to Redis") yield @@ except ImportError: if fail_missing: raise - pytest.skip("redis not installed, skipping redis tests") + pytest.skip("redis not installed; skipping Redis tests") except Exception as e: if fail_missing: raise - pytest.skip(f"Error connecting to Redis server: {e}, skipping redis tests") + pytest.skip(f"Redis not available at {host}:{port}/db{db}: {e}; skipping Redis tests")
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
packages/nvidia_nat_redis/tests/test_redis_object_store.py(1 hunks)
🧰 Additional context used
📓 Path-based instructions (6)
**/*.py
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
**/*.py: Format Python code with yapf (PEP8 base, column_limit=120) run second in the toolchain
Run ruff (ruff check --fix) using configuration in pyproject.toml; fix warnings unless explicitly ignored; ruff is a linter only (not for formatting)
Respect naming: snake_case for functions and variables, PascalCase for classes, UPPER_CASE for constants
Prefer typing/collections.abc abstractions (e.g., Sequence over list)
Use typing.Annotated for units or extra metadata when useful
Treat pyright warnings (configured in pyproject.toml) as errors during development
Prefer httpx with SSL verification enabled by default for HTTP requests; follow OWASP Top-10
Use async/await for I/O-bound work (HTTP, DB, file reads)
Cache expensive computations with functools.lru_cache or an external cache when appropriate
Leverage NumPy vectorized operations when beneficial
Files:
packages/nvidia_nat_redis/tests/test_redis_object_store.py
**/*.{py,md,sh,yml,yaml,toml,json,ini,cfg,txt,rst,pyi}
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
Indent with 4 spaces (no tabs) and ensure every file ends with a single trailing newline
Files:
packages/nvidia_nat_redis/tests/test_redis_object_store.py
**/*.{py,sh,md,yml,yaml,toml,json,ini,txt,cfg,rst,pyi}
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
**/*.{py,sh,md,yml,yaml,toml,json,ini,txt,cfg,rst,pyi}: Every file must start with the standard SPDX Apache-2.0 header
All source files must include the SPDX Apache-2.0 header template
Files:
packages/nvidia_nat_redis/tests/test_redis_object_store.py
**/*.{py,md}
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
Version numbers are derived by setuptools-scm; never hard-code versions in code or documentation
Files:
packages/nvidia_nat_redis/tests/test_redis_object_store.py
**/*
⚙️ CodeRabbit configuration file
**/*: # Code Review Instructions
- Ensure the code follows best practices and coding standards. - For Python code, follow
PEP 20 and
PEP 8 for style guidelines.- Check for security vulnerabilities and potential issues. - Python methods should use type hints for all parameters and return values.
Example:def my_function(param1: int, param2: str) -> bool: passDocumentation Review Instructions - Verify that documentation and comments are clear and comprehensive. - Verify that the documentation doesn't contain any TODOs, FIXMEs or placeholder text like "lorem ipsum". - Verify that the documentation doesn't contain any offensive or outdated terms. - Verify that documentation and comments are free of spelling mistakes, ensure the documentation doesn't contain any
words listed in the
ci/vale/styles/config/vocabularies/nat/reject.txtfile, words that might appear to be
spelling mistakes but are listed in theci/vale/styles/config/vocabularies/nat/accept.txtfile are OK.Misc. - All code should be licensed under the Apache License 2.0, and should contain an Apache License 2.0 header
comment at the top of each file.
- Confirm that copyright years are up-to date whenever a file is changed.
Files:
packages/nvidia_nat_redis/tests/test_redis_object_store.py
packages/**/*
⚙️ CodeRabbit configuration file
packages/**/*: - This directory contains optional plugin packages for the toolkit, each should contain apyproject.tomlfile. - Thepyproject.tomlfile should declare a dependency onnvidia-nator another package with a name starting
withnvidia-nat-. This dependency should be declared using~=<version>, and the version should be a two
digit version (ex:~=1.0).
- Not all packages contain Python code, if they do they should also contain their own set of tests, in a
tests/directory at the same level as thepyproject.tomlfile.
Files:
packages/nvidia_nat_redis/tests/test_redis_object_store.py
🧠 Learnings (1)
📓 Common learnings
Learnt from: willkill07
PR: NVIDIA/NeMo-Agent-Toolkit#649
File: src/nat/cli/commands/object_store/object_store.py:32-42
Timestamp: 2025-08-23T02:55:39.346Z
Learning: In the NAT object store plugin architecture, config classes like S3ObjectStoreClientConfig, MySQLObjectStoreClientConfig, and RedisObjectStoreClientConfig are defined in the object_store.py files (e.g., nat.plugins.s3.object_store), while the implementation classes are in separate files (e.g., nat.plugins.s3.s3_object_store). The registration functions are also in the object_store.py files.
🧬 Code graph analysis (1)
packages/nvidia_nat_redis/tests/test_redis_object_store.py (2)
examples/object_store/user_report/tests/test_user_report_tool.py (2)
builder(33-49)object_store(53-55)packages/nvidia_nat_redis/src/nat/plugins/redis/object_store.py (1)
RedisObjectStoreClientConfig(23-31)
🔇 Additional comments (2)
packages/nvidia_nat_redis/tests/test_redis_object_store.py (2)
1-14: SPDX header and licensing look correct.Header present, correct year (2025), and Apache-2.0 identifiers in place.
48-51: Markers and class wiring look good.integration mark + usefixtures + inheritance from ObjectStoreTests matches the project’s testing pattern.
|
/merge |
Summary
This introduces a unified, pluggable Object Store layer with a new Redis backend, consistent semantics across S3/MySQL, and a consolidated uploader CLI, making it easier to store and retrieve binary artifacts in NeMo Agent Toolkit workflows.
BREAKING CHANGE
The Redis memory provider schema has changed
Key Features Added
Core Object Store Enhancements
packages/nvidia_nat_redis/src/nat/plugins/redis/redis_object_store.py)SET NX) and explicit upsert pathObjectStoreItemvia pickle to preserve content type and metadataToolkit Integrations
RedisObjectStoreClientConfig(name="redis"), updatedS3ObjectStoreClientConfigandMySQLObjectStoreClientConfig@register_object_store, discoverable byWorkflowBuildernat object-store <type> <type-args...> <upload|delete>commands added to NAT CLIDocumentation Updates
docs/source/extend/object-store.mddocs/source/store-and-retrieve/object-store.mdDevelopment & Testing Improvements
packages/nvidia_nat_redis/tests/test_redis_object_store.pyMigration Notes
nat object-store uploadinsteadOld Schema
New Schema
Technical Implementation Details
Configuration Schema
Key Configuration Parameters
bucket_namehost,port,dbendpoint_url,access_key,secret_key,regionhost,port,username,passwordInfrastructure Changes
packages/nvidia_nat_redis/...packages/nvidia_nat_s3/...andpackages/nvidia_nat_mysql/...Bug Fixes & Improvements
Testing
upsert_objectand preserves metadataSample output of upload:
This PR unifies object storage across providers behind a clear interface, adds a production-ready Redis backend, and streamlines examples and docs—reducing setup friction while improving reliability and consistency.
Closes
By Submitting this PR I confirm:
Summary by CodeRabbit
New Features
Documentation
Examples
Tests
Chores