feat: Simple OPA support #644

GatewayJ · 2025-10-11T17:23:22Z

Type of Change

Related Issues

Summary of Changes

对于s3请求实现opa对接

Checklist

I have read and followed the CONTRIBUTING.md guidelines
Passed make pre-commit
Added/updated necessary tests
Documentation updated (if needed)
CI/CD passed (if applicable)

Impact

Breaking change (compatibility)
Requires doc/config/deployment update
Other impact:

Additional Notes

Thank you for your contribution! Please ensure your PR follows the community standards (CODE_OF_CONDUCT.md) and sign the CLA if this is your first contribution.

loverustfs · 2025-10-12T02:17:36Z

Hi @GatewayJ ,

Thank you for your contribution. Could you provide some additional documentation?

Copilot

Pull Request Overview

This PR implements OPA (Open Policy Agent) support for RustFS, enabling external policy evaluation for S3 requests. It adds a new policy plugin system that integrates with OPA for authorization decisions.

Adds OPA integration with HTTP client configuration and policy evaluation
Implements async authorization filtering for bucket operations using OPA
Integrates OPA plugin into the IAM system with configuration management

Reviewed Changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
scripts/run.sh	Adds commented OPA configuration environment variables
rustfs/src/storage/ecfs.rs	Replaces synchronous bucket filtering with async stream processing for OPA authorization
crates/utils/src/certs.rs	Adds explicit type annotation for domain_name variable
crates/policy/src/policy/policy.rs	Removes unnecessary blank line
crates/policy/src/policy/opa.rs	New OPA integration module with client, validation, and policy evaluation
crates/policy/src/policy.rs	Exposes opa module publicly
crates/policy/Cargo.toml	Adds dependencies for OPA functionality (rustfs-config, reqwest, chrono, tracing)
crates/obs/Cargo.toml	Adds opa feature to rustfs-config dependency
crates/iam/src/sys.rs	Integrates OPA plugin into IAM system with lazy initialization and authorization checks
crates/iam/src/lib.rs	Removes unnecessary blank line
crates/iam/Cargo.toml	Adds rustfs-config and once_cell dependencies
crates/ecstore/src/config/mod.rs	Removes unnecessary blank line
crates/config/src/opa/mod.rs	New module defining OPA environment variable constants
crates/config/src/lib.rs	Adds opa module under feature flag
crates/config/Cargo.toml	Adds opa feature flag
.vscode/launch.json	Adds OPA configuration to debug environment

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

crates/policy/src/policy/opa.rs

crates/iam/src/sys.rs

crates/policy/src/policy/opa.rs

GatewayJ · 2025-10-12T11:57:04Z

功能概述

此PR为RustFS实现了OPA（开放策略代理）支持，使S3请求能够进行外部策略判断。

插件架构

它添加了一个新的策略插件系统，用于向外部OPA服务发送鉴权请求。

初始化流程

OPA插件在系统IAM初始化时被触发
初始化时检查OPA环境变量配置是否正确
初始化全局reqwest client单例

鉴权流程

请求鉴权时获取OPA client引用
将结构化参数发送给外部OPA服务进行策略评估

该插件需要的环境变量配置为

RUSTFS_POLICY_PLUGIN_URL（该配置如果存在代表opa插件被开启）
RUSTFS_POLICY_PLUGIN_AUTH_TOKEN（可选项）
两个配置均从环境变量中获取，crates/policy/src/policy/opa.rs:check() -> Result<(), String> 和async fn validate(config: &Args) -> Result<(), String> 对环境变量中的参数进行了安全检查

输出给外部opa服务的参数为:

身份信息

account: 用户账户标识符
groups: 用户所属的用户组
is_owner: 所有权状态标志
claims: 额外的身份声明

资源信息

bucket: 目标存储桶名称
object: 目标对象名称（如适用）
arn: 资源的完整ARN（Amazon Resource Name）表示

操作信息

action: 正在执行的具体操作

上下文信息

conditions: 额外的上下文条件
deny_only: 仅拒绝标志
timestamp: RFC3339格式的请求时间戳

{
  "input": {
    "identity": {
      "account": "string",
      "groups": ["string"],
      "is_owner": "boolean",
      "claims": "object"
    },
    "resource": {
      "bucket": "string",
      "object": "string",
      "arn": "string"
    },
    "action": "string",
    "context": {
      "conditions": "object",
      "deny_only": "boolean",
      "timestamp": "string"
    }
  }
}

注意事项

连接参数配置：reqwest client的连接参数目前未通过环境变量控制，使用基于经验和AI的建议确定的默认值。
异步操作处理：所有网络操作均为异步实现，如需在同步上下文中使用需要注意执行器（futures::executor::block_on）选择。

快速开始指南

拉取镜像
podman run --name opa --publish 8181:8181 docker.io/openpolicyagent/opa:0.40.0-rootless run --server --log-format=json-pretty --log-level=debug --set=decision_logs.console=true
准备权限规则文件

cat rego.rego 
package rustfs.authz

import input

default allow = false

# 允许root用户执行所有操作
allow {
 input.identity.is_owner == true
}

# 其他用户不能执行PutObject操作
allow {
 input.action != "s3:PutObject"
 input.identity.is_owner == false
}

将配置规则应用到opa服务容器

 curl -X PUT --data-binary @rego.rego   localhost:8181/v1/policies/putobject

rustfs的配置为

                "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow"
                "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"

使用s3cmd类似的工具验证即可，注意需要使用子账户的ak sk 。

Copilot

Pull Request Overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

crates/policy/src/policy/opa.rs

crates/iam/src/sys.rs

rustfs/src/storage/ecfs.rs

crates/obs/Cargo.toml

scripts/run.sh

loverustfs · 2025-10-13T03:06:32Z

Could you please add some more test cases?

Great! Thank you.

crates/policy/Cargo.toml

crates/iam/src/sys.rs

houseme

If it is an unused dependency crate, please remove it, thank you

GatewayJ · 2025-10-14T14:15:06Z

Could you please add some more test cases?

Great! Thank you.

@loverustfs Does it refer to the code for unit testing

GatewayJ · 2025-10-14T16:55:39Z

If it is an unused dependency crate, please remove it, thank you

@houseme I've been a bit busy lately and my code is a bit sloppy. Sorry that. I have made some modifications to the code, please review it carefully

crates/policy/src/policy/opa.rs

crates/iam/Cargo.toml

Copilot

Pull Request Overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 4 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

crates/policy/src/policy/opa.rs

Co-authored-by: Copilot <[email protected]>

Copilot

Pull Request Overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 5 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

crates/policy/src/policy/opa.rs

crates/iam/src/sys.rs

crates/policy/src/policy/opa.rs

Copilot · 2025-10-15T07:13:00Z

crates/iam/src/sys.rs

+            match opa::lookup_config().await {
+                Ok(conf) => {
+                    if conf.enable() {
+                        Self::set_policy_plugin_client(opa::AuthZPlugin::new(conf)).await;
+                        info!("OPA plugin enabled");
+                    }
+                }
+                Err(e) => {
+                    error!("Error loading OPA configuration err:{}", e);
+                }
+            };


The spawned task is not awaited or handled, which means initialization errors could be silently ignored and the OPA client might not be properly initialized before authorization requests are made. Consider using a proper initialization mechanism or at least storing the task handle.

Shouldn't OPA initialization errors affect the main process? Provide error logs for easy troubleshooting?

Co-authored-by: Copilot <[email protected]>

GatewayJ · 2025-10-15T15:28:56Z

@houseme I have resolved some issues, please also pay attention to some Copilot comments

houseme

@GatewayJ Thank you. Please add the content related to 'Copyright' in the header of the new file.

GatewayJ · 2025-10-16T01:58:22Z

@GatewayJ Thank you. Please add the content related to 'Copyright' in the header of the new file.

already done

GatewayJ force-pushed the feature-opa branch from a8a9592 to 2540ca8 Compare October 11, 2025 17:26

houseme requested a review from Copilot October 12, 2025 03:07

Copilot AI reviewed Oct 12, 2025

View reviewed changes

crates/policy/src/policy/opa.rs Outdated Show resolved Hide resolved

crates/iam/src/sys.rs Outdated Show resolved Hide resolved

crates/policy/src/policy/opa.rs Outdated Show resolved Hide resolved

crates/policy/src/policy/opa.rs Outdated Show resolved Hide resolved

GatewayJ force-pushed the feature-opa branch 3 times, most recently from 7bab58e to 04f70d4 Compare October 12, 2025 11:18

houseme requested a review from Copilot October 13, 2025 03:00

Copilot AI reviewed Oct 13, 2025

View reviewed changes

crates/policy/src/policy/opa.rs Outdated Show resolved Hide resolved

crates/iam/src/sys.rs Show resolved Hide resolved

rustfs/src/storage/ecfs.rs Show resolved Hide resolved

houseme reviewed Oct 13, 2025

View reviewed changes

crates/obs/Cargo.toml Outdated Show resolved Hide resolved

houseme reviewed Oct 13, 2025

View reviewed changes

scripts/run.sh Outdated Show resolved Hide resolved

houseme reviewed Oct 13, 2025

View reviewed changes

crates/policy/Cargo.toml Outdated Show resolved Hide resolved

houseme reviewed Oct 13, 2025

View reviewed changes

crates/iam/src/sys.rs Outdated Show resolved Hide resolved

houseme reviewed Oct 13, 2025

View reviewed changes

GatewayJ force-pushed the feature-opa branch 3 times, most recently from 6c4b344 to 3c9ffa5 Compare October 14, 2025 16:28

opa-feature

3b7a242

GatewayJ force-pushed the feature-opa branch from 3c9ffa5 to 3b7a242 Compare October 14, 2025 16:37

houseme reviewed Oct 15, 2025

View reviewed changes

crates/policy/src/policy/opa.rs Outdated Show resolved Hide resolved

houseme reviewed Oct 15, 2025

View reviewed changes

crates/policy/src/policy/opa.rs Outdated Show resolved Hide resolved

houseme reviewed Oct 15, 2025

View reviewed changes

crates/iam/Cargo.toml Outdated Show resolved Hide resolved

houseme requested a review from Copilot October 15, 2025 01:42

Merge branch 'main' into feature-opa

6ecd6df

Copilot AI reviewed Oct 15, 2025

View reviewed changes

crates/policy/src/policy/opa.rs Outdated Show resolved Hide resolved

crates/policy/src/policy/opa.rs Outdated Show resolved Hide resolved

crates/policy/src/policy/opa.rs Outdated Show resolved Hide resolved

crates/policy/src/policy/opa.rs Outdated Show resolved Hide resolved

Update crates/policy/src/policy/opa.rs

a34982e

Co-authored-by: Copilot <[email protected]>

Update crates/policy/src/policy/opa.rs

c8d49af

Co-authored-by: Copilot <[email protected]>

houseme requested a review from Copilot October 15, 2025 07:11

Copilot AI reviewed Oct 15, 2025

View reviewed changes

Update crates/policy/src/policy/opa.rs

e42db23

Co-authored-by: Copilot <[email protected]>

GatewayJ force-pushed the feature-opa branch from 22a1cbb to e42db23 Compare October 15, 2025 14:42

GatewayJ and others added 2 commits October 15, 2025 23:00

Apply suggestion from @Copilot

5e16534

Co-authored-by: Copilot <[email protected]>

Merge branch 'main' into feature-opa

c59da7a

houseme reviewed Oct 15, 2025

View reviewed changes

GatewayJ added 2 commits October 16, 2025 09:47

add the content related to 'Copyright'

92e9300

add the content related to 'Copyright'

9bfdb18

weisd merged commit aae768f into rustfs:main Oct 16, 2025
14 checks passed

feat: Simple OPA support #644

feat: Simple OPA support #644

Uh oh!

Conversation

GatewayJ commented Oct 11, 2025

Type of Change

Related Issues

Summary of Changes

Checklist

Impact

Additional Notes

Uh oh!

loverustfs commented Oct 12, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GatewayJ commented Oct 12, 2025

功能概述

插件架构

初始化流程

鉴权流程

该插件需要的环境变量配置为

输出给外部opa服务的参数为:

注意事项

快速开始指南

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

loverustfs commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

houseme left a comment

Choose a reason for hiding this comment

Uh oh!

GatewayJ commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GatewayJ commented Oct 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

GatewayJ Oct 15, 2025

Choose a reason for hiding this comment

Uh oh!

loverustfs commented Oct 13, 2025 •

edited

Loading

GatewayJ commented Oct 14, 2025 •

edited

Loading

houseme left a comment •

edited

Loading