Skip to content

feat(kafka): iam roles anywhere + assume role auth profiles#3606

Merged
yaron2 merged 20 commits into
dapr:mainfrom
sicoyle:feat-iam-roles-anywhere-kafka
Nov 27, 2024
Merged

feat(kafka): iam roles anywhere + assume role auth profiles#3606
yaron2 merged 20 commits into
dapr:mainfrom
sicoyle:feat-iam-roles-anywhere-kafka

Conversation

@sicoyle

@sicoyle sicoyle commented Nov 18, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

This PR:

  • Adds the IAM Roles Anywhere auth profile to Kafka
  • Moves the disjoint Assume Role fields to their own auth profile. This is only used by a few components, so the description is marked as such. However, it is an auth profile that we can add support for in other components, so I've moved it to the auth profile common AWS area to allow us to add this to additional components in the future.
  • Denoted that the originally disjoint Kafka AWS fields become deprecated in Dapr 1.17 as we mv everything to use the common AWS auth profiles that are built in. This meant really gross manual manipulations on the metadata bundle unfortunately...

Metadata bundle changes:
NOTE: This includes all duplicate fields as we deprecate some in the future. Due to this, I've manually made things like accessKey and secretKey that are normally required, not required to not break existing users as we migrate to the common AWS auth fields without the aws prefix.

{
          "title": "AWS: Access Key ID and Secret Access Key",
          "description": "Authenticate using an Access Key ID and Secret Access Key included in the metadata",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "awsAccessKey",
              "description": "Deprecated as of Dapr 1.17. Use 'accessKey' instead.\nIf both fields are set, then 'accessKey' value will be used.\nAWS access key associated with an IAM account.",
              "type": "string",
              "example": "\"AKIAIOSFODNN7EXAMPLE\""
            },
            {
              "name": "awsSecretKey",
              "description": "Deprecated as of Dapr 1.17. Use 'secretKey' instead.\nIf both fields are set, then 'secretKey' value will be used.\nThe secret key associated with the access key.",
              "sensitive": true,
              "type": "string",
              "example": "\"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\""
            },
            {
              "name": "awsSessionToken",
              "description": "Deprecated as of Dapr 1.17. Use 'sessionToken' instead.\nIf both fields are set, then 'sessionToken' value will be used.\nAWS session token to use. A session token is only required if you are using temporary security credentials.",
              "sensitive": true,
              "type": "string",
              "example": "\"TOKEN\""
            },
            {
              "name": "awsIamRoleArn",
              "description": "Deprecated as of Dapr 1.17. Use 'assumeRoleArn' instead.\nIf both fields are set, then 'assumeRoleArn' value will be used.\nIAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "awsStsSessionName",
              "description": "Deprecated as of Dapr 1.17. Use 'sessionName' instead.\nIf both fields are set, then 'sessionName' value will be used.\nRepresents the session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            },
            {
              "name": "awsRegion",
              "description": "The AWS Region where the AWS resource is deployed to.",
              "required": true,
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "accessKey",
              "description": "AWS access key associated with an IAM account",
              "required": true,
              "sensitive": true,
              "example": "\"AKIAIOSFODNN7EXAMPLE\""
            },
            {
              "name": "secretKey",
              "description": "The secret key associated with the access key",
              "required": true,
              "sensitive": true,
              "example": "\"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\""
            },
            {
              "name": "sessionToken",
              "description": "AWS session token to use. A session token is only required if you are using\ntemporary security credentials.",
              "sensitive": true,
              "example": ""
            }
          ]
        },

Note: this is a new auth profile I've made by moving some of the fields here, so this shows some fields that are becoming deprecated:

        {
          "title": "AWS: Assume specific IAM Role",
          "description": "Assume a specific IAM role. Note: This is only supported on Kafka and PostgreSQL components.",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "awsIamRoleArn",
              "description": "Deprecated as of Dapr 1.17. Use 'assumeRoleArn' instead.\nIf both fields are set, then 'assumeRoleArn' value will be used.\nIAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "awsStsSessionName",
              "description": "Deprecated as of Dapr 1.17. Use 'sessionName' instead.\nIf both fields are set, then 'sessionName' value will be used.\nRepresents the session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            },
            {
              "name": "sessionToken",
              "description": "AWS session token to use. A session token is only required if you are using\ntemporary security credentials.",
              "sensitive": true,
              "type": "string",
              "example": "\"TOKEN\""
            },
            {
              "name": "assumeRoleArn",
              "description": "IAM role that has access to AWS resource.\nThis is another option to authenticate with MSK and RDS Aurora aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "sessionName",
              "description": "The session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            }
          ]
        },

Note: This remains as is with the manual manipulation to remove the deprecated fields that are unused for this profile:

        {
          "title": "AWS: Credentials from Environment Variables",
          "description": "Use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the environment",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            }
          ]
        },

Note: new profile added that I manually remove all the fields that we are deprecating bc they are not needed for this auth profile

        {
          "title": "AWS: IAM Roles Anywhere",
          "description": "Use X.509 certificates to establish trust between AWS and your AWS account and the Dapr cluster using AWS IAM Roles Anywhere.",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "trustAnchorArn",
              "description": "ARN of the AWS Trust Anchor in the AWS account granting trust to the Dapr Certificate Authority.",
              "required": true,
              "example": "arn:aws:rolesanywhere:us-west-1:012345678910:trust-anchor/01234568-0123-0123-0123-012345678901"
            },
            {
              "name": "trustProfileArn",
              "description": "ARN of the AWS IAM Profile in the trusting AWS account.",
              "required": true,
              "example": "arn:aws:rolesanywhere:us-west-1:012345678910:profile/01234568-0123-0123-0123-012345678901"
            },
            {
              "name": "assumeRoleArn",
              "description": "ARN of the AWS IAM role to assume in the trusting AWS account.",
              "required": true,
              "example": "arn:aws:iam:012345678910:role/exampleIAMRoleName"
            }
          ]
        }

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #[issue number]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

@sicoyle
sicoyle requested review from a team as code owners November 18, 2024 21:58
@sicoyle sicoyle mentioned this pull request Nov 18, 2024
Closed
3 tasks
@sicoyle

sicoyle commented Nov 19, 2024

Copy link
Copy Markdown
Contributor Author

/ok-to-test

@sicoyle

sicoyle commented Nov 19, 2024

Copy link
Copy Markdown
Contributor Author

unit tests are failing due to wasm issues that are unrelated

@dapr-bot

dapr-bot commented Nov 19, 2024

Copy link
Copy Markdown
Collaborator

Complete Build Matrix

The build status is currently not updated here. Please visit the action run below directly.

🔗 Link to Action run

Commit ref: a1f1c9a

@dapr-bot

dapr-bot commented Nov 19, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Components conformance test

🔗 Link to Action run

Commit ref: a1f1c9a

❌ Some conformance tests failed

These tests failed:

  • bindings.azure.eventgrid

@dapr-bot

dapr-bot commented Nov 19, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Components certification test

🔗 Link to Action run

Commit ref: a1f1c9a

❌ Some certification tests failed

These tests failed:

  • pubsub.aws.snssqs
  • pubsub.gcp.pubsub
  • state.azure.cosmosdb
  • state.sqlserver

famarting
famarting reviewed Nov 19, 2024
View reviewed changes
Comment thread .build-tools/builtin-authentication-profiles.yaml Outdated
famarting
famarting reviewed Nov 19, 2024
View reviewed changes
Comment thread bindings/kafka/metadata.yaml
famarting
famarting reviewed Nov 19, 2024
View reviewed changes
Comment thread common/component/kafka/kafka.go Outdated
famarting
famarting reviewed Nov 19, 2024
View reviewed changes
Comment thread common/component/kafka/kafka.go Outdated
famarting
famarting reviewed Nov 19, 2024
View reviewed changes
Comment thread common/component/kafka/kafka.go Outdated
@famarting

famarting commented Nov 19, 2024

Copy link
Copy Markdown
Contributor

overall lgtm

JoshVanL
JoshVanL requested changes Nov 19, 2024
View reviewed changes
Comment thread common/authentication/aws/x509.go Outdated
@sicoyle

sicoyle commented Nov 20, 2024

Copy link
Copy Markdown
Contributor Author

/ok-to-test

@dapr-bot

dapr-bot commented Nov 20, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Components certification test

🔗 Link to Action run

Commit ref: 16834ed

❌ Some certification tests failed

These tests failed:

  • pubsub.aws.snssqs
  • pubsub.gcp.pubsub
  • state.azure.cosmosdb
  • state.sqlserver

@dapr-bot

dapr-bot commented Nov 20, 2024

Copy link
Copy Markdown
Collaborator

Complete Build Matrix

The build status is currently not updated here. Please visit the action run below directly.

🔗 Link to Action run

Commit ref: 16834ed

@dapr-bot

dapr-bot commented Nov 20, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Components conformance test

🔗 Link to Action run

Commit ref: 16834ed

❌ Some conformance tests failed

These tests failed:

  • bindings.azure.eventgrid
  • state.azure.cosmosdb
  • state.sqlserver

@sicoyle

sicoyle commented Nov 20, 2024

Copy link
Copy Markdown
Contributor Author

final metadata updates

{
          "title": "AWS: Access Key ID and Secret Access Key",
          "description": "Authenticate using an Access Key ID and Secret Access Key included in the metadata",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "awsRegion",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'region' instead.\nThe AWS Region where the AWS Relational Database Service is deployed to.",
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "awsAccessKey",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'accessKey' instead.\nIf both fields are set, then 'accessKey' value will be used.\nAWS access key associated with an IAM account.",
              "type": "string",
              "example": "\"AKIAIOSFODNN7EXAMPLE\""
            },
            {
              "name": "awsSecretKey",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'secretKey' instead.\nIf both fields are set, then 'secretKey' value will be used.\nThe secret key associated with the access key.",
              "sensitive": true,
              "type": "string",
              "example": "\"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\""
            },
            {
              "name": "awsSessionToken",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'sessionToken' instead.\nIf both fields are set, then 'sessionToken' value will be used.\nAWS session token to use. A session token is only required if you are using temporary security credentials.",
              "sensitive": true,
              "type": "string",
              "example": "\"TOKEN\""
            },
            {
              "name": "awsIamRoleArn",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'assumeRoleArn' instead.\nIf both fields are set, then 'assumeRoleArn' value will be used.\nIAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "awsStsSessionName",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'sessionName' instead.\nIf both fields are set, then 'sessionName' value will be used.\nRepresents the session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            },
            {
              "name": "region",
              "description": "The AWS Region where the AWS resource is deployed to.",
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "accessKey",
              "description": "AWS access key associated with an IAM account",
              "sensitive": true,
              "example": "\"AKIAIOSFODNN7EXAMPLE\""
            },
            {
              "name": "secretKey",
              "description": "The secret key associated with the access key",
              "sensitive": true,
              "example": "\"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\""
            },
            {
              "name": "sessionToken",
              "description": "AWS session token to use. A session token is only required if you are using\ntemporary security credentials.",
              "sensitive": true,
              "example": ""
            }
          ]
        },
        {
          "title": "AWS: Assume specific IAM Role",
          "description": "Assume a specific IAM role. Note: This is only supported on Kafka and PostgreSQL components.",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "awsRegion",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'region' instead.\nThe AWS Region where the AWS Relational Database Service is deployed to.",
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "awsIamRoleArn",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'assumeRoleArn' instead.\nIf both fields are set, then 'assumeRoleArn' value will be used.\nIAM role that has access to MSK. This is another option to authenticate with MSK aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "awsStsSessionName",
              "description": "This maintains backwards compatibility with existing fields. \nIt will be deprecated as of Dapr 1.17. Use 'sessionName' instead.\nIf both fields are set, then 'sessionName' value will be used.\nRepresents the session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            },
            {
              "name": "region",
              "description": "The AWS Region where the AWS resource is deployed to.",
              "type": "string",
              "example": "\"us-east-1\""
            },
            {
              "name": "assumeRoleArn",
              "description": "IAM role that has access to AWS resource.\nThis is another option to authenticate with MSK and RDS Aurora aside from the AWS Credentials.",
              "type": "string",
              "example": "\"arn:aws:iam::123456789:role/mskRole\""
            },
            {
              "name": "sessionName",
              "description": "The session name for assuming a role.",
              "type": "string",
              "default": "\"MSKSASLDefaultSession\"",
              "example": "\"MyAppSession\""
            }
          ]
        },
        {
          "title": "AWS: Credentials from Environment Variables",
          "description": "Use AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the environment",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            }
          ]
        },
        {
          "title": "AWS: IAM Roles Anywhere",
          "description": "Use X.509 certificates to establish trust between AWS and your AWS account and the Dapr cluster using AWS IAM Roles Anywhere.",
          "metadata": [
            {
              "name": "authType",
              "description": "Authentication type.\nThis must be set to \"awsiam\" for this authentication profile.",
              "required": true,
              "type": "string",
              "example": "\"awsiam\"",
              "allowedValues": [
                "awsiam"
              ]
            },
            {
              "name": "trustAnchorArn",
              "description": "ARN of the AWS Trust Anchor in the AWS account granting trust to the Dapr Certificate Authority.",
              "required": true,
              "example": "arn:aws:rolesanywhere:us-west-1:012345678910:trust-anchor/01234568-0123-0123-0123-012345678901"
            },
            {
              "name": "trustProfileArn",
              "description": "ARN of the AWS IAM Profile in the trusting AWS account.",
              "required": true,
              "example": "arn:aws:rolesanywhere:us-west-1:012345678910:profile/01234568-0123-0123-0123-012345678901"
            },
            {
              "name": "assumeRoleArn",
              "description": "ARN of the AWS IAM role to assume in the trusting AWS account.",
              "required": true,
              "example": "arn:aws:iam:012345678910:role/exampleIAMRoleName"
            }
          ]
        }

famarting
famarting reviewed Nov 20, 2024
View reviewed changes
Comment thread common/component/kafka/kafka.go
famarting
famarting reviewed Nov 20, 2024
View reviewed changes
Comment thread common/component/kafka/kafka.go Outdated
@sicoyle

sicoyle commented Nov 26, 2024

Copy link
Copy Markdown
Contributor Author

/ok-to-test

@dapr-bot

dapr-bot commented Nov 26, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Components certification test

🔗 Link to Action run

Commit ref: d3d5953

❌ Some certification tests failed

These tests failed:

  • bindings.azure.blobstorage
  • bindings.azure.eventhubs
  • pubsub.aws.snssqs
  • pubsub.gcp.pubsub
  • state.azure.cosmosdb
  • state.sqlserver

@dapr-bot

dapr-bot commented Nov 26, 2024

Copy link
Copy Markdown
Collaborator

Complete Build Matrix

The build status is currently not updated here. Please visit the action run below directly.

🔗 Link to Action run

Commit ref: d3d5953

@dapr-bot

dapr-bot commented Nov 26, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Components conformance test

🔗 Link to Action run

Commit ref: d3d5953

❌ Some conformance tests failed

These tests failed:

  • bindings.azure.eventgrid
  • state.azure.cosmosdb

@sicoyle

sicoyle commented Nov 26, 2024

Copy link
Copy Markdown
Contributor Author

/ok-to-test

@dapr-bot

dapr-bot commented Nov 26, 2024

Copy link
Copy Markdown
Collaborator

Complete Build Matrix

The build status is currently not updated here. Please visit the action run below directly.

🔗 Link to Action run

Commit ref: d3d5953

@dapr-bot

dapr-bot commented Nov 26, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Components conformance test

🔗 Link to Action run

Commit ref: d3d5953

❌ Some conformance tests failed

These tests failed:

  • bindings.azure.eventgrid
  • bindings.azure.eventhubs
  • state.azure.cosmosdb

@dapr-bot

dapr-bot commented Nov 26, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Components certification test

🔗 Link to Action run

Commit ref: d3d5953

❌ Some certification tests failed

These tests failed:

  • bindings.azure.servicebusqueues
  • pubsub.gcp.pubsub
  • state.azure.cosmosdb
  • state.memcached
  • state.sqlserver

@sicoyle
fix: update other closes
7c23636 
Signed-off-by: Samantha Coyle <[email protected]>
@sicoyle
style: last few tweaks
0ce8e18 
Signed-off-by: Samantha Coyle <[email protected]>
@sicoyle

sicoyle commented Nov 27, 2024

Copy link
Copy Markdown
Contributor Author

/ok-to-test

@dapr-bot

dapr-bot commented Nov 27, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Components conformance test

🔗 Link to Action run

Commit ref: 0ce8e18

❌ Some conformance tests failed

These tests failed:

  • bindings.azure.eventgrid
  • state.azure.cosmosdb
  • state.sqlserver

@dapr-bot

dapr-bot commented Nov 27, 2024

Copy link
Copy Markdown
Collaborator

Complete Build Matrix

The build status is currently not updated here. Please visit the action run below directly.

🔗 Link to Action run

Commit ref: 0ce8e18

@dapr-bot

dapr-bot commented Nov 27, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Components certification test

🔗 Link to Action run

Commit ref: 0ce8e18

❌ Some certification tests failed

These tests failed:

  • bindings.azure.eventhubs
  • pubsub.gcp.pubsub
  • state.azure.cosmosdb
  • state.memcached

@sicoyle

sicoyle commented Nov 27, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

this pr is ready pls. All of the certification/conformance test failures are unrelated.

@sicoyle
sicoyle requested review from famarting and yaron2 November 27, 2024 22:31
@yaron2
yaron2 merged commit f48b412 into dapr:main Nov 27, 2024
@sicoyle sicoyle mentioned this pull request Dec 4, 2024
Merged
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yaron2 yaron2 yaron2 approved these changes
@JoshVanL JoshVanL Awaiting requested review from JoshVanL
@famarting famarting Awaiting requested review from famarting

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@sicoyle @dapr-bot @famarting @JoshVanL @yaron2