[RemoteRuntime] Secret token mounting for RemoteRuntime #9050

elbamit · 2025-12-14T09:04:54Z

📝 Description

Add secret token mounting for RemoteRuntime functions.
Save auth as part of the function's spec. Use it to mount the secret into the runtime during function config compilation.

🛠️ Changes Made

Added auth as part of NuclioSpec, ApplicationSpec and ServingSpec
Enrich auth token name on the function during deployment, then mount secret to runtime during function config compilation.
Refactor mount_secret_token_to_runtime to remove existing auth secret volumes/mounts
Refactor _enrich_and_validate_auth_token_name to enrich_and_validate_auth_token_name_on_object to support both runObject and remoteRuntime

✅ Checklist

I updated the documentation (if applicable)
I have tested the changes in this PR
I confirmed whether my changes are covered by system tests
- If yes, I ran all relevant system tests and ensured they passed before submitting this PR
- I updated existing system tests and/or added new ones if needed to cover my changes
If I introduced a deprecation:
- I followed the Deprecation Guidelines
- I updated the relevant Jira ticket for documentation

🧪 Testing

Unit tests + manual tests of deployment of nuclio/application/serving

🔗 References

Ticket link: https://iguazio.atlassian.net/browse/ML-11584
Design docs links:
External links:

🚨 Breaking Changes?

Yes (explain below)
No

🔍️ Additional Notes

As part of https://iguazio.atlassian.net/browse/ML-11599, need to handle redeployment of a function with a different auth token name.

liranbg

Some suggestions and renamings, overall very well. I suggest to run some nuclio system tests to see how it plays and no regressions were made.

liranbg · 2025-12-14T19:02:06Z

server/py/services/api/launcher.py

    # TODO In ML-11600, implement token name resolution and validation + tests
-    def _enrich_and_validate_auth_token_name(self, run: mlrun.run.RunObject):
-        auth = run.spec.auth or {}
+    def enrich_and_validate_auth_token_name_on_object(


Suggested change

def enrich_and_validate_auth_token_name_on_object(

def enrich_and_validate_auth_token_name(

why renaming? also, function name should reflect what it does (enriches/validates the auth token) the added information is subjected to the argument - which can be omitted from the name for redundancy or (TMI)

liranbg · 2025-12-14T19:07:07Z

server/py/services/api/tests/unit/runtimes/test_nuclio.py

                            else url.startswith("http")
                        )

+    def test_compile_function_config_iguazio_v4_mounts_auth_secret(self):


Suggested change

def test_compile_function_config_iguazio_v4_mounts_auth_secret(self):

def test_compile_function_config_with_auth_secret(self):

liranbg · 2025-12-14T19:10:41Z

server/py/services/api/tests/unit/crud/test_secrets.py

+    runtime.spec.volumes = initial_volumes.copy()
+    runtime.spec.volume_mounts = initial_mounts.copy()


why do you need to copy them?

liranbg · 2025-12-14T19:15:58Z

server/py/services/api/launcher.py

+    def enrich_and_validate_auth_token_name_on_object(
+        self, object: Union[mlrun.run.RunObject, mlrun.runtimes.RemoteRuntime]
+    ):
+        # Ensure the auth dictionary exists


Suggested change

# Ensure the auth dictionary exists

code is self explanatory

liranbg · 2025-12-14T19:17:54Z

server/py/services/api/launcher.py

+        # Resolve token name
+        if provided_token_name:
+            token_name = provided_token_name
+            self._validate_token_name(token_name, raise_error_on_failure=True)
+        else:
+            token_name = "default"
+            self._validate_token_name(token_name, raise_error_on_failure=False)


is default token name a constant? I think it should be
also, comment on why you raise an exception?

Suggested change

# Resolve token name

if provided_token_name:

token_name = provided_token_name

self._validate_token_name(token_name, raise_error_on_failure=True)

else:

token_name = "default"

self._validate_token_name(token_name, raise_error_on_failure=False)

# explain me

raise_error_on_failure = bool(provided_token_name)

token_name = provided_token_name or "default"

self._validate_token_name(token_name, raise_error_on_failure=False)

@quaark I can use mlconf.auth_with_oauth_token.token_name, however if user sets this env var prior to importing mlrun, then runs/deploy a function, the token resolution logic will use that instead.
I assume we don't want that behavior right?

Ok so we'll make it a constant not a config

liranbg · 2025-12-14T19:20:09Z

server/py/services/api/crud/secrets.py

+        for vol in volumes:
+            if "secret" in vol:
+                secret = vol["secret"]
+                secret_name = secret.get("secretName", "")
+
+                # Pattern of auth secret volumes
+                if secret_name.startswith("mlrun-auth-secrets"):
+                    volumes_to_remove.add(vol["name"])


Suggested change

for vol in volumes:

if "secret" in vol:

secret = vol["secret"]

secret_name = secret.get("secretName", "")

# Pattern of auth secret volumes

if secret_name.startswith("mlrun-auth-secrets"):

volumes_to_remove.add(vol["name"])

for volume in volumes:

if secret := volume.get("secret"):

secret_name = secret.get("secretName", "")

if secret_name.startswith("mlrun-auth-secrets"):

volumes_to_remove.add(volume["name"])

shorter, with full name

@rokatyy's suggestion is more robust. Using her's instead

liranbg · 2025-12-14T19:22:31Z

server/py/services/api/crud/secrets.py

+        runtime.spec.volumes = [
+            v for v in volumes if v["name"] not in volumes_to_remove
+        ]


Suggested change

runtime.spec.volumes = [

v for v in volumes if v["name"] not in volumes_to_remove

]

runtime.spec.volumes = [

volume

for volume in volumes

if volume["name"] not in volumes_to_remove

]

use full name, makes it more readable

liranbg · 2025-12-14T19:22:37Z

server/py/services/api/crud/secrets.py

+
+        # Filter out matching mounts
+        runtime.spec.volume_mounts = [
+            m for m in mounts if m["name"] not in volumes_to_remove


make it readable

liranbg · 2025-12-14T19:25:36Z

server/py/services/api/crud/secrets.py

        )

+        # Remove any existing auth secret volumes/mounts
+        services.api.crud.secrets.Secrets._remove_auth_secret_volumes(runtime)


seeing these calls (static method) from within same class static method) shows me that the decision to put the logic under Secrets class that it was a bit wrong.
I think we need to change it and move logic under runtime/etc. then we could do

runtime.enrich_and_validate()
function.remove_auth_secret_volumes()

using self, etc

liranbg · 2025-12-14T19:26:13Z

server/py/services/api/crud/runtimes/nuclio/function.py

+        function = services.api.crud.secrets.Secrets.mount_secret_token_to_runtime(
+            function,
+            token_name=function.spec.auth.get("token_name"),
+            username=auth_info.username,
+        )


e.g for https://github.com/mlrun/mlrun/pull/9050/changes#r2617421254

function.mount_secret_token_to_runtime(token_name, auth_info)

rokatyy

minor suggestions

server/py/services/api/crud/secrets.py

…umes' logic from outside the 'Secrets' class into the baseRuntime and the BaseRuntimeHandler. The '_mount_secret_token_to_runtime' is now used inside the 'add_k8s_secrets_to_spec' method which is called both from the runtime handler flow and the nuclio deployment flow. Tests have been updated respectively to match the new functions locations (however not refactored based on other PR comments)

… the mlrun.utils.get_in method, using constant for auth secret name prefix and using full name for variables. The used constant returns a trailing dot in the prefix name which allowed me to discover that the examples in the tests didn't perfectly match real life - so tests have been adjusted accordingly

…d changes. Refactor enrich_and_validate_auth_token_name to run on ig4 only, removed if/else condition and used constant name for default token name

rokatyy

LGTM

rokatyy · 2025-12-16T09:37:20Z

server/py/services/api/crud/runtimes/nuclio/function.py

        silent=True
    ).is_running_inside_kubernetes_cluster():
-        _add_secrets_config_to_function_spec(function)
+        token_name = mlrun.utils.get_in(function.spec, "auth.token_name", None)


do we want to skip if token is empty? Or will it be handled later in the flow?

Later in the flow it will try to list the secrets matching that token, but there won't be such, so we won't perform the mounting.

…ig4 system

…tion was updated to only run on ig4 mode

liranbg

Looks much much better. minor suggestion to keep the coding better

liranbg · 2025-12-17T08:01:54Z

server/py/services/api/crud/runtimes/nuclio/function.py



-def _resolve_env_vars(function, auth_info=None):
+def _enrich_config_spec(function, username):


why not passing auth_info entirely? a better practice as you might late on use its other fields

liranbg · 2025-12-17T08:02:18Z

server/py/services/api/runtime_handlers/base.py

        runtime: mlrun.runtimes.pod.KubeResource,
        project_name: Optional[str] = None,
+        token_name: Optional[str] = None,
+        username: Optional[str] = None,


pass auth_info

liranbg · 2025-12-17T08:02:25Z

server/py/services/api/runtime_handlers/base.py

        project_name: Optional[str] = None,
        encode_key_names: bool = True,
+        token_name: Optional[str] = None,
+        username: Optional[str] = None,


pass auth_info

liranbg · 2025-12-17T08:02:31Z

server/py/services/api/runtime_handlers/base.py


+    @staticmethod
+    def _mount_secret_token_to_runtime(
+        runtime: mlrun.runtimes.base.BaseRuntime, token_name: str, username: str


pass auth_info

liranbg · 2025-12-17T08:03:08Z

server/py/services/api/runtime_handlers/kubejob.py

+            runtime,
+            project_name=run.metadata.project,
+            token_name=(run.spec.auth or {}).get("token_name"),
+            username=auth_info.username,


simply pass the auto info - it allows the function to extend and use the class capabilities later on when we want to identify user not just by username but even perhaps its group

…ow extension in the class capabilities in the future. Fixed tests accordingly.

Add secret token auth to remoteRuntime functions

2263d79

github-actions bot added area/sdk area/server area/api labels Dec 14, 2025

elbamit added 4 commits December 14, 2025 14:58

add auth parameter to servingspec and applicationspec

454feae

lint

36f44cb

add test for _remove_auth_secret_volumes

3601a33

lint

e76ba29

elbamit marked this pull request as ready for review December 14, 2025 18:27

elbamit requested review from a team, TomerShor and adamdelman as code owners December 14, 2025 18:27

liranbg requested changes Dec 14, 2025

View reviewed changes

rokatyy reviewed Dec 15, 2025

View reviewed changes

server/py/services/api/crud/secrets.py Outdated Show resolved Hide resolved

server/py/services/api/crud/secrets.py Outdated Show resolved Hide resolved

elbamit added 3 commits December 15, 2025 16:06

Renaming of functions and variable for better clarity, minor fixes an…

cddb0d1

…d changes. Refactor enrich_and_validate_auth_token_name to run on ig4 only, removed if/else condition and used constant name for default token name

github-actions bot added the area/tests label Dec 16, 2025

lint

a58e5cb

rokatyy approved these changes Dec 16, 2025

View reviewed changes

elbamit added 2 commits December 16, 2025 13:14

Fix failing test. Ensure _mount_secret_token_to_runtime only runs in …

f71b7eb

…ig4 system

Fix failing test by setting authentication mode to ig4 since the func…

e41f33b

…tion was updated to only run on ig4 mode

liranbg requested changes Dec 17, 2025

View reviewed changes

elbamit added 2 commits December 17, 2025 10:38

Refactor functions using 'username' to use 'auth_info' instead to all…

1d88e65

…ow extension in the class capabilities in the future. Fixed tests accordingly.

lint

cc2b3d1

liranbg approved these changes Dec 17, 2025

View reviewed changes

Merge branch 'development' into ML-11584

3a2747a

liranbg merged commit e779888 into mlrun:development Dec 17, 2025
13 checks passed

	def enrich_and_validate_auth_token_name_on_object(
	def enrich_and_validate_auth_token_name(

	def test_compile_function_config_iguazio_v4_mounts_auth_secret(self):
	def test_compile_function_config_with_auth_secret(self):

		runtime.spec.volumes = initial_volumes.copy()
		runtime.spec.volume_mounts = initial_mounts.copy()



		def _resolve_env_vars(function, auth_info=None):
		def _enrich_config_spec(function, username):

[RemoteRuntime] Secret token mounting for RemoteRuntime #9050

[RemoteRuntime] Secret token mounting for RemoteRuntime #9050

Uh oh!

Conversation

elbamit commented Dec 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Description

🛠️ Changes Made

✅ Checklist

🧪 Testing

🔗 References

🚨 Breaking Changes?

🔍️ Additional Notes

Uh oh!

liranbg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elbamit Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rokatyy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rokatyy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liranbg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

elbamit commented Dec 14, 2025 •

edited

Loading

elbamit Dec 15, 2025 •

edited

Loading