📝 Description

This PR addresses a few issues that were found in the project creation flow when MLRun is the project Leader:

1. Ensures that project permissions are properly established after project creation and storage operations, by adding a retry mechanism to wait for permission propagation before returning from the API endpoints, preventing race conditions where clients may immediately try to access the newly created project before permissions are fully available.
   1. Note that this doesn't ensure both chief and worker have the updated copies, but it is a best effort and the delay margins between the two should be small enough.
2. When running operations on all followers (such as create/store project), run them on the sorted list of followers. When the followers are defined ['igz', 'nuclio'] this ensures the project policies are created before the project is created on Nuclio.
   1. This is quite hacky, I agree, but until we provide a more robust project leader mechanism it will have to do.
3. Refactor iguazio v4's store_project to decide if it a "create" or "update" based on if create_project raises a 409 Conflict error. If conflict - it is an "update", if not - it is a "create".
   1. This was needed because in the old implementation which used get_project_policy_assignments, we would get a 403 and not a 404, so we couldn't really tell if that 403 is because the project doesn't exist or we really don't have permissions.

🛠️ Changes Made

• Added ensure_project_permissions() method to AuthVerifier (server/py/framework/utils/auth/verifier.py):

  • New async method that retries project read permission checks with a 1-second backoff and 10-second timeout
  • Handles race conditions where the auth provider may not immediately have permissions available after project creation

• Updated project endpoints (server/py/services/api/api/endpoints/projects.py):

  • Added ensure_project_permissions() call after create_project completes (before returning 201)
  • Added ensure_project_permissions() call after store_project completes

• Refactored iguazio's store_project (server/py/framework/utils/clients/iguazio/v4.py):

  • Resolving create or updated according to a 409 Conflict status

• Sorted followers list when running all on all followers (server/py/framework/utils/projects/leader.py):

  • Minimal effort to ensure igz follower operations run before nuclio follwer.

✅ Checklist

• I updated the documentation (if applicable)
• I have tested the changes in this PR
• I confirmed whether my changes are covered by system tests
  • If yes, I ran all relevant system tests and ensured they passed before submitting this PR
  • I updated existing system tests and/or added new ones if needed to cover my changes
• If I introduced a deprecation:
  • I followed the Deprecation Guidelines
  • I updated the relevant Jira ticket for documentation

🧪 Testing

• Manual testing to verify permissions are available immediately after project creation
• Verified retry mechanism properly waits for permission propagation
• Tested both in IG3 and IG4

🔗 References

• Ticket link: https://iguazio.atlassian.net/browse/IG4-1002, https://iguazio.atlassian.net/browse/IG4-1044
• Design docs links:
• External links:

🚨 Breaking Changes?

• Yes (explain below)
• No

🔍️ Additional Notes