[FEATURE] Allow separation of nested jobs in different bundle files. #1535

Describe the issue

My team has adopted the recent complex variable addition from 0.222.0 which has helped a bit for cleaning up our main bundle file. However, I was trying to split up our job resources into their own files and I had an issue. Our jobs are highly nested with one job calling other jobs through run_job_task and it seems if the sub-job didn't exist in the "root module" then it would throw an error during deployment.

Configuration

make a file structure like

- repo
| - bundle.yml
| - bundle.resources.job2.yml - contains job definition of job2 only
| - bundle.resources.job1.yml - contains job definition of job1 only

in bundle.yml include all bundle*.yml

in job2 have it perform a run_job_task of job1 referencing it by ${resources.jobs.job1.id}

job1 can just be a simple run of a notebook

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

1. Run databricks bundle deploy
2. See error

Expected Behavior

I expected that all of the resources would be merged together so that (in the above example) job2 would be able to reference job1 and deploy without issue.

Actual Behavior

You get an error like the following which says the sub job must be defined in the root module (I assume the root bundle file):

Error: Reference to undeclared resource

  on bundle.tf.json line 571, in resource.databricks_job.wf_run_update.task[1].run_job_task:
 571:               "job_id": "${databricks_job.wf_dimensions.id}"

A managed resource "databricks_job" "wf_dimensions" has not been declared in
the root module.

(this error I copied from our actual error rather than going by the example configuration I gave above).

OS and CLI version

macos 14.3.1 (23D60)
0.222.0

Is this a regression?

n/a

Debug Logs

can provide if needed