Crawler for Externally Orchestrated Jobs with Failing Configuration#395
Crawler for Externally Orchestrated Jobs with Failing Configuration#395
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #395 +/- ##
==========================================
- Coverage 85.67% 82.61% -3.07%
==========================================
Files 42 30 -12
Lines 5311 2490 -2821
Branches 969 445 -524
==========================================
- Hits 4550 2057 -2493
+ Misses 542 326 -216
+ Partials 219 107 -112 ☔ View full report in Codecov by Sentry. |
nfx
left a comment
nfx
left a comment
There was a problem hiding this comment.
needs a passing integration test. please attach a screenshot once you get it to work locally
| failures: str | ||
|
|
||
|
|
||
| @dataclass |
There was a problem hiding this comment.
Do we want to capture only the failing ones or all of them with the failing one anotated?
| def is_custom_image(version_string: str): | ||
| """ | ||
| Is this a custom version? | ||
| """ |
There was a problem hiding this comment.
Does it require implementation?
| pattern = r"^(?P<major>\d+)?\.(?P<minor>\d+)?\.(?P<patch>[\dx]+)?.*" | ||
| lvg = re.match(pattern, left_version) | ||
| rvg = re.match(pattern, right_version) | ||
| left = (int(lvg.group("major")), int(lvg.group("minor"))) |
There was a problem hiding this comment.
Do we have a proper unit test for that?
|
|
||
|
|
||
| def get_job_cluster_from_task( | ||
| task: RunTask, job_run: BaseRun, all_clusters: dict[str, ClusterDetails] |
There was a problem hiding this comment.
Re factor job cluster to use the same mechanism
| self._ws = ws | ||
|
|
||
| def _crawl(self) -> list[ExternallyOrchestratedJobRunWithFailingConfiguration]: | ||
| no_of_days_back = datetime.timedelta(days=30) # todo make configurable in yaml? |
There was a problem hiding this comment.
Make timedelta externally configurable
zpappa
force-pushed
the
feature/external-orchestrator-job-run-crawler
branch
from
02dfe85 to
ea103b4
Compare
zpappa
had a problem deploying
to
account-admin
GitHub Actions
Failure
zpappa
force-pushed
the
feature/external-orchestrator-job-run-crawler
branch
from
ea103b4 to
14daf86
Compare
zpappa
had a problem deploying
to
account-admin
GitHub Actions
Failure
|
|
…estrator-job-run-crawler # Conflicts: # src/databricks/labs/ucx/assessment/crawlers.py # tests/unit/assessment/test_assessment.py
renardeinside
had a problem deploying
to
account-admin
GitHub Actions
Failure
7 participants
Resolves #266
Added ExternallyOrchestratedJobsWithFailingConfigCrawler
Added a crawler to look at JobRuns from the SDK and determine which of the job runs are from the RunsSubmit API
Added Unit Tests
Added tests to cover basic logic and some edge cases
Integration Tests Pending