inject ol_env into ol spark run tags#11786
Conversation
ajyu
changed the title
ajyu
changed the title
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1f3eb913d9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| .contains("_dd.ol_service:databricks.job-cluster.some-run-name") | ||
| } | ||
|
|
||
| def "test setupOpenLineage sets ol_env from dd.env"() { |
There was a problem hiding this comment.
AGENTS.md for this tree explicitly says not to write new Groovy / Spock tests and to migrate touched Groovy tests to JUnit 5. This added Spock test extends AbstractSparkListenerTest.groovy, so the new coverage violates the repository's test policy; please add this coverage in JUnit 5 or migrate the touched test instead.
Useful? React with 👍 / 👎.
|
🎯 Code Coverage (details) 🔗 Commit SHA: cb82b0c | Docs | Datadog PR Page | Give us feedback! |
ajyu
changed the title
ajyu
added
inst: apache spark
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
7762aca
into
master
[do][job] inject ol_env into ol spark run tags move new test to JUnit 5 Co-authored-by: ansha.yu <[email protected]>
2 participants
Fixes DET-109
Adds
_dd.ol_env:<env>to thespark.openlineage.run.tagsset insetupOpenLineage, alongside the existing_dd.ol_service/_dd.ol_app_id.The value comes from
Config.get().getEnv(), and the tag is omitted when envis empty.
Motivation
Downstream (dd-source
lineage-processor) uses the run environment as theSpark application UGP node's
namespace. A node'sentity_idishash(name, type, namespace); today the OpenLineage path and the tracer-onlypath (
djm-span-processor) derivenamespacedifferently, so the same Sparkapplication resolves to two different entities. Carrying env here lets both
paths key the namespace off the same value (env) and converge on one
entity_id._dd.ol_servicealready aligns name; this aligns namespace.When env is unset the tag is omitted so the consumer falls back to the
OpenLineage namespace (no empty-namespace).
Additional Notes
setupOpenLineagetests assert via.contains(...), so the new tagdoesn't affect them; added a test asserting
_dd.ol_env:<env>is present whendd.envis set.