Skip to content

Upgrade Scio to 0.14.12; run CI on Java 17; use magnolify-bigquery in BigDiffy#784

Merged
clairemcginty merged 5 commits into
masterfrom
scio-0.14.12
Feb 12, 2025
Merged

Upgrade Scio to 0.14.12; run CI on Java 17; use magnolify-bigquery in BigDiffy#784
clairemcginty merged 5 commits into
masterfrom
scio-0.14.12

Conversation

@clairemcginty

Copy link
Copy Markdown
Contributor
  • Upgrades Scio to 0.14.12, which unblocks Java 17+
  • Updates CI to run on Java 17 (thus the added JavaOptions.scala file, which adds options required for 17+). Note that the project still has the -target 8 scalaOpt so it will continue to produce Java8-compatible bytecode
  • Replaces BigQueryType usage in BigDiffy with magnolify-bigquery. BigQueryType is deprecated and less performant than Magnolify for BQ writes

Testing: Packaged locally and tested BigDiffy with BQ output to validate that all three tables (_fields, _global, and _keys) are produced correctly.

@codecov

codecov Bot commented Feb 10, 2025

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.

Project coverage is 70.68%. Comparing base (b4e0ddd) to head (f4ec64c).
Report is 6 commits behind head on master.

Files with missing lines Patch % Lines
...in/scala/com/spotify/ratatool/diffy/BigDiffy.scala 0.00% 6 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #784      +/-   ##
==========================================
+ Coverage   70.61%   70.68%   +0.06%     
==========================================
  Files          44       44              
  Lines        1865     1859       -6     
  Branches      309      315       +6     
==========================================
- Hits         1317     1314       -3     
+ Misses        548      545       -3     
Flag Coverage Δ
ratatoolCli 2.89% <0.00%> (+0.01%) ⬆️
ratatoolCommon 0.00% <ø> (ø)
ratatoolDiffy 32.29% <0.00%> (+0.12%) ⬆️
ratatoolExamples 17.20% <0.00%> (+0.05%) ⬆️
ratatoolSampling 60.53% <ø> (-0.11%) ⬇️
ratatoolScalacheck 77.94% <ø> (ø)
ratatoolShapeless 4.16% <0.00%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@benkonz

benkonz commented Feb 11, 2025

Copy link
Copy Markdown
Contributor

thanks @clairemcginty! I tested your changes with BigSampler and BigDiffy. Both succeeded, and the changes look good to me. TY!

@benkonz

benkonz commented Feb 12, 2025

Copy link
Copy Markdown
Contributor

also tested with BigQuery in BigSampler here, also successful!

@clairemcginty

Copy link
Copy Markdown
Contributor Author

also tested with BigQuery in BigSampler here, also successful!

amazing, thanks Ben! 🎉

@benkonz benkonz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@clairemcginty
clairemcginty merged commit 0e1fa3d into master Feb 12, 2025
@clairemcginty
clairemcginty deleted the scio-0.14.12 branch February 12, 2025 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants