Skip to content

fix: Place PRIMARY KEY inside the column list in BigQuery monitoring DDL - #7021

Open
liron-velocity wants to merge 1 commit into
feast-dev:masterfrom
liron-velocity:fix/bigquery-monitoring-primary-key
Open

liron-velocity wants to merge 1 commit into
feast-dev:masterfrom
liron-velocity:fix/bigquery-monitoring-primary-key

Conversation

@liron-velocity

Copy link
Copy Markdown

What this PR does / why we need it:

_bq_ensure_monitoring_tables puts PRIMARY KEY (...) NOT ENFORCED after the closing parenthesis of the column list in all four data quality monitoring CREATE TABLE IF NOT EXISTS statements. GoogleSQL only accepts table constraints inside the table element list (( column | constraint_definition [, ...] )), so BigQuery rejects each statement at parse time:

400 PRIMARY KEY must be defined in the table element list as column attribute or constraint.

As a result, BigQuery monitoring (feast monitor run, the /monitoring REST endpoints, and auto_baseline on feast apply) fails on every BigQuery deployment. This has been the case since v0.64.0.

This PR:

  • moves each PRIMARY KEY (...) NOT ENFORCED inside its column list, after a trailing comma on the last column. The key columns are unchanged, and nothing else in the file changes.
  • adds test_ensure_monitoring_tables_primary_key_inside_column_list to sdk/python/tests/unit/infra/offline_stores/test_bigquery.py. It patches _get_bigquery_client, captures the DDL sent to client.query, and asserts two things: all four monitoring tables are created, and in each CREATE TABLE the PRIMARY KEY ... NOT ENFORCED clause sits inside the outermost parentheses, with nothing after the final ). Until now, no test covered this function: the monitoring integration test mocks ensure_monitoring_tables.

The other offline stores (Redshift, Snowflake, Postgres, Oracle) already put the constraint inside the element list, so only BigQuery needed the fix.

Which issue(s) this PR fixes:

Fixes #7020

Checks

  • I've made sure the tests are passing.
  • My commits are signed off (git commit -s)
  • My PR title follows conventional commits format

Testing Strategy

  • Unit tests
  • Integration tests
  • Manual tests
  • Testing is not required for this change

Unit test, before and after the fix (pytest sdk/python/tests/unit/infra/offline_stores/test_bigquery.py):

  • Without the fix: 1 failed, 16 passed. The new test fails because the feast_monitoring_feature_metrics DDL ends with ... NOT ENFORCED instead of ).
  • With the fix: 17 passed

Lint: ruff check, ruff format --check (ruff 0.16.10, the pinned CI version) on sdk/python/feast/ and sdk/python/tests/, and mypy on both changed files. All clean.

Manual: a BigQuery dry run confirms the parser behavior, and the result doesn't depend on location:

bq query --dry_run --use_legacy_sql=false \
  'CREATE TABLE IF NOT EXISTS `my-project.my_dataset.t` (job_id STRING NOT NULL, status STRING) PRIMARY KEY (job_id) NOT ENFORCED'
# Error: PRIMARY KEY must be defined in the table element list as column attribute or constraint.

bq query --dry_run --use_legacy_sql=false \
  'CREATE TABLE IF NOT EXISTS `my-project.my_dataset.t` (job_id STRING NOT NULL, status STRING, PRIMARY KEY (job_id) NOT ENFORCED)'
# Query successfully validated.

Misc

🤖 Generated with Claude Code

BigQuery only accepts table constraints inside the table element list.
The four data quality monitoring CREATE TABLE statements put
`PRIMARY KEY (...) NOT ENFORCED` after the closing parenthesis, which
BigQuery rejects at parse time, so monitoring failed on every BigQuery
deployment. Move each constraint inside its column list and add a unit
test that checks the generated DDL.

Fixes feast-dev#7020

Signed-off-by: liron-velocity <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 50.96%. Comparing base (386759c) to head (2bc62b4).
⚠️ Report is 1 commits behind head on master.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #7021      +/-   ##
==========================================
+ Coverage   50.94%   50.96%   +0.01%     
==========================================
  Files         455      455              
  Lines       56915    56915              
  Branches     8290     8290              
==========================================
+ Hits        28994    29005      +11     
+ Misses      25940    25928      -12     
- Partials     1981     1982       +1     
Flag Coverage Δ *Carryforward flag
go-feature-server 30.58% <ø> (ø) Carriedforward from 386759c
python-unit 52.42% <ø> (+0.02%) ⬆️

*This pull request uses carry forward flags. Click here to find out more.

Files with missing lines Coverage Δ
sdk/python/feast/infra/offline_stores/bigquery.py 50.71% <ø> (+1.91%) ⬆️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 386759c...2bc62b4. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BigQuery data quality monitoring fails: PRIMARY KEY placed outside the column list in monitoring table DDL

3 participants