Repository navigation
Queries fail to run in full refresh mode #1914
Description
How to reproduce:
- Start a new execution in a Dataform (Cloud) workspace by specifying a tag.
- Select run with full refresh and include dependendents
- Start execution
What happens:
The queries included in the workflow all run without errors. However, the incremental table queries are executed in incremental mode instead of full refresh mode.
In my case, the workflow included both regular tables and incremental tables.
How to fix it:
The issue happens with Dataform core versions starting from 3.0.10.
Downgrading the Dataform core version back to 3.0.9 resolved the issue.
Reacted by Rowan Rodrik van der Molen and jfbeauvais-orange
Activity
bigsmoke commentedon Mar 18, 2025
I can confirm that this problem still exists on Dataform core 3.0.13.
bigsmoke commentedon Mar 18, 2025
And in 3.0.16 as well.
nybach commentedon Mar 19, 2025
I can also confirm this error, I've tested every version between 3.0.9 & 3.0.16. You can see it quite easily via the execution logs, in version 3.0.9 with full refresh option the incremental Dataform script is compiled as a CREATE OR REPLACE TABLE statement in BigQuery, whereas as of version 3.0.10 it gets compiled as a MERGE statement, which only updates or adds data, but does not delete rows that should not be there.
According to a Google Partner, this is linked to the new config parameter 'protected', concerning table deletion protection, which should by default be set to false to allow table recreation by full refresh, but currently defaults to true, blocking this action. The problem can be circumvented by adding the parameter and setting it explicitly to false in the config block of the script (or just use version 3.0.9 for now). A ticket has been logged regarding this wrong default.
Ceridan commentedon Mar 25, 2025
Version 3.0.17 should fix this. Brings back the default behavior for
protectedfield like it was before.