-
-
Notifications
You must be signed in to change notification settings - Fork 64
Allow Direct INSERT in Final Table #35
Copy link
Copy link
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
This would only apply to snapshot mode, or incremental mode with only update_key (no primary_key).
See #33 (comment) for more details.
Changes needed in WriteToDb:
- Add environment variable to capture setting. Something like
SLING_ALLOW_DIRECT_INSERT=TRUE. - Add error handling, if mode is not
snapshotorincrementalmode with onlyupdate_key.- Add config method to determine if direct insert is allowed:
DirectInsertAllowed()
- Add config method to determine if direct insert is allowed:
- Ensure
PreSQLis executed before direct insert. Currently, is executed after successful insert into temp table - Clean up and reorg
WriteToDba bit for easier read
Note about current implementation:
The temporary table is to ensure data quality, such as column typing (sling profiles every value and determines the appropriate column type), check summing, as well as matching row count. Inserting into the final table directly, we cannot check quality at this level. Also, what if the operation is cancelled? So yes, sling handles failure gracefully and drops any temporary file/table if something goes wrong, not affecting the final table, and errors.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request