Skip to content

Allow Direct INSERT in Final Table #35

@flarco

Description

@flarco

This would only apply to snapshot mode, or incremental mode with only update_key (no primary_key).

See #33 (comment) for more details.

Changes needed in WriteToDb:

  • Add environment variable to capture setting. Something like SLING_ALLOW_DIRECT_INSERT=TRUE.
  • Add error handling, if mode is not snapshot or incremental mode with only update_key.
    • Add config method to determine if direct insert is allowed: DirectInsertAllowed()
  • Ensure PreSQL is executed before direct insert. Currently, is executed after successful insert into temp table
  • Clean up and reorg WriteToDb a bit for easier read

Note about current implementation:

The temporary table is to ensure data quality, such as column typing (sling profiles every value and determines the appropriate column type), check summing, as well as matching row count. Inserting into the final table directly, we cannot check quality at this level. Also, what if the operation is cancelled? So yes, sling handles failure gracefully and drops any temporary file/table if something goes wrong, not affecting the final table, and errors.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions