Snowflake/copy into by pawel-big-lebowski · Pull Request #841 · apache/datafusion-sqlparser-rs

pawel-big-lebowski · 2023-03-23T13:12:50Z

Support Snowflake's COPY INTO clause:

/* Standard data load */
COPY INTO [<namespace>.]<table_name>
     FROM { internalStage | externalStage | externalLocation }
[ FILES = ( '<file_name>' [ , '<file_name>' ] [ , ... ] ) ]
[ PATTERN = '<regex_pattern>' ]
[ FILE_FORMAT = ( { FORMAT_NAME = '[<namespace>.]<file_format_name>' |
                    TYPE = { CSV | JSON | AVRO | ORC | PARQUET | XML } [ formatTypeOptions ] } ) ]
[ copyOptions ]
[ VALIDATION_MODE = RETURN_<n>_ROWS | RETURN_ERRORS | RETURN_ALL_ERRORS ]

/* Data load with transformation */
COPY INTO [<namespace>.]<table_name> [ ( <col_name> [ , <col_name> ... ] ) ]
     FROM ( SELECT [<alias>.]$<file_col_num>[.<element>] [ , [<alias>.]$<file_col_num>[.<element>] ... ]
            FROM { internalStage | externalStage } )
[ FILES = ( '<file_name>' [ , '<file_name>' ] [ , ... ] ) ]
[ PATTERN = '<regex_pattern>' ]
[ FILE_FORMAT = ( { FORMAT_NAME = '[<namespace>.]<file_format_name>' |
                    TYPE = { CSV | JSON | AVRO | ORC | PARQUET | XML } [ formatTypeOptions ] } ) ]
[ copyOptions ]

Docs: https://docs.snowflake.com/en/sql-reference/sql/copy-into-table

coveralls · 2023-03-23T13:17:09Z

Pull Request Test Coverage Report for Build 4676350337

294 of 314 (93.63%) changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.1%) to 86.425%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/ast/mod.rs	33	43	76.74%
src/dialect/snowflake.rs	109	119	91.6%

Totals
Change from base Build 4659102766:	0.1%
Covered Lines:	14204
Relevant Lines:	16435

💛 - Coveralls

alamb

Thanks @pawel-big-lebowski -- looks good to me except for a few more tests

alamb · 2023-04-09T11:30:51Z

src/ast/mod.rs

+    /// ```sql
+    /// COPY INTO
+    /// ```
+    /// See <https://docs.snowflake.com/en/sql-reference/sql/copy-into-table>


I agree it makes sense to use a separate struct for system specific syntax

alamb · 2023-04-09T11:31:23Z

src/ast/mod.rs

+                into,
+                from_stage,
+                from_stage_alias,
+                stage_params: _stage_params,


Why are stage_params ignored?

Seems like stage_params were totally forgotten for copy into. Thanks for spotting this.
I've included stage_params support in recent commit version.

alamb · 2023-04-09T11:35:03Z

src/dialect/snowflake.rs

            || ch.is_ascii_digit()
            || ch == '$'
            || ch == '_'
+            || ch == '/'


I didn't see any test coverage of these identifier changes. Could you please add some tests (to make sure we don't break the functionality in some future refactoring)?

This is done within test_snowflake_stage_object_names test.

alamb · 2023-04-09T11:35:26Z

src/dialect/snowflake.rs

+        into,
+        from_stage,
+        from_stage_alias,
+        stage_params: StageParamsObject {


I didn't see any test coverage for stage params -- could you please add some coverage for that as well?

Yes, you're right. I've included test_copy_into_with_stage_params.

Signed-off-by: Pawel Leszczynski <[email protected]>

alamb

Looks good to me -- thank you @pawel-big-lebowski

* Support identifiers beginning with digits in MySQL (apache#856) * support COPY INTO in snowflake (apache#841) Signed-off-by: Pawel Leszczynski <[email protected]> * Add `dialect_from_str` and improve `Dialect` documentation (apache#848) * Add `dialect_from_str` and improve `Dialect` documentation * cleanup * fix compilation with nostd * Support multiple-table DELETE syntax (apache#855) * Support `DISTINCT ON (...)` (apache#852) * Support "DISTINCT ON (...)" * a test * fix the merge * Test trailing commas (apache#859) * test: add tests for trailing commas * tweaks * Add support for query source in COPY .. TO statement (apache#858) * Add support for query source in COPY .. TO statement * Fix compile error * Fix logical merge conflict (apache#865) * Fix tiny typo in custom_sql_parser.md (apache#864) * Make Expr::Interval its own struct (apache#872) * Make Expr::Interval its own struct * Add test interval display * Fix cargo fmt * Include license file in published crate (apache#871) * Add support for multiple expressions, order by in aggregations (apache#879) * Add support for multiple expressions, order by in aggregations * Fix formatting errors * Resolve linter errors * Add parse_multipart_identifier function to parser (apache#860) * Add parse_multipart_identifier function to parser * Update doc for parse_multipart_identifier * Fix conflict * feat: Add custom operator (apache#868) * feat: Add custom operator From apache#863 - It doesn't parse anything — I'm not sure how to parse ` SELECT 'a' REGEXP '^[a-d]';` with `REGEXP` as the operator... (but fine for my narrow purpose) - If we need tests, where would I add them? * Update src/ast/operator.rs --------- Co-authored-by: Andrew Lamb <[email protected]> * feat: Support MySQL's `DIV` operator (apache#876) * feat: MySQL's DIV operator * fix: do not use `_` prefix for used variable --------- Co-authored-by: Andrew Lamb <[email protected]> * truncate: table as optional keyword (apache#883) Signed-off-by: Maciej Obuchowski <[email protected]> * feat: add DuckDB dialect (apache#878) * feat: add DuckDB dialect * formatting * fix conflict * support // in GenericDialect * add DucDbDialect to all_dialects * add comment from suggestion Co-authored-by: Andrew Lamb <[email protected]> * fix: support // in GenericDialect --------- Co-authored-by: Andrew Lamb <[email protected]> * Add support for first, last aggregate function parsing (apache#882) * Add order by parsing to functions * Fix doc error * minor changes * Named window frames (apache#881) * after over clause, named window can be parsed with window ... as after having clause * Lint errors are fixed * Support for multiple windows * fix lint errors * simplifications * rename function * Rewrite named window search in functional style * Test added and some minor changes * Minor changes on tests and namings, and semantic check is removed --------- Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> * Fix merge conflict (apache#885) * Update CHANGELOG for `0.34.0` release (apache#884) * chore: Release sqlparser version 0.34.0 * Update criterion requirement from 0.4 to 0.5 in /sqlparser_bench (apache#890) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: Pawel Leszczynski <[email protected]> Signed-off-by: Maciej Obuchowski <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: AviRaboah <[email protected]> Co-authored-by: pawel.leszczynski <[email protected]> Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Aljaž Mur Eržen <[email protected]> Co-authored-by: Armin Primadi <[email protected]> Co-authored-by: Okue <[email protected]> Co-authored-by: Andrew Kane <[email protected]> Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Jeffrey <[email protected]> Co-authored-by: Maximilian Roos <[email protected]> Co-authored-by: eitsupi <[email protected]> Co-authored-by: Maciej Obuchowski <[email protected]> Co-authored-by: Berkay Şahin <[email protected]> Co-authored-by: Mustafa Akur <[email protected]> Co-authored-by: Mehmet Ozan Kabak <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

pawel-big-lebowski force-pushed the snowflake/copy-into branch 5 times, most recently from dd3f6ff to e6315ad Compare March 27, 2023 10:21

pawel-big-lebowski marked this pull request as ready for review March 27, 2023 10:37

alamb reviewed Apr 9, 2023

View reviewed changes

pawel-big-lebowski force-pushed the snowflake/copy-into branch from e6315ad to 27b5c5b Compare April 12, 2023 08:36

support COPY INTO in snowflake

0561c97

Signed-off-by: Pawel Leszczynski <[email protected]>

pawel-big-lebowski force-pushed the snowflake/copy-into branch from 27b5c5b to 0561c97 Compare April 12, 2023 08:40

alamb approved these changes Apr 27, 2023

View reviewed changes

alamb merged commit d8af925 into apache:main Apr 27, 2023

lustefaniak mentioned this pull request Oct 19, 2023

In Snowflake dialect field/100 tokenizes as Word(field/100) instead of Word(field),Div,Number(100) #1008

Closed

alamb mentioned this pull request Oct 24, 2023

snowflake: Fix handling of /~% in the stage name #1009

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Snowflake/copy into#841

Snowflake/copy into#841
alamb merged 1 commit intoapache:mainfrom
OpenLineage:snowflake/copy-into

pawel-big-lebowski commented Mar 23, 2023 •

edited

Loading

Uh oh!

coveralls commented Mar 23, 2023 •

edited

Loading

Uh oh!

alamb left a comment

Uh oh!

alamb Apr 9, 2023

Uh oh!

alamb Apr 9, 2023

Uh oh!

pawel-big-lebowski Apr 12, 2023

Uh oh!

alamb Apr 9, 2023

Uh oh!

pawel-big-lebowski Apr 12, 2023

Uh oh!

alamb Apr 9, 2023

Uh oh!

pawel-big-lebowski Apr 12, 2023

Uh oh!

alamb left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pawel-big-lebowski commented Mar 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Mar 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 4676350337

💛 - Coveralls

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Apr 9, 2023

Choose a reason for hiding this comment

Uh oh!

alamb Apr 9, 2023

Choose a reason for hiding this comment

Uh oh!

pawel-big-lebowski Apr 12, 2023

Choose a reason for hiding this comment

Uh oh!

alamb Apr 9, 2023

Choose a reason for hiding this comment

Uh oh!

pawel-big-lebowski Apr 12, 2023

Choose a reason for hiding this comment

Uh oh!

alamb Apr 9, 2023

Choose a reason for hiding this comment

Uh oh!

pawel-big-lebowski Apr 12, 2023

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pawel-big-lebowski commented Mar 23, 2023 •

edited

Loading

coveralls commented Mar 23, 2023 •

edited

Loading