Skip to content

Add missing properties in CSV plugin #9399

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
KKcorps merged 2 commits into from
Sep 16, 2022
Merged

Add missing properties in CSV plugin #9399

KKcorps merged 2 commits into from
Sep 16, 2022

Conversation

@KKcorps
Copy link
Contributor

@KKcorps KKcorps commented Sep 14, 2022 •
edited
Loading

Following new configurations are being added to csv reader :

  • skipHeader - Set it to true to skip header line while processing CSV data.

  • ignoreEmptyLines - Set it to true to ignore empty lines, otherwise send them as empty string and let the processing logic take care of it.

  • ignoreSurroundingSpaces - Set to true to ignore the spaces around values.

  • recordSeparator - Character being used to distinguising between multiple lines.

  • quoteCharacter - single character that is being used for quotes in CSV files

  • quoteMode -

    /**
     * Quotes all fields.
     */
    ALL,

    /**
     * Quotes fields which contain special characters such as a delimiter, quote character or any of the characters in
     * line separator.
     */
    MINIMAL,

    /**
     * Quotes all non-numeric fields.
     */
    NON_NUMERIC,

    /**
     * Never quotes fields. When the delimiter occurs in data, it is preceded by the current escape character. If the
     * escape character is not set, printing will throw an exception if any characters that require escaping are
     * encountered.
     */
    NONE

Jackie-Jiang
Jackie-Jiang approved these changes Sep 14, 2022
View reviewed changes
...ormat/pinot-csv/src/main/java/org/apache/pinot/plugin/inputformat/csv/CSVMessageDecoder.java Outdated
public static final String NULL_STRING_VALUE = "nullStringValue";
public static final String SKIP_HEADER = "skipHeader";
public static final String IGNORE_EMPTY_LINES = "ignoreEmptyLines";
public static final String IGNORE_SURROUNDING_SPACE = "ignoreSurroundingSpace";
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang Sep 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
public static final String IGNORE_SURROUNDING_SPACE = "ignoreSurroundingSpace";
public static final String IGNORE_SURROUNDING_SPACES = "ignoreSurroundingSpaces";

@Jackie-Jiang Jackie-Jiang added release-notes Referenced by PRs that need attention when compiling the next release notes Configuration Config changes (addition/deletion/change in behavior) labels Sep 15, 2022
@Jackie-Jiang
Copy link
Contributor

Jackie-Jiang commented Sep 15, 2022

Can you please update the PR with the new added config keys, and the pinot doc?

@KKcorps KKcorps merged commit 3633495 into apache:master Sep 16, 2022
61yao pushed a commit to 61yao/pinot that referenced this pull request Oct 3, 2022
@KKcorps
Add missing properties in CSV plugin (apache#9399)
6348245 
* Add missing properties in CSV plugin

* Fix linting

Co-authored-by: Kartik Khare <[email protected]>
@JerAguilon JerAguilon mentioned this pull request Jun 26, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Jackie-Jiang Jackie-Jiang Jackie-Jiang approved these changes

Assignees

No one assigned

Labels

Configuration Config changes (addition/deletion/change in behavior) release-notes Referenced by PRs that need attention when compiling the next release notes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@KKcorps @Jackie-Jiang