Add generic_jsonl parser #1370
Merged
Add generic_jsonl parser #1370
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Also fix `YOUTUBEDL_EXTRA_ARGS`.
The feedparser packages has 20 years of history and is very good at parsing RSS and Atom, so use that instead of ad-hoc regex and XML parsing. The medium_rss and shaarli_rss parsers weren't touched because they are probably unnecessary. (The special parse for pinboard is just needing because of how tags work.) Doesn't include tests because I haven't figured out how to run them in the docker development setup. Fixes ArchiveBox#1171
pirate
approved these changes
Mar 1, 2024
The feedparser packages has 20 years of history and is very good at parsing RSS and Atom, so use that instead of ad-hoc regex and XML parsing. The medium_rss and shaarli_rss parsers weren't touched because they are probably unnecessary. (The special parse for pinboard is just needing because of how tags work.) Doesn't include tests because I haven't figured out how to run them in the docker development setup. Fixes ArchiveBox#1171
Changes ./bin/test.sh to pass command line options to pytest, and default to only running tests in the tests/ directory instead of everywhere excluding a few directories which is more error-prone. Also keeps the mock_server used in testing quiet so access log entries don't appear on stdout.
Fixes ArchiveBox#1171 Fixes ArchiveBox#870 (probably, would need to test against a Wallabag Atom file to Fixes ArchiveBox#135 Fixes ArchiveBox#123 Fixes ArchiveBox#106
|
Small conflict to fix with your earlier RSS parser changes, then down to merge this! |
|
Okay, I probably made the git history a little wonkier than it needed to be, but the updated PR should be merge cleanly now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a JSONL parser and also fixes the JSON parser to reject what it suspects is a single-line JSONL file.
Changes these areas