Skip to content
This repository was archived by the owner on Apr 8, 2025. It is now read-only.

Add test for empty titles in DPR processor#643

Merged
tholor merged 1 commit intomasterfrom
add_dpr_empty_title_test
Nov 30, 2020
Merged

Add test for empty titles in DPR processor#643
tholor merged 1 commit intomasterfrom
add_dpr_empty_title_test

Conversation

@tholor
Copy link
Copy Markdown
Member

@tholor tholor commented Nov 30, 2020

When using DPR processor with use_fast=True and embed_title=True, we currently get an error if the "title" is an empty string. This is due to the handling of empty strings by FastTokenizers (was already resolved by huggingface/tokenizers#389).
Adding a test case here to cover this in future.

11/30/2020 06:53:05 - ERROR - farm.data_handler.processor -   Error message: Truncation error: Specified max length is too low to respect the various constraints
11/30/2020 06:53:06 - ERROR - farm.data_handler.processor -   Could not create sample(s) from this dict: 
 {'passages': [{'title': '', 'text': '"letters). Some ambiguity exists in the official code page assignment for the Kamenický encoding: Some dot matrix printers of the NEC Pinwriter series, namely the ""P3200""/""P3300"" (""P20""/""P30""), ""P6200""/""P6300"" (""P60""/""P70""), ""P9300"" (""P90""), ""P7200""/""P7300"" (""P62""/""P72""), ""P22Q""/""P32Q"", ""P3800""/""P3900"" (""P42Q""/""P52Q""), ""P1200""/""P1300"" (""P2Q""/""P3Q""), ""P2000"" (""P2X"") and ""P8000"" (""P72X""), supported the installation of optional font EPROMs. The optional ROM #2 ""East Europe"" included this encoding, invokable via escape sequence codice_3 with (n) = 23. While named ""Kamenický"" in the documentation, it was originally advertised by NEC as code page 867 (CP867) or ""Czech"". (However, it was never registered with IBM under that ID, as IBM registered"', 'label': 'positive', 'external_id': 'b9d68e52-72ed-414f-baed-99275dba4767'}]}

@tholor tholor changed the title Add test for empty titles Add test for empty titles in DPR processor Nov 30, 2020
@tholor tholor merged commit 2cc54be into master Nov 30, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant