The two sequoia synsets are not duplicates: one is for the "tree" and the other denotes the "wood" (i.e. material to make furniture). The issue was discussed in OEWN some years ago.

Thanks @fcbond,I should have made it clearer that I was responding to this opening comment above, where @goodmami expresses his sentiment that these two synsets might be duplicates, which they aren't:

There may be duplicate synsets, but the script does not test for these. But here's an example of two synsets that may be duplicates:

WARNING:tsv-duplicates:duplicate of 11640645-n: 'sequoia', 'séquoia'
WARNING:tsv-duplicates:duplicate of 11640898-n: 'sequoia',

Apologies, @ekaf, my wording was imprecise and my example ill-chosen. @fcbond is correct that the script looks for near-duplicate lemmas within a synset and not for multiple synsets that are duplicates of each other (I've updated the issue text above to hopefully make this more clear). The idea is the TSV files have some lemmas that are only trivially different within the same synset and that we'd rather not keep all of them. This example might be more illustrative:

WARNING:tsv-duplicates:duplicate of 15277118-n: 'mortalite', 'mortalité'

If we look at all the lemmas for that synset, we see two others that are more interestingly different:

$ grep 15277118-n wns/fra/wn-data-fra.tab
15277118-n	fra:lemma	taux de mortalité
15277118-n	fra:lemma	mortalite
15277118-n	fra:lemma	morbidité
15277118-n	fra:lemma	mortalité

My guess is that the mortalite without diacritics is redundant and can be removed from the TSV file.

@goodmami, rather than just redundant, "the mortalite without diacritics" is incorrect. But, as you wrote earlier, only native lexicographers can make such corrections:

I don't think we can get around having human annotators to fix the upper/lower case, diacritics, and plurals.

Maybe a spell checker could detect some incorrect forms, but any orthographic editing would need to be approached with great caution.

I created #48 only for the scripts so we can move forward with that. The modifications to the data should happen in other PRs.

I think this PR should be closed without merging since the commits adding the scripts would cause conflicts. The manual modifications to the Icelandic wordnet could be cherry-picked for a new PR. Alternatively, we could repurpose this PR with a force-push that rebases without those commits. Let me know if you want help with either of those options.

@fcbond I want to close this PR but I don't want to lose the manual fixes you made to isl. Unfortunately, git cherry-pick 0881ed9394a53d78ae3781f4bc01c536b1e9ea2c results in a conflict because I've already removed redundant lemmas in #50 and your changes affect the same lines. Furthermore, it looks like you sorted the tab file, so it's hard to identify which were meaningful changes and which were just movements.

Perhaps you can just re-run your isl2tab.py script on a fresh branch, followed by:

$ python3 -m scripts.clean-tsv --in-place wns/isl/wn-data-isl.tab

Done! Sorry for the long delay. It was a very old version of nltk, I had to make a few changes to the script.

I also fixed the entry for föðurafi, móðurafi "grandmother", and stripped off excess white space, so I think there is nothing left to clean :-)

9d4b7f8..930f9ee

Ok great, thanks! I'll close this PR.

But why did you remove the isl-changes.tab file?

Ok, so perhaps you saw it as a record of changes made since the data was imported into the TSV file, and since you've re-imported it with more cleaning steps, those changes are no longer needed? I was using it more as a record of changes since the last release of the OMW data, in which case those changes still exist.

That's fine. I don't think the logs were the best solution anyway. We can probably recreate them using a script that compares versions in git history (which makes storing the logs in git unnecessary).