Thank you @shahrukhx01. Please find requested details as follows -

Few of cleaning feature could be (not extensive, if you have more idea please add them as well) -

lower casing (if possible extra care of Named Entity for example Bush is person and bush is word hence no point is lowering Bush if it appear in sentance)
stop word removal, stemming, punctuation removal (but it should keep sentence as it is ie not return tokens)
Excessive white space remove
link removal or adding filler
Hashtags: remove hashtag or remove only # or replace them with some filer
UserTagging (@someuser): remove user tag or remove only `@ or replace them with some filer
Spelling/grammer correction (if it is heavy model like transformers then it should go to Analyzer)
User provided list of regex (and corresponding substitute) in case he want to customise cleaning
Decoding Unicode characters into a normalized form, such as UTF8
Handling of domain specific words, phrases, and acronyms
Handling or removing numbers, such as dates and amounts
Locating and correcting common typos and misspellings
Transliteration of characters from other languages into one fixed Language (if it is heavy model like transformers then it should go to Analyzer)
Cleaning up of alpha numeric words

Following is design consideration -

Till we finalise on Add DAG support and fix inconsistent naming #8, let's create preprosessor module and add text_cleaner.py
We can have BaseTextCleaner class which can have clean_input as@abstractmethod
Similarly we can have BaseTextCleanerConfig to store configuration of cleaner
clean_input can have following signature -

    def clean_input(
        self,
        input_list: List[AnalyzerRequest],
        config: BaseTextCleanerConfig,
        **kwargs
    ) -> List[AnalyzerRequest]:
        pass
# AnalyzerRequest will change in future for for now let's stick with it

All cleaning feature/config could be optional. User can select any one via config.
Now we can have Specific cleaning class like NLTKTextCleaner, SpacyTextCleaner, TextBobTextCleaner (I am still not sure we can have discussion, for end user it does not matter which library is doing what until their things is getting done). Hence I prefer less cognitive load on end user and for him text cleaner should be text cleaner step how it perform not much interesting to him. So I am open for the suggestions

I know it is very extensive list, it helped me to express my mind. It is not required to have implementation of all. We can start to add basic cleaning first and then enhance it.

Please let me know would you like to work on it and create a PR

Add text cleaner node #75

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions