Dear maintainers,

I am fiddling with FARM to make it work for the very specific use case of multitask token-level regression with initial masking (as for NER), and in doing so I created my custom TokenRegressionProcessor and TokenRegressionHead classes. Alongside those I implemented various other utility methods, including a custom MSE metric that i register through register_metric in metrics.py to average the MSE across masked sentences with variable length in evaluation.

Everything works, except that if I want to evaluate the model during the training, there is no way to set the report parameter of the Evaluator instantiated in the Trainer class to False from the training script (I'm using the ones in examples as references). Currently available reporting functions won't support my use case (same as MSE above), so this makes the Trainer and Evaluator classes unusable for me, forcing me to implement them locally and either change the report to false, or manually add a custom token-level r2 function in the reporting part of the Evaluator.

A solution would be to simply implement a registering system similar to the register_metric mentioned above also for reporting functions, instead of systematically raise NotImplementedError (line 118, eval.py).

If you believe this is interesting, I will create a pull request. When my implementation of token-level regression will be complete I will gladly share my custom classes and methods if you belive if they may be worth including in your library.

Thanks in advance,

Gabriele Sarti