To exclude special attributes from causing MetricsLambda creation #3262
Description
Hi @vfdev-5 ! Long time no see! Happy to be here again!
It might be useful to exclude python special attributes from being subject to creating MetricsLambda in Metric::__getattr__.
This is the case I saw the current behavior might make trouble. I was using an Ignite metric as the compute_metrics argument to HuggingFace Trainer API. There, they do a check on the argument signature using inspect.signature which itself accesses __signature__ attribute of the object which raises an error:
class CustomMetric(Metric):
def reset(self):
...
def update(self, output):
...
def compute(self):
...
def __call__(self, eval_pred, compute_result=False): # HF trainer makes sure this has a `compute_result` arg otherwise raises an error
self.update(eval_pred)
if compute_result:
return self.compute()The check takes place here:
if args.batch_eval_metrics and compute_metrics is not None:
if "compute_result" not in inspect.signature(compute_metrics).parameters.keys():
raise ValueError(
"When using `batch_eval_metrics`, your `compute_metrics` function must take a `compute_result`"
" boolean argument which will be triggered after the last batch of the eval set to signal that the"
" summary statistics should be returned by the function."
)trainer = Trainer(
model=model,
tokenizer=tokenizer,
args=args,
data_collator=data_collator,
train_dataset=dataset,
eval_dataset=dataset,
compute_metrics=CustomMetric(),
)TypeError: unexpected object <function Metric.__getattr__.<locals>.wrapper at 0x7eda2cf82200> in __signature__ attribute