Hi @wyli ,

Thanks for providing the detection metrics example:
https://github.com/rafaelpadilla/Object-Detection-Metrics/blob/master/lib/Evaluator.py#L24
I checked the code and seems we can implement it based on this Metric, something like:

class PascalVOCMetric(Metric):
    def __init__(self, IOUThreshold=0.5, method="EveryPointInterpolation"):
        self.IOUThreshold = IOUThreshold
        self.method = method
        self._ groundtruths = []
        self._detections = []
        self._classes = []

    # here the args and arg type don't match base class
    def _compute(self, bb):
        if bb.getBBType() == BBType.GroundTruth:
            self._groundtruths.append([
                bb.getImageName(),
                bb.getClassId(), 1,
                bb.getAbsoluteBoundingBox(BBFormat.XYX2Y2)
            ])
        else:
            self._detections.append([
                bb.getImageName(),
                bb.getClassId(),
                bb.getConfidence(),
                bb.getAbsoluteBoundingBox(BBFormat.XYX2Y2)
            ])
        if bb.getClassId() not in classes:
            classes.append(bb.getClassId())

    def aggregate(self, data):
        # final metric computation progress

    def reset(self):
        self._ groundtruths = []
        self._detections = []
        self._classes = []

I got several questions with current Metric design:

1. python doesn't support overload for different arg list, how to define the API for both the metrics with pred_y & y and the metrics with only pred_y? The data type of args may be also unknown. Can we define a API interface with uncertain length args? def __call__(...)?
2. Although this PR doesn't include member variables in the Metric base class, I agree we should add a reset() API to unify the logic for member variables.
3. Should we include the multi-gpus distributed syncing logic in Metric or put this task to the upper layer or let users to do it? I have a draft idea is to add a _sync(self, data) interface in Metric, will be called at the beginning of aggregate(), every metric class can implement the real _sync() logic for itself.
4. Should we keep the Metric base class simple, and make several subclasses for different categories of metrics?

Thanks.

This PR should try to close #303 #565 #561 with a lightweight solution, perhaps also try to be compatible with a few existing packages:

• https://github.com/PyTorchLightning/metrics/blob/master/torchmetrics/metric.py
• https://github.com/loli/medpy/tree/master/medpy/metric

So, I think we can have reset and _sync API in a simple base class. could you finish this PR, and then we can get feedback from the working group.

Hi @wyli ,

Thanks for your sharing.
I investigated several other projects and got some ideas, let me try to update this PR ASAP.

Thanks.

/black

/black

Hi @wyli @ericspod ,

I updated the PR based on your comments and some further investigation, could you please help review it again?

Thanks in advance.

/integration-test

/integration-test

/black

/integration-test

Hi @wyli ,

I have totally updated the PR based on the latest design, could you please help review it again?

Thanks in advance.

/black

/integration-test

Hi @wyli ,

Thanks for your review and comments.
I updated the PR based on your comments and enhanced the doc-string of base classes.
I can refine the doc-strings again according to future feedback from users and metrics working group.
Could you please help review it again?

Thanks.

/black

/integration-test

/black

/integration-test