Skip to content

Fix for Saving Evaluation Metrics in Loop Closure Pipeline #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
saurabh1002 merged 1 commit into from
Dec 9, 2024
Merged

Fix for Saving Evaluation Metrics in Loop Closure Pipeline #47

saurabh1002 merged 1 commit into from
Dec 9, 2024

Conversation

@jlaserna
Copy link
Contributor

@jlaserna jlaserna commented Dec 9, 2024

This PR resolves an issue where evaluation metrics were not being correctly saved during the execution of the loop closure pipeline. Specifically, the method self.results.compute_closures_and_metrics() was being called after saving the configuration and logging, resulting in incomplete or missing evaluation metrics in the output file (evaluation_metrics.txt).

The change moves the compute_closures_and_metrics() method to be executed before the configuration is saved and the results are logged, ensuring that all metrics are correctly calculated and included in the output.

Issue Details

Before this fix, running the pipeline with the command:

map_closure_pipeline -e --dataloader mulran ./data/MulRan/KAIST03/ ./output

produced an incomplete evaluation_metrics.txt file with no data:

+----------+----------------+-----------------+-----------------+-----------+--------+----------+
| #Inliers | True Positives | False Positives | False Negatives | Precision | Recall | F1 score |
+==========+================+=================+=================+===========+========+==========+

                                 Loop Closure Evaluation Metrics

After the fix, the metrics are properly calculated and saved, as shown below:

+----------+----------------+-----------------+-----------------+-----------+--------+----------+
| #Inliers | True Positives | False Positives | False Negatives | Precision | Recall | F1 score |
+==========+================+=================+=================+===========+========+==========+
|    5     |     54062      |       502       |     1504324     | 0.9908    | 0.0347 | 0.0670   |
|    6     |     54062      |       502       |     1504324     | 0.9908    | 0.0347 | 0.0670   |
|    7     |     54062      |       502       |     1504324     | 0.9908    | 0.0347 | 0.0670   |
|    8     |     54062      |       502       |     1504324     | 0.9908    | 0.0347 | 0.0670   |
|    9     |     54062      |       502       |     1504324     | 0.9908    | 0.0347 | 0.0670   |
|    10    |     54062      |       502       |     1504324     | 0.9908    | 0.0347 | 0.0670   |
|    11    |     54062      |       502       |     1504324     | 0.9908    | 0.0347 | 0.0670   |
|    12    |     52302      |       16        |     1506084     | 0.9997    | 0.0336 | 0.0649   |
|    13    |     52302      |       16        |     1506084     | 0.9997    | 0.0336 | 0.0649   |
|    14    |     52302      |       16        |     1506084     | 0.9997    | 0.0336 | 0.0649   |
+----------+----------------+-----------------+-----------------+-----------+--------+----------+

                                 Loop Closure Evaluation Metrics

@saurabh1002
Copy link
Collaborator

saurabh1002 commented Dec 9, 2024

Thanks @jlaserna for detecting this bug.
I will merge the PR soon.

@saurabh1002 saurabh1002 added the bug Something isn't working label Dec 9, 2024
@saurabh1002 saurabh1002 merged commit 356d15b into PRBonn:main Dec 9, 2024
10 checks passed
@jlaserna jlaserna deleted the bugfix/evaluation_metricts_results branch December 9, 2024 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@saurabh1002 saurabh1002 saurabh1002 approved these changes

@tizianoGuadagnino tizianoGuadagnino Awaiting requested review from tizianoGuadagnino tizianoGuadagnino is a code owner

Assignees

No one assigned

Labels

bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jlaserna @saurabh1002