Skip to content
This repository was archived by the owner on Apr 8, 2025. It is now read-only.

Fix cosine similarity in DPR training#741

Merged
Timoeller merged 3 commits intomasterfrom
fix_dpr_cosine
Mar 16, 2021
Merged

Fix cosine similarity in DPR training#741
Timoeller merged 3 commits intomasterfrom
fix_dpr_cosine

Conversation

@bogdankostic
Copy link
Copy Markdown
Contributor

@bogdankostic bogdankostic commented Mar 10, 2021

When using cosine similarity as similarity metric in DPR, we get the following error:

  File "/home/ubuntu/pycharm/FARM/examples/dpr_encoder.py", line 159, in <module>
    dense_passage_retrieval()
  File "/home/ubuntu/pycharm/FARM/examples/dpr_encoder.py", line 144, in dense_passage_retrieval
    trainer.train()
  File "/home/ubuntu/pycharm/FARM/farm/train.py", line 301, in train
    per_sample_loss = self.model.logits_to_loss(logits=logits, global_step=self.global_step, **batch)
  File "/home/ubuntu/pycharm/FARM/farm/modeling/biadaptive_model.py", line 323, in logits_to_loss
    all_losses = self.logits_to_loss_per_head(logits, **kwargs)
  File "/home/ubuntu/pycharm/FARM/farm/modeling/biadaptive_model.py", line 307, in logits_to_loss_per_head
    all_losses.append(head.logits_to_loss(logits=logits_for_one_head, **kwargs))
  File "/home/ubuntu/pycharm/FARM/farm/modeling/prediction_head.py", line 1758, in logits_to_loss
    softmax_scores = self._embeddings_to_scores(global_query_vectors, global_passage_vectors)
  File "/home/ubuntu/pycharm/FARM/farm/modeling/prediction_head.py", line 1691, in _embeddings_to_scores
    scores = sim_func(query_vectors, passage_vectors)
  File "/home/ubuntu/pycharm/FARM/farm/modeling/prediction_head.py", line 1650, in cosine_scores
    return nn.functional.cosine_similarity(query_vectors, passage_vectors, dim=1)
RuntimeError: The size of tensor a (4) must match the size of tensor b (8) at non-singleton dimension 0

This PR fixes the computation of cosine similarity between query vectors and passage vectors.

@bogdankostic bogdankostic requested a review from Timoeller March 10, 2021 15:16
Timoeller
Timoeller reviewed Mar 10, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@Timoeller Timoeller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LG!
Only could you update the docstrings above to make clear what n1 and n2 is, please?

@bogdankostic bogdankostic requested a review from Timoeller March 10, 2021 17:27
Timoeller
Timoeller approved these changes Mar 16, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@Timoeller Timoeller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it, thanks

@Timoeller Timoeller merged commit 328f2be into master Mar 16, 2021
@Timoeller Timoeller deleted the fix_dpr_cosine branch March 16, 2021 11:35
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer

@Timoeller Timoeller Timoeller approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bogdankostic @Timoeller