Skip to content
This repository was archived by the owner on Apr 8, 2025. It is now read-only.

Create QA component specific benchmarks#491

Merged
brandenchan merged 10 commits intomasterfrom
benchmarks
Aug 19, 2020
Merged

Create QA component specific benchmarks#491
brandenchan merged 10 commits intomasterfrom
benchmarks

Conversation

@brandenchan
Copy link
Copy Markdown
Contributor

This adds a script and some code that allows the separate benchmarking of preprocessing, language modelling and prediction head processing.

@brandenchan brandenchan requested a review from Timoeller August 12, 2020 14:52
@brandenchan
Copy link
Copy Markdown
Contributor Author

Timing with torch Events instead of time.perf_counter gives pretty much the same times

@brandenchan
Copy link
Copy Markdown
Contributor Author

Baseline speed when running question_answering_components.py

{'language model': 1610.3536376953125,
'model name': 'deepset/bert-base-cased-squad2',
'prediction head': 8769.41357421875,
'preproc': 802.9445190429688,
'question': 'When were the first traces of Human life found in France?',
'total': 11183.6533203125}

{'language model': 595.620849609375,
'model name': 'deepset/minilm-uncased-squad2',
'prediction head': 3255.84228515625,
'preproc': 1075.1549682617188,
'question': 'When were the first traces of Human life found in France?',
'total': 4922.5927734375}

{'language model': 1586.070556640625,
'model name': 'deepset/bert-base-cased-squad2',
'prediction head': 108.00390625,
'preproc': 795.5881042480469,
'question': 'How many pretrained models are available in Transformers?',
'total': 2490.778076171875}

{'language model': 596.1339111328125,
'model name': 'deepset/minilm-uncased-squad2',
'prediction head': 89.099609375,
'preproc': 1066.5256958007812,
'question': 'How many pretrained models are available in Transformers?',
'total': 1755.6483154296875}

Copy link
Copy Markdown
Contributor

@Timoeller Timoeller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it.
Only that we should add a timing mode and then do the synchronize() calls. In normal inference mode this might slow down computations

@brandenchan brandenchan requested a review from Timoeller August 13, 2020 08:52
Copy link
Copy Markdown
Contributor

@Timoeller Timoeller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good - lets put the sample files into another folder. E.g. test/benchmarks/samples?

@Timoeller Timoeller self-requested a review August 19, 2020 14:38
Copy link
Copy Markdown
Contributor

@Timoeller Timoeller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ready to merge

@brandenchan brandenchan merged commit 6fd22b8 into master Aug 19, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants