Lies, Damn Lies, and Selective Benchmarks: Another Case

A few years ago the repository ldlasb was created to clarify some rather misleading points about Rcpp performance that had been claimed by another package. Hence the title about lies, damn lies, and selective benchmarks.

And now we have to do it again, this time to defend RcppArmadillo. As we demonstrate in a longer write-up contained in this repostiory (and package) under docs/, the 'other' package simply handicapped RcppArmadillo when comparing. Nice way to ensure a favourable outcome.

Once we undo this and let the two approaches compete fairly, then the purpoted performance gain actually inverts as can be seen in this first of four graphs in the longer write-up and shown below.

Also note that we did not cherry-pick this benchmark: it, along with a second benchmark, comes from published (and, uggh, peer-reviewed) paper. Still wrong though. It happens.

The first benchmark (as discussed in the longer piece) simply calls a single Armadillo function. The second benchmark (below) evaluates a matrix expression with three terms.

Here the performance gap between the corrected used of RcppArmadillo and the 'other' package is not significant. However, it is worth noting that the incorrectly set-up benchmark where three copies of input data are forced each time does of course noticeably worse. Funny how one wins a race against a competitor one handicapped.

So in summary, we will apply Hanlon's Razor and assume it was lack of knowledge and familiarity with the material judged. As such, this repository and its note may help showing how to use RcppArmadillo correctly---just as over 1200 CRAN packages already do.

Repository

This repository contains all the code that is needed to run the benchmarks (for example via the included quarto notebook) and wraps the supporting code up a small convenience package called armabenchmarks. It cleanly passes R CMD check too. You can install it the usual ways including via, e.g.,

remotes::install_github("eddelbuettel/ldlasb2")

Run Benchmarks

Once the package is installed, you can reach the benchmarks either via the included quarto notebook, or via the script bm.R that predates, or by executing the code snippets manually.

Author

This package was written by Dirk Eddelbuettel.

Code from the 'other' package was used to reproduce / replicate its benchmarks taken from paper directory in its repo.

License

This package is licensed under the GNU GPL, Version 2 or later.