fast cast to matrix using Rfast #4134
Conversation
|
At a glance I would prefer we just wrote our own method in C rather than add a Suggests for a new library with Rcpp... besides the type bumping bit, it should be pretty straightforward & could parallelize on the longer dimension |
mattdowle
changed the title
|
Agree with Michael. Adding a heavy Suggested dep is not much justified for a matrix conversion functions. Why not just use that package instead if someone needs it? |
|
Glancing at the code, there's two bits of complication:
|
|
Some profiling revealed the
With that in mind, I tried to replace |
|
100% agree with the above. What's the best course of action version control wise for dealing with this branch/commit if I wanted to work on creating a C-version? Should I revert the above commit, or just close this pull request and start a new branch? |
|
I would close this & start a new branch. Leaves this idea in the web history at least, without potentially cluttering up the C-level PR. And adding a link to this PR for posterity |
|
some intersection with this functionality and |
5 participants
After waiting several hours for
as.matrix.data.tableto convert a 3000 row 1.7 million column data.table to a matrix, I had a look around to see if there were faster ways of converting a data.frame/data.table to a matrix. This lead me to thedata.frame.to_matrix()function in theRfastpackage, which was able to perform the above task in under a minute.I've set about incorporating this into the
as.matrixfunction in thedata.tablepackage, but this has proved less straightforward than first anticipated. A few things I need to follow-up on:data.frame.to_matrixreturns an error when columns in the data.table are different types, this is intentional, as the code does not do type conversion/pre checks to enhance speed. We would need to do the type conversion ourselves but I'm not sure how best to detect the type that should be converted to across all columns.data.frame.to_matrix` itself is quite janky, it throws a few warnings when it shouldn't due to some bad if-statement checks on their end (hence the somewhat odd choice I've currently made of setting the rownames to NULL post-conversion if there are no rownames)
When supplied integer columns
data.frame.to_matrixconverts these to numerics, causing many tests to fail. This may need to be fixed upstream, and I also need to check how this function works on other non-numeric types (e.g. character vectors, factors, dates, etc) to make sure it returns the equivalent of as.matrix.For now, I've simply added an
Rfast=FALSEargument toas.matrix. If the above points can be solved we can simply remove this, otherwise we could potentially give the user the option of explicitly setting Rfast=TRUE to get a numeric matrix (this would be useful purely for therownamesfunctionality ofas.matrix.data.frame).