Ultimately it make sense to redirect users to use forderv directly when they want to pass DT into forder. If you agree, then maybe a verbose print for now informing about that? because any change here would be significant (and unnecessary) breaking change. Better to provide a verbose (or eventually non-verbose) print to user, so no one will be harmed.

I don't quite follow. There is no breaking change in this respect here so far, afaik. This line that changes (list(DT,...) to list(...)) is basically just maintaining the old usage of forder(DT, -colA, colB) so that form still works. That form isn't what forderv is for, iiuc.

yes, forder has different API but IMO it worth to make distinction, even if not a breaking change, then just suggests to users. So forder is a order replacement and forderv is meant to be used by passing data.table to it.

We can suggest forderv in verbose mode I guess. But although, yes, forderv is meant for passing DT to it, a second aspect of forderv is that it doesn't accept expressions of columns. I was hoping that users can just use DT[order(...), ] as they have been and not need to know that forder and forderv exist. The optimization should hide these things from them and the news item (15) is consistent with that (e.g. it doesn't mention forder vs forderv). If verbose mode for DT[order(...), ] starts to suggest they use forderv it might be a bit confusing because i) they didn't know they were using forder, and ii) neither forder or forderv are exported or documented. Hopefully we never need to export them since optimization can take care of it.

is DT[order(...), ] optimised to forder(DT, ...)? if not, then the verbose message wouldn't be printed. But agree that would be confusing till forder is not exported. Ultimately we want to export it #3447 "so it can be legally used outside of [.data.table"

I see now, so verbose message would just be when first argument is DT. DT[order(...),] is optimized to DT[forder(...),] so wouldn't print even in verbose mode. I added link to #3447 back to this discussion then so we can come back to it.

does it make sense to push down call to colnamesInt from R to C? more related logic next to each other, and (probably very tiny) speed improvement.

colnamesInt is already a .Call to C. You added it quite recently iirc and it's nice. There are 11 calls currently from R level which seems right and nice to me, iiuc.

I mean to call it from forder.c not from forder R

Could do I suppose. I've just made forderv even smaller at R level, so forderv could be a single .Call() eventually. Moving colnamesInt call down to C for that reason might be nice. I feel like I've spent way too long on this already so let's pencil that in for future, if that's ok. Link back here added to #3447.