The following piece of code causes a hard crash very often (not always):

library(data.table)

DT <- data.table(x1 = c(1,1,1,1,1,2,2,2,2,2),
                 x2 = c(1,1,2,2,2,1,1,2,2,2),
                 x3 = c(1,2,1,1,2,1,1,1,1,2),
                 y  = rnorm(10), 
                 key = c("x1", "x2", "x3"))

rm(DT)

detach("package:data.table", unload = TRUE)

I could confirm the problem for the current development version, the CRAN version, both under ubuntu 16.04 with R 3.4.1 and Windows with R 3.3.3.

While this might point to a C memory allocation problem, debugging with valgrind didn't produce any errors.
The following code doesn't cause the crash, so it's definitely the object creation that messes things:

library(data.table)

detach("package:data.table", unload = TRUE)

Maybe I am just reporting known behaviour?

Cheers,
Markus