Right now, if you have (lua)Torch in LD_LIBRARY_PATH, pytorch will try to load that, which will either:

1. fail loading because of symbol lookups
2. load a 1-indexing based torch, also having massive side-effects.

Have guards against this and make it robust

Necessary cmake fixes in #385