joblib currently defaults to md5-hashing its input. For the tasks at hand, a non-cryptographic hash can be significantly faster (see comparison table at http://cyan4973.github.io/xxHash/).
scikit-learn provides an implementation of murmurhash and there's an xxhash implementation available on PyPI.