This has come up a few times, but we apparently haven't created an issue for it: we can keep the masks in the "index" we build instead of the vocab indices. That will save on allocation time.