far afield of OQ, yes, but i for one at least enjoy reading your replies. 
as for practical forum advice… given that the OP was doing a full table scan (and initially not even breaking early), my hunch would be that implementing anything considerably more “esoteric” might be a stretch.
if that 400k table is already considered “very large”, maybe we can assume it won’t be growing much to “truly large” (to the point where some form of encoded storage required). if true, and storage really shouldn’t be an issue, then values-as-keys will likely give most bang-for-buck per loc to implement.