I understand the iterator is lazy (it’s a feature, not a bug etc.). In every circumstance where I have run into this problem I end up limiting the results by creating more specific search querys.
Database engines are much more adept at returning specific, ordered results while self optimizing for speed than almost anything I could ever come up with in a JIT compiled programming language.
I can use the programming language as an intermediate step to apply logic to choose what type of search, search query, etc. is best, but the bulk of the ‘chaff’ of the data should be handled at the db search level.
I can programmatically build a better search query etc, but the database engine is always going to be the go-to the place where speed happens.
This is just in my experience, feel free to disagree. I would love to learn a new method.