It works like this because the pagination uses self.items, not the forms inside the repeating panel.
Your trick solves the crash problem. I may use it as a temporary solution. I will eventually try to use a lazy loading Portable.
I tried creating my own lazy loading class long time ago and failed.
Now Meredydd says it should be possible, so I will give it another try (eventually).