Data table search hanging after exactly 100 results

Hi,

I’m trying to iterate over my Users data table to fix something (I made a design mistake in my data tables so now have to iterate over them to correct the mistake).

My code is essentially this:

counter = 0
for usersRow in app_tables.users.search():
  counter += 1
  dataProgressRow["status"] = f"Starting on row {counter}"
  * do something useful *
  dataProgressRow["status"] = f"Finishing on row {counter}"

The dataProgressRow business is just outputting the status to a data table so I can see what’s happening without hitting a logging limit. This is running as a background task.

The progress always hangs at “Finishing on row 100”. It doesn’t even show “Starting on row 101”. The data table is ~4000 lines long, so it’s not run out of rows. The background task doesn’t stop or give any error - it just stops making progress. Last night it ran for ~12 hours without progress.

In an effort to simplify things I thought I could get all of the rows from the Users data table into a list first, hence this new version of the code:

results = app_tables.users.search():
dataProgressRow["status"] = "Searched"
listOfRows = [row for row in results]
dataProgressRow["status"] = "Completed putting rows into list"
for usersRow in listOfRows:
  * do useful stuff *

However it never reaches the “Completed putting rows into list” line.

It seems I just can’t get through all the rows of my Users table. What could be going on here?

Thanks!

Okay, mystery solved.

I had forgotten that I had included some links to other data table rows as extra columns in my Users data table.

It seems that, to my considerable surprise, the default behaviour is to download the whole content of those linked rows.

When it was reaching a row which was linked to a really large amount of data it was silently failing. When I tried to load that row directly in code it raised an error along the lines of “too much data to transfer” or something similar.

I’m still grumpy that in the context of fetching data table rows no error is shown and the process just stalls.