I agree that Anvil cold do more and with your request.
I just wanted to add that I too use my own ids rather than linked columns because of the problem you mention and especially because they are much faster. Linked tables are often victims of the n+1 problem and trigger database queries when you don’t need them (and if you use them on the client side, they also trigger a round trip, but I don’t use database row objects on the client side at all).
Why not?
The only problem is discipline: you do have pointers, just pay attention not to use them 
And if you want to make the table manager app dependency cleaner and avoid those pointers, you could define a class that manages every access to the table. Then, all the apps that need that table will use the interface defined in that class and not break anything. The class will expose values, not database rows, so there is no way to break anything. The class could also call back some functions so the depending apps could react to row deletions or other events.