Some custom components (eg old versions of anvil-extras) depend on Bootstrap JS/behaviour. Some existing themes’ CSS depends on styling from the Bootstrap CSS (so they will look funny if you remove Bootstrap). Many existing themes’ CSS depends on the old Bootstrap-like names (eg class="btn" rather than class="anvil-btn").
But all of the built-in Anvil components work just fine if you turn off all the legacy options.
This is specifically about sticking random new properties onto instances of Anvil components. That has always been a bad idea – you’re supposed to use the .tag property instead – but when we first released Anvil we didn’t actually prevent you from doing that (it required features we have since added to Skulpt IIRC), so we know there are apps out there that accidentally rely on this behaviour, and we’d break them if we prevented it now. For new apps, it’s safe to prevent them from doing the bad thing 
Yes, that will work, although it might be a little overkill in places. The reason it will work is that deleting your theme CSS (or at least deleting the old theme from it), removes any CSS that could depends on the old class names or on having Bootstrap loaded. It’s a bit “amputate the leg rather than setting the bone”, but it does avoid a bunch of messing around with legacy themes There’s nothing wrong with keeping built-in Anvil components around, but they’ll look a bit plain without theme styling.