To answer your specific question: Yes, you can use Cloudfront with Anvil! However, a bit more information would help determine whether this is the right thing to be doing.
Is this media you’re serving from a Data Table? If you serve media from public API endpoints with constant URLs (we discussed this recently), then those URLs will cache perfectly with Cloudfront or any similar service. (The private, per-session URLs won’t cache well, because they only work from one session.)
However, jumping right to a service like Cloudfront at this point might be using a sledgehammer to crack a walnut – there are usually easier ways to speed up your Anvil app. Can you break down what’s being slow? (If this involves private details, you can try DMing me.)