It would be nice to have a tutorial or updated example but there are no Anvil enhancements needed to make this work. It is a matter of using the relevant Python library in the server code to get responses to questions.
I created a chatbot with LangChain in the server code and streamed responses via background tasks, similar to Mark’s approach but without Firestore. See this thread: How to stream data from server to client?