How to stream data from server to client?

Anvil Q&A
13

I got it to work using the task state without having to use the data tables. I just put most of the server code, including the definition of the custom callback handler, inside the background task. The callback handler then tweaks the task state directly. I then interact with the task state from the client side.

The streaming is alright - not as smooth as what I see in the command line but it is good enough.

Example on the server:

@anvil.server.background_task
def ask_model_new(query, mchat_history=[]):
    anvil.server.task_state['response'] = ''
    class MyTaskHandler(BaseCallbackHandler):
        def __init__(self):
            self.tokens_stream = ""
    
        def on_llm_new_token(self, token: str, **kwargs) -> None:
            self.tokens_stream += token
            anvil.server.task_state['response'] = str(self.tokens_stream)

Example on client (in the button click event):

with anvil.server.no_loading_indicator:
      # this is a server function that launches the background task and returns the task object
      self.task = anvil.server.call('send_msg_call', self.new_message_box.text, self.mchat_history)
      self.new_message_box.text = ""
      while not self.task.is_completed():
          # print(self.task.get_state())
          if 'response' in self.task.get_state():
              self.conversation[-1]['text'] = self.task.get_state()['response']
          self.refresh_conversation()
4 Likes