Uploading LLM to Anvil

Anvil Q&A
1

What I’m trying to do:
Upload a Large Language Model to anvil for using GPT4ALL within the server module to get it to interpret data.

What I’ve tried and what’s not working:
I was trying to upload to the Data Files as well as the app’s Assets. Because the files are around 4GB to above, they seem to be drawing errors due to the file size limits of Anvil. What is my best option at storing the language model and loading it in Anvil?

2

Welcome to the Forum!

This has been discussed before in the Forum, so I recommend using Search. It’s a great way to find answers without waiting.

1 Like
3

Yeah, that’s not going to work! The server environments on Free or self-serve plans aren’t built for LLM hosting. I’d suggest one of three options:

  1. Use a hosted model. As well as the big name-brand APIs, there are plenty of services that will host LLM inference on niche models for you.

  2. Bring your own compute: Use the Uplink to connect your computer/cloud instance to your app, and run LLM inference there

  3. Add Extra Server Resources: With this upgrade (available on the Business Plan), you get a dedicated AWS instance for your backend Python code and data, with as much CPU as you like, and we can definitely get that set up for LLM inference. Drop us a line at sales@anvil.works if you’d like to kick this off!

2 Likes
4

Thanks! I gotcha! I’ll host it on my computer!!