Moving Beyond Anvil (General Discussion)

Anvil Q&A
1

Some context: I’m the sole developer at a media company that has never had a dev on staff before. 18 months ago I’d never coded python, and now have written probably 100s of thousands of lines, with my apps being use for a wide variety of backend admin and user facing products. I’ve been the only user thus far, but by March we will be looking at dozens of end users

So on my journey with Anvil, I’ve found 99% of the things I need to do can be achieved with Anvil in some way. The main issue is looking at scalability, and certain features outside the boundaries placed on Skulpt. While I’m sure stepping beyond a Personal plan will help performance, I imagine at a certain point trying to manage dozens of simultaneous mongodb find and update/replace operations will become a problem, and trying to tie together an increasing number of outside APIs and services will become combersome.

Currently mongodb is the main outside service connected to Anvil, due to the DataTables being mostly useless for my use case (producing 100s of thousands of rows of data weekly). Ive recently added Twilio and SendGrid into the mix, and am slowly working on adding my own Discord bot implementation. I’ve seen a solution on this forum for sending push notifications using Firebase, which would be a super useful addon. I’m also close to beginning work on an S3 integration, as Dropbox and Gdrive is quickly being annoying/useless or expensive.

The main reason for enjoying Anvil is the speed at which I can spin up a working prototype and then easily work that into a production solution (production scale within the context). Being able to quickly create a UI for a management app of some API is so useful, and the business team can see real gains in the way my time is spent. Also having it be cross-platform automatically is great, no need to handle multiple app dev pipelines, with different data models and troubleshooting requirements.

My question I guess is, as the need to expand services and APIs, where should I be looking to move the system beyond what Anvil can currently provide? My current research says Amazon AWS, as the wide array of available services would solve many issues for me. Being able to send events to EventBridge, which routes the event and data exactly where it needs to go, and ensures the task is completed (or retyied upon failure). I will soon have a workflow where an Anvil app will send a command to a Windows EC2 instance to run a script, wait for output from a Photoshop/Premiere render, then upload output to S3 and have notifications sent out that the output is available for consumption. Anvil feels clunky when looking at larger systems like this, but I’d love to still use Anvil as the UI for these systems, utilising the APIs from things like Mongo Atlas App Services API to edit/monitor Triggers, AWS APIs for adding and monitoring events from EventBridge and ensuring EC2 instances and spun up and ready to go, ensuring tracking links in SMS/email are handled correctly etc.

Does anyone have any experience with moving up in scale like this and where should I be focusing my time. I will have someone helping me some time next year, but I don’t like assuming I will have any lighter of a workload :sweat_smile: any advice or recommendations from the geniuses here is greatly appreciated. I don’t wanna get 3 months into a major AWS system build when I find out that gcloud was the way to go. Cost is obviously a factor, but I hope that as time goes on, that becomes less of an issue

2

Looking a bit more locally, there are certain other considerations I’m unsure about. I’ll go through a few here. I’m mostly just throwing stuff at a wall for discussion, I haven’t really spoken to any devs about developing since I’ve started, and I’m entirely self taught so am missing a lot of context and knowledge about developing in general

When I first added mongoDB, I was using the pymongo module. I found quickly that this was rather slow to return results, so moved to using anvil.http and Mongo’s Data API. As I move forward, I’m not sure if I sure consider something like Mongo’s GraphQL API or the Realm SDK, as these may be more performant ways of acquiring and presenting data. While anvil is the only way users can interact with these systems currently, I expect in the future for that to not be the case. Using something like GraphQL will mean I can readily mobile platforms for example. But perhaps I’m complicating it and the Data API is the simplest way to do it

MongoDB has Functions and Triggers, which are similar to Anvil’s Endpoints and Scheduled Tasks, so I’m unsure if these should be moved to be handled by Mongo, instead of Anvil as they currently are. Or when considering AWS, would it make more sense to move it there entirely. There’s the performance, scale, cost, ease of use/implementation to consider

Is continuing to lean on Anvil for creating UIs okay? As things scale there will be a need for more complicated and dynamic UIs. At a certain point, creating nested repeating panels with buttons that look nice and wraps well on mobile is frustrating, but I’m not a UI designer, and not sure whether the time spent on learning that is worth the tradeoff on slower/simpler UIs with Anvil

I’m gonna need the capacity to transfer large files from client to server soon. Like 3GB per user or something. Doing so with Anvil is not feasible from my understanding, this is where S3 and AWS DataSync come in. At what point is using DataSync and AWS Lambda/Elastic Transcoder better/more performant than asking an Anvil Server Function to download many 3GB videos and compress them using ffmpeg and then reupload elsewhere

I’m sure there’s more I wanna discuss, might add more comments below if necessary

3

ML/AI is something that will come to these systems eventually. I will need to ingest live streams that are detected by a model, with annotations being send elsewhere to be stored/processed. The training process will need to be automated, with model training, evaluation and selection, and deployment of models all being as easy as possible. The results from these models need to be easily accessible to be processed and analysed, and the compiled need to be available to end-users in a variety of places in an automated fashion. Imagine attribute detection models collating results over a period, presenting them in a table and producing visualisations from the data
This isn’t stuff coming soon, but definitely something to consider for the future

4

I would also like to discover which cloud DB service offers fast, reliable and cheap backend hosting. Visual drag-and drop query building (just like Anvil forms but for the backend) would be very useful.

I agree with two things: 1) Anvil is an amazing framework for playing with Python, quickly building from idea to a working prototype using nothing but python. 2) One can really apply python programming concepts to something visual - like playing with Lego - so I really learned Python very rapidly, from totally newb to maybe junior intermediate level.

However, I think one area Anvil really needs to offer more in, is a better data service. I find the ability to perform more than simple joins is missing, unless I am missing something that is there. I know they offer a more sophisticated DB with their enterprise plan but that is way more expensive than using an external DB service. About 10X more expensive, which is a deal breaker for most hobbyists imo.

So I would like to know what plans Anvil has for offering a really slick DB - relational and document like Mongo. Interestingly, Wix offers Mongo by default but that is all Javascript World and I wanna be a Pythonista. So I would love for Anvil management to share some inputs on what they can offer. Thanks.

Others – please let me know what great DB backend cloud providers there are out there with visual drag and drop query builders. Thanks.

1 Like
5

Is it feasible to store 3GB per user on a local hard drive hooked up via uplink (with appropriate backup)? A few multiple terabyte drives is very inexpensive, and you can do whatever you want with them (install whatever database system, use for simple file storage, etc.)

2 Likes
6

The solution needs to be as simple as humanly possible. Some end users would lose their own head is it wasn’t screwed on. That 3GB file needs to be uploaded to be distributed, and doing so via a Server Module is not feasible for a number of reasons. First being that you first have to transfer all that data to Anvil, then deal with it, then send it back out. If the file upload goes straight to S3, processing the data and sending it out becomes a different story

1 Like
7

Use Anvil for its productive benefits whenever possible, and trade out pieces, as you’ve done with Mongo, whenever needed. I think Flutter is about as good as it gets for cross platform UI. You can start by connecting Flutter to Anvil APIs, and progress from there if/when Anvil doesn’t meet your needs in other ways.

2 Likes
8

I think you’d find implementing MongoDB into Anvil via pymongo is not that hard. Combine that with the GUI for accessing and monitoring Atlas clusters, you’ve got a very capable, scalable data platform. For me, the issue is performance, Anvil has all the features but less than perfect performance (to clarify not complaining)

1 Like
9

Product page for Flutter looks great, will take a look.

My overarching point with this post was basically: at what point am I trading out every feature of Anvil with something else, and by the end nothing of Anvil is left. Do I bother making that transition sooner rather than later, if I’m going to do it anyway?

10

Flutter connected directly to any scalable DB service (Mongo, Firestore, a hosted Postgres DB, etc.), plus some backend function service is powerful, but I don’t think as productive or pleasant to work with as Anvil. Use Anvil for anything that it can scale for, and enjoy your life :slight_smile: For everything else bring out the big guns.

2 Likes
11

Premature optimization is the root of all evil :).

It sounds like you are finding ways to tie into the services you need, and that Anvil is providing value in being that “nexus” - no small part because it runs Python on the server and anvil.js provides that front-end escape hatch (can you use wix from Anvil.js?). Adding microservices to your Anvil app is totally reasonable in my opinion, where Anvil really is just orchestrating and making API calls or whatever. Same goes with ML/AI etc.

Don’t underestimate your scaling options with Anvil either - sounds like you are on a Personal plan, so you can grow into Business or Dedicated as/when the time comes. Suffice to say, it sounds like you are very far from any “limits” you might face on the Anvil side!

Finally, if you do move to “something else”, the problem you state isn’t solved. If you want to work with MongoDB, integrating ML/AI pipelines, with some AWS services for video processing etc, you’re STILL going to need something to hold it all together.

Aside: You CAN upload very large files in Anvil. See this post by @owen.campbell:

3 Likes
12

Since you said you are just throwing ideas at the wall, check out these two things, they might fit into the puzzle:

  1. Redis is a NoSQL style (mostly) in memory database solution that has many of the things like scheduling, auto-replication, a Pub/Sub notification system, etc. If you are talking about a user uploading 3gb of data that will immediately (or en-que / asap) be processed and then have a subset of data generated that can then be stored, then its possible that writing to a disk is not actually what you want to do until the final product is generated.
  1. Cloudflare - I have not researched this in-depth, but I hear they have some kind of edge computing product that uses automatic services (containers, etc) replication to provide edge computing to reduce the time / distance your 3gb uploads will have to take place over.

I have no idea how either of these will work for you, (I have not used them for anything in production, many others have) but they are probably worth a quick glance.

5 Likes
13

Do you mean

NoSQL-style (-mostly) in-memory database

or

NoSQL-style (mostly-) in-memory database

? That is, is “mostly” modifying what comes before, or after?

14

For UI, also take a look at Quasar:

https://com-pute.com/quasar/quasar-drawer-layout.html
https://com-pute.com/quasar/quasar-table-with-selections.html

It should take you 1 sitting to learn how to hook up Quasar widgets to the MongoDB data API, using Fetch or Axios. The Quasar UMD doesn’t require installing any tooling whatsoever - but then when you want to package for mobile and desktop app stores, the Quasar CLI has you covered.

15

Oops, their site is https://quasar.dev/

16

It used to be 100% in-memory, and it can still be used that way (its main feature) but it now supports point-in-time backups to a disk for redundancy.

17

Hi Kaleb,

Thanks for sharing your thoughts openly.

I’m afraid I cannot offer you any advice concerning complementary backends and databases (my experimentation to this end is limited to external server functions hosted on Python Anywhere – hardly what you’re after :blush:).

However, you also pose the question, “is continuing to lean on Anvil for creating UIs okay?” This resonates with considerations I had myself last year… While I’m sure we all agree about Anvil’s benefits re speedy development, I started to run into constraints especially re styling of Anvil standard components. Although these issues could be fixed, I increasingly found myself spending a lot of time on deciphering the structure of Anvil components and overwriting Bootstrap classes – for challenges that would have been really simple in plain HTML and CSS. Moreover, I was also frustrated by the fact that many examples and tutorials on the web and in the literature needed extra work before they could be put to work in an Anvil context. And many cool browser APIs seemed to be hidden behind a layer of abstraction in Anvil… (although they are actually not).

It came to a point where I flirted with the idea of going with a vanilla JS front-end hosted on e.g., GitHub Pages or Netlify - with a backend still on Anvil. However, working in Python is just SO much nicer (and more efficient, at least for me) than working in JavaScript, and Anvil’s features for client-server communication (i.e., the whole ‘@anvil.server.callable’ story) is just SO much smoother than writing HTTP endpoints and interacting with them with JS ‘fetch’. This conundrum seriously cost me many a sleepless night: Stick with Anvil front-end and accept the limitations (aka extra work to get what you want) or go vanilla JS and miss out on Python in the browser and super-easy client-server communication?

For me the key to resolving this front-end dilemma turned out to be embracing anvil.js. It dawned upon me that with this API (and a little creativity), there are no limits as to what you can do in the front-end with Anvil (in Python!) – compared to the options you have with JavaScript in the wild. My conclusion after a period with one leg in the “JavaScript-front-end-hosted-on-GitHub/Netlify/etc.)”-camp was clear: Stick with Anvil but focus on anvil.js rather than anvil. From my perspective, the only negative aspect of this choice is that I pay a performance price with respect to loading a lot of stuff that I’m not using (jQuery, Bootstrap etc.).

At some point, I also had the thought: “If I want to be able make real good front-end work, I need to use a framework like React, Vue or Svelte, right?” And I did some experiments with Vue in and outside of Anvil. But man, don’t go down that route! Granted, my word on this is not necessarily the truth, but in my experience, it’s not worth it. With modern vanilla technologies like web components, shadow DOM and constructable stylesheets (AND the ability to use client-side Python with Anvil), I personally cannot justify the investment in keeping up with – and paying the performance penalty for using – a JS front-end framework.

Best,

Rufus

4 Likes
18

Definitely appreciate your input Rufus, it’s great to hear from people who have already trodden paths that I am currently looking to be going down. I agree with your conclusion regarding Anvil’s UI needs more massaging as you go on and it makes managing things hard. I haven’t gone as far as rewriting the Bootstrap classes, but definitely struggle to wrap my head around it and the best way to approach issues

Based on suggestions here, Flutter seems to be the UI framework that I am leaning towards. Having an easily multi-platform UI is important to me, a solo dev doesn’t have time to consider the myriad of screen sizes and devices that need to be supported. It also seems to have a myriad of packages and plenty of community support, and seems to solve some specific UI problems I will have. I imagine the best way forward would be to slowly replace my Anvil apps with Flutter apps, but I definitely will continue to use the backend framework that I’ve built for my systems, that Anvil supports. I’m unaware of how flutter differs from perhaps React or Vue as I have essentially zero UI experience, but this feels like the natural progression that needs to happen

One thing I’ve come to realise is that I need to simply focus on the now and not look too far forward with all this, its easy to get lost in the sauce and go around in circles for days with things that appear super helpful or look good, but in the long run don’t achieve anything. Definitely don’t have time for that!

Appreciate all the responses, its given me plenty to think about

2 Likes
19

As already stated in other answers, Anvil is as good as any other tool out there as the director. There is no integrated tool that covers all your needs, so you need a director, you can keep Anvil. Anvil will be the bottleneck in your scaling only when you get really really big.

The only Anvil constraint that is difficult to replace is flexibility of the UI. As @uat said, there is no easy replacement for that part. But it is not a scalability problem.

So my answer about the scalability is keep Anvil.

Taking about UI flexibility, another factor I would keep in mind is your personal productivity. If I understand, you, like most of us hanging on this forum, are flying solo. In my opinion, considering that I am flying solo, I give more importance to getting stuff done than getting stuff pretty. I spent some time (and I asked for some help) long time ago, when I decided how al my apps should look like (my apps are all for my company and share the same look). Since then I just copy the same theme folder to all the new apps and use the form editor with very rare use of anvil.js. I can do that because I have a bunch of custom components I’m familiar with and by now I know where I’m going from day one.

So my answer about the flexibility of the UI is to stick with Anvil and be happy with less than perfect UI until your team grows in size. Then, when you can afford sinking time in making things pretty, but it has to be clear that your team will grow to make things prettier, not to build more apps, you can switch to something more flexible and less productive.

9 Likes
20

Advice here is amazing, you’re correct that I am flying solo and so appreciate the relevant position you put forward. I agree absolutely with the need to stay productive, and this is where Anvil truly shines. I’m on r/python and r/learnpython and I feel like I’ve got this huge secret when I see people have issues that Anvil could solve instantly, especially as a beginner. I remember spending far too long trying to create a useful application using tkinter, once I picked up Anvil I was able to translate that app over within a couple days, when the initial build was easily over a week, considerably uglier and less useful.

I will definitely continue to leverage Anvil for the UI, and the backend, with my first point of improvement being to jump over to the Business plan. My asking about UI is driven by the fact that while currently all my apps are for internal use, that won’t last long. I guess I wanna preempt things that the business team might be asking, or things I should be considering longer term, as I get pulled in 6 different directions all the time so wanna ensure that I have all bases covered. Having opinions from others using all this within a professional sense is great, especially as I further legitimize the things I’m working on, coming from essentially zero formal education on programming.

I dread the day I have to start hiring devs who may know more than me :sweat_smile:

3 Likes