Monorepository vs Microservices for large 1000+ Files Anvil App

Anvil Q&A
1

Good Morning,

So our Core Application is getting fairly large. To give you a hint VS CodeCount results below.

image
image1027Ă—565 37.8 KB

First, for all Anvil doubters, this is running in production for 100 paying customers and anvil is very much capable of handling that. :rocket:

But still we face some issues as our code base keeps growing.
First - the editor is getting very slow which makes the dev experience not as fun as it could be.
Second - the inital load time is around 8 seconds (only 1 roundtrip to check the user)

image
image2313Ă—1419 332 KB

Monorepository vs Microservices
While the initial load time is for sure reducable and the editor will hopefully become more powerfull over time, we are thinking of splitting our codebase up.

I’m curious how you approach larger anvil apps, especially when there are a few developer. Do you build several smaller applications or keep it in a monorepository?

I like the monorepo approach since it makes developing, versioning, staging and testing much easier. Since we would otherwise have about 10 smaller applications which all get frequently updated, have qa, beta, prod environments and get released in 3 week cycles.

Whats your take on that? What where your experiences?

1 Like
3

Hi @mark.breuss, it’s great that you could have 100s paying customer. At one point, my app has about 200 and it was working fine as well.

When one file has about 2K line of codes, the editor becomes unusable in my case, so I just put code into several files, and the issue is fixed.

As for the loading time, there are many techniques in the forum to speeding it up, but admittedly it varies from case to case. Upgrading your current plan to the next one also somehow solves the performance issue as well, but a bit expensive.

I would stick with the monorepo since it is what Anvil is designed for, once you go to the microservice, there are a lot of unexpected issues. I tried, and failed, so good luck then.

1 Like
4

I don’t have experience with Microservices but I maintain a large Anvil App as well so hopefully, these suggestions might help.

  1. I agree with @Tony.Nguyen that splitting code into multiple files will speed up the Anvil Editor.

  2. I investigated your app and noticed your theme.css and code in Native Libraries aren’t really minified. Minifying them can increase the loading by a slight margin. Although this can make development harder for you.
    Hopefully, Anvil will consider implementing Serve Minified Assets

  3. Looks like you use anvil_extras in your project. While it’s great, Anvil Extras can have a big impact on your loading time. So if you see a decent decrease after removing it, consider building a customized version of it that only uses the stuff you need, or try taking things on your own.

  4. Some of the images in your app are directly uploaded from the Anvil Editor. This can have a big difference since these images are loaded during the initial loading. Use Image URLs instead.

These tips should give you a good decrease in speed.

Also, in case you want to go the other way, this might give you some ideas - Server Side Routing Attempt

1 Like
5

Amoni has a shiny new command that will do this for you - and the default starting point is the anvil material design theme.

3 Likes
6

The short version:
I have 100+ apps sharing code via 10+ dependencies.

The long version:
@divyeshlakhotia made a great summary about performance improvement.

I have 100+ small apps, all sharing the same tables, all depending on the same set of small dependency apps.

When I create a new app, I clone it to my computer, replace its theme folder with the theme folder of an app called Standard Theme and push. At this point the new app looks like all the other apps.

Then I add a dependency for each behavior it needs to share with other apps.

For example, most apps need user authentication, so I add the Users service, share the Users table from other apps (need to temporarily switch to the old editor for this), add the dependency to the custom component HeaderWithLogin, drag the custom component on the header of the main form and I can use self.header_with_login_1.user_permissions() and other methods for the user management.

Another example is the KeyValueCache, a dependency that I use to store key/value pairs with an expiration date and with other policies. I use it in 20% of my apps, and when I used it, I need to share the KeyValue table.

Some dependencies depend on other dependencies themselves, like the InputBox that depends on the Validator that depends on Anvil Extras.

I used to have one large dependency called CommonComponents, but it was difficult to maintain and it was forcing every little app to have all the services and tables even if it didn’t use them. I also had a few scary moments when I made a little change that broke dozens of apps. So I have split it in 10+ little dependency apps, each doing its job.

I wish each module of Anvil Extras was available as its own smaller dependency: I don’t like to load 20 modules when I need only 3. But the workaround is easy: make your own stripped down version of Anvil Extras by removing the parts you don’t use (I did it for a short period, but depending on the official Anvil Extras is easier and I am lazy).

I have apps that use pandas and user authentication and tons of other dependencies, and I have apps that work as http endpoints and don’t even have forms, authentication or dependencies.

The bad part of this approach is that I can’t test all in one shot (but, seriously, who tests all :blush:?). One day I might take the amoni road and setup a big test environment for all my apps (but I don’t see that day coming any time soon).

The good part is that I have no performance problems and I can focus on the job of each app without risking to break anything.

Another advantage is that most apps start as a proof of concept prototype, then they quickly become production apps.

EDIT
Immediately after posting this I noticed the announcement about the self-service package installation… here is another reason to use many small apps: you may need to use different environments on each app :slight_smile:

2 Likes
7

Thanks all @Tony.Nguyen @owen.campbell @divyeshlakhotia @stefano.menci for sharing the advice and lessons learned.

Regarding startup speed - I’ll try out the suggestions above and report back what worked for us!

In the case for Monorepo vs Dependencies I’ll stick with the monorepo for now mainly - for the simplicity in managing different versions etc. And until hopefully this request Depend on a specific Branch of a Dependency is implemented.

Thanks all and a great weekend to you!

Mark

2 Likes
8

I just read here that your app has thousands of forms, and I thought I would add 2 more cents to this old post.

It’s unlikely that one user will need thousands of forms during one session, so creating an app with thousands of forms is like saying “dear user, I know you are going to need only 10 forms today, but you’ll need to wait for thousands of forms, including admin ones, obsolete ones and test ones, to be downloaded to your browser because I’m lazy and don’t want to split the mega-app into smaller zippier apps”.

I know I’m stating the obvious here, but I recently did some housekeeping, I deleted dozens of unused apps and left many very rarely used there just because one day I might need them again. Then I saw your feature request mentioning thousands of forms, and I thought “Maybe a good percentage of those forms are just like that percentage of my obsolete + rarely used + test apps”.

Of course I’m flying solo while you have a team, so you may be less sloppy than me :slight_smile:

9

I have not seen many cases where growing the team and product made the code base less sloppy :wink:

But our reality is a bit trickier.
Beeing an ERP System there are many different parts for the user to be presented. And almost all of them share the same data classes.

If we were to split it, it would proboably result in one data_core dependency and about 10 “UI” Apps which all depend on the datacore.

New features will most likely need changes in the data_core app and some change on a “UI” app.

We are currently 4 Engineers, and I do not see a way we could continue to implement features in such a setup. Because the “UI” app either depends on the master wich would result in chaos if 4 peopel merge in paralell while developing. Or on a specific version wich would be very tedious to debug new features.

Unless I am overlooking something at least at the moment we can’t split our codebase.

1 Like