Accelerated Tables for Everyone DELAYED to late April/early May

Announcements
1

Update: This release has now been pushed back until later in April or early May. We will update this post when we have a new date.

Hi all,

Accelerated Tables is becoming generally available! This will bring substantial performance improvements and a bunch of exciting new APIs to everyone.

Even though we expect this to be an unqualified upgrade for everyone, we’re aware that changing table performance underneath your apps is something some folks will want to test carefully before the default changes. Hence:

Accelerated Tables will be enabled for all apps in a few weeks’ time. If you want to test and tune performance, you can do it before then.

In case of problems, it will still be possible to switch back to the old Tables implementation for a while after the changeover, so that you (or if necessary we) can resolve any outstanding issues.

In the meantime, if you haven’t yet enabled Accelerated Tables in your app’s Data Tables settings, do it – it’s great!

Q: I have a large app in production, and I’m worried. What should I test?

  • The biggest change is that by default, Accelerated Tables now loads every column (including Simple Object and multi-link columns). The original Tables did not load those columns until you touched them (causing the infamous extra round trips), but if you were relying on this behaviour to avoid loading large data sets you might want to use q.fetch_only(). (While you’re at it, if you’re only using some of your table’s data, use q.fetch_only() to get only the data you need!)

  • If you have previously been using undocumented internals (eg _spec) to inspect Data Table objects, those internals are about to change, and your code will break! Fortunately, with Accelerated Tables, the operations people were generally using undocumented internals for are now fast using the official API!

  • Remember that, during the beta, the Accelerated Tables switch is stored in your source code - that is to say, if you check the box in the development version of your app, you can test that without changing how your production app behaves. You can then deploy the change to production by publishing your changes, or roll them back.

  • This changeover date does not apply to Anvil Enterprise installations. If you’re using Anvil Enterprise, you’ll receive this change when you take your next update on your normal schedule.

9 Likes
3

Do we have options to get only specific columns when using get_user() now? Because I have enough linked data stored inside the user row to make my app unusable if it tries to get all columns at once.

1 Like
4

I have a similar problem (nothing to do with the users service though) with a set of data that I am not actually using yet in production so I’m not going to panic.
After thinking about the problem for a minute or so I think the pattern that emerges will probably result in something similar to the old behavior through the use of controlling your own set of multiple trips.

Edit: Don’t do any of that, read below this post and use: q.fetch_only()

̶.̶.̶s̶o̶ ̶o̶n̶e̶ ̶s̶e̶a̶r̶c̶h̶ ̶w̶i̶t̶h̶ ̶̶̶̶q̶.̶o̶n̶l̶y̶_̶c̶o̶l̶s̶(̶)̶̶̶̶,̶ ̶a̶n̶d̶ ̶t̶h̶e̶n̶ ̶a̶ ̶s̶e̶c̶o̶n̶d̶ ̶r̶e̶t̶r̶i̶e̶v̶a̶l̶ ̶s̶e̶a̶r̶c̶h̶ ̶f̶o̶r̶ ̶e̶a̶c̶h̶ ̶r̶o̶w̶ ̶t̶h̶a̶t̶ ̶y̶o̶u̶ ̶n̶e̶e̶d̶ ̶t̶h̶e̶ ̶s̶p̶e̶c̶i̶f̶i̶c̶ ̶d̶a̶t̶a̶ ̶f̶r̶o̶m̶ ̶w̶i̶t̶h̶ ̶a̶ ̶d̶i̶f̶f̶e̶r̶e̶n̶t̶ ̶̶̶̶q̶.̶o̶n̶l̶y̶_̶c̶o̶l̶s̶(̶)̶̶̶̶ ̶t̶a̶r̶g̶e̶t̶e̶d̶ ̶a̶t̶ ̶t̶h̶e̶ ̶f̶o̶u̶n̶d̶ ̶r̶o̶w̶ ̶o̶b̶j̶e̶c̶t̶.̶

̶Y̶o̶u̶ ̶w̶o̶u̶l̶d̶ ̶d̶e̶f̶i̶n̶i̶t̶e̶l̶y̶ ̶h̶a̶v̶e̶ ̶t̶o̶ ̶w̶a̶t̶c̶h̶ ̶c̶l̶o̶s̶e̶l̶y̶ ̶h̶o̶w̶ ̶u̶s̶i̶n̶g̶ ̶t̶h̶e̶ ̶d̶a̶t̶a̶ ̶f̶r̶o̶m̶ ̶a̶ ̶r̶o̶w̶ ̶o̶b̶j̶e̶c̶t̶ ̶t̶o̶ ̶r̶e̶-̶r̶e̶t̶r̶i̶e̶v̶e̶ ̶m̶o̶r̶e̶ ̶c̶o̶l̶u̶m̶n̶s̶ ̶f̶r̶o̶m̶ ̶t̶h̶e̶ ̶s̶a̶m̶e̶ ̶r̶o̶w̶ ̶w̶i̶l̶l̶ ̶i̶n̶t̶e̶r̶a̶c̶t̶ ̶w̶i̶t̶h̶ ̶t̶r̶a̶n̶s̶a̶c̶t̶i̶o̶n̶s̶.̶

5

You also have q.fetch_only to play with, to avoid fetching high overhead columns until they’re needed. Still doesn’t help with the user object, though, unless you move all your anvil.user.get_user() calls to the server (which then messes up the password reset mechanism).

6

@jshaffstall you just reminded me we already went over this with a question @rickhurlbatt asked when the beta came out, also we had built some code for “all columns except explicitly specified ones to leave out”
we came up with:

2 Likes
7

Sorry, I typoed up there - I did in fact mean to talk about q.fetch_only(), as that’s how you can replicate the behaviour of “only fetch these columns, but load others on demand if they’re accessed”. This should solve @ianbuywise’s problem.

@divyeshlakhotia - just checking, have you tested your app with Accelerated Tables and determined that this specifically is a problem, or is this speculative for now?

1 Like
8

Thanks for the reply.

Yes, I have checked it out. anvil.users.get_user() Takes about 15-30 seconds on a good internet connection once I enable Accelerated Tables.

And it makes sense too because if we consider linked rows, getting a single user row could mean getting around 50 novels along with it!

1 Like
9

All right – thanks for the input. We’re not going to ship with an issue like that, so we’re either going to ship a fix in the next couple of working days or push back the cut-over date.

Everyone else, please continue to test your apps with Accelerated Tables and report anything that’s an issue!

3 Likes
10

It *sounds like an easy fix, to have the .fetch_only() just grab the email column of the users table and nothing else (To replicate the old behaviour), but inside the “secure zone” of the users service.

*obviously everything security related sounds easy, but hardly ever is. Also nobody asked me, so feel free to ignore my ramblings…

11

Two weeks is not a lot of notice about unavoidable & potentially breaking changes to production, and I’m worried that users who aren’t regular forum readers will miss this entirely. Not all Anvil users have switched over to the new Editor yet, let alone tried the new tables.

2 Likes
12

To sort of play Anvil’s advocate (but not speaking for them, to be clear–just my personal two cents), the default Data Tables caching/fetching behavior that is due to change was never a documented feature, so there wasn’t even that form of implicit assurance that it wouldn’t change. Relying on undocumented implementation behavior is at-your-own-risk.

Also, for anyone concerned about the fetching/caching behavior of Data Tables, there’s been 10 months to experiment with taking greater control of that behavior via the Accelerated Tables Beta. (There hasn’t been a way to take control of get_user fetching, though, so I’m relieved to see that being prioritized now.)

13

I agree, but I also have apps that I forgot they even exist, they just work. You know, no news, good news. I wouldn’t like to get a few calls because all of a sudden they stopped working.

I’m not too worried because I rarely use linked rows, exactly because there is was no control over what’s fetched when.

Perhaps it would be safer to automatically switch (or force) accelerated tables on new apps and, instead of automatically switching existing apps to accelerated tables, send a notification email to the app owners and give them a few months of procrastination time.

3 Likes
14

Thanks for the feedback, all. We’ve updated the plan: On the 14th, all apps will be migrated to Accelerated Tables, but for a while it will be possible to move back to the old tables implementation in case you encounter problems. I’ve updated the post above to reflect that.

We will be updating get_user() so that it does not fetch all link columns on every load (ie it will act as though it’s got a fairly minimal q.fetch_only(...) clause). This will ship in the next few days, and will enable apps such as @divyeshlakhotia’s to run unmodified.

5 Likes
15

Early on, we put quite a few housekeeping columns in our Users table. Probably, a lot of apps did, in order to avoid having a “mirror” table, with all those columns, and the (potential) complications of keeping the two tables in sync. (Those complications seem small to me now!)

So it will be useful to know exactly which columns will (and will not) be accessible from the result of anvil.users.get_user(), in the case where Accelerated Tables is turned on, and which of these will be read-only. Edit: client-side. I wouldn’t expect any restrictions server-side.

App security argues for an absolutely minimal set, with everything read only. Backwards-compatibility argues for returning the entire row, and everything writable. The actual picture is probably somewhere in between. I checked the Accelerated Tables and get_user() documentation, and as of this writing, found nothing on this subject, so I’m asking here.

I can spend a bunch of hours building and testing to find out, but if there’s a quick answer handy, that can be avoided.

1 Like
16

My understanding is that everything in the user row will continue being accessible when get_user is called client-side. That is, we’re not talking about only_cols here, not restricting access to columns. Rather, we’re talking about fetching/caching: which columns will be accessible client-side from get_user without an additional server call.

My guess, seeing @meredydd’s edit, is that get_user will thenceforth fetch and cache all columns that are not linked-row columns. So, subsequently referencing a linked-row column, like user['other_table'], would trigger a server call to retrieve that column, but referencing any other columns (like user['email']) would not trigger a server call. But that’s just a guess, and I second your request for clarity/specificity.

1 Like
17

I’m inclined to agree with your reading, Tim. Given how widely anvil.users.get_user() is used, and how central it is to many apps, it pays to be explicit about every change to that function. Implicit means each developer has a “just see for yourself” situation, which is a potentially open-ended block of work. Explicit allows a developer to scope and plan (and often avoid) that work in advance.

I’m looking forward to that change, whenever it goes live!

2 Likes
18

@hugetim is bang on. The semantics of anvil.users.get_user() will not change - all columns are accessible.

What happened here is an optimisation change that had unwanted results, and has now been reversed. get_user() originally did a shallow load of the table row (no following link columns unless/until you specifically accessed that column); we’re preserving that behaviour.

1 Like
19

Only on old apps or also on apps?

I understand the decision to introduce an inconsistency in order to preserve backward compatibility, but I really don’t like that inconsistency.

The introduction of the new accelerated tables is a great time to do some improvements on both performance and consistency.

Having the old behavior on the get_user() is inconsistent with the other functions and is not intuitive. It wasn’t intuitive in the past, it will be even less in the future. The very fact that you (Anvil) were going to make its behavior consistent with the other new functions confirms it.

This type of stalemate is often addressed with an ugly but effective solution:

  • keep the old get_user()
  • create the new get_user2()
  • document get_user() as obsolete
  • show only get_user2() on the auto completion

Having both get_user() and get_user2() will make everybody happy:

  • old apps will not break, because willl keep the old behavior
  • new apps will use the new behavior, powerful, consistent and intuitive
  • users will understand what’s going on, there will not be confusion on the behavior

Having a 2 at the end of a function name may sound ugly because it is ugly, very ugly. But it is also very common and very effective.

I work with many cad and cam apis, and if they have been around for longer than a few years they all have functions ending with 2. Or with 3. In some cases also with 4. Excel apis have many functions ending in 2, and Excel has even the Value and Value2 functions exposed to the user.

2 Likes
20

Update: We’re pushing this release back a bit. As it’s PyCon next week and a bunch of us will be there, we’ll be rescheduling this when we’re all back at our desks – so probably around the end of April or beginning of May. We’ll update this post when we have a new date for the changeover.

In the meantime, please go click the box and enable Accelerated Tables for your apps!

3 Likes