Temporary disclaimer about stale content
In the What not to do paragraph, the rule Do not return database row objects to the client is the one that I follow most religiously. With the exception of my oldest apps, I have zero linked tables. I even refactored a few old apps not to use linked tables.
Instead of using tables with linked rows, I do something like:
- Do one search to find all the rows from the first table
- Create a list of dictionaries with it
- Scan the list of dictionaries and find all the ids of the rows of the second table that I need
- Do a second search to find all the rows from the second table
- Create a dictionary of dictionaries with it
This way I have 1 access to the database per table instead of 1 access for the first search plus 1 access for every linked row, plus 1 access for every simple object row plus etc. etc.
In other words, this way may be hundreds of times faster than using the Anvil app_tables.
BUT…
The accelerated tables have just been released , and they do what I am doing, but they do it better: they allow you to specify which columns to cache, including simple objects, including linked tables, specifying which columns of each linked table, down to multiple levels.
I don’t really know how my approach will change, but I’m pretty sure my next app will use linked tables.
I don’t know it because I haven’t played with them yet (other than some meaningless tests) and because I have the feeling that the accelerated tables are just the beginning of more goodies that are coming up in the near future.
After my first few apps using the new accelerated tables, I will update this post.
For a long time I have been looking for a way to create apps in a consistent, testable and maintainable way.
I think I have found one, I have used it while developing my latest apps and I thought I would describe it here in the forum, hoping that it will help someone.
Introduction
In some of my apps most of the action happens on the forms, in some the action happens on client modules, in some on server modules. With Anvil it is so easy to create an app that you don’t really need to be consistent. You can be messy, do what you feel is right at the moment, and it is going to work. Maybe it will be less maintainable, but it will work.
I don’t like being messy and I don’t like to think about the right approach every time I start a new app, but I could never identify the best pattern until recently. Now I found one that has been great for the last few apps.
This is not the best practice for everybody, but it has been for me for some time and I think it will be useful to others. I will try to describe it as well as I can in this wiki post. Over time I will try to improve it and I may also add examples. I’m going to publish it before it’s complete, otherwise I would never publish it. So feel free to correct and add to it, as long as your contributions don’t change the basic idea.
What not to do
- In client side modules:
- Do not interact with form components
- In forms:
- Do not manage app logic
- Do not call server side functions
- Do not use dictionaries for
itemor lists of dictionaries foritems
- In server side modules:
- Do not manage app logic
- Do not return database row objects to the client
- Do not fix code in
Globalsif you find out that something does not work while working on the UI - Do not add code in
Globalsif you find out that something is missing while working on the UI
What to do
- In client side modules:
- Create one client side module called
Globalswith the definition of several classes, often with one master class and all the other classes hierarchically related to it - Derive all the classes involved in databinding from
AttributeToKey(see below) - Create a
@propertyfor every class attribute that will be used in databinding. Properties of classes derived fromAttributeToKeybehave as dictionaries and play nicely with databinding - Create one global variable
Globals.all_data, the singleton instance of the top level class Globals.all_datacontains all the data used by the clientGlobals.all_datahas methods that call server functions to load data from and store data to database and access other services- Try to create only one method on the master class that writes to and one that reads from the database. Not a black and white rule, sometimes it does make sense to create more methods for loading and storing different types of data
- In order to save to and load from the database, the classes have a
to_jsonmethod to serialize the current instance and afrom_jsonclass method to deserialize and create a new instance
- Create one client side module called
- In forms:
- Use
Globals.all_datato read and write all the required information - Use
Globals.all_datamembers to manage databinding
- Use
- In server side modules:
- Try to create only one function to read all kinds of data from the database, convert to json and pass to the client. Again, not a black and white rule
- Try to create only one function to write all kinds of data to the database
- Include some logic to protect against malicious users. In simple cases this has nothing to do with business logic and can be done by checking that the user is signed in
- If you find out that something does not work or is missing while working on the UI, go to the test suite and add a test that reproduces the same problem, then work on the code until the test passes, then go back working on the UI
What are the advantages?
- Removing all the logic from the form prevents the forms from having logic bugs. The only problems in the form code will affect the interface
- Removing all the logic from the server modules prevents the server modules from having logic bugs. All the code in the server modules is easily testable by testing
Globals - Since
Globalshas no interactions with forms, with databases or with any other services, it’s easily testable offline. Well, not really offline, often an uplink connection is required to test the interactions with the database and other services, but it’s easy to test locally without running the app on the Anvil server - It is possible to import the singleton in the app console and explore all the app data while the app is running and test the logic with
from AppPackageName.Globals import all_data(whereAppPackageNameis defined at the bottom of the Settings - Dependencies page)
What are the disadvantages?
-
I have the feeling that sometimes the databinding fails silently instead of showing an error.
Anvil does a good job at showing problems with databinding when the databinding is done with a dictionary, while when I make mistakes in the databinding I see nothing. I don’t know yet when this happens, I’m not even sure it does happen. I will keep my eyes open and, maybe it’s just matter of raising the correct exception in
AttributeToKey -
Managing the logic on the client side could be impossible if modules unavailable on the client are required
-
Managing the logic on the client side may expose some aspects of your logic that should be kept secret
AttributeToKey
I mentioned earlier that all the classes involved in databinding should be derived from AttributeToKey.
This is the definition of the AttributeToKey class:
class AttributeToKey:
def __getitem__(self, key):
try:
return self.__getattribute__(key)
except AttributeError:
raise KeyError(str(key))
def __setitem__(self, key, item):
self.__setattr__(key, item)
def get(self, key, default=None):
try:
return self.__getattribute__(key)
except AttributeError:
return default
Classes derived from this AttributeToKey will behave like dictionaries and will play nicely with databinding when the component expects a dictionary.
For example this is how this class will behave:
class Candy(AttributeToKey):
def __init__(self, color, shape):
self.color = color
self.shape = shape
candy = Candy('red', 'round')
print(candy.color, candy.shape)
> red round
print(candy['color'], candy['shape'])
> red round
And this is how this class can be used with databinding on a DataGrid:
Summary: often Anvil assumes the databinding is managed with a dictionary and it is possible to replace self.item['color'] with self.item.color. But in some cases, like with a DataGrid, the dictionary key is required, so it would be impossible to target a class member. The AttributeToKey base class allows to use databinding with any class by behaving like a dictionary and returning instance member values.
Setup the test suite
- Create the app, add a client module called
Globals, add a server module - Clone the app to your computer
- Create the file
Test\test.pywith a test suite to test all the classes that don’t exist yet. Here is what an app looks like at this point on my PC, after opening it with PyCharm:
- Push back to the server. Here is what the commit looks like after the push:
See here for details about how to setup a reliable testing environment: Setting up an environment for testing modules locally
Test-driven vs Messy development
Messy development: create the app, skip the test, hope that it works, hope you don’t need any maintenance
Creating the UI with Anvil is so easy that you often start by creating the UI, then writing the code to make the UI happy. Then, eventually, one day, perhaps, maybe, maybe not, you will try to create some tests. The logic is entangled with the forms and it’s split half on the client and half on the server, and you will give up the testing. Then bugs will start showing up, and you will be scared because you don’t know what changing that line of code will break.
Test-driven development: code your tests rather than test your code and feel safe that everything is tested and any future change will not break the app
Don’t waste time setting up a data grid before you know whether you need a data grid. The same applies to all the forms and UI components.
When you start working on an app you know what you want your forms to do, but don’t know the details. Often you need to completely redesign a form because you didn’t think about certain details of the workflow until you tried to write the code in the form.
With test-driven development, you first create a test to test your classes that do everything your forms should do, but without the UI and without the classes. Those tests will obviously fail, complaining that the classes don’t exist.
Then create the classes and develop them untili the tests stop failing.
When all your tests succeed, you are ready to work on the UI: commit and push to the Anvil server. At this point you know how the app logic works, you know your classes and building the forms will be a breeze.
If you realize that your classes miss something, restart from the test: first add a test for whatever was missing, then add it to Globals, then work on the form.
Example of test.py
Here is what the imports look like:
from unittest import TestCase
# Server is required for the uplink connection
import anvil.server
# The user service can be useful to sign in and find database rows linked
# to the current user.
import anvil.users
# The tables can be used to check test results or to get the user row.
from anvil.tables import app_tables
# This import allows the decorated server callable to run on the uplink.
# Without it, they will run inside the app.
# With this import the server callable will run in a thread different from the
# test. Set a breakpoint on the server callable function to debug.
import server_code.ServerModule
# Different statements to import Globals when running on uplink vs on server.
# Importing Globals will instantiate the global variable that will contain all
# the data used on the client.
# For this example let's make "boxes" that global variable.
if anvil.server.context.type == 'uplink':
# running tests on pc
from client_code.Globals import *
else:
# running on Anvil server
from .Globals import *
Here is what tests look like:
class Tests(TestCase):
def test_1(self):
# test functionalities that don't need to save or read from database
box1 = boxes.add_box('box 1')
candy1 = box1.add_candy('red', 'round')
candy2 = box1.add_candy('red', 'square')
self.assertEqual(candy1.color, candy2.color)
self.assertEqual(2, box1.n_candies)
def test_2(self):
# test interactions with the database
anvil.server.connect(r'server_<uplink-key>')
anvil.users.force_login(app_tables.users.get(email='<test-email>'))
box1 = boxes.add_box('box 1')
candy1 = box1.add_candy('red', 'round')
candy2 = box1.add_candy('red', 'square')
boxes.save()
boxes2 = Boxes()
boxes2.load()
box1 = boxes2['box 1']
candy1 = box1.candies[0]
candy2 = box1.candies[1]
self.assertEqual(candy1.color, candy2.color)
self.assertEqual(2, box1.n_candies)
Example of Globals.py, good enough to pass test_1
After creating the tests shown above, you need to create the classes so those tests will succeed. Here is a quick (ironically enough untested) example.
The imports and the AttributeToKey class:
import anvil.server
class AttributeToKey:
def __getitem__(self, key):
return self.__getattribute__(key)
def __setitem__(self, key, item):
self.__setattr__(key, item)
The Boxes class:
class Boxes(AttributeToKey):
def __init__(self):
self.boxes = []
def add_box(self, name):
self.boxes.append(Box(name))
The Box class:
class Box(AttributeToKey):
def __init__(self, name):
self.name = name
self.candies = []
def add_candy(self, candy):
self.candies.append(Candy(color, shape))
@property
def n_candies(self):
return len(self.candies)
The Candy class:
class Candy(AttributeToKey):
def __init__(self, color, shape):
self.color = color
self.shape = shape
The creation of Global.boxes at the very end of the file:
boxes = Boxes()
Example of Globals.py, good enough to pass test_2
After making sure that test_1 passes, we need to make sure that also test_2 passes. The missing mehtods are Boxes.save() and Boxes.load(), plus all the serialization machinery and the server code with the required callables.
Add to the Boxes class:
def save(self):
json_data = [box.to_json() for box in self.boxes]
anvil.server.call('save', data=json_data)
def load(self):
json_data = anvil.server.call('load')
self.boxes = [Box(box) for box in json_data]
Add to the Box class:
def to_json(self):
[...]
return json
@classmethod
def from_json(cls, json):
box = Box(json['name'])
for candy_json in json['candies']:
box.add_candy(Candy.from_json(candy_json))
return box
The Candy class will get a treatment similar to Box.
The server module will get the save and load callables.
Rendering the form
In my forms the components change the data by setting some properties or calling some methods of Globals.all_data (or one of its descendents), then call the update_interface() form method to re-render the form.
A form that shows the content of a box with a repeating panel for the candies and one for the cookies is structured like this:
class BoxDetails(BoxDetailsTemplate):
def __init__(self, **properties):
self.init_components(**properties)
self.candies.set_event_handler('x-delete-candy', self.delete_candy)
self.cookies.set_event_handler('x-delete-cookie', self.delete_cookie)
def form_show(self, **event_args):
self.box = boxes.get_box_by_[...]([...])
self.update_interface()
def delete_candy(self, candy, **event_args):
self.box.delete_candy(candy)
self.update_interface()
def delete_cookie(self, cookie, **event_args):
self.box.delete_cookie(cookie)
self.update_interface()
def update_interface(self, **event_args):
# initialize the repeating panels
self.candies.items = self.box.candies
self.cookies.items = self.box.cookies
In the above code self.box = boxes.get_box_by_[...]([...]) is inside form_show instead of __init__ because I use the routing module from Anvil Extras and I usually want to reload the data when navigating back to the form. (The code snippets in this page don’t show any code related with the routing module.) The [...] parts of that line could be by_name(name), by_id(id), etc.
Re-rendering the whole form containing many components and repeating panels is obviously less efficient than removing one component as shown in this tutorial, but so far I haven’t had any performance problems. Perhaps when the need comes I will do something like update_interface(candies=False, cookies=False, cupcakes=False) to avoid re-rendering repeating panels that have not changed.
The RowCandiesList form looks like this:
class RowCandiesList(RowCandiesListTemplate):
def __init__(self, **properties):
self.init_components(**properties)
def delete_click(self, **event_args):
self.parent.raise_event('x-delete-candy', candy=self.item)
When the repeating panel is inside a datagrid I use this helper function to make sure the same page is displayed when re-rendering:
def update_data_grid(self, repeating_panel, items):
grid = repeating_panel.parent
page = grid.get_page()
repeating_panel.items = items
grid.set_page(page)
Minimizing round trips
In the examples above the server functions save and load will save and load all the data.
In real apps this may not be the right thing to do. You may need the full list of boxes without the info of their candies or only one box with all the info about its candies, or both. Or you may have other classes like CupCakes and Cookies, and some forms may need info about one or the other.
Creating one server function per class you will almost certainly end up executing several roundtrips on forms that need different types of data.
The best way to minimize the number of roundtrips is to create one save and one load functions, both able to optionally manage all kinds of data. It is good to also optimize the amount of data sent to the client by creating dictionaries that only include the required columns. Something like:
@anvil.server.callable
def load(list_of_boxes=False, box_name='', candies=False,
cookies=False, cupcakes=False):
data = {}
if list_of_boxes:
# return the list of boxes for the current user, only the box name
data['boxes'] = [row['name'] for row in
app_tables.boxes.search(user=anvil.users.get_user())]
if box_name:
# return detailed information for the specified box
row = app_tables.boxes.get(user=anvil.users.get_user(), name=box_name)
data['box'] = {
'name': row['name'],
'candies': row['candies'],
'cookies': row['cookies'],
'cupcakes': row['cupcakes']
}
if candies:
# return the list of candies for the current user
Similarly, the save function will be able to save different kinds of data, but will only save what is provided:
@anvil.server.callable
def save(boxes=None, candies=None, cookies=None, cupcakes=None):
if boxes:
for row in app_table.boxes.search(user=anvil.users.get_user()):
row.delete()
for box in boxes:
app_table.boxes.add_row(user=anvil.users.get_user(), **box)
if candies:
[...]
Making it lazy
Most of my classes are lazy: if they have what they need, they do what they need to do. If they don’t have it, they get it once and keep it for later.
This may trigger undesired roundtrips, for example if the form started by calling boxes.load(candies=True, cookies=False) and later used boxes.list_of_cookies. Here there is one roundtrip when load is called, then a second roundtrip when list_of_cookies is called, because the information is not yet available.
I like to monitor all round trips by adding something like this to all server callables:
@anvil.server.callable
def load(boxes=None, box_name='', candies=None, cookies=None, cupcakes=None):
tokens = []
if boxes: tokens.append('boxes')
if box_name: tokens.append(f'box_name={box_name}')
if candies: tokens.append('candies')
if cookies: tokens.append('cookies')
if cupcakes: tokens.append('cupcakes')
print(f"load({', '.join(tokens)})")
When I open the form that calls both boxes.load(candies=True) and boxes.list_of_cookies, I will see this in the console:
load(boxes)
load(cookies)
So I will change the first line into boxes.load(candies=True, cookies=True). When the next line boxes.list_of_cookies is executed, the data is already loaded and the second round trip will not be executed. The new output will show only one round trip:
load(boxes, cookies)
Working with SQL
I have a dedicated plan, many of my apps use SQL and my uplink scripts don’t have access to SQL.
The workaround is for the server function to call another server function using anvil.server.call() rather than importing it and calling it directly.
I usually have only one SQL query per server function call, so having one function that does the connection and executes the query is not a problem.
In order to test functions that use SQL from the uplink I add a server module called SQL.py containing this:
@anvil.server.callable
def execute_query_and_fetchall(cmd):
cur = connect_to_postgres()
cur.execute(cmd)
return cur.fetchall()
And I have this in the server module:
if anvil.server.context.type == 'uplink':
# running tests on pc: use anvil.server.call and run on the server
def execute_query_and_fetchall(cmd):
return anvil.server.call('execute_query_and_fetchall', cmd=cmd)
else:
# running on Anvil server: import the function directly
from Sql import execute_query_and_fetchall
Easy improvements
When all the tests will pass, the classes will have all the properties required by the forms. The properties will be there available to all the forms to use them, whether they need them or not, without bothering creating a dictionary.
For example on the Truck class of one of my apps I have the following properties. The properties have been used first on the test, before they existed, then they have been added to the class Globals.Truck. Now, when I receive an email asking to add the weight of the truck to a form or to a PDF, the property is there. I add a column to a DataGrid, decide whether to use the numerical or the formatted property, and it works. There is no need to add it to a dictionary or to copy code from one form where the weight was used to the form where I need to add it.
@property
def estimated_weight(self):
return base_weight + sum(crate.weight for crate in self.crates)
@property
def actual_or_estimated_weight(self):
return self.actual_weight or self.estimated_weight
@property
def actual_or_estimated_weight_fmt(self):
return fmt(self.actual_or_estimated_weight)
Major edits
In case you already ready this post in the past, here are the major edits since the first version:
- Added the
What are the disadvantages?section - Added the
Rendering the formsection - Improved the class
AttributeToKey(perhaps this will play more nicely with databinding errors) - Added temporary disclaimer about stale content relative to the new accelerated tables