@lazy_load_imports decorator (for slow loading imports)

[ianb]

A few times I have read about some imports taking a long time to load, causing every server call to be slower than they need to be.

Slow import of googleapiclient

I can’t say whether that’s normal for that particular import, but some imports are time consuming. For optimization, you can move the import away from the global scope and only into the server functions that actually use it. That would at least keep the other server functions from being delayed by an import they don’t use.

The answer to this I have seen a few times is to move the slow import into the function, breaking python convention, and requiring you to remember exactly what to import for each function.

It also requires code duplication if the import is needed on more than one function.

Converting Anvil App to Production Level

So its just for organization and legibility then, I get it. It’s odd that you have to break pythonic convention and put the slow imports inside the functions that use them, but its also not conventional to spin up a new python executable every time you call a single function in a script, so I guess it is pretty understandable.

So I was inspired by the two forum threads above to write a wrapper function that uses a global dictionary of import statements, with the module name you use as the key:

LAZY_IMPORT_DICT = {
                      'json':'import json',
                      'timedelta':'from datetime import timedelta',
                      'datetime':'from datetime import datetime',
                      'date':'from datetime import date',
                      'googlebuild':'from googleapiclient.discovery import build as googlebuild',
                      'pd':'import pandas as pd',
                      'np':'import numpy as np',
                      'plt':'import matplotlib.pyplot as plt',
                      'tf':'import tensorflow as tf',
                      'load_iris':'from sklearn.datasets import load_iris',
                      #'':'',

                        }


from functools import wraps
from dis import get_instructions

def lazy_load_imports(func):
  
  @wraps(func)
  def wrapper_func(*args, **kwargs):
    argval_set = {str(x.argval) for x in get_instructions(func)}
    for import_key in argval_set.intersection( set(LAZY_IMPORT_DICT.keys()) ):
      
      exec(LAZY_IMPORT_DICT[import_key] + "\nglobal " + import_key)

    func(*args, **kwargs)
  return wrapper_func
  

You put your import statement as the string, and the keys in the LAZY_IMPORT_DICT dictionary are the names of the modules as you use them.

Then you decorate any function you wish to use lazy imports on with @lazy_load_imports, and they will only load from the dictionary of imports if they are used in that function (and exist in the dictionary, obviously), reducing load time for large imports to only exactly what is used by the called function.

[ianb]

If anyone is curious what it is doing, I am using the dis module to disassemble the functions instructions and pulling the name of every argument used into a set, and comparing it to the keys in the lazy dictionary. Then it executes the string of code as written in said dictionary, then registers the imported method as global so it can be used by whatever function it @decorated.

For anyone unfamiliar, here is an example of the information provided by dis (called on the wrapper function as written above actually):

[p.colbert]

@ianb

breaking python convention

Actually, IIRC, this is a long-standing Python convention, for dealing with this kind of problem (and others, like circular imports).

[ianb]

Yes, I understand that this is the exception to the normal rule for this particular circumstance, I still think code duplication and having to keep track of imports with two conventions is bad.

Since python is both a beginner and an advanced user language, every time a first-time python tinkerer comes across this, they are more likely to possibly think it the norm, rather than the exception. Since it (only in my opinion) violates the explicit over implicit rule.*

*I know some people think putting the import in the function is more explicit, but it needs to be canonical somewhere, and not in two places depending on arbitrary author learning environments.

I just see plenty of first time python users trying out anvil, and thought maybe being able to put the import statements in a global variable up at the top with all your other imports to negate any extra-time loading penalty might help some out.
…And I would not expect users that wanted this to engage in their own meta-programming on day one.