Anvil-uplink breaking google auth

Anvil Q&A
1

What I’m trying to do:
I am trying to use anvil-uplink within a Managed Notebook launched from the Vertex AI Workbench within a Google Cloud Project. This setup puts the user in a Jupyter Lab environment. I create a custom computing environment (venv) within which I install various python modules. In particular, I install the earthengine-api, which is maintained by Google folks. I go through standard authentication channels and everything is fine. I realize that I forgot to install the anvil-uplink module. I install it into the environment. I restart the kernel and go through the authentication process again. However, the anvil-uplink module has broken something. I receive this trace:

ERROR: Error initializing plugin EntryPoint(name='Google Auth', value='keyrings.gauth', group='keyring.backends').
Traceback (most recent call last):
  File "/home/jupyter/.local/lib/python3.7/site-packages/keyring/backend.py", line 204, in _load_plugins
    init_func = ep.load()
  File "/opt/conda/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 203, in load
    module = import_module(match.group('module'))
  File "/opt/conda/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/conda/lib/python3.7/site-packages/keyrings/gauth.py", line 18, in <module>
    from google.auth.transport import requests
ModuleNotFoundError: No module named 'google.auth.transport'; 'google.auth' is not a package

The trace appears to have broken something at the root level (see below).

What I’ve tried and what’s not working:
I tried to rebuild the environment, placing the anvil-uplink installation before the Google module installations, thinking the latter would replace any broken authentication pathways used in anvil-uplink. This didn’t work. Googling suggested that maybe I need to uninstall + reinstall, or otherwise update, the keyring module. I tried adding this to my workflow, prior to the installation of other modules. It didn’t work, but the above trace was thrown at a different location during environment set-up – after/during handling of argparse from within anvil-uplink.

Requirement already satisfied: anvil-uplink in /home/jupyter/.local/lib/python3.7/site-packages (0.4.0)
Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from anvil-uplink) (1.16.0)
Requirement already satisfied: future in /home/jupyter/.local/lib/python3.7/site-packages (from anvil-uplink) (0.18.2)
Requirement already satisfied: ws4py in /home/jupyter/.local/lib/python3.7/site-packages (from anvil-uplink) (0.5.1)
Requirement already satisfied: argparse in /home/jupyter/.local/lib/python3.7/site-packages (from anvil-uplink) (1.4.0)
ERROR: Error initializing plugin EntryPoint(name='Google Auth', value='keyrings.gauth', group='keyring.backends').
Traceback (most recent call last):
  File "/home/jupyter/.local/lib/python3.7/site-packages/keyring/backend.py", line 204, in _load_plugins
    init_func = ep.load()
  File "/opt/conda/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 203, in load
    module = import_module(match.group('module'))
  File "/opt/conda/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/conda/lib/python3.7/site-packages/keyrings/gauth.py", line 18, in <module>
    from google.auth.transport import requests
ModuleNotFoundError: No module named 'google.auth.transport'; 'google.auth' is not a package

I tried reinstalling google-auth, but it didn’t work.

I would have thought that creating a new environment would have resolved the issue, but it appears that the keyring module is accessed from the global root. Even if I try to update it there, I receive the error. In addition, if I subsequently try to create another new environment, but I remove the installation of anvil-uplink, the same error appears during installation of pycparser. And so, to “fix” this, I end up having to rebuild the entire virtual machine.

I don’t really understand what’s happening (too novice). Could anyone offer me some guidance on where to look for the issue, or otherwise how to structure my environment set-up to avoid whatever conflict is happening? I recognize that my particular setup, within Vertex AI, makes this challenging to debug, but any suggestions would be appreciated.

Thanks!

2

Take this with a grain of salt as I may be more novice than you in the relevant ways, but I doubt the issue has anything to do with anvil-uplink. Rather, I am guessing that something you did the first time when everything worked is not reflected in the notebook you ended up with afterwards (before the addition of the anvil-uplink install). There is “hidden state” in notebooks that is notorious for undermining reproducibility. This is the tip-off (if I’m understanding correctly):

…if I subsequently try to create another new environment, but I remove the installation of anvil-uplink, the same error appears…

I would try to retrace your original steps from scratch. Get it working (without anvil-uplink) in a notebook that works when you “restart kernel and run all cells.” Then add anvil-uplink to that and I suspect it will work fine.

1 Like
3

@hugetim Thanks very much for your thoughts. I agree that the hidden states of notebooks make for some unusual challenges, and my particular set-up makes this even more so. I am told that Google Managed Notebooks have base images that are not modifiable, which makes build errors that much more confusing. I will explore your suggestion of rebuilding a working environment and adding anvil as a post hoc step. In theory I can at least narrow down exactly what action leads to incompatibilities. If I continue to find that anvil-uplink is disrupting an otherwise reproducible and (theoretically) stable notebook set-up, I may post again to see if there are other ideas. Otherwise, I appreciate the input - thanks again.

1 Like
4

Ok, so I finally got around to testing @hugetim’s suggestion. I can create a functional environment in my notebook, and then, as a post hoc step, install anvil-uplink into the dedicated venv virtual environment. The installation completes successfully without any errors. I then restart the kernel and try to load some of the Google-based tools that worked before,

google.auth import compute_engine
import ee # Google's earthengine-api

but am unsuccessful. I receive the error:

cannot import name 'compute_engine' from 'google.auth' (/home/jupyter/test/lib/python3.7/site-packages/google/auth.py)

Naively, it would appear that there is some authentication conflict that is gummed up when anvil-uplink is installed. Does anyone have additional suggestions for how to ascertain what exactly is happening here?

1 Like
5

I apologize if I led you down an unhelpful path. It does seem plausible to me now that anvil-uplink’s version of google auth could be conflicting somehow with your other version of it. I hope someone else will be able to point you towards a way of resolving the issue.

6

Actually, @hugetim, you were indeed correct! I was just testing further and after a call to

%pip install --upgrade google-auth --user

all was well and I could access both the Google tools and my Anvil database. Sorry for the false alarm, but thanks for the prompt reply. Perhaps there are some authentication version incompatibilities that are worth looking into, but ultimately this has been sorted. Thanks again.

1 Like