Using SpaCy from within Anvil (instead of via Uplink)

i.rajeshwari · December 2, 2020, 10:46am

I am looking to upgrade to the Individual plan and was hoping to port my NLP code into Anvil. I use SpaCy too, so will I experience this issue as well (referring to the issue posted [here])(Server error when loading Spacy lib)

Right now I use uplink to call SpaCy related functions. It’s not bad in terms of speed. Will I be better off using Uplink? My NLP code sits in my local machine, so its not ideal to depend on uplink.

brooke · December 4, 2020, 3:17pm

Hi @i.rajeshwari,

You could activate your free trial of the Full Python 3 runtime and try it out. It will also depend on what SpaCy model you are trying to load into memory. In that other post you reference, they were loading the largest model, which requires a fair bit more memory than the medium and small models.

i.rajeshwari · December 7, 2020, 11:04am

Thank you Brooke. Will give this a go as you suggest and report back. I think for now we should be OK with the small model which is 11MB.

i.rajeshwari · December 7, 2020, 2:08pm

Hi ya,

So I have now upgraded and moved to the full Python version in anvil.

In my server module, I have:

    import spacy
    ```
but when I try to load the model:

```python
    @anvil.server.callable
    def loadSpacyModel():
      nlp = spacy.load("en_core_web_sm")
      print('Loaded SpaCy')
    ```
I get the following error:

OSError: [E050] Can't find model 'en_core_web_sm'. It doesn't seem to be a shortcut link, 
a Python package or a valid path to a data directory. at /usr/local/lib/python3.7/site-packages/spacy/util.py, line 119 
called from /usr/local/lib/python3.7/site-packages/spacy/__init__.py, line 21 called from [ServerModule1, line 39](javascript:void(0)) called from [mockTest_template, line 16](javascript:void(0))

Not sure what I am missing. In my local machine, after installing spacy, this code was enough to load the model and work off it. Given it is already installed in Anvil's environment, I don't need to download and install again?

i.rajeshwari · December 7, 2020, 2:35pm

With some googling around, it seems the solution lies in downloading the model again.
For eg. in this post.
But I’m not sure where I’m supposed to do it…

i.rajeshwari · December 9, 2020, 10:03am

Is somebody able to respond to this please? I am unable to make any progress if I do not have SpaCy working in my app. I was hoping that by upgrading, I will not have to depend on Uplink for the NLP aspects, however I have not any luck so far.

Any suggestions much appreciated!
Thank you!

brooke · December 9, 2020, 7:06pm

Hi @i.rajeshwari,

After a little digging myself, I think you are correct that the model needs to be downloaded separately as is implied in the Spacy docs. I will poke the relevant person to download this for you during the (UK) workday tomorrow. Hopefully the Uplink will suffice until then!

i.rajeshwari · December 9, 2020, 7:50pm

Thank you so much Brooke, you are a star. Sorry if I’m being impatient, but my entire app depends on this Will look forward to hearing from you tom.

brooke · December 10, 2020, 5:50pm

Hi @i.rajeshwari,

Thanks for your patience! The en_core_web_sm model is now downloaded. I just tried loading it myself with your code, and it worked

i.rajeshwari · December 10, 2020, 6:00pm

Yes I’m able to load it as well. And it doesn’t take too long.
What joy!

THANK YOU SO VERY MUCH!!

i.rajeshwari · December 16, 2020, 10:45am

Hello,

I had a couple users report that they are experiencing an error when they try using a feature in the app that uses SpaCy. Please refer to the conversation above.

I tried it now and indeed I get this error:

FileNotFoundError: [Errno 2] No such file or directory: ‘/usr/local/lib/python3.7/site-packages/spacy/data/en_core_web_sm/init.py’ at , line 916 called from , line 859 called from , line 724 called from /usr/local/lib/python3.7/site-packages/spacy/compat.py, line 135 called from /usr/local/lib/python3.7/site-packages/spacy/util.py, line 126 called from /usr/local/lib/python3.7/site-packages/spacy/util.py, line 112 called from /usr/local/lib/python3.7/site-packages/spacy/init.py, line 21 called from [ServerModule1, line 265](javascript:void(0)) called from [mockTest_template, line 35](javascript:void(0))

The error is at this line:
nlp = spacy.load(“en_core_web_sm”)

The last time this code was successfully accessed was around noon UK time, dec 15th.

I have not changed anything related to this feature since this version of the app got published. Any ideas please?

i.rajeshwari · December 16, 2020, 1:27pm

Just to update that I tried to use it now and all seems fine.

Please are you able to explain why this might have happened and how can I attend to this in the future?

Thank you

cowen · July 14, 2022, 5:07am

From this thread, I thought a Spacy load would work. When I run this:

import spacy
import en_core_web_sm
NER = spacy.load(“en_core_web_sm”)

I get this traceback:

ModuleNotFoundError: No module named '_swigfaiss'

Need an extra library installed? Click for more info

at /usr/local/lib/python3.7/importlib/__init__.py:127
called from /usr/local/lib/python3.7/site-packages/faiss/swigfaiss.py:16
called from /usr/local/lib/python3.7/site-packages/faiss/swigfaiss.py:17
called from /usr/local/lib/python3.7/site-packages/faiss/__init__.py:19
called from /usr/local/lib/python3.7/site-packages/transformers/models/rag/retrieval_rag.py:42
called from /usr/local/lib/python3.7/site-packages/transformers/models/rag/modeling_rag.py:29
called from /usr/local/lib/python3.7/site-packages/transformers/models/auto/modeling_auto.py:199
called from /usr/local/lib/python3.7/importlib/__init__.py:127
called from /usr/local/lib/python3.7/site-packages/transformers/models/auto/__init__.py:198
called from /usr/local/lib/python3.7/site-packages/transformers/file_utils.py:1699
called from /usr/local/lib/python3.7/site-packages/transformers/file_utils.py:1700
called from /usr/local/lib/python3.7/site-packages/transformers/__init__.py:2487
called from /usr/local/lib/python3.7/site-packages/spacy_transformers/util.py:4
called from /usr/local/lib/python3.7/site-packages/spacy_transformers/data_classes.py:12
called from /usr/local/lib/python3.7/site-packages/spacy_transformers/layers/listener.py:4
called from /usr/local/lib/python3.7/site-packages/spacy_transformers/layers/__init__.py:1
called from /usr/local/lib/python3.7/site-packages/spacy_transformers/architectures.py:6
called from /usr/local/lib/python3.7/site-packages/spacy_transformers/__init__.py:1
called from <frozen importlib._bootstrap>:219
called from <frozen importlib._bootstrap_external>:728
called from <frozen importlib._bootstrap>:677
called from <frozen importlib._bootstrap>:967
called from <frozen importlib._bootstrap>:983
called from <frozen importlib._bootstrap>:1006
called from <frozen importlib._bootstrap>:219
called from <frozen importlib._bootstrap>:953
called from <frozen importlib._bootstrap>:983
called from <frozen importlib._bootstrap>:1006
called from /usr/local/lib/python3.7/importlib/__init__.py:127
called from /usr/local/lib/python3.7/site-packages/catalogue/_importlib_metadata/__init__.py:94
called from /usr/local/lib/python3.7/site-packages/catalogue/__init__.py:124
called from /usr/local/lib/python3.7/site-packages/catalogue/__init__.py:109
called from /usr/local/lib/python3.7/site-packages/spacy/language.py:162
called from /usr/local/lib/python3.7/site-packages/spacy/language.py:1755
called from /usr/local/lib/python3.7/site-packages/spacy/util.py:531
called from /usr/local/lib/python3.7/site-packages/spacy/util.py:488
called from /usr/local/lib/python3.7/site-packages/spacy/util.py:422
called from /usr/local/lib/python3.7/site-packages/spacy/__init__.py:52
called from servicesExcel, line 20

(line 20 is the spacy.load statement)

Any help?

thanks!

ianb · July 14, 2022, 7:06pm

You could wait for someone who knows better to answer but…
Have you heard about the new ability to install packages? You might want to try installing the newest version of SpaCy directly into your anvil server and see if that works?