Use scrapy in Anvil

Anvil Q&A
1

I want to use scrapy in Anvil

I haven’t tried this yet in Anvil but my question is, is this possible?

2

Hi and welcome to the forum.

I haven’t used scrapy, but I’m guessing you can use it, because you can create an environment and install whatever you need: Anvil Docs | Installing Custom Packages

1 Like
3

Appreciate your response @stefano.menci :hand_with_index_finger_and_thumb_crossed:

4

Hi @stefano.menci & @stucork ,

I already am in Personal Plan and have Scapy package installed in my app.

However, I’m just learning Scrapy via a Udemy course and there’s a project format like the image below.

Screenshot 2022-12-09 at 2.50.29 PM
Screenshot 2022-12-09 at 2.50.29 PM1970×986 295 KB

Question is, do I need to have the same folder structure format in Anvil? If so, how?

5

Also, how will I run the spider with a shell command “scrapy crawl [spider.py]” when I click a button in Anvil?

6

The BETA IDE has a “server console” pane you can use to communicate with a Server instance. I haven’t tried it, but others here have.

Edit 1: Not sure it would help in this case. If I remember correctly, it’s a Python Read-Eval-Print Loop (“repl”), not a full command line.

Edit 2: Are you sure that’s the only way to run the spider? Or can you call it from within running Python code, e.g., as a call to a Scrapy function?

7

Hello @p.colbert ,

Thanks! So far my Udemy course only run scrapy using shell.

But I made a quick googling and found this: How to Run Scrapy From a Script. Forget about scrapy’s framework and… | by Aaron S | Towards Data Science.

Seems like the key to running Scrapy using Anvil.

Will keep you posted.

Regards,
Joey

1 Like