I’ve automated a data processing task for staff in my organization. The task involves taking values from a local spreadsheet, passing those values to a website, and then pulling results back from the site and into the local spreadsheet. Selenium (and its Chrome driver) is used to open and interact with the browser (send text, click buttons, etc).
At the moment, staff run a script on their local computers to initiate the automation but obviously this is not ideal since I have to maintain my code on multiple computers.
I was wondering if it is possible to run the script on the Anvil servers since the program could be more easily maintained; just feared that using Selenium to control users’ browsers from within Anvil would be a challenge.
I have not tried anything yet. I just wanted to know your thoughts.
Does the spreadsheet have to live on your staff’s own computer? If so, you’re going to need something to upload your data, whether it’s the user putting a file into a FileLoader or an Uplink script sending data to the server.
If you want to make it a one-click operation for your users, I’d recommend using an Uplink script (use the Client Uplink key, to avoid giving everyone trusted access to your back-end). The Uplink script can read the file into a Media object with anvil.media.from_file(), then pass it via anvil.server.call() to a server function (or Background Task), which process the file however it wants. This means the Uplink script can be nice and simple, and you don’t have to update it when you edit the app it uses.
Yes, the spreadsheet has to live on the staff’s computer (at least that is how they prefer to keep it for now).
My main concern was whether or not Selenium would work when called from the Anvil server. I ask since Selenium requires a driver that is specific to the browser and OS that is being used.
Do you see any issues with this or is it just a matter of installing Selenium on Anvil server and pointing it to its driver (as is normally done on a local machine)?
example of opening a browser with Selenium:
from selenium import webdriver
driver = webdriver.Chrome('path_to_driver/chromedriver_for_linux')
driver.get('https://google.com')
Oh, I see - I thought the website was an Anvil app! If it’s dealing with an external website you’ll want a different approach.
I’m afraid we don’t currently support running Selenium on the Anvil servers (except for Dedicated Plan users) - it launches a whole browser, which uses a ridiculous amount of memory/CPU.
Two options come to mind:
Running Selenium on another computer, using the Uplink. You can still trigger it from another user’s machine (from a FileLoader or from a client Uplink script), but the Selenium run happens on a computer you control - so you can just update the code in one place.
(In this scenario, Anvil is working as an easy way to get files from your users’ computers to your server, rather than doing any of the processing in Anvil.)
Reworking your automation to use requests and BeautifulSoup, which are supported in Anvil server modules.
Thanks @meredydd. Both of those options sound like they would have advantages over my current approach.
(edit) although it does not seem as straightforward to fill in and submit forms (click buttons, choose dropdown items, etc) with requests and beautiful soup. I’ll keep trying though.
Here’s another alternative - have you tried submitting the form with the Network tab of your browser dev tools open? If it’s making a POST request to a particular HTTP endpoint, perhaps you could emulate that rather than filling in the form.
@shaun Very helpful. I can see that it is a POST request. Thanks very much for suggesting this alternative. I’m new to this sort of thing so I will pick away at it as I’d love to move this whole process onto Anvil.