Using GPT 4, I asked it to tell me what a simple site was about but the response is "The website is displaying the error message: “This app has experienced an error. Unfortunately, this means I am unable to access or provide information about the content of this website at the moment.”
I tried even a simple clone of demo sites that anvil publishes. None can be browsed by AI.
AI response is: ‘The website “https://flawless-unwritten-slow-loris.anvil.app/” is also showing an error message stating, “This app has experienced an error.” It seems that there is a technical issue with this site as well, preventing access to its content through my browsing tool. If there’s any other way I can assist you, please let me know.’
Is this because of the way automated crawlers scrape the web? as in, curl, etc?