Hi, thanks for your message.
The requests are made from your web server, so in some cases I think it is possible to get different responses due the location of your server.
For example, here in Spain the link for the delta.com redirects (302) to
https://es.delta.com/
When you receive 403 responses I think it is because the firewall of the destination server detects to many requests and ban your server remote IP. Nothing to do here, it depends about how are configured these filters of the destination servers.
Kind regards,
Pau Iglesias
Thread Starter
jedsta
(@jedsta)
I can curl to these URL’s directly from the web server by hand without issue. I will start a new scan now in the plugin to just make sure it was not something else and update shortly. We only have the one link to these ones which were rejected so it should not be due to multiple requests.
Thread Starter
jedsta
(@jedsta)
I just did a quick curl variation test. The 403 comes back if I include the user agent string from the plugin with my curl request.
Thread Starter
jedsta
(@jedsta)
If I include a referrer with my curl request & the user agent string, then it succeeds.
Ok, what referrer have you included? To take note about what works for this case.
Thread Starter
jedsta
(@jedsta)
I apologize if my syntax is not proper up front, I do not profess to be an expert here. =)
curl -A ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36’ –referer https://www.milesgeek.com https://www.delta.com
Many thanks jedsta,
Maybe the plugin needed a referrer option next to the user-agent to avoid 403 responses from web servers with agressive scraper filters, I will think about it.
Kind regards,
Pau Iglesias
Thread Starter
jedsta
(@jedsta)
Thanks, It certainly would be appreciated. I just purchased the plugin after our correspondence. It would make the end results more usable and clean.
