Cannot load page with Incapsula & recaptcha security #147
Description
Hi,
Im trying to load https://orders.takeaway.com and webclient.getPage(...) stucks and I cant get anything back.
When I debug, I see that there is a async script in the body of the page that causes this behaviour, when I disable the running of this script (with a preScriptProcessor) then the code runs. But when I check the WebResponse, the content I see is the following:
<html style="height:100%"><head><META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"><meta name="format-detection" content="telephone=no"><meta name="viewport" content="initial-scale=1.0"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><script type="text/javascript" src="/_Incapsula_Resource?SWJIYLWA=719d34d31c8e3a6e6fffd425f7e032f3"></script></head><body style="margin:0px;height:100%"><iframe id="main-iframe" src="/_Incapsula_Resource?SWUDNSAI=30&xinfo=7-24816816-0%200NNN%20RT%281584800539592%20508%29%20q%280%20-1%20-1%200%29%20r%281%20-1%29%20B12%2811%2c378703%2c0%29%20U18&incident_id=769001310063609195-103792626402140359&edet=12&cinfo=0b000000&rpinfo=0" frameborder=0 width="100%" height="100%" marginheight="0px" marginwidth="0px">Request unsuccessful. Incapsula incident ID: 769001310063609195-103792626402140359</iframe></body></html>
This is not the page content I would expect, when I check the source code of the url above. It seems that some proxy (Incapsula) is preventing the page to load. And also Google's reCaptcha is getting invoked when I debug the code. You can see this when you add a preScriptProcessor to your webClient.
Can somebody please help me loading this page successfully ?
I'm using HtmlUnit 2.38 & JDK 11.
Thx in advance.