Improvements to heuristic

I ran into a case where some missing parameters were present in the response.

Reponse:

Request malformed: templates parameter not found

It would be nice if the response could be parsed to extract words.

I did a quick hack that worked for my case:

def heuristic(response, paramList):
    soup = BeautifulSoup(response, 'html.parser')
    text = soup.find_all(text=True)[0].split(' ')
    done = []

    for t in text:
        inpName = t
        if inpName not in done:
            if inpName in paramList:
                paramList.remove(inpName)
            done.append(inpName)
            paramList.insert(0, inpName)
            log('%s Heuristic found a potential parameter: %s%s%s' % (good, green, inpName, end))
            log('%s Prioritizing it' % good)

Result:

 Analysing the content of the webpage
 Analysing behaviour for a non-existent parameter
 Reflections: 0
 Response Code: 400
 Content Length: 48
 Plain-text Length: 48
 Parsing webpage for potential parameters
 Heuristic found a potential parameter: Request
 Prioritizing it
 Heuristic found a potential parameter: malformed:
 Prioritizing it
 Heuristic found a potential parameter: templates
 Prioritizing it
 Heuristic found a potential parameter: parameter
 Prioritizing it
 Heuristic found a potential parameter: not
 Prioritizing it
 Heuristic found a potential parameter: found
 Prioritizing it
 Performing heuristic level checks
 Heuristic found 6 potential parameters.
 Scan Completed
 Valid parameter found: templates

heuristic just parses HTML forms and is not intended to fetch words from the webpage text. However, this feature can be provided with a new optional switch.

word extraction is now conditionally used by default in Arjun v2.2.0 🎉