publicwww engine integration by xm1k3 · Pull Request #157 · projectdiscovery/uncover

xm1k3 · 2023-02-15T17:28:19Z

cho "/wp-content/themes/twentysixteen/" | ./uncover  -e publicwww -f host -silent

https://www.mcmaster.ca/
https://fancywap.com/
https://winaero.com/
...

./uncover -e publicwww -q adserver.adtech.de -f host

https://www.anibis.ch/fr
https://fotka.com/
https://ng.se/stockholm
https://www.niederschlagsradar.de/
...

todo, parse and use csv instead of export=urls

ehsandeep

Known issues -

Dedicated search query is not working

./uncover -pw "/wp-content/themes/twentysixteen/"

  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /    
\__,_/_/ /_/\___/\____/|___/\___/_/ v1.0.2

		projectdiscovery.io

[FTL] Program exiting: no query provided

ENV token support, PUBLICWWW_API_KEY
Unpexected output

echo "/wp-content/themes/twentysixteen/" | ./uncover  -e publicwww -v

  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /    
\__,_/_/ /_/\___/\____/|___/\___/_/ v1.0.2

		projectdiscovery.io

[publicwww] :0
[publicwww] :0
[publicwww] :0
[publicwww] :0
[publicwww] :0
[publicwww] :0
[publicwww] :0
[publicwww] :0
[publicwww] :0

xm1k3 · 2023-02-20T22:26:23Z

echo "/wp-content/themes/twentysixteen/" | ./uncover  -e publicwww -v -f host

  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /    
\__,_/_/ /_/\___/\____/|___/\___/_/ v1.0.2

                projectdiscovery.io

[publicwww] www.mcmaster.ca
[publicwww] fancywap.com
[publicwww] winaero.com
[publicwww] getitfree.us
[publicwww] semioffice.com
[publicwww] www.saberia.com
[publicwww] cerkov.ru
[publicwww] bezux.pl
[publicwww] thespacereporter.com
[publicwww] www.cbrc.jp
[publicwww] www.infokids.gr
[publicwww] wiselwisel.com
[publicwww] techmixx.de

xm1k3 · 2023-02-20T22:27:37Z

echo "/wp-content/themes/twentysixteen/" | ./uncover  -e publicwww -v -f url

  __  ______  _________ _   _____  _____
 / / / / __ \/ ___/ __ \ | / / _ \/ ___/
/ /_/ / / / / /__/ /_/ / |/ /  __/ /    
\__,_/_/ /_/\___/\____/|___/\___/_/ v1.0.2

                projectdiscovery.io

[publicwww] https://www.mcmaster.ca/
[publicwww] https://fancywap.com/
[publicwww] https://winaero.com/
[publicwww] https://getitfree.us/

xm1k3 · 2023-02-20T22:33:05Z

[publicwww] :0
[publicwww] :0
[publicwww] :0

@ehsandeep this output is caused because -f by default is ip:port, publicwww csv provide to us url and rank, and with another export also snippet

url;rank;snippet
https://www.anibis.ch/fr;13403;fetch\" href=\"http://adserver.adtech.de\" crossorigin=\"anony

xm1k3 · 2023-02-21T14:55:08Z

echo jira | uncover -e shodan,publicww,fofa

...
[publicwww] cbreresidential.cz
[publicwww] www.dustingetz.com
[publicwww] luccasfraga.com.br
[shodan] 80.188.51.107:443
[shodan] 3.67.24.209:443
[shodan] 159.75.125.176:5800
[shodan] 119.29.10.14:10000
....

uncover/agent/publicwww/publicwww.go

runner/runner.go

tarunKoyalwar · 2023-02-21T17:14:53Z

@xm1k3 , @ehsandeep I think it will be better if we change ratelimit logic in uncover i.e in uncover we use delay to configure ratelimit while we have -rlm and -rl options in nuclei,subfinder,httpx . I think keeping options uniform will help rather than providing different formats for same functionality. Example

If source has ratelimit say 30 req/min

In nuclei we pass -rlm 30
In uncover we have to pass -delay 2

Apart from obvious calculation required another thing we miss is bucket refill rate.

The original library i.e golang.org/x/time/rate implements classic token bucket algorithm allowing a burst of tokens and a refill that happens at a specified ratio by one unit at a time whereas this implementation is a variant that allows a burst of tokens just like "the token bucket" algorithm, but the refill happens entirely at the defined ratio.

runner/runner.go

tarunKoyalwar

Suggesting some improvements .

I think we implement -delay option to avoid hitting ratelimit . since we already know ratelimits for most of the sources (shodan,censys etc) I think we can use a default ratelimit for these sources and leave it for user to configure sources whose ratelimit is not know (ex: publicwww).
we are doing the same in subfinder

tarunKoyalwar · 2023-02-24T11:44:33Z

runner/runner.go

 		)
 		switch engine {
 		case "shodan":
-			agent, err = shodan.NewWithOptions(&uncover.AgentOptions{RateLimiter: shodanRateLimiter})
+			agent, err = shodan.New()
 		case "censys":
-			agent, err = censys.NewWithOptions(&uncover.AgentOptions{RateLimiter: censysRateLimiter})
+			agent, err = censys.New()
 		case "fofa":


@xm1k3 , it depends on your preference , but I think we can remove .New() functions since they are not used anymore and only decrease overall code coverage. I think we can directly append to array

agents = append(agents, &publicwww.Agent{})

tarunKoyalwar · 2023-02-24T11:46:14Z

uncover/agent/censys/censys.go

+	err = session.RateLimits.Take(agent.Name())
+	if err != nil {
+		return nil, err
+	}
 	return session.Do(request)
 }


same here , I think we can modify session.Do to accept a optional sourcename and move ratelimit logic inside Do function

func (s *Session) Do(req *retryablehttp.Request,source ...string)

tarunKoyalwar · 2023-02-24T11:50:21Z

uncover/session.go

+	var err error
+	rateLimitOpts := &ratelimit.Options{
+		MaxCount:    uint(retryMax),
+		Duration:    time.Duration(delay),
+		IsUnlimited: delay == 0,
+	}
+
+	rateLimitOpts.Key = engines[0]


@xm1k3 I think Maxcount should be 1 here and retryMax is configured should be configure inretryablehttp.Client

Mzack9999 · 2023-02-28T08:38:27Z

@tarunKoyalwar I think tracking these changes within a new issue would be better as they are not specific to this implementation, which only adds support for the new source publicwww. Could you create a follow-up ticket?

- session and rate limit logic moved to #165

xm1k3 · 2023-02-28T18:48:58Z

@Mzack9999 @tarunKoyalwar as mentioned in the above message above, the logic that also modifies the other providers has been moved to this issue: #164

tarunKoyalwar

lgtm!

Just a note this source is not very stable due to its ratelimit which will be implemented in #164

uncover/agent/publicwww/publicwww.go

started job

362febc

xm1k3 self-assigned this Feb 15, 2023

xm1k3 added the Type: Enhancement Most issues will probably ask for additions or changes. label Feb 15, 2023

xm1k3 linked an issue Feb 15, 2023 that may be closed by this pull request

publicwww - new api provider #122

Closed

xm1k3 added 2 commits February 15, 2023 23:08

wip: publicwww integration

f7e05ac

added build url logic

ec4c725

xm1k3 marked this pull request as ready for review February 16, 2023 15:23

xm1k3 and others added 3 commits February 17, 2023 12:41

results management

9317c15

small changes

f7e62a9

wip: added url flag, used retryablehttp for request + return raw results

6f3e011

todo, parse and use csv instead of export=urls

ehsandeep requested changes Feb 19, 2023

View reviewed changes

ehsandeep marked this pull request as draft February 19, 2023 10:19

xm1k3 added 4 commits February 20, 2023 09:11

dedicated search query and env support

02390b0

fixes and csv export

dad8e07

tried to use http get as wip to debug with retryablehttp

e9e8559

commented un used code

1475cd4

xm1k3 requested a review from ehsandeep February 20, 2023 22:33

default field for publicwww integration

eced794

ehsandeep marked this pull request as ready for review February 21, 2023 15:15

ehsandeep approved these changes Feb 21, 2023

View reviewed changes

ehsandeep requested a review from tarunKoyalwar February 21, 2023 15:16

tarunKoyalwar requested changes Feb 21, 2023

View reviewed changes

uncover/agent/publicwww/publicwww.go Show resolved Hide resolved

uncover/agent/publicwww/publicwww.go Outdated Show resolved Hide resolved

uncover/agent/publicwww/publicwww.go Outdated Show resolved Hide resolved

runner/runner.go Show resolved Hide resolved

tarunKoyalwar reviewed Feb 21, 2023

View reviewed changes

runner/runner.go Show resolved Hide resolved

xm1k3 added 2 commits February 21, 2023 23:32

started review fixes

e863352

rate limiting logic

5332d77

xm1k3 added 4 commits February 23, 2023 23:16

code refactoring on session management

4ed60d1

error management

164b41d

Merge branch 'dev' into issue-122-publicwww-engine-integration

c4cbce0

go mod - sum

cc0e00b

xm1k3 requested a review from tarunKoyalwar February 23, 2023 22:28

tarunKoyalwar reviewed Feb 24, 2023

View reviewed changes

xm1k3 mentioned this pull request Feb 28, 2023

Change rate limit logic and align the uncover flags the same way they are done on subfinder #164

Closed

restored logic

d593f66

- session and rate limit logic moved to #165

xm1k3 requested a review from tarunKoyalwar February 28, 2023 18:49

tarunKoyalwar approved these changes Mar 1, 2023

View reviewed changes

tarunKoyalwar requested a review from Mzack9999 March 1, 2023 07:17

continue iteration on err

71fd964

Mzack9999 approved these changes Mar 1, 2023

View reviewed changes

uncover/agent/publicwww/publicwww.go Show resolved Hide resolved

Mzack9999 merged commit 0968d71 into dev Mar 1, 2023

Mzack9999 deleted the issue-122-publicwww-engine-integration branch March 1, 2023 07:46

Comments

Conversation

xm1k3 commented Feb 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ehsandeep left a comment • edited by xm1k3 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xm1k3 commented Feb 20, 2023

Uh oh!

xm1k3 commented Feb 20, 2023

Uh oh!

xm1k3 commented Feb 20, 2023

Uh oh!

xm1k3 commented Feb 21, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tarunKoyalwar commented Feb 21, 2023

Uh oh!

Uh oh!

tarunKoyalwar left a comment

Choose a reason for hiding this comment

Uh oh!

tarunKoyalwar Feb 24, 2023

Choose a reason for hiding this comment

Uh oh!

tarunKoyalwar Feb 24, 2023

Choose a reason for hiding this comment

Uh oh!

tarunKoyalwar Feb 24, 2023

Choose a reason for hiding this comment

Uh oh!

Mzack9999 commented Feb 28, 2023

Uh oh!

xm1k3 commented Feb 28, 2023

Uh oh!

tarunKoyalwar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xm1k3 commented Feb 15, 2023 •

edited

Loading

ehsandeep left a comment •

edited by xm1k3

Loading