Conversation
Add search Engine _ Driftnet
WalkthroughThis change introduces support for the Driftnet search engine throughout the project. It adds the Driftnet agent, updates key management, extends CLI options, enhances documentation, and includes Driftnet in the provider and agent registries. The version is incremented to v1.1.0. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant CLI
participant Options
participant Provider
participant DriftnetAgent
participant DriftnetAPI
User->>CLI: Run uncover with -e driftnet -q <query>
CLI->>Options: Parse CLI flags (includes Driftnet)
Options->>Provider: Load Driftnet API key from env
CLI->>DriftnetAgent: Initialize agent with API key
DriftnetAgent->>DriftnetAPI: Send query (IP/CIDR or search term)
DriftnetAPI-->>DriftnetAgent: Return results
DriftnetAgent-->>CLI: Stream results
CLI-->>User: Display results
Assessment against linked issues
Assessment against linked issues: Out-of-scope changes(No out-of-scope changes found.) Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (7)
uncover.go (2)
13-14: Import grouping nitExternal imports are alphabetically sorted elsewhere; insert
driftnetaccordingly to keep diff noise low in future.
195-196: Order of AllAgents listConsider keeping the slice alphabetised to reduce merge conflicts.
- "binaryedge", "onyphe", "driftnet", + "binaryedge", "driftnet", "onyphe",runner/options.go (2)
73-76: Flag help text still says “default shodan”Because
driftnetis now supported, the parenthetical “(default shodan)” may confuse users when they pass-e driftnet.
Recommend clarifying:- (default shodan) + (default: shodan)
153-171: Validation repetition getting unwieldyEach time an engine is added, three nearly identical
genericutil.EqualsAllblocks need updating, which is error-prone.Consider refactoring:
engSlices := [][]string{ options.Engine, options.Shodan, options.Censys, /* … */, options.Driftnet, } if genericutil.EqualsAll(0, slices.Map(engSlices, len)...) { … }Reduces future diff surface.
sources/agent/driftnet/response.go (1)
3-54: Confirm API returns numeric JSON keys
map[int]intassumes port numbers arrive as JSON object keys that can be parsed as integers. If the Driftnet API ever returns them as strings that are not purely numeric (e.g."443/tcp"), unmarshal will fail.sources/agent/driftnet/driftnet.go (2)
122-148: Shadowingresulthampers readabilityThe loop variable
result(typeDriftnetAPIResult) is shadowed byresult := sources.Result{...}a few lines later. Rename one of them to avoid confusion.
235-246:Rawpayload repeats full response for every port
result.Rawstores the entiredriftnetResponse, so identical large blobs are emitted per port. Consider marshalling only the per-IP port slice (or leavingRawempty) to reduce memory/bandwidth.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (9)
README.md(8 hunks)runner/banners.go(1 hunks)runner/options.go(7 hunks)sources/agent/driftnet/driftnet.go(1 hunks)sources/agent/driftnet/response.go(1 hunks)sources/keys.go(2 hunks)sources/provider.go(4 hunks)sources/session.go(1 hunks)uncover.go(3 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
uncover.go (2)
sources/agent/driftnet/driftnet.go (1)
Agent(43-43)sources/agent.go (1)
Agent(8-11)
🔇 Additional comments (9)
runner/banners.go (1)
16-16: Version constant bumped successfullyThe update to
v1.1.0looks good and matches the PR scope.sources/session.go (1)
34-35: Hard-coded rate-limit may breach Driftnet’s ToSA limit of
5 req / secis assumed without referencing Driftnet’s official quota.
If the real quota is lower, users risk 429s or bans.- "driftnet": {Key: "driftnet", MaxCount: 5, Duration: time.Second}, + // TODO: verify and document the official Driftnet rate-limit + "driftnet": {Key: "driftnet", MaxCount: 5, Duration: time.Second},Please double-check Driftnet’s API policy and adjust if necessary.
sources/keys.go (2)
22-23: Key added – remember env / provider wiring
DriftnetTokenis now part of the struct, but ensure:
sources/provider.goloadsDRIFTNET_API_KEYinto this field.- Documentation lists the env-var.
43-45: Nice catch adding missing Onyphe & Driftnet checks
Empty()now covers both keys that were previously forgotten.uncover.go (1)
89-91: Agent wiring complete
driftnet.Agent{}is correctly registered. Confirm that:
Agent.Name()returns"driftnet"(case-match with switch & ratelimit map).- The agent handles
Limitparameter like others.runner/options.go (2)
63-64: Field added – good
Driftnetflag integration mirrors existing engines.
312-313: appendAllQueries updated correctlyDriftnet queries are now aggregated – nice.
sources/provider.go (1)
38-39: Driftnet key integration looks solidThe new slice, env-variable loader, random-selection logic, and
HasKeyscheck are consistent with existing providers. No functional concerns.Also applies to: 124-126, 157-158, 195-196
README.md (1)
52-53: Documentation updates look goodThe Driftnet flags, examples and key sections read clearly – nice addition.
Also applies to: 80-81, 97-98, 169-172, 194-195, 283-317, 382-416
| defer resp.Body.Close() | ||
|
|
||
| // Parse the response | ||
| driftnetResponse := &DriftnetAPIPaginatedResponse{} | ||
| if err := json.NewDecoder(resp.Body).Decode(driftnetResponse); err != nil { | ||
| results <- sources.Result{Source: agent.Name(), Error: err} |
There was a problem hiding this comment.
Close response bodies immediately inside loops
defer resp.Body.Close() inside the page and CIDR loops postpones closing up to 100-200+ responses until the whole function returns, needlessly holding sockets and fds. Close the body right after json.NewDecoder instead.
- defer resp.Body.Close()
- if err := json.NewDecoder(resp.Body).Decode(&dst); err != nil {
+ if err := json.NewDecoder(resp.Body).Decode(&dst); err != nil {
...
}
+ resp.Body.Close()Also applies to: 227-233
🤖 Prompt for AI Agents
In sources/agent/driftnet/driftnet.go around lines 113 to 118, the response body
is deferred to close inside loops, causing many open sockets and file
descriptors to accumulate. Instead of deferring resp.Body.Close(), close the
response body immediately after decoding the JSON response to release resources
promptly. Apply the same fix to lines 227 to 233 where a similar pattern occurs.
* update zoomeye api * close resp.Body * remove host check --------- Co-authored-by: Sandeep Singh <[email protected]>
There was a problem hiding this comment.
Actionable comments posted: 1
🔭 Outside diff range comments (1)
sources/agent/zoomeye/zoomeye.go (1)
30-41:query.Limit == 0prematurely stops pagination
numberOfResults >= query.Limitis true after the first page whenLimitis the zero-value (common when user does not supply-l).
The loop therefore never fetches page 2+.-if numberOfResults >= query.Limit || … +if (query.Limit > 0 && numberOfResults >= query.Limit) || + …That preserves current behaviour when a limit is specified while allowing unlimited fetch by default.
Also applies to: 58-61
🧹 Nitpick comments (4)
README.md (2)
45-46: Inconsistent link-style breaks Markdown-lint & rendered linkZoomEye entry is the only bullet that does not use Markdown link syntax, triggering
MD034and breaking the clickable link:- **ZoomEye (https://www.zoomeye.ai)** + - **[ZoomEye](https://www.zoomeye.ai)**(Driftnet bullet already follows the correct pattern.)
Also applies to: 52-53
382-416: Very long console block – consider collapsing for readabilityThe driftnet example is 👍 but the 30-line ellipsis clutters the README.
Using GitHub’s<details>/</details>or truncating to 4-5 example lines keeps the doc scannable while still demonstrating output.sources/agent/zoomeye/zoomeye.go (2)
14-16: Exported varURLviolates Go naming conventionsA package-level exported identifier named only
URLexposes internal implementation detail and shadows method parameters later on.-var ( - URL = "https://api.zoomeye.ai/v2/search" -) +const apiURL = "https://api.zoomeye.ai/v2/search"Keep it unexported (
apiURL) unless other packages really need it.
105-115: Avoid reallocatingsources.Result– reuse variableInside the tight loop you create a zero value, then fill fields; reusing a local variable avoids one allocation per result.
var out sources.Result out.Source = agent.Name() for _, r := range zoomeyeResponse.Results { out.IP, out.Port, out.Host = r.IP, r.Port, r.Hostname out.Timestamp = time.Now().Unix() // optional but consistent with other agents raw, _ := json.Marshal(r) out.Raw = raw results <- out }Minor but measurable under high volume.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
README.md(9 hunks)sources/agent/zoomeye/response.go(1 hunks)sources/agent/zoomeye/zoomeye.go(3 hunks)sources/keys.go(2 hunks)sources/provider.go(5 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- sources/keys.go
- sources/provider.go
🧰 Additional context used
🧬 Code Graph Analysis (1)
sources/agent/zoomeye/zoomeye.go (6)
sources/agent.go (2)
Query(3-6)Agent(8-11)sources/util.go (1)
NewHTTPRequest(10-17)sources/keys.go (1)
Keys(3-22)sources/session.go (1)
Session(38-43)sources/result.go (1)
Result(8-17)sources/agent/zoomeye/response.go (1)
ZoomEyeResponse(3-6)
🪛 markdownlint-cli2 (0.17.2)
README.md
45-45: Bare URL used
null
(MD034, no-bare-urls)
⏰ Context from checks skipped due to timeout of 90000ms (4)
- GitHub Check: Test Builds (ubuntu-latest, 1.22.x)
- GitHub Check: Test Builds (macOS-13, 1.22.x)
- GitHub Check: Test Builds (windows-latest, 1.22.x)
- GitHub Check: Analyze (go)
🔇 Additional comments (1)
sources/agent/zoomeye/response.go (1)
4-6: Confirm field types against Zoomeye v2 API –totalmay overflowintandportmay come back as stringNice move to a strongly-typed model.
Two details to double-check against the API spec / live traffic:
totalis declaredint. The v2 API returns a 64-bit value ("total": 1234567890). On 32-bit platforms this could overflow.-Total int
json:"total"
+Total int64json:"total"2. Historical JSON samples occasionally wrap `port` in quotes (`"port":"443"`). If that still happens, decoding into `int` will fail the whole response. A custom type with `UnmarshalJSON` or `json.Number` would make the client tolerant to both string and number. Please verify with a live request before shipping. Also applies to: 8-11 </details> </blockquote></details> </details> <!-- This is an auto-generated comment by CodeRabbit for review status -->
| func (agent *Agent) queryURL(session *sources.Session, URL string, zoomeyeRequest *ZoomEyeRequest) (*http.Response, error) { | ||
| zoomeyeURL := fmt.Sprintf(URL, url.QueryEscape(zoomeyeRequest.Query), zoomeyeRequest.Page) | ||
| // Encode query to base64 | ||
| queryBase64 := base64.StdEncoding.EncodeToString([]byte(zoomeyeRequest.Query)) | ||
|
|
||
| requestBody := map[string]interface{}{ | ||
| "qbase64": queryBase64, | ||
| "page": zoomeyeRequest.Page, | ||
| "pagesize": zoomeyeRequest.PageSize, | ||
| } | ||
|
|
||
| jsonBody, err := json.Marshal(requestBody) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
||
| request, err := sources.NewHTTPRequest(http.MethodGet, zoomeyeURL, nil) | ||
| request, err := sources.NewHTTPRequest(http.MethodPost, URL, bytes.NewReader(jsonBody)) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| request.Header.Set("API-KEY", session.Keys.ZoomEyeToken) | ||
| request.Header.Set("Content-Type", "application/json") | ||
| return session.Do(request, agent.Name()) |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Handle non-200 responses & reuse encoded body
- The client proceeds to JSON-decode even when Zoomeye returns
401/429, producing misleading errors. - There is no error message surface back to the caller.
resp, err := agent.queryURL(session, apiURL, zoomeyeRequest)
…
+if resp.StatusCode != http.StatusOK {
+ // Drain and include error body for context (max 1 KiB)
+ b, _ := io.ReadAll(io.LimitReader(resp.Body, 1024))
+ results <- sources.Result{Source: agent.Name(),
+ Error: fmt.Errorf("zoomeye: %s (%d): %s", resp.Status, resp.StatusCode, b)}
+ return nil
+}queryURLre-encodes the query on every call. Pre-encode once outside the loop for minor CPU savings.
Committable suggestion skipped: line range outside the PR's diff.
🤖 Prompt for AI Agents
In sources/agent/zoomeye/zoomeye.go around lines 67 to 88, the function queryURL
does not handle non-200 HTTP responses properly, leading to misleading errors
when Zoomeye returns 401 or 429. Modify the function to check the HTTP response
status code before attempting to decode the JSON body and return a clear error
message if the status is not 200. Additionally, refactor the code to accept the
base64-encoded query as a parameter instead of encoding it inside queryURL, so
the encoding can be done once outside the loop to improve efficiency.
* add proxy option * fix test
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
README.md (1)
45-45: Fix bare URL formatting issue.The static analysis tool flagged a bare URL that should be properly formatted as a markdown link.
Apply this diff to fix the markdown formatting:
- - **ZoomEye (https://www.zoomeye.ai)** + - **[ZoomEye](https://www.zoomeye.ai)**
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
README.md(10 hunks)runner/options.go(9 hunks)runner/runner.go(1 hunks)sources/session.go(2 hunks)sources/session_test.go(1 hunks)uncover.go(5 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- uncover.go
- runner/options.go
🧰 Additional context used
🪛 markdownlint-cli2 (0.17.2)
README.md
45-45: Bare URL used
null
(MD034, no-bare-urls)
🪛 ast-grep (0.38.1)
sources/session.go
[warning] 48-50: MinVersionis missing from this TLS configuration. By default, TLS 1.2 is currently used as the minimum when acting as a client, and TLS 1.0 when acting as a server. General purpose web applications should default to TLS 1.3 with all other protocols disabled. Only where it is known that a web server must support legacy clients with unsupported an insecure browsers (such as Internet Explorer 10), it may be necessary to enable TLS 1.0 to provide support. AddMinVersion: tls.VersionTLS13' to the TLS configuration to bump the minimum version to TLS 1.3.
Context: tls.Config{
InsecureSkipVerify: true,
}
Note: [CWE-327]: Use of a Broken or Risky Cryptographic Algorithm [OWASP A03:2017]: Sensitive Data Exposure [OWASP A02:2021]: Cryptographic Failures [REFERENCES]
https://owasp.org/Top10/A02_2021-Cryptographic_Failures
(missing-ssl-minversion-go)
⏰ Context from checks skipped due to timeout of 90000ms (4)
- GitHub Check: Test Builds (ubuntu-latest, 1.22.x)
- GitHub Check: Test Builds (macOS-13, 1.22.x)
- GitHub Check: Test Builds (windows-latest, 1.22.x)
- GitHub Check: Analyze (go)
🔇 Additional comments (11)
sources/session.go (3)
34-34: LGTM: Driftnet rate limit configuration looks appropriate.The rate limit of 5 requests per second for Driftnet is reasonable and follows the same pattern as other search engines in the DefaultRateLimits map.
53-60: Proxy implementation is secure and well-structured.The proxy configuration logic correctly:
- Prioritizes the provided proxy parameter over environment settings
- Handles URL parsing errors gracefully by falling back to environment proxy
- Uses the standard
http.ProxyURLandhttp.ProxyFromEnvironmentfunctionsThe implementation is secure and follows Go HTTP proxy best practices.
45-45: Function signature change requires verification across codebase.The addition of the
proxyparameter toNewSessionis correctly implemented, but ensure all callers have been updated to match the new signature.#!/bin/bash # Description: Verify all NewSession function calls have been updated to include the proxy parameter # Expected: All calls should have the new proxy parameter ast-grep --pattern 'NewSession($$$)'runner/runner.go (1)
35-35: LGTM: Proxy option forwarding is correctly implemented.The proxy configuration is properly passed from the runner options to the uncover service, maintaining the expected data flow in the application architecture.
sources/session_test.go (1)
22-22: LGTM: Test updated correctly for new function signature.The test properly includes the new proxy parameter with an empty string, which is appropriate for testing without proxy configuration.
README.md (6)
52-52: LGTM: Driftnet integration documented consistently.The documentation correctly adds Driftnet to:
- The list of supported search engines
- Command-line engine options
- Engine-specific flag options
The documentation follows the established pattern for other search engines.
Also applies to: 80-80, 97-97
106-106: LGTM: Proxy flag documentation is clear and concise.The proxy flag documentation is appropriately brief and follows the established format for other configuration flags.
170-172: LGTM: Driftnet API key configuration documented properly.The API key configuration examples for Driftnet follow the same pattern as other search engines, maintaining consistency in the documentation.
Also applies to: 195-195
285-285: LGTM: Usage examples updated consistently.The command-line examples correctly include Driftnet alongside other search engines, maintaining consistency with the existing documentation pattern.
Also applies to: 317-317
383-418: LGTM: New Driftnet section provides valuable context.The new section explaining Driftnet's capability for fast IP/CIDR port lookup is well-structured and provides:
- Clear explanation of the use case
- Practical example with realistic output
- Proper integration with the existing documentation flow
This adds valuable information for users who need fast port scanning capabilities.
198-198: Verify Driftnet registration URL is accessible.The documentation includes a registration link for Driftnet. Please verify this URL is accessible and correct.
Is the Driftnet registration URL https://driftnet.io/auth?state=signup accessible and correct for new user registration?
| MaxIdleConnsPerHost: 100, | ||
| TLSClientConfig: &tls.Config{ | ||
| InsecureSkipVerify: true, |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Address TLS security configuration.
The TLS configuration is missing a minimum version specification. This could allow connections using insecure TLS versions.
Apply this diff to improve TLS security:
TLSClientConfig: &tls.Config{
+ MinVersion: tls.VersionTLS12,
InsecureSkipVerify: true,
},Note: While TLS 1.3 would be ideal, TLS 1.2 is more practical for compatibility with various search engine APIs that may not support the latest TLS version.
🧰 Tools
🪛 ast-grep (0.38.1)
[warning] 48-50: MinVersionis missing from this TLS configuration. By default, TLS 1.2 is currently used as the minimum when acting as a client, and TLS 1.0 when acting as a server. General purpose web applications should default to TLS 1.3 with all other protocols disabled. Only where it is known that a web server must support legacy clients with unsupported an insecure browsers (such as Internet Explorer 10), it may be necessary to enable TLS 1.0 to provide support. AddMinVersion: tls.VersionTLS13' to the TLS configuration to bump the minimum version to TLS 1.3.
Context: tls.Config{
InsecureSkipVerify: true,
}
Note: [CWE-327]: Use of a Broken or Risky Cryptographic Algorithm [OWASP A03:2017]: Sensitive Data Exposure [OWASP A02:2021]: Cryptographic Failures [REFERENCES]
https://owasp.org/Top10/A02_2021-Cryptographic_Failures
(missing-ssl-minversion-go)
🤖 Prompt for AI Agents
In sources/session.go around lines 48 to 50, the TLSClientConfig lacks a minimum
TLS version setting, which can allow insecure TLS versions. To fix this, add a
MinVersion field to the tls.Config struct and set it to tls.VersionTLS12 to
enforce a minimum of TLS 1.2 for better security and compatibility.
3 participants
closes #679
Summary by CodeRabbit
New Features
Documentation
Chores