cleanhttp

A lightweight library to detect and remove known wildcard HTTP web servers. Designed for developers to easily integrate, contribute, and extend signature-based filtering.

Overview

cleanhttp defines signature patterns for HTTP responses that identify wildcard servers. These patterns help detect bad or intentionally wildcarded servers by matching HTTP status codes, headers, titles, and body content.

JSON Structure

Supported Keys:

• http_status_code: Single or Range of status codes (e.g., "500-599").
• http_header: Key-value pairs for HTTP headers.
• http_title: Exact or Partial title match.
• http_title_regex: Regex pattern for matching the title.
• http_body: List of strings that must be contained in the response body.
• http_body_regex: List of regex patterns that must be contained in the response body.

Example:

{
  "services": {
    "cloudflare": {
      "http_status_code": "500-599",
      "http_header": {
        "Server": "cloudflare"
      },
      "http_body": ["error code:"]
    },
    "cloudfront": {
      "http_status_code": "400-499",
      "http_header": {
        "Server": "CloudFront"
      },
      "http_title": "ERROR: The request could not be satisfied",
      "http_body": ["Generated by cloudfront (CloudFront)"]
    },
    "akamai": {
      "http_status_code": "400-499",
      "http_header": {
        "Server": "AkamaiGHost"
      },
      "http_title": "Invalid URL",
      "http_body_regex": ["The requested URL .* invalid"]
    }
  }
}

Contributing

• Follow the JSON structure for adding or updating wildcard server signatures.
• Write tests to verify new pattern matching.
• Submit pull requests with detailed descriptions and adhere to our CONTRIBUTING.md guidelines.

License

Distributed under the MIT License. See LICENSE for details.