What's Changed

🎉 New Features

• Added headless mode support by @Ice3man543 in #1396

You can try out the headless crawler today with the below steps:

# Simple headless crawl, show the browser as well
katana -u https://cloud.projectdiscovery.io -headless -show-browser

# Deeper exploration
katana -u https://ginandjuice.shop -headless -depth 5 -aff

# With debug logging (for investigating state failures, etc) writes diagnostic information in a new directory for debugging purposes along with more verbose logging.
katana -debug -u https://www.github.com -headless -enable-diagnostics

🐞 Bug Fixes

• Fixed concurrency not working as expected in the hybrid crawler by @majiayu000 in #1488

New Contributors

• @majiayu000 made their first contribution in #1488

Full Changelog: v1.3.0...v1.4.0

What's Changed

🎉 New Features

• Added -no-default-ext-filter option to disable default extension filter by @mjkim610 in #1386
• Added -eof and -lof flags for JSON output filtering by @hanXen in #1390

🐞 Bug Fixes

• Fixed -kf option validation by guarding it with pre-defined values by @mjkim610 in #1359
• Fixed nil ptr dereference in response handling by @dogancanbakir in #1407
• Fixed onclick event detection in javascript by @zsbahtiar in #1427
• Fixed nil ptr dereference in response handling in makeRequest function by @zzdboy in #1460
• Fixed -no-scope behaviour by @Deamhan in #1454

Other Changes

• Added crawl depth for each entry in verbose logs by @mjkim610 in #1418
• Updated .gitignore by @mjkim610 in #1395
• Updated formatJSON function to use OrderedMap to preserve order of output fields by @hanXen in #1417
• Improved program exit status handling by @dogancanbakir in #1388
• Improved error handling for for empty response by @mjkim610 in #1409
• Improved depth filtering by updating order of operations by @Deamhan in #1455

New Contributors

• @hanXen made their first contribution in #1390
• @mjkim610 made their first contribution in #1386
• @zsbahtiar made their first contribution in #1427
• @zzdboy made their first contribution in #1460
• @Deamhan made their first contribution in #1455

Full Changelog: v1.2.2...v1.3.0

What's Changed

• Added option to disable unique content filter by @nirohfeld in #1356
• Added cookie support with crawl by @vbisbest in #1345
• Added stale workflow by @dogancanbakir in #1382

New Contributors

• @nirohfeld made their first contribution in #1356
• @vbisbest made their first contribution in #1345

Full Changelog: v1.2.1...v1.2.2

What's Changed

Other Changes

• Bumped Dockerfile golang version by @dogancanbakir in #1339

Full Changelog: v1.2.0...v1.2.1

What's Changed

🎉 New Features

• Added -output-template option to allow users to define custom output formats using templates, providing flexibility in structuring results by @dwisiswant0 in #1197

$ katana -silent -u https://www.google.com/appserve/security-bugs/m2/new -ot "{{email}} - {{url}}"
[email protected] - https://www.google.com/appserve/security-bugs/m2/new

• Introduced OnSkipURL to allow more control over SDK behavior by @dogancanbakir in #1329
• Introduced -path-climb option to allow auto discovering parent paths to crawl by @dogancanbakir in #1334

$ katana -silent -u https://scanme.sh/test/path/to/file.html -path-climb
https://scanme.sh/test/path/to/file.html
https://scanme.sh/test/path
https://scanme.sh/test/path/to
https://scanme.sh/test

🐞 Bug Fixes

• Fixed DSL matcher by @dogancanbakir in #1283

Other Changes

• Replaced global parser state with instance-based parsers to enabling more flexible and isolated parsing behavior by @awerqo in #1275
• Improved network request capturing to include XHR, Fetch, and Script resource types when extraction is enabled by @dwisiswant0 in #1277
• Improved JavaScript file endpoint detection to better handle varying relative path formats, ensuring more accurate matching across a broader range of file paths. by @Autumn-27 in #1242
• Improved error guarding for crawl session initialization by @dogancanbakir in #1300
• Updated release test action to use composite action by @dogancanbakir in #1308

New Contributors

• @awerqo made their first contribution in #1275
• @Autumn-27 made their first contribution in #1242

Full Changelog: v1.1.3...v1.2.0

What's Changed

🐞 Bug Fixes

• Fixed the issue where the "-ef" parameter only filtered at the output level, which increased memory usage by @dogancanbakir in #1203
• Fixed crawl issue by replacing WaitLoad/Idle with WaitStable to ensure all requests are completed by @alban-stourbe-wmx in #1217
• Fixed Dockerfile by adjusting the build environment setup by @LukasForst in #1122

Other Changes

• Made -store-response feature to properly save URLs with ports by @dogancanbakir in #1229
• Fixed typos in both the README and code documentation by @kianmeng in #1208
• Added note to documentation about Go version required by @adaminfinitum in #1216

New Contributors

• @LukasForst made their first contribution in #1122
• @kianmeng made their first contribution in #1208
• @adaminfinitum made their first contribution in #1216

Full Changelog: v1.1.2...v1.1.3

What's Changed

• Fixed higher usages of cpu and memory by katana by @Ice3man543 in #1107
• Added content len field by @dogancanbakir in #1033
• Added validations to handle bad URLs by @SecGus in #1090
• ⚠️ Added -tech-detect as optional feature (previously enabled as default) by @Ice3man543 in #1107

   -td, -tech-detect  enable technology detection (works with jsonl output)

Issue closed in this release - https://github.com/projectdiscovery/katana/milestone/10?closed=1

New Contributors

• @SecGus made their first contribution in #1090

Full Changelog: v1.1.1...v1.1.2

⚠️ Breaking Changes:

The Passive Crawling capability is being removed from katana and released as a separate project urlfinder by @dogancanbakir in #899

PASSIVE:
    -ps, -passive                   enable passive sources to discover target endpoints
    -pss, -passive-source string[]  passive source to use for url discovery (waybackarchive,commoncrawl,alienvault)

What's Changed

🎉 New Features

• Added no-clobber flag to prevent katana from overwriting existing local files. by @dogancanbakir in #827

-ncb, -no-clobber                 do not overwrite output file

• Added -store-field-dir flag to make per-host field directory optional by @dogancanbakir in #877

   -sfd, -store-field-dir string     store per-host field to custom directory

🐞 Bug Fixes

• Fixed improper logging configuration by @dogancanbakir in #825
• Fixed the issue with setting custom headers/cookie headers by @RamanaReddy0M in #813
• Fixed header marshalling issue by preserving the header name casing by @dogancanbakir in #924
• Fixed ignoring form data extraction with -form-extraction when action is "#" by @dogancanbakir in #1052
• Fixed initial host scope checks to allow continued crawling when -crawl-scope is used by @dogancanbakir in #858
• Fixed duplicate field output when multiple value is used with -field flag by @dogancanbakir in #1031
• Fixed Dockerfile build errors by updating Golang by @dualfade in #1034
• Fixed issue where Ctrl-c could not terminate the process in headless mode with -show-browser option by @zrquan in #972
• Fixed the issue with setting custom headers/cookie headers in headless mode by @michael2to3 in #868
• Fixed cookie handling in hybrid mode to include in output by @alban-stourbe-wmx in #936

Other Changes

• The leakless switch is now enabled by default in headless mode, ensuring the browser is terminated when the Go process ends by @dogancanbakir in #831
• Updated README for recently introduced flag -e by @zy9ard3 in #826
• Corrected typos in comments by @eveneast in #851
• Added directive to ensure compatibility with Windows OS or 386 architecture by @Mzack9999 in #841
• Added deduplication for stored the fields on scan completion by @dogancanbakir in #885
• Updated README for installation command by @dwisiswant0 in #1015
• Added a response parser to extract endpoints from htmx attributes by @zrquan in #994
• Updated chrome launcher logic to prioritize the specified Chrome path with -scp over default paths by @zrquan in #979
• Added automatic form filling for select and textarea in forms by @alban-stourbe-wmx in #921
• Disabled headless auto form filling by @dogancanbakir in #918
• Added more JavaScript Libraries to improve parsing coverage by @geeknik in #900
• Corrected path handling with filepath.Join to use OS-specific separators. @ShuBo6 in #883
• Updated Dockerfile to recent Golang and Alpine images by @o6uoq in #886
• Improved form extraction logic to use HTML placeholder if input value is missing by @fmuttis in #957

New Contributors

• @zy9ard3 made their first contribution in #826
• @eveneast made their first contribution in #851
• @ShuBo6 made their first contribution in #883
• @o6uoq made their first contribution in #886
• @michael2to3 made their first contribution in #868
• @alban-stourbe-wmx made their first contribution in #936
• @zrquan made their first contribution in #972
• @fmuttis made their first contribution in #957
• @dualfade made their first contribution in #1034
• @dwisiswant0 made their first contribution in #1015

Full Changelog: v1.1.0...v1.1.1

What's Changed

🎉 New Features

• Introduced passive crawling by @dogancanbakir in #781

   -ps, -passive  enable passive sources to discover target endpoints
   -pss, -passive-source string[]  passive source to use for url discovery (waybackarchive,commoncrawl,alienvault)

Example:

katana -u tesla.com -passive -f qurl
...
https://static-assets-pay.tesla.com/api/payment-schema/creditcard?countrycode=dk
https://static-assets-pay.tesla.com/api/paymenttypes/?clientlibrary=payment-website
https://tradepartnertickets.tesla.com/dist/main.css?029b26e9be3aef4fc82c
https://tradepartnertickets.tesla.com/dist/vendors.vendors.css?029b26e9be3aef4fc82c
https://workforce.tesla.com/auth/callback?code=na_921f11c72db1d416c2fb624ea94ab5e1bad5f803
...
...
...
[INF] Found 208114 endpoints for https://tesla.com in 2m11.65937825s (commoncrawl: 128, alienvault: 1615, waybackarchive: 206371)

• Introduced option to exclude targets to crawl by @dogancanbakir in #731

   -e, -exclude string[]  exclude host matching specified filter ('cdn', 'private-ips', cidr, ip, regex)

🐞 Bug Fixes

• Fixed issue with headless redirect by @Mzack9999 in #823
• Fixed issue with filter and match functionality (SDK) by @dogancanbakir in #775
• Fixed issue with -kf option to skip if the maximum depth is exceeded by @dogancanbakir in #799

Other Changes

• Added conditional support to include go-tree-sitter in build process by @RamanaReddy0M in #814
• Fixed nil pointer check by @dogancanbakir in #773
• Fixed error check on hybrid request parsing by @niudaii in #804
• Updated jsluice to new version with more results by @Ice3man543 in #695
• Updated jsluice by @geeknik in #800

New Contributors

• @beac0n5 made their first contribution in #732
• @toufik-airane made their first contribution in #736
• @niudaii made their first contribution in #804
• @geeknik made their first contribution in #800

Full Changelog: v1.0.5...v1.1.0

What's Changed

🎉 New Features

• Added option to disable redirects with -disable-redirects by @ErikOwen in #630

   -dr, -disable-redirects  disable following redirects (default false)

🐞 Bug Fixes

• Fixed issue with header in output with XHR requests by @ErikOwen in #631
• Fixed with -aff config option by @dogancanbakir in #676
• Fixed issue with -kf option by @dogancanbakir in #655
• Fixed issue with blank field with empty value by @dogancanbakir in #687

Full Changelog: v1.0.4...v1.0.5