feat(iplist2rule): Adds support for extracting addresses from JSON sources - #1833
Conversation
Major crawlers typically publish IP lists in JSON format:
```json
{
"creationTime": "2025-10-30T11:00:00.000000",
"prefixes": [
{
"ipv4Prefix": "132.196.86.0/24"
},
{
"ipv6Prefix": "2001:4860:4801:10::/64"
},
...
]
}
```
These changes try to parse the URL response body as JSON
when the content type is "application/json" or the extension
is ".json".
Assisted-by: Claude Sonnet 5 via Claude Code
Signed-off-by: David Chandek-Stark <[email protected]>
dchandekstark
changed the title
Signed-off-by: David Chandek-Stark <[email protected]>
Xe
left a comment
Xe
left a comment
There was a problem hiding this comment.
I suspect this is gonna end up with a giant list of per-provider format parsers like reputationdb has, but this is a good place to start from. Thanks!
Yeah, I'm sure you've seen much more that I have! Seems like most all the major players use this schema. In any case, probably better to build on this work than to develop the shell commands I used to update IP lists in recents PRs. I may also see about adding an input file option to do updating like I did with yq + jq. |
|
There aren't existing tests for iplist2rule AFAICT, so I added none. Happy to do so if need be. I did run the integration tests, but not sure they're relevant to utils/. |
2 participants
Major crawlers typically publish IP lists in JSON format:
{ "creationTime": "2025-10-30T11:00:00.000000", "prefixes": [ { "ipv4Prefix": "132.196.86.0/24" }, { "ipv6Prefix": "2001:4860:4801:10::/64" }, ... ] }These changes try to parse the URL response body as JSON when the content type is "application/json" or the extension is ".json".
I would also like to add the option to omit the output file to print to stdout -- that can happen as part of this PR, or separately, depending on timing.
Assisted-by: Claude Sonnet 5 via Claude Code
Checklist:
[Unreleased]section of docs/docs/CHANGELOG.mdnpm run test:integration(unsupported on Windows, please use WSL)