Skip to content

Add support for URI remapping#620

Merged
mre merged 9 commits intomasterfrom
remap-url
May 29, 2022
Merged

Add support for URI remapping#620
mre merged 9 commits intomasterfrom
remap-url

Conversation

@mre
Copy link
Copy Markdown
Member

@mre mre commented May 15, 2022

Allow users to remap URIs. Regular expressions are supported. Remappings get evaluated in order of definition. Multiple remappings are allowed.
The format is similar to that of markdown-link-check.

How it works ⚙️

Consider the following input:

file://../../issues
https://example.com
https://example.org
https://example.net

Let's remap these inputs using the --remap parameter:

lychee --remap 'https://example.com http://127.0.0.1:8080' \
       --remap 'https://example.org https://staging.example.com' \
       --remap '../../issues https://github.com/usnistgov/OSCAL/issues' \
       -- input.md

This would check the following URIs:

https://github.com/usnistgov/OSCAL/issues
http://127.0.0.1:8080/
https://staging.example.com/
https://example.net/

Caveats 🚩

  • Be careful when using this feature because checking every link against a large set of regular expressions has a performance impact even though the Rust regex engine is very fast.
  • There are no constraints on the URI mapping, so the rules might contradict each other. It's the user's responsibility to make sure this does not happen.
  • File remappings might be a bit surprising. Files are handled as URIs internally and get prefixed with a file:// scheme. As a consequence, using ^../../issues as a remap pattern would never match. Instead ../../issues (without the ^) would work.

Open Questions 🤔 (please provide feedback)

  • Shall we first remap a URI and then check if it is excluded or vice versa? 🤔

Fixes #547

mre added 5 commits May 15, 2022 15:23
Remaps allow mapping from a URI pattern to a different URI.

The syntax is

```
lychee --remap 'https://example.com http://127.0.0.1'
```

Some use-cases are
- Testing URIs prior to production deployment
- Testing URIs behind a proxy

Be careful when using this feature because checking every link against a
large set of regular expressions has a performance impact. Also there are no
constraints on the URI mapping, so the rules might contradict with each
other.
Remap rules get applied in order of definition to every input URI.
@mre
Copy link
Copy Markdown
Member Author

mre commented May 20, 2022

Any comments before this gets merged?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for domain remapping URLs to custom endpoints (schema-IP-port)

1 participant