Skip to content

Optimize getting relative page URLs#2272

Closed
oprypin wants to merge 5 commits into
mkdocs:masterfrom
oprypin:relpath
Closed

Optimize getting relative page URLs#2272
oprypin wants to merge 5 commits into
mkdocs:masterfrom
oprypin:relpath

Conversation

@oprypin

@oprypin oprypin commented Dec 27, 2020

Copy link
Copy Markdown

This is a custom implementation that's ~5 times faster. That's important because calls to normalize_url on a site with ~300 pages currently take up ~20% of the total run time due to the sheer number of them. The number of calls is at least the number of pages squared.

The old implementation relied on posixpath.relpath, which (among other unnecessary parts) first expands every path with abspath for whatever reason, so actually it becomes even slower if you happen to be building under a deeply nested path.

Profile before/after

profile

In context

profile

This is a custom implementation that's ~5 times faster. That's important because calls to `normalize_url` on a site with ~300 pages currently take up ~20% of the total run time due to the sheer number of them. The number of calls is at least the number of pages squared.

The old implementation relied on `posixpath.relpath`, which (among other unnecessary parts) first expands every path with `abspath` for whatever reason, so actually it becomes even slower if you happen to be building under a deeply nested path.
@waylan

waylan commented Dec 27, 2020

Copy link
Copy Markdown
Contributor

So this change removes calls to the standard lib and replaces them with your own custom implementation. I realize that there may be some standard lib code that we don't need, but the assumption is that the standard lib code is well tested and accurate. Your custom code is an unknown. I suspect the existing tests don't cover nearly enough to be sure this doesn't break various edge cases (admittedly I didn't check). We will always go with accuracy over performance. If you want this accepted, then you will need to demonstrate that this is very well tested.

By the way, you have been providing a lot of performance patches lately. I don't object to that. In fact, I appreciate that someone is looking at the issue, as I don't have any time to dedicate to the issue. In fact, reviewing all these patches which don't improve the project in any perceivable way (from my perspective) just adds more burden on me. Therefore, from my perspective you need to demonstrate that you aren't breaking anything. Your comments always mention how much of a performance improvement this makes. But I don't care much about that. The largest site I have is less than 20 pages. I have the perception that my sites build instantly on my very old machine. Personally I see no need for performance improvements. What I care about is ease of maintenance and accuracy. If you would like me to continue to spend time reviewing your performance improvements, please focus on those two factors first. Any performance improvements will always be second to those in my reviews.

As an example, if the proposed change in this PR were to result in a future bug report, my "fix" would simply be to revert to the previous implementation. I'm not interesting in spending the time to work through the custom code to fix various edge cases. That is more of a maintenance burden than I want to take on. Therefore, before I am willing to accept it, I need to be sure that those bug reports aren't likely to happen in the future.

@oprypin

This comment was marked as abuse.

@waylan

waylan commented Dec 28, 2020

Copy link
Copy Markdown
Contributor

If you would like this merged, then you will need to either point to an existing set of tests (I haven't taken the time to check), or. if they don't exist, provide a comprehensive set of tests. My guess is that you will only find a limited set of tests and need to provide more so that we have a comprehensive set of tests.

@oprypin

This comment was marked as abuse.

@oprypin

This comment was marked as abuse.

@oprypin

This comment was marked as abuse.

@waylan

waylan commented Dec 29, 2020

Copy link
Copy Markdown
Contributor

I still want to see a comprehensive set of unittests included with this patch. And I don't care if some potential inputs will not currently be generated by MkDocs. We can't guarantee that will be the case in the future. If you want to do a custom function, then you ned to provide a complete set of unittests which demonstrate that any potential (past, present or future) input will return the correct normalized result. Until I see that, I'm not spending any more time on this.

@oprypin

This comment was marked as abuse.

@oprypin

This comment was marked as abuse.

@oprypin

This comment was marked as abuse.

@oprypin

This comment was marked as abuse.

@oprypin

This comment was marked as abuse.

@waylan

waylan commented Jan 12, 2021

Copy link
Copy Markdown
Contributor

I have given it a cursory look and it seems okay in principle, but I want to be extra sure your custom replacement for the standard lib function is comely reliable and that is going to take time. Time I havn't been motivated to spend on it.

@oprypin

This comment was marked as abuse.

@waylan

waylan commented Apr 4, 2021

Copy link
Copy Markdown
Contributor

I'm closing this in favor of #2296. This adds our own implementation which is more of a maintenance burden.

@waylan waylan closed this Apr 4, 2021
@oprypin

This comment was marked as abuse.

@waylan

waylan commented Apr 4, 2021

Copy link
Copy Markdown
Contributor

This pull request now 100% exactly matches the implementation at master.

Which means it adds more maintenance burden for no gain.

This change, now compared to master, still cuts off 9% of build times for a site with 142 pages.

Meh. Not a priority. The amount of gain is not worth the extra burden.

@oprypin

This comment was marked as abuse.

@oprypin

This comment was marked as abuse.

@oprypin

This comment was marked as abuse.

@mkdocs mkdocs locked as too heated and limited conversation to collaborators May 13, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants