Optimize getting relative page URLs, now with less custom code#2407
Merged
Conversation
added 2 commits
May 16, 2021 19:22
This is a custom implementation that's significantly faster but gives the exact same results as the current one. The use of `posixpath.relpath` pulled in several path-specific transformations that are never needed here. Efficiency is important because calls to `normalize_url` (e.g.) on a site with ~300 pages currently take up ~10% of the total run time due to the sheer number of them. The number of calls is at least the number of pages squared.
This comment was marked as abuse.
This comment was marked as abuse.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a custom implementation that's significantly faster but always gives the exact same results as the current one. The use of
posixpath.relpathpulled in several path-specific transformations that are never needed here.Efficiency is important because calls to
normalize_url(e.g.) on a site with ~300 pages currently take up ~10% of the total run time due to the sheer number of them. The number of calls is at least the number of pages squared.Generally, approximating that the number of these calls is N×N×2, this is the graph that we end up with:
Full source code how I got this result
This shows the total time spent getting relative paths (Y axis) over the course of building a site with that many pages (X axis).
With red color being "before" and green color being "after".
This does not show the total site build times, rather you can only subtract "red" from "green" to approximate the absolute savings of time.
In all other regards the sites' build times grow linearly, but only this particular place it grows as N^2 (because the templates end up linking to every page on every built page). So, with more and more pages, more and more percentage of time is spent just generating these relative URLs.
So I'm making it very optimal, because it's the place that really matters.
Previously: #2272, #2296 (a big number of tests was already added there)