Skip to content

Zipped HTML includes unnecessary files #9068

Closed

Description

@MikhailRyazanov

Details

Expected Result

We are building online HTML documentation + additional downloadable PDF and zipped HTML. The project uses many plot directives, and in Sphinx's conf.py we specify plot_formats = ['svg', 'png', 'pdf'] such that SVG images can be used for HTML builds and PDF images for the PDF build (PNG images are saved just in case, they shouldn't harm).

When I do sphinx-build -b singlehtml locally, only the necessary SVG images are copied to the output directory (into its _images subdirectory), as expected, producing a clean result. So I expect that the ZIP archive for “Zipped HTML” built by Read the Docs should have only the index.html file and the _images and _static subdirectories with only the necessary files.

Actual Result

The ZIP archive generated on Read the Docs (htmlzip in .readthedocs.yml; -b readthedocssinglehtmllocalmedia in the log file) contains all the generated image files, including absolutely unneedeed PNG and PDF and even .py files related to them, scattered everywhere, including the root (where only index.html should be) and copies of the subdirectories of our doc source tree.

Activity

  1. MikhailRyazanov commented on Apr 2, 2022

    @MikhailRyazanov
    Author

    Forgot to mention: we also have

    # do not add links to source automatically
    plot_html_show_source_link = False
    # do not add links to image formats
    plot_html_show_formats = False

    in conf.py, so there are really no links to any of these files in additional formats, neither to their “sources”.

  2. stsewd commented on Aug 18, 2022

    @stsewd
    Member

    Hi, sorry for the late reply, can you make sure you are using the same sphinx version on RTD as you do locally? See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html

  3. MikhailRyazanov commented on Aug 21, 2022

    @MikhailRyazanov
    Author

    I'm using Sphinx v4.5.0 locally, whereas our last RTD log says:

    Running Sphinx v1.8.6
    

    which seems to be some old branch (I don't know why RTD uses it by default).

    However, when I was doing further investigations and tried to prepare a minimal working example, I've found that the “correct” local behavior apparently depends on the order in which different targets (html, singlehtml, latex) are built, and a significant problem is actually in the Matplotlib plot directive. Then I was distracted and didn't finish, but I'll try to localize the problem and report it to Matplotlib (and update this issue here).

    Nonetheless, maybe you have some workaround? For example, is it possible to modify the output directory structure before it is packed into the ZIP archive? Using some declarative include/exclude list or maybe some imperative hook?

  4. stsewd commented on Aug 22, 2022

    @stsewd
    Member

    @MikhailRyazanov

    which seems to be some old branch (I don't know why RTD uses it by default).

    This is to keep backwards compatibility with old projects that depended on RTD pinning sphinx to a specific version, new projects get the latest version by default, see https://docs.readthedocs.io/en/stable/build-default-versions.html#default-versions-of-dependencies.

    is it possible to modify the output directory structure before it is packed into the ZIP archive?

    We don't support that, but we may add support for this with a process similar to https://docs.readthedocs.io/en/stable/build-customization.html#override-the-build-process I think

  5. MikhailRyazanov commented on Aug 22, 2022

    @MikhailRyazanov
    Author

    new projects get the latest version by default

    Is there any option to switch on “old” project to “new” (and back, if needed)?

    but we may add support for this with a process similar to https://docs.readthedocs.io/en/stable/build-customization.html#override-the-build-process I think

    OK, looks interesting. I'm curious whether post_build mentioned on top of that page is executed before packing the Sphinx output into ZIP or after? Because in the former case it indeed can be easily customized to remove unnecessary files. (On the other hand, even if it runs after, but the location of the resulting ZIP file is known and writable, the archive itself can be cleaned-up, I hope...)

  6. stsewd commented on Aug 22, 2022

    @stsewd
    Member

    Is there any option to switch on “old” project to “new” (and back, if needed)?

    It's possible, this isn't exposed to users, but it can be done via a support request.

    OK, looks interesting. I'm curious whether post_build mentioned on top of that page is executed before packing the Sphinx output into ZIP or after?

    It's executed after.

  7. MikhailRyazanov commented on Aug 23, 2022

    @MikhailRyazanov
    Author

    Thank!

    It's executed after.

    OK. Is then the ZIP file accessible to the post_build commands, such that we can do zip -d ... on it?

  8. stsewd commented on Aug 24, 2022

    @stsewd
    Member

    OK. Is then the ZIP file accessible to the post_build commands, such that we can do zip -d ... on it?

    It probably is, but this isn't something we will guarantee for the file to always be there, I think the file is generated in the ../../artifacts/ directory (from the root of your project)

  9. MikhailRyazanov commented on Aug 24, 2022

    @MikhailRyazanov
    Author

    Yes, I was afraid that this might be fragile. On the other hand, it's not a big problem if zip -d fails, so this might be worth trying as a temporary solution.

    Let me investigate the details with Matplotlib and see what can be done...

  10. humitos commented on Aug 25, 2022

    @humitos
    Member

    @MikhailRyazanov

    OK. Is then the ZIP file accessible to the post_build commands, such that we can do zip -d ... on it?

    It may not be exactly your case, but when working with post_build you may hit this other issue #9179

  11. humitos commented on Jan 31, 2023

    @humitos
    Member

    Hello 👋🏼 We standardized the output paths for all the formats that Read the Docs supports in #9888. This allow people to output, for example, a HTMLZip under _readthedocs/htmlzip/<project-slug>.zip to be recognized by Read the Docs and expose it as a downloadable content (docs.readthedocs.io/en/stable/downloadable-documentation.html) 💯

    This is the exact directory that Read the Docs will read immediately before uploading this .zip file and host it. So, you can use build.jobs.post_build to modify the _readthedocs/htmlzip/<project-slug>.zip file as you want after Read the Docs built it but before uploading it.

    At build.jobs.post_build you will be able to remove all the unnecessary files you don't want to be included in the final .zip file. Let me know if that works for your use case.

    I'm closing the issue, but feel free to comment here or re-open if you consider it's not solved yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions