Skip to content
This repository was archived by the owner on Mar 18, 2024. It is now read-only.
This repository was archived by the owner on Mar 18, 2024. It is now read-only.

Insufficient write access causes temporary directory to grow endlessly #76

Closed

Description

@Frozenlock

When the destination directory cannot be written to (read-only or insufficient rights), an error is thrown and the temporary directory isn't cleaned up. Any subsequent update attempt will make it grow.

Perhaps the temporary directory could be cleaned before any update attempt?

Activity

  1. mordechaim commented on Jan 7, 2020

    @mordechaim
    Contributor

    For context, do you refer to config.updateTemp()?

    Do you have the issue at update time or finalize time?

  2. Frozenlock commented on Jan 7, 2020

    @Frozenlock
    Author

    Correct, it's when using config.updateTemp().

    As far as I can tell an insufficient write access will mess secureMoveFile, which is called from finalizeUpdate.

  3. mordechaim commented on Jan 7, 2020

    @mordechaim
    Contributor

    Ok, so here's the situation: since temp updating is a 2 step process I didn't handle failures as I did in regular update failures. Additionally, the finalization process doesn't utilize the servicing paradigm, where failures are reported as a method callback.

    Since I strived to be the least opinionated and let you decide how to handle things, I simply let it fail with an exception without any cleanup; while in regular updates I deleted the new files and reported to the service provider. The reason for this is since there might be a wide range of reasons to fail, one might still want to finalize at some time later (while for regular updates it's an all-or-nothing process, so there's no point in keeping the files).

    While an ad-hoc solution would be to clear those files manually at your side, I'll think of a clean solution that still fits my agenda of least level of intrusion.

    I'm open for ideas, what do you think?

  4. mordechaim commented on Jan 7, 2020

    @mordechaim
    Contributor

    Keep in mind, an exceptional finalization doesn't void the update. You can just handle it some way and try the finalization process again.

  5. mordechaim commented on Jan 9, 2020

    @mordechaim
    Contributor

    One potential solution could be instead of passing a temp directory path we can pass a file path instead. The whole update would then be zipped into a single file in that location.

    The assumption would be that future updates would reuse the same file name thus overwriting any stale updates.

    The challenge is now to provide a smooth transition with backwards compatibility.

  6. mordechaim commented on Jan 9, 2020

    @mordechaim
    Contributor

    I realized there's a more critical issue here. Namely, for Windows, I check for file locking prior to moving all files (so it shouldn't fail mid-move) but never check for file accessibility on any platform.

    I will start working on that first, test on both Windows and Linux and then get to this issue.

  7. Frozenlock commented on Jan 9, 2020

    @Frozenlock
    Author

    Since I strived to be the least opinionated and let you decide how to handle things, I simply let it fail with an exception without any cleanup.

    Understandable. I'm not even sure an automatic cleanup would be the correct approach. Were update4j used to download a multi-GB update and then failed to move the files, it would be a little sad to download everything again.

    One potential solution could be instead of passing a temp directory path we can pass a file path instead. The whole update would then be zipped into a single file in that location.

    Would it be possible to use the file hash as temporary name and skip the download if it already exists? (After validating the file has the correct hash/checksum.)

    I will start working on that first, test on both Windows and Linux and then get to this issue.

    Alright, I'll let you know if I come up with a brilliant idea in the meantime. Thank you very much!

  8. mordechaim commented on Jan 9, 2020

    @mordechaim
    Contributor

    Would it be possible to use the file hash as temporary name and skip the download if it already exists? (After validating the file has the correct hash/checksum.)

    Ah! smart, this won't require to change the semantics from folder based to zipped-file based. But if a newer config is available just before finalizing (after failure) that file will not be detected/deleted. I still have a gut feeling the zipped file approach is much cleaner and more maintainable.

    As a side note: It might be I'll have to make a breaking change (and increment minor version instead of bugfix), but I think I should do it gradually, but an opt-in flag and make it mandatory only on the next minor release.

  9. Frozenlock commented on Jan 9, 2020

    @Frozenlock
    Author

    What is the advantage of using a (zipped) filepath? Is it to prevent users from providing a directory that could be used for other things, thus preventing us from deleting every file in the directory?

  10. Frozenlock commented on Jan 11, 2020

    @Frozenlock
    Author

    Assuming we can't consider the provided temp directory as under our complete control, here is an approach that would provide a way to:

    1. Avoid duplication of already downloaded files.
    2. Delete files that are no longer needed (past update that was never finalized).
    3. Leave any other file that could have been placed there by the user untouched.

    The temporary files downloaded by update4j could be named with a particular scheme, something like update4j-temp-<file-hash>, where update4j-temp- is a prefix to identify files created and under update4j's responsability. Getting the list of relevant files would be as simple as checking which ones have the correct prefix. Then, the <file-hash> could be used to determine if a file should be kept, deleted, or re-downloaded.

  11. mordechaim commented on Jan 12, 2020

    @mordechaim
    Contributor

    What is the advantage of using a (zipped) filepath?

    Precisely as you noted. When everything is in a single file we know the data is all ours. Additionally, if a client uses a hardcoded name for the temporary file name we know that old data would be overwritten.

    While your idea of using file prefixes is a nice solution, here's what I think would work beautifully, provide more power and control and fits better with developers' expectations:

    • Add a new method group archiveUpdate() with the exact same params as updateTemp(). We will bundle all outdated files, the configuration itself, and some more context information into a single archive.
    • Update should become a full fledged object which is created by reading that archive. Perhaps use a factory method Update.fromArchive(Path). The object will have methods getConfiguration(), getFiles(), and getTimestamp() which would return the configuration used to archive, the FileMetadata objects of said configuration which are included in this archive and the time the update was archived, respectively.
    • Call myUpdate.install() which would unpack the files into their final destination. I think of also using the service provider paradigm by adding an InstallHandler with callbacks. We would not delete the archive ourself on installation, only the DefaultInstallHandler will do it if and only if the installation succeeded.
    • Deprecate updateTemp() and finalizeUpdate().

    As the famous saying goes "There are only two hard things in Computer Science: cache invalidation and naming things." I think the whole temp/finalize naming was a flaw. Installation is what people are used to and expect extracting from an archive.

  12. mordechaim commented on Jan 12, 2020

    @mordechaim
    Contributor

    I can't decide whether the deletion should be managed by the install implementation itself or by the callback, or perhaps add a boolean flag in the install method deleteOnSuccess.

  13. Frozenlock commented on Jan 16, 2020

    @Frozenlock
    Author

    I think DefaultInstallHandler could delete the archive, as you proposed.
    This is probably what most users would want. If they want something else, than they can provide their own InstallHandler.

  14. nemphys commented on Mar 16, 2020

    @nemphys

    As a side note, I recently implemented a custom script that performs the equivalent of finalizeUpdate() in a shutdownHook (so that I can also update the JVM itself), which does not use a delay, but rather waits for the main application's process to exit before running (the fixed delay is a little unsafe, especially on slower machines).

    On Java9+, one can get the process pid using ProcessHandle.current().pid().

    For linux/mac, it is quite simple: the sleep call is replaced by "while ps -p {{processPid}} > /dev/null; do sleep 1; done".

    On Windows, it is a little more complicated. After banging my head trying to find a suitable cmd one-liner script, I resorted to creating a temporary .bat file with the following contents (which is executed and then deleted):

    @echo off
    :Start
    >NUL 2>&1 timeout /T 1 /NOBREAK
    for /F "tokens=1 delims=:" %%G in ('tasklist /FI "PID eq {{processPid}}" /FO csv /NH') do if /I NOT [%%G]==[INFO] goto Start
    ....update commands....

  15. mordechaim commented on Mar 16, 2020

    @mordechaim
    Contributor

    Thanks @nemphys I'll really consider it, and also for deleteOldFiles().

    Is there a reason you had to make a batch file instead of directly executing them?

  16. nemphys commented on Mar 16, 2020

    @nemphys

    Yes, it seems to be impossible to loop through timeout in a one-liner script; goto calls are not supported there and the only alternative seems to be an endless for /l loop, which apparently cannot be broken.

  17. mordechaim commented on Mar 16, 2020

    @mordechaim
    Contributor

    I think the challenge is that timeout on windows doesn't work in non-interactive mode.

    Did you see how I did the pause here? Can you please confirm it will not work with this solution either?

  18. nemphys commented on Mar 17, 2020

    @nemphys

    I was not aware of that (and thought that I found a better alternative to ping).
    Should I consider it luck that it worked fine so far? Is this actually tested?
    If it is, I will have to revert to ping for the timeout on windows.

  19. mordechaim commented on Mar 17, 2020

    @mordechaim
    Contributor

    It works flawlessly and even better than timeout which always snaps to 0 milliseconds (timeout 1 will wait until next instant ending in 3 zeros). Keep in mind you need one more ping than seconds.

    The question remains if by using ping we don't need to create a batch file anymore.

  20. nemphys commented on Mar 17, 2020

    @nemphys

    No, the batch file is still needed. From my research, in order to keep doing the timeout (either way), one can only use a :label & goto, which is not supported in one-liners. I tried with an endless for /l loop (which can be used in a one-liner), but it seems impossible to break it and execute the rest of the code.
    Maybe you can think of something else (?)

  21. mordechaim commented on Apr 8, 2020

    @mordechaim
    Contributor

    With this solution in the works I'm seriously considering deprecating both the standard update and temp update methods for easier code management for this project, since you can easily achieve both using an ephemeral archive. I also plan on make that method much easier to extend and change around in the future without breakage. Of course I will keep those methods for the time being until most people migrate.

    Would this be too harsh?

  22. nemphys commented on Apr 8, 2020

    @nemphys

    Sounds good, as long as 1) nothing breaks (at least for some time) and 2) the new methods are seriously extendable, so that it is possible to achieve the desired result in special cases like mine.

    I have everything working perfectly and would hate to have to rework this :-)

  23. mordechaim commented on Apr 12, 2020

    @mordechaim
    Contributor

    Can you please elaborate in what sense you need extensibility?

    My current plan is to use a parameter class, namely something like UpdateOptions and return UpdateResult so we don't need all those overloads, we can simply add new methods when needed without exponentially adding overloads and breaking anything. This also solves #87 and #89

  24. nemphys commented on Apr 12, 2020

    @nemphys

    Sounds good. We can always iron out the issue after the feature is implemented.

  25. added 2 commits that reference this issue on Aug 19, 2020
    05b4806
    213eaf1
  26. mordechaim commented on Aug 19, 2020

    @mordechaim
    Contributor

    This has been a long one, but finally, implementation is complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions