You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Mar 18, 2024. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 18, 2024. It is now read-only.
Insufficient write access causes temporary directory to grow endlessly #76
When the destination directory cannot be written to (read-only or insufficient rights), an error is thrown and the temporary directory isn't cleaned up. Any subsequent update attempt will make it grow.
Perhaps the temporary directory could be cleaned before any update attempt?
Ok, so here's the situation: since temp updating is a 2 step process I didn't handle failures as I did in regular update failures. Additionally, the finalization process doesn't utilize the servicing paradigm, where failures are reported as a method callback.
Since I strived to be the least opinionated and let you decide how to handle things, I simply let it fail with an exception without any cleanup; while in regular updates I deleted the new files and reported to the service provider. The reason for this is since there might be a wide range of reasons to fail, one might still want to finalize at some time later (while for regular updates it's an all-or-nothing process, so there's no point in keeping the files).
While an ad-hoc solution would be to clear those files manually at your side, I'll think of a clean solution that still fits my agenda of least level of intrusion.
One potential solution could be instead of passing a temp directory path we can pass a file path instead. The whole update would then be zipped into a single file in that location.
The assumption would be that future updates would reuse the same file name thus overwriting any stale updates.
The challenge is now to provide a smooth transition with backwards compatibility.
I realized there's a more critical issue here. Namely, for Windows, I check for file locking prior to moving all files (so it shouldn't fail mid-move) but never check for file accessibility on any platform.
I will start working on that first, test on both Windows and Linux and then get to this issue.
Since I strived to be the least opinionated and let you decide how to handle things, I simply let it fail with an exception without any cleanup.
Understandable. I'm not even sure an automatic cleanup would be the correct approach. Were update4j used to download a multi-GB update and then failed to move the files, it would be a little sad to download everything again.
One potential solution could be instead of passing a temp directory path we can pass a file path instead. The whole update would then be zipped into a single file in that location.
Would it be possible to use the file hash as temporary name and skip the download if it already exists? (After validating the file has the correct hash/checksum.)
I will start working on that first, test on both Windows and Linux and then get to this issue.
Alright, I'll let you know if I come up with a brilliant idea in the meantime. Thank you very much!
Would it be possible to use the file hash as temporary name and skip the download if it already exists? (After validating the file has the correct hash/checksum.)
Ah! smart, this won't require to change the semantics from folder based to zipped-file based. But if a newer config is available just before finalizing (after failure) that file will not be detected/deleted. I still have a gut feeling the zipped file approach is much cleaner and more maintainable.
As a side note: It might be I'll have to make a breaking change (and increment minor version instead of bugfix), but I think I should do it gradually, but an opt-in flag and make it mandatory only on the next minor release.
What is the advantage of using a (zipped) filepath? Is it to prevent users from providing a directory that could be used for other things, thus preventing us from deleting every file in the directory?
Assuming we can't consider the provided temp directory as under our complete control, here is an approach that would provide a way to:
Avoid duplication of already downloaded files.
Delete files that are no longer needed (past update that was never finalized).
Leave any other file that could have been placed there by the user untouched.
The temporary files downloaded by update4j could be named with a particular scheme, something like update4j-temp-<file-hash>, where update4j-temp- is a prefix to identify files created and under update4j's responsability. Getting the list of relevant files would be as simple as checking which ones have the correct prefix. Then, the <file-hash> could be used to determine if a file should be kept, deleted, or re-downloaded.
What is the advantage of using a (zipped) filepath?
Precisely as you noted. When everything is in a single file we know the data is all ours. Additionally, if a client uses a hardcoded name for the temporary file name we know that old data would be overwritten.
While your idea of using file prefixes is a nice solution, here's what I think would work beautifully, provide more power and control and fits better with developers' expectations:
Add a new method group archiveUpdate() with the exact same params as updateTemp(). We will bundle all outdated files, the configuration itself, and some more context information into a single archive.
Update should become a full fledged object which is created by reading that archive. Perhaps use a factory method Update.fromArchive(Path). The object will have methods getConfiguration(), getFiles(), and getTimestamp() which would return the configuration used to archive, the FileMetadata objects of said configuration which are included in this archive and the time the update was archived, respectively.
Call myUpdate.install() which would unpack the files into their final destination. I think of also using the service provider paradigm by adding an InstallHandler with callbacks. We would not delete the archive ourself on installation, only the DefaultInstallHandler will do it if and only if the installation succeeded.
Deprecate updateTemp() and finalizeUpdate().
As the famous saying goes "There are only two hard things in Computer Science: cache invalidation and naming things." I think the whole temp/finalize naming was a flaw. Installation is what people are used to and expect extracting from an archive.
I can't decide whether the deletion should be managed by the install implementation itself or by the callback, or perhaps add a boolean flag in the install method deleteOnSuccess.
I think DefaultInstallHandler could delete the archive, as you proposed.
This is probably what most users would want. If they want something else, than they can provide their own InstallHandler.
As a side note, I recently implemented a custom script that performs the equivalent of finalizeUpdate() in a shutdownHook (so that I can also update the JVM itself), which does not use a delay, but rather waits for the main application's process to exit before running (the fixed delay is a little unsafe, especially on slower machines).
On Java9+, one can get the process pid using ProcessHandle.current().pid().
For linux/mac, it is quite simple: the sleep call is replaced by "while ps -p {{processPid}} > /dev/null; do sleep 1; done".
On Windows, it is a little more complicated. After banging my head trying to find a suitable cmd one-liner script, I resorted to creating a temporary .bat file with the following contents (which is executed and then deleted):
@echo off
:Start
>NUL 2>&1 timeout /T 1 /NOBREAK
for /F "tokens=1 delims=:" %%G in ('tasklist /FI "PID eq {{processPid}}" /FO csv /NH') do if /I NOT [%%G]==[INFO] goto Start
....update commands....
Yes, it seems to be impossible to loop through timeout in a one-liner script; goto calls are not supported there and the only alternative seems to be an endless for /l loop, which apparently cannot be broken.
I was not aware of that (and thought that I found a better alternative to ping).
Should I consider it luck that it worked fine so far? Is this actually tested?
If it is, I will have to revert to ping for the timeout on windows.
It works flawlessly and even better than timeout which always snaps to 0 milliseconds (timeout 1 will wait until next instant ending in 3 zeros). Keep in mind you need one more ping than seconds.
The question remains if by using ping we don't need to create a batch file anymore.
No, the batch file is still needed. From my research, in order to keep doing the timeout (either way), one can only use a :label & goto, which is not supported in one-liners. I tried with an endless for /l loop (which can be used in a one-liner), but it seems impossible to break it and execute the rest of the code.
Maybe you can think of something else (?)
With this solution in the works I'm seriously considering deprecating both the standard update and temp update methods for easier code management for this project, since you can easily achieve both using an ephemeral archive. I also plan on make that method much easier to extend and change around in the future without breakage. Of course I will keep those methods for the time being until most people migrate.
Sounds good, as long as 1) nothing breaks (at least for some time) and 2) the new methods are seriously extendable, so that it is possible to achieve the desired result in special cases like mine.
I have everything working perfectly and would hate to have to rework this :-)
Can you please elaborate in what sense you need extensibility?
My current plan is to use a parameter class, namely something like UpdateOptions and return UpdateResult so we don't need all those overloads, we can simply add new methods when needed without exponentially adding overloads and breaking anything. This also solves #87 and #89
Activity
mordechaim commentedon Jan 7, 2020
For context, do you refer to
config.updateTemp()?Do you have the issue at update time or finalize time?
Frozenlock commentedon Jan 7, 2020
Correct, it's when using
config.updateTemp().As far as I can tell an insufficient write access will mess
secureMoveFile, which is called fromfinalizeUpdate.mordechaim commentedon Jan 7, 2020
Ok, so here's the situation: since temp updating is a 2 step process I didn't handle failures as I did in regular update failures. Additionally, the finalization process doesn't utilize the servicing paradigm, where failures are reported as a method callback.
Since I strived to be the least opinionated and let you decide how to handle things, I simply let it fail with an exception without any cleanup; while in regular updates I deleted the new files and reported to the service provider. The reason for this is since there might be a wide range of reasons to fail, one might still want to finalize at some time later (while for regular updates it's an all-or-nothing process, so there's no point in keeping the files).
While an ad-hoc solution would be to clear those files manually at your side, I'll think of a clean solution that still fits my agenda of least level of intrusion.
I'm open for ideas, what do you think?
mordechaim commentedon Jan 7, 2020
Keep in mind, an exceptional finalization doesn't void the update. You can just handle it some way and try the finalization process again.
mordechaim commentedon Jan 9, 2020
One potential solution could be instead of passing a temp directory path we can pass a file path instead. The whole update would then be zipped into a single file in that location.
The assumption would be that future updates would reuse the same file name thus overwriting any stale updates.
The challenge is now to provide a smooth transition with backwards compatibility.
mordechaim commentedon Jan 9, 2020
I realized there's a more critical issue here. Namely, for Windows, I check for file locking prior to moving all files (so it shouldn't fail mid-move) but never check for file accessibility on any platform.
I will start working on that first, test on both Windows and Linux and then get to this issue.
Frozenlock commentedon Jan 9, 2020
Understandable. I'm not even sure an automatic cleanup would be the correct approach. Were
update4jused to download a multi-GB update and then failed to move the files, it would be a little sad to download everything again.Would it be possible to use the file hash as temporary name and skip the download if it already exists? (After validating the file has the correct hash/checksum.)
Alright, I'll let you know if I come up with a brilliant idea in the meantime. Thank you very much!
mordechaim commentedon Jan 9, 2020
Ah! smart, this won't require to change the semantics from folder based to zipped-file based. But if a newer config is available just before finalizing (after failure) that file will not be detected/deleted. I still have a gut feeling the zipped file approach is much cleaner and more maintainable.
As a side note: It might be I'll have to make a breaking change (and increment minor version instead of bugfix), but I think I should do it gradually, but an opt-in flag and make it mandatory only on the next minor release.
Frozenlock commentedon Jan 9, 2020
What is the advantage of using a (zipped) filepath? Is it to prevent users from providing a directory that could be used for other things, thus preventing us from deleting every file in the directory?
Frozenlock commentedon Jan 11, 2020
Assuming we can't consider the provided temp directory as under our complete control, here is an approach that would provide a way to:
The temporary files downloaded by
update4jcould be named with a particular scheme, something likeupdate4j-temp-<file-hash>, whereupdate4j-temp-is a prefix to identify files created and under update4j's responsability. Getting the list of relevant files would be as simple as checking which ones have the correct prefix. Then, the<file-hash>could be used to determine if a file should be kept, deleted, or re-downloaded.mordechaim commentedon Jan 12, 2020
Precisely as you noted. When everything is in a single file we know the data is all ours. Additionally, if a client uses a hardcoded name for the temporary file name we know that old data would be overwritten.
While your idea of using file prefixes is a nice solution, here's what I think would work beautifully, provide more power and control and fits better with developers' expectations:
archiveUpdate()with the exact same params asupdateTemp(). We will bundle all outdated files, the configuration itself, and some more context information into a single archive.Updateshould become a full fledged object which is created by reading that archive. Perhaps use a factory methodUpdate.fromArchive(Path). The object will have methodsgetConfiguration(),getFiles(), andgetTimestamp()which would return the configuration used to archive, theFileMetadataobjects of said configuration which are included in this archive and the time the update was archived, respectively.myUpdate.install()which would unpack the files into their final destination. I think of also using the service provider paradigm by adding anInstallHandlerwith callbacks. We would not delete the archive ourself on installation, only theDefaultInstallHandlerwill do it if and only if the installation succeeded.updateTemp()andfinalizeUpdate().As the famous saying goes "There are only two hard things in Computer Science: cache invalidation and naming things." I think the whole temp/finalize naming was a flaw. Installation is what people are used to and expect extracting from an archive.
mordechaim commentedon Jan 12, 2020
I can't decide whether the deletion should be managed by the install implementation itself or by the callback, or perhaps add a boolean flag in the install method
deleteOnSuccess.Frozenlock commentedon Jan 16, 2020
I think
DefaultInstallHandlercould delete the archive, as you proposed.This is probably what most users would want. If they want something else, than they can provide their own
InstallHandler.nemphys commentedon Mar 16, 2020
As a side note, I recently implemented a custom script that performs the equivalent of finalizeUpdate() in a shutdownHook (so that I can also update the JVM itself), which does not use a delay, but rather waits for the main application's process to exit before running (the fixed delay is a little unsafe, especially on slower machines).
On Java9+, one can get the process pid using ProcessHandle.current().pid().
For linux/mac, it is quite simple: the sleep call is replaced by "while ps -p {{processPid}} > /dev/null; do sleep 1; done".
On Windows, it is a little more complicated. After banging my head trying to find a suitable cmd one-liner script, I resorted to creating a temporary .bat file with the following contents (which is executed and then deleted):
@echo off
:Start
>NUL 2>&1 timeout /T 1 /NOBREAK
for /F "tokens=1 delims=:" %%G in ('tasklist /FI "PID eq {{processPid}}" /FO csv /NH') do if /I NOT [%%G]==[INFO] goto Start
....update commands....
mordechaim commentedon Mar 16, 2020
Thanks @nemphys I'll really consider it, and also for
deleteOldFiles().Is there a reason you had to make a batch file instead of directly executing them?
nemphys commentedon Mar 16, 2020
Yes, it seems to be impossible to loop through timeout in a one-liner script; goto calls are not supported there and the only alternative seems to be an endless for /l loop, which apparently cannot be broken.
mordechaim commentedon Mar 16, 2020
I think the challenge is that
timeouton windows doesn't work in non-interactive mode.Did you see how I did the pause here? Can you please confirm it will not work with this solution either?
nemphys commentedon Mar 17, 2020
I was not aware of that (and thought that I found a better alternative to ping).
Should I consider it luck that it worked fine so far? Is this actually tested?
If it is, I will have to revert to ping for the timeout on windows.
mordechaim commentedon Mar 17, 2020
It works flawlessly and even better than timeout which always snaps to 0 milliseconds (timeout 1 will wait until next instant ending in 3 zeros). Keep in mind you need one more ping than seconds.
The question remains if by using ping we don't need to create a batch file anymore.
nemphys commentedon Mar 17, 2020
No, the batch file is still needed. From my research, in order to keep doing the timeout (either way), one can only use a :label & goto, which is not supported in one-liners. I tried with an endless for /l loop (which can be used in a one-liner), but it seems impossible to break it and execute the rest of the code.
Maybe you can think of something else (?)
mordechaim commentedon Apr 8, 2020
With this solution in the works I'm seriously considering deprecating both the standard update and temp update methods for easier code management for this project, since you can easily achieve both using an ephemeral archive. I also plan on make that method much easier to extend and change around in the future without breakage. Of course I will keep those methods for the time being until most people migrate.
Would this be too harsh?
nemphys commentedon Apr 8, 2020
Sounds good, as long as 1) nothing breaks (at least for some time) and 2) the new methods are seriously extendable, so that it is possible to achieve the desired result in special cases like mine.
I have everything working perfectly and would hate to have to rework this :-)
mordechaim commentedon Apr 12, 2020
Can you please elaborate in what sense you need extensibility?
My current plan is to use a parameter class, namely something like
UpdateOptionsand returnUpdateResultso we don't need all those overloads, we can simply add new methods when needed without exponentially adding overloads and breaking anything. This also solves #87 and #89nemphys commentedon Apr 12, 2020
Sounds good. We can always iron out the issue after the feature is implemented.
Version bump. Closes #76, closes #87
Version bump. Closes #76, closes #87
mordechaim commentedon Aug 19, 2020
This has been a long one, but finally, implementation is complete.