Skip to content

Use _mkgmtime64 instead of _mktime64 to implement Windows Unix.stat#861

Closed
nojb wants to merge 1 commit intoocaml:trunkfrom
nojb:fix-windows-unix-stat
Closed

Use _mkgmtime64 instead of _mktime64 to implement Windows Unix.stat#861
nojb wants to merge 1 commit intoocaml:trunkfrom
nojb:fix-windows-unix-stat

Conversation

@nojb
Copy link
Contributor

@nojb nojb commented Oct 17, 2016

(Continued from Mantis MPR #7835)

The problem

Unix.stat on Windows returns time stamps which depend on the DST setting of the computer.

How to reproduce

Given an existing file FILE, simply do

$ ocaml unix.cma
> Unix.stat "FILE";;

then change your Windows DST setting and retry. The resulting timestamps should have changed by 3600. (1 hour).

Some information learned so far

Windows Unix.stat time stamps are currently computed as follows:

  1. time stamp is computed in a Windows-specific type (FILETIME, a 64bit value)
  2. map to the local time zone (using FileTimeToLocalFileTime)
  3. map to "structured" date type (SYSTEMTIME, similar to tm structure, using FileTimeToSystemTime)
  4. map to corresponding tm structure "by hand"
  5. map with _mktime64 (takes a tm structure in local time zone)

It looks like the problematic step is 4. -> 5. Indeed, the tm structure has a field tm_isdst, supposed to be set to 0, 1, or -1 according to whether DST is inactive, active, or should be taken from the OS. Since the output of FileTimeToLocalFileTime takes the DST setting from the OS, one should probably set this field to -1 to compensate (incidentally, this field is always set to -1 in the Un*x implementation of Unix.mktime).

Currently, this field is set to 0 instead. So I hoped this was the culprit. However, passing -1 does not seem to have any effect. This behavior is still a mystery.

Also relevant:

Proposed solution

Sidestepping the conversion to local time by using the Windows function _mkgmtime64. This is an avatar of the non-standard GNU C function timegm. The resulting time stamps seem to be correct and invariant under DST changes.

While this change seems to solve the issue, the source of the problem is not yet fully understood. Any comments on the problem and/or the solution, as well as confirmation of the problem would be greatly appreciated!

@gasche
Copy link
Member

gasche commented Oct 17, 2016

(I don't understand the details so apologies if this is a silly question, but) The MSDN documentation indicates that the TZ environment variable may influence the result of these functions. The fact that TZ would influence the result of Unix.gettimeofday was reported as a bug in MPR#6671, is there a risk of a similar issue popping up here?

@nojb
Copy link
Contributor Author

nojb commented Oct 17, 2016

It should not happen with the proposed workaround, since we would not be doing any timezone conversion.

@nojb
Copy link
Contributor Author

nojb commented Oct 17, 2016

We are running OCaml under Cygwin, so the TZ may be involved somehow. I will try to find out more.

@alainfrisch
Copy link
Contributor

This looks good to me.

@dra27 Do you have an opinion on this topic?

@dra27
Copy link
Member

dra27 commented Oct 25, 2016

I need more time to be able to look at this properly - I've got a feeling that the implementation I did was designed for compatibility with the Microsoft C implementation (the code for that is available in any paid for distro of Visual Studio prior to 2015 and in all 2015 distros). I think the idea was that I only enhanced fields (so some fields are populated, such as st_ino which had fixed values and so would never have been used before), I didn't change any. More anon...

@nojb
Copy link
Contributor Author

nojb commented Feb 17, 2017

An update on this issue: this patch does not compile on mingw (maybe only in 32 bits, did not check) as it does not include the function mkgmtime64. I close this PR since I do not see a way around this. However, the bug remains.

@nojb nojb closed this Feb 17, 2017
@alainfrisch
Copy link
Contributor

I think that fixing for the MSVC ports would already be useful, even if having a different behavior between MSVC and Mingw is not great.

@nojb
Copy link
Contributor Author

nojb commented Feb 17, 2017

OK, I will add the necessary preprocessor macros and update the patch.

@dra27
Copy link
Member

dra27 commented Feb 20, 2017

OK, sorry for the long delay on my input here. I have no idea how this slipped through when I developed stat - perhaps it was a combination of my working in a VM (which has a habit of resetting the system clock to the host) or that I developed this while in the Netherlands and got confused from my usual lazy comfort of living in UTC+0!

So, what is wrong with the code is the use of the wrong API function - it's step 2 which is the problem. FileTimeToLocalFileTime does not take DST into account so, giving my own time zone of GMT/BST, during BST it converts a time to UTC+1 literally (i.e. it treats BST as a timezone, rather than a bias). But this is not correct. It means, for example, that if you are trying to convert the stamp 12-Feb-2017 20:43:00 on a computer in BST then FileTimeToLocalFileTime will give 12-Feb-2017 21:43:00 UTC+1 which is correct, but not what you want. All the remaining functions work as intended - they've simply been given the wrong hour.

The correct sequence - noted in MSDN (brief blush for not reading the docs properly...) is FileTimeToSystemTime followed by SystemTimeToTzSpecificLocalTime which does take DST into account and so gives 12-Feb-2017 20:43:00 BST. This fixes the problem:

--- a/otherlibs/win32unix/stat.c
+++ b/otherlibs/win32unix/stat.c
@@ -118,11 +119,11 @@ static value stat_aux(int use_64, __int64 st_ino, struct _stat64 *buf)
 static int convert_time(FILETIME* time, __time64_t* result, __time64_t def)
 {
   SYSTEMTIME sys;
-  FILETIME local;
+  SYSTEMTIME local;

   if (time->dwLowDateTime || time->dwHighDateTime) {
-    if (!FileTimeToLocalFileTime(time, &local) ||
-        !FileTimeToSystemTime(&local, &sys))
+    if (!FileTimeToSystemTime(time, &sys) ||
+        !SystemTimeToTzSpecificLocalTime(NULL, &sys, &local))
     {
       win32_maperr(GetLastError());
       return 0;

The issue with mingw32 is because it only permits the binary API of Visual Studio 6.0 (the full and reasonable explanation is here). It does this with good reason - _mkgmtime64 is not available in msvcrt.dll until Windows Vista. I'm not convinced that this single issue is a good enough reason to drop Windows 2000 and Windows XP completely.

Using _mkgmtime64 is better, though, because _mktime64 can be affected by the TZ environment variable so for my patch above to be safe we'd need to ensure its unset and call _tzset or SystemTimeToTzSpecificLocalTime and _mktime64 could assume different time zones.

My feeling is that both versions should go in - the additional safety offered by _mkgmtime64 should be taken advantage of where possible and the _mktime64 version only used for compatibility.

I've proposed #1057 for this.

@nojb
Copy link
Contributor Author

nojb commented Feb 20, 2017

Thanks @dra27 for the thorough explanation! Closing this to concentrate the discussion at #1057.

@nojb nojb closed this Feb 20, 2017
@nojb nojb deleted the fix-windows-unix-stat branch February 24, 2017 09:10
stedolan pushed a commit to stedolan/ocaml that referenced this pull request Oct 25, 2022
sadiqj pushed a commit to sadiqj/ocaml that referenced this pull request Feb 21, 2023
stedolan pushed a commit to stedolan/ocaml that referenced this pull request Mar 21, 2023
25188da flambda-backend: Missed comment from PR802 (ocaml#887)
9469765 flambda-backend: Improve the semantics of asynchronous exceptions (new simpler version) (ocaml#802)
d9e4dd0 flambda-backend: Fix `make runtest` on NixOS (ocaml#874)
4bbde7a flambda-backend: Simpler symbols (ocaml#753)
ef37262 flambda-backend: Add opaqueness to Obj.magic under Flambda 2 (ocaml#862)
a9616e9 flambda-backend: Add build system hooks for ocaml-jst (ocaml#869)
045ef67 flambda-backend: Allow the compiler to build with stock Dune (ocaml#868)
3cac5be flambda-backend: Simplify Makefile logic for natdynlinkops (ocaml#866)
c5b12bf flambda-backend: Remove unnecessary install lines (ocaml#860)
ff12bbe flambda-backend: Fix unused variable warning in st_stubs.c (ocaml#861)
c84976c flambda-backend: Static check for noalloc: attributes (ocaml#825)
ca56052 flambda-backend: Build system refactoring for ocaml-jst (ocaml#857)
39eb7f9 flambda-backend: Remove integer comparison involving nonconstant polymorphic variants (ocaml#854)
c102688 flambda-backend: Fix soundness bug by using currying information from typing (ocaml#850)
6a96b61 flambda-backend: Add a primitive to enable/disable the tick thread (ocaml#852)
f64370b flambda-backend: Make Obj.dup use a new primitive, %obj_dup (ocaml#843)
9b78eb2 flambda-backend: Add test for ocaml#820 (include functor soundness bug) (ocaml#841)
8f24346 flambda-backend: Add `-dtimings-precision` flag (ocaml#833)
65c2f22 flambda-backend: Add test for ocaml#829 (ocaml#831)
7b27a49 flambda-backend: Follow-up PR#829 (comballoc fixes for locals) (ocaml#830)
ad7ec10 flambda-backend: Use a custom condition variable implementation (ocaml#787)
3ee650c flambda-backend: Fix soundness bug in include functor (ocaml#820)
2f57378 flambda-backend: Static check noalloc (ocaml#778)
aaad625 flambda-backend: Emit begin/end region only when stack allocation is enabled (ocaml#812)
17c7173 flambda-backend: Fix .cmt for included signatures (ocaml#803)
e119669 flambda-backend: Increase delays in tests/lib-threads/beat.ml (ocaml#800)
ccc356d flambda-backend: Prevent dynamic loading of the same .cmxs twice in private mode, etc. (ocaml#784)
14eb572 flambda-backend: Make local extension point equivalent to local_ expression (ocaml#790)
487d11b flambda-backend: Fix tast_iterator and tast_mapper for include functor. (ocaml#795)
a50a818 flambda-backend: Reduce closure allocation in List (ocaml#792)
96c9c60 flambda-backend: Merge ocaml-jst
a775b88 flambda-backend: Fix ocaml/otherlibs/unix 32-bit build (ocaml#767)
f7c2679 flambda-backend: Create object files internally to avoid invoking GAS (ocaml#757)
c7a46bb flambda-backend: Bugfix for Cmmgen.expr_size with locals (ocaml#756)
b337cb6 flambda-backend: Fix build_upstream for PR749 (ocaml#750)
8e7e81c flambda-backend: Differentiate is_int primitive between generic and variant-only versions (ocaml#749)

git-subtree-dir: ocaml
git-subtree-split: 25188da
EmileTrotignon pushed a commit to EmileTrotignon/ocaml that referenced this pull request Jan 12, 2024
* package layout header contains breadcrumbs now

---------

Co-authored-by: Sabine Schmaltz <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants