Skip to content

Synchronized Pervasives implementation between stdlib and threads#943

Merged
damiendoligez merged 1 commit intoocaml:trunkfrom
mmottl:synced-stdlib-threads-pervasives
Jul 21, 2017
Merged

Synchronized Pervasives implementation between stdlib and threads#943
damiendoligez merged 1 commit intoocaml:trunkfrom
mmottl:synced-stdlib-threads-pervasives

Conversation

@mmottl
Copy link
Contributor

@mmottl mmottl commented Nov 29, 2016

While trying to add some other features, I noticed that the Pervasives implementations between the standard library and its threads implementation had diverged. This patch merges the two implementations as much as possible and should make it easier to compare or diff them.

The only changes relevant to code execution are the following:

  • float_of_bits in threads now uses the same external declaration as in stdlib. There does not seem to be anything thread-specific there.
  • bytes_length used the wrong primitive in stdlib. It shouldn't really make a difference with the current implementation, but the patch makes it more future-proof.
  • flush_all in the stdlib now only catches Sys_error exceptions. I believe only assertion failures could be raised otherwise and might go unnoticed without the change. Alternatively, we could also suppress all exceptions in threads, but maybe it's better to be more specific.
  • close_out in threads suppressed exceptions escaping flush, which violates the manual and is inconsistent with stdlib.

Please let me know if you would prefer to merge the two implementations differently.

external out_channel_length : out_channel -> int = "caml_ml_channel_size"
external close_out_channel : out_channel -> unit = "caml_ml_close_channel"
let close_out oc = flush oc; close_out_channel oc
let close_out oc = (try flush oc with _ -> ()); close_out_channel oc
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Buffered data could be silently lost if flush fails. That makes this particular change seem dangerous.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hcarty Note that this patch only makes stdlib and threads consistent. In this case I chose the semantics as already implemented in threads, which I considered saner. It will close the channel for sure, otherwise we would be leaking file descriptors. If a user absolutely wants to make sure that their data has been flushed, they need to explicitly call flush themselves before calling close_out, thus potentially raising Sys_error exceptions. flush_all also suppresses such exceptions, because the user could not possibly know anyway which channel was broken.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mmottl Scope of a surrounding try/with would affect whether there is a leak or not here.

Regardless, this change should be reflected in the changelog and documentation to call out the requirement for an explicit call to flush to guarantee flushing of buffered output.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hcarty Ok, I have looked at the documentation and it actually already specifies that close_out may raise even while flushing and before closing. Users who don't like that behavior should use close_out_noerr, which completely ignores errors. This means that the previous implementation in threads was not compliant with the manual while the stdlib was.

I have just submitted a patch to make the current stdlib behavior the default with both implementations. They must definitely behave the same for that function. It would be very confusing for users if that wasn't the case. If some thread-related library needed the previous closing behavior, that code would have to be changed to use close_out_noerr. I have looked, but haven't seen any such code in the distribution. The semantics as specified in the manual should definitely prevail.

@gasche
Copy link
Member

gasche commented Nov 29, 2016

I'm worried about changes that make stdlib follow threads's behavior and not the other way around, as threads's behavior is much more likely to be untested and not what people have been relying on. It may be the case that it is actually the better behavior (I suspect the %bytes_length vs. %string_length change is an aftermath of #596 and #772, but it looks like reverting to string_length here is an accidental change that was not required), but each would need really careful review -- maybe you should consider having separate PRs for these changes.

@mmottl
Copy link
Contributor Author

mmottl commented Nov 29, 2016

@gasche In this case I actually believe that it's the other way round: the threads behavior is much better tested. I'm pretty sure that a majority of real-world applications, certainly I/O-heavy systems applications, links with the threads version.

Concerning %bytes_length vs %string_length, I'm sure that this is not in any way related to threads. One of the two implementations must be the right one. The type signature of the external declaration suggests that the threads version is the right one to use, otherwise I'd seriously question the naming convention used for the primitives.

@mmottl
Copy link
Contributor Author

mmottl commented Nov 29, 2016

I've just updated the initial comment describing the pull request to reflect changes wrt. close_out.

@mshinwell
Copy link
Contributor

I tend to agree that the threads library is probably in fact better tested.

@gasche
Copy link
Member

gasche commented Dec 9, 2016

This may be true, but for all discrepancies pointed out above, stdlib is in fact the reference version. There is one case (bytes_length) where a stdlib change not reflected in threads was in fact a bad idea, and it's a good idea to revert it, but that does not change my comment that I was worried about changing "stdlib" to match "thread" without a very careful review.

@ghost
Copy link

ghost commented Jan 5, 2017

This patch touches otherlibs/threads, which is for vmthreads. I think none of the code in this directory is used when you are using system threads. I would think that no one uses vmthreads these days so the code in otherlibs/threads is likely to be the less tested one.

('a, 'b, 'c, 'd, 'e, 'f) format6 = "%identity"

let ( ^^ ) (Format (fmt1, str1)) (Format (fmt2, str2)) =
let (^^) (Format (fmt1, str1)) (Format (fmt2, str2)) =
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should fix threads/pervasives to add the spaces rather than remove them here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, it is done.

@mshinwell
Copy link
Contributor

(I made a mistake in my previous comment above; I was thinking that the threads library being referred to was systhreads, but it is not.)

Is there a reason to keep the vmthreads library around? Does anyone use it?

@mmottl
Copy link
Contributor Author

mmottl commented Mar 31, 2017

@mshinwell At the time the vmthreads library was implemented, POSIX threads were not yet that widely or well-supported by operating systems. Xavier even wrote the first Linux implementation of POSIX threads (LinuxThreads), probably just so OCaml could have systhreads 😏.

I guess the question boils down to whether there are still any platforms left that don't have POSIX-threads and are worth supporting. My guess is probably not.

@damiendoligez damiendoligez added this to the 4.06-or-later milestone Apr 5, 2017
@mshinwell
Copy link
Contributor

@xavierleroy Would you be happy removing vmthreads or not?

@alainfrisch
Copy link
Contributor

Does the implementation of vmthreads make sense in the context of compiling the ocaml bytecode interpreter + runtime system to WebAssembly, for instance?

@xavierleroy
Copy link
Contributor

You need to ask the greater OCaml user community to see if vmthreads are still in use.

The only use I know of is in JoCaml, as a more lightweight alternative to system threads. But JoCaml isn't particularly active these days...

Personally, it wouldn't break my heart to remove vmthreads altogether. System threads are enough of a mess already :-)

@ghost
Copy link

ghost commented Apr 10, 2017

I can ask on the caml-list.

At worse I suppose that this could easily be split into a separate library. The only builtin support in the compiler seems to be adding +vmthreads to the search path when -vmthreads is passed. Moreover, this option is only useful for projects that don't use ocamlfind since ocamlfind already interprets -thread/-vmthreads and setups the search path accordingly.

@alainfrisch it seems that browsers are getting support for pthreads in javascript. I guess if this happens WebAssembly will get pthreads as well. In any case, lwt/async are much more popular and provides similar features, so it doesn't seem too bad to me to let anyone interested maintain it as a separate library.

BTW, we'll we need the consurtium approval to drop support for vmthreads?

Copy link
Contributor

@alainfrisch alainfrisch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@mmottl Can you add a Changelog entry, marked as potentially breaking (mentioning the change of behavior for close_out in the thread version)?

@mmottl mmottl force-pushed the synced-stdlib-threads-pervasives branch from afe7af7 to 105880b Compare July 20, 2017 16:21
@mmottl
Copy link
Contributor Author

mmottl commented Jul 20, 2017

@alainfrisch Thanks for the approval, Alain, and to all the reviewers. I have added the Changelog entry and rebased + squashed this branch wrt. the current trunk.

@damiendoligez damiendoligez merged commit 1fc6ccb into ocaml:trunk Jul 21, 2017
stedolan pushed a commit to stedolan/ocaml that referenced this pull request Mar 21, 2023
stedolan pushed a commit to stedolan/ocaml that referenced this pull request Mar 21, 2023
b11eea1 flambda-backend: Introduce Import_info (ocaml#1036)
bc5b135 flambda-backend: Fix `ocamlobjinfo` on flambda2 .cmx files (ocaml#1029)
c8babbd flambda-backend: Compilation_unit optimisations (ocaml#1035)
e8d3e22 flambda-backend: Use 4.14.0 opam switch for building (includes upgrading ocamlformat to 0.24.1) (ocaml#1030)
eb14a86 flambda-backend: Port PR81 from ocaml-jst (ocaml#1024)
131bc12 flambda-backend: Merge ocaml-jst 2022-12-13 (ocaml#1022)
06c189a flambda-backend: Make stack allocation the default (ocaml#1013)
98debd5 flambda-backend: Initial support for value slots not of value kind (ocaml#946)
deb1714 flambda-backend: Add is_last flag to closinfo words (ocaml#938)
d07fce1 flambda-backend: Disable poll insertion in Configure (ocaml#967)
0f1ce0e flambda-backend: Regenerate ocaml/configure autoconf 2.69 (instead of 2.71) (ocaml#1012)
27132d8 flambda-backend: Fix for spurious typing error related to expanding through functor arguments (ocaml#997)
724fb68 flambda-backend: Use `Compilation_unit.t` instead of `Ident.t` for globals (ocaml#871)
396d5b8 flambda-backend: Add a test for frametable setup in natdynlinked libraries (ocaml#983)
b73ab12 flambda-backend: Fix invocation of `caml_shared_startup` in native dynlink (ocaml#980)
7c7d75a flambda-backend: Fix split_default_wrapper which did not trigger anymore with flambda2 (ocaml#970)
8fb75bd flambda-backend: Port ocaml#11727 and ocaml#11732 (ocaml#965)
fdb7987 flambda-backend: Fix include functor issue after 4.14 merge. (ocaml#948)
9745cdb flambda-backend: Print -dprofile/-dtimings output to stdout like 4.12 (ocaml#943)
5f51f21 flambda-backend: Merge pull request ocaml#932 from mshinwell/4.14-upgrade
841687d flambda-backend: Run make alldepend in ocaml/ (ocaml#936)
72a7658 flambda-backend: Remove reformatting changes only in dynlink/dune (preserving PR889 and adjusting to minimise diff)
6d758cd flambda-backend: Revert whitespace changes in dune files, to match upstream
c86bf6e flambda-backend: Remove duplicate tests for polling
971dbeb flambda-backend: Testsuite fixes
32f8356 flambda-backend: Topeval fix for symbols patch
befea01 flambda-backend: Compilation fixes / rectify merge faults
a84543f flambda-backend: Merge ocaml-jst
8e65056 flambda-backend: Merge ocaml-jst
4d70045 flambda-backend: Remove filename from system frametable (amd64) (ocaml#920)
5e57b7d flambda-backend: Bugfix for runtime frame_descr logic for C frames (ocaml#918)
6423d5e flambda-backend: Merge pull request ocaml#914 from mshinwell/merge-ocaml-jst-2022-10-24
ead605c flambda-backend: Add a missing Extract_exception (ocaml#916)
c8f1481 flambda-backend: Resolve conflicts and add specialise/specialised attributes to Builtin_attributes
cf4d0d3 flambda-backend: Merge fixes (ocaml#21)
c2f742f flambda-backend: Re-enable some tests for Flambda2 (ocaml#881)
3d38d13 flambda-backend: Long frames in frametable (ocaml#797)
85aec7b flambda-backend: Add loop attribute to Builtin_attributes
c0f16e3 flambda-backend: Compilation fixes
90dea23 flambda-backend: Merge flambda-backend/main
5acc6ea flambda-backend: Fixes after merge
e501946 flambda-backend: Merge ocaml-jst
115083b flambda-backend: Merge ocaml-jst
9943b2e flambda-backend: Revert "Revert "Transform tail-recursive functions into recursive continuations (ocaml#893)"" (ocaml#909)
ce339f1 flambda-backend: Fix alloc modes and call kinds for overapplications (ocaml#902)
e6a317c flambda-backend: Revert "Transform tail-recursive functions into recursive continuations (ocaml#893)"
853c488 flambda-backend: Transform tail-recursive functions into recursive continuations (ocaml#893)
5a977e4 flambda-backend: Fix missing End_region primitives on switch arms (ocaml#898)
7fa7f9d flambda-backend: Add missing dependencies to Dune files (ocaml#889)
3cd36f0 flambda-backend: Have Lambda `Pgetglobal` and `Psetglobal` take `Compilation_unit.t` (ocaml#896)
7565915 flambda-backend: [@poll error] attribute (ocaml#745)
9eb9448 flambda-backend: Backport the main safepoints PRs (ocaml#740)
689bdda flambda-backend: Add strict mode for ocamldep (ocaml#892)

git-subtree-dir: ocaml
git-subtree-split: b11eea1
EmileTrotignon pushed a commit to EmileTrotignon/ocaml that referenced this pull request Jan 12, 2024
* Create ocamlorg.global library

Directory: src/global
Library Module Name: Ocamlorg
Library Dependency Name: ocamlorg.global
Submodules:
  - Import
  - Url

The goal of this library is to group in a single place stuff used
across several libraries and have a single place for extending Stdlib.

---------
Co-authored-by: Cuihtlauac ALVARADO <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants