Build dylibs, not bundles, on macOS#988
Conversation
Nevermind, I forgot an Output of |
Generally, OCaml creates dynamic libraries in three cases:
* when building bytecode stubs (dllX.so);
* when building a .cmxs plugin;
* when building a .native.so or even .byte.so.
Right now, this results in DLLs on Windows and ELF shared objects on Linux,
all of which can be dynamically loaded (with dlopen() or equivalent) or
linked against (with the -l linker flag or equivalent). However, on macOS,
this is not the case.
macOS has two kinds of dynamic libraries: "bundles" and "dylibs". Prior to
the version 10.4, there have been significant differences between these,
which I will not describe in this commit because 10.4 has long became
irrelevant. After 10.4, there are only two differences:
* rpath is handled slightly differently.
* dylibs can be linked against, with the -l linker flag;
Before this commit, ocamlc/ocamlopt on macOS produce bundles, when using
the .so extension for the output file. After this commit, OCaml on macOS
will produce dylibs, when using the same extension. The rationale is as
follows:
* For bytecode stubs and plugins, the exact structure of which is essentially
an implementation detail of the OCaml runtime, nothing will change because
they can still be dynamically loaded.
* For .native.so and .byte.so objects, there are two changes:
1. The objects can be linked against with the -l flag.
2. The objects can be linked with the -cclib -shared flag, which is
what ocamlbuild and perhaps other buildsystems pass when building
a shared object through -output-obj/-output-complete-obj.
|
I'm perfectly fine with this change. The only reason ocaml use bundles is that, when shared libraries were first introduced in ocaml, this was the only portable way to do it on MacOSX. Of course, this should be tested thoroughly, both for the building/installing and runtime phases. |
| and "0 mod <expr>" in the case when <expr> was a non-constant | ||
| evaluating to zero (Mark Shinwell) | ||
|
|
||
| - MPR#6927, GPR#988: On macOS, when compiling bytecode stubs, plugins, |
|
OK in principle. What about the issue relating to ".so" versus ".dylib"? I tend to think we should follow the platform-specific convention here, which I believe is ".dylib". Idiosyncratic behaviour of OCaml tools isn't likely to help further their adoption, in my opinion. Some of the tools in the Mantis issue (e.g. ocamlbuild, ocamlfind, etc) seem as if they should be easily updated before 4.05 is released. |
There's no real convention. For example, anything you install from homebrew (including OCaml, for that matter) will generally use ".so". macOS-only software will usually stick to ".dylib". Bundles don't have any convention. All in all I don't feel that ".so" will be unexpected, surprising, or confusing here, especially looking as bundles and dylibs nearly equivalent anyway, and the massive fallout (every single makefile for OCaml projects will have to be updated) is not worth it. |
Just a few number on my machine. Note that the list for However that's only for
That doesn't match my experience though. A lot of cross platform stuff I have on my machine uses
However these projects would already have to be updated to use |
There is a lot of projects that special-cases Windows with .exe and .dll, from my cross-compilation experience. |
|
@dbuenzli What you need to count is the number of files called @whitequark I feel you may be exaggerating about "every single Makefile". The ones that are going to break are those that actually build these targets on macOS machines. That must surely be a smaller subset, at least. You mentioned in the Mantis issue about XCode's handling of these files/bundles. Can you confirm whether XCode handles files called ".so" that are actually dylibs correctly, or not? |
Quite a few Makefiles build stubs. |
|
|
I think @dbuenzli 's numbers illustrate that there may be some expectation that |
|
I think that the point here is that shared stub libraries are for use with ocaml only, and are not passed to the system linker. As such, there does not seem to be a strong reason to rename them. And while this would only require changes in the Makefiles for stubs, for me this just means every library I'm maintaining... |
|
@garrigue Does this not affect creation of a normal native shared library for purposes other than bytecode stubs? I suppose we could consider changing it only for those cases, which is indeed when the produced files are more likely to be user-visible. |
I agree that this is less disruptive. However, what are our options here? Consider this:
Frankly, at this point the whole discussion about the extension strikes me as the worst kind of bikeshedding, the one where any choice except of doing nothing has a negative outcome. |
Of course this would be non-sense. We should set |
|
I'm still not really worried about changing the extension wholesale, to be honest. There are plenty of other things that have to be updated in OPAM packages (and elsewhere) between releases, and this just doesn't strike me as a particularly problematic one. I don't think this particular discussion is bikeshedding; as I said above, having different conventions just for OCaml isn't going to further our cause on those platforms. |
|
My shed-related opinion as a long-time mac developer: MacOS by convention uses dylib and MacOS users expect dylib. If as a mac developer I see a "dylib" I understand what to do with it, if I see ".so" there is a moment of pause while I try to figure out if that will work with my other tools. At the other end, at least one of the ancillary tools for OCaml (ocamlbuild, I think?) decides what kind of file to produce by the file extension you ask it for; in the past I believe I've run into confusion because I was trying to ask it for a dylib and it expected me to ask for a .so. Using dylib is more friendly to non-OCaml new users on Macintosh. It is also important to remember OCaml does not exist in isolation. Last fall I attempted to use OCaml in a mixed C/OCaml project, where OCaml was producing a shared library I expected other people to incorporate into their own C programs. The consumer of the shared libraries would not be familiar with OCaml, and if .so is the convention for OCaml on mac they would not be aware of that. For the convenience of my users, I needed to use dylib. I ran into several problems with this project but one was that the OCaml tools at the time emitted .so, and so I had to add an additional step in my make scripts to rename the .so to .dylib. This is minor but these little inconveniences add up. I cannot comment on what existing OCaml users will expect or how disruptive a switch to dylib would be for them. |
|
For what it's worth, I'm fine with ocamlbuild accepting both |
|
After having talked to @mcclure and a few more people on Twitter I have reversed my opinion. It seems that macOS people are divided into two camps: those who are ok with either of so/dylib and those who strongly prefer dylib and consider it the only appropriate choice for the platform. As such I now think that the right way is to change EXT_DLL to dylib. I have also realized that the only way to rewrite Makefiles would be to make them use |
|
There is another OCaml command that creates shared object files: ocamlmklib. Those shared objects contain C stub code and are dynamically linked by ocamlrun. I don't know if this has been considered in this discussion, but I am really not keen to name those shared objects |
|
@whitequark : our messages went past each other, I think. But just to reiterate: I need stronger reasons than "this is what half of a random bunch of Twitters recommend". |
I don't like this condescending attitude. I have put in effort to ask specific people I personally know that have a lot of macOS development experience, much more than me indeed. You have so far simply reiterated what I have said in the very PR description. |
Is "it is what Apple recommends" compelling? Here are the "Dynamic Library Design Guidelines" from Apple's website. They give various guidance on naming dynamic libraries, but they start off with "The filename of a dynamic library normally contains the library’s name with the lib prefix and the .dylib extension". Their dynamic library documentation here uses .dylib throughout, and I do not find any official Apple documentation which suggests .so is allowed. I'm not actually totally sure by what mechanism it is ".so" even works. I have an unconfirmed suspicion ".so" is just an arbitrary three letter string here and the reason it works is dylibs do not in practice require a file extension at all (I did some tests just now with creating and using a mac shared library with the extension ".dll" and this worked just fine). |
Then you were right from the beginning and there is no point in continuing this discussion. |
Arent't these install procedure anyways broken on Windows ? People should be using |
The kernel doesn't care about file extensions at all, it looks at the Mach header only. The only place where it would matter is (non-OCaml) tooling. |
|
@mcclure There used to be a difference, as The fink porting guide has more information on what this meant, but from a fairly old version of OSX. The material differences between the two as described by the guide are:
|
|
@xavierleroy I think the argument for consistency with the standards of the platform is a fairly strong one, especially for platforms like the Mac, where there is probably less tolerance in the community for arbitrary discrepancies than e.g. on Linux. As far as Makefiles go: isn't @dbuenzli right about this, i.e. that we should be using the variable anyway (isn't @shindere 's Unix/Windows Makefile-unification work going to do that)? |
|
We discussed this PR at the developer meeting two days ago, and the consensus was to accept the format change (which is important and non-controversial) and leave the suffix change for later (if/when we reach consensus). |
|
For the suffix change, has the consensus been reached 7 years later? |
|
Not that I know of. A quick search reveals that at least one person is inconvenienced by the use of Does anyone want to propose a PR for changing the extension? With all the work on the makefiles I would expect it to be a lot easier now. |
Well. I'm sure the change will break many Makefiles, including several of mine, that blindly install |
10 participants
Generally, OCaml creates dynamic libraries in three cases:
Right now, this results in DLLs on Windows and ELF shared objects on Linux,
all of which can be dynamically loaded (with dlopen() or equivalent) or
linked against (with the -l linker flag or equivalent). However, on macOS,
this is not the case.
macOS has two kinds of dynamic libraries: "bundles" and "dylibs". Prior to
the version 10.4, there have been significant differences between these,
which I will not describe in this commit because 10.4 has long became
irrelevant. After 10.4, there are only two differences:
Before this commit, ocamlc/ocamlopt on macOS produce bundles, when using
the .so extension for the output file. After this commit, OCaml on macOS
will produce dylibs, when using the same extension. The rationale is as
follows:
an implementation detail of the OCaml runtime, nothing will change because
they can still be dynamically loaded.
what ocamlbuild and perhaps other buildsystems pass when building
a shared object through -output-obj/-output-complete-obj.