Adds a .gitignore to ignore all generated files during `make world.opt' by chaudhuri · Pull Request #8 · ocaml/ocaml

chaudhuri · 2014-02-04T17:28:27Z

Since we are trialing patch reviews on Github, maybe this will make git status a bit less useless...

bobot · 2014-02-05T08:54:42Z

.gitignore

Some must be restricted to the toplevel

/configure /ocaml ....

Good point. Thanks. I will make this change later in the day.

gasche · 2014-02-05T09:06:03Z

This is not only about patch reviews: a lot of people have been using git internally to work with OCaml sources before. I think it's therefore nice to have a .gitignore file and am ready to commit this one. I would like to avoid making commit noise about it, though, so I'll let it sit here for a bit before integrating the final version. People of the Git, please try it and make it rock solid in the meantime.

samoht · 2014-02-05T09:10:51Z

@gasche on a related note (ie. adding files not useful for the SVN repo) will you accept patches for Travis CI integration (to check that the proposed patches compile and pass the tests) ?

bobot · 2014-02-05T09:21:32Z

We should take the svn:ignore properties of all the subdirectories. The testsuite create a lot of files. The only problem is to factor the properties. I use these scripts for getting all the svn:ignore in a suitable format for .gitignore:

create_gitignore_aux.sh:

#!/bin/dash -e

REPO=$1
shift
DIR="$@"
DIR="${DIR#.}/"

svn propget svn:ignore "$REPO$DIR" | sed -e "/^\s*$/d" -e "s&^&$DIR&"

and create_gitignore.sh:

#!/bin/dash -e

CREATE_GITIGNORE_AUX=$(dirname $0)/create_gitignore_aux.sh

REPO=$1
find \( -name .git -prune \) -o \( -type d -exec $CREATE_GITIGNORE_AUX $REPO \{\} \; \) | tee .gitignore

Then ./create_gitignore.sh http://caml.inria.fr/svn/ocaml/trunk in a git clone of trunk gives you a big .gitignore.

This evening I can do the work of factorization if nobody has already a good .gitignore.

There is some fun svn:ignore properties in the svn :) :

/testsuite/makefiles/# svn propset -R svn:ignore -F .svnignore .
/testsuite/makefiles/# find . -name .svnignore -print | while read f; do svn propset svn:ignore -F $f `dirname $f`; done

protz · 2014-02-05T09:30:40Z

Why don't we use the output of

git svn show-ignore

? This should contain everything we want.

I've run it in my local git svn version of ocaml and the output is at
http://gallium.inria.fr/~protzenk/ocaml-gitignore

~ jonathan

On Wed 05 Feb 2014 10:21:33 AM CET, bobot wrote:

We should take the svn:ignore properties of all the subdirectories.
The testsuite create a lot of files. The only problem is to factor the
properties. I use these scripts for getting all the svn:ignore in a
suitable format for .gitignore:

create_gitignore_aux.sh:

|#!/bin/dash -e

REPO=$1
shift
DIR="$@"
DIR="${DIR#.}/"

svn propget svn:ignore "$REPO$DIR" | sed -e "/^\s*$/d" -e "s&^&$DIR&"
|

and create_gitignore.sh:

|#!/bin/dash -e

CREATE_GITIGNORE_AUX=$(dirname $0)/create_gitignore_aux.sh

REPO=$1
find ( -name .git -prune ) -o ( -type d -exec $CREATE_GITIGNORE_AUX $REPO {} ; ) | tee .gitignore
|

Then |./create_gitignore.sh http://caml.inria.fr/svn/ocaml/trunk| in a
git clone of trunk gives you a big .gitignore.

This evening I can do the work of factorization if nobody has already
a good .gitignore.

There is some fun svn:ignore properties in the svn :) :

|/testsuite/makefiles/# svn propset -R svn:ignore -F .svnignore .
/testsuite/makefiles/# find . -name .svnignore -print | while read f; do svn propset svn:ignore -F $f dirname $f; done
|

—
Reply to this email directly or view it on GitHub
#8 (comment).

gasche · 2014-02-05T09:32:00Z

@samoht : I won't take a decision alone on this, I'll ask devs later in the week. If you have a .travis-ci file for an ocaml distribution repository somewhere, would you post a link?

chaudhuri · 2014-02-05T09:34:22Z

@protz - ah, thanks, that is exactly the answer I was hoping already existed somewhere, but I didn't think to look in git-svn yesterday. Is it possible to hook git-svn so that on every rebase it auto-generates the .gitignore from git svn show-ignore? If so, a patch is unnecessary.

avsm · 2014-02-05T09:34:43Z

An old one here for Travis: https://github.com/avsm/ocaml/blob/travis/.travis-ci.sh

Note that it includes the ARM cross-compilation mode as well, so we can test non-x86.

Regarding git-svn, it's worth investigating if we ought to perform this addition in the svn->git scripts to avoid polluting upstream too much. https://github.com/ocaml/ocaml.org-scripts/tree/master/vcs-sync/svn-ocaml

Chris00 · 2014-02-05T09:36:12Z

I thought about that too! I think that would be good.

On Wed, 5 Feb 2014 01:10:54 -0800, Thomas Gazagnaire wrote:

@gasche on a related note (ie. adding files not useful for the SVN repo) will
you accept patches for Travis CI integration (to check that the proposed
patches compile and pass the tests) ?

gasche · 2014-02-05T09:42:16Z

I think github-specific things (such as Travis) would definitely be better located in the svn->git script. For git-specific but repo-agnostic features such as "having a .gitignore", I have no strong opinion. On one hand, having it in the core repo makes life better for people that use git svn directly instead of cloning the github/ocaml/ocaml repo. On the other, it's better to auto-generate the .gitignore and automate its update (than sync it manually), and even if I knew how to do it I don't feel like adding git-specific automation to the upstream repo. So maybe having it in the scripts as well is the most convenient choice for now.

chaudhuri · 2014-02-05T09:56:20Z

Is the svn->git export script written using git-svn? If so, it might be
easiest to add @protz's idea above to the export script and have a
github-mirror-specific .gitignore.

How many people do a git svn clone of the main OCaml repo anyway? Surely
that would bring your svn server to a crawl and should therefore be
disallowed or severely rate-limited for non-Gallium people somehow?

I would wager that 99.99% of people following OCaml using git are just
cloning the github mirror.

On Wed, Feb 5, 2014 at 10:42 AM, gasche [email protected] wrote:

I think github-specific things (such as Travis) would definitely be better
located in the svn->git script. For git-specific but repo-agnostic features
such as "having a .gitignore", I have no strong opinion. On one hand,
having it in the core repo makes life better for people that use git svndirectly instead of cloning the github/ocaml/ocaml repo. On the other, it's
better to auto-generate the .gitignore and automate its update (than sync
it manually), and even if I knew how to do it I don't feel like adding
git-specific automation to the upstream repo. So maybe having it in the
scripts as well is the most convenient choice for now.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/8#issuecomment-34150716
.

@protz

… show-ignore' Based on a suggestion by Jonathan Protzenko (@protz).

amirmc · 2014-02-05T13:44:00Z

I'd concur that most people who want a git repo are simply cloning the mirror.
There used to be stats on the number of git clone's but I think github stopped tracking those.
However, you might find the following indicative: The number of forks of the main mirror [1], currently 58, and the traffic stats of the repo [2] -- the referrer sites are interesting.

[1] https://github.com/ocaml/ocaml/network/members
[2] https://github.com/ocaml/ocaml/graphs/traffic

Amir

On 5 Feb 2014, at 09:56, Kaustuv Chaudhuri [email protected] wrote:

Is the svn->git export script written using git-svn? If so, it might be
easiest to add @protz's idea above to the export script and have a
github-mirror-specific .gitignore.

How many people do a git svn clone of the main OCaml repo anyway? Surely
that would bring your svn server to a crawl and should therefore be
disallowed or severely rate-limited for non-Gallium people somehow?

I would wager that 99.99% of people following OCaml using git are just
cloning the github mirror.

On Wed, Feb 5, 2014 at 10:42 AM, gasche [email protected] wrote:

I think github-specific things (such as Travis) would definitely be better
located in the svn->git script. For git-specific but repo-agnostic features
such as "having a .gitignore", I have no strong opinion. On one hand,
having it in the core repo makes life better for people that use git svndirectly instead of cloning the github/ocaml/ocaml repo. On the other, it's
better to auto-generate the .gitignore and automate its update (than sync
it manually), and even if I knew how to do it I don't feel like adding
git-specific automation to the upstream repo. So maybe having it in the
scripts as well is the most convenient choice for now.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/8#issuecomment-34150716
.

—
Reply to this email directly or view it on GitHub.

dsheets · 2014-02-05T13:53:33Z

The traffic stats are restricted to some privileged GitHub group.

amirmc · 2014-02-05T14:11:26Z

Sorry about that. I've copied the data below.

Site                    Views   Unique visitors
github.com              395     81
Google                  262     90
OCaml.org               108     50
t.co                    58      45
caml.inria.fr           30      16
Gallium.inria.fr        28      17
Stackoverflow.com       22      13
Planet.ocamlcore.org    6       4
en.wikipedia.org        5       4
Feedly.com              5       1

bobot · 2014-04-07T14:02:25Z

Ping and summarize:

It seems that the specific svn->git export script is preferred by the commenters. Who wants to do it?
Who has knowledge about travis and could update and integrate the script previously pointed: https://github.com/avsm/ocaml/blob/travis/.travis-ci.sh ?

Or does someone already started working on one of these point?

avsm · 2014-04-07T14:46:00Z

@bobot if someone can give me a patch to the svn->git export script that doesn't cause history to be rewritten, I can take care of the Travis integration.

I suspect it's far easier just to commit to SVN though. There's already a lot of ancillary stuff in the SVN -- would a few new files (.gitignore and .travis-ci.sh) really make that much of a difference?

chaudhuri · 2014-04-07T14:56:30Z

@avsm :

There's already a lot of ancillary stuff in the SVN -- would a few new files [...] really make that much of a difference?

Leaving a .gitignore in the svn repository is contrary to the DRY principle. Every change to svn:ignore will have to be reflected in this file, which can't even be generated by svn itself as it uses git-svn. The most likely outcome is that the file will quickly go out of sync.

On the other hand, if OCaml moves to git in the future, it will have a .gitignore ready to go.

My own feeling is that the latter has a low enough probability that it's unnecessary to prepare for it right now, so the .gitignore should just be generated by the github mirroring script. In other words, I am proposing to close this pull request and handle it elsewhere.

avsm · 2014-04-07T15:29:09Z

I don't think anyone's explained if adding files to git-svn is actually possible without the entire tree getting rewritten and having to delete everyone's existing forks (not an option). I wonder if adding a manual commit and doing a fast-forward merge in the ocaml.org scripts is the best way.

It would be very helpful if someone could experiment with ocaml/ocaml.org-scripts (which implement the git->svn) and give me a patch. I'm a little tied up on other projects at the moment to look at this in detail.

lpw25 · 2014-04-07T18:08:21Z

Leaving a .gitignore in the svn repository is contrary to the DRY principle.

Tons of stuff in the distribution violates the DRY principle. For things that are not very important (e.g. .gitignore) it is fine. Occasionally the .gitignore will get out of sync with the svn:ignore, but that doesn't really matter and is easily resolved.

I can't help but feel we are over-thinking this one.

bactrian · 2014-04-07T18:41:43Z

I agree with Leo.

On 7 Apr 2014, at 19:08, Leo White [email protected] wrote:

Leaving a .gitignore in the svn repository is contrary to the DRY principle.

Tons of stuff in the distribution violates the DRY principle. For things that are not very important (e.g. .gitignore) it is fine. Occasionally the .gitignore will get out of sync with the svn:ignore, but that doesn't really matter and is easily resolved.

I can't help but feel we are over-thinking this one.

—
Reply to this email directly or view it on GitHub.

dbuenzli · 2014-04-07T18:55:52Z

Damned, the bot has a conscience and talks !

avsm · 2014-04-07T21:27:46Z

It is achieving sentience...

bobot · 2014-04-08T07:17:43Z

😄 How many ocaml svn commiters needs to agree for committing one file? 😄

@avsm, for the travis file. Adding some files in every newly imported commit is not easy. We can use git filter-branch: do nothing on older commits and add the file in new commits. But I think it is error prone and that will be cumbersome to recover once the bad commit will be in the wild.

So I kindly ask the ocaml developers to think of the possibility to commit a .travis.yml and a .travis-ci.sh in the trunk svn for streamlining the git mirror and moving on more interesting pull-request which will be automatically tested.

If you, ocaml developers, agree to add these files, we will come up in this pull-request with satisfying version of these files.

And since I'm not speaking about particular merge-request but to you, ocaml developers, directly: Thank you for all the nice work you do on this lovely language ‼️.

chaudhuri · 2014-04-08T07:23:59Z

@bobot, @avsm, etc.: since you seem to want to take this in a different direction, can you open a pull request or issue yourself? I am not sure I need to be receiving notifications for this PR that I do not support any more myself. I am closing this, but feel free to reopen if there is a significant disagreement.

bobot · 2014-04-08T08:34:13Z

Just, at the top of the pull-request, you can click on unsubscribe in order to not receive notifications for this pull-request. Thanks for starting this PR.

avsm · 2014-04-08T08:37:01Z

I've checked with the development group and they have no problem committing these files upstream, so I'll go ahead and do that shortly

gasche · 2014-05-01T09:52:36Z

I just commited a .gitignore using git svn show-ignore as Jonathan suggested. This is not optimal, but better than the statu quo (which is actually a meager hand-made file that Fabrice stealthy committed).

Output the json metrics as soon as we record them rather than at the final step

@inline

23a7f73 flambda-backend: Fix some Debuginfo.t scopes in the frontend (ocaml#248) 33a04a6 flambda-backend: Attempt to shrink the heap before calling the assembler (ocaml#429) 8a36a16 flambda-backend: Fix to allow stage 2 builds in Flambda 2 -Oclassic mode (ocaml#442) d828db6 flambda-backend: Rename -no-extensions flag to -disable-all-extensions (ocaml#425) 68c39d5 flambda-backend: Fix mistake with extension records (ocaml#423) 423f312 flambda-backend: Refactor -extension and -standard flags (ocaml#398) 585e023 flambda-backend: Improved simplification of array operations (ocaml#384) faec6b1 flambda-backend: Typos (ocaml#407) 8914940 flambda-backend: Ensure allocations are initialised, even dead ones (ocaml#405) 6b58001 flambda-backend: Move compiler flag -dcfg out of ocaml/ subdirectory (ocaml#400) 4fd57cf flambda-backend: Use ghost loc for extension to avoid expressions with overlapping locations (ocaml#399) 8d993c5 flambda-backend: Let's fix instead of reverting flambda_backend_args (ocaml#396) d29b133 flambda-backend: Revert "Move flambda-backend specific flags out of ocaml/ subdirectory (ocaml#382)" (ocaml#395) d0cda93 flambda-backend: Revert ocaml#373 (ocaml#393) 1c6eee1 flambda-backend: Fix "make check_all_arches" in ocaml/ subdirectory (ocaml#388) a7960dd flambda-backend: Move flambda-backend specific flags out of ocaml/ subdirectory (ocaml#382) bf7b1a8 flambda-backend: List and Array Comprehensions (ocaml#147) f2547de flambda-backend: Compile more stdlib files with -O3 (ocaml#380) 3620c58 flambda-backend: Four small inliner fixes (ocaml#379) 2d165d2 flambda-backend: Regenerate ocaml/configure 3838b56 flambda-backend: Bump Menhir to version 20210419 (ocaml#362) 43c14d6 flambda-backend: Re-enable -flambda2-join-points (ocaml#374) 5cd2520 flambda-backend: Disable inlining of recursive functions by default (ocaml#372) e98b277 flambda-backend: Import ocaml#10736 (stack limit increases) (ocaml#373) 82c8086 flambda-backend: Use hooks for type tree and parse tree (ocaml#363) 33bbc93 flambda-backend: Fix parsecmm.mly in ocaml subdirectory (ocaml#357) 9650034 flambda-backend: Right-to-left evaluation of arguments of String.get and friends (ocaml#354) f7d3775 flambda-backend: Revert "Magic numbers" (ocaml#360) 0bd2fa6 flambda-backend: Add [@inline ready] attribute and remove [@inline hint] (not [@inlined hint]) (ocaml#351) cee74af flambda-backend: Ensure that functions are evaluated after their arguments (ocaml#353) 954be59 flambda-backend: Bootstrap dd5c299 flambda-backend: Change prefix of all magic numbers to avoid clashes with upstream. c2b1355 flambda-backend: Fix wrong shift generation in Cmm_helpers (ocaml#347) 739243b flambda-backend: Add flambda_oclassic attribute (ocaml#348) dc9b7fd flambda-backend: Only speculate during inlining if argument types have useful information (ocaml#343) aa190ec flambda-backend: Backport fix from PR#10719 (ocaml#342) c53a574 flambda-backend: Reduce max inlining depths at -O2 and -O3 (ocaml#334) a2493dc flambda-backend: Tweak error messages in Compenv. 1c7b580 flambda-backend: Change Name_abstraction to use a parameterized type (ocaml#326) 07e0918 flambda-backend: Save cfg to file (ocaml#257) 9427a8d flambda-backend: Make inlining parameters more aggressive (ocaml#332) fe0610f flambda-backend: Do not cache young_limit in a processor register (upstream PR 9876) (ocaml#315) 56f28b8 flambda-backend: Fix an overflow bug in major GC work computation (ocaml#310) 8e43a49 flambda-backend: Cmm invariants (port upstream PR 1400) (ocaml#258) e901f16 flambda-backend: Add attributes effects and coeffects (ocaml#18) aaa1cdb flambda-backend: Expose Flambda 2 flags via OCAMLPARAM (ocaml#304) 62db54f flambda-backend: Fix freshening substitutions 57231d2 flambda-backend: Evaluate signature substitutions lazily (upstream PR 10599) (ocaml#280) a1a07de flambda-backend: Keep Sys.opaque_identity in Cmm and Mach (port upstream PR 9412) (ocaml#238) faaf149 flambda-backend: Rename Un_cps -> To_cmm (ocaml#261) ecb0201 flambda-backend: Add "-dcfg" flag to ocamlopt (ocaml#254) 32ec58a flambda-backend: Bypass Simplify (ocaml#162) bd4ce4a flambda-backend: Revert "Semaphore without probes: dummy notes (ocaml#142)" (ocaml#242) c98530f flambda-backend: Semaphore without probes: dummy notes (ocaml#142) c9b6a04 flambda-backend: Remove hack for .depend from runtime/dune (ocaml#170) 6e5d4cf flambda-backend: Build and install Semaphore (ocaml#183) 924eb60 flambda-backend: Special constructor for %sys_argv primitive (ocaml#166) 2ac6334 flambda-backend: Build ocamldoc (ocaml#157) c6f7267 flambda-backend: Add -mbranches-within-32B to major_gc.c compilation (where supported) a99fdee flambda-backend: Merge pull request ocaml#10195 from stedolan/mark-prefetching bd72dcb flambda-backend: Prefetching optimisations for sweeping (ocaml#9934) 27fed7e flambda-backend: Add missing index param for Obj.field (ocaml#145) cd48b2f flambda-backend: Fix camlinternalOO at -O3 with Flambda 2 (ocaml#132) 9d85430 flambda-backend: Fix testsuite execution (ocaml#125) ac964ca flambda-backend: Comment out `[@inlined]` annotation. (ocaml#136) ad4afce flambda-backend: Fix magic numbers (test suite) (ocaml#135) 9b033c7 flambda-backend: Disable the comparison of bytecode programs (`ocamltest`) (ocaml#128) e650abd flambda-backend: Import flambda2 changes (`Asmpackager`) (ocaml#127) 14dcc38 flambda-backend: Fix error with Record_unboxed (bug in block kind patch) (ocaml#119) 2d35761 flambda-backend: Resurrect [@inline never] annotations in camlinternalMod (ocaml#121) f5985ad flambda-backend: Magic numbers for cmx and cmxa files (ocaml#118) 0e8b9f0 flambda-backend: Extend conditions to include flambda2 (ocaml#115) 99870c8 flambda-backend: Fix Translobj assertions for Flambda 2 (ocaml#112) 5106317 flambda-backend: Minor fix for "lazy" compilation in Matching with Flambda 2 (ocaml#110) dba922b flambda-backend: Oclassic/O2/O3 etc (ocaml#104) f88af3e flambda-backend: Wire in the remaining Flambda 2 flags (ocaml#103) 678d647 flambda-backend: Wire in the Flambda 2 inlining flags (ocaml#100) 1a8febb flambda-backend: Formatting of help text for some Flambda 2 options (ocaml#101) 9ae1c7a flambda-backend: First set of command-line flags for Flambda 2 (ocaml#98) bc0bc5e flambda-backend: Add config variables flambda_backend, flambda2 and probes (ocaml#99) efb8304 flambda-backend: Build our own ocamlobjinfo from tools/objinfo/ at the root (ocaml#95) d2cfaca flambda-backend: Add mutability annotations to Pfield etc. (ocaml#88) 5532555 flambda-backend: Lambda block kinds (ocaml#86) 0c597ba flambda-backend: Revert VERSION, etc. back to 4.12.0 (mostly reverts 822d0a0 from upstream 4.12) (ocaml#93) 037c3d0 flambda-backend: Float blocks 7a9d190 flambda-backend: Allow --enable-middle-end=flambda2 etc (ocaml#89) 9057474 flambda-backend: Root scanning fixes for Flambda 2 (ocaml#87) 08e02a3 flambda-backend: Ensure that Lifthenelse has a boolean-valued condition (ocaml#63) 77214b7 flambda-backend: Obj changes for Flambda 2 (ocaml#71) ecfdd72 flambda-backend: Cherry-pick 9432cfdadb043a191b414a2caece3e4f9bbc68b7 (ocaml#84) d1a4396 flambda-backend: Add a `returns` field to `Cmm.Cextcall` (ocaml#74) 575dff5 flambda-backend: CMM traps (ocaml#72) 8a87272 flambda-backend: Remove Obj.set_tag and Obj.truncate (ocaml#73) d9017ae flambda-backend: Merge pull request ocaml#80 from mshinwell/fb-backport-pr10205 3a4824e flambda-backend: Backport PR#10205 from upstream: Avoid overwriting closures while initialising recursive modules f31890e flambda-backend: Install missing headers of ocaml/runtime/caml (ocaml#77) 83516f8 flambda-backend: Apply node created for probe should not be annotated as tailcall (ocaml#76) bc430cb flambda-backend: Add Clflags.is_flambda2 (ocaml#62) ed87247 flambda-backend: Preallocation of blocks in Translmod for value let rec w/ flambda2 (ocaml#59) a4b04d5 flambda-backend: inline never on Gc.create_alarm (ocaml#56) cef0bb6 flambda-backend: Config.flambda2 (ocaml#58) ff0e4f7 flambda-backend: Pun labelled arguments with type constraint in function applications (ocaml#53) d72c5fb flambda-backend: Remove Cmm.memory_chunk.Double_u (ocaml#42) 9d34d99 flambda-backend: Install missing artifacts 10146f2 flambda-backend: Add ocamlcfg (ocaml#34) 819d38a flambda-backend: Use OC_CFLAGS, OC_CPPFLAGS, and SHAREDLIB_CFLAGS for foreign libs (ocaml#30) f98b564 flambda-backend: Pass -function-sections iff supported. (ocaml#29) e0eef5e flambda-backend: Bootstrap (ocaml#11 part 2) 17374b4 flambda-backend: Add [@@Builtin] attribute to Primitives (ocaml#11 part 1) 85127ad flambda-backend: Add builtin, effects and coeffects fields to Cextcall (ocaml#12) b670bcf flambda-backend: Replace tuple with record in Cextcall (ocaml#10) db451b5 flambda-backend: Speedups in Asmlink (ocaml#8) 2fe489d flambda-backend: Cherry-pick upstream PR#10184 from upstream, dynlink invariant removal (rev 3dc3cd7 upstream) d364bfa flambda-backend: Local patch against upstream: enable function sections in the Dune build 886b800 flambda-backend: Local patch against upstream: remove Raw_spacetime_lib (does not build with -m32) 1a7db7c flambda-backend: Local patch against upstream: make dune ignore ocamldoc/ directory e411dd3 flambda-backend: Local patch against upstream: remove ocaml/testsuite/tests/tool-caml-tex/ 1016d03 flambda-backend: Local patch against upstream: remove ocaml/dune-project and ocaml/ocaml-variants.opam 93785e3 flambda-backend: To upstream: export-dynamic for otherlibs/dynlink/ via the natdynlinkops files (still needs .gitignore + way of generating these files) 63db8c1 flambda-backend: To upstream: stop using -O3 in otherlibs/Makefile.otherlibs.common eb2f1ed flambda-backend: To upstream: stop using -O3 for dynlink/ 6682f8d flambda-backend: To upstream: use flambda_o3 attribute instead of -O3 in the Makefile for systhreads/ de197df flambda-backend: To upstream: renamed ocamltest_unix.xxx files for dune bf3773d flambda-backend: To upstream: dune build fixes (depends on previous to-upstream patches) 6fbc80e flambda-backend: To upstream: refactor otherlibs/dynlink/, removing byte/ and native/ 71a03ef flambda-backend: To upstream: fix to Ocaml_modifiers in ocamltest 686d6e3 flambda-backend: To upstream: fix dependency problem with Instruct c311155 flambda-backend: To upstream: remove threadUnix 52e6e78 flambda-backend: To upstream: stabilise filenames used in backtraces: stdlib/, otherlibs/systhreads/, toplevel/toploop.ml 7d08e0e flambda-backend: To upstream: use flambda_o3 attribute in stdlib 403b82e flambda-backend: To upstream: flambda_o3 attribute support (includes bootstrap) 65032b1 flambda-backend: To upstream: use nolabels attribute instead of -nolabels for otherlibs/unix/ f533fad flambda-backend: To upstream: remove Compflags, add attributes, etc. 49fc1b5 flambda-backend: To upstream: Add attributes and bootstrap compiler a4b9e0d flambda-backend: Already upstreamed: stdlib capitalisation patch 4c1c259 flambda-backend: ocaml#9748 from xclerc/share-ev_defname (cherry-pick 3e937fc) 00027c4 flambda-backend: permanent/default-to-best-fit (cherry-pick 64240fd) 2561dd9 flambda-backend: permanent/reraise-by-default (cherry-pick 50e9490) c0aa4f4 flambda-backend: permanent/gc-tuning (cherry-pick e9d6d2f) git-subtree-dir: ocaml git-subtree-split: 23a7f73

da6ff04 Accept [@ocaml.local] without -extension, and move autogenerated attrs to [@extension.local] (ocaml#9) 30ce67d Improve inclusion error messages for [@local_opt] (ocaml#10) f925a62 Remove some uneeded mode variables (ocaml#8) dec721c Local solver speedups (ocaml#7) e9afc49 Fix check_all_arches build 0b9b32a Fix i386 build a515093 Merge flambda-backend changes git-subtree-dir: ocaml git-subtree-split: da6ff04

Bigarrays are printed in the toplevel as `<abstr>`, but ObjTbl.mem computes a hash. However bigarrays are uninitialized when allocated, so this seems to be a genuine bug: ``` ==133712==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x4e6d11 in caml_ba_hash /var/home/edwin/git/ocaml/runtime/bigarray.c:486:45 #1 0x52474a in caml_hash /var/home/edwin/git/ocaml/runtime/hash.c:251:35 ocaml#2 0x599ebf in caml_interprete /var/home/edwin/git/ocaml/runtime/interp.c:1065:14 ocaml#3 0x5a909a in caml_main /var/home/edwin/git/ocaml/runtime/startup_byt.c:575:9 ocaml#4 0x540ccb in main /var/home/edwin/git/ocaml/runtime/main.c:37:3 ocaml#5 0x7f0910abb087 in __libc_start_call_main (/lib64/libc.so.6+0x2a087) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) ocaml#6 0x7f0910abb14a in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2a14a) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) ocaml#7 0x441804 in _start (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x441804) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) Uninitialized value was created by a heap allocation #0 0x47d306 in malloc (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x47d306) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) #1 0x4e7960 in caml_ba_alloc /var/home/edwin/git/ocaml/runtime/bigarray.c:246:12 ocaml#2 0x4e801f in caml_ba_create /var/home/edwin/git/ocaml/runtime/bigarray.c:673:10 ocaml#3 0x59b8fc in caml_interprete /var/home/edwin/git/ocaml/runtime/interp.c:1058:14 ocaml#4 0x5a909a in caml_main /var/home/edwin/git/ocaml/runtime/startup_byt.c:575:9 ocaml#5 0x540ccb in main /var/home/edwin/git/ocaml/runtime/main.c:37:3 ocaml#6 0x7f0910abb087 in __libc_start_call_main (/lib64/libc.so.6+0x2a087) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) ocaml#7 0x7f0910abb14a in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2a14a) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) ocaml#8 0x441804 in _start (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x441804) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) SUMMARY: MemorySanitizer: use-of-uninitialized-value /var/home/edwin/git/ocaml/runtime/bigarray.c:486:45 in caml_ba_hash ``` Suppress it for now until a solution is found. The testsuite passes now: ``` Summary: 1334 tests passed 103 tests skipped 0 tests failed 0 tests not started (parent test skipped or failed) 0 unexpected errors 1437 tests considered ``` Signed-off-by: Edwin Török <[email protected]>

Bigarrays are printed in the toplevel as `<abstr>`, but ObjTbl.mem computes a hash. However bigarrays are uninitialized when allocated, so this seems to be a genuine bug: ``` ==133712==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x4e6d11 in caml_ba_hash /var/home/edwin/git/ocaml/runtime/bigarray.c:486:45 #1 0x52474a in caml_hash /var/home/edwin/git/ocaml/runtime/hash.c:251:35 ocaml#2 0x599ebf in caml_interprete /var/home/edwin/git/ocaml/runtime/interp.c:1065:14 ocaml#3 0x5a909a in caml_main /var/home/edwin/git/ocaml/runtime/startup_byt.c:575:9 ocaml#4 0x540ccb in main /var/home/edwin/git/ocaml/runtime/main.c:37:3 ocaml#5 0x7f0910abb087 in __libc_start_call_main (/lib64/libc.so.6+0x2a087) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) ocaml#6 0x7f0910abb14a in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2a14a) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) ocaml#7 0x441804 in _start (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x441804) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) Uninitialized value was created by a heap allocation #0 0x47d306 in malloc (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x47d306) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) #1 0x4e7960 in caml_ba_alloc /var/home/edwin/git/ocaml/runtime/bigarray.c:246:12 ocaml#2 0x4e801f in caml_ba_create /var/home/edwin/git/ocaml/runtime/bigarray.c:673:10 ocaml#3 0x59b8fc in caml_interprete /var/home/edwin/git/ocaml/runtime/interp.c:1058:14 ocaml#4 0x5a909a in caml_main /var/home/edwin/git/ocaml/runtime/startup_byt.c:575:9 ocaml#5 0x540ccb in main /var/home/edwin/git/ocaml/runtime/main.c:37:3 ocaml#6 0x7f0910abb087 in __libc_start_call_main (/lib64/libc.so.6+0x2a087) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) ocaml#7 0x7f0910abb14a in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2a14a) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) ocaml#8 0x441804 in _start (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x441804) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) SUMMARY: MemorySanitizer: use-of-uninitialized-value /var/home/edwin/git/ocaml/runtime/bigarray.c:486:45 in caml_ba_hash ``` The hashing is only needed to avoid recursion, skip it when the OCaml value doesn't contain further nested OCaml values, i.e. when it wouldn't be scanned by the GC. The testsuite passes now: ``` Summary: 1335 tests passed 102 tests skipped 0 tests failed 0 tests not started (parent test skipped or failed) 0 unexpected errors 1437 tests considered ``` Suggested-by: Gabriel Scherer <[email protected]> Signed-off-by: Edwin Török <[email protected]>

The toplevel printer detects cycles by keeping a hashtable of values that it has already traversed. However, some OCaml runtime types (at least bigarrays) may be partially uninitialized, and hashing them at arbitrary program points may read uninitialized memory. In particular, the OCaml testsuite fails when running with a memory-sanitizer enabled, as bigarray printing results in reads to uninitialized memory: ``` ==133712==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x4e6d11 in caml_ba_hash /var/home/edwin/git/ocaml/runtime/bigarray.c:486:45 #1 0x52474a in caml_hash /var/home/edwin/git/ocaml/runtime/hash.c:251:35 ocaml#2 0x599ebf in caml_interprete /var/home/edwin/git/ocaml/runtime/interp.c:1065:14 ocaml#3 0x5a909a in caml_main /var/home/edwin/git/ocaml/runtime/startup_byt.c:575:9 ocaml#4 0x540ccb in main /var/home/edwin/git/ocaml/runtime/main.c:37:3 ocaml#5 0x7f0910abb087 in __libc_start_call_main (/lib64/libc.so.6+0x2a087) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) ocaml#6 0x7f0910abb14a in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2a14a) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) ocaml#7 0x441804 in _start (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x441804) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) Uninitialized value was created by a heap allocation #0 0x47d306 in malloc (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x47d306) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) #1 0x4e7960 in caml_ba_alloc /var/home/edwin/git/ocaml/runtime/bigarray.c:246:12 ocaml#2 0x4e801f in caml_ba_create /var/home/edwin/git/ocaml/runtime/bigarray.c:673:10 ocaml#3 0x59b8fc in caml_interprete /var/home/edwin/git/ocaml/runtime/interp.c:1058:14 ocaml#4 0x5a909a in caml_main /var/home/edwin/git/ocaml/runtime/startup_byt.c:575:9 ocaml#5 0x540ccb in main /var/home/edwin/git/ocaml/runtime/main.c:37:3 ocaml#6 0x7f0910abb087 in __libc_start_call_main (/lib64/libc.so.6+0x2a087) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) ocaml#7 0x7f0910abb14a in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2a14a) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) ocaml#8 0x441804 in _start (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x441804) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) SUMMARY: MemorySanitizer: use-of-uninitialized-value /var/home/edwin/git/ocaml/runtime/bigarray.c:486:45 in caml_ba_hash ``` The only use of hashing in genprintval is to avoid cycles, that is, it is only useful for OCaml values that contain other OCaml values (including possibly themselves). Bigarrays cannot introduce cycles, and they are always printed as "<abstr>" anyway. The present commit proposes to be more conservative in which values are hashed by the cycle detector to avoid this issue: we skip hashing any value with tag above No_scan_tag -- which may not contain any OCaml values. Suggested-by: Gabriel Scherer <[email protected]> Signed-off-by: Edwin Török <[email protected]>

The toplevel printer detects cycles by keeping a hashtable of values that it has already traversed. However, some OCaml runtime types (at least bigarrays) may be partially uninitialized, and hashing them at arbitrary program points may read uninitialized memory. In particular, the OCaml testsuite fails when running with a memory-sanitizer enabled, as bigarray printing results in reads to uninitialized memory: ``` ==133712==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x4e6d11 in caml_ba_hash /var/home/edwin/git/ocaml/runtime/bigarray.c:486:45 #1 0x52474a in caml_hash /var/home/edwin/git/ocaml/runtime/hash.c:251:35 #2 0x599ebf in caml_interprete /var/home/edwin/git/ocaml/runtime/interp.c:1065:14 #3 0x5a909a in caml_main /var/home/edwin/git/ocaml/runtime/startup_byt.c:575:9 #4 0x540ccb in main /var/home/edwin/git/ocaml/runtime/main.c:37:3 #5 0x7f0910abb087 in __libc_start_call_main (/lib64/libc.so.6+0x2a087) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) #6 0x7f0910abb14a in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2a14a) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) #7 0x441804 in _start (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x441804) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) Uninitialized value was created by a heap allocation #0 0x47d306 in malloc (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x47d306) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) #1 0x4e7960 in caml_ba_alloc /var/home/edwin/git/ocaml/runtime/bigarray.c:246:12 #2 0x4e801f in caml_ba_create /var/home/edwin/git/ocaml/runtime/bigarray.c:673:10 #3 0x59b8fc in caml_interprete /var/home/edwin/git/ocaml/runtime/interp.c:1058:14 #4 0x5a909a in caml_main /var/home/edwin/git/ocaml/runtime/startup_byt.c:575:9 #5 0x540ccb in main /var/home/edwin/git/ocaml/runtime/main.c:37:3 #6 0x7f0910abb087 in __libc_start_call_main (/lib64/libc.so.6+0x2a087) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) #7 0x7f0910abb14a in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x2a14a) (BuildId: 8f53abaad945a669f2bdcd25f471d80e077568ef) #8 0x441804 in _start (/var/home/edwin/git/ocaml/runtime/ocamlrun+0x441804) (BuildId: 7a60eef57e1c2baf770bc38d10d6c227e60ead37) SUMMARY: MemorySanitizer: use-of-uninitialized-value /var/home/edwin/git/ocaml/runtime/bigarray.c:486:45 in caml_ba_hash ``` The only use of hashing in genprintval is to avoid cycles, that is, it is only useful for OCaml values that contain other OCaml values (including possibly themselves). Bigarrays cannot introduce cycles, and they are always printed as "<abstr>" anyway. The present commit proposes to be more conservative in which values are hashed by the cycle detector to avoid this issue: we skip hashing any value with tag above No_scan_tag -- which may not contain any OCaml values. Suggested-by: Gabriel Scherer <[email protected]> Signed-off-by: Edwin Török <[email protected]> Co-authored-by: Edwin Török <[email protected]>

Adds a .gitignore to ignore all generated files during `make world.opt'

16d6e2d

bobot reviewed Feb 5, 2014
View reviewed changes

Replace the manually generated .gitignore with the output of 'git svn…

c3f8d19

… show-ignore' Based on a suggestion by Jonathan Protzenko (@protz).

chaudhuri closed this Apr 8, 2014

poechsel pushed a commit to poechsel/ocaml that referenced this pull request Jun 30, 2021

Speedups in Asmlink (ocaml#8)

9cf1250

poechsel pushed a commit to poechsel/ocaml that referenced this pull request Jul 2, 2021

Speedups in Asmlink (ocaml#8)

a1b9b81

stedolan pushed a commit to stedolan/ocaml that referenced this pull request Dec 13, 2021

flambda-backend: Speedups in Asmlink (ocaml#8)

db451b5

punchagan pushed a commit to punchagan/ocaml that referenced this pull request Mar 7, 2022

Merge pull request ocaml#8 from punchagan/cb-stream-json-output

892102a

Output the json metrics as soon as we record them rather than at the final step

stedolan added a commit to stedolan/ocaml that referenced this pull request Mar 22, 2022

Remove some uneeded mode variables (ocaml#8)

f925a62

andreas-schwab mentioned this pull request Nov 28, 2022

ocamlopt.opt crashes from stack overflow while building coccinelle on RISC-V #11765

Closed

Conversation

chaudhuri commented Feb 4, 2014

Uh oh!

bobot Feb 5, 2014

Choose a reason for hiding this comment

Uh oh!

chaudhuri Feb 5, 2014

Choose a reason for hiding this comment

Uh oh!

gasche commented Feb 5, 2014

Uh oh!

samoht commented Feb 5, 2014

Uh oh!

bobot commented Feb 5, 2014

Uh oh!

protz commented Feb 5, 2014

Uh oh!

gasche commented Feb 5, 2014

Uh oh!

chaudhuri commented Feb 5, 2014

Uh oh!

avsm commented Feb 5, 2014

Uh oh!

Chris00 commented Feb 5, 2014

Uh oh!

gasche commented Feb 5, 2014

Uh oh!

chaudhuri commented Feb 5, 2014

Uh oh!

amirmc commented Feb 5, 2014

Uh oh!

dsheets commented Feb 5, 2014

Uh oh!

amirmc commented Feb 5, 2014

Uh oh!

bobot commented Apr 7, 2014

Uh oh!

avsm commented Apr 7, 2014

Uh oh!

chaudhuri commented Apr 7, 2014

Uh oh!

avsm commented Apr 7, 2014

Uh oh!

lpw25 commented Apr 7, 2014

Uh oh!

bactrian commented Apr 7, 2014

Uh oh!

dbuenzli commented Apr 7, 2014

Uh oh!

avsm commented Apr 7, 2014

Uh oh!

bobot commented Apr 8, 2014

Uh oh!

chaudhuri commented Apr 8, 2014

Uh oh!

bobot commented Apr 8, 2014

Uh oh!

avsm commented Apr 8, 2014

Uh oh!

gasche commented May 1, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants