I like the change, but I'm less convinced by the ! versions of the operator. My understanding is that you need it to not special-case the way the array-sugar uses unsafe_get sometimes, but the downside is that the bang-using operators do not have a sensible "applied syntax", contrarily to the other.

That sounds quite an interesting extension! Seems that can encode python-like dictionaries using your {}:

let ( .{} ) = Hashtbl.find
let ( .{} <- ) = Hashtbl.update

let () =
  let t = Hashtbl.create 1024 in
  t.{ "foo" } <- 1;
  t.{ "bar" } <- 2;
  t.{ "foo" }

Awesome idea! I look forward to that being implemented in the standard distribution.

Would it be possible to add custom operators ? Following the ocaml convention, something like "start with {, ( or [, end with the corresponding closing symbol". .(| ... |), .[^ ... ^] and so on.

A previous related proposal (for monadic operators rather than index-access operators) was shot down as too complex after we started arguing about which custom operators were reasonable. If I were you, I would first concentrate on a minimum viable proposal, and revisit the "custom operators" idea a few versions later.

@gasche You're probably right, I don't want to start this kind of discussion.

@gasche I agree that the ! version of the operators are less than ideal and even potentially
dangerous. They are here to mirror exactly the current behavior of the -unsafe compiler option.
In presence of this option, the compiler switch the desugaring of a.(b) to Array.unsafe_get
rather than Array.get. I don't see right now a way to implement this behavior without two different
variants of each index operators. However, if this behavior was deprecated, one possibility
would be to add an Unsafe submodule to the concerned modules: In order to emulate the previous
behavior of the -unsafe compiler option, one would only need to open the Array/String/Bytes/Etc.Unsafe module. The granularity would be better but it would imply even more modifications to existing code.

@Drup The problem with adding custom operators is also that it becomes more difficult
to predict the interaction with all the specialized parenthesis, present and future. For instance, what about
[% or [@?

@Octachron I believe you could avoid the unsafe variants by adding a module camlinternalArray to the stdlib which contains declarations like:

external get : 'a array -> 'a = "%array_opt_get"
external set : 'a array -> 'a -> unit = "%array_opt_set"

and exposes them as externals in the interface. By exposing the externals in the interface, they are guaranteed to be inlined when used. This means that the behaviour of the %array_opt_get and %array_opt_set compiler primitives can be controlled by the -unsafe option.

EDIT: Actually, I suppose you are defining .() in Pervasives, so using a camlinternalArray module would place the inlining in the wrong place. Instead .() should itself be exposed as an external by Pervasives. This might affect code which used (module type of Pervasives), but hopefully no one actually does that (and if they do it's their own fault).

@lpw25 Thank you for your suggestion. Using new compiler primitives for the index operators
works. With this, the ! versions of the index operator can be removed.
@hcarty I have updated the documentation.

This new version removes the ! version of the index operators. The -unsafe compiler option
is handled using new compiler primitives for string, array and bigarray index operators.
This new primitives are then transformed to the standard safe/unsafe get/set depending on
the -unsafe compiler option. Consequently, the -unsafe can no longer trigger compiler error
when the unsafe ! operators have been forgotten.

In fact this can break some code of users relying on the orignal 'feature'.
I had, and know at least one other person who used that trick:

module S = struct
  module Bigarray = struct
    module Array1 = struct
      let get x y = Hashtbl.find x y
      let set x y z = Hashtbl.add x y z
    end
  end
end

let t = Hashtbl.create 1
let _ = S.(t.{1} <- 'x'; t.{1})

True. This change would break any code relying on the (undocumented?) original desugaring
to redefine the index operators. However, translations to the new syntax should be simple.

If really needed, I think a ppx extension could even do the translation automatically.

Anyway, this incompatibility should be documented.

The case of python-like hashtable accessors is terrific, can't wait to see it in the official distribution.

This PR have been discussed during the last developers' meeting, where I had the pleasure to be. To sum-up:

• Problem with evaluation time as @thomasblanc commented, must be fixed
• Otherwise reasonable, ok for integration.

Nice to see this is considered for integration!

@bolot thank you for the reports on the various pull request, It's very nice for people who can't go to developers meetings.

@bobot Thanks for the information

I have rebased the pull request and cleaned it a little following the suggestions of @thomasblanc.
The -unsafe option should now produce the expected segmentation fault.

The 4 levels of the bigarray operators is a bit arbitrary. Could we instead expose just:

• 1-tuple => .{} : arity 2 [ +1 for assignment ] operator
• 2 and more tuples => .{,} : arity 2 [ +1 for assignment] operator, the tuple is passed as an array.

and then add a simple optimisation for the %caml_ba_indexop_ref_n primitive to detect the case with 2 or 3 arguments (in a similar way to how the safe/unsafe variants now work).

The only downside I can see is that custom operators wouldn't be able to have optimised versions for 2 and 3 indices, but they already can't optimise for 4 or higher so it doesn't seem too bad. Ideally, inlining and unboxing optimisations would remove the extra cost from array allocation anyway.

I like @lpw25's suggestion because, with that approach, it would be simpler to support the array form for all indexing operators. There's nothing special about the .{} operator at this point.

@lpw25 I agree that the 4 levels of bigarray operators are a bit too arbitrary.
I don't really like to have that many operators but they are necessary for backward compatibility with
Bigarray:

.{} : Bigarray.Array1.t -> ... 
.{,}: Bigarray.Array2.t -> ...

and the Bigarray.Array{n}.t types are abstract.
It is not an ideal situation but I am not completely convinced either by the division between .{} and .{,}. If we were breaking compatibility withBigarray, I would prefer a more uniform solution.

@Octachron If the current "1, 2, 3, array" approach is kept for Bigarray compatibility is there any reason not to extend the same support to the other index operators?

@hcarty The implementation of this extension would be quite straightforward.

However, this feature would interfere a little with the simple version of the .()/.[] operator.
Without this feature,

 a.[x,y] 

and

let p = x, y in 
 a.[p] 

are equivalent. There are not equivalent anymore when .[,] is introduced because

a.[x,y] = (.[,]) a x y

whereas

let p = x,y in 
 a.[p] 
= (.[]) a p

Of course, fixing this problem is easy enough: if .[] takes a tuple in argument, the corresponding
.[,..] operator must be also defined. But this does mean that the complex option leak into the simple one.

Another point is that I have the impression that most of the use of .{,..} could be solved using a mixture of GADT and phantom types.
For instance, in Bigarray, .{} could be implemented as

type z=Nil_zero
type 'a succ = Nil_succ
type ('ph,'a,'b,'c) Genarray.t = ( 'a,'b,'c) Genarray.t
type 'ph n_list = T : z n_list | Cons : int * 'n n_list  -> 'n 'succ n_list
let ( *: ) (x:int) l =  Cons x l
let ( .{} ) a:('ph,'a,'b,'c) indices:'ph n_list   = ...
...

Then a.{ 1 *: 2 *: T } would have more static guaranteed than the current version of Bigarray.

Speaking about Bigarray, there is a way to implement Bigarray with simply (.{}) and (.{,}) without losing the distinction between the Array{n}.t type : we can add a phantom type to Genarray.t and a type witness at the parser level. Something like

(.{} ) : ('type_integer, 'a,'b,'c ) Genarray.t -> 'type_integer * int array -> 'b 

It would complicate the Bigarray module but it should be possible to preserve the current semantic. But like the 4 version variant, we still lose the equivalence a.{x,y} = a.{2-tuple}.

In brief, I see 3 possible stopping points

• Statu-quo: A 4-version operator for .{}, only one version for .[] and .(). The operator .{} is the only one which breaks the equivalence between a.{x,y} and a.{tuple}
• Second option: All three operators implement the 4 versions.
• Third option: All three operators implements the 2 versions .{} and .{,} where the second argument of ".{,}" is a pair (array,type witness to the number of argument).

Personally, I quite like the third option. But it sounds too complex to be integrated in this pull request.
Simultaneously, I am quite neutral concerning the second option, do you have any use case in mind @hcarty?
Anyway, if the need arise in the future, it should be relatively straightforward to extend the current version to one of these two extensions.

You must keep it simple, and I think the statu quo is the best option in this regard. Multi-dimensional bigarrays are by now a historical curiosity, they can stay a bit peculiar.

Sorry for the terse writing above (I was trying to give feedback as soon as possible to the many change proposals that have been discussed, but evidently did a bad job at communicating here).

There is nothing secret about Leo's proposal, except that he prepared some slides/presentation for the meeting that are not online yet ( @lpw25 , could you make them publicly available somewhere?). It is work that originated from an effort to remove the special-casing of float arrays in the current OCaml runtime. You can find some early description of the idea in this comment on a proposal by Alain Frisch related to float arrays. Leo then developed a more complete proposal of what he calls "array data types", whose implementation can be found in his array-data-types branch, where a new type constructor for arrays is introduced with attributes to provoke various kind of specializations (mutable/immutable, unboxing, zero-termination, with the default being generic, non-float-specialized arrays).

The key point of his proposal that is of concern here is to consider these specialized array types as glorified records, where the array-indexing operation .() and .()<-, and a length projection .length have a status similar to record field projections -- in particular, they are disambiguated in a type-directed way, instead of being overloaded by scope as in your proposal.

It is not yet clear that this aspect of the proposal is bound to stay, but I think it is interesting enough (obviously for record projections type-based disambiguation is much more usable than scope-based usage, and as the complex discussion on shadowing warnigns show, local open are no panacea) to give us pause and some time to think harder about where we want to take indexing operators.

Having .() type-disambiguated and .[] scope-disambiguated is a proposal of Leo, and honestly I think it makes no sense. If we find out that the type-directed strategy is indeed better, it will surely be better for .[] as well (think of bytes vs string for example), and conversely if scope-based is a better design in terms of user applicability, then I think this aspect of Leo's proposal should be rejected. We are trying to find a good design, not play politics and make compromises to please the involved parties. (I'm not saying I would be against all hybrid designs, but there would need to be a very strong justification for mixing both strategies.)

We are trying to move to a shorter release cycles, ideally six-month releases. The choice here is not to definitely reject your feature (scope-based user-defined indexing overloading), but rather to not include it in the next release (4.03) and postpone an inclusion choice to 4.04 -- hoping to have advanced in the consideration of these competing designs. Language design is a conservative affair.

When I speak of a branch, I am thinking of having a scoped-indexing-operators branch in the development repository, that would be synchronized with trunk -- so that at any point in time it is easy to build a version of the distribution with the feature enabled, for experimentation or for specialized usage. I'm not asking you to maintain it -- but I'm not promising either that I will personally have the time to do so.

(Let me point out that the Bigarray-related compatibilty woes were not instrumental in taking this decision. Nor are their your responsibility, it's clear that at the time the feature was integrated there was consensus that not special-casing Bigarray was the right choice -- but this was proved wrong by practical experience with trunk compatibility. In fact we had discussed them separately before taking this reversion decision, and had decided to go the conservative way of implicitly opening Bigarray.Operators an .{..} operator was used with none in scope.)

@lpw25 , could you make them publicly available somewhere?

I didn't have any actual slides, but the branch is available at https://github.com/lpw25/ocaml/tree/array-data-types. It doesn't have any documentation to explain the syntax, since I haven't added tests yet, so that may not be very useful for understanding the feature.

Having .() type-disambiguated and .[] scope-disambiguated is a proposal of Leo, and honestly I think it makes no sense.

I disagree. Whilst the current patch doesn't do this yet, it is sufficient to use ".()" for both regular arrays and strings (and bytes etc.), in fact that is a benefit of the proposal: ".()" becomes an explicit projection from an array(-like) block. However, it means that ".()" is always restricted to types which are implemented as array(-like) blocks. Since many other types may usefully have some notion of projection it makes sense to also separately provide an operator that can be over-loaded for those types and it seems reasonable for ".[]" to be that operator.

As a side-note, my array data types proposal -- in common with ordinary labels -- supports the syntax:

foo.FloatArray.(0)

which avoids the need for local open. I think a similar syntax would be useful for ".[]".

I have trouble to understand how you can't do all that with the current proposal + implicits (and an Indexable signature), and without more insight, I don't see how @lpw25's proposal is likely to get integrated before implicits.

I have trouble to understand how you can't do all that with the current proposal + implicits (and an Indexable signature)

It's difficult to explain without fully explaining the proposal -- so best to wait until I've found time to do that.

I don't see how @lpw25's proposal is likely to get integrated before implicits.

The proposal is already mergeable whilst implicits still requires a huge amount of work.

@gasche:

Thanks for taking the time to detail your comment. I have no doubt that you were well meaning in your previous comment. However, communicating decisions without the context of the decision is bound to create some frictions from time to time. If there is any plan to have some publicly available minutes of the developer meeting? It would be very nice; the difference between a secret and an information shared only with a select circle is tenuous at best.

After reading the explanation for Leo proposal, it sounds like a nice feature; that does not cover any of my uses case for custom index operators (e.g dictionary, slice, multidimentionnal array or statistically checked array ) where either the data is not exactly an array block or the index is not a simple integer. In these more complex situations, I still think that it makes more sense to have scope-overloaded index operators. So I agree with @lpw45 that it might make sense to have .() for "true" array type and .[] for indexable type ( and .{} for use cases that match bigarray semantic accidentally ).

We are trying to move to a shorter release cycles, ideally six-month releases. The choice here is not to definitely reject your feature (scope-based user-defined indexing overloading), but rather to not include it in the next release (4.03) and postpone an inclusion choice to 4.04 -- hoping to have advanced in the consideration of these competing designs. Language design is a conservative affair.

The shorter release cycles sounds like a good news. This is also the kind of news for which having minutes would be nice. And like I said before, I understand the thirst for further considerations -- just does not expect me to be perfectly happy if these considerations take place behind closed doors.

When I speak of a branch, I am thinking of having a scoped-indexing-operators branch in the development repository, that would be synchronized with trunk -- so that at any point in time it is easy to build a version of the distribution with the feature enabled, for experimentation or for specialized usage. I'm not asking you to maintain it -- but I'm not promising either that I will personally have the time to do so.

In this context, i.e. limited period and well-defined objective, I will be glad to lend an hand to maintain this branch.

@lpw45:

I agree that the syntax array.Module.{..} is a nice way to avoid both local open and custom ascii art operators. Originally, I was thinking to propose this extended syntax after 4.03. Let's say that I am a little bit wary about such extensions now.

If there is any plan to have some publicly available minutes of the developer meeting?

I am currently in the process of "exporting" the decisions that have been made as publicly-visible comments on the tickets that have been discussed. I did a bit yesterday (obviously I over-favoured quantity over quality of the reports), and hope to finish today. (It's actually quite a bit of work.)

We do need to improve communication on release cycle decisions and such (I'm trying to mention it explicitly in the present reports), but I'm uncomfortable with the idea of exporting minutes for a meeting without having had agreement of the various participants beforehand. For this one meeting I think I'll keep a summary-only approach. (I don't expect to be present in similar meetings in the short- to medium-term future, but maybe @bobot or other participants could take a decision on this. )

but I'm uncomfortable with the idea of exporting minutes for a meeting without having had agreement of the various participants beforehand.

I think a summary of the discussions (rather than full minutes) would be useful in addition to the individual feedback to PRs as it gives a single place people can read about the various discussions. I also agree that the agreement of the participants is needed before releasing such a summary. Perhaps if you wrote a small summary from your notes, and asked on caml-devel for the approval of the participants, they could be posted somewhere (although I'm not sure where the best place would be).

The minutes would also be useful for caml-devel itself since not all core developers could attend the meeting. For a larger diffusion, we would also need to get the green light from non-core devel participants.

@damiendoligez As far as I recall from the recent meeting it was decided that this should not be merged; do I remember correctly? If so we should probably close this.

@mshinwell I have more mixed memories on this discussion, however it would be clearer to close this PR (and #622) in favor of its latest incarnation #1064 .

My notes agree with @Octachron