Skip to content

Conversation

@lrytz
Copy link
Member

@lrytz lrytz commented Apr 12, 2016

No description provided.

JUnit 4 does not support running `@Test` methods defined as default
methods in parent interfaces. JUnit 5 will, but is not yet available.

Currently scalac emits a forwarder to every trait method inherited by
a class, so tests are correctly executed. The fix for SD-98 will change
this.
@sjrd
Copy link
Member

sjrd commented Apr 12, 2016

I did not really read/understand the changes, but I'm guessing this could cause the opposite "problem" in Scala.js: that classes (symbols) read from classfiles do have the synthetic parents, but that symbols being compiled do not have the synthetic parents at the time Scala.js runs.

The Scala.js IR does not have the restriction that super calls may only call directly inherited interfaces, so the Scala.js back-end would not add synthetic parents to fix the problem.

I'm not exactly sure whether what I'm saying makes any sense, but maybe it will trigger some thought.

@lrytz
Copy link
Member Author

lrytz commented Apr 12, 2016

classes (symbols) read from classfiles do have the synthetic parents

Nothing has changed for symbols and frontend types. the additional interfaces only show up in ClassBTypes, which scala.js is probably not using at all.

we have two ways of filling up the details of a ClassBType: either we get the information (parents, flags, etc) from the frontend symbol or from the classfile. to ensure we have all the necessary information in the latter case, some metadata is saved in the classfile in the InlineInfoAttribute, for example what methods have the @inline annotation. The goal is basically to get a consistent ClassBType without requiring the corresponding symbol, because it's not always straightforward to get the correct symbol for a given class internal name (name mangling, synthetic classes, etc).

@sjrd
Copy link
Member

sjrd commented Apr 12, 2016

Nothing has changed for symbols and frontend types. the additional interfaces only show up in ClassBTypes, which scala.js is probably not using at all.

OK then my concern is moot. Scala.js does not use ClassBTypes, indeed. Thanks for the explanation.

lrytz added 2 commits April 12, 2016 12:01
In most cases when a class inherits a concrete method from a trait we
don't need to generate a forwarder to the default method in the class.

t5148 is moved to pos as it compiles without error now. the error
message ("missing or invalid dependency") is still tested by t6440b.
It tests an internal debugging tool which does not appear to work as
intented. If anyone can compile and run that test and get an output
that looks like the check file, I'd be interested to know. Origins does
not seem to support the kind of stack traces that scalac currently
emits.
@lrytz lrytz force-pushed the traitParents branch 4 times, most recently from 748c5e9 to f36aa1c Compare April 13, 2016 13:28
@lrytz
Copy link
Member Author

lrytz commented Apr 13, 2016

here's an observation while working on this PR: there's no good way to generate a valid tree for a static method defined in scala.

for comparison, the selection of a java-defined static method A.f is Select(owner, "f"), owner.symbol is the companion ModuleSymbol for A.

assume we have a static method in a scala class:

  • if the owner is an ordinary ClassSymbol c, Select(c.thisType, "m") is not really valid as it's not a selection on the instance. a selection on the companion (if it exists) is also not valid because the static member is in the class, not the companion - for the backend, a member of a companion object would end up in the companion's ModuleClass.
  • if the owner is a ModuleClassSymbol we can encode it as a selection on the ModuleSymbol, but that is also inconsistent: the member is static in the module class, and we don't need to load the module to access it.

in any case, the backend will identify the selection of a static method and not generate any bytecode for the Select's qualifier.

these are not blockers, just a little weird. we should keep them in mind while adding more static methods (https://issues.scala-lang.org/browse/SI-9390)

@lrytz lrytz force-pushed the traitParents branch 3 times, most recently from 057309d to 6fd8275 Compare April 14, 2016 09:13
if (sym == definitions.Array_clone) {
// Special-case Array.clone, introduced in 36ef60e. The goal is to generate this
// call as "[I.clone" instead of "java/lang/Object.clone" (which would also work).
// Not sure why this is useful, maybe for performance. The commit message says
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's because Object.clone() is protected. I guess trying to call it would be an illegal access on the JVM.

Copy link
Member Author

@lrytz lrytz Apr 15, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested without the special case:

object Test {
  def foo(a: Array[Int]) = a.clone()
  def main(args: Array[String]): Unit = {
    println(foo(Array(1,2,3)).toList)
  }
}

gives

public final class Test$ {
  public foo([I)[I
    ALOAD 1
    INVOKEVIRTUAL java/lang/Object.clone ()Ljava/lang/Object;
    CHECKCAST [I
    ARETURN
}

and it runs fine. but i'd have to check the spec of protected again..

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found this part of the JVM spec hard to follow. clone is overriden as public in each of the Array classes (which also implement Clonable). Seems like its fine to refer the the protected method in Object in the invokevirtual in this case.

Toying around with this via reflection:

scala> java.lang.invoke.MethodHandles.publicLookup.findVirtual(classOf[Object], "clone", java.lang.invoke.MethodType.methodType(classOf[Object]))
java.lang.IllegalAccessException: member is protected: java.lang.Object.clone()Object/invokeVirtual, from java.lang.Object/public
  at java.lang.invoke.MemberName.makeAccessException(MemberName.java:852)

scala> def lookupOf(cls: Class[_]) = { import java.lang.invoke._; val constructor = classOf[MethodHandles.Lookup].getDeclaredConstructors.head; constructor.setAccessible(true); constructor.newInstance(cls).asInstanceOf[MethodHandles.Lookup] }
lookupOf: (cls: Class[_])java.lang.invoke.MethodHandles.Lookup

scala> lookupOf(classOf[Array[Int]]).findVirtual(classOf[Object], "clone", java.lang.invoke.MethodType.methodType(classOf[Object]))
res4: java.lang.invoke.MethodHandle = MethodHandle(int[])Object

Perhaps related:

https://bugs.openjdk.java.net/browse/JDK-4750641
https://bugs.openjdk.java.net/browse/JDK-4962591

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

@lrytz lrytz Apr 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked again the JVM spec

Access Control: protected method R, declared in C, is accessible to a class or interface D if

  • D is a subclass of C, and
  • R must contain a symbolic reference to a class T, such that T is either a subclass of D, a superclass of D, or D itself

"symbolic reference to class T" means the receiver class in the invocation instruction.

But there's more: "This discussion omits a related restriction [...] is checked as part of the verification process (§4.10.1.8); it is not part of link-time access control."

4.10.1.8. Type Checking for protected Members encodes that a protected method C.m can be accessed in a subclass D only if the receiver object's type conforms to D. This corresponds to the Java spec 6.6.2.1.

Assume we have

class C { override def clone(): Object = "hi" }

We cannot emit def f(c: C) = c.clone() as Object.clone(). he JVM would perform the following checks:

I tested this with a compiler hack, we get java.lang.VerifyError indeed.

So the reason Object.clone works for arrays is really the special cases you point to above. Given that javac also emits [I.clone we better stick to that.

Finally, to make the discussion complete, here's the example with traits:

trait T { override def clone(): Object = "hu" }
trait U extends T
class K extends U
object T1 { def f1(u: U) = u.clone() }

We have to emit u.clone as T.clone. Using U.clone gives NoSuchMethodError: U.clone. Looking at 5.4.3.4. Interface Method Resolution I actually don't see why this is the case, it might be a bug in the JVM.

Copy link
Member Author

@lrytz lrytz Apr 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To stress that point ("it might be a bug in the JVM"), I can get javac to produce bytecode that crashes with a NoSuchMethodError:

public class A {
  public interface I1 {
    default Object clone() { return "hi"; }
  }

  public interface I2 extends I1 { }

  public static class K implements I2 {
    public Object clone() { return "Ka"; }
  }

  public static void f(I2 i) { System.out.println(i.clone()); }

  public static void main(String[] args) {
    f(new K());
  }
}
$ javac A.java
$ java A
Exception in thread "main" java.lang.NoSuchMethodError: A$I2.clone()Ljava/lang/Object;
    at A.f(A.java:12)
    at A.main(A.java:15)

@retronym
Copy link
Member

there's no good way to generate a valid tree for a static method defined in scala.

I remember noticing the same problem once before when working in Delambdafy. I think I tried Select(EmptyTree, "m"), which I thought was more honest than the version useing This, but that broke something down the line, so I backed out of the change. We could try that again, perhaps things work now that GenASM is gone.

Another possibility would be be to use a null constant for the qualifier.

@sjrd
Copy link
Member

sjrd commented Apr 18, 2016

if the owner is a ModuleClassSymbol we can encode it as a selection on the ModuleSymbol, but that is also inconsistent: the member is static in the module class, and we don't need to load the module to access it.

FYI, currently the Scala.js backend expects this. It does not do anything special to identify static methods. It compiles them as calls to methods on the companion object, just as the tree says.

That's what it's supposed to do, because Java statics end up being re implemented in Scala as methods on the companion object.

@retronym
Copy link
Member

retronym commented Apr 18, 2016

@sjrd Here's a case where Scala 2.11 currently emits a static method within a class. This is done as a special case in lambdalift to avoid a verify error. Might be interesting to look at this from the ScalaJS perspective.

scala> class C(a: Any); class D extends C({ def foo = 1; foo})
defined class C
defined class D

scala> :javap -private -c D
Compiled from "<console>"
public class D extends C {
  private static final int foo$1();
    Code:
       0: iconst_1
       1: ireturn

  public D();
    Code:
       0: aload_0
       1: invokestatic  #11                 // Method foo$1:()I
       4: invokestatic  #17                 // Method scala/runtime/BoxesRunTime.boxToInteger:(I)Ljava/lang/Integer;
       7: invokespecial #20                 // Method C."<init>":(Ljava/lang/Object;)V
      10: return
}

@sjrd
Copy link
Member

sjrd commented Apr 18, 2016

@retronym That's ... interesting ^^ I never saw that pop up. It turns out it works fine because the method is also emitted as non-static, and that is not a problem for the Scala.js IR:

> helloworld/scalajsp helloworld/D.sjsir
class Lhelloworld_D extends Lhelloworld_C {
  def foo$1__p2__I(): int = {
    1
  }
  def init___() {
    this.Lhelloworld_C::init___O(this.foo$1__p2__I())
  }
}

lineNumber(tree)
fieldStore(lhs.symbol)
// receiverClass is used in the bytecode to access the field. using sym.owner may lead to IllegalAccessError, SI-4283
val receiverClass = qual.tpe.typeSymbol
Copy link
Member

@retronym retronym Apr 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks consistent with javac.

public class Test {
    private static class A {
        int i;
        static int j;
        void m() {};
        static void n() {};
    }
    static class B extends A {};

    public static void main(String[] args) {
        B b = new B();
        C.test(b);
        assert(b.i == 1);
    }
}
class C {
    public static void test(Test.B b) {
        b.i = b.i + 1;
        b.j = b.j + 1;
        b.m();
        b.n();
    }
}
  public static void test(Test$B);
    Code:
       0: aload_0
       1: aload_0
       2: getfield      #2                  // Field Test$B.i:I
       5: iconst_1
       6: iadd
       7: putfield      #2                  // Field Test$B.i:I
      10: aload_0
      11: pop
      12: aload_0
      13: pop
      14: getstatic     #3                  // Field Test$B.j:I
      17: iconst_1
      18: iadd
      19: putstatic     #3                  // Field Test$B.j:I
      22: aload_0
      23: invokevirtual #4                  // Method Test$B.m:()V
      26: aload_0
      27: pop
      28: invokestatic  #5                  // Method Test$B.n:()V
      31: return

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scala doesn't allow static references via an instance or a subclass qualifier. I just noticed that this makes it impossible to express some static field/method selections.

class D {
  def test(b: Test.B) {
    b.i = b.i + 1;

    b.j = b.j + 1; // j not a member
    Test.B.j = Test.B.j + 1; // j not a member
    Test.A.j = Test.A.j + 1; // A is not accessible

    b.m();
    b.n(); // n is not a member
    Test.B.n(); // n is not a member
    Test.A.n(); // A is not accessible
  }
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For static access, the qual.tpe.typeSymbol will be the (fictional) module class representing the static perspective of the Java class. Looks like this is correctly handled by def internalName, so this looks correct.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scala doesn't allow static references via an instance or a subclass qualifier

I didn't know that was possible in Java..!

@retronym
Copy link
Member

retronym commented Apr 19, 2016

We may need a related change in erasure related to the casts inserted to implement refinements (is it just refinements that require this treatment?).

public class Test {
    private static class A {
        int i;
    }
    static class B extends A {
    };
class D {
  def test(b: Other with Test.B) {
    b.i = b.i + 1;
  }
}

class Other
// scalac-2.11.8 -Xprint:posterasure  sandbox/test.scala
[[syntax trees at end of               posterasure]] // test.scala
package <empty> {
  class D extends Object {
    def test(b: Other): Unit = b.$asInstanceOf[Test.A]().i = b.$asInstanceOf[Test.A]().i.+(1)
  };
}

Here, we have cast the qualifier to the owner of the field.

I expect this would be pretty hard to fix, though, and it is something of a corner case so we might just have to leave it.

@lrytz
Copy link
Member Author

lrytz commented Apr 19, 2016

Good observation..! I managed to turn this into a test:

$ cat B.java
package p1;

interface A {
  int i();
}

public interface B extends A { }

$ cat Test.scala
package p2 {
  class K
}

object Test {
  def t(x: p2.K with p1.B) = x.i()
  def main(args: Array[String]): Unit = {
    println(t(new p2.K with p1.B { def i() = 1 }))
  }
}

$ rm -rf p1 p2 *.class
$ javac B.java -d .
$ qsc Test.scala
$ qs Test
java.lang.IllegalAccessError: tried to access class p1.A from class Test$
    at Test$.t(Test.scala:6)

@lrytz
Copy link
Member Author

lrytz commented Apr 19, 2016

Echoing the earlier discussion about [I.clone vs Object.clone, because it's hidden now after a force-push:

Jason:

Seems like OpenJDK has a special case in method resolution to allow this. Probably best not to rely on it.
http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/tip/src/share/vm/interpreter/linkResolver.cpp#l439

The special case basically considers Object.clone as public if the dynamic receiver is an array.

Lukas:

Checked again the JVM spec

Access Control: protected method R, declared in C, is accessible > to a class or interface D if

  • D is a subclass of C, and
  • R must contain a symbolic reference to a class T, such that T is either a subclass of D, a superclass of D, or D itself

"symbolic reference to class T" means the receiver class in the invocation instruction.

But there's more: "This discussion omits a related restriction [...] is checked as part of the verification process (§4.10.1.8); it is not > part of link-time access control."

4.10.1.8. Type Checking for protected Members encodes that a > protected method C.m can be accessed in a subclass D only if the receiver object's type conforms to D. This corresponds to the Java spec > 6.6.2.1.

Assume we have

class C { override def clone(): Object = "hi" }

We cannot emit def f(c: C) = c.clone() as Object.clone(). he JVM would perform the following checks:

I tested this with a compiler hack, we get java.lang.VerifyError indeed.

So the reason Object.clone works for arrays is really the special cases you point to above. Given that javac also emits [I.clone we > better stick to that.

Finally, to make the discussion complete, here's the example with traits:

trait T { override def clone(): Object = "hu" }
trait U extends T
class K extends U
object T1 { def f1(u: U) = u.clone() }

We have to emit u.clone as T.clone. Using U.clone gives NoSuchMethodError: U.clone. Looking at 5.4.3.4. Interface Method Resolution I actually don't see why this is the case, it might be a bug in the JVM.

Lukas:

To stress that point ("it might be a bug in the JVM"), I can get javac to produce bytecode that crashes with a NoSuchMethodError:

public class A {
  public interface I1 {
    default Object clone() { return "hi"; }
  }

  public interface I2 extends I1 { }

  public static class K implements I2 {
    public Object clone() { return "Ka"; }
  }

  public static void f(I2 i) { System.out.println(i.clone()); }

  public static void main(String[] args) {
    f(new K());
  }
}
$ javac A.java
$ java A
Exception in thread "main" java.lang.NoSuchMethodError: A$I2.clone()Ljava/lang/Object;
  at A.f(A.java:12)
  at A.main(A.java:15)

@lrytz
Copy link
Member Author

lrytz commented Apr 19, 2016

@retronym I updated the comments here and here and filed https://issues.scala-lang.org/browse/SI-9761, so this PR should be ready now.

@retronym
Copy link
Member

LGTM

lrytz added 3 commits April 20, 2016 08:53
The code was patched many times in the history and became a bit
scattered.

When emitting a virtual call, the receiver in the bytecode cannot just
be the method's owner (the class in which it is declared), because that
class may not be accessible at the callsite. Instead we use the type
of the receiver. This was basically done to fix
  - aladdin bug 455 (9954eaf)
  - SI-1430 (0bea2ab) - basically the same bug, slightly different
  - SI-4283 (8707c9e) - the same for field reads

In this patch we extend the fix to field writes, and clean up the code.

This patch basically reverts 6eb55d4, the fix for SI-4560, which was
rather a workaround than a fix. The underlying problem was that in some
cases, in a method invocation `foo.bar()`, the method `bar` was not
actually a member of `foo.tpe`, causing a NoSuchMethodErrors. The
issue was related to trait implementation classes. The idea of the fix
was to check, at code-gen time, `foo.tpe.member("bar")`, and if that
returns `NoSymbol`, use `barSym.owner`. With the new trait encoding
the underlying problem seems to be fixed - all tests still pass
(run/t4560.scala and run/t4560b.scala).
A super call (invokespecial) to a default method T.m is only allowed if
the interface T is a direct parent of the class. Super calls are
introduced for example in Mixin when generating forwarder methods:

    trait T { override def clone(): Object = "hi" }
    trait U extends T
    class C extends U

The class C gets a forwarder that invokes T.clone(). During code
generation the interface T is added as direct parent to class C. Note
that T is not a (direct) parent in the frontend type of class C.

This commit stores interfaces that are added to a class during code
generation in the InlineInfo classfile attribute. This allows filtering
the interface list when constructing a ClassBType from a classfile.
For some reason this was not the case, leading to spurious inliner
warnings (no inline info found for method O$lzycompute).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants