Tasty Reader with Simple Trees in Annotations [ci: last-only] #8865

bishabosha · 2020-04-08T09:22:14Z

This pull request adds the ability for the SymbolTable to be populated from a classfile that is linked to a TASTy file by reading its contents. Similar to how Pickles provide symbols. The TASTy reader is not a tree provider, however there is limited support for reading expressions, enough to read annotation trees with constants and paths. Future work would be to add full support for creating trees for defs, vals, templates etc.

I would greatly appreciate feedback on a few things that I will point out in the diffs, but mainly how I handle adaption of value class extension methods, and dotty's enum constants.

Steps to Review

i. Be sure to try out the flag -Ydebug-tasty which gives a lot of information which you can use to trace where in a tasty file you are currently traversing

About half of the lines changed and the majority of files are related to testing
It is probably not worth reviewing commit by commit as many designs are refactored or cut along the way
Begin within src/compiler/scala/tools/tasty/ - these classes provide machinery for reading bytes from TASTy, and some simple data structures that represent static data in TASTy:
- TastyFormat.scala, TastyHeaderUnpickler.scala, TastyRefs.scala, UnpickleException.scala, TastyReader.scala - these are duplicates of sources from the dotty repo, unchanged except to remove dead code. These purely deal with traversing TASTy bytes
- TastyFlags.scala: A static type representing a bitset of flags that are encoded in a TASTy file, along with some standard flags inferred from context, such as Method and Deferred.
- TastyName.scala: This is a data structure representing semantic names. TastyName is the interface that TASTy uses to select members from a type, providing more information than simple strings, such as selecting a Types over Terms, companion module instead of a class, or if a term is a default getter. Names can also be paired with a MethodSignature[ErasedTypeRef], which is used to select an overloaded method.
- ErasedTypeRef.scala: This is a wrapper around a TastyName.TypeName that adds a count of the number of array dimensions the referenced type has. It carries the invariant that it references the erasure of a fully qualified class or module, and never any other definition.
- Signature.scala: Currently only encodes MethodSignature, which represents the erased signature of a method, along with any interspersed type parameter lists
Next in src/compiler/scala/tools/nsc/tasty are compiler specific definitions for reading tasty:
- TastyModes.scala: A static type representing a bitset of modes that affect the interpretation of a TASTy file, such as distinguishing between reading the parents of a class, or an annotation tree.
- package.scala: provides SafeEq, which is a implicit value class adding equality operators asserting at compiletime that the rhs is a subtype of the lhs.
- TastyUniverse.scala: provides a wrapper around scala.tools.nsc.symbtab.SymbolTable adapting it to the API TASTy requires to resolve types and symbols and create trees. Uses a cake with dependent types from symbolTable.
- ScalacUnpickler.scala: the entry point to TASTy unpickling for nsc, initialises a TastyUniverse#Context with the root symbols of a top level class, then delegates to TastyUnpickler to parse names from a TASTy file and divide it into sections, before entering symbols from the ASTs section with TreeUnpickler
- TastyUnpickler.scala: Divides a TASTy file into sections, reading names into a cache of TastyName. Responsible for providing a TastyReader associated with a section to a SectionUnpickler.
- TreeUnpickler.scala: Responsible for traversing all trees in a TASTy file, representing the definitions inside the classfile associated with the root class/module and extracting the public API into the symbolTable. Public API includes annotations where they are simple trees. Where possible, TreeUnpickler should not directly manipulate values created by the symbolTable, but use operations provided by TastyUniverse
Next in src/compiler/scala/tools/nsc/tasty/bridge/ are the sources that make up the TastyUniverse cake, divided by responsibility:
- TastyCore.scala: the base of the cake, provides aliases to types from scala.reflect at the same import level as new TASTy specific types.
- ContextOps.scala: This is the most important trait, this contains the definition for Context, which is used to maintain state through traversal of a TASTy file, such as the current owner - the container of the current definition; the traversal mode; and the root owners and source path for the TASTy file. It also provides operations for manipulation of the symbol table, such as creating/updating symbols and finding fully qualified classes/modules. the file also provides some standard error throwing capabilities with formatted errors that depend on the context mode.
- AnnotationOps.scala: adds support for creating annotations from trees
- FlagOps.scala: Handles encoding of TastyFlags to scala.reflect flags and witnessing which flags do not map directly from TASTy.
- NameOps.scala: This layer handles encoding of TastyName to symbolTable.Name, escaping any specially handled names. Also contains definitions of names for handling special dotty internal symbols within TreeUnpickler.
- SymbolOps.scala: This layer deals with selecting a member symbol from a type using a TastyName, also adds factories for making type references to symbols
- TreeOps.scala: This layer adds factories that construct typed scala.reflect Trees in the shapes that TASTy expects
- TypeOps.scala This layer adds factories that construct scala.reflect Types in the shapes that TASTy expects. Additionally provides operations to select a type from a prefix, or a type with an additional prefix, using a TastyName.
Additionally we have some extra changes to internals outside of the new tasty packages
- src/compiler/scala/tools/nsc/symtab/classfile/ClassfileParser.scala: Adds logic to look for the TASTY attribute in a classfile and then delegate to ScalacUnpickler, similar to what is done for UnPickler
- src/compiler/scala/tools/nsc/transform/ExtensionMethods.scala: Moved logic for creating extension method symbols to internal.Types, as it needs to be triggered from within the TreeUnpickler. I believe it would be too inefficient to modify ExtensionMethods.scala to inspect all select nodes to check if the symbol owner is a value class. Perhaps there is another efficient way to generate extension methods for value classes from TASTy.
- src/compiler/scala/tools/nsc/transform/patmat/PatternMatching.scala: Added logic that checks the selector of a match if it has the annotation scala.annotation.internal.Child, which is the dotty encoding of sealed children. This has potential to be moved to be driven from within TreeUnpickler but there is the problem of forcing the annotation at the correct time to prevent errors from reading a forward reference, and forcing each annotation to ensure exhaustivity.
- src/reflect/scala/reflect/internal/Constants.scala: Added a new DottyEnumTag which is used to give a precise type to a dotty enum constant before erasure. After erasure these should be treated the same as a java enum. If dotty enum constants are not typed with a constant type then pattern match warnings use a wildcard pattern and not the constants name - perhaps pattern match logic could be adjusted rather than adding a new constant kind. This change also affects BCodeBodyBuilder.scala, internal/Definitions.scala and internal/Importers.scala. - this change has been reversed.
- src/reflect/scala/reflect/internal/transform/Erasure.scala: added a check that when a dotty enum singleton is erased, it is no longer a module class, but erased like the intersection of its parents. This is matched by a change to src/compiler/scala/tools/nsc/backend/jvm/BCodeBodyBuilder.scala which is modified to that accesses to Dotty enum singletons are to a field in the wrapping object, not a field within the module class of the symbol
Testing - A new testing framework TastyTest, has been added in src/tastytest/scala/tools/tastytest/ which recreates (pos/neg/run) scenarios which the tasty reader is intended to be used for: typically, compiling some Scala 3 code with dotc and using the Dotty library, then depending on it from Scala 2 code. the modes are explained in doc/internal/tastytest.md. TastyTest also has test kinds to force dependencies compiled with dotc to not be available on the classpath from scala 2. Alternative: Partest could be adopted to integrate these new workflows, or perhaps it already has a mode for dependent compilations using a different compiler between stages.
- build.sbt several task keys have been added to assist with testing individual scenarios in the tasty_reader, perhaps these pollute the build?
- test/tasty This directory consists exclusively of test sources to be compiled by TastyTest, which is automated by a JUnit suite: test/tasty/test/scala/tools/tastytest/TastyTestJUnit.scala, which wraps TastyTest suites with customised options.

Dotty Support:

There is an attempt to provide some more information, when possible, about unsupported dotty features, (currently, union types can not be pretty printed, with more work it is possible to flatten nested union types for the purpose of better error messages)

build.sbt

src/compiler/scala/tools/nsc/settings/ScalaSettings.scala

bishabosha · 2020-04-08T10:24:54Z

src/compiler/scala/tools/nsc/tasty/TastyReader.scala

+ *  @param end      The position one greater than the last byte to be read
+ *  @param base     The index referenced by the logical zero address Addr(0)
+ */
+class TastyReader(val bytes: Array[Byte], start: Int, end: Int, val base: Int = 0) {


These classes are duplicated from the tasty-core project in dotty https://github.com/lampepfl/dotty/tree/master/tasty/src/dotty/tools/tasty I hoped we could depend on it, but its compiled with scala 2.13.1, so probably not allowed

bishabosha · 2020-04-08T10:28:18Z

src/compiler/scala/tools/nsc/tasty/TreeUnpickler.scala

+    nameAtRef: NameRef => TastyName,
+    splices: Seq[Any])(implicit
+    val tasty: Tasty) { self =>
+  import tasty._, FlagSets._


main aim here is that tasty provides some type aliases and we don't directly select from tasty.symbolTable, not yet converted to abstract types due to directly selecting on symbols here

bishabosha · 2020-04-08T10:30:38Z

src/compiler/scala/tools/nsc/tasty/bridge/ContextOps.scala

+import scala.tools.nsc.tasty.TastyName
+import scala.tools.nsc.tasty.TastyModes._
+
+trait ContextOps { self: TastyUniverse =>


once TreeUnpickler can only see abstract types, then all the FooOps traits will still see the concrete types of symbolTable

src/compiler/scala/tools/nsc/tasty/bridge/NameOps.scala

bishabosha · 2020-04-08T10:41:53Z

src/compiler/scala/tools/nsc/tasty/bridge/TastyCore.scala

+import nsc.symtab, nsc.tasty.TastyUniverse
+
+abstract class TastyCore { self: TastyUniverse =>
+  import self.{symbolTable => u}


Ideally this should extend some version where all the types and operations are abstract and thats what TreeUnpickler sees, but perhaps that would be useless, depending on what shared API is extracted for reading TASTy.

bishabosha · 2020-04-08T10:43:46Z

src/compiler/scala/tools/nsc/tasty/package.scala

+import annotation.unchecked.uncheckedVariance
+
+package object tasty {
+  implicit final class SafeEq[-T](private val t: T @uncheckedVariance) extends AnyVal {


assert rhs is subtype of lhs

bishabosha · 2020-04-08T14:03:25Z

src/compiler/scala/tools/nsc/transform/ExtensionMethods.scala

-    *  and a method for re-use elsewhere, because nobody will get this right without
-    *  some higher level facilities.
-    */
-    def extensionMethInfo(extensionMeth: Symbol, origInfo: Type, clazz: Symbol): Type = {


I moved this to internal.Types as I was not sure how to transform symbols coming from tasty as ExtensionMethods only works on trees, and there is no flag to detect if a symbol of a tree is from tasty

src/compiler/scala/tools/nsc/tasty/ErasedTypeRef.scala

src/library/scala/collection/mutable/StringBuilder.scala

bishabosha · 2020-04-17T17:36:01Z

This is ready for review now

lrytz · 2020-04-20T09:37:48Z

This is ready for review now

Great - it'll take me a week or two to get to, and through it.

bishabosha · 2020-04-20T13:47:07Z

Be sure to try out the flag -Ydebug-tasty which gives a lot of information which you can use to trace where in a tasty file you are currently traversing

bishabosha · 2020-05-04T09:46:25Z

The last three commits were important bug fixes but I'll stop pushing to this branch now

lrytz

some feedback before i forget:

Unpickler class names: ScalacUnpickler is maybe not the best name - put Tasty in it somewhere? SectionUnpickler, TreeSectionUnpickler: do we need those small wrapper classes?
The explanations you wrote in the PR description are really excellent and should make their way into the source code.
Maybe you can extend tastyreader.md or tastytest.md (side-note: merge the two?) with some workflow advice how to effectively work on the tasty rader code.

lrytz · 2020-06-08T14:07:05Z

src/compiler/scala/tools/nsc/symtab/classfile/ClassfileParser.scala

      unpickler.unpickle(bytes, 0, clazz, staticModule, file.name)
+    } else if (isTASTY) {
+
+      def parseTASTYFile(): Array[Byte] = file.underlyingSource match { // TODO: simplify when #3552 is fixed


Suggested change

def parseTASTYFile(): Array[Byte] = file.underlyingSource match { // TODO: simplify when #3552 is fixed

def parseTASTYFile(): Array[Byte] = file.underlyingSource match { // TODO: simplify when lampepfl/dotty#3552 is fixed

in case that comment applies here

lrytz · 2020-06-11T13:39:46Z

src/compiler/scala/tools/nsc/tasty/bridge/ContextOps.scala

+          for (decl <- cls.info.decls if decl.isMethod) {
+            if (decl.isParamAccessor) decl.makeNotPrivate(cls)
+            if (!decl.isClassConstructor) {
+              val extensionMeth = decl.newExtensionMethodSymbol(cls.companion, u.NoPosition)


just a question: is cls.companion guaranteed to be defined?

dotty manually defines equals$extension and hashCode$extension in the companion so I assume so

Actually, this seems to be false, so I should guard that

actually the companion object is still made even for a basic class Box(val a: Any) extends AnyVal, just not the extension methods for equals and hashCode

bishabosha · 2020-06-11T14:42:34Z

@lrytz The organisation of ScalacUnpickler, SectionUnpickler and TreeSectionUnpickler are a leftover from the dotty implementation which also has a CommentUnpickler and PositionUnpickler but they could be merged

bishabosha · 2020-06-11T20:49:06Z

src/compiler/scala/tools/nsc/tasty/bridge/TypeOps.scala

+      bounds
+  }
+
+  private[bridge] def resolveErasedTypeRef(ref: ErasedTypeRef)(implicit ctx: Context): Type = {


this is not correct for converting an ErasedTypeRef of an inner class of a class, in another branch I instead convert Type => ErasedTypeRef

bishabosha · 2020-06-26T16:02:08Z

This has been rebased for scala 2.13.3

NthPortal · 2020-06-29T22:53:26Z

I don't have the compiler knowledge to review this, and I don't have the capacity to obtain the relevant knowledge right now. sorry

bishabosha · 2020-06-30T08:10:56Z

@NthPortal sorry, I forced pushed some code and GitHub requested a review

NthPortal · 2020-06-30T17:47:03Z

no worries 💜

lrytz

This looks really great!

As discussed offline, let's try to assign the children of sealed types more early. Doing it late (patmat) causes sym.children to be incomplete if someone (a macro, compiler plugin) happens to call it early, and it's also a bit ad-hoc.

Ideally it would happen already during tasty-unpickling, similar to what's done in ordinary unpickling. If that fails because completing annotation arguments causes cycles, I can implement what was discussed here: #9023 (comment), which we should do anyway.

I don't know why the travis build failed... On jenkins it's green. Do you have an idea?

compiling sources in /home/travis/build/scala/scala/test/tasty/pos/src-3 with dotc.
exception occurred while compiling /home/travis/build/scala/scala/test/tasty/pos/src-3/dotty_i3816.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/IO.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/Defaults.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/Reader.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/Functor.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/Monad.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/Annotated.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/printing/printers.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/rootAnnot.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/HelloWorld.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/DelayedParameterised.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/AnythingIsPossible.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/Delayed.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/TraitWithSideEffects.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/symbolicAnnot.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/DelayedInternal.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest_>>>/package.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/InEmpty.scala
java.lang.AssertionError: assertion failed: no extension method found for:
  method ->:[B](y: B): (A, B) with signature Signature(List(1, java.lang.Object),scala.Tuple2) in  <none>
 Candidates:
 
 Candidates (signatures normalized):
  while compiling /home/travis/build/scala/scala/test/tasty/pos/src-3/dotty_i3816.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/IO.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/Defaults.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/Reader.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/Functor.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/Monad.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/Annotated.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/printing/printers.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/rootAnnot.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/HelloWorld.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/DelayedParameterised.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/AnythingIsPossible.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/Delayed.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/TraitWithSideEffects.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/symbolicAnnot.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest/DelayedInternal.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/tastytest_>>>/package.scala, /home/travis/build/scala/scala/test/tasty/pos/src-3/InEmpty.scala
[error] Test scala.tools.tastytest.TastyTestJUnit.pos failed: java.lang.reflect.InvocationTargetException: null, took 2.296 sec
[error]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]     at java.lang.reflect.Method.invoke(Method.java:498)
[error]     at scala.tools.tastytest.Dotc$.$anonfun$processMethod$3(Dotc.scala:19)
[error]     at scala.tools.tastytest.Dotc$.$anonfun$processMethod$2(Dotc.scala:18)
[error]     at scala.tools.tastytest.Dotc$.dotc(Dotc.scala:38)
[error]     at scala.tools.tastytest.TastyTest$.dotcPos(TastyTest.scala:145)
[error]     at scala.tools.tastytest.TastyTest$.$anonfun$posSuite$8(TastyTest.scala:141)
[error]     at scala.util.Success.flatMap(Try.scala:258)
[error]     at scala.tools.tastytest.TastyTest$.$anonfun$posSuite$7(TastyTest.scala:32)
[error]     at scala.util.Success.flatMap(Try.scala:258)
[error]     at scala.tools.tastytest.TastyTest$.$anonfun$posSuite$4(TastyTest.scala:31)
[error]     at scala.util.Success.flatMap(Try.scala:258)
[error]     at scala.tools.tastytest.TastyTest$.posSuite(TastyTest.scala:29)
[error]     at scala.tools.tastytest.TastyTestJUnit.pos(TastyTestJUnit.scala:23)
[error]     ...
[error] Caused by: java.lang.AssertionError: assertion failed: no extension method found for:
[error] 
[error]   method ->:[B](y: B): (A, B) with signature Signature(List(1, java.lang.Object),scala.Tuple2) in  <none>
[error] 
[error]  Candidates:
[error] 
[error]  
[error] 
[error]  Candidates (signatures normalized):
[error] 
[error]  
[error]     at dotty.DottyPredef$.assertFail(DottyPredef.scala:17)
[error]     at dotty.tools.dotc.transform.ExtensionMethods$.extensionMethod$$anonfun$1(ExtensionMethods.scala:195)
[error]     at dotty.tools.dotc.core.Phases.atPhase$$anonfun$1(Phases.scala:35)
[error]     at dotty.tools.dotc.core.Periods.atPhase(Periods.scala:25)
[error]     at dotty.tools.dotc.core.Phases.atPhase(Phases.scala:35)
[error]     at dotty.tools.dotc.core.Contexts$Context.atPhase(Contexts.scala:75)
[error]     at dotty.tools.dotc.transform.ExtensionMethods$.extensionMethod(ExtensionMethods.scala:208)
[error]     at dotty.tools.dotc.transform.VCInlineMethods.rewire(VCInlineMethods.scala:69)
[error]     at dotty.tools.dotc.transform.VCInlineMethods.rewireIfNeeded(VCInlineMethods.scala:96)
[error]     at dotty.tools.dotc.transform.VCInlineMethods.transformApply(VCInlineMethods.scala:106)
[error]     at dotty.tools.dotc.transform.MegaPhase.goApply(MegaPhase.scala:611)
[error]     at dotty.tools.dotc.transform.MegaPhase.goApply(MegaPhase.scala:612)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:272)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:396)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTrees$$anonfun$1(MegaPhase.scala:420)
[error]     at scala.collection.immutable.List.mapConserve(List.scala:472)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTrees(MegaPhase.scala:420)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:328)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:396)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTrees$$anonfun$1(MegaPhase.scala:420)
[error]     at scala.collection.immutable.List.mapConserve(List.scala:472)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTrees(MegaPhase.scala:420)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:271)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:396)
[error]     at dotty.tools.dotc.transform.MegaPhase.mapValDef$1(MegaPhase.scala:234)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:238)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:394)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformStat$2(MegaPhase.scala:404)
[error]     at dotty.tools.dotc.transform.MegaPhase.$anonfun$1(MegaPhase.scala:409)
[error]     at scala.collection.immutable.List.mapConserve(List.scala:472)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:409)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:339)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:396)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:251)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:394)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformStat$2(MegaPhase.scala:404)
[error]     at dotty.tools.dotc.transform.MegaPhase.$anonfun$1(MegaPhase.scala:409)
[error]     at scala.collection.immutable.List.mapConserve(List.scala:472)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:409)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:286)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:396)
[error]     at dotty.tools.dotc.transform.MegaPhase.mapDefDef$1(MegaPhase.scala:245)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:248)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:394)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformStat$2(MegaPhase.scala:404)
[error]     at dotty.tools.dotc.transform.MegaPhase.$anonfun$1(MegaPhase.scala:409)
[error]     at scala.collection.immutable.List.mapConserve(List.scala:472)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:409)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:339)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:396)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformNamed$1(MegaPhase.scala:251)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:394)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformStat$2(MegaPhase.scala:404)
[error]     at dotty.tools.dotc.transform.MegaPhase.$anonfun$1(MegaPhase.scala:409)
[error]     at scala.collection.immutable.List.mapConserve(List.scala:472)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformStats(MegaPhase.scala:409)
[error]     at dotty.tools.dotc.transform.MegaPhase.mapPackage$1(MegaPhase.scala:356)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformUnnamed$1(MegaPhase.scala:359)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformTree(MegaPhase.scala:396)
[error]     at dotty.tools.dotc.transform.MegaPhase.transformUnit(MegaPhase.scala:415)
[error]     at dotty.tools.dotc.transform.MegaPhase.run(MegaPhase.scala:427)
[error]     at dotty.tools.dotc.core.Phases$Phase.runOn$$anonfun$1(Phases.scala:318)
[error]     at scala.collection.immutable.List.map(List.scala:250)
[error]     at dotty.tools.dotc.core.Phases$Phase.runOn(Phases.scala:319)
[error]     at dotty.tools.dotc.Run.runPhases$4$$anonfun$4(Run.scala:167)
[error]     at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:15)
[error]     at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:10)
[error]     at scala.collection.ArrayOps$.foreach$extension(ArrayOps.scala:1323)
[error]     at dotty.tools.dotc.Run.runPhases$5(Run.scala:177)
[error]     at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:185)
[error]     at dotty.runtime.function.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
[error]     at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:64)
[error]     at dotty.tools.dotc.Run.compileUnits(Run.scala:192)
[error]     at dotty.tools.dotc.Run.compileSources(Run.scala:129)
[error]     at dotty.tools.dotc.Run.compile(Run.scala:112)
[error]     at dotty.tools.dotc.Driver.doCompile(Driver.scala:36)
[error]     at dotty.tools.dotc.Driver.process(Driver.scala:189)
[error]     at dotty.tools.dotc.Driver.process(Driver.scala:158)
[error]     at dotty.tools.dotc.Driver.process(Driver.scala:170)
[error]     at dotty.tools.dotc.Main.process(Main.scala)
[error]     ... 49 more

doc/internal/tastyreader.md

src/compiler/scala/tools/nsc/symtab/classfile/ClassfileParser.scala

bishabosha · 2020-07-02T16:51:14Z

so I now force the sealed children annotations as soon as all definitions are seen in a scope

src/compiler/scala/tools/nsc/tasty/TreeUnpickler.scala

bishabosha · 2020-07-06T15:19:18Z

~~forcing all annotations early is causing some cyclic referencing which needs to be looked at, such as this code~~

//object DefAnnots {
//
//  def withAnnotatedAnnot(@Wrapper.annotatedAnnot arg: Any): Any = arg
//
//  object Wrapper {
//    class annot extends StaticAnnotation
//    @annot class annotatedAnnot extends StaticAnnotation
//  }
//
//}

EDIT: resolved in commit

This commit is a squash of several commits to atomically add the feature: When indexing a scope for the first time to set up completers, add lazy annotations to symbols. At the end of the indexing, force the annotations and if one is scala.annotation.internal.Child[T], add the symbol of T as a sealed child. If we are not indexing statements, such as completing the a method and reading its parameters, force annotations as soon as each symbol is initialised, this is ok because the owner tree for the definition is also initialised so forward references can be safely accessed. force sealed children after indexing scope cache RepeatedAnnot force all annotations early, catching more errors Also force annotations for lookahead contexts. It would be good to see if perhaps we can share a single annotation cache per tasty file and force from one place. Force annotations for each new symbol Additionally, when sequencing statements, only force after the whole sequence is indexed. Forceable annotations are now stored in and forced from the initial context. Made it more lazy to print parents of a class. If a LambdaPolyType is in a New tree, unwrap it. force annotations at end of createMemberSymbol only make lazy annotation if we delay its evaluation

… class info. Before this, inherited members may not be visible at the point of forcing the annotation on a definition in the template. Also moved symbol cache to the initial context so that child contexts all pass to the same cache.

Inner classes can not be found with Mirror getRequiredClass, so we should have another way to compare types to signatures, this proposes to instead convert method signatures to Signature[ErasedTypeRef]

Read inner classes in signatures, force annotations after template parents

Fix JavaUniverseForce

scala-jenkins added this to the 2.13.3 milestone Apr 8, 2020

bishabosha commented Apr 8, 2020

View reviewed changes

build.sbt Outdated Show resolved Hide resolved

dwijnand changed the title ~~Tasty Reader with Simple Trees in Annotations~~ Tasty Reader with Simple Trees in Annotations [ci: last-only] Apr 8, 2020

bishabosha commented Apr 8, 2020

View reviewed changes

src/compiler/scala/tools/nsc/settings/ScalaSettings.scala Outdated Show resolved Hide resolved

bishabosha commented Apr 8, 2020

View reviewed changes

src/compiler/scala/tools/nsc/tasty/bridge/NameOps.scala Outdated Show resolved Hide resolved

bishabosha commented Apr 8, 2020

View reviewed changes

bishabosha commented Apr 9, 2020

View reviewed changes

src/compiler/scala/tools/nsc/tasty/ErasedTypeRef.scala Outdated Show resolved Hide resolved

bishabosha force-pushed the tasty_reader branch from fa5d042 to 86361f8 Compare April 9, 2020 17:52

bishabosha commented Apr 15, 2020

View reviewed changes

src/library/scala/collection/mutable/StringBuilder.scala Outdated Show resolved Hide resolved

lrytz self-assigned this Apr 20, 2020

lrytz self-requested a review April 20, 2020 09:37

SethTisue modified the milestones: 2.13.3, 2.13.4 May 12, 2020

lrytz reviewed Jun 11, 2020

View reviewed changes

bishabosha commented Jun 11, 2020

View reviewed changes

bishabosha force-pushed the tasty_reader branch from f290e46 to 586c2c2 Compare June 26, 2020 10:06

bishabosha requested a review from NthPortal as a code owner June 26, 2020 10:06

bishabosha force-pushed the tasty_reader branch 3 times, most recently from c88224c to d42d672 Compare June 26, 2020 12:02

bishabosha added 3 commits June 26, 2020 16:46

update documentation

50cf0fc

eliminate Ident and Annotation from TastyCore

a9b52fe

insert parens to nillary applications

1fc48cc

bishabosha force-pushed the tasty_reader branch from f6d087a to 1fc48cc Compare June 26, 2020 14:54

NthPortal removed their request for review June 29, 2020 22:53

add more refinement tests

1835f94

lrytz reviewed Jul 1, 2020

View reviewed changes

doc/internal/tastyreader.md Outdated Show resolved Hide resolved

src/compiler/scala/tools/nsc/symtab/classfile/ClassfileParser.scala Outdated Show resolved Hide resolved

src/compiler/scala/tools/nsc/symtab/classfile/ClassfileParser.scala Outdated Show resolved Hide resolved

TASTY is classfile attribute

ae1b4f6

bishabosha force-pushed the tasty_reader branch from 9a7b98d to 81376fa Compare July 3, 2020 15:17

bishabosha commented Jul 3, 2020

View reviewed changes

src/compiler/scala/tools/nsc/tasty/TreeUnpickler.scala Outdated Show resolved Hide resolved

bishabosha force-pushed the tasty_reader branch 4 times, most recently from 63fa421 to 7ae768b Compare July 6, 2020 22:00

bishabosha force-pushed the tasty_reader branch from 7ae768b to 85a574e Compare July 7, 2020 08:55

bishabosha and others added 5 commits July 7, 2020 16:44

Find overloads by converting types to ErasedTypeRef.

f8c2212

Inner classes can not be found with Mirror getRequiredClass, so we should have another way to compare types to signatures, this proposes to instead convert method signatures to Signature[ErasedTypeRef]

Merge pull request #106 from scalacenter/tasty/pick-erased-type-ref

0721788

Read inner classes in signatures, force annotations after template parents

Fix JavaUniverseForce

0104ffe

Merge pull request #107 from lrytz/pr8865

f2a49fd

Fix JavaUniverseForce

bishabosha closed this Jul 10, 2020

bishabosha mentioned this pull request Jul 10, 2020

Add the Tasty Reader [ci: last-only] #9109

Merged

SethTisue removed this from the 2.13.4 milestone Jul 23, 2020

	def parseTASTYFile(): Array[Byte] = file.underlyingSource match { // TODO: simplify when #3552 is fixed
	def parseTASTYFile(): Array[Byte] = file.underlyingSource match { // TODO: simplify when lampepfl/dotty#3552 is fixed

Tasty Reader with Simple Trees in Annotations [ci: last-only] #8865

Tasty Reader with Simple Trees in Annotations [ci: last-only] #8865

Uh oh!

Conversation

bishabosha commented Apr 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps to Review

Dotty Support:

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bishabosha Apr 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bishabosha commented Apr 17, 2020

Uh oh!

lrytz commented Apr 20, 2020

Uh oh!

bishabosha commented Apr 20, 2020

Uh oh!

bishabosha commented May 4, 2020

Uh oh!

lrytz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bishabosha commented Jun 11, 2020

Uh oh!

bishabosha Jun 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bishabosha commented Jun 26, 2020

Uh oh!

NthPortal commented Jun 29, 2020

Uh oh!

bishabosha commented Jun 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NthPortal commented Jun 30, 2020

Uh oh!

lrytz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bishabosha commented Jul 2, 2020

Uh oh!

Uh oh!

bishabosha commented Jul 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

bishabosha commented Apr 8, 2020 •

edited

Loading

bishabosha Apr 8, 2020 •

edited

Loading

bishabosha Jun 11, 2020 •

edited

Loading

bishabosha commented Jun 30, 2020 •

edited

Loading

bishabosha commented Jul 6, 2020 •

edited

Loading