[ffigen,jnigen] symbols.json -> symbols.dart

@HosseinYousefi

~~2025-08-18 Changed to generate annotations instead of a Dart file.~~ Changed back to Dart file.

@HosseinYousefi @liamappelbe Should we consider generating Dart instead of JSON for the symbols file?

Reasoning: Now that we're transitioning to the config file being Dart code, you could argue that the imported symbols from other bindings are also configuration and Dart code. We'd basically use a very big const object literal.

Pros:

Less worrying about version skew of the JSON format, you can rely on the Dart analyzer for if fields exist. (Just avoid making a field of an object get different semantics.)
No file-system gymnastics to find the symbols.dart, it can simply be imported as package:other_package/jnigen_symbols.dart.

Cons:

Public API gets an extra jnigenSymbols global var cluttering the API docs. (We could consider @nodoc.) But maybe this is actually a pro instead of con. No magic.
Not compatible with with yaml config. (Probably fine, you're likely only importing symbols on a new project, so you can opt to use the Dart config.)

WDYT @HosseinYousefi @liamappelbe? Any concerns? Thoughts?

(Context: Inspired by the discussions on running code generators on multiple packages dart-lang/sdk#56512 (comment))

SGTM. Ffigen currently uses yaml for its symbol files. So adding a Dart API format feel less redundant and confusing than adding a JSON format.

I see two problems with this:

If we want to continue using yaml for simple use cases then we can't import symbols anymore.
Not only we expose jnigenSymbols but we also expose the internal jnigen classes used for API stability alongside the classes used for transformations. So users will see both StabilityMethod and Method. We could expose jnigenSymbols in a json-like Map<String, dynamic> format that then gets parsed internally but we lose the pro about the version skew.

So users will see both StabilityMethod and Method.

This is kind of a very similar issue to #2135, the same concept existing in different phases. I think we should probably just embrace that and come up with good names. E.g. Flutter has XxxWidget and XxxElement.

If we want to continue using yaml for simple use cases then we can't import symbols anymore.

Yeah, we should deprecate and remove them at the same time.

I was thinking that source code annotations are after all probably better than both .json and .dart symbol files because they also automatically account for other custom added symbols. For example if a user just adds a ListMixin, sure they have to implement some of the abstract methods so they know about those and can manually add the fact that they added those methods somewhere like usedNames but they also don't know about a bunch of implicitly added methods that ListMixin adds (unless they go look up the exact definition of the mixin) but even then it's annoying to add all of those names (so the renamer automatically avoids colliding with them).

If we use the analyzer we just see all of the members, the annotated ones are added by jnigen, the non-annotated ones are added by the user, and we use this information when generating the bindings (either in regen or in import).

I opened #2506 to further discuss.

Some notes from our discussion and thoughts:

The annotations themselves can be type-safe, just how a symbols.dart with a const object would be type safe. Evolving the format can be done by adding nullable fields. This makes it unlikely for us to accidentally introduce breaking changes.
The same cannot be said for finding the annotations. If we significantly change the way our code is generated (classes to extension methods, maybe we start to use augmentations, ...) the annotation parser will need to still read the old way of doing things.
How do we discover which files to read for annotations? Especially if we generated bindings into multiple files?
I'd love for the Dart-config if you want to import generated bindings from another package that you'd simply refer to a generated type. That means no file-uri-gymnastics for users, just a normal Dart import.
I'd probably opt for a single annotation per generated "kind": @JniGenClassInfo(...), JniGenMethodInfo(...) rather than multiple annotations on a single declaration. Multiple annotations do not document if they are optional or not. But fields in a const constructor do.

Pros:

Works with augmentations/mixins/etc. A solution that actually works with the end-result curated by users.
Only half worrying about version skew. (The annotations are type checked.)
No file-system gymnastics to find the symbols.dart, if we implement that you can simply refer to a type.
Public API gets no extra jnigenSymbols global var cluttering the API docs.
Compatible with with yaml config. (But not if we want to get rid of file system gymnastics.)
The annotations serve as documentation, for humans (and LLMs).
Can serve as the input for link hooks and tree shaking.

Cons:

Half worrying about version skew. (Discovering all generated things and all annotations can break on breaking change.)
File system gymnastics, at least for yaml.
We start depending on the analyzer, but if we want to support custom classes etc we need to do this.

I opened #2506 to further discuss.

Let's just repurpose this one so we have context!

I'd probably opt for a single annotation per generated "kind": @JniGenClassInfo(...), JniGenMethodInfo(...) rather than multiple annotations on a single declaration.

For jnigen I think that the only thing needed in the annotation is the JVM name and the descriptor for all members.

@JvmClass('some/package/ClassName')
class ClassName extends JObject {
  @JvmMethod('foo', '(Ljava/lang/String;I)V')
  void foo(JString s, int i) { /* ... */ }

  @JvmField('bar', 'J')
  int get bar;
}

Everything else can really be identified from the generated bindings: Did we change this method to be a getter? Did we rename this? What is the superclass of this class? ...

I'd set these up with (1) A name that cleary says JniGen, and (2) use required named fields. I think (1) JniGen in the annotation names is useful for documentation purposes for users, and (2) I'm pretty sure we're going to add stuff later.

@JniGenClass(
  originalName: 'some/package/ClassName',
)
class ClassName extends JObject {
  @JniGenMethod(
    originalName: 'foo',
    jvmSignature: '(Ljava/lang/String;I)V',
  )
  void foo(JString s, int i) { /* ... */ }

  @JniGenField(
    originalName: 'bar',
    jvmType: 'J',
  )
  int get bar;
}

Everything else can really be identified from the generated bindings: Did we change this method to be a getter? Did we rename this? What is the superclass of this class? ...

True, but maybe there are some cases where generated code can end up being identical from different sources. For example lets say we generate enums to classes or something, and the code looks very very similar to how we generate classes to classes. (Hypothetical use case.) If we can clearly state something in a simple field that would require a complex navigation of the ast, it might be worth it.

(The reason I'm thinking this is that I've been trying to reverse-engineer certain patterns in package:json_syntax_generator from JSON schemas. And sometimes a different pattern uses similar keywords in a JSON schema, making it hard to pin down exactly was it A or B.)

I'd set these up with (1) A name that cleary says JniGen[...] I think (1) JniGen in the annotation names is useful for documentation purposes for users

I personally don't like it aesthetically.
These annotations will only exist in generated files that start with an autogeneration notice anyway, so repeating JniGen doesn't add any extra information here.
Kotlin uses similar annotations like JvmName JvmField, JvmRecord, etc.
JvmXyz actually better documents the fact that these are related to Java bytecode.
originalName is not quite the right word for it. It's the name that is found in the bytecode that we use to correctly match these elements. It's possible that a Kotlin element has a different "originalName" than the "bytecode name" meaning that in the original language (Kotlin) it was named differently.

and (2) use required named fields. [...] (2) I'm pretty sure we're going to add stuff later.

I don't think we're going to add stuff later. And having the first one or two be positional doesn't interfere with the rest anyways.

True, but maybe there are some cases where generated code can end up being identical from different sources. For example lets say we generate enums to classes or something, and the code looks very very similar to how we generate classes to classes. (Hypothetical use case.) If we can clearly state something in a simple field that would require a complex navigation of the ast, it might be worth it.

Let's cover two cases that we want to solve with this: API stability, and importing other generated bindings.

For API stability, we only really need the name and the type of the Dart element and a way to match the element in the code with the generated element. The bytecode identifiers are perfect for this matching to happen.

For importing generated bindings it's important to note that we cannot change the already generated bindings we're trying to import. We just want to use it. So what we actually want is the Dart code structure which is perfectly represented—well, in Dart AST! Sure, com.example.MyEnum is an enum but it has been generated as a Dart class, so now we should treat it as such. (In this specific example we probably have another JvmEnum annotation but for the sake of argument I'm assuming it's a kind of change we do not capture with annotations alone.)

We're completely bound by the limitations set by Dart when trying to interact with the already generated code. If there is a getter named foo in a superclass we're trying to import, we simply cannot have another element with the same name in our generated bindings even if Java allows it. Storing other information like signature¹ is not really helping us because even if we know what the original generic type structure was, we're limited by how the Dart code is.

Overall, I think by restricting the annotations this way, we'll develop codegen tools that are more resilient to version changes even when downgrading (maybe because the latest version has had a bug but we still want to use the generated source for API stability).

Java Signature differs from the descriptor as it also has generic type information in it. For example, for a method List<T extends String> foo() the descriptor is ()Ljava/util/List; while the signature is ()Ljava/util/List<+Ljava/lang/String;>; ↩

Coming back to this:

I'd probably opt for a single annotation per generated "kind": @JniGenClassInfo(...), JniGenMethodInfo(...) rather than multiple annotations on a single declaration. Multiple annotations do not document if they are optional or not. But fields in a const constructor do.

I don't think that documenting if they're optional is necessary. If we were to add other parameters, I assume to not break backwards compatibility we will end up making all the other parameters optional anyways which is in effect like having multiple annotations. Also the annotations reside in package:jni and even if we want to make another parameter required, it should really only be required when we're trying to "import" but changing the original annotation and adding another required parameter means that existing bindings will stop working (even though we never really wanted to import them, we only wanted to use them).

Basically what I mean is that annotations are somewhat weird, in that they're in reality closer to dev-dependencies because we don't really care about them in runtime, but since they exist in the source we have to have them as dependencies. To fix this issue, we should basically never break them if we're not also breaking the runtime.

I think in ObjC's case we need the original name, and whatever unique ID clang gives us, since the name isn't globally unique. I don't think we need the signature, but there's something analogous to those Java encoded signatures in ObjC, so I'll probably add it anyway: Type Encodings.

@liama

Cycling back here with the notes from an offline discussion.

Pros for using annotations:

Enables dealing with user adding members. (Without the user having to specify the conflicting names in the transformer.)
The generated code has a lot of implicit information that we don't need to think about up front. (open world)

Cons for using annotations:

Annotation on every member requires reverse engineering the generated structure, and user added members (open world assumption, how do we cover everything?)
The generated code has a lot of implicit information that we don't think about might be accessed in reading of the annotations, leading to accidental breaking changes. (open world)
Annotations on the classes or the library require repeating the member names and class names.
@liama: The annotation parser is prohibitively slow. This is a hard blocker.

So, we go back to Dart consts. Which means we need to deprecate the YAML config.

I was thinking that this is good for imports where the user explicitly provides the const object to our API but how do we want to use it for stability information? The user has to know that after the first generation, they have to change their config to include an object from the generated stability file. I'm not sure if this is desirable.

Another aspect to consider is the workflow when accidentally bricking your JNIgen/FFIgen setup due to a wrong config or a bug in FFIgen/JNIgen: If you run the generator, and it generates a broken symbols file, and you use it for the name stability, you cannot run that same generator anymore. The best practice would probably be to revert the stability file.

This impacts the various options slightly differently:

Annotations in the generated file: Most likely to brick because the surface area is huge.
JSON jfile: Smaller surface area, failure only visible at runtime.
Dart consts file: Smaller surface area, failure visible in editor with dart analyze. Incompatible changes might be caught by dart_apitool on the constructors of the consts.

@liamappelbe

RE: #3617

Now that we use Dart files for the config, we could consider emitting a big Dart const for the symbols rather than parsing the yaml. Then the package importing the symbols of another package simply refers to a globally visible Dart top level const.

I don't remember all the considerations we had previously about it. But some implications:

One can depend on another package via pub, and pull in the Dart const. (Rather than having to find the package root via the package config and finding the yaml file in that package.)
We cannot do accidental breaking changes to the yaml format, it's Dart objects.
We cannot do a gradual breaking change where we support two formats, all future formats need to be additive. (Mitigation, we could make a toplevel object that holds a "v1".)

@liamappelbe WDYT would it make sense to emit a big const and import it to get rid of all file paths?

Note: the classes for the symbols file should probably live in package:jnigen_symbols and package:ffigen_symbols, or potentially in package:jni. Because if we export a symbols Dart object from a package, the classes for that object should be dependencies not dev_dependencies.

Also if you run generators in sequence, we need to run them with Process.run. However, I suspect that even in mono-repos with multiple packages, the generators will likely be in different packages. So they will have different hook/generate.dart entry points anyways. Only in the case where we have multiple generators dependent on each other within a single package will we need to use Process.run.

The ImportedType? Function(Declaration declaration) importType; approach is so general that we can easily support the legacy yaml file and any kind of generated Dart object.

Now I'm wondering if the top level Dart object should just be a Function with that signature, or if we should go with the top level const object idea and add a conversion util that turns it into a Function (like we do for the symbol yaml file). Probably doesn't matter much either way.

I'd probably make it a toplevel const object for efficiency reasons.

We can make the toplevel const object class have a getter that returns a closure with the right signature. That's probably easiest to discover. (A toplevel helper conversion function is harder to discover. Since the signature is ImportedType? Function(Declaration declaration) and not an interface with a member with that type, we can't put a named factory or something.)

Navigation Menu

Repository navigation

[ffigen,jnigen] symbols.json -> symbols.dart #2453

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[ffigen,jnigen] symbols.json -> symbols.dart #2453

Description

Activity

liamappelbe commented on Jul 29, 2025

HosseinYousefi commented on Aug 1, 2025

dcharkes commented on Aug 4, 2025

HosseinYousefi commented on Aug 18, 2025

dcharkes commented on Aug 18, 2025

HosseinYousefi commented on Aug 18, 2025

dcharkes commented on Aug 18, 2025

HosseinYousefi commented on Aug 19, 2025

Footnotes

HosseinYousefi commented on Aug 19, 2025

liamappelbe commented on Aug 20, 2025

dcharkes commented on Sep 2, 2025

HosseinYousefi commented on Sep 16, 2025

dcharkes commented on Sep 16, 2025

dcharkes commented on Sep 7, 2026

liamappelbe commented on Sep 7, 2026

dcharkes commented on Sep 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions