☂️ JNIgen transformations and possibly YAML config removal

Note

All of this could be used for package:jni itself, as it already contains a lot of hand-written code for List, Map, ByteBuffer, … I plan to add support for exceptions and it's really useful to write transformers for common exception types and have them either in package:jni or even another package:java_library.

Use Cases

Differences between Java and Dart styles

For example, Java naming convention recommends constants to be SCREAMING_SNAKE_CASE, while Dart uses lowerCamelCase.

=> A way to easily change the case.

Some words are keywords in Dart

For example, the word in is a keyword in Dart and cannot be used as an argument name in a method, but it can be in Java. The current behavior is to add 0 to the end of these reserved names. So in becomes in0.

=> A way to specify certain words to rename.

Some methods and fields are used in `Object`

Like hashCode and toString. toString is also used in Java, however the return type of the method in Java is JString. The current behavior is to add 1 to the end of the existing methods. A prettier renaming would be to change toString to toJString or toJavaString.

=> A way to change method names, this of course needs to take the subclasses into account as well.

Some names have a different meaning in Dart semantics

In Dart, Errors are classes that we normally don't catch, in contrast to Exceptions which we do catch. This is largely the same in Java, however the exceptions that are subclasses of RuntimeException are more similar to Dart's Errors as they're "unchecked exceptions". One might want to rename classes such as NullPointerException to NullPointerError to better align with Dart semantics.

=> A way to change class names.

Some classes have the same names as `dart:core` classes

Even though it's technically possible to either import dart:core as core, or import <generated> as gen, it's probably easier to have a different name for classes like Object and String that exist both in Dart and Java. For example, renaming Object to JObject.

=> A way to change class names.

Purely for aesthetics

Self explanatory!

Java supports method overloading, but Dart does not

This means that int foo(), int foo(int a), and int foo(int a, int b) will turn into foo, foo1, and foo2 in Dart. An alternative for this would be to let users rename each foo or to write a custom logic with optional parameters.

=> A way to change method names.

Dart supports operator overloading, but Java does not

A good example of this is java.lang.BigInteger. Since Java doesn't support operator overloading, it has methods like add, multiply, divide, … These methods could be changed to their respective operators in Dart to match the language better. List's get and put also correlate with operators [] and []=.

=> A way to convert methods to operators.

Dart has getters and setters, but Java does not

Java is full of getFoos and setFoos. We can change these methods to getters and setters in Dart. We might only want to do this if we have a private foo field. Similarly some methods are better suited to be getters in Dart, an obvious example of this is hashCode() that can be converted to hashCode getter.

=> A way to convert methods to setters/getters.

We might not want to include certain classes/methods/fields

So, we need to have an option to exclude elements from the generated bindings.

=> A way to exclude elements.

Ability to add methods/fields to classes

For example, a toDartString could be a good method to add to JString, which is not already available. Of course, users could use extensions but those have their own limitations for example in case of static methods or constructors. This can be as simple as the ability to add any user-defined string to any class.

=> A way to add custom code.

Argument names are not available in the bytecode

When parsing from the bytecode, the argument names in methods are not available. This means that a method like int foo(int bar, double baz) gets turned into int foo(int i, double d).

=> A way to rename argument names.

Ability to implement a Dart class

For example we might want java.util.List to implement or mixin ListMixin as it is already the case for package:jni's JList.

=> A way to add custom code and to modify the class definition.

Approaches

Yaml config with regex support

This is the approach already used by ffigen. The syntax is something like:

The point here is not to specify an exact syntax for the yaml, but to give you a general idea of how it can exclude and rename.

Pros

Relatively easy to do easy things.
Declarative.

Cons

Impossible to do complicated things, such as checking for some classes' structure or superclass when deciding about renaming. One such example is when trying to change all (.*)Exception classes to $1Error classes when these classes extend RuntimeException. This and many other such examples are simply not possible to do with a yaml config.
Regex is notoriously difficult to debug.
It's not obvious in which order the changes happen. Does only one change happen like a switch statement? Or will they happen in a chain from top to bottom?
Fully qualified names in Java include periods and dollar signs (for inner classes) like a.b.Foo$Bar which need to be escaped since we allow regex: a\.b\.Foo\$Bar which makes it ugly (and error-prone) to do simple things.

Overall, I think it's not a good idea to have all of the transformation logic in yaml.

Passing functions to Dart config object

Another option would be to ditch the yaml configuration for more complex tasks and write Dart code. We already support this for normal use cases, and we can extend it to do transformations as well.

We could consider exposing a Method object instead of only the name to access things such as the name of the enclosing class.

This is better, we wrap the actual internal Method into a wrapper (called Method again!) with only the useful information and actions to be exposed to the user. This way any changes to the internal representation of Method will not break users' config code.

Pros

Still easy to do easy things.
Possible to do more complicated things.

Cons

Having a single method for transforming methods, could lead to a messy code with a bunch of branchings.

Using transformer visitors

This is basically the same as passing functions to Dart config but could be chained together and can suggest better code organization habits.

Pros

Almost the same as passing functions, just packaged differently

Cons

Could be a bit unfamiliar for users who are not used to the visitor pattern.

Another Question, is yaml useful?

Suppose we go with the transformer option, will yaml be still useful? It would be weird to start off with yaml, and transfer everything to a Dart config to add a transformer. Another solution would be to "import" a dart file within the Yaml config like

But what do we gain from keeping the yaml around? Is it that much simpler than having a Dart file for beginners? Is JNIgen ever going to be used by beginners?

I would argue that most actual JNIgen use cases are going to be complicated enough to need some form of transformation.

Things to keep in mind

Imported bindings

Users cannot modify the imported bindings as they are already generated. This will impose certain limitations on what they can and cannot do. For example, one cannot rename an imported class, so we probably will have two classes Class and MutableClass, where Class is not renamable. Or have an imported boolean getter and throw when trying to modify an imported class.

What about fields and methods? The same method could be used in an imported class, and we cannot change its name if we're extending or implementing that class.

Reserved Names

Users might want to add certain names to the reserved namespace of a class. These names should not be reused in the subclasses. This happens when a package author adds custom code to introduce useful fields/methods.

Issues to tackle

More use cases:

Renaming without breaking user's code

We could rename and deprecate. The previously used names (stored in a json file), can still work but be annotated with @Deprecated. (Could we also add a quick fix for this?)

Generate CHANGELOG

Renaming public API breaks any binding that depends on our binding, and any code that uses it. Transformers could produce a changelog to communicate the breaking changes.

The ideal solution must balance four key requirements:

User Empowerment: Users must have powerful and flexible control to rename entities, change method signatures (to getters/setters/operators), exclude elements, and add custom code.
Predictability: The system's behavior should be easy to understand. Users should not be surprised by the final generated code. The result of their transformations should be respected.
Stability: Generated bindings must be stable across runs. Re-running jnigen after a Java class is updated (e.g., with new overloaded methods) should not break existing, valid code by arbitrarily changing the names of previously generated entities.
Correctness: The final generated Dart code must always be syntactically valid, free of name collisions, and respect Dart's language rules (keywords, etc.).

Proposed Architectures

We will evaluate three primary models for integrating the user visitor.

Option A: Pure User-as-Final-Arbiter (Visitor at the End)

In this model, the user has the final say on all naming. The system generates baseline names, and then the user visitor runs to finalize the output. This is a "raw" implementation where the user's transformer code bears the full responsibility for correctness.

Pipeline: ... -> renamer -> **userVisitor** -> dartGenerator

Cons

High Complexity Burden: The user's transformer is responsible for all correctness logic: manual conflict resolution, keyword handling, and manually reading/respecting renaming.json for stability.
Very High Barrier to Entry: Simple tasks are difficult and error-prone.
Difficult Transformer Composition: Chaining multiple transformers is fragile.

Option B: System-as-Final-Arbiter (Visitor at the Beginning)

In this model, the user provides their intent early in the pipeline. The system uses this intent as a high-priority input for its own robust renaming logic, guaranteeing correctness and stability while trying to honor the user's wishes.

Pipeline: ... -> **userVisitor** -> renamer -> dartGenerator

Pros

Simplicity & Safety: Users can write simple, focused transformers. The system acts as a safety net, guaranteeing valid output and managing stability automatically. This is the most accessible model.
Safe Composition: The final renamer step can safely arbitrate and gracefully resolve conflicts introduced by a chain of transformers.

Cons

Potential for Surprise: The system might override a user's desired name to resolve a conflict. While this can be mitigated with warnings, it violates the principle of "what you set is what you get" and can lead to poor, but valid, APIs (e.g., recordId$1).
Less Absolute Power: Users cannot implement a completely custom renaming scheme that might conflict with the system's core correctness rules.

Option C: Guided User-as-Final-Arbiter (Visitor at the End with Proactive API)

This model builds on Option A, keeping the user as the final decision-maker, but equips them with powerful, proactive tools built into the transformation API to mitigate the risks.

Pipeline: ... -> renamer -> **userVisitor** -> dartGenerator

Key Enhancements to the Visitor API:

Proactive Validation: The name setter on a transformation node will immediately throw an exception if the proposed name is a reserved keyword or is already in use by another member in the same class. The user gets instant feedback within a try/catch block.
Transparent Stability Management: The user-facing nodes will have a read-only stableName property, pre-populated from renaming.json. This allows the user's code to easily respect or intentionally override stable names.
Changelog & Override Tracking: When a user's code successfully sets a name that overrides a pre-existing stableName, this action can be internally flagged to generate a changelog.md file, explicitly documenting breaking changes.

Pros

Maximum Control with Guardrails: Achieves the predictability of Option A while solving its most critical safety flaws. The user has the final say, but the API helps them avoid common mistakes, forcing them to produce a clean API.
Lowers Barrier to Entry: Compared to Option A, it's far easier to write a correct transformer.
Turns Stability into a Feature: Stability is no longer a manual chore but an explicit, controllable part of the API.

Cons

Remaining User Responsibility: While many risks are mitigated, the user is still fundamentally responsible for correctness, especially for conflicts with superclass members.
Brittle Transformer Composition: While the API will detect and prevent a final name collision by throwing an exception, it does not resolve it. This makes chaining transformers fragile: a conflict caused by two independent transformers will fail the build, forcing the user to manually coordinate the logic between them.

Recommendation

The choice is between Option B (Safety by Default) and Option C (Power with Guardrails).

Option B is the simplest and safest path. It guarantees a successful build, making it a good choice if the primary goal is accessibility. However, it does so at the risk of producing a poor API (recordId$1), which is a significant flaw for a library generation tool.
Option C is a more advanced, powerful alternative. It provides the ultimate control that expert library authors require. It prioritizes API elegance over automatic build success by failing loudly when a conflict arises, forcing the developer to make a conscious, high-quality decision.

Recommendation: Pursue Option C: Guided User-as-Final-Arbiter.

Given that jnigen is an expert-level tool for complex interoperability scenarios, the target audience is well-suited to handle the responsibilities of Option C in exchange for its power and transparency. The proactive validation and "fail-on-conflict" approach ensure a higher-quality final API, which is the primary goal of the tool. The benefits of absolute predictability and control, when combined with strong API guardrails, outweigh the "always-succeeds" but potentially "ugly-API" model of Option B.

However, it does so at the risk of producing a poor API (recordId$1), which is a significant flaw for a library generation tool.

I wonder if our renamer should be opt-in instead of default. And the end of the pipeline we should have an error if there are any conflicting names.

The user-facing nodes will have a read-only stableName property, pre-populated from renaming.json.

This would be useful both for user-defined transformers as well as for our current renamer. 👍 I like that we put this in the system and not in a transformer.

From a conceptual point of view, I believe our current renamer should not have any special privileges over the user-defined visitors. E.g. I'd want our users to be able to write a similar visitor that postfixes argument types instead of $123. So the visitor API should be powerfull enough to do the kind of renames we currently do (e.g. walking the class hierarchy).

Users then chose option C or B by combining our renamer with their own, or only use ours or only use theirs. (The Dart API should take a list of visitors that are run in sequence.) This design makes the visitors composable and lets users decide what to do:

... -> our renamer -> dartGenerator (current behavior)
... -> dartGenerator (might fail due to duplicate names being output) (should this be the new default?)
... -> **userVisitor** -> dartGenerator (user just wants to have different renaming, error on emitting duplicate names)
... -> **userVisitor** -> our renamer -> dartGenerator (partial user-renaming, rest done by our renamer, lazy-mode)
... -> our renamer -> **userVisitor** -> dartGenerator (a different lazy mode)

I was thinking the API would look more like:

bindings.findClass('com.example.Bar').findMethod('...').allInstances.name = 'foobar';

The .allInstances returns a List<Method>, and the set name(value) is an extension method on List<Method>.

The way to make it discoverable is to add in the doc comment on String get name for the individual methods that you should likely rename all related methods together with a small snippet of how to do it.

I was thinking the API would look more like:

bindings.findClass('com.example.Bar').findMethod('...').allInstances.name = 'foobar';
The .allInstances returns a List<Method>, and the set name(value) is an extension method on List<Method>.

The way to make it discoverable is to add in the doc comment on String get name for the individual methods that you should likely rename all related methods together with a small snippet of how to do it.

So an individual member doesn't have a set name?

Probably an extension on List<Method> is not the thing we want then, we want to have a dedicated class for the allinstances. Otherwise someone could do [method].name = 'something' and go around the API!

Yeah, something like a MethodGroup, or MethodPrototype that you can describe on that it is all methods everywhere in the hierarchy that are related to each other through overrides and implements clauses.

But then both of these classes are basically doing the same thing. So we can either just document that a class doesn't have individual methods but it just has method prototypes and keep the same Method class.

Or change the way we do renaming to make it more obvious, e.g. instead of set name use renameInAllInstances(String).

Or change the way we do renaming to make it more obvious, e.g. instead of set name use renameInAllInstances(String).

I was thinking that you might want to do various versions of .allInstances that give different groups. E.g. for renaming you want to follow all overrides up and down and all implements transitively. But maybe if you want to add nullability on a library that's not there you only want to do it down. So we decouple the "getting a group" and the "setting of something".

I was thinking that you might want to do various versions of .allInstances that give different groups. E.g. for renaming you want to follow all overrides up and down and all implements transitively. But maybe if you want to add nullability on a library that's not there you only want to do it down. So we decouple the "getting a group" and the "setting of something".

Good point, so this kinda means that we also have MethodParameterGroups. Tightening the nullability of one should tighten the nullability of everything under it.

I think all this makes the transformer stuff too complicated for my liking though. Something about just setting .name and JNIgen handling it everywhere in the hierarchy appeals to me for most use cases. It's very likely that the new name suggested by the user doesn't really conflict with anything anyways.

Likewise for the method parameters, I'd rather handle the nullability sync automatically (as you say, it should only go downwards).

Those "getting a group" various things should also make it trivial to write validators:

final group = oneMethod.allInstances(implemented: true, implements: true, overridden: true, overrides: true, transitive: true);
final distinctNames = group.instances.map((e) => e.name).distinct();
if (distinctNames.length > 1) {
  //print warning
}

Yeah maybe the setter on the individual parameter and method instances should simply do void set name(value) => allInstances(with right params).name = value; then we can have the default case be covered and it's very easy for users to see how to get different behavior by simply looking at the setter implementation.

So we decouple the "getting a group" and the "setting of something".

Another point is getting the "only down" group and then setting .name is wrong. So by splitting this into two phases we introduced an incorrect combination of phases which wouldn't exist if we had .renameInAllInstances and .changeNullabilityOfAllSubMembers

An important scenario to test is running the generator -> removing a method named foo and regen -> adding another method named foo and regen.

In this case the second foo must be named differently that the first foo, even though the first foo has been completely removed from the bindings. This means that our exporter logic should not only look at the existing member and their stable names but also the previous members that are non-existing but are available in the symbols file and keep them available in the next iterations.

Another argument is that removing foo is already a breaking change so it doesn't matter that another foo with the same name is taking over its place so it's not necessary to keep a stable name after removing a member.

We use the fact that this member has been renamed or not quite a lot in the naming conflict resolution algorithms. We have three API options:

Make the member.name getter nullable. If member.name is null, this member has not been renamed (or even named) yet.
member.name is nullable but defaults to stableName. This moves the logic of "if the stable name is available for this member, then use that" from the transformer layer into the graph builder or importer layer. We could do this if we think our users always want to have this logic anyways.
member.name is not nullable but defaults to stableName ?? originalName or originalName. Users have to manually keep track of which method prototypes have been renamed. It's important to note that member.name == originalName does not tell us if the method name has been decided or not. It could be that we decided to keep the original method name which we expect to do most of the times.

I think I want to go with the second option. Having the name getter be nullable actually further discourages users to use member.name in their APIs which is ideal since they should really be using .identifier or .originalName getters instead.

Another possible scenario is that a method can be provided in an implementer of an interface by extending another class. For example:

public class Player {
  // lower your head!
  public void duck() {...}
}

public interface Ducking {
  void duck();
}

public class DuckingPlayer extends Player implements Ducking {
  // No need for anything, implementing Ducking is already satisfied by extending the class Player.
}

public class DuckOwningPlayer extends Player {
  public Duck duck;
}

This means that in the bindings the name of the duck method in Player and the name of the duck method in the Ducking interface must be the same, since they're shared as one in the DuckingPlayer. This is different than the concept of MethodPrototypes. These two methods are technically not each other's super-methods at all (although they of course share the same signature and original name) but since somewhere in the hierarchy, one is used to implement the other, their states must be shared everywhere in the tree.

This can connect multiple unrelated imethods together if they are all implemented or extended in the same class and they share the same method name+signature.

Another point that came up in #2453 is instead of using symbols.json or similar, let's use annotations on the actual generated bindings and use Dart analyzer to extract stability and symbol information from the source. This way we can see all of the other potential members added by the user and there is only a single source of truth—the Dart source.

☂️ JNIgen transformations and possibly YAML config removal #1171

Description

Note

Use Cases

Differences between Java and Dart styles

Some words are keywords in Dart

Some methods and fields are used in Object

Some names have a different meaning in Dart semantics

Some classes have the same names as dart:core classes

Purely for aesthetics

Java supports method overloading, but Dart does not

Dart supports operator overloading, but Java does not

Dart has getters and setters, but Java does not

We might not want to include certain classes/methods/fields

Ability to add methods/fields to classes

Argument names are not available in the bytecode

Ability to implement a Dart class

Approaches

Yaml config with regex support

Pros

Cons

Passing functions to Dart config object

Pros

Cons

Using transformer visitors

Pros

Cons

Another Question, is yaml useful?

Things to keep in mind

Imported bindings

Reserved Names

Issues to tackle

Activity

HosseinYousefi commented on Sep 10, 2024

Renaming without breaking user's code

Generate CHANGELOG

HosseinYousefi commented on Jun 4, 2025

Proposed Architectures

Option A: Pure User-as-Final-Arbiter (Visitor at the End)

Cons

Option B: System-as-Final-Arbiter (Visitor at the Beginning)

Pros

Cons

Option C: Guided User-as-Final-Arbiter (Visitor at the End with Proactive API)

Pros

Cons

Recommendation

dcharkes commented on Jun 4, 2025

15 remaining items

dcharkes commented on Jul 24, 2025

HosseinYousefi commented on Jul 24, 2025

dcharkes commented on Jul 24, 2025

HosseinYousefi commented on Jul 24, 2025

dcharkes commented on Jul 24, 2025

HosseinYousefi commented on Jul 24, 2025

dcharkes commented on Jul 24, 2025

HosseinYousefi commented on Jul 24, 2025

HosseinYousefi commented on Aug 1, 2025

HosseinYousefi commented on Aug 1, 2025

HosseinYousefi commented on Aug 1, 2025

HosseinYousefi commented on Aug 19, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Some methods and fields are used in `Object`

Some classes have the same names as `dart:core` classes