Skip to content

Conversation

@tneotia
Copy link
Contributor

@tneotia tneotia commented Jul 28, 2021

This PR adds support for image keyboard support on Android, utilizing the commitContent function.

Partially addresses #20796. Associated engine PR: flutter/engine#27763.

cc @justinmc and @gspencergoog.

Pre-launch Checklist

  • I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
  • I read the [Tree Hygiene] wiki page, which explains my responsibilities.
  • I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement].
  • I signed the [CLA].
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt.
  • All existing and new tests are passing.

@flutter-dashboard flutter-dashboard bot added f: cupertino flutter/packages/flutter/cupertino repository f: material design flutter/packages/flutter/material repository. framework flutter/packages/flutter repository. See also f: labels. tool Affects the "flutter" command-line tool. See also t: labels. labels Jul 28, 2021
@google-cla
Copy link

google-cla bot commented Jul 28, 2021

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

@google-cla google-cla bot added the cla: no label Jul 28, 2021
@tneotia
Copy link
Contributor Author

tneotia commented Jul 28, 2021

@googlebot I fixed it

@google-cla
Copy link

google-cla bot commented Jul 28, 2021

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

@zlshames
Copy link

@googlebot I consent.

@google-cla google-cla bot added cla: yes and removed cla: no labels Jul 28, 2021
@gspencergoog
Copy link
Contributor

@tneotia You'll need to do the same rebase or squash here as in the engine PR.

@zlshames
Copy link

zlshames commented Jul 28, 2021

@tneotia You'll need to do the same rebase or squash here as in the engine PR.

Hey, so when I originally checked-out the repo to make the changes, this is what I did (for the engine):

For the framework, I did a similar thing, but instead of checking out a hash, i checked-out a tag. The engine repo doesn't have those tags. The reason I did the reset was because I wanted to make sure that the existing commits were only ones that existed in the 2.2.3 tag, and drop anything newer. And the reason we only wanted the 2.2.3 changes (instead of master) was because we wanted only stable/officially released code that was compatible with the flutter framework v 2.2.3

git checkout <sha256 hash of the cherry picks for 2.2.3>
git reset --hard <sha256 hash>
git branch ......

# made my changes

git commit ......
git push ......

Should this have been done a different way? I am also no git expert, but i generally know the basic actions

@gspencergoog
Copy link
Contributor

Somewhere in there you need to either squash the commits down to one commit, or rebase them so that the only changes between the master branch and your branch are your changes. The CLA bot doesn't really like any PR where more than one author appears in the list of changes.

The workflow I use is:

# Make a new branch
git fetch origin
git fetch upstream
git checkout master
git checkout -b git-insertion-pr
git push -u origin git-insertion-pr
# Make changes and push
git commit -m "message"
git push
# More changes
git commit -m "message"
# Incorporate changes made to master
git fetch upstream
git rebase upstream/master
# Fix conflicts
git push --force origin git-insertion-pr

With git, there are a number of ways you can accomplish the same thing (sadly), but this one works for me.

I think the problem here is that you need to rebase on top of master, so the relevant commands would be:

git fetch upstream
git rebase upstream/master
# Fix conflicts
git push --force origin git-insertion-pr

This all assumes that the output of git remote -v looks like:

origin	[email protected]:BlueBubblesApp/flutter.git (fetch)
origin	[email protected]:BlueBubblesApp/flutter.git (push)
upstream	[email protected]:flutter/flutter.git (fetch)
upstream	[email protected]:flutter/flutter.git (push)

If that's not the case, then you'll probably have to at least set up the upstream that way and try rebasing.

@tneotia tneotia force-pushed the tanay/gif-insertion-pr branch from 5e19404 to 54762a0 Compare July 28, 2021 17:33
@tneotia
Copy link
Contributor Author

tneotia commented Jul 28, 2021

@gspencergoog this one should be fixed now as well. Again thanks for your help, learned something new today!

@zlshames
Copy link

@gspencergoog this one should be fixed now as well. Again thanks for your help, learned something new today!

Beautiful, thank you for doing that

@justinmc
Copy link
Contributor

I've left a review (flutter/engine#27763 (review)) on the engine PR in order to find consensus on this approach before I get too detailed with a review here.

@zlshames
Copy link

zlshames commented Jul 29, 2021

I've left a review (flutter/engine#27763 (review)) on the engine PR in order to find consensus on this approach before I get too detailed with a review here.

Hopefully this isn't a dumb question, but is there a way to use the engine in "dev" mode? So I don't need to rebuild the engine each time I make a change and need to test it? That's how I've been doing it because I never found a way to use the engine without it being compiled first.

In the meantime... I will be recompiling the engine with some changes to test how changing the mimeType support list will work.

Edit 1: So, I just rebuilt the engine after removing all the mimeTypes from the setContentMimeType list. For whatever reason, GIF insertion still worked. This leads me to believe that calling setContentMimeType doesn't do anything... I actually pulled it from Anton here: #20796 (comment).

Now, I am rebuilding the engine with only 1 mime type (png) to see if i can still insert GIFs... if that still works, I'm going to try and remove the setContentMimeType call entirely and see what happens...

@jonahwilliams jonahwilliams removed the tool Affects the "flutter" command-line tool. See also t: labels. label Jul 29, 2021
@justinmc
Copy link
Contributor

@zlshames Sorry I missed your question before, but no, unfortunately you must rebuild the engine with ninja as in the compiling the engine wiki page each time you make a change, the run your Flutter app with --local-engine=.... If you've already built it once, it should be much faster than the first time.

@zlshames
Copy link

@zlshames Sorry I missed your question before, but no, unfortunately you must rebuild the engine with ninja as in the compiling the engine wiki page each time you make a change, the run your Flutter app with --local-engine=.... If you've already built it once, it should be much faster than the first time.

That's what I figured, and have been doing. No worries! Been busy working on our latest app release, but will get back to working on this PR (hopefully) shortly

@justinmc
Copy link
Contributor

Sounds good, happy to take a look whenever you have time to update it 👍

@flutter-dashboard
Copy link

This pull request executed golden file tests, but it has not been updated in a while (20+ days). Test results from Gold expire after as many days, so this pull request will need to be updated with a fresh commit in order to get results from Gold.

For more guidance, visit Writing a golden file test for package:flutter.

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

@Hixie
Copy link
Contributor

Hixie commented Nov 30, 2021

@zlshames No pressure, just curious if you've had a chance to look at this?

@zlshames
Copy link

zlshames commented Dec 1, 2021

@zlshames No pressure, just curious if you've had a chance to look at this?

Apologies! I have not forgotten about it. Just am super busy with the holiday season right now. And previously, with working on updates to a different part of the app that uses this custom code. I know @tneotia updated our code, I just need to test setting custom MIME Types so users can select which they want to handle (vs. just automatically handling every type). I would like to do it by the end of the year.

If no one sees an update by me nearing Christmas, give me a ping and I'll try to get on it :)

@mrorabau
Copy link

mrorabau commented Jan 5, 2022

@zlshames I hope you had a nice holiday. Just checking in, I know lots of people are looking forward to this PR, and it seems that you were fairly close with only a few failing checks. Any update or ETA?

@zlshames
Copy link

@justinmc I made a bunch of changes and comments to your comments. Let me know if there is anything else.

@zlshames
Copy link

Alright, someone might have to help me out here. Admittedly, I'm not a full-time Dart developer. I can't for the life of me figure out what's wrong in our template comment's sample code. It's saying that it's receiving an unexpected character, which I would have assumed would have been due to the unescaped single-quote, but it still fails when the quote is escaped.

Copy link
Contributor

@justinmc justinmc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zlshames I think your dartpad example needs to be a snippet sample (see below) instead, since it won't work on the web. Let me know if that doesn't fix the analyzer errors.

void performAction(TextInputAction action);

/// Notify client about new content insertion from Android keyboard.
void commitContent(Map<String, dynamic> content);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to split this out of TextInputClient, maybe onto its own method channel? We're trying to keep TextInputClient slim to avoid breaking people that have implemented it. I'm working on a better set up for this.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is the preferred location?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So there has been some discussion about this, we now have a design doc. That doc is still up for debate, so feel free to chime in if you have opinions.

If it makes sense for commitContent to be separate from TextInputClient, then I think I would recommend creating an entirely new method channel in system_channels.dart. But does it make sense on its own, or is it inherently tied to TextInputClient? Maybe it's not possible to use commitContent without opening an input connection and everything first via TextInputClient.

CC @tneotia

/// onContentCommitted: (CommittedContent data) async {
/// if (data.mimeType == "image/gif" && data.data != null) {
/// //handle Uint8List (e.g. upload to server, display a MemoryImage, etc)
/// ...
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using @tool dartpad above means that this sample will actually be turned into a live example on the docs website, so this code needs to compile. You'll have to either comment out this ... and/or make it a @tool snippet to just show the code in a codeblock on the docs site.

Actually this won't work on the web, right? So it should probably be a snippet.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, you'll want to use {@tool sample} instead of {@tool snippet}, since it's a complete example that just doesn't run on the web. sample API doc samples are non-dartdoc full application examples that can be compiled and run on the platforms that support them. snippet API doc samples are partial examples that don't compile/run as an application.

Also, you'll want these to be non-inline code without using a template: templates are no longer supported. Take a look at https://github.com/flutter/flutter/blob/master/examples/api/README.md for more details.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the clarification!

@justinmc
Copy link
Contributor

@zlshames @tneotia Are you all still available to help get this PR merged? Would you be able to merge in master (or rebase) to update the branch and get the checks to run again? It looks like the analyzer failure is an infrastructure thing.

@zlshames
Copy link

zlshames commented Aug 10, 2022

@justinmc I could have sworn we discussed needing to refactor it and decouple it from the TextInputClient code. Our other PR was closed due to inactivity as well as that they weren't sure if it was how they would want to architect the solution. @tneotia correct me if I'm misremembering. We can always reopen the PR, but I just don't think this is the official solution they wanted.

If anyone from Google wants to chime in and offer their opinion, we are open! Obviously, we would love to get this native in Flutter. But at the same time, I'm not sure where to refactor it so that it's decoupled

@justinmc
Copy link
Contributor

After catching back up on the code here and in the engine PR, I think the current approach with TextInputClient is already the best option 👍 . @zlshames I think if you re-open the engine PR and get both PRs to pass CI, then I can re-review and get this moving again without any major changes.

There was a strong argument made for decoupling this from TextInputClient here, but I think that this feature is fundamentally a part of TextInputClient. Sending rich image content from the platform keyboard requires TextInputClient's input connection. Very custom behavior is still possible (say for someone writing a custom text editor) by overriding commitContent.

In general we should work to decouple our platform channel methods as discussed in my design doc, but in this case the logic belongs in TextInputClient.

CC @LongCatIsLooong, I believe you and I discussed this a few weeks ago and came to the same conclusion.

@tneotia
Copy link
Contributor Author

tneotia commented Aug 19, 2022

I am going back to school shortly, but I will definitely have some time next week to mess with this and bring it back up to date. I may re-PR this just because there are way, way, way too many commits here.

Building our app with a custom flutter engine has been.... well... not great. We'd like to get this into a stable Flutter release ASAP! :)

@justinmc
Copy link
Contributor

Closing this in favor of #110052. Will leave a review over there soon. Thanks!

@justinmc justinmc closed this Aug 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

f: cupertino flutter/packages/flutter/cupertino repository f: material design flutter/packages/flutter/material repository. framework flutter/packages/flutter repository. See also f: labels.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants