Skip to content

Conversation

@GaryQian
Copy link
Contributor

@GaryQian GaryQian commented Apr 14, 2022

MigrateUtils is the utility class that handles the git and flutter commands used by the migrate tool.

MigrateManifest is a working metadata file that tracks the files to be changed by the migrate tool.

This code is inaccessible by the rest of the tool as of now.

@flutter-dashboard flutter-dashboard bot added the tool Affects the "flutter" command-line tool. See also t: labels. label Apr 14, 2022
@flutter-dashboard
Copy link

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For testing, don't use dart:io's Process.run, but instead the ProcessManager from the context: https://github.com/flutter/flutter/blob/master/packages/flutter_tools/lib/src/context_runner.dart#L306. In addition, direct inject the process manager via the constructor, so we can unit test it with a FakeProcessManager.

The IOSDeploy class is an example that does this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, for direct injection, these static methods should probably be moved to instance methods.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, thanks for the feedback. I figured I would have to change the static utils class setup, this sounds good, thanks for the example to reference.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Clones a copy of the flutter repo into the destination directory. Returns false if unsucessful.
/// Clones a copy of the flutter repo into the destination directory. Returns false if unsucessful.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to clone a new copy of the repo?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it because we will be calling flutter create from a previous commit?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we require the full flutter tool from previous versions to be able to accurately reproduce reference apps to migrate off of. Unfortunately, the only way to do this is to clone flutter. The migrate tool isn't run frequently, so the overhead here is more palatable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why --shallow-exclude=v1.0.0?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And why --single-branch?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These parameters greatly reduce the time it takes to clone the repo by using a shallow clone of the master branch. We don't claim to support migration apps created before 1.0, so this hard limit significantly improves performance of the command.

I'm actually still experimenting with the --single-branch option, so I may remove that for now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not convinced that shallow cloning is faster (I benchmarked it locally and they were exactly the same, which seems surprising), and the Flutter CLI tool is not guaranteed to work in a shallow clone environment (certain operations depend on access to the git history, I'm not sure I can guarantee flutter create won't crash in some situations).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following up, I don't think this is the correct way to shallow clone; I actually did get a clone speedup from doing git clone $REMOTE --depth 1, however, again, I wouldn't recommend it as that voids your warranty on the Flutter CLI tool.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, i'll remove this for now then until we can come up with a better way for performance improvements.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit. can you construct this as a single list literal using collection for and collection if? something like:

final List<String> cmdArgs = <String>[
  'create',
  if (!legacyNameParameter) '--project-name=$name',
  '--android-language=$androidLanguage',
  '--ios-language=$iosLanguage',
  if (platforms.isNotEmpty) '--platforms=${platforms.join(',')}',
  '--no-pub',
  if (legacyNameParameter) name,
  else outputDirectory,
];

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than recursively calling this function, could you first check the Flutter version, then configure the parameters accordingly the first time?

I also worry that we could recurse infinitely here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flutter version tag is not always reliable in representing what parameters are available as it is not fine enough, I also tried using the hash to see if the boundaries between commits that add parameters can reliably obtained, but depending on the branch, this method was also not reliable due to various branches and releases.

This technique is the most robust way to approach it. Since we are effectively limiting the backwards compatibility to version v1.0.0, this should be able to self-pare its way down to the minimal runnable command. I can add a counter also to restrict this from infinite looping if you would like.

Copy link
Contributor

@christopherfujino christopherfujino Apr 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flutter version tag is not always reliable in representing what parameters are available as it is not fine enough

Can't we just determine the first tagged version that had a specific change? GitHub will even do this for you if you look at a specific commit in the web UI:
The bold tag on the left is the latest release with this commit, the tag on the right is the earliest tag with this commit, which we want.

Copy link
Contributor Author

@GaryQian GaryQian Apr 26, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not uncommon for projects to be created off of a custom branch with a commit hash that is not on the master branch. This is especially true for our first party sample projects which were largely created with squashed local commits.

In these cases, we can't really say what version of flutter it was created with or the version we are using in reference to the tags and if we get this wrong, the command will fail and we would have to fall back to this system anyways.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why only limit to 1.0? Where does the --platforms flag first appear? If it helps to delete this logic to instead require creation after 2.0 for migration to work, I think that would be totally reasonable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels like this whole class should just be an enum, and the base constructor case a different data type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class is meant to represent the output of a git diff-filescommand including stuff like exit code. I've changed the is into an enum though!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about making MergeResult an abstract class, and implement it with either a StringMergeResult or BinaryMergeResult

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

packages/flutter_tools/test/general.shard is for unit tests, which should not use the LocalFileSystem. We probably do want to call git to verify these commands work, so I recommend moving this to packages/flutter_tools/test/integration.shard.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't use dart:io

Suggested change
import 'dart:io';

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I probably shouldn't review this test file further until the other changes have been made. Let me know when I should take another look!

@GaryQian
Copy link
Contributor Author

@christopherfujino This is ready for another round, whenever you have the time. Thanks!

@christopherfujino
Copy link
Contributor

@christopherfujino This is ready for another round, whenever you have the time. Thanks!

I'll take a look tomorrow, thanks for the ping!

@GaryQian GaryQian force-pushed the migrateutils branch 2 times, most recently from fe9c124 to 21926b0 Compare April 22, 2022 20:47
Fix test

Fix windows

Platforms

Kick tests

Platforms

windows
import 'migrate_result.dart';
import 'migrate_utils.dart';

/// Represents the mamifest file that tracks the contents of the current
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Represents the mamifest file that tracks the contents of the current
/// Represents the manifest file that tracks the contents of the current

valid = valid && (addedFilesYaml is YamlList || addedFilesYaml == null);
valid = valid && (deletedFilesYaml is YamlList || deletedFilesYaml == null);
if (!valid) {
throwToolExit('Invalid .migrate_manifest file in the migrate working directory. Entry is not a Yaml list. Fix the manifest or abandon the migration and try again.', exitCode: 1);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the user ever hand-editing this, or is it just the tool? If the former, maybe we should try to help them by telling them what exactly made this invalid, if the latter maybe we should just throw an exception so we get crash reporting because it's a bug in the tool.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is not meant to be hand-edited. If this file becomes invalid, it is from the user accidentally modifying it on their own without our blessing.

throwToolExit('Invalid .migrate_manifest file in the migrate working directory. File is not a Yaml map.', exitCode: 1);
}
final YamlMap map = yamlContents;
bool valid = map.containsKey('merged_files') && map.containsKey('conflict_files') && map.containsKey('added_files') && map.containsKey('deleted_files');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you make these keys constants at the top level scope of the file?

}
}

/// Returns true if the file does not contain any git conflit markers.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Returns true if the file does not contain any git conflit markers.
/// Returns true if the file does not contain any git conflict markers.

@GaryQian GaryQian requested a review from christopherfujino May 6, 2022 22:32
@GaryQian
Copy link
Contributor Author

GaryQian commented May 9, 2022

@christopherfujino Ready for another pass! Thanks!

Copy link
Contributor

@christopherfujino christopherfujino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@christopherfujino
Copy link
Contributor

Idk why, but multiple times on this PR I could have sworn I submitted feedback, but then apparently I didn't...

Copy link
Member

@zanderso zanderso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DBC

I understand this is probably part of a multi-PR sequence, but I'd still expect us to follow our usual code hygiene practices. In particular, this PR has a bunch of dead code checked in. I'd expect that even if the code we check in isn't used directly, yet, it should at least be covered by tests.

// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.

// import 'package:process/process.dart';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DBC:

Please do not check in commented out code.

}
}

Future<bool> gitRepoExists(String projectDirectory, Logger logger, MigrateUtils migrateUtils) async {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this called somewhere? If not, it should be held back and only checked in once it's used/tested somewhere.

return false;
}

Future<bool> hasUncommittedChanges(String projectDirectory, Logger logger, MigrateUtils migrateUtils) async {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto comment

}

/// Prints a command to logger with appropriate formatting.
void printCommandText(String command, Logger logger) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only called once, above in gitRepoExists. It could be a library private function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will soon be used in all of the subcommands

}

/// Returns the manifest file given a migration workind directory.
static File getManifestFileFromDirectory(Directory workingDir) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only used in this class. Can it be library-private?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will soon be used in all of the subcommands


/// Returns true if the file does not contain any git conflict markers.
bool _conflictsResolved(String contents) {
if (contents.contains('>>>>>>>') && contents.contains('=======') && contents.contains('<<<<<<<')) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably also fail if the markers have been partially removed. For example if the user removes the '>>>' and '===', but leaves in the '<<<'. Given that, I think this can be simplified as:

@visibleForTesting
bool conflictsResolved(String contents) {
  final bool hasMarker = contents.contains('>>>>>>>') ||
                         contents.contains('=======') ||
                         contents.contains('<<<<<<<');
  return !hasMarker;
}

This should also have a unit test.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh, he actually had it that way, and I asked him to use &&. But this makes sense, my bad.

/// Returns true if the migration working directory has all conflicts resolved and prints the migration status.
///
/// The migration status printout lists all added, deleted, merged, and conflicted files.
bool checkAndPrintMigrateStatus(MigrateManifest manifest, Directory workingDir, {bool warnConflict = false, Logger? logger}) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not called or tested anywhere. Please don't check in dead / untested code.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why only limit to 1.0? Where does the --platforms flag first appear? If it helps to delete this logic to instead require creation after 2.0 for migration to work, I think that would be totally reasonable.

bool exit = true,
bool silent = false
}) {
if (!allowedExitCodes.contains(result.exitCode) && !allowedExitCodes.contains(-1)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flip the sense, and unindent.

}

/// Returns true if the file does not contain any git conflit markers.
bool conflictsResolved(String contents) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't studied how this code is organized deeply, but I'm sure that it's likely possible to avoid duplicating methods like this across libraries.

@GaryQian
Copy link
Contributor Author

Addressed Zach's comments in #103466

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Aug 30, 2022
engine-flutter-autoroll added a commit to engine-flutter-autoroll/plugins that referenced this pull request Aug 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tool Affects the "flutter" command-line tool. See also t: labels.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants