-
Notifications
You must be signed in to change notification settings - Fork 29.7k
MigrateUtils and MigrateManifest classes #101937
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!). If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix? Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
importing dart:io is banned from the tool. Instead, use https://github.com/flutter/flutter/blob/master/packages/flutter_tools/lib/src/base/io.dart
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For testing, don't use dart:io's Process.run, but instead the ProcessManager from the context: https://github.com/flutter/flutter/blob/master/packages/flutter_tools/lib/src/context_runner.dart#L306. In addition, direct inject the process manager via the constructor, so we can unit test it with a FakeProcessManager.
The IOSDeploy class is an example that does this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, for direct injection, these static methods should probably be moved to instance methods.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, thanks for the feedback. I figured I would have to change the static utils class setup, this sounds good, thanks for the example to reference.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| // Clones a copy of the flutter repo into the destination directory. Returns false if unsucessful. | |
| /// Clones a copy of the flutter repo into the destination directory. Returns false if unsucessful. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need to clone a new copy of the repo?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it because we will be calling flutter create from a previous commit?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, we require the full flutter tool from previous versions to be able to accurately reproduce reference apps to migrate off of. Unfortunately, the only way to do this is to clone flutter. The migrate tool isn't run frequently, so the overhead here is more palatable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why --shallow-exclude=v1.0.0?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And why --single-branch?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These parameters greatly reduce the time it takes to clone the repo by using a shallow clone of the master branch. We don't claim to support migration apps created before 1.0, so this hard limit significantly improves performance of the command.
I'm actually still experimenting with the --single-branch option, so I may remove that for now.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not convinced that shallow cloning is faster (I benchmarked it locally and they were exactly the same, which seems surprising), and the Flutter CLI tool is not guaranteed to work in a shallow clone environment (certain operations depend on access to the git history, I'm not sure I can guarantee flutter create won't crash in some situations).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Following up, I don't think this is the correct way to shallow clone; I actually did get a clone speedup from doing git clone $REMOTE --depth 1, however, again, I wouldn't recommend it as that voids your warranty on the Flutter CLI tool.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, i'll remove this for now then until we can come up with a better way for performance improvements.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit. can you construct this as a single list literal using collection for and collection if? something like:
final List<String> cmdArgs = <String>[
'create',
if (!legacyNameParameter) '--project-name=$name',
'--android-language=$androidLanguage',
'--ios-language=$iosLanguage',
if (platforms.isNotEmpty) '--platforms=${platforms.join(',')}',
'--no-pub',
if (legacyNameParameter) name,
else outputDirectory,
];There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rather than recursively calling this function, could you first check the Flutter version, then configure the parameters accordingly the first time?
I also worry that we could recurse infinitely here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The flutter version tag is not always reliable in representing what parameters are available as it is not fine enough, I also tried using the hash to see if the boundaries between commits that add parameters can reliably obtained, but depending on the branch, this method was also not reliable due to various branches and releases.
This technique is the most robust way to approach it. Since we are effectively limiting the backwards compatibility to version v1.0.0, this should be able to self-pare its way down to the minimal runnable command. I can add a counter also to restrict this from infinite looping if you would like.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is not uncommon for projects to be created off of a custom branch with a commit hash that is not on the master branch. This is especially true for our first party sample projects which were largely created with squashed local commits.
In these cases, we can't really say what version of flutter it was created with or the version we are using in reference to the tags and if we get this wrong, the command will fail and we would have to fall back to this system anyways.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why only limit to 1.0? Where does the --platforms flag first appear? If it helps to delete this logic to instead require creation after 2.0 for migration to work, I think that would be totally reasonable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It feels like this whole class should just be an enum, and the base constructor case a different data type.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class is meant to represent the output of a git diff-filescommand including stuff like exit code. I've changed the is into an enum though!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about making MergeResult an abstract class, and implement it with either a StringMergeResult or BinaryMergeResult
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
packages/flutter_tools/test/general.shard is for unit tests, which should not use the LocalFileSystem. We probably do want to call git to verify these commands work, so I recommend moving this to packages/flutter_tools/test/integration.shard.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
don't use dart:io
| import 'dart:io'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I probably shouldn't review this test file further until the other changes have been made. Let me know when I should take another look!
|
@christopherfujino This is ready for another round, whenever you have the time. Thanks! |
I'll take a look tomorrow, thanks for the ping! |
fe9c124 to
21926b0
Compare
Fix test Fix windows Platforms Kick tests Platforms windows
| import 'migrate_result.dart'; | ||
| import 'migrate_utils.dart'; | ||
|
|
||
| /// Represents the mamifest file that tracks the contents of the current |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| /// Represents the mamifest file that tracks the contents of the current | |
| /// Represents the manifest file that tracks the contents of the current |
| valid = valid && (addedFilesYaml is YamlList || addedFilesYaml == null); | ||
| valid = valid && (deletedFilesYaml is YamlList || deletedFilesYaml == null); | ||
| if (!valid) { | ||
| throwToolExit('Invalid .migrate_manifest file in the migrate working directory. Entry is not a Yaml list. Fix the manifest or abandon the migration and try again.', exitCode: 1); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the user ever hand-editing this, or is it just the tool? If the former, maybe we should try to help them by telling them what exactly made this invalid, if the latter maybe we should just throw an exception so we get crash reporting because it's a bug in the tool.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file is not meant to be hand-edited. If this file becomes invalid, it is from the user accidentally modifying it on their own without our blessing.
| throwToolExit('Invalid .migrate_manifest file in the migrate working directory. File is not a Yaml map.', exitCode: 1); | ||
| } | ||
| final YamlMap map = yamlContents; | ||
| bool valid = map.containsKey('merged_files') && map.containsKey('conflict_files') && map.containsKey('added_files') && map.containsKey('deleted_files'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can you make these keys constants at the top level scope of the file?
| } | ||
| } | ||
|
|
||
| /// Returns true if the file does not contain any git conflit markers. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| /// Returns true if the file does not contain any git conflit markers. | |
| /// Returns true if the file does not contain any git conflict markers. |
|
@christopherfujino Ready for another pass! Thanks! |
christopherfujino
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
|
Idk why, but multiple times on this PR I could have sworn I submitted feedback, but then apparently I didn't... |
zanderso
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
DBC
I understand this is probably part of a multi-PR sequence, but I'd still expect us to follow our usual code hygiene practices. In particular, this PR has a bunch of dead code checked in. I'd expect that even if the code we check in isn't used directly, yet, it should at least be covered by tests.
| // Use of this source code is governed by a BSD-style license that can be | ||
| // found in the LICENSE file. | ||
|
|
||
| // import 'package:process/process.dart'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
DBC:
Please do not check in commented out code.
| } | ||
| } | ||
|
|
||
| Future<bool> gitRepoExists(String projectDirectory, Logger logger, MigrateUtils migrateUtils) async { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this called somewhere? If not, it should be held back and only checked in once it's used/tested somewhere.
| return false; | ||
| } | ||
|
|
||
| Future<bool> hasUncommittedChanges(String projectDirectory, Logger logger, MigrateUtils migrateUtils) async { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ditto comment
| } | ||
|
|
||
| /// Prints a command to logger with appropriate formatting. | ||
| void printCommandText(String command, Logger logger) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is only called once, above in gitRepoExists. It could be a library private function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will soon be used in all of the subcommands
| } | ||
|
|
||
| /// Returns the manifest file given a migration workind directory. | ||
| static File getManifestFileFromDirectory(Directory workingDir) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is only used in this class. Can it be library-private?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will soon be used in all of the subcommands
|
|
||
| /// Returns true if the file does not contain any git conflict markers. | ||
| bool _conflictsResolved(String contents) { | ||
| if (contents.contains('>>>>>>>') && contents.contains('=======') && contents.contains('<<<<<<<')) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should probably also fail if the markers have been partially removed. For example if the user removes the '>>>' and '===', but leaves in the '<<<'. Given that, I think this can be simplified as:
@visibleForTesting
bool conflictsResolved(String contents) {
final bool hasMarker = contents.contains('>>>>>>>') ||
contents.contains('=======') ||
contents.contains('<<<<<<<');
return !hasMarker;
}This should also have a unit test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ahh, he actually had it that way, and I asked him to use &&. But this makes sense, my bad.
| /// Returns true if the migration working directory has all conflicts resolved and prints the migration status. | ||
| /// | ||
| /// The migration status printout lists all added, deleted, merged, and conflicted files. | ||
| bool checkAndPrintMigrateStatus(MigrateManifest manifest, Directory workingDir, {bool warnConflict = false, Logger? logger}) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not called or tested anywhere. Please don't check in dead / untested code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why only limit to 1.0? Where does the --platforms flag first appear? If it helps to delete this logic to instead require creation after 2.0 for migration to work, I think that would be totally reasonable.
| bool exit = true, | ||
| bool silent = false | ||
| }) { | ||
| if (!allowedExitCodes.contains(result.exitCode) && !allowedExitCodes.contains(-1)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Flip the sense, and unindent.
| } | ||
|
|
||
| /// Returns true if the file does not contain any git conflit markers. | ||
| bool conflictsResolved(String contents) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I haven't studied how this code is organized deeply, but I'm sure that it's likely possible to avoid duplicating methods like this across libraries.
|
Addressed Zach's comments in #103466 |

MigrateUtils is the utility class that handles the git and flutter commands used by the migrate tool.
MigrateManifest is a working metadata file that tracks the files to be changed by the migrate tool.
This code is inaccessible by the rest of the tool as of now.