cc @godofredoc @CaseyHillers @christopherfujino how do i enable something like this for the LUCI shards as well?

@Hixie should this be run on all of the Flutter checks? Would it make sense to add this to bots/test.dart and run it at the end of every recipe?

bots/test.dart isn't run for every shard, is it? e.g. I don't think it gets run for the customer_testing shard.
For Cirrus I just added it as a line that gets run at the end of every Cirrus shard regardless.

It's not, but we could add it to the missing builds. That test could be run at the end of every bots/test.dart invocation.

If we're adding something, easier to just add git diff --exit-code :-)
My question is more about how to do that. Presumably long-term the move-of-test-descriptions-to-the-repo-under-test will make this trivial but right now we have to edit a recipe somewhere?

Yes, you'll need to add steps to the recipes. You can run the git command as a shell step in those recipes until we support go/flutter-single-recipe-builds

The recipes repo is hosted on git-on-borg at https://flutter.googlesource.com/recipes/

We have been calling these use case ad hoc validations and we have been implementing them in the following way:

1. Create a script in this location: https://cs.opensource.google/flutter/recipes/+/master:recipe_modules/adhoc_validation/resources/
2. Then add the script name in this list: https://cs.opensource.google/flutter/recipes/+/master:recipe_modules/adhoc_validation/api.py;l=18

If this is something that won't run as an independent task and instead we want to run it on every task we will need to add it as a step to the relevant recipes:

• flutter
• devicelab

The idea is to run it at the end of every job that ran anything, to make sure the test didn't affect the git workspace.

In any case it looks like it might be simpler to just wait until the recipes and such aren't written in python in another repo... I'm a little worried that for example if I made the change here it would catch a bug and break everything (unless there's a way to test the recipe changes ahead of landing them?).

The idea is to run it at the end of every job that ran anything, to make sure the test didn't affect the git workspace.

In any case it looks like it might be simpler to just wait until the recipes and such aren't written in python in another repo... I'm a little worried that for example if I made the change here it would catch a bug and break everything (unless there's a way to test the recipe changes ahead of landing them?).

Recipe changes are tested on presubmit using builds from framework, devicelab hostonly and engine. Here is an example: https://ci.chromium.org/p/flutter/builders/try/recipes-with-led/b8853549532636159696?

I've filed #78265 for now, going to close this. I'm unlikely to get to this in the near future unfortunately.