Invalidate the build based on external dependency changes #967

Activity

1. nshahan

   added

   type-enhancementA request for a change that isn't a bugA request for a change that isn't a bug

   package:build_runner

   on Feb 6, 2018
2. natebosch commented on Feb 6, 2018

   

   natebosch

   on Feb 6, 2018

   Member

   More actions

   Two possibilities I can think of:

   • An "environment check" (Allow builder authors to define custom "environment checks" #961) that checks the analyzer version and can somehow communicate that the summaries are invalid (maybe change an artificial "input" to the summary builders?)
   • When the builder runs catch this specific exception and somehow communicate back that the inputs it read are invalid.

   Either of these would require new APIs.
3. jakemac53 commented on Feb 6, 2018

   

   jakemac53

   on Feb 6, 2018

   Contributor

   More actions

   This is more a general issue with using external dependencies I think. Today if you are using any external executable and invoking it inside a scratch space we don't know if/when that external executable changes.

   In this case, its even more complicated because I believe the the dartanalyzer executable is actually just a shell script that runs the analyzer snapshot with the dart vm. We would need to track the dart vm itself, plus the snapshot and shell script. Computing hashes of some of these larger files would also likely be slow.

   We could also just invalidate builds whenever the dart sdk version changes (add that as an encoded field on the asset graph). This would solve things for analyzer/dartdevc but not other external executables.
4. matanlurey commented on Feb 7, 2018

   

   matanlurey

   on Feb 7, 2018

   Last edited by matanlurey

   Contributor

   More actions

   Could builders expose an optional externalDependencies?

   class UsesDartSdkBuilder extends Builder {
     // Used as part of the Builder digest?
     @override
     Future<List<String>> get externalDependencies async => [
       _getDartSdkVersion(),
     ];
   }

   Optionally, I'd like to see some sort of easily-consumable pattern for a builder that executes via IPC so others can write stuff using other external dependencies more easily, maybe that can simplify this?

   👀React with 👀1Reacted by Frenco
5. matanlurey

   changed the title [-]Make sure summaries are invalidated between SDK and analyzer version changes.[/-] [+]Invalidate the build based on external dependency changes[/+] on Feb 7, 2018
6. matanlurey

   added

   P1A high priority bug; for example, a single project is unusable or has many test failuresA high priority bug; for example, a single project is unusable or has many test failures

   on Feb 7, 2018
7. jakemac53

   closed this as completedin #1009on Feb 13, 2018
8. jakemac53

   reopened this on Feb 13, 2018
9. jakemac53 commented on Feb 13, 2018

   

   jakemac53

   on Feb 13, 2018

   Contributor

   More actions

   leaving this open, we do track dart sdk versions now but a more general purpose thing would be better
10. natebosch commented on Jun 5, 2018

    

    natebosch

    on Jun 5, 2018

    Member

    More actions

    After some discussion we think we could solve this by allowing you to configure an environment_check per builder definition in build.yaml. This would be the name of a top-level or static method conforming to the signature Digest Function(). We'd call this once per build (if that builder has any actions possible). We'd guarantee that the log variable from package:build is available, so you could log.severe if there is a missing binary or anything like that, and the Digest you return would be an expression of whatever dependencies you have (tool version information, files you might read non-hermetically with dart:io once per build, etc).

    We could migrate our tools that are dependent on the SDK version to use this and be a bit more optimal than today. It would also satisfy both the environment check and build invalidation problems.
11. jakemac53 commented on Jun 5, 2018

    

    jakemac53

    on Jun 5, 2018

    Last edited by jakemac53

    Contributor

    More actions

    Presumably we would not attempt the build if you log.severe? Or should we require a throw? Or use the setting for --fail-on-severe?
12. natebosch commented on Jun 5, 2018

    

    natebosch

    on Jun 5, 2018

    Member

    More actions

    I think we'd not attempt the build (or maybe attempt the rest of the build but mark the actions using that builder as failures without trying to run them) with either log.severe or a throw. I'd not bother honoring the setting for --fail-on-severe since it's a new API and we want to drop that flag.
13. jakemac53 commented on Jun 5, 2018

    

    jakemac53

    on Jun 5, 2018

    Contributor

    More actions

    or maybe attempt the rest of the build but mark the actions using that builder as failures without trying to run them

    Interesting idea... we could also just require the user to disable the builder explicitly (but would need global config). My only real concern with either of those approaches is you may very well get a lot of cascading build failures from steps that try to read those files.
14. natebosch commented on Jun 5, 2018

    

    natebosch

    on Jun 5, 2018

    Member

    More actions

    Yeah the tradeoff is between the two sharp edge scenarios: "Why did you bother running this build when it's all going to fail" vs "Why can't I just run the build and get DDC output to test things even though this inconsequential other thing is going to fail"
15. frencojobs commented on Oct 30, 2020

    

    frencojobs

    on Oct 30, 2020

    More actions

    Could builders expose an optional externalDependencies?

    class UsesDartSdkBuilder extends Builder {
      // Used as part of the Builder digest?
      @override
      Future<List<String>> get externalDependencies async => [
        _getDartSdkVersion(),
      ];
    }

    Optionally, I'd like to see some sort of easily-consumable pattern for a builder that executes via IPC so others can write stuff using other external dependencies more easily, maybe that can simplify this?

    Any update on the API like this? I'm building a package that will generate source code for a dart file using the data from the .env file. So, I need to invalidate the cache if not only the .dart file but also when the .env file is changed. I really need this API. I'm having to manually clean the snapshot when the .env file is updated.
16. jakemac53 commented on Oct 30, 2020

    

    jakemac53

    on Oct 30, 2020

    Contributor

    More actions

    Does this .env file live outside the package?
17. frencojobs commented on Oct 30, 2020

    

    frencojobs

    on Oct 30, 2020

    More actions

    Yes, it does. But with #2851 it won't be a problem, right?
18. jakemac53 commented on Oct 30, 2020

    

    jakemac53

    on Oct 30, 2020

    Contributor

    More actions

    That PR doesn't allow you to reach outside the package, it just allows you to expose files outside the lib dir of your package to other packages.
19. frencojobs commented on Oct 30, 2020

    

    frencojobs

    on Oct 30, 2020

    More actions

    Okay, I think I misunderstood the PR then. But may I know why does it matter? Because, as long as there is an API like the suggested one, I can still make a function using dart:io to read the file, which is how I currently implemented.

    I just need a way to invalidate the snapshot once the specified asset has been modified.
20. jakemac53 commented on Oct 30, 2020

    

    jakemac53

    on Oct 30, 2020

    Contributor

    More actions

    Right, if you use dart:io then the build system doesn't track that and won't invalidate things. You will have to somehow manage that.

    In an ideal world this would not mean deleting the entire build cache, as only a few actions are likely affected, and also ideally the user themselves wouldn't have to do something manually here.
21. frencojobs commented on Oct 31, 2020

    

    frencojobs

    on Oct 31, 2020

    More actions

    Okay, I think I see your point. If I could read the value of .env with buildStep.readAsString, it would solve the problem. But in return, I might have to move the .env into the lib so that it can be read with an assetId. Right?

    But is it possible to have a workaround for reading the .env file outside of the lib? It really can't be moved into the lib.

    👍React with 👍1Reacted by zacharypuulsedev
22. jakemac53 commented on Nov 2, 2020

    

    jakemac53

    on Nov 2, 2020

    Contributor

    More actions

    There is not a real solution for what you want to do today, the closest thing would be using a symlink to the file in your package, but I don't think the file watcher would work so you would have to restart the build whenever that file changed, at which point we should see that it has been modified I think.

    Note that once the pull request you linked above is published you could move the file outside of lib but it would still have to live under the package directory.

    One potential workaround although I have no idea if it would be possible for your use case, is to make the .env file be a part of an actual package (but not your root package). You would then depend on that package wherever you want to use that file, using a path override:

    dependencies:
      env:
        path: /some/place/env

    And then you could construct an AssetId to read that file?
23. davidmorgan commented on Jul 7, 2025

    

    davidmorgan

    on Jul 7, 2025

    Contributor

    More actions

    Closing as stale.
24. davidmorgan

    closed this as completedon Jul 7, 2025