Skip to content

[cocoon] Not reporting "success" after retrying a build. #116226

Closed

Description

@ditman

Cocoon is not reporting success for manually retried LUCI builds, for example in this run:

https://github.com/flutter/plugins/runs/9782441143

I retried manually here through the Cocoon "RERUN" button. That set the Github status "in progress" again, but once the build successfully passed, the Github issue never went green again. Instead, it's "updating" every few minutes, but it never goes to green.

Flutter Ecosystem tooling relies on the tree being all green for CI automation to keep going. For Flutter Rolls this is not a big issue, because they never trigger a release of any package, but if a feature PR lands and ends up in this state, our release procedure must be performed manually (we have no automation override), and it's sub-optimal.

Can Cocoon report successful retried builds to Github as a success? Is this not a Cocoon problem and are we doing something wrong?

Activity

  1. drewroengoogle commented on Nov 30, 2022

    @drewroengoogle
    Contributor

    For my understanding, did Cocoon report successful retried builds to Github in the past and is failing to now, or is this a request for an additional feature?

    /cc @CaseyHillers potentially, who would know more than me

  2. CaseyHillers commented on Nov 30, 2022

    @CaseyHillers
    Contributor

    @ditman can you screenshot what you're clicking? I see "Retry Build" in that link, which does not work. You need to use flutter-dashboard.appspot.com to retry postsubmit builds.

    We could add support for the postsubmit GitHub checks rerun functionality as well.

  3. ditman commented on Nov 30, 2022

    @ditman
    MemberAuthor

    can you screenshot what you're clicking?

    @CaseyHillers I clicked this button:

    Screen Shot 2022-11-30 at 2 37 18 PM

    That updated the github status to "in progress" (as expected), and kicked off the LUCI build, but when the build finished successfully, the success was not reported to the github run.

    (If you take a look at the run now, github thinks it started 5 minutes ago (??) This seems to attempt to refresh every 5 or 6 minutes?)

  4. drewroengoogle commented on Dec 1, 2022

    @drewroengoogle
    Contributor

    I talked with Casey a bit and it sounds like this may be a feature that Keyong was working on that is a part of moving the plugin team to LUCI.

    @stuartmorgan Do you know if this feature would be a part of that work? (Asking since it looks like you are taking over that work from Keyong)

  5. ditman commented on Dec 1, 2022

    @ditman
    MemberAuthor

    @drewroengoogle how is this different from Cocoon reporting build successes when I don't have to restart the build by hand? That bit works fine :/

  6. CaseyHillers commented on Dec 1, 2022

    @CaseyHillers
    Contributor

    @drewroengoogle how is this different from Cocoon reporting build successes when I don't have to restart the build by hand? That bit works fine :/

    Up until recently, Cocoon never supported GitHub checks for postsubmit. This sounds like a bug as the feature is still in development.

  7. ditman commented on Dec 2, 2022

    @ditman
    MemberAuthor

    Roger that! Thanks for the details @CaseyHillers!

  8. drewroengoogle commented on Dec 2, 2022

    @drewroengoogle
    Contributor

    @CaseyHillers Does it make sense to keep this issue open but move it out of the queue based the info you've provided?

  9. stuartmorgan-g commented on Dec 3, 2022

    @stuartmorgan-g
    Contributor

    @stuartmorgan Do you know if this feature would be a part of that work?

    I wasn't aware that there was any remaining issue with the post-submit checkin; as far as I knew it was done. The expectation we set was was that post-submit reporting from LUCI was a precondition for migration of any CI to LUCI, so I'm presuming Keyong didn't know that this didn't work either.

    (Asking since it looks like you are taking over that work from Keyong)

    The plan was for Keyong to come back to finish the migration; there were a few individual tests I was considering migrating in the meantime, but that's it.

    This issue puts us in a bad place: we can't revert the migration because the Cirrus macOS Intel support that we migrated these tasks away from will be eliminated on January 1st, and we can't migrate the tasks to Cirrus ARM because of bugs in the Cirrus ARM hosts that haven't been resolved, but the current situation will cause us a lot of manual pain for the team due to the disruption to autoreleases described above. Is there anyone else with Cocoon expertise who could look into this bug in the meantime?

  10. CaseyHillers commented on Dec 5, 2022

    @CaseyHillers
    Contributor

    @ricardoamador or @nehalvpatel can you take a look?

  11. nehalvpatel commented on Dec 5, 2022

    @nehalvpatel
    Contributor

    Taking a look.

  12. nehalvpatel commented on Dec 7, 2022

    @nehalvpatel
    Contributor

    I think this will require the datastore Task entity to be updated with a new column for check_run_id. It looks like the LUCI build doesn't store it (maybe we can put it in there?). https://cr-buildbucket.appspot.com/rpcexplorer/services/buildbucket.v2.Builds/GetBuild?request={%20%20%20%20%22id%22:%20%228796130005986382449%22}

    Does that sound right, @CaseyHillers?

  13. CaseyHillers commented on Dec 7, 2022

    @CaseyHillers
    Contributor

    SGTM!

  14. godofredoc commented on Dec 7, 2022

    @godofredoc
    Contributor

    We may need to refactor the entire data models soon, adding new columns will complicate that migration. Do we know what the issue is? is there a way to fix it without adding new columns to data models?

  15. nehalvpatel commented on Dec 8, 2022

    @nehalvpatel
    Contributor

    I discussed with Godofredo on chat and he suggested a different approach. I'll be working on that. Anywho, I don't want to turn this thread into an implementation discussion so I'll make a new issue. 😄

  16. self-assigned this
    on Dec 12, 2022
  17. nehalvpatel commented on Jan 23, 2023

    @nehalvpatel
    Contributor

    Awaiting final review on this PR which should fix this: flutter/cocoon#2387

  18. nehalvpatel commented on Jan 26, 2023

    @nehalvpatel
    Contributor

    This should be fixed now. flutter/cocoon#2387 (comment)

    Please reopen if the issue continues!

  19. ricardoamador commented on Jan 26, 2023

    @ricardoamador
    Contributor

    Nice work @nehalvpatel !

  20. ditman commented on Feb 1, 2023

    @ditman
    MemberAuthor

    Have you ever been excited about a new CI feature? Because I'm excited about a new CI feature! Thanks people!! 🤟

  21. nehalvpatel commented on Feb 1, 2023

    @nehalvpatel
    Contributor

    Thank you for the kind words :)

  22. github-actions commented on Mar 3, 2023

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  23. locked as resolved and limited conversation to collaborators on Mar 3, 2023
  24. moved this from Verified to To be archived in [DEPRECATED] Infra Ticket Queueon Dec 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

team-infraOwned by Infrastructure team

Type

No type

Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions