13:52 <•SimonSapin> #18338 is being a pain
13:52 <crowbot> Issue #18338: Buildbot jobs fail intermittently with "exceptions.Exception: Actual commit (…) differs from requested commit (…)" - #18338
13:52 <nox> SimonSapin: AFAIK it's a garbage lockfile somewhere.
13:53 <•SimonSapin> git lock?
13:53 <nox> SimonSapin: http://build.servo.org/builders/mac-rel-wpt1/builds/5713/steps/git/logs/stdio
13:53 <nox> SimonSapin: Yes.
New messages
13:53 <nox> SimonSapin: I suspect this happened because we forced some PRs,
13:53 <nox> and forcing stuff just stops the builds without cleaning up.
13:53 <•SimonSapin> comment in the issue?
13:53 <nox> So if you force during a Git checkout, nothing works anymore afterwards.
13:53 <nox> Imma copypasta

There are two issues here:

1. the first step git update should not be green - there is this line in there:

fatal: Unable to create '/Users/servo/buildbot/slave/mac-rel-wpt1/build/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.

I'm assuming this is the result of a force or otherwise cancelled build that needs to be cleaned up better.

2. It can be fixed manually with something like I did in:
   servo-mac7 can't complete git checkout saltfs#680 (comment)

at the risk of "wow, that seems dangerous" I believe that admins can do something like:

salt servo-mac* cmd.run 'find /Users/servo/buildbot/slave -name index.lock | xargs rm'

And fix it up for the whole cluster. Or burn it all to the ground :-)

cc @edunham

@aneeshusa @edunham Any reason you can think of for this behavior? Can't cmd.run right now:

root@servo-master1:~# salt 'servo-mac1' test.ping
servo-mac1:
    True
root@servo-master1:~# salt 'servo-mac1' cmd.run 'find /Users/servo/buildbot/slave index.lock | xargs echo'
servo-mac1:
    Minion did not return. [No response]
root@servo-master1:~# 

Works for me right now:

root@servo-master1:~# salt 'servo-mac*' cmd.run 'echo hi'
servo-mac2:
    hi
servo-mac4:
    hi
servo-mac3:
    hi
servo-mac8:
    hi
servo-mac9:
    hi
servo-mac7:
    hi
servo-mac5:
    hi
servo-mac6:
    hi
servo-mac1:
    hi
root@servo-master1:~#

Depending on how long moving to Taskcluster takes (e.g. #17580, servo/saltfs#559 and co), we may want to fork Buildbot and write/backport a better git plugin that's less prone to these issues or teach Buildbot to clean the git lock after a force.

Also, I think a slightly safter version of your command would be

root@servo-master1:~# salt 'servo-mac*' cmd.run 'find /Users/servo/buildbot/slave -path "*/.git/index.lock"'

followed by

root@servo-master1:~# salt 'servo-mac*' cmd.run 'find /Users/servo/buildbot/slave -path "*/.git/index.lock" -delete'

You didn't pass a -name or -path before, so that might just delete everything...

Aha, nice tweaks - I had noticed the -name thing before running it, luckily :-)

I just did:

ssh [email protected] 'find /Users/servo/buildbot/slave -name index.lock | xargs rm'

For now, which is less than awesome. I didn't know about the -delete arg - that's fantastic!

cc @aneeshusa

The solution is documented in https://github.com/servo/servo/wiki/Buildbot-administration, so no need to keep this open.