I decided to just run our ansible worker create script across our cluster today in light of the sudo security flaw that was announced. In the process I've encountered a bunch of errors that we should probably look in to. Recording here as a list so we can tick them off. I'd appreciate help in dealing with these.

I didn't run any updates on:

• Windows - shouldn't need it, I think they all do auto updates and regular restarts anyway
• IBM platforms (including ibmi, aix, rhel-s390x, zos) I'll let @nodejs/build IBM folks deal with that
• macOS - @AshCripps would you mind doing these? I'm a little afraid to just run the scripts against the existing infra.

Errors:

Failed 'not secret' (Jenkins secret not in secrets/inventory.yml)

• test-digitalocean-debian9-x64-1
• test-digitalocean-centos5-x86-1 Removed
• test-softlayer-centos5-x64-2 Removed
• test-softlayer-centos5-x64-1 Removed
• release-digitalocean-centos5-x64-1 Removed
• release-digitalocean-centos6-x86-1 Removed ansible: add RHEL 8 x64 instances #2886
• release-softlayer-centos5-x86-1 Removed
• test-digitalocean-ubuntu1404-x64-1
• test-digitalocean-ubuntu1404-x86-1 Removed ansible: add RHEL 8 x64 instances #2886
• test-digitalocean-ubuntu1604-x86-2
• test-digitalocean-ubuntu1804-x64-1
• test-rackspace-ubuntu1204-x64-1 (should remove)
• test-requireio_rvagg-ubuntu1404-arm64_odroidxu-1 (should remove)
• test-requireio_rvagg-ubuntu1404-arm64_odroidxu-2 (should remove)
• test-requireio_rvagg-ubuntu1404-arm64_odroidxu3-1 (should remove)
• release-scaleway-ubuntu1604-armv7l-1 (should remove)
• release-scaleway-ubuntu1604-armv7l-2 (should remove)

Unreachable

• test-joyent-smartos18-x64-1 Removed
• test-joyent-smartos18-x64-2 Removed
• test-digitalocean-freebsd10-x64-1 (should remove?)
• test-joyent-freebsd10-x64-1 Removed
• test-joyent-freebsd10-x64-2 Removed
• test-rackspace-freebsd10-x64-1 (should remove?)
• test-digitalocean-ubuntu1204-x64-1 (should remove)
• test-digitalocean-ubuntu1204-x64-2 (should remove)
• test-scaleway-ubuntu1804-armv7l-2 (should remove)
• test-scaleway-ubuntu1804-armv7l-3 (should remove)
• test-mininodes-ubuntu1604-arm64_odroid_c2-1 (should remove)
• test-mininodes-ubuntu1604-arm64_odroid_c2-2 (should remove)
• test-mininodes-ubuntu1604-arm64_odroid_c2-3 (should remove)

Failed on 'baselayout : run ccache installer'

• test-joyent-smartos15-x64-3 (gone: Replacing machines from Joyent data center #2552)
• test-joyent-smartos15-x64-4 (gone: Replacing machines from Joyent data center #2552)
• test-joyent-smartos16-x64-3 (gone: Replacing machines from Joyent data center #2552)
• test-joyent-smartos16-x64-4 (gone: Replacing machines from Joyent data center #2552)
• test-joyent-smartos17-x64-3 "baselayout : run ccache installer" fails running ansible against LinuxONE/smartos #2533
• test-joyent-smartos17-x64-4 "baselayout : run ccache installer" fails running ansible against LinuxONE/smartos #2533
• release-joyent-smartos15-x64-2 (gone: Replacing machines from Joyent data center #2552)
• release-joyent-smartos17-x64-2 "baselayout : run ccache installer" fails running ansible against LinuxONE/smartos #2533

Failed update

• test-digitalocean-debian8-x64-1 (jessie-backports 404)
• test-softlayer-centos7-x64-1 (?) ansible: fix centos7 x64 ansible errors #2651
• test-digitalocean-freebsd11-x64-2 ("pkg: repository meta /var/db/pkg/FreeBSD.meta has wrong version 2") Errors running Ansible across current cluster #2531 (comment)
• test-nearform_intel-ubuntu1604-x64-1 Errors running Ansible across current cluster #2531 (comment)

Failed 'bootstrap : install libselinux-python bindings'

• test-digitalocean-fedora30-x64-1 ansible: fix multiple register vars on Fedora #2654
• test-digitalocean-fedora30-x64-2 ansible: fix multiple register vars on Fedora #2654
• release-softlayer-centos6-x64-1 removed ansible: add RHEL 8 x64 instances #2886
• test-softlayer-centos6-x64-1 removed ansible: add RHEL 8 x64 instances #2886
• test-softlayer-centos6-x64-2 removed ansible: add RHEL 8 x64 instances #2886

Failed 'set hostname'

(We should remove these entirely)

• test-softlayer-ubuntu1404-x64-1 (remove?)
• test-softlayer-ubuntu1404-x86-1 (remove?)

Failed 'baselayout : centos7 | install ius'

"Name or service not known", is IUS still a thing?

• test-rackspace-centos7-x64-1 ansible: fix centos7 x64 ansible errors #2651
• release-digitalocean-centos7-x64-1 ansible: fix centos7 x64 ansible errors #2651

Failed 'jenkins-worker : install tap2junit'

• test-digitalocean-freebsd11-x64-1 ansible: fixes for freebsd #2652

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'pkg_resources'
{"changed": false, "msg": "Failed to import the required Python library (setuptools) on test-digitalocean-freebsd11-x64-1's Python /usr/local/bin/python. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}

Failed uninstall node

• test-digitalocean-ubuntu1604-x86-1 Errors running Ansible across current cluster #2531 (comment)

FAILED! => {"changed": false, "cmd": "node -v", "delta": "0:00:01.462308", "end": "2021-01-27 20:56:35.373427", "failed_when_result": true, "rc": 0, "start": "2021-01-27 20:56:33.911119", "stderr": "", "stderr_lines": [], "stdout": "v9.11.2", "stdout_lines": ["v9.11.2"]}

No "containers" config

(these probably need to be removed, they were part of my next-gen containerisation experiment iirc)

• test-digitalocean-ubuntu2004_docker-x64-1
• test-digitalocean-ubuntu2004_docker-x64-2