We noticed that UI & api are very unresponsive.

Log showed increase rate of auth and queue service calls, most of them started to 401 with various hawk errors.

Further investigation showed that few tasks were trying to upload ~100k artifacts at once creating high load on services side.

Related links:

https://firefox-ci-tc.services.mozilla.com/tasks/OxOwThZjQ5Cz3XfOoADCiA

https://hg-edge.mozilla.org/try/rev/9213806cbc7dc367020d0dc8adb3e91058f0773e#l3.17