Improve test reliability


tests are still failing too often. Since we don't have CI stats for github, I tried to crawl workflow results using a rudimentary [bash script](https://gist.github.com/mbien/6d5f82f6fbb6c67bfbb702bf91fe3c0c) calling into the github API.

The script scans workflow runs and filters merges to master which failed, then scans re-runs too and prints the failed steps. Rest is just `sort` and `uniq`.


```
     14 "ide/project.ant" - done
     13 "java.hints batch1"
     11 "java.file.launcher" - done
     10 "ide/css.editor"
      7 "groovy/groovy.editor" - done
      6 "webcommon/html.knockout"
      3 "platform/o.n.bootstrap"
      2 "java/maven"
      2 "java.hints batch2"
      1 "web.beans"
      1 "platform/openide.nodes"
      1 "java.editor"
```
note: this does not take the retry script into account which might be active on [some steps](https://github.com/apache/netbeans/blob/cbfd04788a960b293b2b16521629aabdafee1e84/.github/workflows/main.yml#L701)


here the results with the links pointing directly to the workflow run:

<details>

run: https://github.com/apache/netbeans/actions/runs/12936028241 (4 attempts)

 - `groovy/groovy.editor`
 - `groovy/groovy.editor`
 - `groovy/groovy.editor`
 
run: https://github.com/apache/netbeans/actions/runs/12934523662 (2 attempts)

 - `java.file.launcher`
 
run: https://github.com/apache/netbeans/actions/runs/12933263724 (3 attempts)

 - `ide/project.ant`
 - `ide/project.ant`
 - `java.editor`
 - `java/maven`
 
run: https://github.com/apache/netbeans/actions/runs/12933078681 (4 attempts)

 - `java.hints batch1`
 - `java.hints batch2`
 - `platform/o.n.bootstrap`
 - `java.hints batch1`
 - `java.hints batch1`

run: https://github.com/apache/netbeans/actions/runs/12912321783 (3 attempts)

 - `java.file.launcher`
 - `java.hints batch1`
 - `platform/o.n.bootstrap`
 - `ide/css.editor`
 - `ide/css.editor`

run: https://github.com/apache/netbeans/actions/runs/12872488235 (3 attempts)

 - `webcommon/html.knockout`
 - `webcommon/html.knockout`

run: https://github.com/apache/netbeans/actions/runs/12869051716 (3 attempts)

 - `java.hints batch1`
 - `ide/css.editor`
 - `webcommon/html.knockout`
 
run: https://github.com/apache/netbeans/actions/runs/12857668868 (2 attempts)

 - `ide/css.editor`
 
run: https://github.com/apache/netbeans/actions/runs/12851818621 (3 attempts)

 - `ide/css.editor`
 - `webcommon/html.knockout`
 
run: https://github.com/apache/netbeans/actions/runs/12846158264 (2 attempts)

 - `ide/project.ant`
 - `ide/css.editor`
 
run: https://github.com/apache/netbeans/actions/runs/12836525165 (2 attempts)

 - `java.file.launcher`
 
run: https://github.com/apache/netbeans/actions/runs/12836146985 (2 attempts)

 - `java/maven`
 - `java.file.launcher`
 
run: https://github.com/apache/netbeans/actions/runs/12826345964 (3 attempts)

 - `ide/project.ant`
 - `platform/o.n.bootstrap`
 - `webcommon/html.knockout`
 - `ide/project.ant`
 
run: https://github.com/apache/netbeans/actions/runs/12825920011 (3 attempts)

 - `ide/project.ant`
 - `groovy/groovy.editor`
 - `ide/project.ant`
 
run: https://github.com/apache/netbeans/actions/runs/12825908774 (2 attempts)

 - `ide/project.ant`
 - `java.file.launcher`
 
run: https://github.com/apache/netbeans/actions/runs/12795877604 (2 attempts)

 - `ide/project.ant`
 - `java.hints batch1`
 - `java.file.launcher`
 - `ide/css.editor`
 
run: https://github.com/apache/netbeans/actions/runs/12794975127 (2 attempts)

 - `groovy/groovy.editor`
 
run: https://github.com/apache/netbeans/actions/runs/12791936642 (2 attempts)

 - `ide/project.ant`
 - `groovy/groovy.editor`
 
run: https://github.com/apache/netbeans/actions/runs/12765772367 (2 attempts)

 - `ide/project.ant`
 - `java.hints batch1`
 
run: https://github.com/apache/netbeans/actions/runs/12764904231 (2 attempts)

 - `ide/project.ant`
 - `java.file.launcher`
 - `platform/openide.nodes`
 
run: https://github.com/apache/netbeans/actions/runs/12762278637 (3 attempts)

 - `java.file.launcher`
 - `ide/css.editor`
 - `ide/css.editor`
 
run: https://github.com/apache/netbeans/actions/runs/12762263024 (3 attempts)

 - `java.hints batch1`
 - `java.file.launcher`
 - `java.file.launcher`
 - `java.hints batch1`
 
run: https://github.com/apache/netbeans/actions/runs/12749609504 (2 attempts)

 - `java.hints batch1`
 - `java.hints batch2`
 - `java.file.launcher`
 
run: https://github.com/apache/netbeans/actions/runs/12722000825 (4 attempts)

 - `ide/project.ant`
 - `web.beans`
 - `groovy/groovy.editor`
 - `java.hints batch1`
 - `ide/css.editor`
 - `java.hints batch1`
 - `java.hints batch1`
 
run: https://github.com/apache/netbeans/actions/runs/12721993365 (2 attempts)

 - `ide/project.ant`
 - `webcommon/html.knockout`


</details>


Navigation Menu

Uh oh!

Improve test reliability #8183

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Uh oh!

Improve test reliability #8183

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions