Node.js Testing WG Meeting 2016-02-05

Links

• Recording: https://www.youtube.com/watch?v=6wnWsUN8Usw

Present

• Rich Trott
• Alexis Campailla
• João Reis
• Myles Borins

Standup

• João: Visual C++ tools in CI just completed. One more compiler in CI!
• Alexis: Nothing related to testing.
• Myles: Continuing CITGM, adding more bits into it. First instance of CITGM finding a regression landing on master this week.
• Rich: test-dgram-pingpong flakiness on Raspberry Pi

Minutes

Initial Draft of Docs

• Alexis: Propose landing at least the README and perhaps the whole PR just to have some material in the repo

Single stress testing of parallel tests in parallel

• João: Perhaps we can copy the stress test job in Jenkins and Santiago or someone else can experiment with it to see about getting it working in parallel
• Alexis: We can open an issue on the build repo to ask for access for Santiago
• Rich: Will check that this sounds good to Santi and then move forward

CI results searching via automation or…

• Rich: Santi’s POC: https://github.com/santigimeno/ci-test-results
• Alexis: Do we think that the REST API is the way to go? Ideally there would be a place that Jenkins put TAP results after a test run (a database, a file, something) that this script could access.

Flaky Tests

test-tick-processor:

• Rich: Might be a problem for a long time.
• Alexis: This is basically the issue of how to motivate people to fix these tests. This might be something this WG can make a proposal.
• Ref: nodejs/build#248 (comment)

test-dgram-pingpong:

• Rich talked about what he’s doing. Nothing to see here. Move along.

Revise Flaky Tests Wiki Page

• Alexis: Feel free to update it. It should be part of the collaborating onboard guide too.
• Rich: I’ll try.

Skipped tests

• Alexis: We don’t track them and perhaps we ought to. Sometimes tests are just skipped because they are unreliable, sometimes they are skipped for good reasons. We would need to go through the list of skipped tests and figure out which ones are being skipped for not-so-great reasons. Maybe someone is motivated to make progress on this right now? Anyone? Maybe move the skips that can be moved to the status file (that is, skips based on platform and not on runtime checks) to the status file and include comments explaining why the test is skipped.

Meeting cadence

• Rich: Once every three weeks?
• Myles: Sounds good.
• Alexis: Build group does this. Should be good here.
• João: Okay with it too.