[Testing:Developer] Code coverage for Component tests by jndlansh · Pull Request #12959 · Submitty/Submitty

jndlansh · 2026-06-25T19:54:29Z

Why is this Change Important & Necessary?

Closes #12915
This PR adds code coverage for Cypress Component Tests, allowing developers to measure how much of the Vue component source code is exercised during testing. This helps identify untested code paths, improves confidence in new component tests, and provides visibility into frontend test coverage.

What is the New Behavior?

Enables coverage in CI
Uploads the reports to Codecov
Adds a dedicated cypress-component Codecov flag
Surfaces the coverage directly in the PR comment

New dependencies introduced with this PR :

vite-plugin-istanbul prepares the code for coverage.
@cypress/code-coverage captures the coverage while Cypress runs.
nyc turns that captured data into reports you can read.

Running Component Tests with:

VITE_COVERAGE=true npx cypress run --component

now generates coverage information, including HTML and lcov reports.

Normal development (vite) and production builds (npm run build) continue to behave exactly as before and are not instrumented.

What steps should a reviewer take to reproduce or test the bug or new feature?

Sync changes and install dependencies if required
Run Cypress Component Tests with coverage enabled:

In bash: 
VITE_COVERAGE=true npx cypress run --component
or 
VITE_COVERAGE=true npx cypress open --component

Verify that:

Component tests pass successfully.
Coverage is collected and summarized in the terminal.
Coverage reports are generated in the coverage/ directory.

Automated Testing & Documentation

Verified successful production build.
Verified Cypress Component Tests pass with coverage enabled.
Verified coverage reports are generated successfully.
Confirmed that instrumentation is only enabled when VITE_COVERAGE=true is set.

Other information

Not a Breaking change
No Migrations needed.

codecov · 2026-06-25T19:56:04Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 21.44%. Comparing base (01f8919) to head (8875d3d).

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12959      +/-   ##
============================================
+ Coverage     21.35%   21.44%   +0.09%     
  Complexity     9990     9990              
============================================
  Files           275      277       +2     
  Lines         37317    37361      +44     
  Branches        505      512       +7     
============================================
+ Hits           7969     8013      +44     
  Misses        28847    28847              
  Partials        501      501

Flag	Coverage Δ
autograder	`21.16% <ø> (ø)`
js	`1.98% <ø> (ø)`
migrator	`100.00% <ø> (ø)`
php	`20.41% <ø> (ø)`
python_submitty_utils	`80.08% <ø> (ø)`
submitty_daemon_jobs	`91.13% <ø> (ø)`
vue	`100.00% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

jndlansh · 2026-06-26T09:19:13Z

+
+    return {
+        plugins,
    define: {


Fix indentation

williamjallen

@jndlansh Please make sure coverage is turned on in CI:

jndlansh · 2026-06-27T04:31:45Z

Thanks @williamjallen, I have added code-coverage in CI, missed out initially.

jndlansh · 2026-06-27T04:35:42Z

Now, shall I add Cypress code coverage to codecov?

JManion32

As we discussed in our meeting, please add this to the codecov.yml file, and make sure to update the PR description accordingly.

I see you added several new dependencies to create this. I think it is good practice, especially if there are multiple, to explain the need for adding each of them. nyc and vite-plugin-istanbul especially caught my eye when I first looked over this PR. I looked into this a little bit, and they actually have an interesting history:
The original JavaScript coverage tool was called Istanbul (named after the city). When the maintainers rewrote and modernized the command-line interface, they named it nyc—another famous city, and the common abbreviation for New York City. - ChatGPT
And this is what each new dependency does:

vite-plugin-istanbul prepares the code for coverage.
@cypress/code-coverage captures the coverage while Cypress runs.
nyc turns that captured data into reports you can read.
Just add something to the description so it is clear to other reviewers.

I noticed this in the CI output before the coverage report:

Opening Cypress...
Warning: The allowCypressEnv configuration option is enabled. This allows any browser code to read values from Cypress.env(). This is insecure and will be removed in a future major version.

1. Replace Cypress.env() calls with cy.env() (for sensitive values) or Cypress.expose() (for public configuration)
2. Set allowCypressEnv: false in your Cypress configuration to disable Cypress.env()

Learn more: https://on.cypress.io/cypress-env-migration

I verified this is new to this PR. Can we address it here?

jndlansh · 2026-06-29T06:49:50Z

Now for this warning that is displayed on CI/cypress component :
I don't think it is not caused by the code coverage changes. I removed the @cypress/code-coverage/support import and the warning still appears. Would like to know your opinion.

I checked it on main branch and got this same warning in the terminal atleast

JManion32 · 2026-06-29T20:04:45Z

Now for this warning that is displayed on CI/cypress component : I don't think it is not caused by the code coverage changes. I removed the @cypress/code-coverage/support import and the warning still appears. Would like to know your opinion.

I checked it on main branch and got this same warning in the terminal atleast

Yes, you are correct, I just verified the same.

JManion32

CodeCov is currently saying that the coverage is 100% for cypress-component. This is not possible, since the popup test is 28/29. I dug into this a bit, and it looks like this may be caused by the path you are using in codecov.yml. There are no Vue files at site/vue, so maybe you want to adjust that to site/vue/src or even site/vue/src/components?

I am not sure if this is the definite solution, but it is a good starting point. I just know that CodeCov shouldn't be reporting 100% since our only merged test is 28/29.

…into code-coverage

williamjallen

Seems reasonable to me. Good to merge, once @JManion32 is happy with the changes.

JManion32

Code coverage now displays for each test and is a part of CodeCov.

We currently have 2:

Popup

StatusBanner

This will be helpful for ensuring our component tests are thorough. Nice work!

Add code coverage support for Cypress component tests

83c5662

github-project-automation Bot moved this to Seeking Reviewer in Submitty Development Jun 25, 2026

github-project-automation Bot added this to Submitty Development Jun 25, 2026

jndlansh requested a review from JManion32 June 26, 2026 03:53

automateprojectmangement Bot moved this from Seeking Reviewer to In Review in Submitty Development Jun 26, 2026

jndlansh commented Jun 26, 2026

View reviewed changes

Comment thread site/vue/vite.config.mts Outdated

return {

plugins,

define: {

jndlansh Jun 26, 2026

Copy link
Copy Markdown

Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix indentation

williamjallen requested changes Jun 27, 2026

View reviewed changes

github-project-automation Bot moved this from In Review to Work in Progress in Submitty Development Jun 27, 2026

jndlansh added 3 commits June 27, 2026 09:40

add code-coverage to CI and debugging

de13c02

Debug Cleanup

f494456

fix JS lint

ca17eb5

jndlansh requested a review from williamjallen June 27, 2026 17:05

automateprojectmangement Bot moved this from Work in Progress to In Review in Submitty Development Jun 27, 2026

jndlansh commented Jun 29, 2026

View reviewed changes

Comment thread .gitignore Outdated

JManion32 requested changes Jun 29, 2026

View reviewed changes

Comment thread .gitignore Outdated

github-project-automation Bot moved this from In Review to Work in Progress in Submitty Development Jun 29, 2026

remove public path from .gitignore

52ab5dd

jndlansh added 2 commits June 29, 2026 23:17

add code-coverage to codecov - 1

e4832b5

refactor codecov configuration and clean up CI workflow

e648fc0

jndlansh requested a review from JManion32 June 29, 2026 19:05

automateprojectmangement Bot moved this from Work in Progress to In Review in Submitty Development Jun 29, 2026

Merge branch 'main' into code-coverage

f619120

JManion32 requested changes Jun 29, 2026

View reviewed changes

github-project-automation Bot moved this from In Review to Work in Progress in Submitty Development Jun 29, 2026

FIx component path

dec3435

jndlansh added 3 commits June 30, 2026 05:46

Merge branch 'code-coverage' of https://github.com/jndlansh/Submitty …

c16c530

…into code-coverage

Fix path for cypress code coverage in Codecov

3a58ecb

Fix path - debug

99fb0fd

williamjallen requested changes Jun 30, 2026

View reviewed changes

Comment thread .codecov.yml Outdated

Comment thread .codecov.yml

add cypress component to coverage status

0d20519

williamjallen reviewed Jun 30, 2026

View reviewed changes

Comment thread .codecov.yml Outdated

change flag name from cypress-component to vue

788b879

jndlansh requested review from JManion32 and williamjallen June 30, 2026 15:27

automateprojectmangement Bot moved this from Work in Progress to In Review in Submitty Development Jun 30, 2026

williamjallen approved these changes Jul 1, 2026

View reviewed changes

jndlansh added 3 commits July 1, 2026 23:15

Fix 100% coverage issue

d16343d

Merge remote-tracking branch 'upstream/main' into code-coverage

77c9ca4

add 'cwd' setting in .nycrc

8b10180

JManion32 approved these changes Jul 2, 2026

View reviewed changes

github-project-automation Bot moved this from In Review to Awaiting Maintainer Review in Submitty Development Jul 2, 2026

Merge branch 'main' into code-coverage

8875d3d

bmcutler merged commit a67bbb7 into Submitty:main Jul 3, 2026
25 of 28 checks passed

Uh oh!

Conversation

jndlansh commented Jun 25, 2026 • edited by JManion32 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why is this Change Important & Necessary?

What is the New Behavior?

What steps should a reviewer take to reproduce or test the bug or new feature?

Automated Testing & Documentation

Other information

Uh oh!

codecov Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jndlansh Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

williamjallen left a comment

Choose a reason for hiding this comment

Uh oh!

jndlansh commented Jun 27, 2026

Uh oh!

jndlansh commented Jun 27, 2026

Uh oh!

Uh oh!

JManion32 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jndlansh commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JManion32 commented Jun 29, 2026

Uh oh!

JManion32 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

williamjallen left a comment

Choose a reason for hiding this comment

Uh oh!

JManion32 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jndlansh commented Jun 25, 2026 •

edited by JManion32

Loading

codecov Bot commented Jun 25, 2026 •

edited

Loading

jndlansh commented Jun 29, 2026 •

edited

Loading