-
Notifications
You must be signed in to change notification settings - Fork 9.6k
Closed
Labels
Description
We have ~5 current flakes in the smoke tests: => #11341 (comment) <=
First, yarn smoke perf reliable fails for me locally.
$ node lighthouse-cli/test/smokehouse/frontends/smokehouse-bin.js perf
Running ONLY smoketests for: perf
perf smoketest starting…
perf smoketest results:
Doing a run of 'http://localhost:10200/preload.html'...
1 assertion(s) failed.
$ node lighthouse-cli/index.js http://localhost:10200/preload.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-xy5Duc/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-xy5Duc/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-xy5Duc/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-xy5Duc/config.json
✘ difference at uses-rel-preconnect audit.warnings.length
expected: 1
found: 2
found result:
{
"id": "uses-rel-preconnect",
"title": "Preconnect to required origins",
"description": "Consider adding `preconnect` or `dns-prefetch` resource hints to establish early connections to important third-party origins. [Learn more](https://web.dev/uses-rel-preconnect/).",
"score": 1,
"scoreDisplayMode": "numeric",
"warnings": [
"A preconnect <link> was found for \"https://fonts.googleapis.com\" but was not used by the browser. Check that you are using the `crossorigin` attribute properly.",
"A preconnect <link> was found for \"https://fonts.gstatic.com\" but was not used by the browser. Only preconnect to important origins that the page will certainly request."
]
}
Correctly passed 10 assertions
Failed 1 assertion
Doing a run of 'http://localhost:10200/perf/perf-budgets/load-things.html'...
$ node lighthouse-cli/index.js http://localhost:10200/perf/perf-budgets/load-things.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-IsBVxu/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-IsBVxu/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-IsBVxu/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-IsBVxu/config.json
Correctly passed 5 assertions
Doing a run of 'http://localhost:10200/perf/fonts.html'...
$ node lighthouse-cli/index.js http://localhost:10200/perf/fonts.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-DxhyAA/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-DxhyAA/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-DxhyAA/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-DxhyAA/config.json
Correctly passed 5 assertions
Doing a run of 'http://localhost:10200/perf/trace-elements.html'...
$ node lighthouse-cli/index.js http://localhost:10200/perf/trace-elements.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bKgbXx/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bKgbXx/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bKgbXx/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bKgbXx/config.json
[0] failed chrome version check, pruning expectation: {
"_minChromiumMilestone": 86,
"name": "anim",
"failureReasonsMask": 8224,
"unsupportedProperties": [
"background-color"
]
} Actual Chromium version: 84
Correctly passed 7 assertions
Doing a run of 'http://localhost:10200/perf/trace-elements.html?missing'...
1 assertion(s) failed.
$ node lighthouse-cli/index.js http://localhost:10200/perf/trace-elements.html?missing --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bgnFcz/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bgnFcz/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bgnFcz/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bgnFcz/config.json
✘ difference at layout-shift-elements audit.scoreDisplayMode
expected: "notApplicable"
found: "informative"
found result:
{
"id": "layout-shift-elements",
"title": "Avoid large layout shifts",
"description": "These DOM elements contribute most to the CLS of the page.",
"score": null,
"scoreDisplayMode": "informative",
"displayValue": "1 element found",
"details": {
"type": "table",
"headings": [
{
"key": "node",
"itemType": "node",
"text": "Element"
},
{
"key": "score",
"itemType": "numeric",
"granularity": 0.001,
"text": "CLS Contribution"
}
],
"items": [
{
"node": {
"type": "node",
"path": "",
"selector": "h1",
"nodeLabel": "Please don't move me",
"snippet": "<h1>",
"boundingRect": {
"top": 0,
"bottom": 0,
"left": 0,
"right": 0,
"width": 0,
"height": 0
}
},
"score": 0.0023515625
}
]
}
}
Correctly passed 4 assertions
Failed 1 assertion
Doing a run of 'http://localhost:10200/perf/animations.html'...
$ node lighthouse-cli/index.js http://localhost:10200/perf/animations.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-CXKQVu/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-CXKQVu/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-CXKQVu/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-CXKQVu/config.json
[non-composited-animations] failed chrome version check, pruning expectation: {
"_minChromiumMilestone": 86,
"score": null,
"displayValue": "1 animated element found",
"details": {
"items": [
{
"node": {
"type": "node",
"path": "2,HTML,1,BODY,1,DIV",
"selector": "body > div#animated-boi",
"nodeLabel": "div",
"snippet": "<div id=\"animated-boi\">"
},
"subItems": {
"items": [
{
"failureReason": "Unsupported CSS Property: width"
},
{
"failureReason": "Unsupported CSS Property: height",
"animation": "alpha"
},
{
"failureReason": "Unsupported CSS Property: background-color",
"animation": "beta"
}
]
}
}
]
}
} Actual Chromium version: 84
Correctly passed 3 assertions
perf smoketest complete.
4 test(s) passing
2 test(s) failing
34 expectations passing
2 expectations failing
We have 1 failing smoketests: perf
error Command failed with exit code 1.
Only tried with stable Chrome.
example for TraceElements: https://github.com/GoogleChrome/lighthouse/runs/1039274323?check_suite_focus=true#step:8:276 https://github.com/GoogleChrome/lighthouse/pull/11302/checks?check_run_id=1058028788#step:8:277