Skip to content

smoke tests are flaky #11341

@connorjclark

Description

@connorjclark

We have ~5 current flakes in the smoke tests: => #11341 (comment) <=



First, yarn smoke perf reliable fails for me locally.

$ node lighthouse-cli/test/smokehouse/frontends/smokehouse-bin.js perf
Running ONLY smoketests for: perf

perf smoketest starting…

perf smoketest results:
Doing a run of 'http://localhost:10200/preload.html'...
1 assertion(s) failed.
$ node lighthouse-cli/index.js http://localhost:10200/preload.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-xy5Duc/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-xy5Duc/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-xy5Duc/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-xy5Duc/config.json

  ✘ difference at uses-rel-preconnect audit.warnings.length
              expected: 1
                 found: 2

          found result:
      {
        "id": "uses-rel-preconnect",
        "title": "Preconnect to required origins",
        "description": "Consider adding `preconnect` or `dns-prefetch` resource hints to establish early connections to important third-party origins. [Learn more](https://web.dev/uses-rel-preconnect/).",
        "score": 1,
        "scoreDisplayMode": "numeric",
        "warnings": [
          "A preconnect <link> was found for \"https://fonts.googleapis.com\" but was not used by the browser. Check that you are using the `crossorigin` attribute properly.",
          "A preconnect <link> was found for \"https://fonts.gstatic.com\" but was not used by the browser. Only preconnect to important origins that the page will certainly request."
        ]
      }

  Correctly passed 10 assertions
  Failed 1 assertion


Doing a run of 'http://localhost:10200/perf/perf-budgets/load-things.html'...
$ node lighthouse-cli/index.js http://localhost:10200/perf/perf-budgets/load-things.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-IsBVxu/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-IsBVxu/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-IsBVxu/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-IsBVxu/config.json
  Correctly passed 5 assertions


Doing a run of 'http://localhost:10200/perf/fonts.html'...
$ node lighthouse-cli/index.js http://localhost:10200/perf/fonts.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-DxhyAA/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-DxhyAA/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-DxhyAA/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-DxhyAA/config.json
  Correctly passed 5 assertions


Doing a run of 'http://localhost:10200/perf/trace-elements.html'...
$ node lighthouse-cli/index.js http://localhost:10200/perf/trace-elements.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bKgbXx/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bKgbXx/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bKgbXx/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bKgbXx/config.json
[0] failed chrome version check, pruning expectation: {
  "_minChromiumMilestone": 86,
  "name": "anim",
  "failureReasonsMask": 8224,
  "unsupportedProperties": [
    "background-color"
  ]
} Actual Chromium version: 84
  Correctly passed 7 assertions


Doing a run of 'http://localhost:10200/perf/trace-elements.html?missing'...
1 assertion(s) failed.
$ node lighthouse-cli/index.js http://localhost:10200/perf/trace-elements.html?missing --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bgnFcz/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bgnFcz/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bgnFcz/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-bgnFcz/config.json

  ✘ difference at layout-shift-elements audit.scoreDisplayMode
              expected: "notApplicable"
                 found: "informative"

          found result:
      {
        "id": "layout-shift-elements",
        "title": "Avoid large layout shifts",
        "description": "These DOM elements contribute most to the CLS of the page.",
        "score": null,
        "scoreDisplayMode": "informative",
        "displayValue": "1 element found",
        "details": {
          "type": "table",
          "headings": [
            {
              "key": "node",
              "itemType": "node",
              "text": "Element"
            },
            {
              "key": "score",
              "itemType": "numeric",
              "granularity": 0.001,
              "text": "CLS Contribution"
            }
          ],
          "items": [
            {
              "node": {
                "type": "node",
                "path": "",
                "selector": "h1",
                "nodeLabel": "Please don't move me",
                "snippet": "<h1>",
                "boundingRect": {
                  "top": 0,
                  "bottom": 0,
                  "left": 0,
                  "right": 0,
                  "width": 0,
                  "height": 0
                }
              },
              "score": 0.0023515625
            }
          ]
        }
      }

  Correctly passed 4 assertions
  Failed 1 assertion


Doing a run of 'http://localhost:10200/perf/animations.html'...
$ node lighthouse-cli/index.js http://localhost:10200/perf/animations.html --output-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-CXKQVu/smokehouse.report.json --output=json -G=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-CXKQVu/artifacts/ -A=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-CXKQVu/artifacts/ --quiet --port=0 --config-path=/var/folders/9z/q2z3wm4d629705qmsn35mr9m00kp_v/T/smokehouse-CXKQVu/config.json
[non-composited-animations] failed chrome version check, pruning expectation: {
  "_minChromiumMilestone": 86,
  "score": null,
  "displayValue": "1 animated element found",
  "details": {
    "items": [
      {
        "node": {
          "type": "node",
          "path": "2,HTML,1,BODY,1,DIV",
          "selector": "body > div#animated-boi",
          "nodeLabel": "div",
          "snippet": "<div id=\"animated-boi\">"
        },
        "subItems": {
          "items": [
            {
              "failureReason": "Unsupported CSS Property: width"
            },
            {
              "failureReason": "Unsupported CSS Property: height",
              "animation": "alpha"
            },
            {
              "failureReason": "Unsupported CSS Property: background-color",
              "animation": "beta"
            }
          ]
        }
      }
    ]
  }
} Actual Chromium version: 84
  Correctly passed 3 assertions


perf smoketest complete.
  4 test(s) passing
  2 test(s) failing

34 expectations passing
2 expectations failing
We have 1 failing smoketests: perf
error Command failed with exit code 1.

Only tried with stable Chrome.

example for TraceElements: https://github.com/GoogleChrome/lighthouse/runs/1039274323?check_suite_focus=true#step:8:276 https://github.com/GoogleChrome/lighthouse/pull/11302/checks?check_run_id=1058028788#step:8:277

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions