TLDR: The emulation flags are awkward and there's implicit assumptions made. We should fix that and support our varied usecases with first-class settings. Key idea: get rid of emulatedFormFactor=none.

DevTools, Calibre, and WPT are three lighthouse clients that handle emulation previous to lighthouse running. In DevTools' case, only screen emulation is applied, and lighthouse is expected to apply its own network emulation. And calibre's case, both emulations are applied and the intent is for a lighthouse to not apply double emulation anywhere. LH on WPT supports a few approaches and everyone's been confused at some point.

We introduced the illustrious internalDisableDeviceScreenEmulation flag (#9377) to solve this. (However even with this, LH will still overwrite any previously set userAgentOverride). But if emulatedFormFactor (#6098) is none and external UA emulation is applied, you don't get correct scoring.

This is messy and I think that there's an opportunity to completely redefine this configuration to be more clear.

Goals

1. no surprises wrt mobile or desktop scoring being used
2. a more first-class config for this external-emulation setup
3. continue to support the no-emulation-lh-is-running-against-a-real-phone scenario

Starting Proposal

• Rename emulatedFormFactor to formFactor. This is the key flag and the value must be either mobile or desktop, none (née provided) is no longer a valid option. Our perf metrics and 3 SEO metrics change their behavior based on mobile v desktop, so it seems important to make this an explicit user contract.
• Add two more flags: disableScreenEmulation and disableNetworkEmulation.
• Remove internalDisableDeviceScreenEmulation. Throw if this or emulatedFormFactor are used.
• Internally, we drop the guessed TestedAsMobileDevice bool and just use the formFactor enum of 'desktop'|'mobile'.

The combination of these allows us to support the below user stories with clear expectations. It also prevents the case of a using incorrect scoring accidentally.

User stories

I believe we have 4 user stories to support (right?)

1. Typical LH (we apply emulation): formFactor is set.
2. DevTools: formFactor is set via the mobile/desktop radio, and screenEmulation is false.
3. Calibre and WPT: formFactor is set, and both screenEmulation: false, throttlingMethod: 'provided'. emulatedUserAgent is set as desired.
4. LH on mobile device: --no-screenEmulation and --throttling.cpuSlowdownMultiplier=1. (--formFactor=mobile is the default already)

Extra thoughts and questions

• Given this situation and our different desktop scoring, it definitely raises the priority of communicating mobile/desktop run near the score. Add visual indicator to score to warn when it was run with non-default settings #8178 Report needs visible indicator of mobile or desktop run  #9379 ☂️Report changes and goals for v6 #9438
• Does the CLI default to --formFactor=mobile or do we make it (annoyingly) required?
  • the TestedAsMobileDevice logic correctly handles real-mobile-device, but doesn't handle external UA emulation. We could detect there's external UA emulation being applied and throw if we don't see configuration that matches. Or with the proposal implemented, we may just log a warning to let them know we see it.
  • docs: Testing on a mobile device still works correctly in v6, works in v6 even with emulated-form-factor=none. We could also detect a potential double-emulation scenario here and warn to make sure flags are correctly setup.
• core: add settings.internalDisableDeviceScreenEmulation #9377 (review) has some discussion around custom UA/metrics/touch/etc. I think the above proposal lays good groundwork for allowing that configurability.
• Throttling is related, but I haven't yet considered if we should also reconfigure those settings to better align with these. Probably not, but we could optionally detect and warn on unexpected emulation/throttling pairings.
  • Desktop throttling preset.. Going through all this I see that clients(devtools): use the same desktop throttling as lightrider #10322 made it so devtools/desktop use the same throttling constants as lr-desktop, but CLI/desktop doesn't. It probly should? Will file as a separate issue.
• I was a lil concerned the LH v6 on WPT might have broke something. But looks like Set --emulated-form-factor=desktop for LH tests without device emulation catchpoint/WebPageTest.agent#297 which was merged last week fixes things (judging by Joseph's last comment) - Update: there's a bug but only for the scorecalc handoff, actual scoring is all good.

References

• Probably related emulation-y issues: Report Score not matching the Calculator Score (webpagetest.org) - desktop #10836 LHCI fails to run puppeteer script lighthouse-ci#321 Set --emulated-form-factor=desktop for LH tests without device emulation catchpoint/WebPageTest.agent#297
• lol: Remove disableDeviceEmulation flag before v5 #7044
• Mobile run (with external screen & UA emulation), scored as desktop: https://googlechrome.github.io/lighthouse/viewer/?gist=701d46b57b8252011a25025b2442ca81