Tell us about your PDF experience.

Microsoft Playwright Testing Preview

documentation
Learn how to run end-to-end tests for your web apps at scale with Microsoft Playwright
Testing Preview. Run Playwright tests with high parallelization across different operating
system and browser combinations simultaneously. Tutorials, code examples, GitHub
Actions, Azure Pipelines, and more.

About Microsoft Playwright Testing

ｅ OVERVIEW

What is Microsoft Playwright Testing?

Try Microsoft Playwright Testing for free

Get started

ｆ QUICKSTART

Run end-to-end web tests at scale

Set up continuous end-to-end testing

Troubleshoot using rich reports and artifacts

ｐ CONCEPT

Determine optimal test suite configuration

Manage workspaces

ｃ HOW-TO GUIDE

Optimize regional latency

Manage workspaces
Security

ｃ HOW-TO GUIDE

Manage access tokens

Manage workspace access

Manage authentication
What is Microsoft Playwright Testing
Preview?
Article • 09/16/2024

Microsoft Playwright Testing Preview is a fully managed service for end-to-end testing
built on top of Playwright. With Playwright, you can automate end-to-end tests to
ensure your web applications work the way you expect it to, across different web
browsers and operating systems. The service abstracts the complexity and infrastructure
for running Playwright tests and managing results and artifacts. The service runs tests
with high parallelization and stores test results and artifacts to help you ship features
faster and troubleshoot easily.

Run your Playwright test suite in the cloud, without changes to your test code or
modifications to your tooling setup. Use the Playwright Test Visual Studio Code
extension for a rich editor experience, or use the Playwright CLI to add automation
within your continuous integration (CI) workflow.

Get started with Quickstart: run your Playwright tests at scale with Microsoft Playwright
Testing.

To learn more about how to create end-to-end tests with the Playwright framework, visit
the Getting started documentation on the Playwright website.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Accelerate tests with parallel remote browsers

As your application becomes more complex, your test suite increases in size. The time to
complete your test suite also grows accordingly. Use parallel remote browsers to
shorten the overall test suite completion time.

Distribute your tests across many parallel browsers, hosted on cloud infrastructure.

Scale your tests beyond the processing power of your developer workstation, local
infrastructure, or CI agent machines.
 Consistent regional performance by running your tests on browsers in an Azure
region that's closest to your client machine.

Learn more about how you can configure for optimal performance.

Test consistently across multiple operating

systems and browsers
Modern web apps need to work flawlessly across numerous browsers, operating
systems, and devices.

Run tests simultaneously across all modern browsers on Windows, Linux, and
mobile emulation of Google Chrome for Android and Mobile Safari.

Using service-managed browsers ensures consistent and reliable results for both
functional and visual regression testing, whether tests are run from your team's
developer workstations or CI pipeline.

Microsoft Playwright Testing supports all browsers supported by Playwright .

Troubleshoot tests easily using reporting and

artifacts
As applications grow, maintaining quality is crucial. Use the reporting feature of the
service to troubleshoot test results with rich artifacts.

Publish test results and artifacts to the service and view them in the service portal
for faster troubleshooting.
Integrate reporting with CI pipelines to get rich, consolidated reports.

Endpoint testing
Use cloud-hosted remote browsers to test web applications regardless of where they're
hosted, without having to allow inbound connections on your firewall.

Test publicly and privately hosted applications.

During the development phase, run tests against a localhost development server.

Playwright support
Microsoft Playwright Testing is built on top of the Playwright framework.

Support for multiple versions of Playwright with each new Playwright release.

Integrate your existing Playwright test suite without changing your test code.

Use the Playwright Test Visual Studio Code extension for a rich editor experience.

Continuous end-to-end testing by using the Playwright CLI to integrate with

continuous integration (CI) tools.

How it works
Microsoft Playwright Testing instantiates cloud-hosted browsers across different
operating systems. Playwright runs on the client machine and interacts with Microsoft
Playwright Testing to run your Playwright tests on the hosted browsers. The client
machine can be your developer workstation or a CI agent machine if you run your tests
as part of your CI workflow. The Playwright test code remains on the client machine
during the test run.

After a test run completes, the test results, trace files, and other test run files are
available on the client machine. These are then published to the service from the client
machine and can be viewed in the service portal.

To run existing tests with Microsoft Playwright Testing requires no changes to your test
code, install the Microsoft Playwright Testing service package and specify the service
endpoint for your workspace.
Learn more about how to determine the optimal configuration for optimizing test suite
completion.

In-region data residency & data at rest

Microsoft Playwright Testing doesn't store or process customer data outside the region
you deploy the workspace in. When you use the regional affinity feature, the metadata is
transferred from the cloud hosted browser region to the workspace region in a secure
and compliant manner.

Microsoft Playwright Testing automatically encrypts all data stored in your workspace
with keys managed by Microsoft (service-managed keys). For example, this data includes
workspace details, Playwright test run meta data like test start and end time, test
minutes, who ran the test, and test results and artifacts generated by Playwright which
are published to the service.

Next step
Quickstart: Run Playwright tests at scale

Feedback
Was this page helpful?  Yes  No

Provide product feedback

Try Microsoft Playwright Testing
Preview for free
Article • 09/16/2024

Microsoft Playwright Testing Preview is a fully managed service for end-to-end testing
built on top of Playwright. With the free trial, you can try Microsoft Playwright Testing
for free for 30 days, 100 test minutes, and 1,000 test results. In this article, you learn
about the limits of the free trial, how to get started, and how to track your free trial
usage.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Prerequisites
An Azure account with an active subscription. If you don't have an Azure
subscription, create a free account before you begin.
Your Azure account needs the Owner, Contributor, or one of the classic
administrator roles.

Limits to free trial

The following table lists the limits for the Microsoft Playwright Testing free trial.

ﾉ Expand table

Resource Limit

Duration of trial 30 days

Total test minutes¹ 100 minutes

Total test results¹ 1,000 results

Number of workspaces²³ 1
¹ If your usage exceeds either the free test minute limit or the free test result limit, only
the overage counts toward the pay-as-you-go billing model. The two features are billed
separately. See Microsoft Playwright Testing preview pricing

² These limits only apply to the first workspace you create in your Azure subscription.
Any subsequent workspaces you create in the subscription automatically uses the pay-
as-you-go billing model.

³ If you delete the free trial workspace, you can't create a new free trial workspace
anymore.

Ｕ Caution

If you exceed any of these limits, the workspace is automatically converted to the
pay-as-you-go billing model. Learn more about the Microsoft Playwright Testing
pricing .

Create a workspace
The first time you create a workspace in your Azure subscription, the workspace is
automatically enrolled in the free trial.

To create a workspace in the Playwright portal:

1. Sign in to the Playwright portal with your Azure account.

2. Select Create workspace.

If this is the first workspace you create in the Azure subscription, you see a
message that the workspace is eligible for the free trial.
 

3. Provide the following information to create the workspace:

ﾉ Expand table

Field Description

Workspace Enter a unique name to identify your workspace.

name The name can only consist of alphanumerical characters, and have a
length between 3 and 64 characters.

Azure Select the Azure subscription that you want to use for this Microsoft
subscription Playwright Testing workspace.

Region Select a geographic location to host your workspace.

This location is where the test run data is stored for the workspace.

4. Select Create workspace.

Track your free trial usage

You can track the usage of the free trial for a workspace in either of these ways:

Select the settings icon and then select Billing.

 

Select the In free trial menu item.

In the list of all workspaces, you can view a banner message that indicates if a
workspace is in the free trial.

Upgrade your workspace

When you exceed any of the limits of the free trial, your workspace is automatically
converted to the pay-as-you-go billing model.

All test runs, test results, and other artifacts linked to your workspace remain available.

Next step
Quickstart: Run Playwright tests at scale

Feedback
Was this page helpful?  Yes  No

Provide product feedback

Quickstart: Run end-to-end tests at
scale with Microsoft Playwright Testing
Preview
Article • 11/19/2024

In this quickstart, you learn how to run your Playwright tests with highly parallel cloud
browsers and troubleshoot failed tests easily using Microsoft Playwright Testing Preview.
Use cloud infrastructure to validate your application across multiple browsers, devices,
and operating systems. Publish the results and artifacts generated by Playwright to the
service and view them in the service portal.

After you complete this quickstart, you have a Microsoft Playwright Testing workspace
to run your Playwright tests at scale and view test results and artifacts in the service
portal.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Prerequisites
An Azure account with an active subscription. If you don't have an Azure
subscription, create a free account before you begin.
Your Azure account needs the Owner, Contributor, or one of the classic
administrator roles.
A Playwright project. If you don't have project, create one by using the Playwright
getting started documentation or use our Microsoft Playwright Testing sample
project .
Azure CLI. If you don't have Azure CLI, see Install Azure CLI.

Create a workspace
To get started with running your Playwright tests at scale on cloud browsers, you first
create a Microsoft Playwright Testing workspace in the Playwright portal.
1. Sign in to the Playwright portal with your Azure account.

2. If you already have a workspace, select an existing workspace, and move to the
next step.

 Tip

If you have multiple workspaces, you can switch to another workspace by

selecting the workspace name at the top of the page, and then select Manage
all workspaces.

3. If you don't have a workspace yet, select + New workspace, and then provide the
following information:

ﾉ Expand table

Field Description

Workspace Enter a unique name to identify your workspace.

name The name can only consist of alphanumerical characters, and have a
length between 3 and 64 characters.

Azure Select the Azure subscription that you want to use for this Microsoft
subscription Playwright Testing workspace.

Region Select a geographic location to host your workspace.

This is the location where the test run data is stored for the workspace.

4. Select Create workspace to create the workspace in your subscription.

 During the workspace creation, a new resource group and a Microsoft Playwright
Testing Azure resource are created in your Azure subscription.

When the workspace creation finishes, you're redirected to the setup guide.

Install Microsoft Playwright Testing package

To use the service, install Microsoft Playwright Testing package.

npm

npm init @azure/microsoft-playwright-testing@latest

This generates playwright.service.config.ts file which serves to:

Direct and authenticate Playwright to the Microsoft Playwright Testing service.

Adds a reporter to publish test results and artifacts.

If you already have this file, the package asks you to override it.

Configure the service region endpoint

In your setup, you have to provide the region-specific service endpoint. The endpoint
depends on the Azure region you selected when creating the workspace.

To get the service endpoint URL, perform the following steps:

1. In Add region endpoint in your setup, copy the region endpoint for your
workspace.

The endpoint URL matches the Azure region that you selected when creating the
workspace. Make sure this URL is available in PLAYWRIGHT_SERVICE_URL environment
variable.

Set up your environment

To set up your environment, you have to configure the PLAYWRIGHT_SERVICE_URL
environment variable with the value you obtained in the previous steps.

We recommend that you use the dotenv module to manage your environment. With
dotenv , you define your environment variables in the .env file.

1. Add the dotenv module to your project:

shell

npm i --save-dev dotenv

2. Create a .env file alongside the playwright.config.ts file in your Playwright

project:

PLAYWRIGHT_SERVICE_URL={MY-REGION-ENDPOINT}

Make sure to replace the {MY-REGION-ENDPOINT} text placeholder with the value you
copied earlier.

Set up Authentication
To run your Playwright tests in your Microsoft Playwright Testing workspace, you need
to authenticate the Playwright client where you're running the tests with the service. This
could be your local dev machine or CI machine.

The service offers two authentication methods: Microsoft Entra ID and Access Tokens.

Microsoft Entra ID uses your Azure credentials, requiring a sign-in to your Azure account
for secure access. Alternatively, you can generate an access token from your Playwright
workspace and use it in your setup.

Set up authentication using Microsoft Entra ID

Microsoft Entra ID is the default and recommended authentication for the service. From
your local dev machine, you can use Azure CLI to sign-in

CLI

az login
 ７ Note

If you're a part of multiple Microsoft Entra tenants, make sure you sign in to the
tenant where your workspace belongs. You can get the tenant ID from Azure portal.
See Find your Microsoft Entra Tenant. Once you get the ID, sign-in using the
command az login --tenant <TenantID>

Set up authentication using access tokens

You can generate an access token from your Playwright Testing workspace and use it in
your setup. However, we strongly recommend Microsoft Entra ID for authentication due
to its enhanced security. Access tokens, while convenient, function like long-lived
passwords and are more susceptible to being compromised.

1. Authentication using access tokens is disabled by default. To use, Enable access-

token based authentication

2. Set up authentication using access tokens

Ｕ Caution

We strongly recommend using Microsoft Entra ID for authentication to the service.

If you are using access tokens, see How to Manage Access Tokens

Enable artifacts in your Playwright setup

In the playwright.config.ts file of your project, make sure you are collecting all the
required artifacts.

TypeScript

use: {
trace: 'on-first-retry',
video:'retain-on-failure',
screenshot:'on'
}

Run your tests at scale and troubleshoot easily

with Microsoft Playwright Testing
You've now prepared the configuration for running your Playwright tests in the cloud
with Microsoft Playwright Testing. You can either use the Playwright CLI to run your
tests, or use the Playwright Test Visual Studio Code extension .

Run a single test with the service

With Microsoft Playwright Testing, you get charged based on the number of total test
minutes and number of test results published. If you're a first-time user or getting
started with a free trial, you might start with running a single test at scale instead of
your full test suite to avoid exhausting your free trial limits.

７ Note

Reporting feature is enabled by default for existing workspaces. This is being rolled
out in stages and will take a few days. To avoid failures, confirm that Rich
diagnostics using reporting setting is ON for your workspace before proceeding.

See, Enable reporting for workspace.

After you validate that the test runs successfully, you can gradually increase the test load
by running more tests with the service.

Perform the following steps to run a single Playwright test with Microsoft Playwright
Testing:

Playwright CLI

To use the Playwright CLI to run your tests with Microsoft Playwright Testing, pass
the service configuration file as a command-line parameter.

1. Open a terminal window.

2. Enter the following command to run your Playwright test on remote browsers
in your workspace:

Replace the {name-of-file.spec.ts} text placeholder with the name of your

test specification file.

Bash

npx playwright test {name-of-file.spec.ts} --

config=playwright.service.config.ts
 After the test completes, you can view the test status in the terminal.

Output

Running 1 test using 1 worker

1 passed (2.2s)

To open last HTML report run:

npx playwright show-report

You can now run multiple tests with the service, or run your entire test suite on remote
browsers.

Ｕ Caution

Depending on the size of your test suite, you might incur additional charges for the
test minutes and test results beyond your allotted free test minutes and free test
results.

Run a full test suite with the service

Now that you've validated that you can run a single test with Microsoft Playwright
Testing, you can run a full Playwright test suite at scale.

Perform the following steps to run a full Playwright test suite with Microsoft Playwright
Testing:

Playwright CLI

When you run multiple Playwright tests or a full test suite with Microsoft Playwright
Testing, you can optionally specify the number of parallel workers as a command-
line parameter.

1. Open a terminal window.

2. Enter the following command to run your Playwright test suite on remote
browsers in your workspace:

Bash

npx playwright test --config=playwright.service.config.ts --

 workers=20

Depending on the size of your test suite, this command runs your tests on up
to 20 parallel workers.

After the test completes, you can view the test status in the terminal.

Output

Running 6 tests using 6 workers

6 passed (18.2s)

Test report:
https://playwright.microsoft.com/workspaces/<workspace-
id>/runs/<run-id>

View test runs and results in the Playwright

portal
You can now troubleshoot the failed test cases in the Playwright portal.

1. After your test run completes, a link to the Playwright Portal is generated. Open
this link to view detailed test results and associated artifacts. The portal displays
essential information, including:

CI build details
Overall test run status
The commit ID linked to the test run


2. The Playwright portal provides all the necessary information for troubleshooting.
You can:

Switch between retries.

View detailed error logs, test steps, and attached artifacts such as screenshots
or videos.
Navigate directly to the Trace Viewer for deeper analysis.

3. The Trace Viewer allows you to step through your test execution visually. You can:

Use the timeline to hover over individual steps, revealing the page state
before and after each action.
Inspect detailed logs, DOM snapshots, network activity, errors, and console
output for each step.

 Tip
 You can use Microsoft Playwright Testing service features independently. You can
publish test results to the portal without using the cloud-hosted browsers feature
and you can also use only cloud-hosted browsers to expedite your test suite
without publishing test results.

７ Note

The test results and artifacts that you publish are retained on the service for 90
days. After that, they are automatically deleted.

Optimize parallel worker configuration

Once your tests are running smoothly with the service, experiment with varying the
number of parallel workers to determine the optimal configuration that minimizes test
completion time.

With Microsoft Playwright Testing, you can run with up to 50 parallel workers. Several
factors influence the best configuration for your project, such as the CPU, memory, and
network resources of your client machine, the target application's load-handling
capacity, and the type of actions carried out in your tests.

You can specify the number of parallel workers on the Playwright CLI command-line, or
configure the workers property in the Playwright service configuration file.

Learn more about how to determine the optimal configuration for optimizing test suite
completion.

Next step
You've successfully created a Microsoft Playwright Testing workspace in the Playwright
portal and run your Playwright tests on cloud browsers.

Advance to the next quickstart to set up continuous end-to-end testing by running your
Playwright tests in your CI/CD workflow.

Set up continuous end-to-end testing in CI/CD

Feedback
Was this page helpful?  Yes  No

Provide product feedback

Quickstart: Set up continuous end-to-
end testing with Microsoft Playwright
Testing Preview
Article • 12/30/2024

In this quickstart, you set up continuous end-to-end testing with Microsoft Playwright
Testing Preview to validate that your web app runs correctly across different browsers
and operating systems with every code commit and troubleshoot tests easily using the
service dashboard. Learn how to add your Playwright tests to a continuous integration
(CI) workflow, such as GitHub Actions, Azure Pipelines, or other CI platforms.

After you complete this quickstart, you have a CI workflow that runs your Playwright test
suite at scale and helps you troubleshoot tests easily with Microsoft Playwright Testing.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Prerequisites
An Azure account with an active subscription. If you don't have an Azure
subscription, create a free account before you begin.

A Microsoft Playwright Testing workspace. Complete the quickstart: run Playwright

tests at scale and create a workspace.

GitHub Actions

A GitHub account. If you don't have a GitHub account, you can create one for
free .
A GitHub repository that contains your Playwright test specifications and
GitHub Actions workflow. To create a repository, see Creating a new
repository .
A GitHub Actions workflow. If you need help with getting started with GitHub
Actions, see create your first workflow
 Set up authentication from GitHub Actions to Azure. See Use GitHub Actions
to connect to Azure

Get the service region endpoint URL

In the service configuration, you have to provide the region-specific service endpoint.
The endpoint depends on the Azure region you selected when creating the workspace.

To get the service endpoint URL and store it as a CI workflow secret, perform the
following steps:

1. Sign in to the Playwright portal with your Azure account.

2. On the workspace home page, select View setup guide.

 Tip

If you have multiple workspaces, you can switch to another workspace by

selecting the workspace name at the top of the page, and then select Manage
all workspaces.

3. In Add region endpoint in your setup, copy the service endpoint URL.

The endpoint URL matches the Azure region that you selected when creating the
workspace.

4. Store the service endpoint URL in a CI workflow secret:

ﾉ Expand table

Secret name Value

PLAYWRIGHT_SERVICE_URL Paste the endpoint URL you copied previously.

Add service configuration file

If you don't have Playwright tests configured to run with the service, add a service
configuration file to your repository. In the next step, you then specify this service
configuration file on the Playwright CLI.
1. Create a new file playwright.service.config.ts alongside the
playwright.config.ts file.

Optionally, use the playwright.service.config.ts file in the sample repository .

2. Add the following content to it:

TypeScript

import { defineConfig } from '@playwright/test';

import { getServiceConfig, ServiceOS } from '@azure/microsoft-
playwright-testing';
import config from './playwright.config';

/* Learn more about service configuration at https://aka.ms/mpt/config

*/
export default defineConfig(
config,
getServiceConfig(config, {
exposeNetwork: '<loopback>',
timeout: 30000,
os: ServiceOS.LINUX,
useCloudHostedBrowsers: true
}),
{
/*
Playwright Testing service reporter is added by default.
This will override any reporter options specified in the base
playwright config.
If you are using more reporters, please update your configuration
accordingly.
*/
reporter: [['list'], ['@azure/microsoft-playwright-
testing/reporter']],
}
);

By default, the service configuration enables you to:

Accelerate build pipelines by running tests in parallel using cloud-hosted

browsers.
Simplify troubleshooting with easy access to test results and artifacts
published to the service.

However, you can choose to use either of these features or both. See How to use
service features and update the service configuration file as per your requirements.

3. Save and commit the file to your source code repository.

Update package.json file
Update the package.json file in your repository to add details about Microsoft
Playwright Testing service package in devDependencies section.

TypeScript

"devDependencies": {
"@azure/microsoft-playwright-testing": "^1.0.0-beta.6"
}

Enable artifacts in Playwright configuration

In the playwright.config.ts file of your project, make sure you're collecting all the
required artifacts.

TypeScript

use: {
trace: 'on-first-retry',
video:'retain-on-failure',
screenshot:'on'
},

Set up authentication
The CI machine running Playwright tests must authenticate with Playwright Testing
service to get the browsers to run the tests and to publish the test results and artifacts.

The service offers two authentication methods: Microsoft Entra ID and Access Tokens.
We strongly recommend using Microsoft Entra ID to authenticate your pipelines.

Set up authentication using Microsoft Entra ID

GitHub Actions

If you're using GitHub Actions, you can connect to the service using GitHub OpenID
Connect. Follow the steps to set up the integration:

Prerequisites
Option 1: Microsoft Entra application

Create a Microsoft Entra application with a service principal by Azure portal,

Azure CLI, or Azure PowerShell.

Copy the values for Client ID, Subscription ID, and Directory (tenant) ID to
use later in your GitHub Actions workflow.

Assign the Owner or Contributor role to the service principal created in the
previous step. These roles must be assigned on the Playwright Testing
workspace. For more details, see how to manage access.

Configure a federated identity credential on a Microsoft Entra application to

trust tokens issued by GitHub Actions to your GitHub repository.

Option 2: User-assigned managed identity

Create a user-assigned managed identity.

Copy the values for Client ID, Subscription ID, and Directory (tenant) ID to
use later in your GitHub Actions workflow.

Assign the Owner or Contributor role to the user-assigned managed identity

created in the previous step. These roles must be assigned on the Playwright
Testing workspace. For more details, see how to manage access.

Configure a federated identity credential on a user-assigned managed identity

to trust tokens issued by GitHub Actions to your GitHub repository.

Create GitHub secrets

Add the values you got in the previous step as secrets to your GitHub
repository. See set up GitHub Action Secret. These variables are used in the
GitHub Action workflow in subsequent steps.

ﾉ Expand table

GitHub Secret Source (Microsoft Entra Application or Managed Identity)

AZURE_CLIENT_ID Client ID

AZURE_SUBSCRIPTION_ID Subscription ID

AZURE_TENANT_ID Directory (Tenant) ID

 ７ Note

For enhanced security, it is strongly recommended to use GitHub Secrets to

store sensitive values rather than including them directly in your workflow file.

Set up authentication using access tokens

Ｕ Caution

We strongly recommend using Microsoft Entra ID for authentication to the service.

If you're using access tokens, see How to manage access tokens

You can generate an access token from your Playwright Testing workspace and use it in
your setup. However, we strongly recommend Microsoft Entra ID for authentication due
to its enhanced security. Access tokens, while convenient, function like long-lived
passwords and are more susceptible to being compromised.

1. Authentication using access tokens is disabled by default. To use, Enable access-

token based authentication.

2. Set up authentication using access tokens.

3. Store the access token in a CI workflow secret and use it in the GitHub Actions
workflow or Azure Pipeline yaml file.

ﾉ Expand table

Secret name Value

PLAYWRIGHT_SERVICE_ACCESS_TOKEN Paste the value of Access Token you created previously.

Update the workflow definition

Update the CI workflow definition to run your Playwright tests with the Playwright CLI.
Pass the service configuration file as an input parameter for the Playwright CLI. You
configure your environment by specifying environment variables.

1. Open the CI workflow definition

2. Add the following steps to run your Playwright tests in Microsoft Playwright
Testing.

The following steps describe the workflow changes for GitHub Actions or Azure
Pipelines. Similarly, you can run your Playwright tests by using the Playwright CLI in
other CI platforms.

GitHub Actions

yml

# This step is to sign-in to Azure to run tests from GitHub

Action workflow.
# Choose how to set up authentication to Azure from GitHub
Actions. This is one example.
name: Playwright Tests (Microsoft Playwright Testing)
on:
push:
branches: [main, master]
pull_request:
branches: [main, master]

permissions: # Required when using Microsoft Entra ID to

authenticate
id-token: write
contents: read

jobs:
test:
timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Login to Azure with AzPowershell (enableAzPSSession

true)
uses: azure/login@v2
with:
client-id: ${{ secrets.AZURE_CLIENT_ID }} # GitHub Open
ID connect values copied in previous steps
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
enable-AzPSSession: true

- name: Install dependencies

working-directory: path/to/playwright/folder # update
accordingly
run: npm ci

- name: Run Playwright tests

working-directory: path/to/playwright/folder # update
 accordingly
env:
# Regional endpoint for Microsoft Playwright Testing
PLAYWRIGHT_SERVICE_URL: ${{
secrets.PLAYWRIGHT_SERVICE_URL }}
# PLAYWRIGHT_SERVICE_ACCESS_TOKEN: ${{
secrets.PLAYWRIGHT_SERVICE_ACCESS_TOKEN }} # Not recommended, use
Microsoft Entra ID authentication.
run: npx playwright test -c playwright.service.config.ts
--workers=20

- name: Upload Playwright report

uses: actions/upload-artifact@v3
if: always()
with:
name: playwright-report
path: path/to/playwright/folder/playwright-report/ #
update accordingly
retention-days: 10

3. Save and commit your changes.

When the CI workflow is triggered, your Playwright tests run in your Microsoft
Playwright Testing workspace on cloud-hosted browsers, across 20 parallel
workers. The results and artifacts collected are published to the service and can be
viewed on service portal.

The settings for your test run can be defined in .runsettings file. For more information,
see how to use service package options

７ Note

Reporting feature is enabled by default for existing workspaces. This is being rolled
out in stages and will take a few days. To avoid failures, confirm that Rich
diagnostics using reporting setting is ON for your workspace before proceeding.

For more information, see, Enable reporting for workspace.

Ｕ Caution

With Microsoft Playwright Testing, you get charged based on the number of total
test minutes consumed and test results published. If you're a first-time user or
getting started with a free trial, you might start with running a single test at scale
instead of your full test suite to avoid exhausting your free test minutes and test
results.
After you validate that the test runs successfully, you can gradually increase the test load
by running more tests with the service.

You can run a single test with the service by using the following command-line:

PowerShell

npx playwright test {name-of-file.spec.ts} --

config=playwright.service.config.ts

View test runs and results in the Playwright

portal
You can now troubleshoot the CI pipeline in the Playwright portal,

1. After your test run completes, a link to the Playwright Portal is generated. Open
this link to view detailed test results and associated artifacts. The portal displays
essential information, including:

CI build details
Overall test run status
The commit ID linked to the test run

2. The Playwright portal provides all the necessary information for troubleshooting.
You can:

Switch between retries.

View detailed error logs, test steps, and attached artifacts such as screenshots
or videos.
Navigate directly to the Trace Viewer for deeper analysis.
 

3. The Trace Viewer allows you to step through your test execution visually. You can:

Use the timeline to hover over individual steps, revealing the page state
before and after each action.
Inspect detailed logs, DOM snapshots, network activity, errors, and console
output for each step.

 Tip

You can use Microsoft Playwright Testing service features independently. You can
publish test results to the portal without using the cloud-hosted browsers feature
and you can also use only cloud-hosted browsers to expedite your test suite
without publishing test results. For details, see How to use service features.

７ Note
 The test results and artifacts that you publish are retained on the service for 90
days. After that, they are automatically deleted.

Related content
You've successfully set up a continuous end-to-end testing workflow to run your
Playwright tests at scale on cloud-hosted browsers.

Grant users access to the workspace

Manage your workspaces

Feedback
Was this page helpful?  Yes  No

Provide product feedback

Quickstart: Troubleshoot tests with
Microsoft Playwright Testing Preview
Article • 11/19/2024

In this quickstart, you learn how to troubleshoot your Playwright tests easily using
reports and artifacts published on Microsoft Playwright Testing Preview. Additionally,
this guide demonstrates how to utilize the reporting feature, regardless of whether
you're running tests on the cloud-hosted browsers provided by the service.

After you complete this quickstart, you'll have a Microsoft Playwright Testing workspace
to view test results and artifacts in the service portal.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Background
Microsoft Playwright Testing service enables you to:

Accelerate build pipelines by running tests in parallel using cloud-hosted browsers.

Simplify troubleshooting by publishing test results and artifacts to the service,
making them accessible through the service portal.

These two features of the service can be used independently or together, and each has
its own pricing plan . You have the flexibility to:

Expedite test runs and streamline troubleshooting by using both features: running
tests in cloud-hosted browsers and publishing results to the service.
Run tests only in cloud-hosted browsers to finish test runs faster.
Publish test results to the service while continuing to run tests locally for efficient
troubleshooting.

７ Note
 This article focuses on how you can publish test results to the service without using
cloud-hosted browsers. If you want to learn how to also accelerate your test runs,
see quickstart: run Playwright tests at scale

Prerequisites
An Azure account with an active subscription. If you don't have an Azure
subscription, create a free account before you begin.
Your Azure account needs the Owner, Contributor, or one of the classic
administrator roles.
A Playwright project. If you don't have project, create one by using the Playwright
getting started documentation or use our Microsoft Playwright Testing sample
project .
Azure CLI. If you don't have Azure CLI, see Install Azure CLI.

Create a workspace
To get started with publishing test results on Playwright Testing service, first create a
Microsoft Playwright Testing workspace in the Playwright portal.

1. Sign in to the Playwright portal with your Azure account.

2. If you already have a workspace, select an existing workspace, and move to the
next step.

 Tip

If you have multiple workspaces, you can switch to another workspace by

selecting the workspace name at the top of the page, and then select Manage
all workspaces.

3. If you don't have a workspace yet, select + New workspace, and then provide the
following information:

ﾉ Expand table

Field Description

Workspace Enter a unique name to identify your workspace.

name The name can only consist of alphanumerical characters, and have a
length between 3 and 64 characters.
 Field Description

Azure Select the Azure subscription that you want to use for this Microsoft
subscription Playwright Testing workspace.

Region Select a geographic location to host your workspace.

This is the location where the test run data is stored for the workspace.

4. Select Create workspace to create the workspace in your subscription.

During the workspace creation, a new resource group and a Microsoft Playwright
Testing Azure resource are created in your Azure subscription.

When the workspace creation finishes, you're redirected to the setup guide.

Install Microsoft Playwright Testing package

To use the service, install the Microsoft Playwright Testing package.

npm

npm init @azure/microsoft-playwright-testing@latest

７ Note

Make sure your project uses @playwright/test version 1.47 or above.

This command generates playwright.service.config.ts file which serves to:

 Direct and authenticate your Playwright client to the Microsoft Playwright Testing
service.
Adds a reporter to publish test results and artifacts.

If you already have this file, the prompt asks you to override it.

To use only reporting feature for the test run, disable cloud-hosted browsers by setting
useCloudHostedBrowsers as false.

TypeScript

export default defineConfig(

config,
getServiceConfig(config, {
timeout: 30000,
os: ServiceOS.LINUX,
useCloudHostedBrowsers: false // Do not use cloud hosted browsers
}),
{
reporter: [['list'], ['@azure/microsoft-playwright-testing/reporter']],
// Reporter for Microsoft Playwright Testing service
}
);

Setting the value as false ensures that cloud-hosted browsers aren't used to run the
tests. The tests run on your local machine but the results and artifacts are published on
the service.

 Tip

If you wish to accelerate your test run using cloud-hosted browser, you can set
useCloudHostedBrowsers as true. This will run your tests on the service managed

browsers.

Configure the service region endpoint

In your setup, you have to provide the region-specific service endpoint. The endpoint
depends on the Azure region you selected when creating the workspace.

To get the service endpoint URL, perform the following steps:

1. In Add region endpoint in your setup, copy the region endpoint for your
workspace.
 The endpoint URL matches the Azure region that you selected when creating the
workspace. Make sure this URL is available in PLAYWRIGHT_SERVICE_URL environment
variable.

Set up your environment

To set up your environment, you have to configure the PLAYWRIGHT_SERVICE_URL
environment variable with the value you obtained in the previous steps.

We recommend that you use the dotenv module to manage your environment. With
dotenv , you define your environment variables in the .env file.

1. Add the dotenv module to your project:

shell

npm i --save-dev dotenv

2. Create a .env file alongside the playwright.config.ts file in your Playwright

project:

PLAYWRIGHT_SERVICE_URL={MY-REGION-ENDPOINT}

Make sure to replace the {MY-REGION-ENDPOINT} text placeholder with the value you
copied earlier.

Set up authentication
To publish test results and artifacts to your Microsoft Playwright Testing workspace, you
need to authenticate the Playwright client where you're running the tests with the
service. The client could be your local dev machine or CI machine.

The service offers two authentication methods: Microsoft Entra ID and Access Tokens.
Microsoft Entra ID uses your Azure credentials, requiring a sign-in to your Azure account
for secure access. Alternatively, you can generate an access token from your Playwright
workspace and use it in your setup.

Set up authentication using Microsoft Entra ID

Microsoft Entra ID is the default and recommended authentication for the service. From
your local dev machine, you can use Azure CLI to sign-in

CLI

az login

７ Note

If you're a part of multiple Microsoft Entra tenants, make sure you sign in to the
tenant where your workspace belongs. You can get the tenant ID from Azure portal.
For more information, see Find your Microsoft Entra Tenant. Once you get the ID,
sign-in using the command az login --tenant <TenantID>

Set up authentication using access tokens

You can generate an access token from your Playwright Testing workspace and use it in
your setup. However, we strongly recommend Microsoft Entra ID for authentication due
to its enhanced security. Access tokens, while convenient, function like long-lived
passwords and are more susceptible to being compromised.

1. Authentication using access tokens is disabled by default. To use, Enable access-

token based authentication

2. Set up authentication using access tokens

Ｕ Caution

We strongly recommend using Microsoft Entra ID for authentication to the service.

If you are using access tokens, see How to Manage Access Tokens

Enable artifacts in your Playwright setup

In the playwright.config.ts file of your project, make sure you are collecting all the
required artifacts.

TypeScript

use: {
trace: 'on-first-retry',
video:'retain-on-failure',
screenshot:'on'
}

Run your tests and publish results on Microsoft

Playwright Testing
You've now prepared the configuration for publishing test results and artifacts with
Microsoft Playwright Testing. Run tests using the newly created
playwright.service.config.ts file and publish test results and artifacts to the service.

Bash

npx playwright test --config=playwright.service.config.ts

７ Note

For the Reporting feature of Microsoft Playwright Testing, you get charged based
on the number test results published. If you're a first-time user or getting started
with a free trial, you might start with publishing single test result instead of your
full test suite to avoid exhausting your free trial limits.

After the test completes, you can view the test status in the terminal.

Output

Running 6 test using 2 worker

5 passed, 1 failed (20.2s)

Test report: https://playwright.microsoft.com/workspaces/<workspace-

id>/runs/<run-id>

Ｕ Caution
 Depending on the size of your test suite, you might incur additional charges for the
test results beyond your allotted free test results.

View test runs and results in the Playwright

portal
You can now troubleshoot the failed test cases in the Playwright portal.

1. After your test run completes, a link to the Playwright Portal is generated. Open
this link to view detailed test results and associated artifacts. The portal displays
essential information, including:

CI build details
Overall test run status
The commit ID linked to the test run

2. The Playwright portal provides all the necessary information for troubleshooting.
You can:

Switch between retries.

View detailed error logs, test steps, and attached artifacts such as screenshots
or videos.
Navigate directly to the Trace Viewer for deeper analysis.
 

3. The Trace Viewer allows you to step through your test execution visually. You can:

Use the timeline to hover over individual steps, revealing the page state
before and after each action.
Inspect detailed logs, DOM snapshots, network activity, errors, and console
output for each step.

 Tip

You can also use Microsoft Playwright Testing service to run tests in parallel using
cloud-hosted browsers. Both Reporting and cloud-hosted browsers are
independent features and are billed separately. You can use either of these or both.
For details, see How to use service features

７ Note
 The test results and artifacts that you publish are retained on the service for 90
days. After that, they are automatically deleted.

Next step
You've successfully created a Microsoft Playwright Testing workspace in the Playwright
portal and run your Playwright tests on cloud browsers.

Advance to the next quickstart to set up continuous end-to-end testing by running your
Playwright tests in your CI/CD workflow.

Set up continuous end-to-end testing in CI/CD

Feedback
Was this page helpful?  Yes  No

Provide product feedback

Determine the optimal test suite
configuration
Article • 10/04/2023

Microsoft Playwright Testing Preview enables you to speed up your Playwright test
execution by increasing parallelism at cloud scale. Several factors affect the completion
time for your test suite. Determining the optimal configuration for reducing test suite
completion time is application-specific and requires experimentation. This article
explains the different levels to configure parallelism for your tests, the factors that
influence test duration, and how to determine your optimal configuration to minimize
test completion time.

In Playwright, you can run tests in parallel by using worker processes. By using Microsoft
Playwright Testing, you can further increase parallelism by using cloud-hosted browsers.
In general, adding more parallelism reduces the time to complete your test suite.
However, adding more worker processes doesn't always result in shorter test suite
completion times. For example, the client machine computing resources, network
latency, or test complexity might also affect test duration.

The following chart gives an example of running a test suite. By running the test suite
with Microsoft Playwright Testing instead of locally, you can significantly increase the
parallelism and reduce the test completion time. Notice that, when running with the
service, the completion time reaches a minimum limit, after which adding more workers
only has a minimal effect. The chart also shows how using more computing resources on
the client machine positively affects the test completion time for tests running with the
service.
 

Worker processes
In Playwright , all tests run in worker processes. These processes are OS processes,
running independently, in parallel, orchestrated by the Playwright Test runner. All
workers have identical environments and each process starts its own browser.

Generally, increasing the number of parallel workers can reduce the time it takes to
complete the full test suite. You can learn more about Playwright Test parallelism in
the Playwright documentation.

As previously shown in the chart, the test suite completion time doesn't continue to
decrease as you add more worker processes. There are other factors that influence the
test suite duration.

Run tests locally

By default, @playwright/test limits the number of workers to 1/2 of the number of CPU
cores on your machine. You can override the number of workers for running your test.

When you run tests locally, the number of worker processes is limited to the number of
CPU cores on your machine. Beyond a certain point, adding more workers leads to
resource contention, which slows down each worker and introduces test flakiness.

To override the number of workers using the --workers command line flag :

Bash
 npx playwright test --workers=10

To specify the number of workers in playwright.config.ts using the workers setting:

TypeScript

export default defineConfig({

...
workers: 10,
...
});

Run tests with the service

When you use Microsoft Playwright Testing, you can increase the number of workers at
cloud-scale to larger numbers. When you use the service, the worker processes continue
to run locally, but the resource-intensive browser instances are now running remotely in
the cloud.

Because the worker processes still run on the client machine (developer workstation or
CI agent machine), the client machine might still become a bottleneck for scalable
execution as you add more workers. Learn how you can determine the optimal
configuration.

You can specify the number of workers on the command line with the --workers flag:

Bash

npx playwright test --config=playwright.service.config.ts --workers=30

Alternately you can specify the number of workers in playwright.service.config.ts

using the workers setting:

TypeScript

export default defineConfig({

...
workers: 30,
...
});

Factors that influence completion time

In addition to the number of parallel worker processes, there are several factors that
influence the test suite completion time.

Factor Effects on test duration

Client machine The worker processes still run on the client machine (developer
compute resources workstation or CI agent machine) and need to communicate with the
remote browsers. Increasing the number of parallel workers might
result in resource contention on the client machine, and slow down
tests.

Complexity of the test As the complexity of the test code increases, the time to complete the
code tests might also increase.

Latency between the The workers run on the client machine and communicate with the
client machine and the remote browsers. Depending on the Azure region where the browsers
remote browsers are hosted, the network latency might increase. Learn how you can
optimize regional latency in Microsoft Playwright Testing.

Playwright Playwright settings such as service timeouts, retries, or tracing can

configuration settings adversely affect the test completion time. Experiment with the optimal
configuration for these settings when running your tests in the cloud.

Target application's Running tests with Microsoft Playwright Testing enables you to run
load-handling capacity with higher parallelism, which results in a higher load on the target
application. Verify that the application can handle the load that is
generated by running your Playwright tests.

Learn more about the workflow for determining the optimal configuration for
minimizing the test suite duration.

Workflow for determining your optimal

configuration
The optimal configuration for minimizing the test suite completion time is specific to
your application and environment. To determine your optimal configuration, experiment
with different levels of parallelization, client machine hardware configuration, or test
suite setup.

The following approach can help you find the optimal configuration for running your
tests with Microsoft Playwright Testing:

1. Determine your test completion time goal

Determine what is an acceptable test suite completion time and associated cost per test
run.

Depending on the scenario, your requirements for test completion might be different.
When you're running your end-to-end tests with every code change, as part of a
continuous integration (CI) workflow, minimizing test completion time is essential. When
you schedule your end-to-end tests in a (nightly) batch run, you might have
requirements that are less demanding.

2. Verify that your tests run correctly on the client

machine
Before you run your Playwright test suite with Microsoft Playwright Testing, make sure
that your tests run correctly on your client machine. If you run your tests as part of a CI
workflow, validate that your tests run correctly on the CI agent machine. Ensure that you
run your tests with a minimum of two parallel workers to verify that your tests are
properly configured for parallel execution. Learn more about parallelism in Playwright .

3. Run with cloud-hosted browsers on Microsoft

Playwright Testing
Once your tests run correctly, add the service configuration to run your tests on cloud-
hosted browsers with the service. Validate that your tests continue to run correctly from
your client machine (developer workstation or CI agent machine).

Get started with the Quickstart: run Playwright tests at scale with Microsoft Playwright
Testing.

4. Verify the Azure region remote browsers

Microsoft Playwright Testing can use remote browsers in the Azure region that's nearest
to your client machine, or use the fixed region on which your workspace was created.

Learn how you can optimize regional latency for your workspace.

5. Experiment with the number of parallel workers

Experiment with the number of parallel workers to run your tests. Measure the test
completion time and compare against the target goal you set previously.
Notice at which point the test completion time no longer reduces as you add more
workers. Move to the next step to further optimize your setup.

７ Note

While the service is in preview, the number of parallel workers per workspace is
limited to 50. You can request an increase of this limit for your workspace .

6. Scale the client

As you increase parallelism, the client machine might experience compute resource
contention. Increase the computing resources on the client machine, for example by
selecting larger GitHub-hosted runners .

Alternatively, if you have hardware limitations, you can shard your client tests.

Rerun your tests and experiment with the number of parallel workers.

7. Update your Playwright test configuration settings

Configure your Playwright test configuration settings, such as test timeouts , trace
settings, or retries .

Related content
Run your Playwright tests at scale with Microsoft Playwright Testing
What is Microsoft Playwright Testing
Optimize regional latency for a
workspace in Microsoft Playwright
Testing Preview
Article • 10/04/2023

Learn how to minimize the network latency between the client machine and the remote
browsers for a Microsoft Playwright Testing Preview workspace.

Microsoft Playwright Testing lets you run your Playwright tests on hosted browsers in
the Azure region that's nearest to your client machine. The service collects the test
results in the Azure region of the remote browsers, and then transfers the results to the
workspace region.

By default, when you create a new workspace, the service runs tests in an Azure region
closest to the client machine. When you disable this setting on the workspace, the
service uses remote browsers in the Azure region of the workspace.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Configure regional settings for a workspace

You can configure the regional settings for your workspace in the Azure portal.

1. Sign in to the Playwright portal with your Azure account.

2. Select the workspace settings icon, and then go to the General page to view the
workspace settings.
 

3. Select Select region to go to your workspace in the Azure portal.

Alternately, you can go directly to the Azure portal and select your workspace:
a. Sign in to the Azure portal .
b. Enter Playwright Testing in the search box, and then select Playwright Testing
in the Services category.
c. Select your Microsoft Playwright Testing workspace from the list.

4. In your workspace, select Region Management in the left pane.

5. Choose the corresponding region selection option.

By default, the service uses remote browsers in the Azure region that's closest to
the client machine to minimize latency.

Related content
Learn more about how to determine the optimal configuration for optimizing test
suite completion.

Manage a Microsoft Playwright Testing workspace

Understand how Microsoft Playwright Testing works

Manage workspaces in Microsoft
Playwright Testing Preview
Article • 10/04/2023

In this article, you create, view, and delete Microsoft Playwright Testing Preview
workspaces. You can access and manage a workspace in the Azure portal or in the
Playwright portal.

The following table lists the differences in functionality, based on how you access
Microsoft Playwright Testing:

Functionality Azure Playwright Learn more

portal portal

Create a workspace Yes Yes Quickstart: run Playwright test in

the cloud

View the list of workspaces Yes Yes View all workspaces

View the workspace activity No Yes Display activity log

log

Delete a workspace Yes Yes Delete a workspace

Configure region affinity Yes No Configure region affinity

Grant or revoke access to a Yes No Manage workspace access

workspace

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Prerequisites
An Azure account with an active subscription. Create an account for free .

Create a workspace
To get started with running your Playwright tests on cloud browsers, you first create a
Microsoft Playwright Testing workspace. You can create a workspace in either the Azure
portal or the Playwright portal.

Playwright portal

When you create a workspace in the Playwright portal, the service creates a new
resource group and a Microsoft Playwright Testing Azure resource in your Azure
subscription. The name of the new resource group is based on the workspace
name.

1. Sign in to the Playwright portal with your Azure account.

2. If you already have a workspace, select an existing workspace, and move to

the next step.

 Tip

If you have multiple workspaces, you can switch to another workspace by

selecting the workspace name at the top of the page, and then select
Manage all workspaces.

3. If you don't have a workspace yet, select + New workspace, and then provide
the following information:

Field Description

Workspace Enter a unique name to identify your workspace.

name The name can only consist of alphanumerical characters, and have a
length between 3 and 64 characters.

Azure Select the Azure subscription that you want to use for this
subscription Microsoft Playwright Testing workspace.

Region Select a geographic location to host your workspace.

This is the location where the test run data is stored for the
workspace.
 

4. Select Create workspace to create the workspace in your subscription.

During the workspace creation, a new resource group and a Microsoft

Playwright Testing Azure resource are created in your Azure subscription.

Display the list of workspaces

To get the list of Playwright workspaces that you have access to:

Playwright portal

1. Sign in to the Playwright portal with your Azure account.

2. Select your current workspace in the top of the screen, and then select
Manage all workspaces.


 3. On the Workspaces page, you can now see all the workspaces that you have
access to.

The page shows a card for each of the workspaces in the currently selected
Azure subscription. You can switch to another subscription by selecting a
subscription from the list.

 Tip

Notice that the workspace card indicates if the workspace is included in a

free trial.

4. Select a workspace to view the workspace details and activity log.

Display the workspace activity log

You can view the list of test runs for the workspace in the Playwright portal. Microsoft
Playwright Testing only stores test run metadata, and doesn't store the test code, test
results, trace files, or other artifacts.

The workspace activity log lists for each test run the following details:

Total test duration of the test suite

Maximum number of parallel browsers
Total time across all parallel browsers. This is the time that you're billed for the test
run.

To view the list of test runs in the Playwright portal:

 1. Sign in to the Playwright portal with your Azure account.

2. Optionally, switch to another workspace by selecting your current workspace in the

top of the screen, and then select Manage all workspaces.

3. On the workspace home page, you can view the workspace activity log.

Delete a workspace
To delete a Playwright workspace:

Playwright portal

1. Sign in to the Playwright portal with your Azure account.

2. Select your current workspace in the top of the screen, and then select
Manage all workspaces.

3. On the Workspaces page, select the ellipsis (...) next to your workspace, and
then select Delete workspace.


 4. On the Delete Workspace page, select Delete to confirm the deletion of the
workspace.

２ Warning

Deleting a workspace is an irreversible action. The workspace and the

activity log can't be recovered.

Related content
Optimize regional latency for a workspace

Manage workspace access

Get started with running Playwright tests at scale

Learn more about the Microsoft Playwright Testing resource limits

Manage workspace access tokens in
Microsoft Playwright Testing Preview
Article • 10/04/2023

In this article, you learn how to manage workspace access tokens in Microsoft
Playwright Testing Preview. You use access tokens to authenticate and authorize access
to your workspace.

Access tokens are associated with a user account and workspace. When you use an
access token for running Playwright tests, the service checks your Azure role-based
access control (Azure RBAC) role to verify if you're granted access to run tests on the
service. Learn more about workspace access in Microsoft Playwright Testing.

You can create multiple access tokens per workspace, for example to distinguish
between running tests interactively or as part of your continuous integration (CI)
workflow. When you create an access token, the token has a limited lifespan.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Prerequisites
An Azure account with an active subscription. Create an account for free .
A Microsoft Playwright Testing workspace. To create a workspace, see Quickstart:
Run Playwright tests at scale.
To create or delete access tokens, your Azure account needs to have the
Contributor or Owner role at the workspace level. Learn more about managing
access to a workspace.

Protect your access tokens

Your workspace access tokens are similar to a password for your Microsoft Playwright
Testing workspace. Always be careful to protect your access tokens. Avoid distributing
access tokens to other users, hard-coding them, or saving them anywhere in plain text
that is accessible to others.

Revoke and recreate your tokens if you believe they may have been compromised.

View all workspace access tokens

You can view the list of access tokens for your workspace in the Playwright portal. For
each token, the list displays the token name, status, and expiration date. You can't
retrieve the access token value after it has been created.

You can only view the list of access tokens for the workspaces you have access to.

To view the list of workspace access tokens:

1. Sign in to the Playwright portal with your Azure account.

2. Select your workspace.

3. Select the settings icon on the home page to go to the workspace settings.

4. Select the Access tokens page to view the list of access tokens.

The Access tokens page displays the list of access tokens for the workspace.

Generate a workspace access token

Create an access token to authorize access to your Microsoft Playwright Testing
workspace, and to run existing Playwright tests in your workspace. You can create
multiple access tokens for your workspace. When you create an access token, you have
to specify an expiration date for the token. After a token expires, you need to create a
new access token.

When you use an access token, the service checks the Azure RBAC role of the user that
is associated with the access token to verify that the required permissions are granted.
For example, if you have the Reader role, you can't run Playwright tests but you can view
the test results. When there are role assignment changes, the service checks the
permissions at the time you perform the action.

To create a new workspace access token:

1. Sign in to the Playwright portal with your Azure account.

2. Select your workspace.

3. Select the settings icon on the home page to go to the workspace settings.

4. On the Access tokens page, select Generate new token.

5. Enter the access token details, and then select Generate token.
 

6. Copy the access token for the workspace.

You can save the access token in a CI/CD secrets store or use it in an environment
variable for running tests interactively.

） Important

You can only access the token value immediately after you've created it. You
can't access the token value anymore at a later time.

７ Note
 The number of access tokens per user and per workspace is limited. For more
information, see the Microsoft Playwright Testing service limits.

Delete an access token

You can only delete access tokens that you created in a workspace. To create an access
token:

1. Sign in to the Playwright portal with your Azure account.

2. Select your workspace.

3. Select the settings icon on the home page to go to the workspace settings.

4. On the Access tokens page, select Delete next to the access token that you want
to delete.

5. Select Delete on the deletion confirmation page.

Ｕ Caution

You can't undo the delete operation of an access token. Any existing scripts for
running tests with this token will fail after deleting the access token.

Related content
Learn more about managing access to a workspace.
 Learn more about managing authentication to the workspace

Feedback
Was this page helpful?  Yes  No

Provide product feedback

Manage access to a workspace in
Microsoft Playwright Testing Preview
Article • 10/11/2023

In this article, you learn how to manage access to a workspace in Microsoft Playwright
Testing Preview. The service uses Azure role-based access control (Azure RBAC) to
authorize access rights to your workspace. Role assignments are the way you control
access to resources using Azure RBAC.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Prerequisites
To assign roles in Azure, your account needs the User Access Administrator, Owner,
or one of the classic administrator roles.

To verify your permissions in the Azure portal:

1. In the Azure portal , go to your Microsoft Playwright Testing workspace.

2. On the left pane, select Access Control (IAM), and then select View my
access.

Default roles
Microsoft Playwright Testing workspaces uses three Azure built-in roles. To grant users
access to a workspace, you assign them one of the following Azure built-in roles:

Role Access level

Reader - Read-only access to the workspace in the Playwright portal.

- View test results for the workspace.
- Can't create or delete workspace access tokens.
Can't run Playwright tests on the service.
 Role Access level

Contributor - Full access to manage the workspace in the Azure portal but can't assign roles in
Azure RBAC.
- Full access to the workspace in the Playwright portal.
- Create and delete their access tokens.
- Run Playwright tests on the service.

Owner - Full access to manage the workspace in the Azure portal, including assigning roles
in Azure RBAC.
- Full access to the workspace in the Playwright portal.
- Create and delete their access tokens.
- Run Playwright tests on the service.

） Important

Before you assign an Azure RBAC role, determine the scope of access that is
needed. Best practices dictate that it's always best to grant only the narrowest
possible scope. Azure RBAC roles defined at a broader scope are inherited by the
resources beneath them. For more information about scope for Azure RBAC role
assignments, see Understand scope for Azure RBAC.

Grant access to a user

You can grant a user access to a Microsoft Playwright Testing workspace by using the
Azure portal:

1. Sign in to the Playwright portal with your Azure account.

2. Select the workspace settings icon, and then go to the Users page.

3. Select Manage users for your workspace in the Azure portal to go to your
workspace in the Azure portal.
 Alternately, you can go directly to the Azure portal and select your workspace:
a. Sign in to the Azure portal .
b. Enter Playwright Testing in the search box, and then select Playwright Testing
in the Services category.
c. Select your Microsoft Playwright Testing workspace from the list.
d. On the left pane, select Access Control (IAM).

4. On the Access Control (IAM) page, select Add > Add role assignment.

If you don't have permissions to assign roles, the Add role assignment option is
disabled.

5. On the Role tab, select Privileged administrator roles.

6. Select one of the Microsoft Playwright Testing default roles, and then select Next.

7. On the Members tab, make sure User, group, or service principal is selected.

8. Select Select members, find and select the users, groups, or service principals.
 

9. Select Review + assign to assign the role.

For more information about how to assign roles, see Assign Azure roles using the
Azure portal.

Revoke access for a user

You can revoke a user's access to a Microsoft Playwright Testing workspace using the
Azure portal:

1. In the Azure portal , go to your Microsoft Playwright Testing workspace.

2. On the left pane, select Access Control (IAM), and then select Role assignments.

3. In the list of role assignments, add a checkmark next to the user and role you want
to remove, and then select Remove.


 4. Select Yes in the confirmation window to remove the role assignment.

For more information about how to remove role assignments, see Remove Azure
role assignments.

(Optional) Use Microsoft Entra security groups

to manage workspace access
Instead of granting or revoking access to individual users, you can manage access for
groups of users using Microsoft Entra security groups. This approach has the following
benefits:

Avoid the need for granting team or project leaders the Owner role on the
workspace. You can grant them access only to the security group to let them
manage access to the workspace.
You can organize, manage and revoke users' permissions on a workspace and
other resources as a group, without having to manage permissions on a user-by-
user basis.
Using Microsoft Entra groups helps you to avoid reaching the subscription limit on
role assignments.

To use Microsoft Entra security groups:

1. Create a security group.

2. Add a group owner. This user has permissions to add or remove group members.
The group owner isn't required to be group member, or have direct RBAC role on
the workspace.

3. Assign the group an RBAC role on the workspace, such as Reader or Contributor.

4. Add group members. The added members can now access to the workspace.

Create a custom role for restricted tenants

If you're using Microsoft Entra tenant restrictions and users with temporary access, you
can create a custom role in Azure RBAC to manage permissions and grant access to run
tests.

Perform the following steps to manage permissions with a custom role:

1. Follow these steps to create an Azure custom role.

 2. Select Add permissions, and enter Playwright in the search box, and then select
Microsoft.AzurePlaywrightService.

3. Select the microsoft.playwrightservice/accounts/write permission, and then

select Add.

4. Follow these steps to add a role assignment for the custom role to the user
account.

The user can now continue to run tests in the workspace.

Troubleshooting
Here are a few things to be aware of while you use Azure role-based access control
(Azure RBAC):

When you create a resource in Azure, such as a workspace, you are not
automatically the owner of the resource. Your role is inherited from the highest
scope role that you're authorized against in that subscription. As an example, if
you're a Contributor for the subscription, you have the permissions to create a
Microsoft Playwright Testing workspace. However, you would be assigned the
Contributor role against that workspace, and not the Owner role.

When there are two role assignments to the same Microsoft Entra user with
conflicting sections of Actions/NotActions, your operations listed in NotActions
from one role might not take effect if they're also listed as Actions in another role.
 To learn more about how Azure parses role assignments, read How Azure RBAC
determines if a user has access to a resource.

It can sometimes take up to 1 hour for your new role assignments to take effect
over cached permissions.

Related content
Get started with running Playwright tests at scale

Learn how to manage Playwright Testing workspaces

Use cloud-hosted browsers for locally
deployed or privately hosted apps with
Microsoft Playwright Testing Preview
Article • 11/19/2024

Learn how to use Microsoft Playwright Testing Preview to run end-to-end tests for
locally deployed applications. Microsoft Playwright Testing uses cloud-hosted, remote
browsers for running Playwright tests at scale. You can use the service to run tests for
apps on localhost, or that you host on your infrastructure.

Playwright enables you to expose networks that are available on the client machine to
remote browsers. When you expose a network, you can connect to local resources from
your Playwright test code without having to configure additional firewall settings.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Configure Playwright to expose local networks

To expose local networks and resources to remote browsers, you can use the
exposeNetwork option in Playwright. Learn more about the exposeNetwork option in
the Playwright documentation.

You can specify one or multiple networks by using a list of rules. For example, to expose
test/staging deployments and localhost : *.test.internal-domain,*.staging.internal-
domain,<loopback> .

You can configure the exposeNetwork option in playwright.service.config.ts . The

following example shows how to expose the localhost network by using the
<loopback> rule. You can also replace localhost with a domain that you want to
enable for the service.

TypeScript
 import { getServiceConfig, ServiceOS } from "@azure/microsoft-playwright-
testing";
import { defineConfig } from "@playwright/test";
import { AzureCliCredential } from "@azure/identity";
import config from "./playwright.config";

export default defineConfig(

config,
getServiceConfig(config, {
exposeNetwork: '<loopback>', // Allow service to access the localhost.
}),
);

You can now reference localhost in the Playwright test code, and run the tests on
cloud-hosted browsers with Microsoft Playwright Testing:

Bash

npx playwright test --config=playwright.service.config.ts --workers=20

Related content
Run Playwright tests at scale with Microsoft Playwright Testing
Learn more about writing Playwright tests in the Playwright documentation

Feedback
Was this page helpful?  Yes  No

Provide product feedback

Configure visual comparisons with
Microsoft Playwright Testing Preview
Article • 10/04/2023

In this article, you learn how to properly configure Playwright's visual comparison tests
when using Microsoft Playwright Testing Preview. Unexpected test failures may occur
because Playwright's snapshots differ between local and remote browsers.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Background
The Playwright Test runner uses the host OS as a part of the expected screenshot path. If
you're running tests using remote browsers on a different OS than your host machine,
the visual comparison tests fail. Our recommendation is to only run visual comparisons
when using the service. If you're taking screenshots on the service, there's no need to
compare them to your local setup since they don't match.

Configure ignoreSnapshots
You can use the ignoreSnapshots option to only run visual comparisons when using
Microsoft Playwright Testing.

1. Set ignoreSnapshots: true in the original playwright.config.ts that doesn't use

the service.
2. Set ignoreSnapshots: false in playwright.service.config.ts .

When you're using the service, its configuration overrides playwright.config.ts , and
runs visual comparisons.

Configure the snapshot path

To configure snapshot paths for a particular project or the whole config, you can set
snapshotPathTemplate option .

JavaScript

// This path is exactly like the default path, but replaces OS with
hardcoded value that is used on the service (linux).
config.snapshotPathTemplate = '{snapshotDir}/{testFileDir}/{testFileName}-
snapshots/{arg}{-projectName}-linux{ext}'

// This is an alternative path where you keep screenshots in a separate

directory, one per service OS (linux in this case).
config.snapshotPathTemplate =
'{testDir}/__screenshots__/{testFilePath}/linux/{arg}{ext}';

Example service config

Example service config that runs visual comparisons and configures the path for
snapshotPathTemplate :

typeScript

import { defineConfig } from '@playwright/test';

import { getServiceConfig, ServiceOS } from '@azure/microsoft-playwright-
testing';
import config from './playwright.config';

/* Learn more about service configuration at https://aka.ms/mpt/config */

export default defineConfig(
config,
getServiceConfig(config, {
exposeNetwork: '<loopback>',
timeout: 30000,
os: ServiceOS.LINUX
}),
{
/*
Playwright Testing service reporter is added by default.
This will override any reporter options specified in the base playwright
config.
If you are using more reporters, please update your configuration
accordingly.
*/
reporter: [["list"], ['@azure/microsoft-playwright-testing/reporter']],
ignoreSnapshots: false,
// Enable screenshot testing and configure directory with expectations.
snapshotPathTemplate:
`{testDir}/__screenshots__/{testFilePath}/${ServiceOS.LINUX}/{arg}{ext}`,
 }
);

Related content
Learn more about Playwright Visual Comparisons .

Feedback
Was this page helpful?  Yes  No

Provide product feedback

Monitoring Microsoft Playwright Testing
Preview
Article • 10/04/2023

When you have critical applications and business processes relying on Azure resources,
you want to monitor those resources for their availability, performance, and operation.
This article describes the monitoring data generated by Microsoft Playwright Testing
Preview.

Microsoft Playwright Testing creates monitoring data using Azure Monitor, which is a full
stack monitoring service in Azure. Azure Monitor provides a complete set of features to
monitor your Azure resources. It can also monitor resources in other clouds and on-
premises. Learn more about monitoring Azure resources with Azure Monitor.

 Tip

To understand costs associated with Azure Monitor, see Usage and estimated
costs . To understand the time it takes for your data to appear in Azure Monitor,
see Log data ingestion time.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Monitoring data
Microsoft Playwright Testing collects the same kinds of monitoring data as other Azure
resources that are described in Monitoring data from Azure resources.

See Monitor Microsoft Playwright Testing data reference for detailed information on
logs metrics created by Microsoft Playwright Testing.

Collection and routing

Platform metrics and the Activity log are collected and stored automatically, but can be
routed to other locations by using a diagnostic setting. Resource Logs are not collected
and stored until you create a diagnostic setting and route them to one or more
locations.

See Create diagnostic setting to collect platform logs and metrics in Azure for the
detailed process for creating a diagnostic setting using the Azure portal, CLI, or
PowerShell. When you create a diagnostic setting, you specify which categories of logs
to collect.

The log categories for Microsoft Playwright Testing are listed in Monitor Microsoft
Playwright Testing data reference.

Analyzing logs
Data in Azure Monitor Logs is stored in tables where each table has its own set of
unique properties. All resource logs in Azure Monitor have the same fields followed by
service-specific fields. The common schema is outlined in Azure Monitor resource log
schema.

You can find the schema for Microsoft Playwright Testing resource logs in the Monitor
Microsoft Playwright Testing data reference.

The Activity log is a type of Azure platform log that provides insight into subscription-
level events. You can view it independently or route it to Azure Monitor Logs, where you
can do much more complex queries using Log Analytics.

For a list of resource logs types collected for Microsoft Playwright Testing, see Monitor
Microsoft Playwright Testing data reference.

Related content
See Monitor Microsoft Playwright Testing data reference for a reference of the
metrics, logs, and other important values created by Microsoft Playwright Testing.

See Monitoring Azure resources with Azure Monitor for details on monitoring
Azure resources.
Use Microsoft Playwright Testing
Reporting preview with Playwright
sharding
Article • 12/30/2024

In this article, you learn how to use the Microsoft Playwright Testing service's reporting
feature with test runs that use Playwright's sharding features .

Playwright's sharding enables you to split your test suite to run across multiple
machines simultaneously. This feature helps running tests in parallel.

You can use Playwright Testing's reporting feature to get a consolidated report of a test
run with sharding. Ensure the runId of the test run is same across all shards.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Prerequisites
An Azure account with an active subscription. If you don't have an Azure
subscription, create a free account before you begin.

Set up continuous end-to-end testing. Complete the Quickstart: Set up continuous

end-to-end testing with Microsoft Playwright Testing Preview to set up continuous
integration (CI) pipeline.

Update playwright service configuration file

Add or update runId to playwright.service.config.ts file in your setup.

TypeScript

export default defineConfig(

config,
 getServiceConfig(config, {
runId: process.env.PLAYWRIGHT_SERVICE_RUN_ID, //Set runId for the test run
}),
);

You can use PLAYWRIGHT_SERVICE_RUN_ID variable in your setup to ensure the runId
remains same across all shards.

Set up variables
The runId setting is used as an identifier by Playwright Testing service to distinguish
between test runs. The results from multiple runs with same runId are reported to the
same run on the Playwright portal.

By default, a test run that uses reporting feature automatically generates a unique runId
unless you explicitly set the value yourself. If the value of the variable remains same
across runs, the results are reported together in the same run on the Playwright portal.

 Tip

If you use the cloud-hosted browsers provided by Microsoft Playwright Testing

service to run your tests, you might have already set this variable. To avoid
overwrites, make sure you set it only once.

While using sharding, make sure the same runId is set across all the shards for the
results to be reported together. Use the variable PLAYWRIGHT_SERVICE_RUN_ID and set the
value same across all shards.

Here's an example of how you can set it in your pipeline via GitHub Actions.

yml

name: Playwright Tests (Microsoft Playwright Testing)

on:
push:
branches: [ main, master ]
pull_request:
branches: [ main, master ]
workflow_dispatch:
strategy:
fail-fast: false
matrix:
shardIndex: [1, 2, 3, 4]
shardTotal: [4]
 permissions: # Required when using Microsoft Entra ID to authenticate
id-token: write
contents: read
jobs:
test:
timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: 18
# This step is to sign-in to Azure to run tests from GitHub Action
workflow.
# You can choose how set up Authentication to Azure from GitHub
Actions, this is one example.
- name: Login to Azure with AzPowershell (enableAzPSSession true)
uses: azure/login@v2
with:
client-id: ${{ secrets.AZURE_CLIENT_ID }} # Required when using
Microsoft Entra ID to authenticate
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
enable-AzPSSession: true

- name: Install dependencies

working-directory: path/to/playwright/folder # update accordingly
run: npm ci

- name: Run Playwright tests

working-directory: path/to/playwright/folder # update accordingly
env:
# Regional endpoint for Microsoft Playwright Testing
PLAYWRIGHT_SERVICE_URL: ${{ secrets.PLAYWRIGHT_SERVICE_URL }}
PLAYWRIGHT_SERVICE_RUN_ID: ${{ github.run_id }}-${{
github.run_attempt }}-${{ github.sha } #This Run_ID will be unique and will
remain same across all shards
run: npx playwright test --shard=${{ matrix.shardIndex }}/${{
matrix.shardTotal }}

Feedback
Was this page helpful?  Yes  No

Provide product feedback

Manage authentication and
authorization for Microsoft Playwright
Testing preview
Article • 11/19/2024

In this article, you learn how to manage authentication and authorization for Microsoft
Playwright Testing preview. Authentication is required to run Playwright tests on cloud-
hosted browsers and to publish test results and artifacts to the service.

By default, Microsoft Entra ID is used for authentication. This method is more secure and
is the recommended authentication method. You can't disable authentication using
Microsoft Entra ID. However, you can also use access tokens to authenticate and
authorize.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Background
Microsoft Playwright Testing Preview is built on the Playwright open-source framework.
It runs Playwright tests on cloud-hosted browsers and publishes reports and artifacts
back to the service.

To use the service, the client must authenticate with the service to access the browsers.
Similarly, publishing results and artifacts requires authenticated API interactions. The
service offers two authentication methods: Microsoft Entra ID and access tokens.

Microsoft Entra ID uses your Azure credentials, requiring a sign-in to your Azure account
for secure access. Alternatively, you can generate an access token from your Playwright
workspace and use it in your setup. However, we strongly recommend Microsoft Entra
ID for authentication due to its enhanced security. Access tokens, while convenient,
function like long-lived passwords and are more susceptible to being compromised.

Enable authentication using access-tokens

Microsoft Playwright Testing service also supports authentication using access tokens.
This authentication method is less secure. We recommend using Microsoft Entra ID to
authenticate to the service.

Ｕ Caution

Your workspace access tokens are similar to a password for your Microsoft
Playwright Testing workspace. Always be careful to protect your access tokens.
Avoid distributing access tokens to other users, hard-coding them, or saving them
anywhere in plain text that is accessible to others.

Revoke and recreate your tokens if you believe they are compromised.

To enable authentication using access tokens:

1. Sign in to the Playwright portal with your Azure account and select your
workspace.

2. Select the settings icon on the home page to go to the workspace settings.

3. Select the Authentication page and turn on Enable authentication using Access
tokens

Ｕ Caution

Authentication using access tokens is less secure. Learn how to manage access
tokens

Set up authentication using access-tokens

1. While running the tests, enable access token auth in the
playwright.service.config.ts file in your setup.
 TypeScript

/* Learn more about service configuration at https://aka.ms/mpt/config

*/
export default defineConfig(config, getServiceConfig( config {
serviceAuthType:'ACCESS_TOKEN'
}));

2. Create access token

Follow the steps to create an access token. Copy the value of the access token
generated.

3. Set up your environment

To set up your environment, configure the PLAYWRIGHT_SERVICE_ACCESS_TOKEN

environment variable with the value you obtained in the previous steps. Ensure this
environment variable is available in your setup where you are running tests.

We recommend that you use the dotenv module to manage your environment.
With dotenv , you define your environment variables in the .env file.

a. Add the dotenv module to your project:

shell

npm i --save-dev dotenv

b. Create a .env file alongside the playwright.config.ts file in your Playwright

project:

PLAYWRIGHT_SERVICE_ACCESS_TOKEN={MY-ACCESS-TOKEN}

Make sure to replace the {MY-ACCESS-TOKEN} text placeholder with the value you
copied earlier.

Run tests on the service and publish results

Run Playwright tests against cloud-hosted browsers and publish the results to the
service using the configuration you created above.

TypeScript
 npx playwright test --config=playwright.service.config.ts --workers=20

Related content
Learn more about managing access tokens.

Feedback
Was this page helpful?  Yes  No

Provide product feedback

Use features of Microsoft Playwright
Testing preview
Article • 11/19/2024

In this article, you learn how to use the features provided by Microsoft Playwright
Testing preview.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Prerequisites
An Azure account with an active subscription. Create an account for free .
A Microsoft Playwright Testing workspace. To create a workspace, see Quickstart:
Run Playwright tests at scale.
To manage features, your Azure account needs to have the Contributor or Owner
role at the workspace level. Learn more about managing access to a workspace.

Background
Microsoft Playwright Testing preview allows you to:

Accelerate build pipelines by running tests in parallel using cloud-hosted browsers.

Simplify troubleshooting by publishing test results and artifacts to the service,
making them easily accessible through the service portal.

These features have their own pricing plans and are billed separately. You can choose to
use either feature or both. These features can be enabled or disabled for the workspace
or for any specific run. To know more about pricing, see Microsoft Playwright Testing
preview pricing

Manage feature for the workspace

1. Sign in to the Playwright portal with your Azure account.
 2. Select the workspace settings icon, and then go to the General page to view the
workspace settings.

3. Navigate to Feature management section.

4. Choose the features you want to enable for your workspace.

Currently, you can choose to enable or disable only reporting feature of the
service. By default, reporting is enabled for the workspace.

Manage features while running tests

You can also choose to use either feature or both for a test run.

） Important

You can only use a feature in a test run if it is enabled for the workspace.
1. In your Playwright setup, go to playwright.service.config.ts file and use these
settings for feature management.

TypeScript

import { getServiceConfig, ServiceOS } from "@azure/microsoft-playwright-

testing";
import { defineConfig } from "@playwright/test";
import { AzureCliCredential } from "@azure/identity";
import config from "./playwright.config";

export default defineConfig(

config,
getServiceConfig(config, {
useCloudHostedBrowsers: true, // Select if you want to use cloud-hosted
browsers to run your Playwright tests.
}),
{
reporter: [
["list"],
["@azure/microsoft-playwright-testing/reporter"], //Microsoft
Playwright Testing reporter
],
},
);

useCloudHostedBrowsers :
Description: This setting allows you to choose whether to use cloud-hosted
browsers or the browsers on your client machine to run your Playwright tests. If
you disable this option, your tests run on the browsers of your client machine
instead of cloud-hosted browsers, and you do not incur any charges. You can
still configure reporting options.
Default Value: true
Example:

TypeScript

useCloudHostedBrowsers: true

reporter

Description: The playwright.service.config.ts file extends the Playwright

configuration file of your setup. This option overrides the existing reporters and
sets the Microsoft Playwright Testing reporter. You can add or modify this list to
include the reporters you want to use. You are billed for Microsoft Playwright
Testing reporting if you add @azure/microsoft-playwright-testing/reporter .
This feature can be used independently of cloud-hosted browsers, meaning you
 don’t have to run tests on service-managed browsers to get reports and
artifacts on the Playwright portal.
Default Value: ["@azure/microsoft-playwright-testing/reporter"]
Example:

TypeScript

reporter: [
["list"],
["@azure/microsoft-playwright-testing/reporter"]],

Related content
Learn more about Microsoft Playwright Testing preview pricing .

Feedback
Was this page helpful?  Yes  No

Provide product feedback

Use options available in service package
with Microsoft Playwright Testing
preview
Article • 11/19/2024

This article shows you how to use the options available in the
playwright.service.config.ts file that was generated for you. If you don't have this file

in your code, follow Quickstart: Run end-to-end tests at scale with Microsoft Playwright
Testing Preview

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Prerequisites
Follow the Quickstart guide and set up a project to run with Microsoft Playwright
Testing service. See, Quickstart: Run end-to-end tests at scale with Microsoft
Playwright Testing Preview

Here's version of the playwright.service.config.ts file with all the available options:

TypeScript

import { getServiceConfig, ServiceOS } from "@azure/microsoft-playwright-

testing";
import { defineConfig } from "@playwright/test";
import { AzureCliCredential } from "@azure/identity";
import config from "./playwright.config";

export default defineConfig(

config,
getServiceConfig(config, {
serviceAuthType:'ACCESS_TOKEN' // Use this option when you want to
authenticate using access tokens. This mode of auth should be enabled for
the workspace.
os: ServiceOS.WINDOWS, // Select the operating system where you want to
run tests.
runId: new Date().toISOString(), // Set a unique ID for every test run
 to distinguish them in the service portal.
credential: new AzureCliCredential(), // Select the authentication
method you want to use with Entra.
useCloudHostedBrowsers: true, // Select if you want to use cloud-hosted
browsers to run your Playwright tests.
exposeNetwork: '<loopback>', // Use this option to connect to local
resources from your Playwright test code without having to configure
additional firewall settings.
timeout: 30000 // Set the timeout for your tests.
}),
{
reporter: [
["list"],
[
"@azure/microsoft-playwright-testing/reporter",
{
enableGitHubSummary: true, // Enable/disable GitHub summary in
GitHub Actions workflow.
},
],
],
},
);

Settings in playwright.service.config.ts file

serviceAuthType :

Description: This setting allows you to choose the authentication method you
want to use for your test run.
Available Options:
ACCESS_TOKEN to use access tokens. You need to enable authentication using
access tokens if you want to use this option, see manage authentication.
ENTRA_ID to use Microsoft Entra ID for authentication. It's the default mode.

Default Value: ENTRA_ID

Example:

TypeScript

serviceAuthType:'ENTRA_ID'

os :

Description: This setting allows you to choose the operating system where the
browsers running Playwright tests are hosted.
Available Options:
ServiceOS.WINDOWS for Windows OS.
 ServiceOS.LINUX for Linux OS.

Default Value: ServiceOS.LINUX

Example:

TypeScript

os: ServiceOS.WINDOWS

runId :

Description: This setting allows you to set a unique ID for every test run to
distinguish them in the service portal. Using the same runId for multiple test
runs results in error. For sharding, keep this same across all shards.
Example:

TypeScript

runId: new Date().toISOString()

credential :

Description: This setting allows you to select the authentication method you
want to use with Microsoft Entra ID.
Example:

TypeScript

credential: new AzureCliCredential()

useCloudHostedBrowsers

Description: This setting allows you to choose whether to use cloud-hosted

browsers or the browsers on your client machine to run your Playwright tests. If
you disable this option, your tests run on the browsers of your client machine
instead of cloud-hosted browsers, and you don't incur any charges.
Default Value: true
Example:

TypeScript

useCloudHostedBrowsers: true

exposeNetwork

Description: This setting allows you to connect to local resources from your
Playwright test code without having to configure another firewall settings. To
 learn more, see how to test local applications
Example:

TypeScript

exposeNetwork: '<loopback>'

timeout

Description: This setting allows you to set timeout for your tests connecting to
the cloud-hosted browsers.
Example:

TypeScript

timeout: 30000,

reporter

Description: The playwright.service.config.ts file extends the playwright

config file of your setup. This option overrides the existing reporters and sets
Microsoft Playwright Testing reporter. You can add or modify this list to include
the reporters that you want to use. You're billed for Microsoft Playwright Testing
reporting if you add @azure/microsoft-playwright-testing/reporter .
Default Value: ["@azure/microsoft-playwright-testing/reporter"]
Example:

TypeScript

reporter: [
["list"],
["@azure/microsoft-playwright-testing/reporter"],

enableGitHubSummary :

Description: This setting allows you to configure the Microsoft Playwright

Testing service reporter. You can choose whether to include the test run
summary in the GitHub summary when running in GitHub Actions.
Default Value: true
Example:

TypeScript

reporter: [
["list"],
[
 "@azure/microsoft-playwright-testing/reporter",
{
enableGitHubSummary: true,
},
],
]

Feedback
Was this page helpful?  Yes  No

Provide product feedback

Tutorial: Run end-to-end Playwright
tests with Microsoft Playwright Testing
service
Article • 12/19/2024

In this tutorial, you learn how to integrate your Playwright test suite with Microsoft
Playwright Testing, run tests faster using cloud-hosted browsers, and troubleshoot
efficiently using the service's reporting features. You simulate a Playwright test suite,
connect it to the service for faster execution, and use reporting tools for streamlined
troubleshooting.

In this tutorial, you:

＂ Set up a Playwright test suite.

＂ Integrate the Playwright test suite with Microsoft Playwright Testing service.
＂ Run the test suite with the service for faster execution and efficient troubleshooting.

Prerequisites
An Azure account with an active subscription. If you don't have an Azure
subscription, create a free account before you begin.
The Azure CLI installed on your local computer.
Azure CLI version 2.2.0 or later. Run az --version to check the installed version on
your computer. If you need to install or upgrade the Azure CLI, see How to install
the Azure CLI.
Visual Studio Code. If you don't have it, download and install it .
Git. If you don't have it, download and install it .

Prerequisites check
Before you start, validate your environment:

Sign in to the Azure portal and check that your subscription is active.

Check your version of the Azure CLI in a terminal or command window by running
az --version . For the latest version, see the latest release notes.

If you don't have the latest version, update your installation by following the
installation guide for your operating system or platform.
Set up Playwright test suite
In this step, you are creating a Playwright test suite which is integrated with the service.

1. Clone the sample repository and navigate to test folder.

PowerShell

git clone https://github.com/microsoft/playwright-testing-service

cd playwright-testing-service/samples/get-started

2. Install dependencies.

PowerShell

npm install

3. Run Playwright tests.

Run this command to execute tests locally, outside of the service, to identify any
problems before integrating with the service. This project is used in the next steps to
integrate with the service.

PowerShell

npx playwright test

Integrate Playwright test suite with Microsoft

Playwright Testing service
Integrate the Playwright test suite you created in the previous tutorial with Playwright
Testing service.

Follow these steps to set up the service and integrate the test suite.

Create a Playwright Testing workspace

To get started with running your Playwright tests at scale on cloud browsers, you first
create a Microsoft Playwright Testing workspace in the Playwright portal.

1. Sign in to the Playwright portal with your Azure account.

2. If you already have a workspace, select an existing workspace, and move to the
next step.

 Tip

If you have multiple workspaces, you can switch to another workspace by

selecting the workspace name at the top of the page, and then select Manage
all workspaces.

3. If you don't have a workspace yet, select + New workspace, and then provide the
following information:

ﾉ Expand table

Field Description

Workspace Enter a unique name to identify your workspace.

name The name can only consist of alphanumerical characters, and have a
length between 3 and 64 characters.

Azure Select the Azure subscription that you want to use for this Microsoft
subscription Playwright Testing workspace.

Region Select a geographic location to host your workspace.

This is the location where the test run data is stored for the workspace.

4. Select Create workspace to create the workspace in your subscription.

During the workspace creation, a new resource group and a Microsoft Playwright
Testing Azure resource are created in your Azure subscription.
When the workspace creation finishes, you're redirected to the setup guide.

Install Microsoft Playwright Testing package

To install service package, navigate to the location of your test suite you created in the
previous tutorial and run this command:

npm

npm init @azure/microsoft-playwright-testing@latest

This command generates playwright.service.config.ts file which serves to:

Direct and authenticate Playwright to the Microsoft Playwright Testing service.

Adds a reporter to publish test results and artifacts.

Configure the service region endpoint

In your setup, you have to provide the region-specific service endpoint. The endpoint
depends on the Azure region you selected when creating the workspace.

To get the service endpoint URL, perform the following steps:

1. In Add region endpoint in your setup, copy the region endpoint for your
workspace.

The endpoint URL matches the Azure region that you selected when creating the
workspace. Ensure this URL is available in PLAYWRIGHT_SERVICE_URL environment
variable.

Set up your environment

To set up your environment, you have to configure the PLAYWRIGHT_SERVICE_URL
environment variable with the value you obtained in the previous steps.

We recommend that you use the dotenv module to manage your environment. With
dotenv , you define your environment variables in the .env file.
 1. Add the dotenv module to your project:

shell

npm i --save-dev dotenv

2. Create a .env file alongside the playwright.config.ts file in your Playwright

project:

PLAYWRIGHT_SERVICE_URL={MY-REGION-ENDPOINT}

Make sure to replace the {MY-REGION-ENDPOINT} text placeholder with the value you
copied earlier.

Set up authentication
To run your Playwright tests in your Microsoft Playwright Testing workspace, you need
to authenticate the Playwright client where you're running the tests with the service.
Authenticate with the dev workstation where you are running the Playwright tests.

Microsoft Entra ID is the default and recommended authentication for the service. Use
Azure CLI to sign-in

CLI

az login

７ Note

If you're a part of multiple Microsoft Entra tenants, make sure you sign in to the
tenant where your workspace belongs. You can get the tenant ID from Azure portal.
See Find your Microsoft Entra Tenant. Once you get the ID, sign-in using the
command az login --tenant <TenantID>

Enable artifacts in your Playwright setup

In the playwright.config.ts file of your project, ensure you are collecting all the
required artifacts.
 TypeScript

use: {
trace: 'on-first-retry',
video:'retain-on-failure',
screenshot:'on'
}

Run your tests at scale and troubleshoot easily

with Microsoft Playwright Testing
You've now prepared the configuration for running your Playwright tests in the cloud
with Microsoft Playwright Testing.

Run Playwright tests with the service

With Microsoft Playwright Testing, you get charged based on the number of total test
minutes and number of test results published. If you're a first-time user or getting
started with a free trial, you may run a single test at scale instead of your full test suite
to avoid exhausting your free trial limits.

Follow these steps to run Playwright tests with Microsoft Playwright Testing:

1. Open a terminal window.

2. Enter the following command to run your Playwright test suite on remote browsers
and publish test results to your workspace.

Bash

npx playwright test --config=playwright.service.config.ts --workers=20

After the test completes, you can view the test status in the terminal.

Output

Running 600 tests using 20 workers

600 passed (3m)

Test report: https://playwright.microsoft.com/workspaces/<workspace-

id>/runs/<run-id>

View test runs and results in the Playwright portal

You can now troubleshoot the failed test cases in the Playwright portal.

1. After your test run completes, a link to the Playwright Portal is generated. Open
this link to view detailed test results and associated artifacts. The portal displays
essential information, including:

CI build details
Overall test run status
The commit ID linked to the test run

2. The Playwright portal provides all the necessary information for troubleshooting.
You can:

Switch between retries.

View detailed error logs, test steps, and attached artifacts such as screenshots
or videos.
Navigate directly to the Trace Viewer for deeper analysis.

3. The Trace Viewer allows you to step through your test execution visually. You can:
 Use the timeline to hover over individual steps, revealing the page state
before and after each action.
Inspect detailed logs, DOM snapshots, network activity, errors, and console
output for each step.

 Tip

You can use Microsoft Playwright Testing service features independently. You can
publish test results to the portal without using the cloud-hosted browsers feature
and you can also use only cloud-hosted browsers to expedite your test suite
without publishing test results.

７ Note

The test results and artifacts that you publish are retained on the service for 90
days. After that, they are automatically deleted.

Next steps
Set up continuous end-to-end testing in CI/CD
Explore service configuration options
Determine the optimal test suite configuration

Feedback
Was this page helpful?  Yes  No

Provide product feedback

[Resolved] AADSTS7000112: Application
'b1fd4ebf-2bed-4162-be84-
97e0fe523f64'(PlaywrightServiceAADLo
gin) is disabled.
Article • 10/04/2023

Symptoms
When using Microsoft Playwright Testing, you fail to sign into the Playwright portal. You
receive the following error message:

AADSTS7000112: Application 'b1fd4ebf-2bed-4162-be84-

97e0fe523f64'(PlaywrightServiceAADLogin) is disabled.

Cause
This issue occurs if the service principal for Microsoft Playwright Testing is disabled for
the tenant.

Resolution
To resolve this issue, you need to enable the service principal for Microsoft Playwright
Testing for the tenant.

） Important

To enable the service principal, you need to be a tenant admin.

Follow these steps to enable the Microsoft Playwright Testing service principal:

1. Open an elevated Windows PowerShell command prompt (run Windows

PowerShell as an administrator).

2. Install the Microsoft Azure Active Directory module for Windows PowerShell by
running the following cmdlet:

PowerShell
 Install-Module MSOnline

3. Connect to Microsoft Entra ID for your Microsoft 365 subscription by running the
following cmdlet:

PowerShell

Connect-MsolService

4. Check the current status of the service principal for Microsoft Playwright Testing by
running the following cmdlet:

PowerShell

(Get-MsolServicePrincipal -AppPrincipalId b1fd4ebf-2bed-4162-be84-

97e0fe523f64).accountenabled

5. Enable the service principal for Microsoft Playwright Testing by running the
following cmdlet:

PowerShell

Get-MsolServicePrincipal -AppPrincipalId b1fd4ebf-2bed-4162-be84-

97e0fe523f64 | Set-MsolServicePrincipal -AccountEnabled $true
Troubleshoot issues with running tests
with Microsoft Playwright Testing
preview
Article • 10/04/2023

This article addresses issues that might arise when you run Playwright tests at scale with
Microsoft Playwright Testing Preview.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Tests are failing with a 401 Unauthorized error

Your access token may be invalid or expired. Make sure you're using the correct access
token or generate a new access token.

Tests run slow

Microsoft Playwright Testing hosts the remote browsers in specific Azure regions. If your
client machine or target web application is outside these regions, you might experience
increased network latency. Learn how you can optimize regional latency for your
workspace.

Tests seem to hang

Your tests might hang due to a piece of code that's unintentionally paused the test
execution. For example, you might have added pause statements while debugging your
test.

Search for any instances of pause() statements in your code and comment them out.

Tests are failing because of a timeout

Your tests could be timing out because of the following reasons:

Your client machine is in a different region than the browsers.

Connecting to service-hosted browsers introduces network latency. You might

need to increase your timeout settings in the Playwright configuration . Start with
increasing the test timeout setting in playwright.service.config.ts .

Trace files cause performance issues (currently a known problem).

Sending the Playwright trace files from the service to the client machine can create
congestion, which can cause tests to fail due to a timeout.You can disable tracing
in the Playwright configuration file .

Unable to test web applications hosted behind

firewall
Ensure that you set the exposeNetwork option in the playwright.service.config.ts file
to make the network available on the client machine to the cloud browser. Example
values for this option are: <loopback> for the localhost network, * to expose all
networks, or the IP address/DNS of the application endpoint.

Learn how more about how to test locally deployed applications.

The time displayed in the browser is different

from my local time
Web applications often display the time based on the user's location. When you run
tests with Microsoft Playwright Testing, the client machine and the service browsers may
be in different regions.

You can mitigate the issue by specifying the time zone in the Playwright configuration
file .

Test fails with Path is not available when

connecting remotely
You might encounter the Path is not available when connecting remotely error when
you run your Playwright tests on remote browsers with Microsoft Playwright Testing. For
example, when you're testing the functionality to download a file in your test code.
The cause of this issue is that the path() function on the download file instance is not
available when run on remote browsers.

To resolve this issue, you should use the saveAs() function to save a local copy of the
file on your client machine. Learn more about downloads in the Playwright
documentation .

The following code snippet gives an example of how to use saveAs() instead of path()
for reading the contents of a downloaded file:

TypeScript

const downloadPromise = page.waitForEvent('download');

await page.getByText('Download file').click();

const download = await downloadPromise;

// FAILS: download.path() fails when connecting to a remote browser

// const result = fs.readFileSync(await download.path(), 'utf-8');

// FIX: use saveAs() to download the file, when connecting to a remote

browser
await download.saveAs('/path/to/save/at/' + download.suggestedFilename());

Related content
Manage workspace access
Optimize regional latency a workspace
Service limits in Microsoft Playwright
Testing Preview
Article • 12/24/2024

Azure uses limits and quotas to prevent budget overruns due to fraud, and to honor
Azure capacity constraints. Consider these limits as you scale for production workloads.
In this article, you learn about:

Default limits on Azure resources related to Microsoft Playwright Testing Preview.

Limitations for the Playwright test code
Supported operating systems and browsers
Requesting quota increases.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Default resource quotas

While the service is in preview, the following limits apply on a per-subscription basis.

ﾉ Expand table

Resource Limit

Workspaces per region per subscription 2

Parallel workers per workspace 100

Access tokens per user per workspace 10

Test code limitations

The service supports Playwright OSS version 1.47 or higher.
The service supports the Playwright test runner and the NUnit test runner only.
Supported operating systems and browsers
The service supports running hosted browsers on Linux and Windows.
Supports all browsers that Playwright supports .

Other limitations
Moving a workspace to another resource group is not yet supported.
The Playwright Testing service portal is only available in English. Localization in
other languages is currently in progress.

Request quota increases

To raise the resource quota above the default limit for your subscription, create an issue
in the Playwright Testing GitHub repository .

Outbound IP addresses
This section lists the outbound IP address ranges that Microsoft Playwright Testing
requires to communicate through your firewall.

ﾉ Expand table

Azure region IP address range

East US 52.190.15.208/28

West US3 20.172.9.112/28

East Asia 20.24.220.64/28

West Europe 98.71.172.224/28

Related content
Get started and run Playwright tests at scale

Feedback
Was this page helpful?  Yes  No

Provide product feedback

Monitor Microsoft Playwright Testing
Preview data reference
Article • 10/04/2023

Learn about the data and resources collected by Azure Monitor from your workspace in
Microsoft Playwright Testing Preview. See Monitor Microsoft Playwright Testing for
details on collecting and analyzing monitoring data.

） Important

Microsoft Playwright Testing is currently in preview. For legal terms that apply to
Azure features that are in beta, in preview, or otherwise not yet released into
general availability, see the Supplemental Terms of Use for Microsoft Azure
Previews .

Resource logs
This section lists the types of resource logs you can collect for Microsoft Playwright
Testing.

Operational logs
Operational log entries include elements listed in the following table:

Name Description

Time Date and time when the record was created

ResourceId Azure Resource Manager resource ID

Location Azure Resource Manager resource location

OperationName Name of the operation attempted on the resource

Category Category of the emitted log

ResultType Indicates if the request was successful or failed

ResultSignature HTTP status code of the API response

ResultDescription Additional details about the result

Name Description

DurationMs The duration of the operation in milliseconds

CorrelationId Unique identifier to be used to correlate logs

Level Security Level of the log

Related content
See Monitor Microsoft Playwright Testing for a description of monitoring Microsoft
Playwright Testing.
See Monitor Azure resources with Azure Monitor for details on monitoring Azure
resources.