Skip to content

fix modular_filesystem call itself - #45682

Merged
copybara-service[bot] merged 1 commit into
tensorflow:masterfrom
vnghia:fs-plugin
Dec 28, 2020
Merged

copybara-service[bot] merged 1 commit into
tensorflow:masterfrom
vnghia:fs-plugin

Conversation

@vnghia

@vnghia vnghia commented Dec 15, 2020 •

Copy link
Copy Markdown
Contributor

@google-ml-butler google-ml-butler Bot added the size:S CL Change Size: Small label Dec 15, 2020
@google-cla google-cla Bot added the cla: yes label Dec 15, 2020
@gbaned gbaned self-assigned this Dec 15, 2020
@gbaned gbaned added the awaiting review Pull request awaiting review label Dec 21, 2020
@vnghia

vnghia commented Dec 22, 2020

Copy link
Copy Markdown
Contributor Author

@mihaimaruseac Could you take a look at this PR ? Thank you!

@google-ml-butler google-ml-butler Bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Dec 27, 2020
@kokoro-team kokoro-team removed the kokoro:force-run Tests on submitted change label Dec 27, 2020
@gbaned gbaned removed the awaiting review Pull request awaiting review label Dec 28, 2020
@copybara-service
copybara-service Bot merged commit c06cd62 into tensorflow:master Dec 28, 2020
@vnghia
vnghia deleted the fs-plugin branch December 29, 2020 00:49
@mihaimaruseac

Copy link
Copy Markdown
Contributor

Let's also cherrypick this on the r2.4 branch (if not already)

copybara-service Bot pushed a commit that referenced this pull request Jul 20, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

Reverts changelist 949495726

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45682 from nbhargava:pjrt-non-two-phase-prepare-error 1e96c1e75c504728fe20c950b4d46ea7ad464b59
PiperOrigin-RevId: 950753335
copybara-service Bot pushed a commit that referenced this pull request Jul 20, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

Reverts changelist 949495726

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45682 from nbhargava:pjrt-non-two-phase-prepare-error 1e96c1e75c504728fe20c950b4d46ea7ad464b59
PiperOrigin-RevId: 950753335
copybara-service Bot pushed a commit that referenced this pull request Jul 20, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

Reverts changelist 949495726

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45682 from nbhargava:pjrt-non-two-phase-prepare-error 1e96c1e75c504728fe20c950b4d46ea7ad464b59
PiperOrigin-RevId: 950753335
copybara-service Bot pushed a commit that referenced this pull request Jul 20, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

Reverts changelist 949495726

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45682 from nbhargava:pjrt-non-two-phase-prepare-error 1e96c1e75c504728fe20c950b4d46ea7ad464b59
PiperOrigin-RevId: 950753335
copybara-service Bot pushed a commit that referenced this pull request Jul 21, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45682 from nbhargava:pjrt-non-two-phase-prepare-error 1e96c1e75c504728fe20c950b4d46ea7ad464b59
PiperOrigin-RevId: 950753335
copybara-service Bot pushed a commit that referenced this pull request Jul 21, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45682 from nbhargava:pjrt-non-two-phase-prepare-error 1e96c1e75c504728fe20c950b4d46ea7ad464b59
PiperOrigin-RevId: 950753335
copybara-service Bot pushed a commit that referenced this pull request Jul 21, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45682 from nbhargava:pjrt-non-two-phase-prepare-error 1e96c1e75c504728fe20c950b4d46ea7ad464b59
PiperOrigin-RevId: 950753335
copybara-service Bot pushed a commit that referenced this pull request Jul 21, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45682 from nbhargava:pjrt-non-two-phase-prepare-error 1e96c1e75c504728fe20c950b4d46ea7ad464b59
PiperOrigin-RevId: 950753335
copybara-service Bot pushed a commit that referenced this pull request Jul 23, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45682 from nbhargava:pjrt-non-two-phase-prepare-error 1e96c1e75c504728fe20c950b4d46ea7ad464b59
PiperOrigin-RevId: 950753335
copybara-service Bot pushed a commit that referenced this pull request Jul 23, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45682 from nbhargava:pjrt-non-two-phase-prepare-error 1e96c1e75c504728fe20c950b4d46ea7ad464b59
PiperOrigin-RevId: 950753335
copybara-service Bot pushed a commit that referenced this pull request Jul 23, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45682 from nbhargava:pjrt-non-two-phase-prepare-error 1e96c1e75c504728fe20c950b4d46ea7ad464b59
PiperOrigin-RevId: 950753335
copybara-service Bot pushed a commit that referenced this pull request Jul 23, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45682 from nbhargava:pjrt-non-two-phase-prepare-error 1e96c1e75c504728fe20c950b4d46ea7ad464b59
PiperOrigin-RevId: 950753335
copybara-service Bot pushed a commit that referenced this pull request Jul 23, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45682 from nbhargava:pjrt-non-two-phase-prepare-error 1e96c1e75c504728fe20c950b4d46ea7ad464b59
PiperOrigin-RevId: 950753335
copybara-service Bot pushed a commit that referenced this pull request Jul 24, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#45682 from nbhargava:pjrt-non-two-phase-prepare-error 1e96c1e75c504728fe20c950b4d46ea7ad464b59
PiperOrigin-RevId: 950753335
copybara-service Bot pushed a commit that referenced this pull request Jul 24, 2026
…ice Execute

Imported from GitHub PR openxla/xla#45682

## The bug

`CommonPjRtLoadedExecutable::Execute`'s multi-device path calls `ExecutePrepareWithOomRetries` per device, then — only when `supports_two_phase_launch()` is true — checks the returned status inside the two-phase barrier block. `CpuClient` overrides `supports_two_phase_launch()` to `false`, so on a multi-device CpuClient a Prepare failure is never checked and execution falls through to `ExecuteLaunch(*launch_args, ...)` with `launch_args.executable` still null → null-deref crash instead of a clean error.

## The fix

Add an `else if (!launch_status.ok())` branch after the two-phase block: record the failure in `results[i]`, decrement `launching` under `mu`, return. Same shape as the two-phase failure path minus the barrier it doesn't participate in.

## Scope

`CpuClient` is the only in-tree client with `supports_two_phase_launch()==false`. Two-phase clients (GPU/TPU stream-executor) are unchanged — the diff is purely additive.

## Test

`MultiDevicePrepareFailurePropagatesError` in `[cpu_client_test.cc](http://cpu_client_test.cc/)`: 2-device CpuClient, 2-replica identity program, wrong-size buffer on device 1 so `CheckBufferCompatibilities` rejects it during Prepare. Asserts `Execute()` returns an error (not a crash) and that the error comes from the per-device Prepare path.
Copybara import of the project:

--
1e96c1e75c504728fe20c950b4d46ea7ad464b59 by Nikhil Bhargava <[email protected]>:

pjrt: propagate Prepare failure on non-two-phase multi-device Execute

When supports_two_phase_launch()==false (CpuClient), the multi-device
Execute path never checked the status returned by
ExecutePrepareWithOomRetries — the only check was inside the two-phase
branch. A Prepare failure fell through to ExecuteLaunch with
launch_args.executable still null and crashed.

Add the missing status check on the non-two-phase path: record the
failure in results[i], decrement the launching counter under mu, and
return — mirroring the two-phase failure handling without the barrier.

Test: MultiDevicePrepareFailurePropagatesError in cpu_client_test.cc
creates a 2-device CpuClient, passes a wrong-size buffer on one device
so CheckBufferCompatibilities rejects it during Prepare, and asserts
Execute() returns an error rather than crashing.

Merging this change closes #45682

PiperOrigin-RevId: 953266494
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla: yes ready to pull PR ready for merge process size:S CL Change Size: Small

Projects

None yet

Development

Successfully merging this pull request may close these issues.

New filesystem plugins throw segfault

4 participants