Skip to content

fix: follow pysher thread lifecycle in wait_forever - #4829

Merged
Alberto Schiabel (jkomyno) merged 2 commits into
ComposioHQ:nextfrom
bunnysayzz:fix/wait-forever-pysher-lifecycle-4811
Oct 13, 2026
Merged

Alberto Schiabel (jkomyno) merged 2 commits into
ComposioHQ:nextfrom
bunnysayzz:fix/wait-forever-pysher-lifecycle-4811

Conversation

@bunnysayzz

Copy link
Copy Markdown
Contributor

Summary

TriggerSubscription.wait_forever() in the Python SDK ends on network drops that pysher goes on to recover from, and never ends after pysher gives up for good. It reads websocket-client's has_errored flag, which stays set through pysher's reconnect backoff, so it cannot tell a recoverable drop from a dead connection.

Fixes #4811

Changes

  • has_errored() now follows pysher's connection thread instead of the socket flag: the thread stays alive across reconnects and exits once the connection is dead for good (unrecoverable pusher:error, or stop()).
  • Regression tests covering both directions.

Type of change

  • Bug fix
  • New feature
  • Refactor/Chore
  • Documentation
  • Breaking change

How Has This Been Tested?

  • 4 new tests in TestTriggerSubscriptionStop: socket flagged but thread alive -> not errored; thread dead -> errored; wait_forever returns on thread death; wait_forever survives a 2.5s reconnect backoff and unblocks on stop().
  • Full python/tests/test_triggers.py: 128 passed.
  • ruff check and ruff format clean with the repo's python/config/ruff.toml.
  • Verified against installed pysher 1.0.8 / websocket-client 1.9.2 sources: Connection is a Thread, run() exits when pysher gives up.

Checklist

  • I have read and understand every line of this diff, including any LLM-generated code
  • I ran linters/tests locally and they passed
  • I added tests or explain why not applicable
  • I updated documentation as needed (no doc changes needed)
  • I added a changeset (Python-only change, changesets are for TS packages)

TriggerSubscription.has_errored() read websocket-client's has_errored
flag, which stays set through pysher's reconnect backoff, so
wait_forever() returned on drops pysher would have recovered from,
and never returned after pysher gave up for good (e.g. pusher:error
4000-4099). Track the pysher connection thread instead: it stays
alive across reconnects and exits once the connection is dead.

Fixes ComposioHQ#4811

Signed-off-by: bunnysayzz <[email protected]>
@vercel

vercel Bot commented Oct 13, 2026

Copy link
Copy Markdown

Azhar (@bunnysayzz) is attempting to deploy a commit to the Composio Team on Vercel.

A member of the Team first needs to authorize it.

@parameterai

parameterai Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Risk: Low · 1 low

This round only replaces restart()'s disconnect/_connect sequence with a single reconnect(reconnect_interval=0) request on the running pysher worker (plus a docstring and tests); it removes a thread-affinity bug but does not change has_errored() or wait_forever(). The prior issue stands: because has_errored() mirrors pysher's worker-thread liveness (triggers.py:875-878) and pysher retries plain failures forever, wait_forever() never returns on a permanently dead connection unless the caller calls stop().

Sentinel reviewed 1b05f30 · Review settings

@greptile-apps

greptile-apps Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable new issue was established.

Summary

The PR makes has_errored() follow pysher’s connection thread so wait_forever() can keep waiting during reconnects and return when the worker exits.

Since the previous review:

  • Trigger waits follow the pysher worker through retries and shutdown.
  • Manual restarts reconnect through the existing pysher worker.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A["wait_forever()"] --> B{"Subscription alive?"}
  B -->|No| E["Return"]
  B -->|Yes| C{"Connection worker alive?"}
  C -->|No| E
  C -->|Yes| D["Sleep one second"]
  D --> B
  R["restart()"] --> F["Request reconnect on running worker"]
Loading

Reviews (2) · Last reviewed commit: "fix(python): preserve trigger worker acr..." · Reviewed by Greptile

connection = getattr(self, "_connection", None)
if connection is None:
return True
return not connection.is_alive()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 wait_forever() now hangs forever while pysher endlessly retries a permanently dead connection

A websocket link that never comes back (network partition, or a hostile/compromised realtime endpoint that keeps dropping the socket) leaves pysher's connection thread retrying forever, so has_errored() never fires and a caller parked in wait_forever() is wedged indefinitely with no error surfaced — where the previous code returned after the first drop. pysher only stops retrying on disconnect() or an unrecoverable pusher:error (4000–4099) frame; plain failures retry every reconnect_interval with no cap, as the SDK's own comment at python/composio/core/models/triggers.py:1119-1123 states ("reconnects every reconnect_interval until disconnect is called").

How:

  1. The endpoint or network drops the websocket repeatedly without sending an unrecoverable pusher:error frame.
  2. pysher's run loop keeps reconnecting; the connection thread stays alive across the retries (this PR's own premise).
  3. not connection.is_alive() at python/composio/core/models/triggers.py:878 therefore stays False forever.
  4. The loop at python/composio/core/models/triggers.py:886 never exits; the consumer process blocks until stop() is called from another thread.

Fix: Cap the reconnect attempts (or treat repeated failed reconnects / socket.has_errored while not connected as terminal) so wait_forever() can return once the link is permanently down.


React with 👍 if useful or 👎 if not

Use pysher's reconnect path so thread-liveness checks remain valid after
restart. Cover the real Connection lifecycle and bound terminal-wait tests.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the connection lifecycle and prior feedback. The restart regression is fixed, the real pysher Connection lifecycle test covers worker preservation and terminal exit, and 129 trigger tests plus Ruff checks pass. Indefinite retry matches issue #4811.

@jkomyno
Alberto Schiabel (jkomyno) merged commit 909d3ab into ComposioHQ:next Oct 13, 2026
5 of 6 checks passed

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 1b05f30. Configure here.

"""Request a reconnect on the running subscription worker."""
# Disconnect joins the worker; calling _connect afterwards runs the
# socket on the caller while has_errored sees a terminated worker.
self._connection.reconnect(reconnect_interval=0)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restart disables reconnect backoff

Medium Severity

Calling restart forwards reconnect_interval=0 into pysher reconnect, which stores that value as the worker's ongoing backoff. After one manual reconnect, later recoverable drops retry with no delay, which can spin the worker and get the socket closed for good by Pusher rate limits.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 1b05f30. Configure here.

Alberto Schiabel (jkomyno) added a commit that referenced this pull request Oct 15, 2026
…n handler rejections (#4903)

This PR:
- supersedes #4814 and
#4813, combining both into
one coherent subscription lifecycle and callback error boundary
- closes #4812
- fixes the TypeScript half of
#4810 (the Python half
stays open; see Limitations)
- gives each subscription period (first `subscribe()` → next
`unsubscribe()`) its own pusher-js client: concurrent `subscribe()`
calls share one connection attempt, and `unsubscribe()` aborts setup
still in flight, unbinds the SDK's handlers and disconnects
- routes trigger handlers and `onSubscriptionError` through one
`invokeUserCallback()` boundary that contains synchronous throws and
rejected promises and logs a printable description (`describeThrown()`)
- keeps the containment of async `pusher:subscription_error` failures
from #4448, and also closes a
remaining gap there: a rejection whose value `String()` cannot convert
used to throw inside the `.catch` and escape again
- adds a patch changeset for `@composio/core` / `@composio/slim`

Designated PR reviewer: [@jkomyno](mailto:[email protected]).

## Root causes

These are two independent defects.

**Connection never closed
(#4812
`PusherService.unsubscribe()` unsubscribed the channel but nothing
called `disconnect()`, so the pusher-js websocket kept Node's event loop
alive. It also called `getPusherClient()`, so an `unsubscribe()` before
any `subscribe()` fetched credentials and opened a connection.
`getPusherClient()` checked `this.pusherClient` before two awaits
(credentials, `import('pusher-js')`), so concurrent first `subscribe()`
calls each created a client and the last assignment orphaned the others,
which `unsubscribe()` could never reach.

Reusing one client across periods (the alternative discussed in
#4812) is unsafe with
pusher-js 8.6.0:
- `disconnect()` keeps every channel in the client's table, with its
bound handlers, and the next `connect()` resubscribes all of them
(`subscribeAll()`).
- `unsubscribe()` on a channel whose auth is still pending only sets
`subscriptionCancelled`; the channel stays, and a later `subscribe()` of
the same name reinstates it with the old handlers. The harness below
shows this on `next`: a handler removed by `unsubscribe()` still
received the next event.
- A channel-auth reply requested for the old socket can arrive after a
reconnect and be sent on the new socket.

A fresh client per period isolates all three: late replies and cancelled
channels belong to a disconnected client whose `send_event` is a no-op,
and its handlers are unbound.

**Async handler rejections
(#4810
`Triggers.subscribe()` and `PusherService`'s `safeCallback` called the
handler inside a synchronous `try/catch`. An `async` handler returns a
promise, so its rejection skipped the `catch`, nothing handled it, and
Node's default `unhandledRejection` behavior ended the process.
Separately, the logger `JSON.stringify`s object arguments, so a thrown
`Error` was logged as `{}`.

## Ownership and cancellation model

- `PusherService` holds at most one `attempt` (an `AbortController` plus
the shared `ready` promise) and, once created, the `connection` (client,
channel, channel name). Both are cleared together by `unsubscribe()`.
- `openConnection()` checks `signal.aborted` after its last await; the
client creation and channel subscription after it are synchronous, so
`unsubscribe()` cannot interleave with them.
- A `subscribe()` whose attempt was aborted resolves without
subscribing, including when the aborted setup later fails (for example,
the credentials request rejects). A `subscribe()` that resumes after the
client was created but after `unsubscribe()` ran installs no handlers.
- A failed (not aborted) attempt is forgotten, so the next `subscribe()`
retries.
- `unsubscribe()` ends every subscription of the `Triggers` instance, as
documented ("Unsubscribe from all the triggers"), so it always closes
the connection; there is no other channel on that client to keep it
open.

## Verification

Focused regression tests
(`ts/packages/core/test/services/pusher.test.ts`,
`ts/packages/core/test/models/triggers.test.ts`), using a fake pusher-js
that keeps its binding semantics. 15 of them fail on `next` source and
pass with this change:
- disconnects and unbinds on unsubscribe; unsubscribe first creates no
client and fetches no credentials
- concurrent subscribes share one client and credentials request
- unsubscribe while credentials are pending, and when that pending
request then fails
- unsubscribe after the client exists but before the waiting
`subscribe()` resumes
- unsubscribe interleaved with pending setup keeps only the later
subscription
- resubscribe after cleanup uses a new client; late events and auth
errors on the old client reach nothing
- five cycles leave every client disconnected and unbound; retry after a
credentials failure
- sync throw and async rejection from trigger handlers, non-`Error` and
unprintable rejection values, and an unprintable rejection from
`onSubscriptionError`

Built SDK against unmodified `pusher-js` 8.6.0 and a local
Pusher-protocol (v7) server over TLS, which validates channel signatures
against the socket they arrive on. A preload only redirects the TLS
connection for the fake cluster host to localhost. Node 24.17.0, one
process per scenario; "exits" means the process ended on its own after
the scenario finished:

| Scenario | `next` | This PR |
| --- | --- | --- |
| subscribe → event → unsubscribe | still running, socket open (killed
after 5 s) | exits in ~6 ms, 0 open sockets |
| unsubscribe before subscribe | 1 credentials request, 1 client, still
running | 0 requests, 0 clients, exits |
| unsubscribe → resubscribe → event | both receive, still running | both
receive on separate connections, exits |
| two concurrent subscribes | 2 credentials requests, 2 sockets, both
left open | 1 request, 1 socket, each handler once, exits |
| unsubscribe during initialization | event delivered after unsubscribe,
2 sockets open | no socket opened, nothing delivered, exits |
| async handler rejects | process exits with code 1 (unhandled
rejection) | logs `❌ Error in trigger callback: <message>`, keeps
running, exits 0 |
| auth reply delayed past unsubscribe → resubscribe | unsubscribed
handler still receives the event | unsubscribed handler receives
nothing, 0 stale or invalid subscribes, exits |
| 100 subscribe/unsubscribe cycles | 1 client, never exits | 100 clients
created, 0 connected, 0 open sockets, exits |

Commands:
- `pnpm --filter @composio/core test`: 73 files, 1842 passed, 2 expected
failures
- `pnpm --filter @composio/core typecheck`
- `oxlint` on the changed files: no new warnings (one existing warning
in `Triggers.ts`)
- `pnpm validate:changesets`

## Limitations

- pusher-js 8.6.0 never releases a client: the constructor pushes it
onto the static `Pusher.instances` array, and its `ConnectionManager`
binds `online`/`offline` listeners on the module-level `Network` object
that `disconnect()` does not remove, so each disconnected client stays
reachable. Neither has a documented release API, and the listeners can't
be told apart from other clients' listeners. With pusher-js alone, each
connect/subscribe/disconnect cycle retained about 14 KB. Through the
SDK, the per-cycle growth was about 14.6 KB over 300 cycles, which is
mostly that client. The SDK unbinds its own handlers, so retained
clients don't keep user callbacks alive. A script that subscribes once
is unaffected; an app that cycles subscribe/unsubscribe many times keeps
one disconnected client per cycle.
- Each subscription period fetches realtime credentials again.
- Python is unchanged. `TriggerSubscription.stop()` already disconnects,
and #4829 covered the
connection-thread lifecycle. The Python half of
#4810 (an `async def`
handler is never awaited) is a separate defect that needs a decision
between rejecting async handlers and running them, so this PR doesn't
close that issue.
- `triggers.subscribe()` keeps its `(data) => void` handler type; it
already accepts async functions, and widening the return type would
reject handlers that return other values.

## Credits

Thanks to @Hugo-DDT for #4814
and to @TanishGudise for
#4813 and the investigation
in #4812 and
#4810. Both are co-authors
of the commit.
- From #4814: disconnecting
on unsubscribe, the no-client-on-bare-unsubscribe rule, shared
initialization for concurrent subscribers, and treating setup errors
after cancellation as cancellation. This PR replaces the generation
counter and `allChannels()` check with an aborted attempt per period and
a fresh client per period, which also covers a `subscribe()` that
resumes after `unsubscribe()` and the cancelled-channel and late-auth
cases.
- From #4813: the
`Promise.resolve(...).catch(...)` containment and the
failure-description fallbacks, now shared by both callback paths as
`describeThrown()`. Its regression tests are included unchanged.
- The issue analysis, especially the `Pusher.instances` and `Network`
retention finding and the late-auth concern, shaped the design and the
harness above. Thanks also to @VANDRANKI for the community reviews.

Co-authored-by: DDT <[email protected]>
Co-authored-by: Tanish Gudise <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Python wait_forever() ends on network drops pysher recovers from, and never ends after pysher gives up

2 participants