Skip to content

Dev to main, 21st Oct, 2026 - #283

Merged
atharva-bhange merged 9 commits into
mainfrom
dev
Oct 21, 2026
Merged

atharva-bhange merged 9 commits into
mainfrom
dev

Conversation

@khushalsonawat

@khushalsonawat khushalsonawat commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

What does this PR do?

Release the Python traceAI-livekit shutdown-flush fix (TH-8730). The integration flushes pending spans at LiveKit job shutdown and process exit, preserving agent_session and job_shutdown spans that were previously lost when Linux forkserver children exited. Import-time setup re-arms the exit finalizer after multiprocessing clears it. Each shutdown/exit flush waits at most three seconds; multiple providers flush independently.

Adds the public flush_on_job_shutdown() helper for tracing configured outside a job, plus regression tests for real job processes and import-time initialization.

Closes TH-8730

How was it tested?

Existing GitHub Python tests CI is green. The LiveKit job ran 82 tests on Python 3.11 (run 39619546426), including forkserver job shutdown and local OTLP collector integration tests. Core fi_instrumentation checks also passed.

Additional release preparation is being validated on Python 3.10–3.13 before publishing. This PR does not publish to PyPI automatically.

Release scope

Only traceAI-livekit from python/frameworks/livekit will be published. PyPI currently has 0.1.1; pyproject.toml and traceai_livekit.version already agree on 0.1.2. No other package version bump or publication is required. The fi-instrumentation-otel >=1.1.0 dependency is already available on PyPI.

khushalsonawat and others added 6 commits October 19, 2026 22:37
chore: merge main into dev (Python tests CI and the LiveKit fix)
chore: merge main into dev (install failures fail the check, time limits, pipefail)
…-8730)

LiveKit voice traces reached Future AGI without their root
agent_session span. traceAI exports through a batch processor that sends
every 5 s; on Linux a LiveKit job runs in a forkserver process that ends
with os._exit(), which skips atexit, and LiveKit flushes only its own
processors at job end. So the spans that end last, the root among them,
were dropped.

enable_* now adds a shutdown callback to the job it runs in that
flushes the tracer provider. The flush runs in a daemon thread and job
shutdown waits for it at most 10 s, because OpenTelemetry's batch
force_flush ignores its timeout and LiveKit joins the loop's default
executor before the process exits. A new flush_on_job_shutdown() covers
set-up done outside a job; enable_* warns when it runs outside one. On
any livekit-agents 1.x the hook either registers or is skipped, never
raising into the entrypoint.

Tests cover each behaviour, including a real job process started with
forkserver in which agent_session must reach the exporter.

Signed-off-by: khushalsonawat <[email protected]>
…TH-8730)

Review follow-up on the job-shutdown flush.

- The job-shutdown flush now waits at most 3 s, well inside LiveKit's
  10 s exit budget, instead of 10 s.
- A second flush runs when the job process exits (a multiprocessing
  finalizer, which forkserver children run before os._exit). It sends
  the spans LiveKit ends after the shutdown callbacks, such as
  job_shutdown, and covers set-up done outside a job. Also capped at 3 s.
- The once-per-job guard is now per job and per provider, so a second
  provider in the same job gets its own flush.
- Outside a job, a debug line instead of a warning; the comment on
  get_job_context now says required= arrived in livekit-agents 1.5.3.
- Tests for the pre-1.5.3 API during a job, a second provider, the exit
  flush (registered once, flushes every provider, bounded), and the
  real-process test now also requires job_shutdown.
- README and CHANGELOG describe both flushes and their limits.

Signed-off-by: khushalsonawat <[email protected]>
…H-8730)

On Linux a LiveKit job process is a forkserver child. A module it
imports while unpickling its target runs before multiprocessing clears
the child's exit finalizers, so tracing set up at the top of an agent
module lost its exit flush, and the pid check stopped it being
registered again: no spans were sent at all.

- Keep the finalizer and make it again when it is gone or belongs to
  another process; an after-fork hook re-arms it in fork and forkserver
  children right after the clear.
- Register under a lock, so jobs enabling at once in one process
  (LiveKit's thread executor) register one finalizer.
- Flush each provider in its own thread under one deadline, so a hung
  provider does not hold up the others.
- A child keeps the providers it inherited: they are the copies its
  spans go to.
- README: the exit flush sends job_entrypoint only when tracing is set
  up before the job starts.
- Tests: a real forkserver job process with tracing set up at import of
  a module only the child loads, plus the hook, the lock, the pid change
  and the per-provider flush.

Signed-off-by: khushalsonawat <[email protected]>
…shutdown

fix(livekit): flush traceAI's spans when a LiveKit job shuts down (TH-8730)
@khushalsonawat khushalsonawat self-assigned this Oct 21, 2026
atharva-bhange
atharva-bhange previously approved these changes Oct 21, 2026
chore(livekit): prepare and validate 0.1.2 release
@atharva-bhange
atharva-bhange merged commit 4a91b2d into main Oct 21, 2026
12 checks passed
nik13 added a commit that referenced this pull request Oct 22, 2026
The repo now releases dev -> main (#283), and #271/#272 target dev, so
this PR moves to dev too. Changing the base does not start the
pull_request workflows; this empty commit does.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants