Skip to content

fix(discord): keep live voice transcription bounded - #42611

Open
joelneleber wants to merge 1 commit into
NousResearch:mainfrom
joelneleber:codex/fix-discord-voice-transcription
Open

fix(discord): keep live voice transcription bounded#42611
joelneleber wants to merge 1 commit into
NousResearch:mainfrom
joelneleber:codex/fix-discord-voice-transcription

Conversation

@joelneleber

@joelneleber joelneleber commented Jun 9, 2026

Copy link
Copy Markdown

Brain-Context: none present in Hermes checkout; read AGENTS.md.

Root cause summary

Discord live voice capture already cleared VoiceReceiver buffers after check_silence() emitted a completed utterance, so the receiver was not intentionally retranscribing the same in-memory audio forever. The expensive slowdown came from the runtime feeding every completed transcript snippet through adapter.handle_message(), which creates a full agent turn by default. Long meetings also had an inactivity timer risk: the Discord voice timeout was reset on join/playback, but not on inbound audio/transcript activity, so an active listening session could still age toward disconnect.

What changed

  • Kept completed transcript side-chat posting in deterministic gateway/runtime code by default.
  • Added HERMES_DISCORD_VOICE_TRANSCRIPT_AGENT_TURNS and discord.voice_transcript_agent_turns opt-in behavior for deployments that do want every transcript snippet to invoke the agent.
  • Refreshed the Discord voice inactivity timer on inbound audio/transcript activity so live sessions stay alive while the voice connection is active.
  • Added a max utterance duration in VoiceReceiver to bound per-speaker PCM buffers during long monologues.
  • Documented the new env/config flag in .env.example and cli-config.yaml.example.
  • Added focused tests for default no-agent transcript posting, opt-in agent dispatch, config bridging, and completed utterance buffer clearing.

Are we trimming/resetting audio after transcription, or retranscribing everything every time?

We are trimming/resetting. VoiceReceiver.check_silence() returns a completed (user_id, pcm_bytes) utterance and then clears that SSRC buffer plus its last-packet timestamp. The new regression test asserts the completed utterance is returned once and a second check_silence() does not re-emit the same audio. This PR also bounds long utterances so hot-path audio buffers do not grow for an entire meeting.

Tests run

  • scripts/run_tests.sh tests/gateway/test_voice_command.py tests/gateway/test_stt_config.py tests/gateway/test_discord_voice_mixer.py -- -q
  • /Users/j/.hermes/hermes-agent/venv/bin/python -m py_compile gateway/run.py gateway/config.py plugins/platforms/discord/adapter.py

@joelneleber

joelneleber commented Jun 9, 2026

Copy link
Copy Markdown
Author

Runtime note: this PR is now deployed on Joey's ssh-linux Hermes gateway via isolated worktree /home/j/.hermes/hermes-agent/.claude/worktrees/runtime-discord-voice-pr42611 at 5bb0eb6. Branch protection still prevents joelneleber from merging or enabling auto-merge; maintainer merge still needed upstream.

@liuhao1024

liuhao1024 commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

✅ Verified — Discord voice transcription bounded with opt-in agent dispatch

Reviewed the diff across all 7 files.

  • Buffer bounding: MAX_UTTERANCE_DURATION = 30.0 prevents unbounded audio buffer growth during long monologues. Utterances are flushed at 30s even without silence detection — correct for memory safety.
  • Timeout rate-limiting: _note_voice_activity() with VOICE_ACTIVITY_TIMEOUT_REFRESH = 30.0 prevents creating a new asyncio task on every audio packet. Timer only resets after 30s of monotonic clock drift — clean throttle.
  • Opt-in agent dispatch: Default behavior now posts transcripts to the side text channel without invoking the agent. This is a deliberate behavioral change (previously every snippet triggered an agent turn). Gated via HERMES_DISCORD_VOICE_TRANSCRIPT_AGENT_TURNS env var or voice_transcript_agent_turns config key.
  • Config bridge: gateway/config.py correctly bridges the new key from platform config to the merged extra dict.
  • Test coverage: 6+ tests covering default no-agent behavior, opt-in via env/config, duplicate suppression, and buffer clearing semantics.

The behavioral change from always-dispatch to opt-in is well-documented in .env.example and cli-config.yaml.example. No issues found.

@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery platform/discord Discord bot adapter tool/tts Text-to-speech and transcription P2 Medium — degraded but workaround exists labels Jun 9, 2026
@joelneleber joelneleber mentioned this pull request Jun 9, 2026
Open
teknium1
teknium1 reviewed Jul 14, 2026
View reviewed changes

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating the voice-buffer and per-transcript agent-turn costs. The current implementation still waits for silence before completing an utterance (plugins/platforms/discord/adapter.py:664) and sends completed transcripts through the normal agent pipeline (gateway/run.py:13070-13081), so the core issue remains worth salvaging.

Problems

  • .env.example adds HERMES_DISCORD_VOICE_TRANSCRIPT_AGENT_TURNS as a user-facing behavior setting. This conflicts with the repository policy in AGENTS.md:102-107: behavioral configuration belongs in config.yaml, not a new HERMES_*/.env setting.
  • The added clearing test exercises the existing silence path, not the new duration cap. It should prove a fresh active speaker is emitted at MAX_UTTERANCE_DURATION.
  • Current Discord YAML handling is plugin-owned by plugins/platforms/discord/adapter.py:_apply_yaml_config (:8229-8352); salvage should validate the config path there as well as gateway/config.py.

Suggested changes

  • Retain a config.yaml-only setting and remove the public environment-variable path.
  • Add the active-monologue duration-cap regression test and document the config setting in website/docs/user-guide/messaging/discord.md.

Automated hermes-sweeper review.

Comment thread .env.example
# should also become an agent turn.
# HERMES_DISCORD_VOICE_TRANSCRIPT_AGENT_TURNS=false

# =============================================================================

@teknium1 teknium1 Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes a new non-secret HERMES_* variable the user-facing behavior switch. Repository policy requires behavioral settings to be configured through config.yaml; please remove this .env mechanism and keep the supported configuration path YAML-only.

Comment thread plugins/platforms/discord/adapter.py

SILENCE_THRESHOLD = 1.5 # seconds of silence → end of utterance
MIN_SPEECH_DURATION = 0.5 # minimum seconds to process (skip noise)
MAX_UTTERANCE_DURATION = 30.0 # bound hot-path audio buffers during long monologues

@teknium1 teknium1 Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a regression test where last_packet_time is fresh but the PCM buffer reaches MAX_UTTERANCE_DURATION; the current added test only covers the pre-existing silence-triggered clearing path.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 14, 2026
Tranquil-Flow pushed a commit to Tranquil-Flow/hermes-agent that referenced this pull request Jul 15, 2026
Hermes Salvage
fix(discord): config.yaml-only STT duration cap, remove env var (Nous…
a67f41e 
…Research#42611)
@teknium1 teknium1 mentioned this pull request Jul 28, 2026
Merged
@alt-glitch alt-glitch mentioned this pull request Aug 4, 2026
Open
@leandrogg leandrogg mentioned this pull request Aug 4, 2026
Open
1 task
leandrogg added a commit to leandrogg/hermes-agent that referenced this pull request Aug 4, 2026
@leandrogg @claude
refactor(gateway): align transcript config key with NousResearch#42611 …
0bad9da 
…vocabulary

Rename discord.voice_transcripts_to_agent to voice_transcript_agent_turns
to match the key name used by the reviewed implementation in NousResearch#42611,
easing future reconciliation (default here stays true, preserving
existing behavior; NousResearch#42611 proposes flipping it). Also document the three
new keys in cli-config.yaml.example.

Co-Authored-By: Claude Fable 5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@teknium1 teknium1 teknium1 left review comments
+1 more reviewer
@HotGirlLo HotGirlLo HotGirlLo approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/discord Discord bot adapter sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state tool/tts Text-to-speech and transcription type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@joelneleber @liuhao1024 @teknium1 @HotGirlLo @alt-glitch