Skip to content

Heartbeat does not reload HEARTBEAT.md after gateway restart #51542

Description

@Suidge

Labels: bug, heartbeat, gateway


Background

We initially configured HEARTBEAT.md to require HEARTBEAT_OK at the beginning of the heartbeat message. However, this caused persistent errors where the heartbeat system failed to recognize the acknowledgment properly.

Original HEARTBEAT.md requirement (causing errors):

**[Final Output Format Requirements]**
If no urgent matters are found, reply exactly: `HEARTBEAT_OK` at the beginning of your message.

❌ **Incorrect Example**:

📧 No new emails
🔍 No failed commands
HEARTBEAT_OK


✅ **Correct Example**:

HEARTBEAT_OK

📧 No new emails
🔍 No failed commands

Observed Errors (with HEARTBEAT_OK at beginning):

Actual heartbeat message received (2026-03-21 13:38, before HEARTBEAT.md update):

Another heartbeat check at 1:38 PM, 30 minutes after the last one. Everything remains unchanged - no new emails, no failures.
HEARTBEAT_OK

**巡检时间**: 2026-03-21 13:38 (Asia/Shanghai)
**📧 邮件检查**
- 无新邮件
**🔍 对话复盘**
- 过去 1 小时无失败命令
- 无需要记录的纠正或改进
**💬 互动间隔**
- 约 30 分钟前有互动
- 无需问候
**状态**: ✅ 一切正常

Problem: HEARTBEAT_OK appears in the middle of the message (after the intro sentence, before the inspection details), not at the beginning as required. This caused the heartbeat acknowledgment to not be properly recognized by the system.

To resolve this, we modified HEARTBEAT.md to require HEARTBEAT_OK at the end (last line) of the message. However, after making this change and performing a complete gateway restart, we discovered that the heartbeat content did not change at all — it continued using the old prompt format.


Problem Description

After modifying HEARTBEAT.md, the heartbeat session continues to use the old prompt even after a complete gateway restart. The new HEARTBEAT.md content is not dynamically loaded.

Specific Symptom:

  • Modified HEARTBEAT.md to require HEARTBEAT_OK on the last line of the message
  • After complete gateway restart, heartbeat still sends HEARTBEAT_OK in the middle of the message, not on the last line
  • Heartbeat message format remained completely unchanged despite the config modification

Environment

  • OpenClaw Version: v2026.3.13+ (to be confirmed)
  • OS: macOS 15.3 (Darwin 25.3.0, arm64)
  • Node Version: v25.8.1
  • Gateway Mode: LaunchAgent (managed by launchctl)
  • Heartbeat Config: 30-minute interval, Feishu DM target

Reproduction Steps

1. Modify HEARTBEAT.md

# HEARTBEAT.md — Monitoring Task List

> ⚠️ **System-level Interception Rule**:
> If there are no urgent matters requiring immediate user attention, the system must silently intercept the reply to avoid disturbing the user.
> **The absolute technical condition for interception to take effect: `HEARTBEAT_OK` must appear on the [last line] of your entire reply content, and there must be absolutely no characters, punctuation, newlines, or status summaries after it.**

## 📋 Required Tasks for Each Heartbeat

1. **Email Check** — Run `himalaya envelope list`, only report if there are [family] or [work-urgent] related new emails
2. **Conversation Review** — Quickly review the past 1 hour of conversation: any failed commands or methods that need improvement? If yes, record to daily memory
3. **8-Hour No Interaction** — Carefully check context. Only send a simple greeting if it has been more than 8 hours since the last message from [human user]

---

**[Final Output Format Requirements]**
If no urgent matters are found, you can output your Chinese/English inspection briefing first, but **the last line of the entire reply must end precisely with `HEARTBEAT_OK`**.

❌ **Incorrect Example** (never put the keyword in the middle):

Another heartbeat check...
HEARTBEAT_OK
Inspection time: 13:38


✅ **Correct Example** (briefing first, last line收尾):

Inspection time: 13:38
📧 No new emails
🔍 No failed commands in the past 1 hour
💬 No greeting needed
Status: All normal
HEARTBEAT_OK

Modification Time: 2026-03-21 13:50:58 (Asia/Shanghai)

2. Restart Gateway

# 1. Unload LaunchAgent
launchctl bootout gui/$UID/ai.openclaw.gateway

# 2. Remove plist
rm ~/Library/LaunchAgents/ai.openclaw.gateway.plist

# 3. Kill process (if still running)
kill -9 <PID>

# 4. Start again
openclaw gateway start

3. Observe Heartbeat Output

Wait for the next heartbeat trigger (30-minute interval) and observe the output format.

Expected Result:

Inspection time: 17:41
📧 No new emails
🔍 No failed commands in the past 1 hour
💬 No greeting needed
Status: All normal
HEARTBEAT_OK

Actual Result:

🌙 [Title]

**Summary**:
- Key point 1
- Key point 2

**Status**: ✅ Complete / ⏳ In Progress
📄 Full report: `file path.md`
HEARTBEAT_OK

---

**Inspection Time**: 2026-03-21 17:41 (Asia/Shanghai)
📧 Email Check
- No new emails
🔍 Conversation Review
- No failed commands in the past 1 hour
- Gateway restart message sent
💬 Interaction Interval
- Interaction a few minutes ago
- No greeting needed
**Status**: ✅ All normal

Problem: HEARTBEAT_OK appears in the middle of the message, not on the last line.


Timeline Investigation

Time Event Description
2026-03-17 19:22 Gateway started PID 78440
2026-03-21 13:50 HEARTBEAT.md modified 725 bytes, added last-line requirement
2026-03-21 13:53 Config hot reload ackMaxChars=2000
2026-03-21 14:13 Heartbeat started gateway.log entry
2026-03-21 16:16 Heartbeat started gateway.log entry
2026-03-21 17:01 Discovered gateway not restarted PID 78440 running since Mar 17
2026-03-21 17:11 Complete gateway restart launchctl bootout + kill -9
2026-03-21 17:13 Gateway new process PID 84268
2026-03-21 17:41 Heartbeat triggered Still using old prompt

Root Cause Analysis

Hypothesis 1: Gateway caches HEARTBEAT.md at startup

Speculated Mechanism:

Gateway starts
    ↓
Reads HEARTBEAT.md content
    ↓
Caches to heartbeat session prompt
    ↓
Heartbeat session reuses cached prompt

Evidence:

  • Gateway ran from Mar 17 to Mar 21 (4 days), during which HEARTBEAT.md was modified
  • Config hot reload only updates partial config (e.g., ackMaxChars), does not re-read HEARTBEAT.md
  • Problem persists after complete gateway restart

Contradiction:

  • If gateway reads HEARTBEAT.md at startup, it should read new content after restart
  • But problem persists after restart (new PID 84268)

Hypothesis 2: Heartbeat Session has independent cache

Speculated Mechanism:

Heartbeat is a long-running cron session
    ↓
Reads HEARTBEAT.md on first start
    ↓
Session state persists (not cleared on gateway restart)
    ↓
Subsequent heartbeats reuse the same session state

Evidence:

  • gateway.log shows heartbeat sessions have independent session IDs
  • Heartbeat sessions might be "persistent", not recreated on each trigger

To Confirm:

  • Are heartbeat sessions really persistent?
  • Where is session state stored?

Hypothesis 3: HEARTBEAT.md read path issue

To Confirm:

  • Which HEARTBEAT.md does heartbeat actually read?
  • Is it workspace/HEARTBEAT.md or another path?
  • Are there multiple HEARTBEAT.md files?

Confirmed:

  • ~/.openclaw/workspace/HEARTBEAT.md exists (1593 bytes, modified 2026-03-21 13:50)
  • ~/.openclaw/HEARTBEAT.md does not exist
  • OpenClaw docs state: HEARTBEAT.md defaults to agent workspace

Attempted Solutions

Solution Operation Result
Config hot reload openclaw config set agents.defaults.heartbeat.ackMaxChars 2000 ❌ Invalid (only updates config, does not re-read HEARTBEAT.md)
Complete gateway restart launchctl bootout + kill -9 + openclaw gateway start ❌ Invalid (new PID 84268, problem persists)
Confirm file path Check ~/.openclaw/workspace/HEARTBEAT.md exists and content is correct ✅ File is correct, but not being read

Open Questions

  1. Heartbeat prompt loading timing:

    • Is it loaded once at gateway startup?
    • Or dynamically read on each heartbeat trigger?
    • Or is there another caching mechanism?
  2. Heartbeat session lifecycle:

    • Is heartbeat session created new on each trigger, or is it a long-running session?
    • Is session state persisted? Where is it stored?
  3. HEARTBEAT.md read path:

    • Is there a config file that specifies the HEARTBEAT.md path?
    • Is hot reload supported?

Expected Behavior

  1. Dynamic Read: Heartbeat should read the latest HEARTBEAT.md content on each trigger
  2. Or Support Reload: At minimum, should re-read HEARTBEAT.md after gateway restart
  3. Or Document: If this is by design, clearly document how to refresh heartbeat prompt

Workaround

No reliable workaround currently exists.

Possible workarounds (unverified):

  1. Clear all session state (if there is a cache directory)
  2. Modify openclaw.json to force re-initialization
  3. Wait for heartbeat session to naturally expire (if there is a time limit)

Impact

  • Severity: Medium (affects heartbeat behavior customization, but does not affect core functionality)
  • Affected Users: All users who customize HEARTBEAT.md
  • Trigger Condition: After modifying HEARTBEAT.md, expecting changes to take effect

Suggested Fix Directions

  1. Dynamic Read: Read HEARTBEAT.md on each heartbeat trigger (recommended)
  2. Session Reset: Force reset all heartbeat sessions on gateway restart
  3. Cache Invalidation: Detect HEARTBEAT.md file changes, automatically invalidate cache
  4. Documentation: If this is by design, clearly document how to refresh heartbeat prompt

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions