Repository navigation
Fix/bettering the validation - #7
Conversation
There was a problem hiding this comment.
Pull request overview
This PR strengthens Finny’s algorithm submission pipeline by adding a structured validation retry mechanism, expanding validator checks via Python AST + behavioral smoke tests + config cross-referencing, and updating UI/docs to reflect the stricter save flow.
Changes:
- Add
RetryOrchestratorand integrate it intofinny_algorithm_save, including a newalgorithm.regeneratingbus event and TUI suppression/banner behavior for transient retries. - Expand
Validateto run Python diagnostics (AST analyzer + smoke test) and enforce config key usage, with updated tool/docs prompts describing new error codes and retry protocol. - Update algorithm templates/backtest metrics for RSI flat-market handling and sample-variance calculations; add mean-reversion USO example strategy files.
Reviewed changes
Copilot reviewed 16 out of 18 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/opencode/strategies/mean_reversion_uso.py | Adds a legacy (on_tick) mean-reversion USO example strategy. |
| packages/opencode/strategies/mean-reversion-uso/strategy.py | Adds a broker-API (on_bar) mean-reversion USO example strategy with sizing. |
| packages/opencode/strategies/mean-reversion-uso/config.json | Adds config for the new mean-reversion USO strategy example. |
| packages/opencode/src/tool/algorithm-validate.txt | Updates validator tool documentation to describe strict AST/smoke/config checks and codes. |
| packages/opencode/src/tool/algorithm-save.txt | Updates save tool documentation with strict validation + mandatory retry protocol. |
| packages/opencode/src/tool/algorithm-save.ts | Implements retry-aware validation flow and publishes regenerating events. |
| packages/opencode/src/cli/cmd/tui/routes/session/index.tsx | Adds “validation failed — regenerating” banner and hides transient tool parts. |
| packages/opencode/src/cli/cmd/tui/component/prompt/index.tsx | Prevents session replay from overwriting the user’s active model selection. |
| packages/opencode/src/backtest/runner.ts | Switches return variance to sample variance (N-1). |
| packages/opencode/src/algorithm/validate.ts | Adds config cross-reference, Python AST + smoke-test runners, and new diagnostic code unions. |
| packages/opencode/src/algorithm/templates.ts | Fixes template RSI flat-market behavior and sample variance. |
| packages/opencode/src/algorithm/smoke_test.py | Adds behavioral smoke test runner and invariant checks. |
| packages/opencode/src/algorithm/retry-orchestrator.ts | Adds centralized per-session/per-algo retry attempt tracking and messaging. |
| packages/opencode/src/algorithm/index.ts | Adds algorithm.regenerating bus event definition. |
| packages/opencode/src/algorithm/ast_analyzer.py | Adds AST-based static analysis checks (lookahead, same-bar bias, sizing, etc.). |
| packages/opencode/src/algorithm/pycache/smoke_test.cpython-313.pyc | (Accidental) compiled Python cache artifact added to repo. |
| packages/opencode/src/agent/prompt/finny-build.txt | Updates agent prompt with strict validator rules and mandatory retry protocol. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| /** | ||
| * Spawn a Python script, feed `code` to stdin, and parse its JSON stdout as diagnostics. | ||
| * Returns [] on any failure — the Python script is advisory; upstream syntax/structure | ||
| * checks are authoritative for blocking save. | ||
| */ | ||
| async function runPythonDiagnostic( | ||
| scriptPath: string, | ||
| code: string, | ||
| timeoutMs: number, | ||
| extraArgs: string[] = [], | ||
| ): Promise<Diagnostic[]> { | ||
| try { | ||
| const proc = Process.spawn( | ||
| ["python3", scriptPath, ...extraArgs], | ||
| { stdin: "pipe", stdout: "pipe", stderr: "pipe", timeout: timeoutMs }, | ||
| ) | ||
| proc.stdin!.write(code) | ||
| proc.stdin!.end() | ||
|
|
||
| const timeout = new Promise<number>((resolve) => setTimeout(() => { | ||
| try { proc.kill("SIGKILL") } catch {} | ||
| resolve(124) | ||
| }, timeoutMs)) | ||
|
|
||
| const [exitCode, stdout] = await Promise.all([ | ||
| Promise.race([proc.exited, timeout]), | ||
| new Promise<string>((resolve) => { | ||
| const chunks: Buffer[] = [] | ||
| proc.stdout!.on("data", (c: Buffer) => chunks.push(c)) | ||
| proc.stdout!.on("end", () => resolve(Buffer.concat(chunks).toString())) | ||
| }), | ||
| ]) | ||
|
|
||
| if (exitCode !== 0) return [] | ||
| const raw = stdout.trim() | ||
| if (!raw) return [] | ||
| const parsed = JSON.parse(raw) | ||
| if (!Array.isArray(parsed)) return [] | ||
|
|
||
| const diags: Diagnostic[] = [] | ||
| for (const d of parsed) { | ||
| if (typeof d !== "object" || d === null) continue | ||
| const code = String(d.code ?? "") | ||
| const severity = d.severity === "error" ? "error" : "warning" | ||
| if (severity === "error" && !KNOWN_ERROR_CODES.has(code as ErrorCode)) continue | ||
| if (severity === "warning" && !KNOWN_WARNING_CODES.has(code as WarningCode)) continue | ||
| diags.push({ | ||
| code: code as ErrorCode | WarningCode, | ||
| severity, | ||
| message: String(d.message ?? ""), | ||
| line: typeof d.line === "number" ? d.line : undefined, | ||
| fix: typeof d.fix === "string" ? d.fix : undefined, | ||
| }) | ||
| } | ||
| return diags | ||
| } catch { | ||
| return [] | ||
| } |
There was a problem hiding this comment.
runPythonDiagnostic currently returns [] on any failure (spawn error, timeout, non-zero exit, parse error). That means the AST analyzer / smoke test can silently be skipped and invalid algorithms may pass validation even though the docs describe these checks as strict. Consider surfacing a blocking diagnostic (or at least a visible warning) when these scripts can’t be executed so validation can’t be bypassed by an environment issue.
| const timeout = new Promise<number>((resolve) => setTimeout(() => { | ||
| try { proc.kill("SIGKILL") } catch {} | ||
| resolve(124) | ||
| }, timeoutMs)) | ||
|
|
||
| const [exitCode, stdout] = await Promise.all([ | ||
| Promise.race([proc.exited, timeout]), | ||
| new Promise<string>((resolve) => { | ||
| const chunks: Buffer[] = [] | ||
| proc.stdout!.on("data", (c: Buffer) => chunks.push(c)) | ||
| proc.stdout!.on("end", () => resolve(Buffer.concat(chunks).toString())) | ||
| }), | ||
| ]) |
There was a problem hiding this comment.
The manual timeout setTimeout(...proc.kill...) is never cleared when the process exits normally. That leaves dangling timers and can attempt to SIGKILL an already-finished process later. Capture the timer handle and clearTimeout it after proc.exited resolves (or rely on Process.spawn’s built-in timeout).
| def _looks_like_sizing_division(assign): | ||
| """True if RHS is a division whose denominator name hints at stop/risk distance.""" | ||
| if not isinstance(assign, ast.Assign): | ||
| return False | ||
| value = assign.value | ||
| if not (isinstance(value, ast.BinOp) and isinstance(value.op, (ast.Div, ast.FloorDiv))): | ||
| return False | ||
| denom_names = _collect_names_from_expr(value.right) | { | ||
| a.attr for a in ast.walk(value.right) | ||
| if isinstance(a, ast.Attribute) and isinstance(a.value, ast.Name) and a.value.id == "self" | ||
| } | ||
| return any(any(h in n.lower() for h in SIZING_DENOM_HINTS) for n in denom_names) |
There was a problem hiding this comment.
_looks_like_sizing_division only matches assignments whose RHS is a bare risk / stop_distance BinOp. Common whole-share patterns like qty = int(risk_amount / stop_distance) or qty = math.floor(risk_amount / stop_distance) won’t be detected, which can bypass POSITION_SIZE_UNCAPPED (and also affects FRACTIONAL_SHARES_EQUITY). Consider unwrapping simple call wrappers (int(...), math.floor(...), etc.) before checking for the division so the validator still enforces the cap even when quantities are floored.
| """ | ||
| Mean-reversion strategy for USO using Bollinger Bands with RSI filter. | ||
|
|
||
| Buys when price touches lower Bollinger Band and RSI confirms oversold. | ||
| Sells when price reaches upper band or RSI confirms overbought. | ||
| Includes volatility filter and proper position sizing based on stop loss. | ||
| """ | ||
|
|
||
| from collections import deque | ||
|
|
||
|
|
||
| class Strategy: | ||
| def __init__(self): | ||
| """Initialize the mean-reversion strategy.""" | ||
| # Bollinger Bands parameters | ||
| self.bollinger_period = 20 | ||
| self.bollinger_std = 2.0 | ||
|
|
||
| # RSI parameters | ||
| self.rsi_period = 14 | ||
| self.rsi_oversold = 30 | ||
| self.rsi_overbought = 70 | ||
|
|
||
| # Risk management parameters | ||
| self.volatility_threshold = 0.02 | ||
| self.max_risk_per_trade_percent = 1.0 | ||
| self.stop_loss_pct = 0.02 | ||
| self.take_profit_pct = 0.04 |
There was a problem hiding this comment.
The module docstring claims “proper position sizing based on stop loss”, but the implementation only uses self.position as a 0/1 flag and never uses max_risk_per_trade_percent (or equity) to size trades. Either implement sizing consistent with the description or update the description/unused config fields so this example doesn’t mislead users.
| # Volatility calculation (returns) | ||
| self.returns = deque(maxlen=20) | ||
|
|
||
| # Trade tracking - use -1 to indicate no position (avoid [0,100] range) |
There was a problem hiding this comment.
Minor: the comment says “use -1 to indicate no position”, but self.position is initialized to 0 and treated as a 0/1 flag throughout. Update the comment or the sentinel so it matches actual behavior.
| # Trade tracking - use -1 to indicate no position (avoid [0,100] range) | |
| # Trade tracking - position uses 0/1 (0 = no position, 1 = in position); | |
| # price levels use -1.0 to indicate an unset value. |
| price = bar["open"] | ||
| self.prices.append(price) | ||
|
|
There was a problem hiding this comment.
on_bar appends the current bar’s open to self.prices and then computes Bollinger Bands / triggers buy/sell off that updated series. With the new validator this is same-bar lookahead (SAME_BAR_EXECUTION_BIAS) and will be rejected. Compute signals from the prior window first, then append/update state after the decision (or execute the signal on the next bar).
| price = bar["open"] | ||
| self.prices.append(price) | ||
|
|
||
| if len(self.prices) < self.period: | ||
| return "HOLD" | ||
|
|
||
| mean = sum(self.prices) / self.period | ||
| variance = sum((p - mean) ** 2 for p in self.prices) / self.period | ||
| variance = sum((p - mean) ** 2 for p in self.prices) / (self.period - 1) | ||
| std = math.sqrt(variance) |
There was a problem hiding this comment.
This mean-reversion template still updates self.prices with the current bar’s price and then computes bands / issues BUY/SELL based on that updated series. With the new AST rule (SAME_BAR_EXECUTION_BIAS), this is same-bar lookahead and will be rejected by the validator. Compute the signal from prior-bar state first (using the existing deque contents), then append the current price (or delay execution to the next bar).
| current_price = bar["open"] # Use open for decisions to avoid lookahead bias | ||
|
|
||
| # Update price history | ||
| self.prices.append(current_price) | ||
|
|
There was a problem hiding this comment.
on_tick appends the current bar’s price to self.prices before computing Bollinger Bands and potentially returning BUY/SELL. This makes the indicator include the same tick you’re trading on, which is exactly the SAME_BAR_EXECUTION_BIAS the new validator blocks (and is a subtle lookahead in backtests). Make the trade decision using prior-bar state first, then append/update state after the decision (or execute the signal on the next tick).
Fixes all 17 Copilot comments: Telemetry opt-out (#1, #2, #3, #9, #16): - tracker.ts now reads FINNY_TELEMETRY=0 / OPENCODE_TELEMETRY=0 at module load, so the opt-out applies in every entrypoint (TUI worker, server, CLI), not just the bootstrap path. - Email dialog auto-prompt and /subscribe slash command also gated on Analytics.isEnabled() — email POST goes to the same Convex deployment. Tracker correctness (#5, #14, #17): - drain() now also waits for the DeviceProfile.userId() resolution promise, not just the post-send promise. - worker rpc.shutdown() and main-thread thread.ts both call Analytics.drain() before tearing down — neither beforeExit nor SIGTERM fire on worker.terminate() / process.exit(0). - beforeExit handler runs at most once per process, preventing an infinite drain loop when a hung telemetry request later resolves and re-empties the loop. Privacy (#10, #11, #12): - Drop process.argv from tui.worker.booted (could include --prompt text and project paths). - error.surfaced now sends only error.name; error messages frequently embed user prompt content and local file paths. - Don't persist the submitted email address to local KV (no reader, just unnecessary PII retention). Email dialog state machine (#4, #6, #8): - Wait for kv.ready before evaluating the prompt gate so we don't re-show the dialog to users who already submitted/skipped. - Conditional dismiss handler — only marks "skipped" when no terminal status has been set, so a successful submit isn't overwritten by the dismiss callback that fires on dialog.clear() and isn't suppressed by another dialog replacing this one. Server-side validation (#15): - subscribe mutation now rejects empty / oversized / regex-failing emails with a ConvexError, so the public Convex client can't bypass the TUI regex. Command coverage (#13): - Hook Analytics.track() in DialogCommand.trigger() instead of the bus event handler. trigger() is the central activation point for palette, slash, keybind, plugin, and bus paths — single hook, full coverage, no double-counting. Comment wording (#7): - bootstrap.ts: "opt-in by default" → "enabled by default" (correct description of opt-out semantics). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This pull request introduces a robust validation and retry system for algorithm submissions, enforces stricter and more accurate validation rules, and improves the clarity and correctness of both the validation logic and the algorithm templates. The main focus is on ensuring that only valid algorithms are saved, providing actionable feedback for failed attempts, and maintaining version stability. It also updates the documentation to clearly communicate the validator's requirements and retry protocol.
Validation and Retry System Enhancements:
RetryOrchestratormodule that tracks per-session-per-algorithm validation attempts, enforces a maximum of 3 retries, and generates structured retry/exhaustion signals for the agent. This ensures failed attempts do not increment algorithm versions and provides targeted feedback for corrections.algorithm.regeneratingevent in the event bus to communicate retry attempts and errors.Validator Improvements:
on_tick) and broker-API (on_bar) entry methods, with improved error reporting for missing or malformed signatures. [1] [2] [3]Algorithm Template Corrections:
N-1denominator) rather than population variance. [1] [2]Documentation Updates:
Other Improvements:
These changes collectively ensure a more reliable, user-friendly, and transparent algorithm submission process, reducing the risk of subtle bugs and improving the quality of strategies accepted by the system.