Skip to content

Fix/bettering the validation - #7

Merged
Jaiminp007 merged 2 commits into
devfrom
fix/bettering-the-validation
Apr 24, 2026
Merged

Jaiminp007 merged 2 commits into
devfrom
fix/bettering-the-validation

Conversation

@Jaiminp007

Copy link
Copy Markdown
Collaborator

This pull request introduces a robust validation and retry system for algorithm submissions, enforces stricter and more accurate validation rules, and improves the clarity and correctness of both the validation logic and the algorithm templates. The main focus is on ensuring that only valid algorithms are saved, providing actionable feedback for failed attempts, and maintaining version stability. It also updates the documentation to clearly communicate the validator's requirements and retry protocol.

Validation and Retry System Enhancements:

  • Added a RetryOrchestrator module that tracks per-session-per-algorithm validation attempts, enforces a maximum of 3 retries, and generates structured retry/exhaustion signals for the agent. This ensures failed attempts do not increment algorithm versions and provides targeted feedback for corrections.
  • Introduced a new algorithm.regenerating event in the event bus to communicate retry attempts and errors.

Validator Improvements:

  • Expanded the validator to include new error and warning codes for stricter enforcement, such as checks for state resets, lookahead bias, equity updates, and position sizing. Also added config cross-reference and smoke-test error codes. [1] [2]
  • Implemented Python-based AST and smoke-test diagnostic runners, integrating their results into the validation process for comprehensive static and behavioral checks.
  • Enhanced structure checks to support both legacy (on_tick) and broker-API (on_bar) entry methods, with improved error reporting for missing or malformed signatures. [1] [2] [3]

Algorithm Template Corrections:

  • Updated the RSI calculation in the template to correctly handle flat markets (returns 50 instead of 100 when both gains and losses are zero), and fixed the variance calculation to use sample variance (N-1 denominator) rather than population variance. [1] [2]

Documentation Updates:

  • Added comprehensive documentation outlining the validator's strict requirements, the mandatory retry protocol, common pitfalls, and best practices for both legacy and broker-API strategies. This serves as essential guidance for anyone submitting algorithms.

Other Improvements:

  • Removed outdated lookahead bias checks from the validator in favor of more robust AST-based detection.
  • Added utility imports and improved path handling for Python diagnostic scripts.

These changes collectively ensure a more reliable, user-friendly, and transparent algorithm submission process, reducing the risk of subtle bugs and improving the quality of strategies accepted by the system.

Copilot AI review requested due to automatic review settings April 24, 2026 04:50
@Jaiminp007
Jaiminp007 merged commit 4c42278 into dev Apr 24, 2026
1 check passed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR strengthens Finny’s algorithm submission pipeline by adding a structured validation retry mechanism, expanding validator checks via Python AST + behavioral smoke tests + config cross-referencing, and updating UI/docs to reflect the stricter save flow.

Changes:

  • Add RetryOrchestrator and integrate it into finny_algorithm_save, including a new algorithm.regenerating bus event and TUI suppression/banner behavior for transient retries.
  • Expand Validate to run Python diagnostics (AST analyzer + smoke test) and enforce config key usage, with updated tool/docs prompts describing new error codes and retry protocol.
  • Update algorithm templates/backtest metrics for RSI flat-market handling and sample-variance calculations; add mean-reversion USO example strategy files.

Reviewed changes

Copilot reviewed 16 out of 18 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
packages/opencode/strategies/mean_reversion_uso.py Adds a legacy (on_tick) mean-reversion USO example strategy.
packages/opencode/strategies/mean-reversion-uso/strategy.py Adds a broker-API (on_bar) mean-reversion USO example strategy with sizing.
packages/opencode/strategies/mean-reversion-uso/config.json Adds config for the new mean-reversion USO strategy example.
packages/opencode/src/tool/algorithm-validate.txt Updates validator tool documentation to describe strict AST/smoke/config checks and codes.
packages/opencode/src/tool/algorithm-save.txt Updates save tool documentation with strict validation + mandatory retry protocol.
packages/opencode/src/tool/algorithm-save.ts Implements retry-aware validation flow and publishes regenerating events.
packages/opencode/src/cli/cmd/tui/routes/session/index.tsx Adds “validation failed — regenerating” banner and hides transient tool parts.
packages/opencode/src/cli/cmd/tui/component/prompt/index.tsx Prevents session replay from overwriting the user’s active model selection.
packages/opencode/src/backtest/runner.ts Switches return variance to sample variance (N-1).
packages/opencode/src/algorithm/validate.ts Adds config cross-reference, Python AST + smoke-test runners, and new diagnostic code unions.
packages/opencode/src/algorithm/templates.ts Fixes template RSI flat-market behavior and sample variance.
packages/opencode/src/algorithm/smoke_test.py Adds behavioral smoke test runner and invariant checks.
packages/opencode/src/algorithm/retry-orchestrator.ts Adds centralized per-session/per-algo retry attempt tracking and messaging.
packages/opencode/src/algorithm/index.ts Adds algorithm.regenerating bus event definition.
packages/opencode/src/algorithm/ast_analyzer.py Adds AST-based static analysis checks (lookahead, same-bar bias, sizing, etc.).
packages/opencode/src/algorithm/pycache/smoke_test.cpython-313.pyc (Accidental) compiled Python cache artifact added to repo.
packages/opencode/src/agent/prompt/finny-build.txt Updates agent prompt with strict validator rules and mandatory retry protocol.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +166 to +223
/**
* Spawn a Python script, feed `code` to stdin, and parse its JSON stdout as diagnostics.
* Returns [] on any failure — the Python script is advisory; upstream syntax/structure
* checks are authoritative for blocking save.
*/
async function runPythonDiagnostic(
scriptPath: string,
code: string,
timeoutMs: number,
extraArgs: string[] = [],
): Promise<Diagnostic[]> {
try {
const proc = Process.spawn(
["python3", scriptPath, ...extraArgs],
{ stdin: "pipe", stdout: "pipe", stderr: "pipe", timeout: timeoutMs },
)
proc.stdin!.write(code)
proc.stdin!.end()

const timeout = new Promise<number>((resolve) => setTimeout(() => {
try { proc.kill("SIGKILL") } catch {}
resolve(124)
}, timeoutMs))

const [exitCode, stdout] = await Promise.all([
Promise.race([proc.exited, timeout]),
new Promise<string>((resolve) => {
const chunks: Buffer[] = []
proc.stdout!.on("data", (c: Buffer) => chunks.push(c))
proc.stdout!.on("end", () => resolve(Buffer.concat(chunks).toString()))
}),
])

if (exitCode !== 0) return []
const raw = stdout.trim()
if (!raw) return []
const parsed = JSON.parse(raw)
if (!Array.isArray(parsed)) return []

const diags: Diagnostic[] = []
for (const d of parsed) {
if (typeof d !== "object" || d === null) continue
const code = String(d.code ?? "")
const severity = d.severity === "error" ? "error" : "warning"
if (severity === "error" && !KNOWN_ERROR_CODES.has(code as ErrorCode)) continue
if (severity === "warning" && !KNOWN_WARNING_CODES.has(code as WarningCode)) continue
diags.push({
code: code as ErrorCode | WarningCode,
severity,
message: String(d.message ?? ""),
line: typeof d.line === "number" ? d.line : undefined,
fix: typeof d.fix === "string" ? d.fix : undefined,
})
}
return diags
} catch {
return []
}

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

runPythonDiagnostic currently returns [] on any failure (spawn error, timeout, non-zero exit, parse error). That means the AST analyzer / smoke test can silently be skipped and invalid algorithms may pass validation even though the docs describe these checks as strict. Consider surfacing a blocking diagnostic (or at least a visible warning) when these scripts can’t be executed so validation can’t be bypassed by an environment issue.

Copilot uses AI. Check for mistakes.
Comment on lines +185 to +197
const timeout = new Promise<number>((resolve) => setTimeout(() => {
try { proc.kill("SIGKILL") } catch {}
resolve(124)
}, timeoutMs))

const [exitCode, stdout] = await Promise.all([
Promise.race([proc.exited, timeout]),
new Promise<string>((resolve) => {
const chunks: Buffer[] = []
proc.stdout!.on("data", (c: Buffer) => chunks.push(c))
proc.stdout!.on("end", () => resolve(Buffer.concat(chunks).toString()))
}),
])

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The manual timeout setTimeout(...proc.kill...) is never cleared when the process exits normally. That leaves dangling timers and can attempt to SIGKILL an already-finished process later. Capture the timer handle and clearTimeout it after proc.exited resolves (or rely on Process.spawn’s built-in timeout).

Copilot uses AI. Check for mistakes.
Comment on lines +689 to +700
def _looks_like_sizing_division(assign):
"""True if RHS is a division whose denominator name hints at stop/risk distance."""
if not isinstance(assign, ast.Assign):
return False
value = assign.value
if not (isinstance(value, ast.BinOp) and isinstance(value.op, (ast.Div, ast.FloorDiv))):
return False
denom_names = _collect_names_from_expr(value.right) | {
a.attr for a in ast.walk(value.right)
if isinstance(a, ast.Attribute) and isinstance(a.value, ast.Name) and a.value.id == "self"
}
return any(any(h in n.lower() for h in SIZING_DENOM_HINTS) for n in denom_names)

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_looks_like_sizing_division only matches assignments whose RHS is a bare risk / stop_distance BinOp. Common whole-share patterns like qty = int(risk_amount / stop_distance) or qty = math.floor(risk_amount / stop_distance) won’t be detected, which can bypass POSITION_SIZE_UNCAPPED (and also affects FRACTIONAL_SHARES_EQUITY). Consider unwrapping simple call wrappers (int(...), math.floor(...), etc.) before checking for the division so the validator still enforces the cap even when quantities are floored.

Copilot uses AI. Check for mistakes.
Comment on lines +1 to +28
"""
Mean-reversion strategy for USO using Bollinger Bands with RSI filter.

Buys when price touches lower Bollinger Band and RSI confirms oversold.
Sells when price reaches upper band or RSI confirms overbought.
Includes volatility filter and proper position sizing based on stop loss.
"""

from collections import deque


class Strategy:
def __init__(self):
"""Initialize the mean-reversion strategy."""
# Bollinger Bands parameters
self.bollinger_period = 20
self.bollinger_std = 2.0

# RSI parameters
self.rsi_period = 14
self.rsi_oversold = 30
self.rsi_overbought = 70

# Risk management parameters
self.volatility_threshold = 0.02
self.max_risk_per_trade_percent = 1.0
self.stop_loss_pct = 0.02
self.take_profit_pct = 0.04

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The module docstring claims “proper position sizing based on stop loss”, but the implementation only uses self.position as a 0/1 flag and never uses max_risk_per_trade_percent (or equity) to size trades. Either implement sizing consistent with the description or update the description/unused config fields so this example doesn’t mislead users.

Copilot uses AI. Check for mistakes.
# Volatility calculation (returns)
self.returns = deque(maxlen=20)

# Trade tracking - use -1 to indicate no position (avoid [0,100] range)

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: the comment says “use -1 to indicate no position”, but self.position is initialized to 0 and treated as a 0/1 flag throughout. Update the comment or the sentinel so it matches actual behavior.

Suggested change
# Trade tracking - use -1 to indicate no position (avoid [0,100] range)
# Trade tracking - position uses 0/1 (0 = no position, 1 = in position);
# price levels use -1.0 to indicate an unset value.

Copilot uses AI. Check for mistakes.
Comment on lines +53 to +55
price = bar["open"]
self.prices.append(price)

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on_bar appends the current bar’s open to self.prices and then computes Bollinger Bands / triggers buy/sell off that updated series. With the new validator this is same-bar lookahead (SAME_BAR_EXECUTION_BIAS) and will be rejected. Compute signals from the prior window first, then append/update state after the decision (or execute the signal on the next bar).

Copilot uses AI. Check for mistakes.
Comment on lines 101 to 109
price = bar["open"]
self.prices.append(price)

if len(self.prices) < self.period:
return "HOLD"

mean = sum(self.prices) / self.period
variance = sum((p - mean) ** 2 for p in self.prices) / self.period
variance = sum((p - mean) ** 2 for p in self.prices) / (self.period - 1)
std = math.sqrt(variance)

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mean-reversion template still updates self.prices with the current bar’s price and then computes bands / issues BUY/SELL based on that updated series. With the new AST rule (SAME_BAR_EXECUTION_BIAS), this is same-bar lookahead and will be rejected by the validator. Compute the signal from prior-bar state first (using the existing deque contents), then append the current price (or delay execution to the next bar).

Copilot uses AI. Check for mistakes.
Comment on lines +133 to +137
current_price = bar["open"] # Use open for decisions to avoid lookahead bias

# Update price history
self.prices.append(current_price)

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on_tick appends the current bar’s price to self.prices before computing Bollinger Bands and potentially returning BUY/SELL. This makes the indicator include the same tick you’re trading on, which is exactly the SAME_BAR_EXECUTION_BIAS the new validator blocks (and is a subtle lookahead in backtests). Make the trade decision using prior-bar state first, then append/update state after the decision (or execute the signal on the next tick).

Copilot uses AI. Check for mistakes.
Jaiminp007 pushed a commit that referenced this pull request May 2, 2026
Fixes all 17 Copilot comments:

Telemetry opt-out (#1, #2, #3, #9, #16):
- tracker.ts now reads FINNY_TELEMETRY=0 / OPENCODE_TELEMETRY=0 at module
  load, so the opt-out applies in every entrypoint (TUI worker, server,
  CLI), not just the bootstrap path.
- Email dialog auto-prompt and /subscribe slash command also gated on
  Analytics.isEnabled() — email POST goes to the same Convex deployment.

Tracker correctness (#5, #14, #17):
- drain() now also waits for the DeviceProfile.userId() resolution promise,
  not just the post-send promise.
- worker rpc.shutdown() and main-thread thread.ts both call
  Analytics.drain() before tearing down — neither beforeExit nor SIGTERM
  fire on worker.terminate() / process.exit(0).
- beforeExit handler runs at most once per process, preventing an infinite
  drain loop when a hung telemetry request later resolves and re-empties
  the loop.

Privacy (#10, #11, #12):
- Drop process.argv from tui.worker.booted (could include --prompt text
  and project paths).
- error.surfaced now sends only error.name; error messages frequently
  embed user prompt content and local file paths.
- Don't persist the submitted email address to local KV (no reader, just
  unnecessary PII retention).

Email dialog state machine (#4, #6, #8):
- Wait for kv.ready before evaluating the prompt gate so we don't re-show
  the dialog to users who already submitted/skipped.
- Conditional dismiss handler — only marks "skipped" when no terminal
  status has been set, so a successful submit isn't overwritten by the
  dismiss callback that fires on dialog.clear() and isn't suppressed by
  another dialog replacing this one.

Server-side validation (#15):
- subscribe mutation now rejects empty / oversized / regex-failing emails
  with a ConvexError, so the public Convex client can't bypass the TUI
  regex.

Command coverage (#13):
- Hook Analytics.track() in DialogCommand.trigger() instead of the bus
  event handler. trigger() is the central activation point for palette,
  slash, keybind, plugin, and bus paths — single hook, full coverage,
  no double-counting.

Comment wording (#7):
- bootstrap.ts: "opt-in by default" → "enabled by default" (correct
  description of opt-out semantics).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants