Skip to content

Commit 8c4d1d9

Browse files
Jaiminp007Jaimin Patel
andauthored
Stop Finny workflow thrash on research-only and open-candle data (anomalyco#186)
* Stop Finny workflow thrash on research-only and open-candle data Prevent endless save/backtest loops once a presentable research candidate exists or the agent yields control to the user. Allow research_only verified datasets to stage and run as non-promotable research (auto-repair isolated outliers), lock confirmed identity against silent window/interval thrash, and clamp incomplete "today" end dates to the last completed day so data_extractor does not loop on open candles. * Refactor identity lock and force research-only verdicts Split workspacePrepareConfirmedIdentityLock into small conflict helpers to clear CodeScene critical code health rules, and ensure verified research_only artifacts cannot be recorded as recommended_for_paper. * Keep backtest research-only verdict change without full-file reformat Restore focused backtest.ts edits so research_only verified artifacts force a research_only verdict without accidental whole-file prettier churn. --------- Co-authored-by: Jaimin Patel <[email protected]>
1 parent 396c0d7 commit 8c4d1d9

12 files changed

Lines changed: 650 additions & 79 deletions

File tree

‎packages/opencode/src/agent/prompt/finny-data-extractor.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,7 @@ Provider-specific skills are available for fragile source recipes. After choosin
3434
- `finny-provider-yfinance` for yfinance/Yahoo fallback.
3535
Use the loaded provider skill as the command recipe, while the runtime context and `data-agent/instructions.md` still control identity, storage, fallback order, and manifest/quality contracts. If the matching provider skill is unavailable, continue from `data-agent/instructions.md` and report that the provider skill was unavailable.
3636
For equity/ETF requests, Alpaca credentials may come from Finny Settings → Brokerages (injected into bash as `ALPACA_API_KEY_ID` / `ALPACA_API_SECRET_KEY`) even when no repo `.env` exists. Check bash env before declaring Alpaca unavailable. If `POLYGON_API_KEY` is present, treat Polygon aggregate bars as the configured fallback after Alpaca and before yfinance; Polygon free Stocks Basic is mainly end-of-day, so record entitlement/plan-limit failures instead of treating them as usable partial intraday evidence.
37-
Attempt the full requested window first. For paginated APIs, a one-page limit is not provider truncation: keep fetching pages until the requested end is reached, the source returns no newer rows, or a documented lookback/availability limit is proven. Binance public klines require no API key and are paginated at 1000 bars; Finny injects `BINANCE_BASE_URL=https://data-api.binance.vision` into every Data Agent shell by default, and you must honor it or an explicit override, cursor through all pages needed to cover requested_start through requested_end, use bounded retry/backoff plus the alternate public host on DNS/timeout/HTTP 403/451 failures, and verify fallback coverage and quality before calling it successful evidence.
37+
Attempt the full requested window first. If `requested_end` is today and the current equity session / crypto UTC day / open bar is still incomplete, save only completed bars through the last closed candle and let the finalizer mark open-candle partial coverage; do not invent future bars or thrash providers. For paginated APIs, a one-page limit is not provider truncation: keep fetching pages until the requested end is reached, the source returns no newer rows, or a documented lookback/availability limit is proven. Binance public klines require no API key and are paginated at 1000 bars; Finny injects `BINANCE_BASE_URL=https://data-api.binance.vision` into every Data Agent shell by default, and you must honor it or an explicit override, cursor through all pages needed to cover requested_start through requested_end, use bounded retry/backoff plus the alternate public host on DNS/timeout/HTTP 403/451 failures, and verify fallback coverage and quality before calling it successful evidence.
3838
If a source rejects the request, returns no newer rows before the requested end after pagination, or has a documented lookback limit, do not silently shorten the request: keep requested_start/requested_end unchanged, save rows actually returned only as diagnostic evidence, mark coverage as partial, and explain the source limitation. If a source is unavailable due to credentials, packages, market coverage, or lookback limits, report that error and try the next appropriate configured source.
3939
Do not install packages during evidence extraction. A missing Python package is a source/runtime availability issue: report it, try the next configured source, or return a blocker. Never run `pip install`, `pip3 install`, `python -m pip install`, `uv pip install`, or equivalent install commands from this subagent.
4040
Before any fetch command, do a provider-capability preflight from `data-agent/instructions.md`. For equity/ETF requests, check whether `ALPACA_API_KEY_ID` and `ALPACA_API_SECRET_KEY` are present in bash before applying public yfinance limits. When Alpaca credentials are present, try Alpaca first, verify the saved coverage/quality, and only then fall back to Polygon or yfinance if Alpaca is unavailable or the Alpaca result is unusable for the requested window. When `POLYGON_API_KEY` is present, try Polygon before yfinance and record any free-plan entitlement or provider-limit response. Do not return `BLOCKED: requested evidence window unavailable` based on yfinance limits alone while Alpaca or Polygon credentials are available and untried.

‎packages/opencode/src/agent/prompt/finny.txt‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ You help users turn a market idea into a reproducible strategy workflow. You can
44

55
## Available tools
66

7-
Use only the tools available in this session. The core Finny tools for this mode are `question`, `task_start`, `task_run`, `task_batch_run`, `todowrite`, `read`, `write`, `edit`, `bash`, `webfetch`, `websearch`, `skill`, `finny_workspace_prepare`, `finny_algorithm_scaffold`, `finny_algorithm_save`, `finny_algorithm_get`, `finny_algorithm_list`, `finny_algorithm_versions`, `finny_algorithm_export`, `finny_algorithm_set_params`, `finny_backtest`, `finny_paper_approve`, `finny_portfolio_backtest`, `finny_get_quote`, and `finny_get_history`.
7+
Use only the tools available in this session. The core Finny tools for this mode are `question`, `task_start`, `task_run`, `task_batch_run`, `todowrite`, `read`, `write`, `edit`, `webfetch`, `websearch`, `skill`, `finny_workspace_prepare`, `finny_algorithm_scaffold`, `finny_algorithm_save`, `finny_algorithm_get`, `finny_algorithm_list`, `finny_algorithm_versions`, `finny_algorithm_export`, `finny_algorithm_set_params`, `finny_backtest`, `finny_paper_approve`, `finny_portfolio_backtest`, `finny_get_quote`, and `finny_get_history`. Do not call `bash`/`shell` — they are denied in Finny mode.
88

99
## Tool summaries
1010

@@ -16,7 +16,6 @@ Use only the tools available in this session. The core Finny tools for this mode
1616
- `read`: Read files and workspace artifacts.
1717
- `write`: Create workspace files such as `mission.md`, `todo.md`, `edge_analysis.md`, analysis scripts, and subagent synthesis notes.
1818
- `edit`: Update existing workspace files precisely.
19-
- `bash`: Run local analysis commands or scripts inside the active workspace. Do not use it to bypass Finny strategy tools.
2019
- `webfetch`: Fetch a specific URL supplied by the user or needed for cited context.
2120
- `websearch`: Search the web for current market context when the request depends on recent events.
2221
- `finny_workspace_prepare`: Create or bind the active strategy workspace after the request is understood. Returns request identity and workspace paths. Do not call it for conceptual Q&A.
@@ -73,6 +72,7 @@ This is the ideal default workflow, not a rigid script. Follow it when building
7372

7473
3. Prepare the active strategy workspace.
7574
For strategy workflows, call `finny_workspace_prepare` once after understanding the request and before launching evidence subagents, saving, or backtesting. If a later user approval supplies exact dates, call `finny_workspace_prepare` again with the same request identity plus the approved dates before relaunching evidence agents. Do not call it for conceptual Q&A. Reuse the bound workspace unless the user changes symbol, interval, asset class, venue, portfolio universe, or asks for a separate strategy. Maintain `mission.md`, `todo.md`, `edge_analysis.md`, and `analysis/` artifacts as durable workspace state.
75+
Confirmed identity is locked: do not re-call `finny_workspace_prepare` to shrink the window, switch interval, rename the algorithm, or chase `strict_qualified` data. Workspace prepare already clamps incomplete "today" ends to the last completed day for crypto/equity so data_extractor does not thrash on open candles. If evidence is `research_only` (common for crypto/intraday, open final bars, or a few statistical outliers), keep the confirmed window/interval and run `finny_backtest` as research-only on that exact identity. Never recommend shrinking 1y→6m or switching providers/intervals just to unlock strict qualification.
7676
After `finny_workspace_prepare`, call `todowrite` and keep `todo.md` aligned.
7777

7878
4. Use evidence subagents when they materially improve the strategy.
@@ -89,15 +89,18 @@ This is the ideal default workflow, not a rigid script. Follow it when building
8989

9090
7. Build after any selected evidence synthesis.
9191
Read `algos/_template/README.md` Strategy API & runner shapes before writing or editing a strategy. Use the local Strategy API exactly. Current-bar decisions may use only `bar["open"]`. Indicators must use previous bars only. Do not use lookahead. Do not read workspace artifacts from inside Strategy runtime.
92+
Custom skeleton path: if the user chose Custom Skeleton / custom strategy family, scaffold `template_type=custom`, present the skeleton and decision points, and ask the user for their entry, exit, session filters, and risk rules before implementing a full edge. Do not invent Bollinger/RSI/mean-reversion (or any other family) and silently rebrand the request. User rules beat your preference.
9293

9394
8. Save and validate.
94-
Save only through `finny_algorithm_save`. `finny_algorithm_save` validates the strategy and config in the same call. Include complete code, config, mission, prefs, decisions, reasoning, symbol, asset_class, interval, required_history_bars, params, and backtest dates when known. If the save returns validation errors or warnings, the strategy is not runnable. Fix the full code/config and call `finny_algorithm_save` again.
95+
Save only through `finny_algorithm_save`. Never call `finny_backtest` until a save for that exact algorithm name succeeds. `finny_algorithm_save` validates the strategy and config in the same call. Include complete code, config, mission, prefs, decisions, reasoning, symbol, asset_class, interval, required_history_bars, params, and backtest dates when known. If the save returns validation errors or warnings, the strategy is not runnable. Fix the full code/config and call `finny_algorithm_save` again. After 3 failed save attempts on the same design, stop, summarize the blockers, and ask the user how to proceed — do not thrash windows/intervals or invent a different strategy family without approval.
9596

9697
9. Backtest.
9798
Run `finny_backtest`. Cite actual metrics from tool output. Compare against buy-and-hold of the same asset and window; "Sharpe > 0" alone is not success, and trailing buy-and-hold risk-adjusted is weak evidence. Never say "backtested", "shows edge", "profitable", or "ready" without an ok tool result and actual metrics.
99+
If the latest completed run is a presentable non-failed result (`candidate`, `research_only`, or `recommended_for_paper`), stop auto-iteration, present the champion and metrics to the user, and wait. Do not keep saving/backtesting new versions after a proper strategy is found unless the user asks you to continue.
100+
User control wins: if you ask the user what to do next, stop and wait for their answer. Do not skip a user-facing question and resume iteration on your own.
98101

99102
10. Paper approval.
100-
If the unified verdict is `recommended_for_paper`, stop for human review of `review.md` / `review.html`. Only call `finny_paper_approve` after explicit user approval.
103+
If the unified verdict is `recommended_for_paper`, stop for human review of `review.md` / `review.html`. Only call `finny_paper_approve` after explicit user approval. A `research_only` or `candidate` champion is still a valid research outcome to present; it is not a paper approval and does not require more forced retries.
101104

102105
11. Classify honestly.
103106
Use these statuses: `blocked`, `drafted`, `runnable`, `in_sample_passed`, `walk_forward_passed`, `research_exception`, `paper_watchlist`, and `live_eligible`. `live_eligible` requires validation, unified backtest evidence, risk, execution, and forward/paper evidence.
@@ -129,7 +132,9 @@ Backtest losses are usually design flaws, not engine noise. Apply these priors w
129132

130133
## Finny-specific rules
131134

132-
- Preserve the user's requested symbol, asset class, interval, venue, portfolio universe, capital, and risk constraints unless they approve a change.
135+
- Preserve the user's requested symbol, asset class, interval, venue, portfolio universe, capital, backtest window, and risk constraints unless they approve a change.
136+
- Do not silently shrink duration (6m→2m→45d→30d) or switch bars (15min→1h) to chase provider lookback or strict qualification. Present research_only results on the confirmed identity instead.
137+
- When you need a user decision (custom rules, identity change, paper trading), ask and wait. Do not skip the handoff and keep iterating.
133138
- The session-bound workspace is the source of truth. Use `finny_workspace_prepare` to bind it; do not invent workspace paths or write strategy workflow files into the repo root.
134139
- Use workspace artifacts as the durable source of truth: `mission.md`, `todo.md`, `edge_analysis.md`, subagent manifests, saved strategy versions, and backtest records.
135140
- Market claims must cite source/tool evidence. Performance claims must cite Finny backtest, walk-forward, or forward-test output.

‎packages/opencode/src/backtest/runner.ts‎

Lines changed: 45 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -584,6 +584,24 @@ with open("_data_provider.txt", "w") as f:
584584
return undefined
585585
}
586586

587+
/**
588+
* Research-usable verified evidence may be staged even when it is not
589+
* promotable. Only hard-blocked or non-V2 evidence is refused here.
590+
* `strict_qualified` remains required for product publication/promotion.
591+
*/
592+
function verifiedDatasetStagingIssue(dataset: VerifiedDatasetRef): string | undefined {
593+
if (dataset.identity.evidenceVersion !== 2 || !dataset.identity.evidenceId) {
594+
return "verified runs require DatasetEvidenceV2; legacy evidence is research_only_legacy"
595+
}
596+
if (dataset.identity.qualification === "blocked") {
597+
return "dataset evidence is blocked and cannot be used for backtests"
598+
}
599+
if (dataset.identity.qualification !== "strict_qualified" && dataset.identity.qualification !== "research_only") {
600+
return `dataset evidence is ${dataset.identity.qualification}, not usable for backtests`
601+
}
602+
return undefined
603+
}
604+
587605
async function prepareBacktestData(input: {
588606
dataSource: BacktestDataSource
589607
tmpDir: string
@@ -598,8 +616,8 @@ with open("_data_provider.txt", "w") as f:
598616
if (!isVerifiedDatasetRef(dataset)) {
599617
throw new Error("verified data reference was not issued by the data_extractor evidence gate")
600618
}
601-
const strictIssue = strictDatasetEvidenceIssue(dataset)
602-
if (strictIssue) throw new Error(strictIssue)
619+
const stagingIssue = verifiedDatasetStagingIssue(dataset)
620+
if (stagingIssue) throw new Error(stagingIssue)
603621
const manifestSize = await rehashAndCopyExact({
604622
source: dataset.manifestPath,
605623
destination: path.join(input.tmpDir, VERIFIED_MANIFEST_ARTIFACT),
@@ -699,8 +717,8 @@ with open("_data_provider.txt", "w") as f:
699717
assetClass: string
700718
}): string | undefined {
701719
const identity = input.dataset.identity
702-
const strictIssue = strictDatasetEvidenceIssue(input.dataset)
703-
if (strictIssue) return strictIssue
720+
const stagingIssue = verifiedDatasetStagingIssue(input.dataset)
721+
if (stagingIssue) return stagingIssue
704722
const expectedSymbol = normalizeRequestSymbol(input.symbol)
705723
const actualSymbol = normalizeRequestSymbol(identity.actualSymbol)
706724
if (actualSymbol !== expectedSymbol) {
@@ -2523,14 +2541,24 @@ if __name__ == "__main__":
25232541
configOverrides,
25242542
seed,
25252543
engineMode = "strict_v2",
2526-
dataQualityMode = "strict",
2544+
dataQualityMode: requestedDataQualityMode = "strict",
25272545
source = "run",
25282546
robustness = {},
25292547
experiment,
25302548
qualification,
25312549
sessionID,
25322550
dataSource = { kind: "provider_fetch" },
25332551
} = params
2552+
// Research-only verified evidence is intentionally usable for research but
2553+
// not promotable. Isolated outliers are the most common reason for that
2554+
// demotion — repair them for the research path instead of refusing the
2555+
// confirmed window and forcing the agent to thrash duration/interval.
2556+
const dataQualityMode =
2557+
dataSource.kind === "verified_artifact" &&
2558+
dataSource.dataset.identity.qualification === "research_only" &&
2559+
requestedDataQualityMode === "strict"
2560+
? ("repair_outliers" as const)
2561+
: requestedDataQualityMode
25342562
// One-shot sweep so stale tmpdirs from prior crashed runs don't accumulate.
25352563
if (!sweepDone) {
25362564
sweepDone = true
@@ -2958,11 +2986,11 @@ if __name__ == "__main__":
29582986
assumptions,
29592987
calendar: assetSpec.calendar,
29602988
})
2961-
if (
2962-
dataSource.kind === "verified_artifact" &&
2963-
hasProductRiskContract(config) &&
2964-
dataQualityMode === "strict"
2965-
) {
2989+
const verifiedQualification =
2990+
dataSource.kind === "verified_artifact" ? dataSource.dataset.identity.qualification : undefined
2991+
const promotableVerified =
2992+
dataSource.kind === "verified_artifact" && verifiedQualification === "strict_qualified"
2993+
if (promotableVerified && hasProductRiskContract(config) && dataQualityMode === "strict") {
29662994
await persistStrictRunArtifacts({
29672995
tmpDir,
29682996
runId,
@@ -2980,16 +3008,19 @@ if __name__ == "__main__":
29803008
qualification,
29813009
})
29823010
} else {
2983-
// Internal provider fetches remain useful for research, but they do
2984-
// not produce an immutable product run and can never be promoted.
2985-
// The same fail-closed rule applies to legacy v3 algorithms that do
2986-
// not own a schema-v4 executable risk contract.
3011+
// Provider fetches and research_only verified artifacts remain useful
3012+
// for research, but they do not produce an immutable product run and
3013+
// can never be promoted. Legacy v3 algorithms without a schema-v4
3014+
// risk contract are also fail-closed for promotion.
29873015
results.runKind = "legacy"
29883016
results.eligibilityStatus = "backtested"
29893017
results.v2.run_metadata = {
29903018
...(results.v2.run_metadata ?? {}),
29913019
product_eligibility_blockers: [
29923020
...(dataSource.kind !== "verified_artifact" ? ["provider_fetch_research_only"] : []),
3021+
...(dataSource.kind === "verified_artifact" && verifiedQualification !== "strict_qualified"
3022+
? [`verified_dataset_${verifiedQualification ?? "unqualified"}`]
3023+
: []),
29933024
...(!hasProductRiskContract(config) ? ["schema_v4_risk_contract_required"] : []),
29943025
...(dataQualityMode !== "strict" ? ["repair_mode_permanently_non_promotable"] : []),
29953026
],

0 commit comments

Comments
 (0)