Repository navigation
feat(acquisition): qualify provider/cache migration and legacy data utilities - #129
Merged
Merged
Conversation
added 10 commits
October 20, 2026 22:36
…h native providers Add scalar FRED and adjusted-close Yahoo paths, explicit Fama-French cache policy and an opt-in Empyrical acquisition adapter. Reuse native provider recovery and locking; preserve original missing observations and distinguish legacy presentation from source facts. Replay official daily/monthly ZIP bytes and independent legacy captures through actual parser/cache paths. Atomic single-file publication preserves valid caches on interrupted refresh. Remaining migration options, installed consumers and final gates stay tracked in #121.
Retain explicit annual-percent Treasury semantics and fail when source metadata does not establish those units. Reject infinite simple returns instead of publishing undefined zero-denominator results. Actual captured-provider tests fail on4060 and pass with the guards; add default-weekday, no-new-yield-row and native-versus-legacy endpoint workflows.
…bels An installed immutable Empyrical0.7.0/pandas-datareader0.11.1 replay confirms that mid-month bounds include the labeled monthly observation and return a UTC nanosecond index. Preserve those semantics in the opt-in adapter while leaving native provider filtering unchanged. Concatenate acquired tables using the legacy column/sample rules rather than silently suffixing overlapping factor names. Do not label breakpoint datasets as decimal returns.
Replay of immutable Empyrical0.7.0 and pandas-datareader0.11.1 confirms a Date index name and pandas-dependent period timestamp unit: ns on2.3.3,us on3.0.6. Reuse pandas period presentation instead of forcing ns and retain the actual Yahoo SDK index name in response metadata.178 focused provider/compatibility tests pass after main reconciliation; earlier unit-assumption drafts are not qualified evidence.
… presentation Add explicit public CSV mode through the existing FRED request/validation path; keep API credentials required by default and never fall back after configured authentication failures. Preserve unknown generic units/publication and label documented Treasury units separately. Replay actual anonymous Treasury and paired Yahoo DST captures, restoring actual SDK clock units. Cover overlapping FF3/FF5 columns and real ragged1926 factor histories including Saturday observations.186 focused provider/compatibility tests pass.
…lid returns The captured official source header reserves only-99.99 and-999 as missing. The previous less-than-90percent cutoff turned a contract-valid-95percent factor into a gap. A full ZIP/HTTP/parser/compatibility workflow fails on ba41 and passes with exact sentinel matching;187 focused provider and migration tests pass. The controlled numeric variant is not represented as an actual historical return capture.
…tion workflows The adjusted-close route explicitly keeps missing SDK rows before deriving no-fill returns; otherwise yfinance's default row removal silently bridges gaps. Preserve existing OHLCV defaults and shared recovery. Add a runnable offline provider/cache/import migration recipe and API/provider guidance.189 focused provider/migration tests,static/type checks,offline recipe and strict docs pass. Installed consumers and complete final-revision gates remain pending under #121.
…nsumers The former Yahoo range admitted SDK releases missing the exceptions used by public empty-download recovery. Reproduce the native failure with installed yfinance 0.2.1 and require the qualified 1.5.2 range; retain the existing resolved lock version. Record actual isolated wheel and sdist consumer journeys and their immutable legacy revisions.
Preserve the legacy Yahoo read-failure warning while retaining Data provider exception types. The installed 01afffd public transport-failure regression fails without the warning and the repaired 190-case provider/acquisition lane passes. Seven concrete producing-code failure controls are covered with no waivers or deferrals. Full parent source lanes each pass 4489 tests; final exact-head hosted full/platform/security qualification remains required before merge. Closes #121.
stefan-jansen
added this to the
Successor migration - Data inputs and provenance milestone
stefan-jansen
marked this pull request as ready for review
October 21, 2026 04:06
Hosted Windows 3.13 reproduces 13 cache publication failures because fsync rejects the read-only staging handle. Open the existing staged file read/write before flushing; keep fsync and atomic replacement intact. Two actual CSV publication controls reproduce the writable-handle requirement using zero-byte OS writes and pass with the fix. Pin the new offline example link to its verified immutable source commit so PR documentation does not require a file that is not on main yet.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Acquisition and cache migration
Closes #121.
Add FRED scalar observations, explicit Fama-French cache policy and opt-in
ml4t.data.compat.empyricalacquisition utilities. Existing provider/OHLCV defaults and nativepaths remain unchanged. No analytical facade or mandatory Reloaded dependency is added.
The compatibility route preserves factor/RF scales and complete-case samples, Treasury annual
percent units, Yahoo adjusted-price basis and exclusive bounds, actual CSV filenames and mtime
freshness, and the Pyfolio return-function hook. Unknown source publication remains unknown.
Native source nulls remain available. Yahoo returns do not fill gaps or deduct dividends twice.
FRED public CSV access is explicit; configured API errors never silently fall back to it.
Preserve valid cache files after failed refreshes using existing locks and atomic publication.
Validate source keys/schema/values before publishing. The Yahoo optional dependency now requires
the tested SDK range because older admitted versions fail public empty-download recovery.
Evidence
provider parsers and files. Original immutable Empyrical/PDR replays establish pandas 2/3
date-index shapes and period-bound behavior.
package coverage above 85%, ruff/format, ty, strict docs, build, actionlint and all hooks.
acquisition/provider lane. Exact-head hosted full/platform/security gates remain mandatory.
Pyfolio's registered callback and Alphalens forward-return preparation. Published Diagnostic
0.1.9 and immutable Pyfolio/Alphalens source revisions are identified in the migration guide.
pandas-market-calendars 5.4.0 and yfinance 1.5.2.
examples/acquisition_migration.pyruns offline without credentials or Reloaded packages.The migration guide states material definitions and qualified versions. These are unreleased
source APIs. This PR does not publish a package or deploy documentation.
AI assistance was used for implementation, tests, source review and migration documentation.
The Data maintainer agent reviewed the concrete code and executed the recorded workflows.