Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: ml4t/data
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.2.0
Choose a base ref
...
head repository: ml4t/data
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v0.2.1
Choose a head ref
  • 8 commits
  • 113 files changed
  • 5 contributors

Commits on Sep 25, 2026

  1. docs: publish asset-class data-source references (#71)

    * docs: split market sources by asset class
    
    Closes #68.
    
    * docs: add economic and specialized source references
    
    Closes #69.
    
    * test: enforce source-reference contracts
    
    Closes #70.
    
    * ci: refresh ecosystem qualification snapshot
    
    ---------
    
    Co-authored-by: Stefan Jansen <[email protected]>
    stefan-jansen and Stefan Jansen authored Sep 25, 2026
    Configuration menu
    Copy the full SHA
    885bc11 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2026

  1. feat: prediction-market resolution outcomes, price and trade history …

    …(Kalshi, ForecastEx, Polymarket, Polymarket US) (#76)
    
    * feat(kalshi): enumerate resolved markets and trades across live and archive tiers
    
    KalshiProvider gains iter_markets/fetch_markets and iter_trades/fetch_trades.
    Both page through the live endpoints (/markets, /markets/trades) and the
    archive (/historical/markets, /historical/trades), using /historical/cutoff
    to skip a tier that cannot match and dropping duplicates. Filters the
    archive does not accept (status, close-time window, multivariate exclusion
    combined with a ticker filter) are applied on the client.
    
    fetch_markets returns a typed frame with a normalized resolution outcome
    (result in yes/no/void/other or null, settlement_value, settlement_ts), and
    fetch_trades returns trade_id, ticker, timestamp, price, count, taker_side,
    is_block_trade. Both the *_dollars/*_fp fields and legacy unsuffixed fields
    are read. list_markets and the other existing methods are unchanged.
    
    BaseProvider gets a shared rate-limited _request/_request_json helper that
    maps 429/404/5xx/transport failures to the shared exceptions and retries
    transient ones under the existing provider retry policy, now exposed as
    provider_retry. prediction_markets.py holds the outcome vocabulary and trade
    schema shared by the prediction-market adapters.
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
    
    * feat(providers): add ForecastEx and Polymarket US event-contract providers
    
    ForecastExProvider reads the free daily CSVs in the public
    forecastex-public-data bucket: fetch_prices (daily OHLC, marks, settlement
    price), fetch_trades (every matched YES/NO pair) and fetch_products. Its
    fetch_markets derives the resolution outcome: a contract absent from the
    next day's prices file has expired, and a final YES settlement price of
    1.00/0.00 is the payout. Intraday contracts that drop out with a daily mark
    instead of the payout are reported as expired with a null result rather
    than guessed.
    
    PolymarketUSProvider reads the public gateway of the CFTC-regulated
    Polymarket US exchange (separate from the global PolymarketProvider, which
    is unchanged): offset-paginated markets with a normalized outcome taken
    from the long market side, /settlement, and price history in fixed or
    24-hour windows. The gateway's outcomes labels are not always in
    outcomePrices order, so labels are never zipped with prices.
    
    Both are registered as "events" providers outside DataManager routing.
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
    
    * docs(providers): document prediction-market history and outcome loaders
    
    Add provider pages for ForecastEx and Polymarket US, extend the Kalshi page
    with resolved-market enumeration, the normalized outcome columns and trade
    history, and list both new venues in the provider index, the
    prediction-market source reference, provider selection and the navigation.
    
    Rename the new offline test modules so their basenames do not collide with
    the integration modules of the same venue.
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
    
    * build(deps): lock patched multidict, urllib3 and virtualenv
    
    The dependency audit now flags multidict 6.7.1 (CVE-2026-104874),
    urllib3 2.7.0 (PYSEC-2026-4175..4177) and virtualenv 21.7.1
    (PYSEC-2026-4011..4014). Upgrade the three transitive pins to 6.9.1,
    2.8.0 and 21.14.6; python-discovery moves with virtualenv.
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
    
    * feat(kalshi): expose mve_filter and keep series, previous price and taker sides
    
    Replace the exclude_multivariate flag with Kalshi's mve_filter parameter
    (None, "exclude", "only") on iter_markets/fetch_markets, so combo (KXMVE*)
    markets can be included, excluded or selected. /markets receives the value
    directly; /historical/markets rejects "only" and takes "exclude" only
    without a ticker filter, so the selection is completed on the client.
    
    The normalized market frame gains series (series filter or event-ticker
    prefix) and previous_price; the trade frame keeps taker_outcome_side and
    taker_book_side next to taker_side.
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
    
    * feat(kalshi): tier-aware candlesticks, streaming market frames, live contracts
    
    Price history now follows the market's tier. fetch_candles (and
    fetch_ohlcv, which shares the routing) reads archived markets from
    /historical/markets/{ticker}/candlesticks and live ones from
    /series/{series}/markets/{ticker}/candlesticks. An explicit tier or
    series_ticker skips detection; otherwise a live 404 falls back to the
    archive, and a second 404 looks the series up from the market's event,
    which fixes old tickers whose prefix is not their series (FED-23DEC is in
    KXFED). Ranges beyond Kalshi's 5000-candle limit are split.
    
    Prices are converted per field by encoding: *_dollars values and
    unsuffixed strings are dollars, unsuffixed integers are legacy cents. The
    previous blanket /100 on unsuffixed fields would have scaled archive
    candles down 100x. fetch_candles returns trade OHLC, mean and previous,
    YES bid/ask close, volume, open interest and the serving tier.
    
    iter_market_frames streams fetch_markets' normalized rows in frames of
    chunk_size rows for listings too large to materialize; fetch_markets now
    delegates to it.
    
    The provider-contracts public job runs the new Kalshi, ForecastEx and
    Polymarket US live smoke tests, and the workflow contract test checks it.
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
    
    * feat(polymarket): bulk resolved-market listing, windowed price history and trades
    
    iter_markets / iter_market_frames / fetch_markets page through Gamma's
    GET /markets/keyset (GET /markets refuses offsets beyond 2000), passing
    next_cursor back as after_cursor; any other parameter name silently
    returns page 1, so a repeated cursor raises instead of looping. The
    server honours closed, volume_num_min, end_date_min and end_date_max
    (verified 2026-10-08) and caps pages at 100; the filters are re-applied
    on the client. include_tag=true brings the market tags, the reliable
    topic field now that category is mostly null.
    
    Rows are normalized to the shared resolution vocabulary. Gamma markets
    are binary and ordered: the first outcome is the YES side whatever its
    label, final prices [1,0]/[0,1] are yes/no, [0.5,0.5] is void, a UMA
    "resolved" market with other prices is "other", the rest is null.
    settlement_ts is closedTime (else umaEndDate), and the frame carries
    token ids, event slug/title, neg-risk and the raw fee fields.
    
    fetch_candles always sends startTs/endTs with fidelity, split into
    windows of at most 15 days: interval=max returns nothing for resolved
    markets below daily fidelity, and longer spans are rejected with 400.
    
    fetch_trades pages data-api /trades newest first. Past the offset cap of
    10000 it moves the inclusive end bound to the oldest second seen and
    drops the boundary trades already yielded; a repeated page or a second
    too dense to page past raises. Prices are converted to the YES price
    and the taker side to yes/no from outcome index and BUY/SELL.
    
    Offline tests use responses recorded on 2026-10-08; each was checked to
    fail with its defect planted.
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
    
    * fix(polymarket): fetch_ohlcv returned nothing for past date ranges
    
    The CLOB treats interval as a range ending now that overrides startTs
    and endTs, so fetch_ohlcv's interval=1h chunks came back empty for any
    past window (reproduced live on 2026-10-08 for a market resolved in
    2025). Chunks now send their bounds with the interval's step as
    fidelity. YYYY-MM-DD bounds were also converted in the machine's local
    zone; they are UTC days now, and "max" no longer builds a single
    multi-year window the CLOB rejects.
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
    
    * docs(polymarket): document resolved history and run its live contracts
    
    The Polymarket provider page covers fetch_markets / iter_market_frames
    (keyset paging, normalized columns, first-outcome-is-YES mapping),
    fetch_candles (15-day windows, the empty interval=max history of
    resolved markets) and fetch_trades (offset cap and end-bound paging),
    and replaces the outdated 7-day range advice. The prediction-market
    reference lists Polymarket among the venues with normalized outcomes.
    
    TestPolymarketResolvedHistory smoke-tests the three calls against a
    market resolved on 2026-01-01; the provider-contracts public job runs
    it and the workflow contract test checks that it does.
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
    
    * test(polymarket): skip the local-zone test where time.tzset is missing
    
    Windows has no time.tzset, so the ecosystem qualification job errored in
    the fixture that switches the local zone to America/New_York.
    
    Co-Authored-By: Claude Opus 5.5 <[email protected]>
    
    ---------
    
    Co-authored-by: Stefan Jansen <[email protected]>
    Co-authored-by: Claude Opus 5.5 <[email protected]>
    3 people authored Oct 8, 2026
    Configuration menu
    Copy the full SHA
    1855154 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2026

  1. ci: bump the github-actions group across 1 directory with 3 updates (#81

    )
    
    Bumps the github-actions group with 3 updates in the / directory: [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv), [actions/upload-artifact](https://github.com/actions/upload-artifact) and [actions/download-artifact](https://github.com/actions/download-artifact).
    
    
    Updates `astral-sh/setup-uv` from 10.1.0 to 10.3.0
    - [Release notes](https://github.com/astral-sh/setup-uv/releases)
    - [Commits](astral-sh/setup-uv@bec219d...1c37ad0)
    
    Updates `actions/upload-artifact` from 7.0.1 to 7.0.2
    - [Release notes](https://github.com/actions/upload-artifact/releases)
    - [Commits](actions/upload-artifact@043fb46...cf430e0)
    
    Updates `actions/download-artifact` from 8.0.1 to 8.0.2
    - [Release notes](https://github.com/actions/download-artifact/releases)
    - [Commits](actions/download-artifact@3e5f45b...9000827)
    
    ---
    updated-dependencies:
    - dependency-name: astral-sh/setup-uv
      dependency-version: 10.3.0
      dependency-type: direct:production
      update-type: version-update:semver-minor
      dependency-group: github-actions
    - dependency-name: actions/upload-artifact
      dependency-version: 7.0.2
      dependency-type: direct:production
      update-type: version-update:semver-patch
      dependency-group: github-actions
    - dependency-name: actions/download-artifact
      dependency-version: 8.0.2
      dependency-type: direct:production
      update-type: version-update:semver-patch
      dependency-group: github-actions
    ...
    
    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    dependabot[bot] authored Oct 14, 2026
    Configuration menu
    Copy the full SHA
    dfcce14 View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2026

  1. docs: correct stale guidance and enforce link integrity (#78)

    * docs: enforce published link integrity
    
    * docs: replace stale contributor guidance
    
    * fix: retry deployed documentation probes
    
    * docs: remove stale legacy guidance
    
    * fix: tolerate transient external link failures
    
    * docs: describe supported Massive financial statements
    
    * docs: make session validation examples executable
    
    ---------
    
    Co-authored-by: Stefan Jansen <[email protected]>
    stefan-jansen and Stefan Jansen authored Oct 15, 2026
    Configuration menu
    Copy the full SHA
    8055e70 View commit details
    Browse the repository at this point in the history
  2. ci: enforce minimum package coverage (#80)

    * docs: enforce published link integrity
    
    * docs: replace stale contributor guidance
    
    * fix: retry deployed documentation probes
    
    * docs: remove stale legacy guidance
    
    * fix: tolerate transient external link failures
    
    * ci: enforce minimum package coverage
    
    * docs: align agent test command with coverage gate
    
    * docs: document enforced coverage lane
    
    * ci: retain current setup-uv pin in coverage job
    
    ---------
    
    Co-authored-by: Stefan Jansen <[email protected]>
    stefan-jansen and Stefan Jansen authored Oct 15, 2026
    Configuration menu
    Copy the full SHA
    bc24706 View commit details
    Browse the repository at this point in the history
  3. docs: adopt shared focused-contribution policy (#83)

    * docs: adopt shared contribution policy
    
    * docs: align contribution checks with coverage gate
    
    ---------
    
    Co-authored-by: Stefan Jansen <[email protected]>
    stefan-jansen and Stefan Jansen authored Oct 15, 2026
    Configuration menu
    Copy the full SHA
    72b5e9b View commit details
    Browse the repository at this point in the history
  4. fix: restore inclusive date ranges and complete Massive histories (#85)

    Include the real month end in Fama-French filters, include the stored end-date bar in gap detection, request the inclusive macro end date from Yahoo, interpret Finnhub date bounds in UTC, and follow Massive aggregate pagination.
    
    Closes #73
    
    Preserves the five applicable fixes contributed by quantum-like in #74. The Kalshi and Polymarket UTC fixes from that contribution are already supplied by #76.
    
    The regressions failed against the original base and passed with the fixes. The replacement retains the original contribution commits for review and passed the supported CI checks in #85.
    
    Co-authored-by: quantum-like <[email protected]>
    stefan-jansen and quantum-like authored Oct 15, 2026
    Configuration menu
    Copy the full SHA
    f2a2252 View commit details
    Browse the repository at this point in the history
  5. docs: add verified roic.ai fundamentals source (#86)

    Addresses #84 with the documented free annual-history restriction and point-in-time caveat.
    
    Co-authored-by: Stefan Jansen <[email protected]>
    stefan-jansen and Stefan Jansen authored Oct 15, 2026
    Configuration menu
    Copy the full SHA
    6888629 View commit details
    Browse the repository at this point in the history
Loading