pfeed: The Single Source of Truth for Algo-Trading Data. Uniting Traders to Clean Once & Share with All.

TL;DR: use pfeed to manage your trading data; other traders will help you clean it

Problem

Starting algo-trading requires reliable and clean data, but traders often work in silos, each writing duplicated code to clean the same datasets—wasting time and effort. Accessing clean, ready-to-use data is a challenge, forcing traders to handle tedious data tasks before they can even start trading.

Solution

pfeed leverages modern data engineering tools to centralize data cleaning efforts, automate ETL/ELT, store data in a local data lakehouse (Parquet + DuckLake), and output backtesting-ready data, allowing traders to focus on strategy development.

---

pfeed (/piː fiːd/) is the data engine for trading, serving as a pipeline between raw data sources and traders. It enables you to download historical data, stream real-time data, and store cleaned data in a local data lake for quantitative analysis, supporting both batch processing and streaming workflows through streamlined data collection, cleaning, transformation, and storage.

Core Features

• Download or stream reliable, validated and clean data for research, backtesting, or live trading
• Get historical data (dataframe) or live data in standardized formats by just calling one single function
• Own your data — store locally now, write directly to the cloud later
• Build a local data lakehouse (Parquet + DuckLake) on your own machine
• Interact with different kinds of data (including TradFi, CeFi and DeFi) using a unified interface

---

Table of Contents

• Installation
• Quick Start
  • Download Historical Data
  • Retrieve Stored Data
  • Stream Live Data
  • Replay Stored Data as a Stream
• Supported Data Sources
• Supported IOs
• Related Projects
• Disclaimer

Installation

For more installation options, please refer to the documentation.

# pick your data source, e.g. bybit, and add ray for parallel processing (recommended).
pip install "pfeed[bybit,ray]"

Important

Windows users: enable long paths first, or storing data fails with [WinError 206] The filename or extension is too long. pfeed's data folders are deeper than Windows' default 260-character path limit. Run once in PowerShell as Administrator, then open a new terminal:

New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem" -Name "LongPathsEnabled" -Value 1 -PropertyType DWORD -Force

Quick Start

Want to try first? Run pfeed in your browser — no install needed — at pfeed.pfund.ai.

import pfeed as pe

bybit = pe.Bybit()
feed = bybit.market_feed

1. Download Historical Data

Download historical data

result = feed.download(
    product="BTC_USDT_PERP",  # or BTC_USDT_PERPETUAL in full
    resolution="1minute",  # '1tick'/'1t' or '2second'/'2s' etc.
    start_date="2026-01-01",
    end_date="2026-01-01",
    io=pe.ParquetIO(),  # None to skip storing
)
df = result.data.collect()

2. Retrieve Stored Data

Retrieve stored data

result = feed.retrieve(
    product="BTC_USDT_PERP",
    resolution="1m",  # 1-minute data
    start_date="2026-01-01",
    end_date="2026-01-01",
    io=pe.ParquetIO(),
)
df = result.data.collect()

3. Stream Live Data

Get streaming data

feed.stream(
    product="BTC_USDT_CRYPTO",
    resolution="1tick",  # or just 'tick' for tick data
    # only DuckLake supports storing streamed data for now, set it to None to skip storing
    io=pe.DuckLakeIO(),
    callback=lambda msg: print(msg),
)

4. Replay Stored Data as a Stream

Replay stored data message by message, like a live stream

feed.replay(
    product="BTC_USDT_PERP",
    resolution="1m",
    start_date="2026-01-01",
    end_date="2026-01-01",
    io=pe.ParquetIO(),  # where the data was stored
    callback=lambda msg: print(msg),
)

Supported Data Sources

Data Source	Data Categories	Download Historical Data	Stream Live Data
Hyperliquid	Market Data	🔴	🔴
Bybit	Market Data	🟢	🟢
OKX	Market Data	🔴	🔴
Binance	Market Data	🔴	🔴

Supported IOs

IO Format	Status
Parquet	🟢
DuckLake	🟢
Lance	🔴

🟢 = finished
🟡 = in progress
🔴 = todo
⚪ = not applicable
* = paid data

Related Projects

• alphafund - AI-Powered Personal Hedge Fund. Be Your Own Fund Manager.
• pfund — A Complete Algo-Trading Framework for Machine Learning, TradFi, CeFi and DeFi ready. Supports Vectorized and Event-Driven Backtesting, Paper and Live Trading
• pytrade - An Educational Hub for Trading with Python. Where Python Trading Begins

Disclaimer

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF, OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

This framework is intended for educational and research purposes only. It should not be used for real trading without understanding the risks involved. Trading in financial markets involves significant risk, and there is always the potential for loss. Your trading results may vary. No representation is being made that any account will or is likely to achieve profits or losses similar to those discussed on this platform.

The developers of this framework are not responsible for any financial losses incurred from using this software. This includes but not limited to losses resulting from inaccuracies in any financial data output by pfeed. Users should conduct their due diligence, verify the accuracy of any data produced by pfeed, and consult with a professional financial advisor before engaging in real trading activities.