Skip to content

Add optional retry mechanism for transient network errors - #2627

Merged
ValueRaider merged 1 commit into
ranaroussi:devfrom
axisrow:feature/retry-mechanism
Dec 2, 2025
Merged

ValueRaider merged 1 commit into
ranaroussi:devfrom
axisrow:feature/retry-mechanism

Conversation

@axisrow

@axisrow axisrow commented Nov 14, 2025 •

Copy link
Copy Markdown
Contributor

Summary

  • Adds optional retries parameter to download() function for handling transient network errors
  • Implements exponential backoff (1s, 2s, 4s...) for network/timeout errors
  • Only retries on transient errors (socket, timeout), skips permanent errors (invalid ticker)
  • Backward compatible - retries=0 by default means no change to existing behavior

Changes

  • Added retries parameter to download() with default value 0
  • Implemented _is_transient_error() helper function to distinguish error types
  • Modified _download_one() to support retry loop with exponential backoff
  • Propogated retries parameter through threading and sync code paths

Testing

  • Added test_download_retry_on_transient_error - verifies retry on network errors
  • Added test_download_no_retry_on_permanent_error - verifies no retry for invalid ticker
  • All tests pass successfully

Usage Examples

yf.download('AAPL', period='1y') # No retry (default behavior)

yf.download('AAPL', period='1y', retries=2) # With 2 retries for network errors

yf.download(['AAPL', 'MSFT', 'GOOG'], retries=1) # Bulk download with retry

@ValueRaider

ValueRaider commented Nov 14, 2025 •

Copy link
Copy Markdown
Collaborator

curl_cffi can't do this?

@axisrow

axisrow commented Nov 18, 2025 •

Copy link
Copy Markdown
Contributor Author

curl_cffi doesn’t provide the high-level retry logic that yfinance needs. It can retry some low-level transport errors, but it cannot distinguish transient vs. permanent ticker errors or propagate retry settings through threaded/bulk downloads — especially when using Threads. I download around 1500 tickers in different environments once per day, and 2–20 of them consistently fail due to timeouts. I’ve already written my own wrapper, but adding this logic directly inside the library is just a couple of lines of code and avoids having to parse curl_cffi’s textual output, which is very inconvenient.

There is example:
2025-11-18 17:24:10,492 [ERROR] ['MYFW']: Timeout('Failed to perform, curl: (28) Connection timed out after 10002 milliseconds. See https://curl.se/libcurl/c/libcurl-errors.html first for more details.')
2025-11-18 17:24:10,492 [ERROR] ['VRNS']: Timeout('Failed to perform, curl: (28) Connection timed out after 10001 milliseconds. See https://curl.se/libcurl/c/libcurl-errors.html first for more details.')

Such timed outs are always fixed by second retry. If second retry is inside of library no need to write an outer fix.

@ValueRaider

ValueRaider commented Nov 19, 2025 •

Copy link
Copy Markdown
Collaborator

Fair retort. Can it be moved into YfData?

I'm working on adding a new config class here #2525 https://github.com/ranaroussi/yfinance/blob/feature/expose-exceptions-v2/yfinance/config.py.

So then won't need to add any function parameters, just set once at top of script. But first I need to get it merged in.

@axisrow

axisrow commented Nov 21, 2025

Copy link
Copy Markdown
Contributor Author

Fair retort. Can it be moved into YfData?

I'm working on adding a new config class here #2525 https://github.com/ranaroussi/yfinance/blob/feature/expose-exceptions-v2/yfinance/config.py.

So then won't need to add any function parameters, just set once at top of script. But first I need to get it merged in.

Done

Do you need any help? I can write docs. Do all dirty job.

@ValueRaider

ValueRaider commented Nov 23, 2025 •

Copy link
Copy Markdown
Collaborator

#2525 now merged in so you can incorporate new YfConfig. You probably want to rebase to dev HEAD.

@axisrow
axisrow force-pushed the feature/retry-mechanism branch from 3a70d67 to 11ff7b1 Compare November 24, 2025 02:13
@axisrow

axisrow commented Nov 24, 2025

Copy link
Copy Markdown
Contributor Author

Done

@ValueRaider

Copy link
Copy Markdown
Collaborator

Did you explore moving the retry logic into YfData? Then all fetches will benefit.

@axisrow

axisrow commented Nov 27, 2025

Copy link
Copy Markdown
Contributor Author

Did you explore moving the retry logic into YfData? Then all fetches will benefit.

Retry logic has been moved from multi.py to data.py
Now all API calls (download, history, info, earnings, options, news) receive a retry.
This is exactly what you asked for.

@ValueRaider

Copy link
Copy Markdown
Collaborator

@ValueRaider

ValueRaider commented Nov 29, 2025 •

Copy link
Copy Markdown
Collaborator

And update the docs https://ranaroussi.github.io/yfinance/development/documentation.html

@axisrow
axisrow force-pushed the feature/retry-mechanism branch from bf6d629 to 542878a Compare December 1, 2025 00:36
@axisrow

axisrow commented Dec 1, 2025

Copy link
Copy Markdown
Contributor Author

Done
United all commits into one
Updated documentation for retries parameter

@axisrow
axisrow force-pushed the feature/retry-mechanism branch from 542878a to 944d103 Compare December 1, 2025 00:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants