You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A user reported some website loading issues at scipy/scipy#23756. I've also noticed these, especially when I traverse into the SciPy stable release docs via i.e., a Google search for some specific object, like ConvexHull.
As far as action items, we could perhaps touch base with the folks hosting the stable release docs (Enthought folks still? I have some old email chains b/w Ralf and an admin there IIRC). I'll make a calendar reminder to look through those emails when I have access to Gmail (not at work, unfortunately).
FWIW, I set up a monitoring system using UptimeRobot to get data on when the downtime / slowness was. It works by making an HTTP request to https://docs.scipy.org/doc/scipy/building/index.html?uptime-robot-monitoring-contact-nickodell-for-issues=None and measuring how long it takes and whether it succeeds. Since I started tracking this data on Sep 29th, on average, it shows about 99% uptime and a mean of 2700 ms latency.
I haven't been able to see any patterns in this graph so far.
@dpinte You were able to help us last time from the Enthought side--are you still the correct contact for docs.scipy.org hosting? It seems unusually slow lately per several user reports and we just wanted to check in on that.
@tylerjereddy yes, I am. If I'm not responsive, please ping me by email. I'm flooded with github notifications ... I'm looking into the deployement and check what's going on.
Nothing special reported on the server. It's not a very beefy machine but system stats look good (cpu, memory and storage) Some maintenance will be required at some point though (OS upgrade).
+1 for the CDN. I am still puzzled on to why we do not have control on this website and the domain. I asked several years ago and it was not clear and people did not want me to help out so I let it roll sadly.
@Carreau that would definitely be a good idea. I can't promise anything in terms of my availabilities to do it.
@tupui the domain is managed by NumFocus and switched to CloudFare in 2023. And the website has never been transferred with the rest of the ressources. Very happy to support a migration if this is requested. Happy to transfer anything to anywhere and/or provide access to specific ressources.
Very happy to support a migration if this is requested. Happy to transfer anything to anywhere and/or provide access to specific ressources.
I could look into a migration. I would need some information:
What is the CPU / RAM / storage of the current set up? This would be useful for figuring out what the cost of the new service would be. (It's easier to ask for money if I know how much it will be. :) )
Can you provide an approximate number of page hits per 24 hours? This would be useful for capacity planning. Right now it's unclear to me if SciPy has out-grown its current server, or if something is misconfigured.
Could you provide SSH access to allow me to rsync the files that are currently present on the server?
On Fri, 17 Oct 2025, 18:37 Nick ODell, ***@***.***> wrote:
*nickodell* left a comment (scipy/docs.scipy.org#102)
<#102 (comment)>
Very happy to support a migration if this is requested. Happy to transfer
anything to anywhere and/or provide access to specific ressources.
I could look into a migration. I would need some information:
1. What is the CPU / RAM of the current set up? This would be useful
for figuring out what the cost of the new service would be. (It's easier to
ask for money if I know how much it will be. :) )
2. Can you provide an approximate number of page hits per 24 hours?
This would be useful for capacity planning. Right now it's unclear to me if
SciPy has out-grown its current server, or if something is misconfigured.
3. Could you provide SSH access to allow me to rsync the files that
are currently present on the server?
—
Reply to this email directly, view it on GitHub
<#102 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAEDL6FAPZ6FJVL2VKWTXF33YFVTTAVCNFSM6AAAAACI36QVTKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTIMJXGQZTEMJZHA>
.
You are receiving this because you were mentioned.Message ID: <scipy/docs.
***@***.***>
It is possible that some of the load is on fetching sphinxobj.inv (other projects building docs) which may not appear in page views depending on the method.
I don't think we should do a migration to a new server if we don't understand the problem. The current setup has its issues with deployment, but it works the way it works for a reason.
We had numpy docs on there as well until 2020. We migrated those to static deployment with GitHub Pages, using the https://github.com/numpy/doc repo. That is the alternative here I'd say. The issue with that is that the repo size grows with every minor release. Repo size can be up to 1 GB, for NumPy it's now 400 MB or so - see numpy/numpy#24736 for the latest discussion. The math is fairly simple (I can't find the previous issue about this so quickly, if anyone remembers please post a link). scipy docs are larger than numpy docs, so we can grow it for O(5 years) until we have to reshuffle the repo. Maybe that is the right option now, not sure.
This is the space we're using on the server now (6.7 GB):
We have a habit of uploading new docs for every bug fix release though, and that seems unnecessary - numpy doesn't do that (saves effort, and saves repo space).
Size per release:
$ cd /srv/docs_scipy_org/doc
$ ls -lh scipy-1.15.0/
total 71M
We could decide to store only docs the last 10 feature releases, reshuffle the repo when we hit 11 by dropping the oldest set of docs, and use GitHub Pages. That saves maintenance effort of the server and hitting a problem there once a year or so.
As for the server itself, Enthought - with @dpinte the point person, and one of their sysadmins helped out regularly in the past too - has been maintaining it for us for many years, which is much appreciated.
Server maintenance can be a painful chore, and when there's a problem it's usually not at a convenient time. We migrated https://scipy.org/ and the mailing list away from that server years ago; the docs are so bulky that it has been harder to make that decision. I think we can investigate the current issue, or choose static hosting, or perhaps put a CDN in front to patch over the problem if we can't find it. Migrating the current setup to a new server unchanged and taking on admin responsibility seems like the least attractive of all the options.
I don't think we should do a migration to a new server if we don't understand the problem. The current setup has its issues with deployment, but it works the way it works for a reason.
I agree - the best solution is to understand what's broken before any fixes are attempted.
We had numpy docs on there as well until 2020. We migrated those to static deployment with GitHub Pages, using the https://github.com/numpy/doc repo. That is the alternative here I'd say. The issue with that is that the repo size grows with every minor release. Repo size can be up to 1 GB, for NumPy it's now 400 MB or so - see numpy/numpy#24736 for the latest discussion. The math is fairly simple (I can't find the previous issue about this so quickly, if anyone remembers please post a link). scipy docs are larger than numpy docs, so we can grow it for O(5 years) until we have to reshuffle the repo. Maybe that is the right option now, not sure.
[...]
I think we can investigate the current issue, or choose static hosting, or perhaps put a CDN in front to patch over the problem if we can't find it. Migrating the current setup to a new server unchanged and taking on admin responsibility seems like the least attractive of all the options.
As an alternative to running a server, what would you think about hosting the docs on a Cloudflare R2 bucket with a custom domain? This is an S3-like object storage service. We could optionally put a Cloudflare CDN in front of that.
In terms of cost, this would cost $0.20 per month for storage, plus $0.36 per million HTTP requests. (See pricing.) The workflow for updating the docs would also change: instead of rsyncing to a cloud server, a release manager would use rclone or an S3 compatible client. (Alternatively, the update could be made through a GitHub action triggered by a push to a maintenance branch.)
This would have the advantage of providing unlimited storage space, so we wouldn't need to delete old docs. I see the old docs as being worth the expense.
Here's an update to the monitoring statistic that I posted last week. Latency has improved, but average uptime has dropped to 97% over the past 24 hours.
As for the server itself, Enthought - with @dpinte the point person, and one of their sysadmins helped out regularly in the past too - has been maintaining it for us for many years, which is much appreciated. Server maintenance can be a painful chore, and when there's a problem it's usually not at a convenient time.
I agree. I've worked on this project for less than a year. I am not familiar with the history of the project, and the hard work that many volunteers have put into running the services that the SciPy community depends on. I am sorry if I came across as unappreciative in my previous comment. I do appreciate the work that Enthought and @dpinte have put into this server.
We have a habit of uploading new docs for every bug fix release though, and that seems unnecessary - numpy doesn't do that (saves effort, and saves repo space).
Ok, noted. Maybe in the distant future when O(5 lines) change in the docs there will be a way to effectively update the docs in an efficient diff/git-like manner that doesn't require copying everything up again.
That said, a lot of us are probably more excited about writing a new algorithm for SciPy or similar, which is already hard to find time for even with partial funding, which probably gets back to the original comment about avoiding new server burdens.
I just ran all of those steps. A few tools aren't installed, and I don't have enough permissions to look at the access log. Hopefully there is something in there that is helpful:
As an alternative to running a server, what would you think about hosting the docs on a Cloudflare R2 bucket with a custom domain?
That seems like a good option potentially. It'd be nice to get a sense for the number of requests/month, since that'll be most of the cost. I just tried some free website analytics tools, but that gives very little info. Counting lines in /var/log/apache2/access.log seems easiest, maybe you can do that @dpinte?
Same for MNE-Python, today has been a particularly bad day for it (I would say over half my CI runs have died on the intersphinx resolution, but not all of them). Maybe knowing that is useful for log pulls or something!
The inventory file https://static.scipy.org/doc/scipy/objects.inv is equivalent to https://docs.scipy.org/doc/scipy/objects.inv. This file is automatically mirrored to static.scipy.org by a cron job on docs.scipy.org every hour, and it is served from more reliable infrastructure (R2.)
@leofang Thank you for the correction. I'd forgotten that this also determines the base URL of the documentation. Here's a fixed version that splits the INV file (which we mirror) from the HTML (which we do not.)
This will generate sphinx docs where the target is set to the stable docs at https://docs.scipy.org/doc/scipy/.
Activity
nickodell commentedon Oct 10, 2025
FWIW, I set up a monitoring system using UptimeRobot to get data on when the downtime / slowness was. It works by making an HTTP request to
https://docs.scipy.org/doc/scipy/building/index.html?uptime-robot-monitoring-contact-nickodell-for-issues=Noneand measuring how long it takes and whether it succeeds. Since I started tracking this data on Sep 29th, on average, it shows about 99% uptime and a mean of 2700 ms latency.I haven't been able to see any patterns in this graph so far.
tylerjereddy commentedon Oct 11, 2025
@dpinte You were able to help us last time from the Enthought side--are you still the correct contact for
docs.scipy.orghosting? It seems unusually slow lately per several user reports and we just wanted to check in on that.dpinte commentedon Oct 12, 2025
@tylerjereddy yes, I am. If I'm not responsive, please ping me by email. I'm flooded with github notifications ... I'm looking into the deployement and check what's going on.
dpinte commentedon Oct 12, 2025
Nothing special reported on the server. It's not a very beefy machine but system stats look good (cpu, memory and storage) Some maintenance will be required at some point though (OS upgrade).
Carreau commentedon Oct 17, 2025
Can you maybe front by cloudflare /fastly and cache ? Or similar ?
tupui commentedon Oct 17, 2025
+1 for the CDN. I am still puzzled on to why we do not have control on this website and the domain. I asked several years ago and it was not clear and people did not want me to help out so I let it roll sadly.
dpinte commentedon Oct 17, 2025
@Carreau that would definitely be a good idea. I can't promise anything in terms of my availabilities to do it.
@tupui the domain is managed by NumFocus and switched to CloudFare in 2023. And the website has never been transferred with the rest of the ressources. Very happy to support a migration if this is requested. Happy to transfer anything to anywhere and/or provide access to specific ressources.
nickodell commentedon Oct 17, 2025
I could look into a migration. I would need some information:
dpinte commentedon Oct 17, 2025
Carreau commentedon Oct 18, 2025
It is possible that some of the load is on fetching sphinxobj.inv (other projects building docs) which may not appear in page views depending on the method.
rgommers commentedon Oct 18, 2025
I don't think we should do a migration to a new server if we don't understand the problem. The current setup has its issues with deployment, but it works the way it works for a reason.
We had
numpydocs on there as well until 2020. We migrated those to static deployment with GitHub Pages, using the https://github.com/numpy/doc repo. That is the alternative here I'd say. The issue with that is that the repo size grows with every minor release. Repo size can be up to 1 GB, for NumPy it's now 400 MB or so - see numpy/numpy#24736 for the latest discussion. The math is fairly simple (I can't find the previous issue about this so quickly, if anyone remembers please post a link).scipydocs are larger thannumpydocs, so we can grow it for O(5 years) until we have to reshuffle the repo. Maybe that is the right option now, not sure.This is the space we're using on the server now (6.7 GB):
We have a habit of uploading new docs for every bug fix release though, and that seems unnecessary -
numpydoesn't do that (saves effort, and saves repo space).Size per release:
Number of releases we store:
We could decide to store only docs the last 10 feature releases, reshuffle the repo when we hit 11 by dropping the oldest set of docs, and use GitHub Pages. That saves maintenance effort of the server and hitting a problem there once a year or so.
As for the server itself, Enthought - with @dpinte the point person, and one of their sysadmins helped out regularly in the past too - has been maintaining it for us for many years, which is much appreciated.
Server maintenance can be a painful chore, and when there's a problem it's usually not at a convenient time. We migrated https://scipy.org/ and the mailing list away from that server years ago; the docs are so bulky that it has been harder to make that decision. I think we can investigate the current issue, or choose static hosting, or perhaps put a CDN in front to patch over the problem if we can't find it. Migrating the current setup to a new server unchanged and taking on admin responsibility seems like the least attractive of all the options.
rgommers commentedon Oct 18, 2025
I'm not seeing problems right now by the way. The site won't be as responsive as https://scipy.github.io/devdocs/ (because that's on GitHub pages rather than a lone Apache server), but it seems to function fine. If someone sees unusual latencies, it'd be great to post some stats here.
nickodell commentedon Oct 18, 2025
I agree - the best solution is to understand what's broken before any fixes are attempted.
I wrote a document on troubleshooting steps that I think might help. If there's anyone who has the time/access to try them, that would be great.
As an alternative to running a server, what would you think about hosting the docs on a Cloudflare R2 bucket with a custom domain? This is an S3-like object storage service. We could optionally put a Cloudflare CDN in front of that.
In terms of cost, this would cost $0.20 per month for storage, plus $0.36 per million HTTP requests. (See pricing.) The workflow for updating the docs would also change: instead of rsyncing to a cloud server, a release manager would use rclone or an S3 compatible client. (Alternatively, the update could be made through a GitHub action triggered by a push to a maintenance branch.)
This would have the advantage of providing unlimited storage space, so we wouldn't need to delete old docs. I see the old docs as being worth the expense.
Here's an update to the monitoring statistic that I posted last week. Latency has improved, but average uptime has dropped to 97% over the past 24 hours.
If anyone is interested in live updates to this monitor, I have added a status page here: https://stats.uptimerobot.com/oKUpNEbIwJ
I agree. I've worked on this project for less than a year. I am not familiar with the history of the project, and the hard work that many volunteers have put into running the services that the SciPy community depends on. I am sorry if I came across as unappreciative in my previous comment. I do appreciate the work that Enthought and @dpinte have put into this server.
tylerjereddy commentedon Oct 18, 2025
Ok, noted. Maybe in the distant future when O(5 lines) change in the docs there will be a way to effectively update the docs in an efficient diff/git-like manner that doesn't require copying everything up again.
That said, a lot of us are probably more excited about writing a new algorithm for SciPy or similar, which is already hard to find time for even with partial funding, which probably gets back to the original comment about avoiding new server burdens.
rgommers commentedon Oct 20, 2025
You didn't! Thanks for digging into the current issue so thoroughly 🙏🏼
I just ran all of those steps. A few tools aren't installed, and I don't have enough permissions to look at the access log. Hopefully there is something in there that is helpful:
Details
That seems like a good option potentially. It'd be nice to get a sense for the number of requests/month, since that'll be most of the cost. I just tried some free website analytics tools, but that gives very little info. Counting lines in
/var/log/apache2/access.logseems easiest, maybe you can do that @dpinte?126 remaining items
larsoner commentedon Sep 3, 2026
Same for MNE-Python, today has been a particularly bad day for it (I would say over half my CI runs have died on the intersphinx resolution, but not all of them). Maybe knowing that is useful for log pulls or something!
tylerjereddy commentedon Sep 3, 2026
@nickodell @jychuah suggestions?
nickodell commentedon Sep 4, 2026
The major suggestion I would make to fix builds is to get the file from
static.scipy.org, our static file hosting.Add this to your sphinx configuration file:
The inventory file
https://static.scipy.org/doc/scipy/objects.invis equivalent tohttps://docs.scipy.org/doc/scipy/objects.inv. This file is automatically mirrored to static.scipy.org by a cron job on docs.scipy.org every hour, and it is served from more reliable infrastructure (R2.)larsoner commentedon Sep 4, 2026
Okay opened Quansight-Labs/intersphinx_registry#123 for those of us using
intersphinx_registry!docs: fetch the scipy intersphinx inventory from a CDN-backed mirror
docs: fetch the scipy intersphinx inventory from a CDN-backed mirror
docs: fetch the scipy intersphinx inventory from a CDN-backed mirror (#…
docs: fetch the scipy intersphinx inventory from a CDN-backed mirror (#…
docs: use static.scipy.org mirror for scipy intersphinx
leofang commentedon Sep 4, 2026
@nickodell unfortunately it doesn't work, because while the docs build fine (
objects.invis served), the generated links all get wrong and point to https://static.scipy.org/doc/scipy/. As an example, see the "See also" link in this CuPy doc that is built with this suggestion: https://docs.cupy.dev/en/latest/reference/generated/cupyx.scipy.interpolate.BarycentricInterpolator.html#cupyx.scipy.interpolate.BarycentricInterpolator.cc @rgommers @seberg for vis
leofang commentedon Sep 4, 2026
It seems Qiskit's tweaks would fix the base url issue, we're looking into it: Qiskit/qiskit-fermions#319
nickodell commentedon Sep 4, 2026
@leofang Thank you for the correction. I'd forgotten that this also determines the base URL of the documentation. Here's a fixed version that splits the INV file (which we mirror) from the HTML (which we do not.)
This will generate sphinx docs where the target is set to the stable docs at
https://docs.scipy.org/doc/scipy/.cc @larsoner for intersphinx
docs: point scipy intersphinx target back at docs.scipy.org
Carreau commentedon Sep 5, 2026
Fixed, merged and published.
lucascolley commentedon Sep 5, 2026
should we be advertising this fix in a better place than buried in this thread?