Repository navigation
Description
Background
Pangeo Forge recipes operate on "inputs", string URLs that point to individual source files. These inputs are generated by the recipe FilePattern object.
Currently, Pangeo Forge relies 100% on fsspec to download these files. This is great because fsspec has a ton of implementations for just about every file transfer / storage protocol in the world.
However, fsspec does not support Globus...
Globus provides a secure, unified interface to your research data. Use Globus to 'fire and forget' high-performance data transfers between systems within and across organizations.
...and it probably never will. Globus is a system for transferring whole files and nothing else, so wrapping it in a filesystem-like abstraction probably just doesn't make sense. (@martindurant correct me if I'm wrong.) However, so much scientific data is accessible over Globus. Our impact will be much greater if we can interoperate with Globus somehow.
However, using Globus will likely require us to refactor our approach to file transfers; Globus operates at a higher level, so much of the logic we have for opening, streaming, and saving files will be unnecessary (also discussed in #95).
Questions about Globus
To move forward, we need to answer some basic questions about Globus. I'm going to tag @rpwagner, who has been very helpful in this area in the past and see if he bites.
- Is there such thing as a "global namespace" for the entire Globus federation? Can I refer to a specific file at a specific endpoint via a string url, e.g
globus://my-globus-endpoint.columbia.edu/my/specific_file.nc? - If we wanted to use Globus to transfer data from user-specified HPC systems to our cloud buckets via automated batch jobs, how would we handle authentication? Does Globus support "service accounts," long-lived tokens, etc,...any way of storing credentials persistently for months / years without requiring interactive login? (Obviously such credentials would have to be managed securely, but that's the case for any credentials.)
- Would we need a paid Globus subscription to transfer data to buckets in S3 / GCS?
- Does Globus work with OSN?
- What does the python code look like to create and wait on a file transfer using Globus?
- Is it possible to rename the file as it is transferred to the target endpoint?
Another useful reference is the Parsl workflow tool, which already supports Globus file transfer. There is lots to learn from reading their docs and looking at their code.
Next Steps
Depending on the answer to those questions, here are some specific tasks we could take towards implementing Globus support in Pangeo Forge.
- Refactor file downloading / caching (e.g.
cache_inputsstep) out ofXarrayZarrRecipeand into a standalone mixin that can be added to any Recipe class (e.g.FsspecFileCacherMixin).- Create unit tests for this class.
- Make the current fsspec-based file cacher an implementation of a more generic base class for file caching (e.g.
FileCacherMixin) - Implement the
GlobusFileCacherMixin
Activity
rabernat commentedon Mar 9, 2022
@MaxGrover recently pointed me to the Foundry project, from which I was able to track down this useful piece of code for transferring globus files.
https://github.com/materials-data-facility/toolbox/blob/master/mdf_toolbox/globus_transfer.py
ricardobarroslourenco commentedon Mar 9, 2022
Just a comment on :
Perhaps it would be interesting to integrate with Globus ID . Perhaps @rpwagner may refer to a more detailed, official, documentation.
rpwagner commentedon Mar 9, 2022
My apologies, I meant to comment on this a while ago. The short answer to most of the questions about "can we do this?" is yes, you can. Then there are the details.
For the specific example of having services do automated transfers, it's not about the identity provider, like Globus ID, but rather having a client (service, script, etc.) use its own credentials. This can be combined with permissions or roles on a guest collection to let the client (script, portal, JupyterHub instance, etc.) do things like read and write data, or even manage permissions for users.
The practical aspect of implementation may be around which HPC centers allow guest collections. The ALCF model would enable this, as do several other XSEDE and DOE centers, and many campus research IT environments.
I would also suggest we bring some Globus folks into this thread, namely @lliming and @kylechard. (I'm back at UCSD.) Amongst us, we've seen a few potential architectures. Most of my recent experience is more from the genomics and bioinformatics side, but I'm glad to share them.
rabernat commentedon Mar 9, 2022
Thanks a lot for the reply @rpwagner. I appreciate your reply.
If anyone you tagged can provide specific replies to my "Questions about Globus" in #222 (comment) I will literally buy you a 🍺 or the beverage of your choice. 😉
ricardobarroslourenco commentedon Mar 10, 2022
@rabernat it would be great to have @kylechard opinion on this (hi Kyle!). If such a dialog between STAC endpoints (even if it is just a proxy) and Globus happens will be very interesting (but I don't know if is feasible or even desirable from a systems perspective).
I was at a talk by @ianfoster today, and he mentioned that they are moving a huge amount of data (If I'm not wrong, CMIP and MODIS), to be stored on Argonne ALCF, and it seems that they will be discoverable with Globus. So I think that this topic may be interesting for Globus folks too.
ianfoster commentedon Mar 11, 2022
It is good to see this thread. @rabernat alerted me to it almost 2 months ago, and I am ashamed to say that I let myself be busy with other things. We will get answers to the questions not already answered, and to others as they arise. About @ricardobarroslourenco's data question: I provide some context in these slides, https://www.slideshare.net/ianfoster/big-data-big-computing-ai-and-environmental-science. We have replicated 700 TB of MODIS data to Argonne (slide 26), and are in the process of replicating 8.6 PB of ESGF CMIP data (slide 28). The idea is that they can then be computed on at high speed, as e.g. Takuya Kurihana is doing (https://ieeexplore.ieee.org/iel7/36/9633014/09497325.pdf).
rabernat commentedon Mar 11, 2022
Thanks for everyone who has chimed in here. Encouraged by @blaiszik, I am trying to organize an informal cross-project meeting between Pangeo and Globus, not specifically about this issue but about how our projects might collaborate more broadly. See details here - https://discourse.pangeo.io/t/pangeo-globus-labs-meeting-and-discussion/2308 - if you'd like to join!
kylechard commentedon Mar 11, 2022
@rabernat certainly interested in joining the call and hearing more about your work. Here are some answers to the questions you posed above.
All Globus endpoints have a unique ID and thus files can be referenced directly. In Parsl and other systems we compose URLs like your example (globus://..).
When using the latest Globus Connect version 5 endpoints, data access is via what we call "collections". Each collection has an associated DNS name, such that you can refer to collections directly. By default collections are assigned DNS names in the data.globus.org subdomain, but they can also be customized by deployments.
v5 also supports HTTP/S access to data, while enforcing a common security policy across the access mechanism. For example, here is a publicly accessible file: https://a4969.36fe.dn.glob.us/public/read-only/logo.png. For non-public data, users must authenticate (and be authorized to access the data) before it can be downloaded via HTTPS.
Yes to all of the above: transfers can be automated, Globus supports long running transfers, and you can retrieve long-lived refresh tokens.
There are two authentication layers with Globus. a) Authenticating to the Globus service so it can manage and track you transfers; and b) authenticating to the storage system with the needed credentials and meeting their policy.
For (a), for complete automation, application credentials can be used. This leverages OAuth’s model of an application obtaining credentials with an authorization server. These credentials grant the application an identity in the Globus ecosystem, and permissions for data and other roles can be granted.
For data access, (b) the level of automation that can be achieved is determined by policy on the storage system. Globus supports shared endpoints (or guest collections in v5) that allow full automation of transfers (like the example Ian mentioned above for replicating the ESGF data).
Transferring to/from cloud storage requires a Globus subscription. 150+ institutions are existing subscribers, so you should check if you already have access to a subscription via your institution.
Yes. Globus has connectors for Ceph/S3 compatible object stores. Here is an example search portal that uses Globus to access data on OSN https://osn.globusdemo.org/.
Documentation about the Globus Python SDK is here: https://github.com/globus/globus-sdk-python.
Typically users will a submit a transfer and check the status asynchronously (rather than waiting).
There are some good examples in the tutorial notebooks: https://github.com/globus/globus-jupyter-notebooks/blob/master/Transfer_API_Exercises.ipynb and several automation examples that might match your use case : https://github.com/globus/automation-examples
Yes, you can specify both the source and destination name in a transfer.
rabernat commentedon Mar 28, 2022
To test things out, I have created a sample public collection here:
https://app.globus.org/file-manager?origin_id=f0267e3e-f246-449c-8303-037a3b8bd233&origin_path=%2F
How do I access this data over HTTPS? Can someone point me to the relevant documentation for the HTTPS feature?
ricardobarroslourenco commentedon Mar 28, 2022
@rabernat take a look here: https://docs.globus.org/globus-connect-server/v5/https-access-collections/
rabernat commentedon Mar 28, 2022
@ricardobarroslourenco - thanks for the tip. I attempted to follow those directions and found some inconsistencies. But I was able to move forward with a little bit of trial and error. I can successfully access my data via HTTPS! 🎉
Trying to run the example from the globus docs
I installed the globus cli and logged in
globus loginusing my institutional credentials. Then I tried to run the example CLI code from the docsSo I tried to follow the advice
So I tried
globus login --gcs 0b3dfa5b-5572-41e2-a61e-dee56a9a87eaand used the browser to authenticate again. Then I tried runningMy conclusion is that the example in the documentation is either broken or points to data that is not part of a public collection. I am using the latest globus CLI version (3.4.0).
I believe the globus docs need an update. Is there a place I can make a PR?
Try to make the command work for my collection
At this point, I hypothesize that I don't have the proper credentials to access the data from the example, so I am trying again with my public guest collection linked above (https://app.globus.org/file-manager?origin_id=f0267e3e-f246-449c-8303-037a3b8bd233&origin_path=%2F).
Great! 🎉 I found an
HTTPS_URLTry to figure out the correct URL for HTTPS
I now assumed that the directory would be accessible via this HTTPS endpoint. And it works!
I am also able to load the data via fsspec, xarray, and zarr!
And plot it using lazy loading
This is a huge milestone. Thanks so much for your support.
rabernat commentedon Mar 28, 2022
One final subtle issue i have noticed: subdirectories in this HTTPS endpoint do not have the correct content-type
httpd/unix-directory. Compare the rootto the subdirectory
This will cause some issues for our downstream tooling. I would consider it a bug in the globus HTTPS implementation. Where would be the best place to report this bug? Thanks.
ricardobarroslourenco commentedon Mar 28, 2022
Happy to help @rabernat ! @kylechard any comments on how Ryan should proceed?
sirosen commentedon Mar 28, 2022
Yes, looking at this, and speaking as one of the maintainers of the globus-cli, we need to fix this doc.
The docs.globus.org source is private, but I'll make sure we take care of this. If you're looking for a place to file upstream issues, we're happy to take any bug reports either via [email protected] or on the CLI issue tracker if they're globus-cli related.
This doc was correct until the release of globus-cli v3.0.0. In 3.0, we added features for handling GCS version 5 which made the command inaccurate (it should have been updated to
globus collection show ...), and we didn't catch it.rabernat commentedon Mar 28, 2022
Hi @sirosen - thanks for the response! Since you have noted the issue already, I'm not going to file a ticket for the doc fixes.
However, I would be very interested in opening one for the
Content-Type: httpd/unix-directoryHTTPS issue above. Would [email protected] be the right venue for that?ianfoster commentedon Mar 28, 2022
kylechard commentedon Mar 28, 2022
Hi @rabernat I checked with the team on the content-type problem. At present directory listing is not a supported feature via the HTTPS interface. Instead you would need to list the directory via the Globus API.
rabernat commentedon Mar 28, 2022
Thanks Kyle. I don't want to actually list the directory. I just want Globus to return a
Content-Type: httpd/unix-directoryheader when I call GET on a directory. It's fine if the content is empty. Globus is already doing this for the top-level URL of the collection: https://g-c70289.0da32.08cc.data.globus.org/. Just not for the subdirectory https://g-c70289.0da32.08cc.data.globus.org/pyqg_ensemble/001.zarr/. Do you think that is feasible?rabernat commentedon Mar 28, 2022
Actually, Kyle, perhaps this is not such a big deal, because changing the headers would not actually resolve zarr-developers/zarr-python#993.