You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 137067f
Browse filesBrowse the repository at this point in the historyBrowse files
Provide documentation of what path interpolation parameters you can use in Globus download methods (#207)
* Typos
* Add basic description of path keys in data download section
* Add page on path interpolation keys with list of available ones
* Move description and table to net file
* Add changelog
* Replace hard-coded table in docs with function to print the table
* Add function to path-sanitize keys and use in FileManager.download
* Apply suggestions from code review
* Use astropy table and copy key format from sunpy
* Factor out inventory keys as argument to format function
* Test path formatting with defined input and output
Co-authored-by: Stuart Mumford <[email protected]>
Copy file name to clipboardExpand all lines: docs/guide/net.rst
+25Lines changed: 25 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -124,6 +124,9 @@ Once we have loaded the dataset, if we wish to transfer all the FITS files a sin
124
124
If this is the first time you have run this method, or your authentication has expired, a login page should open in your webbrowser to authenticate with the Globus system.
125
125
By default this call will download all the FITS files to the current value of ``ds.files.basepath``, i.e. by default in the same directory as the loaded ASDF file.
126
126
You can override this behaviour by using the ``path=`` keyword argument.
127
+
The path argument can contain keys which will be replaced by the corresponding values in the dataset's metadata.
128
+
For example, setting `path="~/data/dkist/{instrument}/"` will download all files and save them in separate folders named for the instrument.
129
+
A full list of the available keys can be found in :ref:`interpolation-keys`.
127
130
128
131
The real power of using ``download()`` however, is that you don't have to transfer the FITS files for the frames you do not wish to study (yet).
129
132
For instance, imagine the situation where you wish to first inspect the Stokes I profile to asses the viability of the data for your analysis, using this download method you can do this and your transfer will take about a quarter of the time.
@@ -183,3 +186,25 @@ Finally, if you know the dataset ID of a dataset you wish to download, you can j
183
186
.. code-block:: python
184
187
185
188
transfer_complete_datasets("AAAAA")
189
+
190
+
.. _interpolation-keys:
191
+
192
+
Path interpolation keys
193
+
-----------------------
194
+
195
+
When downloading DKIST data with ``ds.files.download()`` or ``Fido.fetch(), the ``path=`` keyword argument can be used to specify the target folder for the download.
196
+
This path can include keys corresponding to metadata entries, and those values are then used to complete the download path.
197
+
For example, to download a dataset into its own folder named with the dataset ID, with separate subfolders for each instrument in the dataset, you could set ``path="~/data/dkist/{dataset_id}/{instrument}/"``.
198
+
This would take the values for the dataset ID and instrument name from either the ASDF file or the search results.
199
+
200
+
Here is a full list of the metadata keywords available for this purpose and their corresponding path interpolation keys:
201
+
202
+
.. generate:: html
203
+
204
+
from dkist.utils.inventory import _path_format_table
205
+
print(_path_format_table())
206
+
207
+
The complete list of keys can be accessed using the function ``dkist.utils.inventory.path_format_keys()``::
208
+
209
+
>>> from dkist.utils.inventory import path_format_keys
0 commit comments