Skip to content

Commit c4f2487

Browse files
authored
Merge pull request #4 from shikharsg/master
sync with master
2 parents 0013f95 + b70c348 commit c4f2487

23 files changed

Lines changed: 469 additions & 131 deletions

‎.pyup.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# pyup.io config file
2+
# see https://pyup.io/docs/configuration/ for all available options
3+
4+
schedule: every month
5+
6+
requirements:
7+
- requirements.txt:
8+
pin: False
9+
update: False
10+
- requirements_test.txt:
11+
pin: False
12+
update: False
13+
- requirements_rtfd.txt:
14+
pin: False
15+
update: False
16+
- requirements_dev.txt:
17+
pin: True
18+
update: all
19+
- requirements_dev_npy.txt:
20+
pin: True
21+
update: all
22+
- requirements_dev_optional.txt:
23+
pin: True
24+
update: all

‎appveyor.yml‎

Lines changed: 6 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -12,49 +12,29 @@ environment:
1212

1313
matrix:
1414

15-
- PYTHON: "C:\\Python27"
16-
PYTHON_VERSION: "2.7"
17-
NUMPY_VERSION: "1.15.2"
18-
1915
- PYTHON: "C:\\Python27-x64"
2016
PYTHON_VERSION: "2.7"
21-
NUMPY_VERSION: "1.15.2"
2217
DISTUTILS_USE_SDK: "1"
2318

24-
- PYTHON: "C:\\Python35"
25-
PYTHON_VERSION: "3.5"
26-
NUMPY_VERSION: "1.15.2"
27-
2819
- PYTHON: "C:\\Python35-x64"
2920
PYTHON_VERSION: "3.5"
30-
NUMPY_VERSION: "1.15.2"
31-
32-
- PYTHON: "C:\\Python36"
33-
PYTHON_VERSION: "3.6"
34-
NUMPY_VERSION: "1.15.2"
3521

3622
- PYTHON: "C:\\Python36-x64"
3723
PYTHON_VERSION: "3.6"
38-
NUMPY_VERSION: "1.15.2"
39-
40-
- PYTHON: "C:\\Python37"
41-
PYTHON_VERSION: "3.7"
42-
NUMPY_VERSION: "1.15.2"
4324

4425
- PYTHON: "C:\\Python37-x64"
4526
PYTHON_VERSION: "3.7"
46-
NUMPY_VERSION: "1.15.2"
4727

4828
install:
4929
- "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
30+
- "%CMD_IN_ENV% python -m pip install -U pip setuptools wheel"
31+
- "%CMD_IN_ENV% python -m pip install -rrequirements_test.txt"
32+
- "%CMD_IN_ENV% python -m pip install -rrequirements_dev_npy.txt"
33+
- "%CMD_IN_ENV% python -m pip install --no-binary=numcodecs -rrequirements_dev.txt"
34+
- "%CMD_IN_ENV% python setup.py install"
35+
- "%CMD_IN_ENV% python -m pip freeze"
5036

5137
build: off
5238

5339
test_script:
54-
- "%CMD_IN_ENV% python -m pip install -U pip setuptools wheel"
55-
- "%CMD_IN_ENV% python -m pip install numpy==%NUMPY_VERSION%"
56-
- "%CMD_IN_ENV% python -m pip install cython==0.29"
57-
- "%CMD_IN_ENV% python -m pip install -v --no-binary=numcodecs numcodecs==0.5.5"
58-
- "%CMD_IN_ENV% python -m pip install -rrequirements_dev.txt"
59-
- "%CMD_IN_ENV% python setup.py install"
6040
- "%CMD_IN_ENV% python -m pytest -v --pyargs zarr"

‎docs/api/convenience.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,5 @@ Convenience functions (``zarr.convenience``)
1010
.. autofunction:: copy_all
1111
.. autofunction:: copy_store
1212
.. autofunction:: tree
13+
.. autofunction:: consolidate_metadata
14+
.. autofunction:: open_consolidated

‎docs/api/storage.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,8 @@ Storage (``zarr.storage``)
2727
.. automethod:: invalidate_values
2828
.. automethod:: invalidate_keys
2929

30+
.. autoclass:: ConsolidatedMetadataStore
31+
3032
.. autoclass:: LRUChunkCache
3133

3234
.. automethod:: invalidate

‎docs/release.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,13 @@ Release notes
99
Enhancements
1010
~~~~~~~~~~~~
1111

12+
* Add "consolidated" metadata as an experimental feature: use
13+
:func:`zarr.convenience.consolidate_metadata` to copy all metadata from the various
14+
metadata keys within a dataset hierarchy under a single key, and
15+
:func:`zarr.convenience.open_consolidated` to use this single key. This can greatly
16+
cut down the number of calls to the storage backend, and so remove a lot of overhead
17+
for reading remote data. By :user:`Martin Durant <martindurant>`, :issue:`268`.
18+
1219
* Support has been added for structured arrays with sub-array shape and/or nested fields. By
1320
:user:`Tarik Onalan <onalant>`, :issue:`111`, :issue:`296`.
1421

‎docs/tutorial.rst‎

Lines changed: 43 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -778,9 +778,11 @@ chunk size, which will reduce the number of chunks and thus reduce the number of
778778
round-trips required to retrieve data for an array (and thus reduce the impact of network
779779
latency). Another option is to try to increase the compression ratio by changing
780780
compression options or trying a different compressor (which will reduce the impact of
781-
limited network bandwidth). As of version 2.2, Zarr also provides the
782-
:class:`zarr.storage.LRUStoreCache` which can be used to implement a local in-memory cache
783-
layer over a remote store. E.g.::
781+
limited network bandwidth).
782+
783+
As of version 2.2, Zarr also provides the :class:`zarr.storage.LRUStoreCache`
784+
which can be used to implement a local in-memory cache layer over a remote
785+
store. E.g.::
784786

785787
>>> s3 = s3fs.S3FileSystem(anon=True, client_kwargs=dict(region_name='eu-west-2'))
786788
>>> store = s3fs.S3Map(root='zarr-demo/store', s3=s3, check=False)
@@ -828,6 +830,44 @@ interface to the storage.
828830

829831
.. _tutorial_copy:
830832

833+
Consolidating metadata
834+
~~~~~~~~~~~~~~~~~~~~~~
835+
836+
(This is an experimental feature.)
837+
838+
Since there is a significant overhead for every connection to a cloud object
839+
store such as S3, the pattern described in the previous section may incur
840+
significant latency while scanning the metadata of the dataset hierarchy, even
841+
though each individual metadata object is small. For cases such as these, once
842+
the data are static and can be regarded as read-only, at least for the
843+
metadata/structure of the dataset hierarchy, the many metadata objects can be
844+
consolidated into a single one via
845+
:func:`zarr.convenience.consolidate_metadata`. Doing this can greatly increase
846+
the speed of reading the dataset metadata, e.g.::
847+
848+
>>> zarr.consolidate_metadata(store) # doctest: +SKIP
849+
850+
This creates a special key with a copy of all of the metadata from all of the
851+
metadata objects in the store.
852+
853+
Later, to open a Zarr store with consolidated metadata, use
854+
:func:`zarr.convenience.open_consolidated`, e.g.::
855+
856+
>>> root = zarr.open_consolidated(store) # doctest: +SKIP
857+
858+
This uses the special key to read all of the metadata in a single call to the
859+
backend storage.
860+
861+
Note that, the hierarchy could still be opened in the normal way and altered,
862+
causing the consolidated metadata to become out of sync with the real state of
863+
the dataset hierarchy. In this case,
864+
:func:`zarr.convenience.consolidate_metadata` would need to be called again.
865+
866+
To protect against consolidated metadata accidentally getting out of sync, the
867+
root group returned by :func:`zarr.convenience.open_consolidated` is read-only
868+
for the metadata, meaning that no new groups or arrays can be created, and
869+
arrays cannot be resized. However, data values with arrays can still be updated.
870+
831871
Copying/migrating data
832872
----------------------
833873

‎requirements.txt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
asciitree
2-
pytest
3-
numpy
42
fasteners
53
numcodecs
4+
numpy
5+
pytest

‎requirements_dev.txt‎

Lines changed: 0 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -1,60 +1,3 @@
11
asciitree==0.3.3
2-
asn1crypto==0.24.0
3-
atomicwrites==1.2.1
4-
attrs==18.2.0
5-
bleach==3.0.2
6-
boto3==1.9.26
7-
botocore==1.12.26
8-
certifi==2018.10.15
9-
cffi==1.11.5
10-
chardet==3.0.4
11-
cmarkgfm==0.4.2
12-
configparser==3.5.0
13-
coverage==4.5.1
14-
coveralls==1.5.1
15-
cryptography==2.3.1
16-
Cython==0.29
17-
docopt==0.6.2
18-
docutils==0.14
19-
enum34==1.1.6
202
fasteners==0.14.1
21-
filelock==3.0.9
22-
flake8==3.5.0
23-
funcsigs==1.0.2
24-
future==0.16.0
25-
h5py==2.8.0
26-
idna==2.7
27-
ipaddress==1.0.22
28-
jmespath==0.9.3
29-
mccabe==0.6.1
30-
monotonic==1.5
31-
more-itertools==4.3.0
32-
msgpack-python==0.5.6
333
numcodecs==0.5.5
34-
pathlib2==2.3.2
35-
pkginfo==1.4.2
36-
pluggy==0.8.0
37-
py==1.7.0
38-
pycodestyle==2.3.1
39-
pycparser==2.19
40-
pyflakes==1.6.0
41-
Pygments==2.2.0
42-
pyOpenSSL==18.0.0
43-
pytest==3.9.1
44-
pytest-cov==2.6.0
45-
python-dateutil==2.7.3
46-
readme-renderer==22.0
47-
requests==2.19.1
48-
requests-toolbelt==0.8.0
49-
s3fs==0.1.6
50-
s3transfer==0.1.13
51-
scandir==1.9.0
52-
six==1.11.0
53-
toml==0.10.0
54-
tox==3.5.2
55-
tox-travis==0.11
56-
tqdm==4.27.0
57-
twine==1.12.1
58-
urllib3==1.23
59-
virtualenv==16.0.0
60-
webencodings==0.5.1

‎requirements_dev_npy.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# Break this out into a separate file to allow testing against
2+
# different versions of numpy. This file should pin to the latest
3+
# numpy version.
4+
numpy==1.15.4

‎requirements_dev_optional.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
1+
# These packages are currently not available on Windows.
12
bsddb3==6.2.6
23
lmdb==0.94

0 commit comments

Comments
 (0)