Skip to content

Update downloading from MinIO to accommodate for new structure #1304

@PGijsbers

Description

@PGijsbers

Previously, each dataset had their own bucket: https://openml1.win.tue.nl/datasets61/dataset_61.pq

But we were advised to reduce the amount of buckets and
favor hosting many objects in hierarchical structure, so
we now have instead some prefixes to divide up the
dataset objects into separate subdirectories: https://openml1.win.tue.nl/datasets/0000/0061/dataset_61.pq

I started work on it here: https://github.com/openml/openml-python/tree/fix/new_minio
It works but is ugly and I didn't run any tests.
Just trying to get it to work for now so Taniya and Prabhant can continue on with their deep learning integration, but we need to integrate this in the next release.

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions