@mfeurer since you mentioned removing or fixing split pickling, I only commented out the code as a simple solution. I can remove it completely if you are ok with it. @janvanrijn we are using double quotes for docstrings.

I think removing it is fine. Reading those files isn't as costly as reading in the data.

@mfeurer sure, but we will not have a way to disable loading from cache.

I don't think we have this for anything else. As a split should never change I don't see a reason to keep this functionality.

@mfeurer while working in the java repo, there was a case when the server was having problems and it was returning a bad value for a certain request. The value was cached and it would fail a procedure we were running as it would run on the cached object. In this case the flag was necessary (apart from the fact that the files could be removed locally). However given that we validate the object, this should not happen.

Yes, we have this piece of code for downloading the task:

        try:
            task = _get_task_description(task_id)
            dataset = get_dataset(task.dataset_id)
            class_labels = dataset.retrieve_class_labels(task.target_name)
            task.class_labels = class_labels
            task.download_split()
        except Exception as e:
            openml.utils._remove_cache_dir_for_id(TASKS_CACHE_DIR_NAME, tid_cache_dir)
            raise e

One last thing, could you please remove the try/except around the pickle file loading. We don't react on any exception, so the try/except is actually useless.