Co-authored-by: adel <[email protected]>

* Cache _list_all we don't need the latest list

The test does not require the list of flows to be updated, to a single
cached version will do fine (this call otherwise would take ~40
seconds).

* Reduce the amount of verified runs

Downloading a run takes a non-significant amount of time (est. 300ms on
my current setup). It is unnecessary to compare against all >=100 runs,
while a handful should do fine (perhaps even just one should do).

* Increase the batch size to avoid more than 2 pages

The batch size required in some pages over 40 pages to be loaded, which
increased the workload unnecessarily. These changing preserve pagination
tests while lowering the amount of round trips required.

* Mark as test_get_run_trace as skip

Since it is already covered by test_run_and_upload_randomsearch.

* Filter on dataset id serverside

Speeds up ~25x, and reduces network traffic.

* Reduce the amount of pages loaded

Loading a page takes ~600ms. I don't think testing with 3 pages is any
worse than 10. I also think this is an ideal candidate of test that
could be split up into (1) testing the url is generated correctly, (2)
testing a pre-cached result is parsed correctly and (3) testing the url
gives the expected response (the actual integration test).

* Simplify model tested in swapped parameter test

If the test is that swapped parameters work, we don't need a complicated pipeline or dataset.

* Add a cli flag to toggle short/long scenarios

Some tests support both, by checking e.g. only a few runs vs all runs.

* Skip time measurement on any Windows machine

* Invoke the --long versions on the COVERAGE job

* Add long/short versions for some long tests

* Check the trace can be retrieved individually

To cover for the skipping of test_get_run_trace

* Remove old test

* Use patch isolate list_all caching to one test

* Fix decorator call

* Initial changes

* Deleting TODO that will addressed by #968

* [skip ci] removing redundant imports

* [skip ci] Simplifying flow to generate prediction probablities

* Triggering unit tests

* Fixing mypy and flake issues

* [skip ci] Replacing HistGradientBoostingClassifier

* Simplifying examples

* Minor typo fix

* Improving the performance of check_datasets_active, modifying unit test

* Adding changes to doc/progress

* Addressing Pieter's comments

Co-authored-by: PGijsbers <[email protected]>

* Add CI through Github Actions

Initial attempt, not convinced this works.

* Add scikit-learn to matrix

* Fix syntax

* Complete job matrix

* Turn off fail-fast behavior

And continues to run other jobs even if one already failed. Kind of needed with our current flakey setup.

* Add conditional scipy install for sklearn 0.18

* Move scipy requirement to correct scikit-learn

* Throttle parallel jobs to avoid server issues

* Remove travis jobs covered by Github Actions

Currently TEST_DIST check is no longer executed.
I will add it to one of the Github Action workflows.

#978)

* add validation for ignore_attributes and default_target_attribute at craete_dataset

* update naming convetions and adding type hints. using pytest parametrize with attribute validation

* formating long lines and update types hint for return values

* update test_attribute_validations to use pytest.mark.parametrize

* add more tests for different input types for attribute validation

* update formatting

Co-authored-by: adel <[email protected]>

…added unit test, fixed typo (#983)

* retry on database error to reduce number of test failures

* take into account Pieter's suggestions, unfortunately, some changes by black, too

* Add pre-commit workflow, rename test workflow

* Fix formatting

* Add a dist check workflow

Checks the dist can be built and installed and the long description is
rendered correctly on pypi

* Determine $last_dist at each step

* Remove duplicate "dist/"

* Downgrade to Python 3.8 since no wheel for 3.9

Not all wheels are available for Py3.9 yet, so CI stalls on installing
some packages (see also
docker-library/python#540)

* Add PEP 561 compliance check

* Add workflow to build and deploy docs

* Add code coverage reporting to py3.8/sk0.23.1

* Improve naming, make it consistent

Step names start with a capital. Clarified names to indicate what is
done.

* Check no files left behind after test

* Avoid upload coverage on non-coverage job

* Fix the conditional for codecov upload

* Remove Travis files

* Add optional dependencies for building docs

PGijsbers: I had forgotten to make sure @a-moadel  had included his updates to `progress.rst`. I am merging this despite failures because it only updates the progress file which can not be the cause of the failures.

Co-authored-by: adel <[email protected]>

* docs: update README.md [skip ci]

* docs: create .all-contributorsrc [skip ci]

* docs: update README.md [skip ci]

* docs: update .all-contributorsrc [skip ci]

Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>

* docs: update README.md [skip ci]

* docs: update .all-contributorsrc [skip ci]

* docs: update README.md [skip ci]

* docs: update .all-contributorsrc [skip ci]

Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>

* randomize test order

* reduce noise in the output to better see the issues

* deprecate format argument to OpenMLDataset

* fix file upload

* further reduce warnings

* fix test which failed due to deleting a dataset on the test server

* re-add test randomization (due to rebase)

* try if random test order causes all problems by removing it

* improve lbfgs test

* distribute tests better

* reduce randomness in lbfgs test

* add requested commits

* sklearn model fit check

* Reordering warning

* Update openml/extensions/sklearn/extension.py

Co-authored-by: PGijsbers <[email protected]>

* Adding function to ext for checking model fit

* Removing junk file

* Fixing sklearn version compatibility issue

Co-authored-by: Matthias Feurer <[email protected]>
Co-authored-by: PGijsbers <[email protected]>

* cache dataset features and qualities as pickle

* incorporate feedback

* Fix unit tests

* black, pep8 etc

* Remove unused imports

Co-authored-by: PGijsbers <[email protected]>

* Remove extra period
* Refactor copyright message generation
* Remove other u-strings

* Making some unit tests work

* Waiting for dataset to be processed

* Minor test collection fix

* Template to handle missing tasks

* Accounting for more missing tasks:

* Fixing some more unit tests

* Simplifying check_task_existence

* black changes

* Minor formatting

* Handling task exists check

* Testing edited check task func

* Flake fix

* More retries on connection error

* Adding max_retries to config default

* Update database retry unit test

* Print to debug hash exception

* Fixing checksum unit test

* Retry on _download_text_file

* Update datasets_tutorial.py

* Update custom_flow_tutorial.py

* Update test_study_functions.py

* Update test_dataset_functions.py

* more retries, but also more time between retries

* allow for even more retries on get calls

* Catching failed get task

* undo stupid change

* fix one more test

* Refactoring md5 hash check inside _send_request

* Fixing a fairly common unit test fail

* Reverting loose check on unit test

Co-authored-by: Matthias Feurer <[email protected]>

* Refactor flow of loading/compressing data

There was a lot of code duplication, and the general flow of
loading/storing the data in compressed format was hard to navigate.

* Only set data file members for files that exist

* Call get_data to create compressed pickle

Otherwise the data would actually be loaded from arff (first load).

* Add data load refactor

* Revert aggressive text replacement from PyCharm

My editor incorrectly renamed too many instances of 'data_file' to
'arff_file'.

* Avoid duplicate exists/isdir

* Adding importable helper functions

* Changing import of cat, cont

* Better docstrings

* Adding unit test to check ColumnTransformer

* Refinements from @mfeurer

* Editing example to support both NumPy and Pandas

* Unit test fix to mark for deletion

* Making some unit tests work

* Waiting for dataset to be processed

* Minor test collection fix

* Template to handle missing tasks

* Accounting for more missing tasks:

* Fixing some more unit tests

* Simplifying check_task_existence

* black changes

* Minor formatting

* Handling task exists check

* Testing edited check task func

* Flake fix

* More retries on connection error

* Adding max_retries to config default

* Update database retry unit test

* Print to debug hash exception

* Fixing checksum unit test

* Retry on _download_text_file

* Update datasets_tutorial.py

* Update custom_flow_tutorial.py

* Update test_study_functions.py

* Update test_dataset_functions.py

* more retries, but also more time between retries

* allow for even more retries on get calls

* Catching failed get task

* undo stupid change

* fix one more test

* Refactoring md5 hash check inside _send_request

* Fixing a fairly common unit test fail

* Reverting loose check on unit test

* Fixing integer type check to allow np.integer

* Trying to loosen check on unit test as fix

* Examples support for pandas=1.2.1

* pandas indexing as iloc

* fix example: actually load the different tasks

* Renaming custom flow to disable tutorial (#1019)

Co-authored-by: Matthias Feurer <[email protected]>
Co-authored-by: PGijsbers <[email protected]>

* Fix #883 #884 #906 #972

* Address Mitar's comments

* rework for Windows/OSX, some mypy pleasing due to pre-commit

* type fixes and removing unused code

* update getdatasets function to give possibility to not download the dataset qualities

* make download qualities defaulted to True

* Using cahced version if exist

* Updated the comments for get_dataset and get_datasets to include new parameter

* Update openml/datasets/functions.py

Co-authored-by: PGijsbers <[email protected]>

* Update openml/datasets/functions.py

Co-authored-by: PGijsbers <[email protected]>

* update get_dataset_qualities to have consistent output regardless the cache status , adding unit test for get_dataset_qualities

* run pre-commit

* fix parameter passing

* Updated the comments for get_dataset and get_datasets to include new parameter, remove unnecessarily call for download qualities

Co-authored-by: Mohamed Adel <[email protected]>
Co-authored-by: PGijsbers <[email protected]>
Co-authored-by: mohamed adel <[email protected]>

* Adding importable helper functions

* Changing import of cat, cont

* Better docstrings

* Adding unit test to check ColumnTransformer

* Refinements from @mfeurer

* Editing example to support both NumPy and Pandas

* Unit test fix to mark for deletion

* Making some unit tests work

* Waiting for dataset to be processed

* Minor test collection fix

* Template to handle missing tasks

* Accounting for more missing tasks:

* Fixing some more unit tests

* Simplifying check_task_existence

* black changes

* Minor formatting

* Handling task exists check

* Testing edited check task func

* Flake fix

* More retries on connection error

* Adding max_retries to config default

* Update database retry unit test

* Print to debug hash exception

* Fixing checksum unit test

* Retry on _download_text_file

* Update datasets_tutorial.py

* Update custom_flow_tutorial.py

* Update test_study_functions.py

* Update test_dataset_functions.py

* more retries, but also more time between retries

* allow for even more retries on get calls

* Catching failed get task

* undo stupid change

* fix one more test

* Refactoring md5 hash check inside _send_request

* Fixing a fairly common unit test fail

* Reverting loose check on unit test

* Updating examples to run on sklearn 0.24

* Spawning tests for sklearn 0.24

* Adding numpy import

* Fixing integer type check to allow np.integer

* Making unit tests run on sklearn 0.24

* black fix

* Trying to loosen check on unit test as fix

* simplify examples

* disable test for old python version

Co-authored-by: Matthias Feurer <[email protected]>
Co-authored-by: PGijsbers <[email protected]>
Co-authored-by: neeratyoy <>

* improve path detection

* simplify code a bit

* Squashing commits

* All flow dependencies for sklearn>0.24 will change now

* Dep. string change only for OpenML>v0.11

* Black fix + removal of untested unit test

* Black fix

* More unit tests

* Docstrings + unit test robustness

* Skipping unit test for lower sklearn versions

* Skipping unit test for lower sklearn versions

* Refining unit tests

* fix merge conflict

Co-authored-by: Matthias Feurer <[email protected]>

* Store the minio_url from description xml

* Add minio dependency

* Add call for downloading file from minio bucket

* Allow objects to be located in directories

* add parquet equivalent of _get_dataset_arff

* Store parquet alongside arff, if available

* Deal with unknown buckets, fix path expectation

* Update test to reflect parquet file is downloaded

* Download parquet file through lazy loading

i.e. if the dataset was initially retrieved with download_data=False,
make sure to download the dataset on first get_data call.

* Load data from parquet if available

* Update (doc) strings

* Cast to signify url is str

* Make cache file path generation extension agnostic

Fixes a bug where the parquet files would simply be overwritten.
Also now only save the local files to members only if they actually
exist.

* Remove return argument

* Add clear test messages, update minio urls

* Debugging on CI with print

* Add pyarrow dependency for loading parquet

* Remove print

* initial commit

* private functions

* [skip ci] addressing #248

* Unit test to test existence of refit time

* Refactoring unit test

* Fixing unit test failures

* Unit test fixing + removing redundant parameter

* Debugging stochastic failure of test_joblib_backends unit test

* Unit test fix with decorators

* Flaky for failing unit test

* Adding flaky reruns for unit tests

* Fixing setup big

* pytest rerun debug

* Fixing coverage failure

* Debugging coverage failure

* Debugging coverage failure

* Adding __init__ files in test/ for pytest-cov

* Debugging coverage failure

* Debugging lean unit test

* Debugging loky failure in unit tests

* Clean up of debugging stuff

* Test setup_string is stored and retrievable

* Add setup_string to run dictionary representation

* Add fix to release notes

* Test setup_string in xml without roundtrip

Also moved the test to OpenMLRun, since it mainly tests the OpenMLRun
behavior, not a function from openml.runs.functions.

* Serialize run_details

* Update with merged PRs since 11.0

* Prepare for run_details being provided by the server

* Remove pipeline code from setup_string

Long pipelines (e.g. gridsearches) could lead to too long setup strings.
This prevented run uploads.
Also add mypy ignores for old errors which weren't yet vetted by mypy.

* skip two unit tests on Windows

* make tests less strict for Windows