Have appveyor build relevant versions of Python. by matteius · Pull Request #5887 · celery/celery

matteius · 2019-12-22T22:16:42Z

Description

Recent PR's fail their AppVeyor builds. Python 2 is failing with a warning that AppVeyor will remove support after January 1st, and python 3.4 build is failing on our own setup.py requirement that " Celery 4.0 requires CPython 3.5 or later".

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support

Traceback (most recent call last):
  File "setup.py", line 239, in <module>
    'celery = celery.contrib.pytest',
  File "C:\Python27\lib\site-packages\setuptools\__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "C:\Python27\lib\distutils\core.py", line 151, in setup
    dist.run_commands()
  File "C:\Python27\lib\distutils\dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "C:\Python27\lib\distutils\dist.py", line 972, in run_command
    cmd_obj.run()
  File "C:\Python27\lib\site-packages\setuptools\command\test.py", line 225, in run
    installed_dists = self.install_dists(self.distribution)
  File "C:\Python27\lib\site-packages\setuptools\command\test.py", line 209, in install_dists
    tr_d = dist.fetch_build_eggs(dist.tests_require or [])
  File "C:\Python27\lib\site-packages\setuptools\dist.py", line 721, in fetch_build_eggs
    replace_conflicting=True,
  File "C:\Python27\lib\site-packages\pkg_resources\__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (python-dateutil 2.8.1 (c:\projects\celery\.eggs\python_dateutil-2.8.1-py2.7.egg), Requirement.parse('python-dateutil<2.8.1,>=2.1'), set(['botocore']))

See also for python 3.4: https://ci.appveyor.com/project/ask/celery/builds/29484693/job/dsnjeiykm2u4262d

Executing: C:\Python34/python setup.py test
Traceback (most recent call last):
  File "setup.py", line 50, in <module>
    raise Exception(E_UNSUPPORTED_PYTHON % (PYIMP, '3.5'))
Exception: 
----------------------------------------
 Celery 4.0 requires CPython 3.5 or later
----------------------------------------
- For CPython 2.6, PyPy 1.x, Jython 2.6, CPython 3.2->3.3; use Celery 3.1:
    $ pip install 'celery<4'
- For CPython 2.5, Jython 2.5; use Celery 3.0:
    $ pip install 'celery<3.1'
- For CPython 2.4; use Celery 2.2:
    $ pip install 'celery<2.3'
Command exited with code 1

codecov · 2019-12-22T22:24:05Z

Codecov Report

Merging #5887 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #5887   +/-   ##
=======================================
  Coverage   83.19%   83.19%           
=======================================
  Files         145      145           
  Lines       17126    17126           
  Branches     2110     2110           
=======================================
  Hits        14248    14248           
  Misses       2663     2663           
  Partials      215      215

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ad5592...c63d602. Read the comment docs.

…iles exist)

auvipy · 2019-12-23T03:12:44Z

why 2.7 is being dropped?

matteius · 2019-12-23T12:32:23Z

@auvipy I mentioned the answer in the description above, but I will reiterate.

why 2.7 is being dropped?

Appveyor is a Windows CI so it builds everything in Windows and is already failing to fetch the python 2.7 environment, it prints out the deprecation warning:

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support

Why should having appveyor (windows builds) always fail the build because it is configured to never work properly, be somehow a proof that the project still can run on python 2?

matteius · 2019-12-23T12:34:40Z

Perhaps the real issue with 2.7 for Appveyor is:

pkg_resources.ContextualVersionConflict: (python-dateutil 2.8.1 (c:\projects\celery.eggs\python_dateutil-2.8.1-py2.7.egg), Requirement.parse('python-dateutil<2.8.1,>=2.1'), set(['botocore']))

but I am not sure there is a great fix to that either -- perhaps figure out what those things have to be pinned to and call out 2.7 with a different set of requirements.txt in the tox.ini ? Just doesn't seem worth it for something that is going away -- have the Travis CI show that the 2.7 unit tests still pass should be sufficient evidence the code can be run on 2.7.

auvipy · 2019-12-23T13:13:31Z


-    - PYTHON: "C:\\Python27"
-      PYTHON_VERSION: "2.7.x"
-      PYTHON_ARCH: "32"


just try to use 64bit and keep it, please. so that this can be merged.

auvipy

should we wait for appveyor to be green or this can be merged as is now?

matteius · 2019-12-23T15:59:45Z

It can be merged as incremental progress -- it further reveals the next set of issues related to the windows celery build's. After the latest run build failures are:
python2.7 -- couchbase will never support py2 on windows (requires py3)
python3.5 and 3.6 get the furthest along -- they eventually fail on the pylibmc install not finding the libmemcached headers:
c:\users\appveyor\appdata\local\temp\1\pip-install-jw3v0vds\pylibmc\src\_pylibmcmodule.h(42): fatal error C1083: Cannot open include file: 'libmemcached/memcached.h': No such file or directory
python3.7 and 3.8 fail earlier on the pycurl because the wheel files don't exist for those builds of python and so it fails to build Re: libcurl (we have only been able to work around this with appveyor by pinning to versions with prebuilt whl files that are compatible).

auvipy

OK thanks.

dejlek · 2019-12-24T12:55:06Z

PyPy will continue supporting v2. Perhaps we should add PyPy 2 and 3 versions?

auvipy · 2019-12-24T13:03:05Z

we will keep python 2 until the end of 2020

matteius · 2019-12-24T14:40:00Z

We really need to differentiate project level support vs. appveyor build. My chief argument here is that we have two build CIs right now: Travis CI and Appveyor. While Travis CI is designed to pass, we are locking ourselves into patterns that will always fail Appveyor builds, because Appveyor builds are Windows system builds.

If the project dependency cannot build on Windows easily, then it will fail in Appveyor which is the problem we have today with all of the Appveyor builds. We should design the happy path build case such that the master code repository passes as much as possible on both CIs -- why even have Appveyor if we aren't going to refine to have a happy path Green build of it?

I have been attempting through agile commit patterns tried to refine the exact blocker for each python version in the Appveyor output preventing it from building without introducing an additional impediment that would affect all builds to fail at an earlier step (in other words, reduce impediments to the builds succeeding).

For the builds that have gotten past the py2 issue, or the issue of finding a wheel for pycurl (some py3 builds have this), the latest issue is that none of the wheels line up for pylibmc. You can tell because those builds run longer before failing in appveyor.
Its not that pylibmc has no wheels, it has some, but I think the weird naming pattern prevents them from being installed by default? Still trying to figure that one out, but if we do, we might be able to get a few of the appveyor build configs to build green and that would be a victory in my book.

auvipy · 2019-12-24T14:56:29Z

We got a plan to move from appveyor

matteius · 2019-12-29T13:15:03Z

@auvipy What is the planned replacement for appveyor? My rational argument would be to keep appveyor as a "Here is what a windows build looks like" CI and then add on any additional CI for the purpose appveyor was often perceived to of had.

auvipy · 2019-12-29T14:10:25Z

check it out celery/ceps#15 (comment)

* Fix serialization and deserialization of nested exception classes (#5717) * Fix #5597: chain priority (#5759) * adding `worker_process_shutdown` to __all__ (#5762) * Fix typo (#5769) * Reformat code. * Simplify commands to looking for celery worker processes (#5778) * update doc- celery supports storage list. (#5776) * Update introduction.rst * Update introduction.rst * Fail xfailed tests if the failure is unexpected. * Added integration coverage for link_error (#5373) * Added coverage for link_error. * Use pytest-rerunfailed plugin instead of rolling our own custom implementation. * Added link_error with retries. This currently fails. * Remove unused import. * Fix import on Python 2.7. * retries in link_error do not hang the worker anymore. * Run error callbacks eagerly when the task itself is run eagerly. Fixes #4899. * Adjust unit tests accordingly. * Grammar in documentation (#5780) * Grammar in documentation * Address review. * pypy 7.2 matrix (#5790) * removed extra slashes in CELERY_BROKER_URL (#5792) The Celery broker URL in settings.py had 2 slashes in the end which are not required and can be misleading. so I changed :- CELERY_BROKER_URL = 'amqp://guest:guest@localhost//' to CELERY_BROKER_URL = 'amqp://guest:guest@localhost' * Fix #5772 task_default_exchange & task_default_exchange_type not work (#5773) * Fix #5772 task_default_exchange & task_default_exchange_type not work * Add unit test: test_setting_default_exchange * Move default_exchange test to standalone class * Run integration suite with memcached results backend. (#5739) * Fix hanging forever when fetching results from a group(chain(group)) canvas. (#5744) PR #5739 uncovered multiple problems with the cache backend. This PR should resolve one of them. PR #5638 fixed the same test case for our async results backends that support native join. However, it did not fix the test case for sync results backends that support native join. * Fix regression in PR #5681. (#5753) See comment in the diff for details. * Grammatical fix to CONTRIBUTING.rst doc (#5794) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs (#5795) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs * Add unit test: test_get_connection_with_authmechanism * Add unit test: test_get_connection_with_authmechanism_no_username * Fix errors in Python 2.7 Remove "," after "**" operator * Revert "Revert "Revert "Added handle of SIGTERM in BaseTask in celery/task.py to prevent kill the task" (#5577)" (#5586)" (#5797) This reverts commit f79894e. * Add Python 3.8 Support (#5785) * Added Python 3.8 to the build matrix. * Ensure a supported tblib version is installed for Python 3.8 and above. In addition, modernize the relevant tests. * Workaround patching problem in test. * py 3.8 in clasifier * ubuntu bionic (#5799) * ubuntu bionic * fast finish * sync bumversion with pypi release * Dev.req (#5803) * update docker config * undo hardpin * devr req install from github master * update docker config (#5801) * update docker config * make dockerfile to install from github master dev branch by default * update download link * Isort. * Grammatical & punctuation fixes for CONTRIBUTING.rst document (#5804) * update dockerfile * switched to ubuntu bionic * update docker * keep it empty until we reconfigure it again with autopep8 * Fixed Dockerfile (#5809) * Update document CONTRIBUTING.rst & fix Dockerfile typo (#5813) * Added an issue template for minor releases. * reference gocelery Go Client/Server for Celery (#5815) * Add enterprise language (#5818) * Fix/correct minor doc typos (#5825) * Correct a small typo * Correct bad contributing documentation links * Preserve the task priority in case of a retry (#5820) * Preserve the task priority in case of a retry * Created test case for retried tasks with priority * Implement an integration test for retried tasks with priorities * bump kombu * basic changelog for celery 4.4.0rc4 * bump celery 4.4.0rc4 * events bootstep disabled if no events (#5807) * events bootstep disabled if no events * Added unit tests. * update bug report template * fixing ascii art to look nicer (#5831) * Only rerun flaky tests when failures can be intermediary. * Rename Changelog to Changelog.rst * The test_nested_group_chain test can run without native_join support. (#5838) * Run integration tests with Cassandra (#5834) * Run integration tests with Cassandra. * Configure cassandra result backend * Pre-create keyspace and table * Fix deprecation warning. * Fix path to cqlsh. * Increase connection timeout. * Wait until the cluster is available. * SQS - Reject on failure (#5843) * reject on failure * add documentation * test fix * test fix * test fix * Add a concurrency model with ThreadPoolExecutor (#5099) * Add a concurrency model with ThreadPoolExecutor * thread model test for pypy * Chain primitive's code example fix in canvas documentation (Regression PR#4444) (#5845) * Changed multi-line string (#5846) This string wasn't rendering properly and was printing the python statement too. Although the change isn't as pretty code-wise, it gets rid of an annoyance for the user. * Add auto expiry for DynamoDB backend (#5805) * Add auto expiry for DynamoDB backend This adds auto-expire support for the DynamoDB backend, via the DynamoDB Time to Live feature. * Require boto3>=1.9.178 for DynamoDB TTL support boto3 version 1.9.178 requires botocore>=1.12.178. botocore version 1.12.178 introduces support for the DynamoDB UpdateTimeToLive call. The UpdateTimeToLive call is used by the DynamoDB backend to enable TTL support on a newly created table. * Separate TTL handling from table creation Handle TTL enabling/disabling separately from the table get-or-create function. Improv

matteius added 2 commits December 22, 2019 17:10

Have appveyor build relevant and buildable versions of Python.

6554408

Appveyor is missing CI requirements to build.

8303db3

Pin pycurl to version that will build with appveyor (because wheels f…

33053d5

…iles exist)

matteius force-pushed the improve-appveyor branch from 2534814 to 33053d5 Compare December 22, 2019 22:52

auvipy requested changes Dec 23, 2019

View reviewed changes

auvipy added this to the 4.4.x milestone Dec 23, 2019

Restrict python 2.7 64 bit version of python-dateutil for parse.

c63d602

auvipy reviewed Dec 23, 2019

View reviewed changes

auvipy approved these changes Dec 23, 2019

View reviewed changes

auvipy merged commit 47d3ef1 into celery:master Dec 23, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Have appveyor build relevant versions of Python.#5887

Have appveyor build relevant versions of Python.#5887
auvipy merged 4 commits into
celery:masterfrom
matteius:improve-appveyor

matteius commented Dec 22, 2019 •

edited

Loading

Uh oh!

codecov Bot commented Dec 22, 2019 •

edited

Loading

Uh oh!

auvipy commented Dec 23, 2019

Uh oh!

matteius commented Dec 23, 2019

Uh oh!

matteius commented Dec 23, 2019

Uh oh!

auvipy Dec 23, 2019

Uh oh!

auvipy left a comment

Uh oh!

matteius commented Dec 23, 2019

Uh oh!

auvipy left a comment

Uh oh!

dejlek commented Dec 24, 2019

Uh oh!

auvipy commented Dec 24, 2019

Uh oh!

matteius commented Dec 24, 2019

Uh oh!

auvipy commented Dec 24, 2019

Uh oh!

matteius commented Dec 29, 2019

Uh oh!

auvipy commented Dec 29, 2019

Uh oh!

Uh oh!

Uh oh!

Conversation

matteius commented Dec 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

codecov Bot commented Dec 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

auvipy commented Dec 23, 2019

Uh oh!

matteius commented Dec 23, 2019

Uh oh!

matteius commented Dec 23, 2019

Uh oh!

auvipy Dec 23, 2019

Choose a reason for hiding this comment

Uh oh!

auvipy left a comment

Choose a reason for hiding this comment

Uh oh!

matteius commented Dec 23, 2019

Uh oh!

auvipy left a comment

Choose a reason for hiding this comment

Uh oh!

dejlek commented Dec 24, 2019

Uh oh!

auvipy commented Dec 24, 2019

Uh oh!

matteius commented Dec 24, 2019

Uh oh!

auvipy commented Dec 24, 2019

Uh oh!

matteius commented Dec 29, 2019

Uh oh!

auvipy commented Dec 29, 2019

Uh oh!

matteius commented Dec 22, 2019 •

edited

Loading

codecov Bot commented Dec 22, 2019 •

edited

Loading