Skip to content

grpcio-tools and grpcio-health-checking namespace packages break grpcio in nonstandard installs #8058

@marienz

Description

@marienz

grpcio-health-checking currently cannot be installed separately, while using grpcio (and grpcio-tools) installed system-wide (or in any directory other than the one grpcio-health-checking is installed into).

grpcio-tools and grpcio-health-checking use "grpc" as a namespace package, which is intended to make the above work. That is: they pass namespace_packages=['grpc'] to setuptools.setup.

As documented, namespace packages have the following requirements:

  • You must NOT include any other code and data in a namespace package’s __init__.py
  • You must include the declare_namespace() line in the __init__.py of every project that has contents for the namespace package in question

grpcio-tools and grpcio-health-checking meet these requirements. However, grpcio itself uses "grpc" as a normal package: most of its code is in grpc/__init__.py, it does not call declare_namespace, and it does not have namespace_packages=['grpc'].

This mostly works, but in more unusual situations (involving installs of grpcio and the grpcio-* subpackages in different directories), it breaks a bit. For example:

  • Install grpcio (and possibly grpcio-tools) into system site-packages (using favorite package manager).
  • Create a virtualenv with --system-site-packages.
  • Use this virtualenv's pip to install grpcio-health-checking.

If I do this using Python 3.5 and setuptools 26.1.1, it results in the virtualenv's "grpc" package shadowing the system one (because of .nspkg.pth files created in the virtualenv site-packages by setuptools). If setuptools were to change to start relying on PEP 420's implicit namespace packages instead of dropping a .nspkg.pth file, or if grpcio-health-checking was installed outside of site-packages (so .pth files are not executed), it would break in the other direction: the system grpc package gets picked up as a standard package because it has __init__.py, shadowing the __init__.py-less namespace package used by grpcio-health-checking.

I would expect pip to produce the same broken install if it is asked to install some package that depends on a grpcio-* subpackage rather than grpcio-* itself. There are probably more failing scenarios (but I'm out of practice with namespace packages, so I don't know what they all are).

Maybe the answer is "install all grpcio packages the same way in the same location to avoid these problems". But that would somewhat defeat the purpose of splitting the packages up in the first place. And especially if subpackages like grpcio-health-checking are pure Python, there is some value in being able to install them separately while using a system-wide grpcio, avoiding the need for a working C compiler and Python headers.

Possible mostly backwards-compatible fix:

  • Move grpcio-tools and grpcio-health-checking to a proper namespace package that does not rely on code in __init__.py.
  • Include grpc/health.py and grpc/tools.py in the main grpcio package. When imported, they import the subpackages from their new location and replace themselves. If the subpackages are not installed, you get an ImportError.
  • Optionally, modify the forwarding modules to produce a deprecation warning on import after some deprecation period.
  • After a suitable deprecation period, remove the forwarding modules from grpc/__init__.py.

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions