package data in subdirectory causes warning

@inducer

setuptools version

62.3.2

Python version

3.10

OS

Debian with conda

Additional environment information

No response

Description

pyopencl has OpenCL files and some headers in a subdirectory pyopencl/cl and they are included as package_data so that the python module can find them.

package_data={
                    "pyopencl": [
                        "cl/*.cl",
                        "cl/*.h",
                        "cl/pyopencl-random123/*.cl",
                        "cl/pyopencl-random123/*.h",
                        ]
                    },

With new setuptools, there is a warning saying


    ############################
    # Package would be ignored #
    ############################
    Python recognizes 'pyopencl.cl' as an importable package, however it is
    included in the distribution as "data".
    This behavior is likely to change in future versions of setuptools (and
    therefore is considered deprecated).

    Please make sure that 'pyopencl.cl' is included as a package by using
    setuptools' `packages` configuration field or the proper discovery methods
    (for example by using `find_namespace_packages(...)`/`find_namespace:`
    instead of `find_packages(...)`/`find:`).

    You can read more about "package discovery" and "data files" on setuptools
    documentation page.

cc @inducer

Expected behavior

No warning

How to Reproduce

clone https://github.com/inducer/pyopencl
install numpy
Run python setup.py install

Output

$ python setup.py install
running install
/home/idf2/miniforge3/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/home/idf2/miniforge3/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
writing pyopencl.egg-info/PKG-INFO
writing dependency_links to pyopencl.egg-info/dependency_links.txt
writing requirements to pyopencl.egg-info/requires.txt
writing top-level names to pyopencl.egg-info/top_level.txt
reading manifest file 'pyopencl.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'pyopencl.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
/home/idf2/miniforge3/lib/python3.10/site-packages/setuptools/command/build_py.py:153: SetuptoolsDeprecationWarning:     Installing 'pyopencl.cl' as data is deprecated, please list it in `packages`.
    !!


    ############################
    # Package would be ignored #
    ############################
    Python recognizes 'pyopencl.cl' as an importable package, however it is
    included in the distribution as "data".
    This behavior is likely to change in future versions of setuptools (and
    therefore is considered deprecated).

    Please make sure that 'pyopencl.cl' is included as a package by using
    setuptools' `packages` configuration field or the proper discovery methods
    (for example by using `find_namespace_packages(...)`/`find_namespace:`
    instead of `find_packages(...)`/`find:`).

    You can read more about "package discovery" and "data files" on setuptools
    documentation page.


!!

  check.warn(importable)
running build_ext

@isuruf

Hi @isuruf thank you for bringing this subject for discussion.

This change (and the warning) was intentional, so in the future we will be able to solve a related problem with find_packages(..., exclude=...) and include_package_data.

Since PEP 420, there is no way for really differentiating a folder from a package in Python... People that install pyopencl will be able to successfully run import pyopencl.cl, which means that the interpreter does consider properly.cl a package. Nevertheless configuration file seems to want to consider cl a "data directory", which is not currently a well stablished concept in the ecosystem...

Would it possible for you guys to use find_namespace_packages() as indicated in the warning message?

@abravalheri

I'm facing the same problem, and I'm not clear on the nature of the change suggested by @abravalheri. Are you saying that directories in the package hierarchy that contain only data files and no Python code should be included in the project's list of packages despite not actually being Python packages?

@jwodder

Hi @jwodder, thank you very much for the input. Please find my comments bellow:

directories in the package hierarchy that contain only data files and no Python ... not actually being Python packages

I think that this assumption¹ no longer holds.

According to PEP 420, package is a term that "refers to Python packages as defined by Python’s import statement".
An evidence that a directory without Python code is treated by the Python import statement as a package is given by the following snippets:

rm -rf /tmp/example
mkdir -p /tmp/example/mypkg1
touch /tmp/example/mypkg1/file.txt
mkdir -p /tmp/example/mypkg2/subpkg
touch /tmp/example/mypkg2/__init__.py
touch /tmp/example/mypkg2/subpkg/file.txt
python3.10

>>> import sys
>>> sys.path.append("/tmp/example")
>>> import mypkg1
>>> mypkg1
<module 'mypkg1' (<_frozen_importlib_external._NamespaceLoader object at 0x7f3683fac2e0>)>
>>> mypkg1.__path__
_NamespacePath(['/tmp/example/mypkg1'])
>>> mypkg1.__loader__.is_package('mypkg1')
>>> import mypkg2.subpkg
>>> mypkg2.subpkg.__path__
_NamespacePath(['/tmp/example/mypkg2/subpkg'])
>>> mypkg2.subpkg.__loader__.is_package('mypkg2.subpkg')
True

Note in this example that both mypkg1 and mypkg2.subpkg can be imported, and are treated as regular packages (both are imported as module objects with the attribute __path__ set), despite not containing any Python file.
I added a mypkg.subpkg to the example to demonstrate that it does not matter whether or not the directory is nested inside another "traditional package".

Moreover, the ecosystem also considers those folders, as demonstrated by the implementation of importlib.resources:

>>> import importlib.resources
>>> importlib.resources.contents('mypkg1')
['file.txt']
>>> importlib.resources.contents('mypkg2.subpkg')
['file.txt']

I understand that not all tools in the ecosystem embrace PEP 420 by default and that sometimes it might be frustrating to see tools evolving some interfaces evolving. That is the reason why I decided to add a deprecation warning first and a transition period, instead of simply changing things in a major version bump.

This change in setuptools was implemented due to a single, very pragmatic, objective: to fix internal inconsistencies (as presented in #3260), which are caused by this division between packages and directories that don't contain Python code (which is not a division recognized by Python's import system).

For the time being, I don't plan on removing the deprecation warning, since the only way I know how to fix the internal inconsistencies requires us to eliminate this arbitrary division. However if anyone in the community finds a different way of solving the problem and are willing to provide a backward-compatible PR, I am more than happy to consider an alternative.

Namely the assumption that a directory with no Python code is not a package. ↩

@abravalheri

@abravalheri So you are saying that "data-only packages" should be included in the project's list of packages? Is that the only change you're currently recommending to authors of such projects? Does it matter whether the data files are included via package_data or MANIFEST.in+include_package_data?

Yes, I recommend adding all the sub-directories to the list of packages, even if they only include data files. On the bright side that can be done with find_namespace_packages (setup.py) or find_namespace: (setup.cfg).

It should make no difference if you are using package_data or MANIFEST.in (although I am not sure if users are going to see the warning if they don't use include_package_data=True).

Do you have a suggestion about how to improve the warning message to make it more clear?

@abravalheri

@abravalheri I find the first paragraph of the warning to be a bit confusing. I would rewrite it from:

Python recognizes '...' as an importable package, however it is included in the distribution as "data". This behavior is likely to change in future versions of setuptools (and therefore is considered deprecated).

to something more like:

Python recognizes '...' as an importable package, but it is not listed in "packages". It is included in the distribution because it contains package data, but this behavior is likely to change in future versions of setuptools (and therefore is considered deprecated).

Use find_namespace: instead of find:

I was following this issue because I was very confused (as a newbie) by the warning that came up. Can I suggest a further edit to the warning message:
Currently {importable!r} is only added to the distribution because it may
contain data files...
-->
Currently {importable!r} has been automatically added to the distribution because it may
contain data files...
This is my understanding of what is actually happening (i.e. automatic inclusion). If this suggested change is wrong, then I'm still not following how this works...

@abravalheri

Do you have any go-to suggestion for suppressing the warning in such cases?

UPD: turned out that it was include_package_data=True that confused setuptools. There was really no need for it and removing the setting fixed the problem.

@abravalheri I think that adding a hint suggesting to delete include_package_data=True and replace it with a fine-grained package-data mapping would be useful.

In my case, a more straightforward suggestion would've made it clearer what to do, because I wasn't the one to set up initial packaging in the project and had to attempt figuring out how and why things were set up that way.

Another source of confusion was the fact that the error message was unhappy with multidict._multilib, suggesting to use helpers like find_namespace_packages() while it did list the packages explicitly with no discovery needed, via a static list: packages=["multidict"]. This also confused a contributor who attempted adding "multidict._multilib" to that list, not understanding immediately that C-headers aren't to be installed in site-packages/ (or distributed in wheels, for that matter).

@abravalheri

@abravalheri

I appreciate your efforts to solve what is a quite challenging situation due to recent changes (PEP 420). You said very clearly early in this issue: "Since PEP 420, there is no way for really differentiating a folder from a package in Python."

But this problematic new state is not one that developers should learn to live with, they should not be compelled to adopt insecure quick fixes. The packaging distinction between code and data should be preserved. Because to do otherwise, to conflate between the two, and to make data executable by default, is to introduce additional security risks and to basically undermine the concept of having an "api".

The proposed fix which resolves the deprecation is apparently to use find_namespace_packages. But this is just wrong. Since namespace packages are meant to contain importable executable code.

In my current project which wraps a third party library. The directory structure is slightly more complex to accommodate two build variants: a default dynamically-linked build and an optional statically-linked build variant.

The default build's code is in src/cyfaust and the static build code is in src/static/cyfaust. The project consists of cython extensions and a significant amount of data which is placed in a resources directory hierarchy in the project root and softlinked into each build variant's folder so it can be picked up as below:

.
├── docs
├── include
├── resources
│    ├── architecture
│    │  ├── AU
│    │  ├── VST
│    │  ├── api
│    │  ├── daisy
│    │  ├── faust
│    │  ├── juce
│    │  ├── max-msp
│    │  ├── sam
│    │  ├── teensy
│    │  └── vcvrack
│    └── libraries
│        └── examples
├── scripts
├── src
│    ├── cyfaust
│    │  └── resources -> ../../resources
│    └── static
│        └── cyfaust
│               └── resources -> ../../../resources
└── tests

Within the resources folder structure (which is specified by the library I'm wrapping) there are 775 files, 10.4 MB in total. Most of these are templated source files and all are part of the thirdparty library itself. But all are precisely data, not to be imported in the project package, only to be used for the purposes of code generation.

The relevant part of setup.py captures this package code, data directory distinction simply and effectively:

# ...

if STATIC:

    # forces cythonize in this case
    subprocess.call("cythonize cyfaust.pyx", cwd="src/static/cyfaust", shell=True)

    with open("MANIFEST.in", "w") as f:
        f.write("graft src/static/cyfaust/resources\n")
        f.write("exclude src/static/cyfaust/*.cpp\n")

    extensions = [
        mk_extension("cyfaust.cyfaust", 
            sources=["src/static/cyfaust/cyfaust.pyx"] + RTAUDIO_SRC,
            define_macros=DEFINE_MACROS,
        ),
    ]

    setup(
        name='cyfaust',
        version=VERSION,
        ext_modules=cythonize(
            extensions,
            language_level="3",
        ),
        package_dir = {"cyfaust": "src/static/cyfaust"},
        packages=['cyfaust'],
        include_package_data=True
    )

# ----------------------------------------------------------------------------

else:

    with open("MANIFEST.in", "w") as f:
        f.write("graft src/cyfaust/resources\n")
        f.write("exclude src/cyfaust/*.cpp\n")

    extensions = [
        mk_extension("cyfaust.interp", 
            sources=["src/cyfaust/interp.pyx"] + RTAUDIO_SRC,
            define_macros=DEFINE_MACROS,
        ),
        mk_extension("cyfaust.common", ["src/cyfaust/common.pyx"]),
        mk_extension("cyfaust.signal",["src/cyfaust/signal.pyx"]),
        mk_extension("cyfaust.box", ["src/cyfaust/box.pyx"]),
    ]

    setup(
        name='cyfaust',
        version=VERSION,
        ext_modules=cythonize(
            extensions,
            language_level="3",
        ),
        package_dir = {"cyfaust": "src/cyfaust"},
        packages=['cyfaust'],
        include_package_data=True
    )

Right now, and everything just works as expected. The only problem I have is the deprecation warning and the fact that I'm put off by the proposed solution to use find_namespace_packages for the reasons given above.

May I suggest that the deprecation warning be dropped or even abbreviated somewhat until the data / code packaging distinction can be restored and preserved by some new PEP or other. It makes no sense to promote solutions, in the interim, which incorrectly conflate the two.

S

@shakfu

Hi @shakfu, thank you very much for sharing your thoughts. Please see my comments below:

[...] quite challenging situation due to recent changes (PEP 420) [...] But this problematic new state is not one that developers should learn to live with [...]

Please note that the "directory is a package" behaviour introduced in PEP 420 is actually quite old. The PEP has been approved in 19/Apr/2012 and the specified behaviour implemented in Python 3.3, over 10 years ago which is a lot in "software development years".
It is safe to assume that this behaviour is stable and that at some point in their carreers Python developers will indeed learn what a Python package means and how adding a directory nested somewhere under one of the entries of sys.path corresponds effectively to create a Python package that can be regularly imported as any other package via an import statement.

[...] they should not be compelled to adopt insecure quick fixes.

Could you please clarify in what sense the solutions presented in the warning message are insecure?
The error message presents to the user a couple of suggestions (to be chosen accordingly to what fits better their use case) which include to manually add the missing entries to the packages configuration option, or to use a convenience function provided by setuptools. In both cases, the procedure is very mature and stable.

The proposed fix which resolves the deprecation is apparently to use find_namespace_packages. But this is just wrong. Since namespace packages are meant to contain importable executable code.

I understand that this is a popular interpretation of what the concept of packages (and/or namespace packages) might mean for Python and I see where it comes from. But I don't think this interpretation is backed by the Python implementation and the way it works...

You can import directories that don't contain .py files, and having packages for holding non-Python files is actually a very useful feature¹! I never found official documentation saying that packages/namespace packages are meant to contain importable executable code and cannot be used to contain only non-Python files, I don't think there is an official stance on that.

Within the resources folder structure (which is specified by the library I'm wrapping) there are 775 files, 10.4 MB in total. Most of these are templated source files and all are part of the thirdparty library itself. But all are precisely data, not to be imported in the project package, only to be used for the purposes of code generation.

If these non-Python files are not meant to be installed in the end-user's machine, I believe it is a matter of properly configuring packages/package_data/include_package_data/exclude_package_data/MANIFEST.in so that they are part of the sdist but not part of the wheel. Otherwise, if they end up nested somewhere under an entry of sys.path, they will be import packages, effectively.

If you really don't like the idea of having these directories as importable packages, then the alternative is to use data-files, which will translate into a special directory in the wheel file ({name}-{version}.data/data/). In turn, pip will stall them in a different location that will not be nested somewhere in sys.path. It is a lot of effort to align expectation and implementation, for most people it might just be worth to adapt their expectations.

May I suggest that the deprecation warning be dropped or even abbreviated somewhat until the data / code packaging distinction can be restored and preserved by some new PEP or other. It makes no sense to promote solutions, in the interim, which incorrectly conflate the two.

I think that simply dropping the warning would be unwise.
The purpose of the warning is for developers to align their configuration to their expectations.
If they want certain directories to be installed somewhere under sys.path, this effectively mean that they are asking setupotools to include certains packages/subpackages into the wheel. Within setuptools, that desire is captured by the packages configuration option.

We need users to start clarifying their configuration, because the next step is to fix other related bugs (see #3340 (comment)), and it would be bad if suddenly some folders are missing from packages.

I don't know of anyone currently attempting to introduce a data / code distinction (and what that will mean for Python packages and import system) via a new PEP. Personally, I like the status quo and I think it works quite well. Since there is no concrete plans for such change in the ecosystem, there is no foreseeable risk of conflation, and we don't need to treat this situation as "interim". As far as we know this is the stable behaviour that we should be targetting to achieve after 10 years of transition.

There is another approach² that I have absolute no problems in considering and actually would welcome with open arms: if a member of the community is willing to contribute (i.e. design, discuss, find consensus, implement, document, fix, support ...) a different way of configuring setuptools that is more conceptually self-evident and less prone to ambiguity than packages/package_data/include_package_data/exclude_package_data/MANIFEST.in³. Extra requirements for such solution are: backward compatibility and easy maintenance.

It is a tough challenge which I don't have the resources to tackle myself, but I would be very grateful if someone else can.

They make it really easy to find non-Python files them runtime using importlib.resources. You can also implement extensions/plugin systems on top of it, and etc... ↩
But that is orthogonal to the warning and next steps discussed here. ↩
In some sense, the automatic discovery (when the user does not specify packages) that was introduced a couple of years ago is meant to be easier and less confusing. But automatic discovery is not a fit for all and edge cases still require playing with packages/package_data/include_package_data/exclude_package_data. ↩

@abravalheri

@abravalheri I think that adding a hint suggesting to delete include_package_data=True and replace it with a fine-grained package-data mapping would be useful.

That is nice idea. Depending on what the developer is trying to achieve that will indeed solve the problem. Currently the message does mention include-package-data=False:

[...]
If you don't want {importable!r} to be distributed and are
already explicitly excluding {importable!r} via
`find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
you can try to use `exclude_package_data`, or `include-package-data=False` in
combination with a more fine grained `package-data` configuration.

But probably that can be rephrased to be more clear. Would you like to give it a shot?
(P.S.: I consider include-package-data=False a better text for the warning message than omitting, because in pyproject.toml the default value of this config is True, so it just saves the time to explain the nuances).

Another source of confusion was the fact that the error message was unhappy with multidict._multilib, suggesting to use helpers like find_namespace_packages() while it did list the packages explicitly with no discovery needed, via a static list: packages=["multidict"]. This also confused a contributor who attempted adding "multidict._multilib" to that list, not understanding immediately that C-headers aren't to be installed in site-packages/ (or distributed in wheels, for that matter).

I understand that, but unfortunately there is no single fix for all scenarios. For some people include_package_data=False is the fix, for others find_namespace_packages() will do what they are trying to achieve...

And that is the problem with the ambiguous¹ configuration, setuptools should not have to guess what the user wants to do, that is why we are asking the user to tell us.

"Ambiguous" while setuptools does not take a final action on the "directories with non-Python files are not packages" controversy... Once the warning manage to compel devs to clarify their config, we can move past that problem and actually allow people that are intentionally omitting entries in packages to use include-package-data. ↩

@abravalheri

Hi @abravalheri, appreciate your extended and thoughtful reply to my early post (please see my comments below):

Please note that the "directory is a package" behaviour introduced in PEP 420 is actually quite old. The PEP has been approved in 19/Apr/2012 and the specified behaviour implemented in Python 3.3, over 10 years ago which is a lot in "software development years".

It is safe to assume that this behaviour is stable and that at some point in their carreers Python developers will indeed learn what a Python package means and how adding a directory nested somewhere under one of the entries of sys.path corresponds effectively to create a Python package that can be regularly imported as any other package via an import statement.

I've been working in Python since a little bit before version 1.5.2, so this is certainly not new to me. Nonetheless, I admittedly did not express myself clearly: I have no issue with the principle of "directory is a package" or that packages are importable. This is fundamental to Python since way back.

I'm also happy with having a data folder within a python package such that a module within that package can programmatically always find this data folder by retrieving its path relative to __file__. Indeed this is precisely what I'm doing in the project I described earlier: in the cyfaust.common module I have the following code:

def get_package_resources() -> tuple[str, str, str, str]:
    """provides the paths of package architecture and library folders."""
    resources = os.path.join(os.path.dirname(__file__), "resources")
    archs = os.path.join(resources, "architecture")
    libs = os.path.join(resources, "libraries")
    return ("-A", archs, "-I", libs)

PACKAGE_RESOURCES = get_package_resources()

After reading your prior reply, I appreciate that the current situation (and your understanding of it) is a lot more nuanced than I previously considered, and that the confusion is probably stemming from a couple of things here:

1. My failure to understand the full implications of PEP 420 Namespace Packages:

This PEP makes everything in a package potentially importable, even when that is not the intent!

For a hypothetical example in my project, a user of my package can (currently) import a python file that is only intended to be used as a template.

But due to PEP 420, this is possible:

from cyfaust.resources.architecture.templates import abctemplate

Now, this can be prevented by renaming resources to a non-python friendly name such as pkg-resources.

So now this is impossible:

from cyfaust.pkg-resources.architecture.templates import abctemplate

I did not realize that this was the default situation now... and it has nothing to do with setuptools. That's my bad.

2. The Documentation + the Deprecation + the Multiplicity of Configuration options + Different ways of doing the same thing are confusing:

In my case, this is how I tried to resolve the deprecation:

My initial working setup.py:

setup(
    name='cyfaust',
    version=VERSION,
    ext_modules=cythonize(
        extensions,
        language_level="3",
    ),
    packages=['cyfaust'],
    package_dir = {"cyfaust": "src/cyfaust"},
    include_package_data=True
)

Then I saw the deprecation notice, and to figure out how to resolve it I read the relevant datafiles section of the setuptools documentation.

I chose the initial recommended include_package_data method.

This led to the following:

setup(
    name='cyfaust',
    version=VERSION,
    ext_modules=cythonize(
        extensions,
        language_level="3",
    ),
    packages=find_packages(where="src"),
    package_dir = {"cyfaust": "src/cyfaust"},
    include_package_data=True
)

And it works, it produces the same outcomes as my prior working case, but... the deprecation notice is still there.

So after some further reading and research on the web, the advice is use find_namespace_packages so I choose the "Subdirectory for Data Files" method

I naively just changed find_packages to find_namespace_packages since it is seemingly the simplest solution

setup(
    name='cyfaust',
    version=VERSION,
    ext_modules=cythonize(
        extensions,
        language_level="3",
    ),
    packages=find_namespace_packages(where="src"),
    package_dir = {"cyfaust": "src/cyfaust"},
    include_package_data=True
)

and then I get an error:

running build
running build_py
error: package directory 'static' does not exist

At this point the frustration of palpable. All I want to do is specify a subdirectory of my package as a data directory to be included in whole.

The more complex way in this method is to add a package_data entry, but filling it out seems laborious given that I have 775 files data files accross many directories with different types etc..

At this stage, I decide to just live with the deprecation!

You asked for suggestions on how to improve this api. Here are my two cents:

Ideally it should be as simple as something like this:

setup(
    name='cyfaust',
    version=VERSION,
    ext_modules=cythonize(
        extensions,
        language_level="3",
    ),
    packages = {
        'cyfaust': {
            'path': "src/cyfaust",
            'data_dirs': ["cyfaust/pkg-resources"],
        }
    }
 )

or just:

setup(
    name='cyfaust',
    version=VERSION,
    ext_modules=cythonize(
        extensions,
        language_level="3",
    ),
    packages=['cyfaust'],
    package_dir = {"cyfaust": "src/cyfaust"},
    include_package_data=True
)

In the above ideal cases, MANIFEST.in or similar is only configured if you need more granular control over what is included or excluded. It is not required in the simplest case where you want everything included.

@abravalheri

@abravalheri

As an addendum to my last post I finally found a way to remove the deprecation warning. I could not get it done with find_namespace_packages and the only way was to run os.walk in my resources folder and then manually add folders which had python friendly names to the packages list as per the below code. Note that this was the only solution I could arrive at to remove the deprecation warning. If there is a better way, please let me know otherwise, my setup.py file looks a little ridiculous. 😟

setup(
    name='cyfaust',
    version=VERSION,
    ext_modules=cythonize(
        extensions,
        language_level="3",
    ),
    package_dir = {"cyfaust": "src/cyfaust"},
    packages = [
        "cyfaust",
            # NOTE: The entries below are not really packages but data folders
            # They have been added to this list to remove a recent deprecation
            # warning caused by the deprecatation of the category of data files
            # within packages. These are now just called packages 
            # (even if that is not the intention).
            # DO NOT IMPORT, even though they are importable!!
        "cyfaust.resources.libraries",
        "cyfaust.resources.libraries.examples",
        "cyfaust.resources.libraries.examples.ambisonics",
        "cyfaust.resources.libraries.examples.analysis",
        "cyfaust.resources.libraries.examples.autodiff",
        "cyfaust.resources.libraries.examples.autodiff.delay",
        "cyfaust.resources.libraries.examples.autodiff.gain",
        "cyfaust.resources.libraries.examples.autodiff.gain_dc",
        "cyfaust.resources.libraries.examples.autodiff.gain_exp",
        "cyfaust.resources.libraries.examples.autodiff.gain_pow",
        "cyfaust.resources.libraries.examples.autodiff.gain_pow_trig",
        "cyfaust.resources.libraries.examples.autodiff.gain_sq",
        "cyfaust.resources.libraries.examples.autodiff.mem",
        "cyfaust.resources.libraries.examples.autodiff.one_zero",
        "cyfaust.resources.libraries.examples.autodiff.recursion",
        "cyfaust.resources.libraries.examples.autodiff.tremolo",
        "cyfaust.resources.libraries.examples.delayEcho",
        "cyfaust.resources.libraries.examples.dynamic",
        "cyfaust.resources.libraries.examples.filtering",
        "cyfaust.resources.libraries.examples.gameaudio",
        "cyfaust.resources.libraries.examples.generator",
        "cyfaust.resources.libraries.examples.misc",
        "cyfaust.resources.libraries.examples.old",
        "cyfaust.resources.libraries.examples.old.rewriting",
        "cyfaust.resources.libraries.examples.phasing",
        "cyfaust.resources.libraries.examples.physicalModeling",
        "cyfaust.resources.libraries.examples.physicalModeling.fds",
        "cyfaust.resources.libraries.examples.physicalModeling.old",
        "cyfaust.resources.libraries.examples.pitchShifting",
        "cyfaust.resources.libraries.examples.psychoacoustic",
        "cyfaust.resources.libraries.examples.quantizing",
        "cyfaust.resources.libraries.examples.reverb",
        "cyfaust.resources.libraries.examples.smartKeyboard",
        "cyfaust.resources.libraries.examples.smartKeyboard.associatedEffects",
        "cyfaust.resources.libraries.examples.spat",
        "cyfaust.resources.architecture",
        "cyfaust.resources.architecture.api",
        "cyfaust.resources.architecture.api.doc",
        "cyfaust.resources.architecture.AU",
        "cyfaust.resources.architecture.AU.AUPublic",
        "cyfaust.resources.architecture.AU.AUPublic.AUBase",
        "cyfaust.resources.architecture.AU.AUPublic.AUEffectBase",
        "cyfaust.resources.architecture.AU.AUPublic.AUInstrumentBase",
        "cyfaust.resources.architecture.AU.AUPublic.Utility",
        "cyfaust.resources.architecture.AU.PublicUtility",
        "cyfaust.resources.architecture.AU.Source",
        "cyfaust.resources.architecture.AU.Source.AUSource",
        "cyfaust.resources.architecture.AU.Source.CocoaUI",
        "cyfaust.resources.architecture.daisy",
        "cyfaust.resources.architecture.faust",
        "cyfaust.resources.architecture.faust.au",
        "cyfaust.resources.architecture.faust.audio",
        "cyfaust.resources.architecture.faust.dsp",
        "cyfaust.resources.architecture.faust.gui",
        "cyfaust.resources.architecture.faust.gui.Styles",
        "cyfaust.resources.architecture.faust.midi",
        "cyfaust.resources.architecture.faust.unity",
        "cyfaust.resources.architecture.faust.vst",
        "cyfaust.resources.architecture.juce",
        "cyfaust.resources.architecture.juce.plugin",
        "cyfaust.resources.architecture.juce.standalone",
        "cyfaust.resources.architecture.sam",
        "cyfaust.resources.architecture.teensy",
        "cyfaust.resources.architecture.vcvrack",
        "cyfaust.resources.architecture.vcvrack.template",
        "cyfaust.resources.architecture.vcvrack.template.res",
        "cyfaust.resources.architecture.vcvrack.template.src",
        "cyfaust.resources.architecture.VST",
    ],
    include_package_data=True
)

@shakfu

@shakfu, have you tried something like the following?

common = {
   ...,
   "include_package_data": True,
   "exclude_package_data": {"": "*.cpp"},
}

if STATIC:
   setup(
       **common,
       ext_modules=[...],
       package_dir={"": "src/static"},
       packages=find_namespace_packages(where="src/static"),
   )
else:
   setup(
       **common,
       ext_modules=[...],
       package_dir={"": "src"},
       packages=find_namespace_packages(where="src", include=["cyfaust*"]),
   )

The example above requires MANIFEST.in to be configured so that all the files necessary for the build process are present in the sdist. Alternatively if you are OK with all the files in your VCS to also be present in your sdist you can use a plugin like setuptools-scm (which potentially simplifies a lot the configuration).

@abravalheri

@abravalheri Thanks for the tip, I'll try it out to see if I can improve it as per your suggestions.

Incidentally, because I have two build variant in my setup.py file, I generate the MANIFEST.in file in each case:

    with open("MANIFEST.in", "w") as f:
        f.write("graft src/cyfaust/resources\n")
        f.write("exclude src/cyfaust/*.cpp\n")

@abravalheri

@abravalheri

Incidentally, I deleted my prior post as it included an error that should not have been flagged here. In any case, the setup.py file is working, checked-in and the deprecation is gone. Thanks very much for your help!

  packages=find_namespace_packages(where="src", include=["cyfaust*"]),

It does not seem good if docs have to recommend the definition of namespace packages for projects that are not using namespace packages.

It does not seem good if docs have to recommend the definition of namespace packages for projects that are not using namespace packages.

This is the essence of the issue: the deprecation makes it necessary to use a more complex solution (find_namespace_packages) to a problem (bundling a data folder in one's package) that was previously relatively simple to resolve.

My case above is illustrative.

@merwok

It does not seem good if docs have to recommend the definition of namespace packages for projects that are not using namespace packages.

@merwok, please note that if a distribution install a directory nested somewhere under sys.path and this directory that does not contain an __init__.py, then that is indeed a namespace package, and the project is, effectively, using namespace packages. You can use your REPL to check that the directory is importable and the REPL will show you something like <module '...' (namespace)> if you print the object you imported.

The assumption that certain directories that don't contain Python files are exempt of this behaviour is not backed by the implementation, and as far as I know, by the Python docs (the PEP introducing implicit namespaces pretty much says the opposite).

The warning message discussed in this issue has to deal with 2 situations:

When the developer does want such directories to be included in the installation and is trying to rely on include_package_data to do so, but is forgetting to list such directories in packages (therefore, it is providing an incorrect configuration that fails match file structure and intent).
When the developer does not want certain directories to be included and is specifying the correct configuration to do so (either by manually listing packages or by using find_namespace_packages(..) with the include/exclude options, or by consciously relying on the fact that find_packages(...) skip directories without __init__.py). At the same time, the developer is correctly specifying include_package_data such that files in other directories are included.

Currently there is a bug in setuptools that will incorrectly handle both situations.

For situation 1, setuptools will include packages that are not listed in packages, and therefore fail to fulfil the configuration that has been passed.

For situation 2, setuptools will not skip certain directories that are intentionally omitted from packages, and therefore also fail to fulfil the configuration that has been passed.

To fix this bug we need the warning message to bring awareness for users that fall into category 1: that their configuration does not match the intent, and for users that fall into category 2: that they will need to use a workaround while the fix has not been implemented yet.

the project is, effectively, using namespace packages

I understand that, but my viewpoint is about author intent.
In the examples here the developers have regular packages, source files and data files.
They need a clean way to have their packages packaged up and installed, and the data files also packaged up.

They need a clean way to have their packages packaged up and installed, and the data files also packaged up.

The clean way is to list any package they are using to host data files in the packages directive. This mental model is aligned with the Python implementation, docs, and work great with importlib.resources.
There is no need for differentiating between types of directories, that is only conceptual overhead that does not really match the way Python works.

BUILD: Resort to namespace packaging for (pseudo-)data directories

Navigation Menu

Repository navigation

package data in subdirectory causes warning #3340

setuptools version

Python version

OS

Additional environment information

Description

Expected behavior

How to Reproduce

Output

48 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

package data in subdirectory causes warning #3340

Description

setuptools version

Python version

OS

Additional environment information

Description

Expected behavior

How to Reproduce

Output

Activity

abravalheri commented on May 29, 2022

jwodder commented on Jun 3, 2022

abravalheri commented on Jun 4, 2022

Footnotes

jwodder commented on Jun 4, 2022

abravalheri commented on Jun 4, 2022

abravalheri commented on Jun 4, 2022

jwodder commented on Jun 4, 2022

cdfarrow commented on Jun 9, 2022

48 remaining items

webknjaz commented on Dec 30, 2023

shakfu commented on Jan 10, 2024

abravalheri commented on Jan 11, 2024

Footnotes

abravalheri commented on Jan 11, 2024

Footnotes

shakfu commented on Jan 12, 2024

shakfu commented on Jan 13, 2024

abravalheri commented on Jan 13, 2024

shakfu commented on Jan 13, 2024

shakfu commented on Jan 14, 2024

merwok commented on Jan 14, 2024

shakfu commented on Jan 14, 2024

abravalheri commented on Jan 14, 2024

merwok commented on Jan 15, 2024

abravalheri commented on Jan 16, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions