Tag: Python

Solve pytest import error (while python can)

After refactoring a formerly monolithic python script into several files, I started getting problems related to module imports. I shifted the source code into a package directory, and the test/ directory was now parallel to the package source directory. The full directory layout was like this:

<packagename>
|- <packagename>/
|  |- __init__.py
|  |- module_1.py
|  `- ...
|- test/
|  |- __init__.py
|  |- test_module1.py
|  `- ...
|- __init__.py
|- README.md
`- packagename.py

When running “pytest <packagename>”, I got:

ImportError while importing test module '...'

However, when running “packagename.py” (executable python script via shebang), everything worked fine. Both the test modules and the main script contained the same import statement:

from packagename import module_1

A google search turned up: https://stackoverflow.com/questions/41748464/pytest-cannot-import-module-while-python-can, which pointed me in the direction that the issue must be related to __init__.py

However, the solution was not to delete __init__.py in the test folder, but to delete __init__.py in the main package folder. I came to this solution based on the documentation of pytest: https://docs.pytest.org/en/latest/explanation/goodpractices.html#tests-as-part-of-application-code

The working layout is:

<packagename>
|- <packagename>/
|  |- __init__.py
|  |- module_1.py
|  `- ...
|- test/
|  |- __init__.py
|  |- test_module1.py
|  `- ...
|- README.md
`- packagename.py

Disclaimer:

I do not want to advertise this directory layout for python packages. In my use case, I have a python script inside a very large repository that is not distributed as a package outside of this repository; I just wanted to improve maintainability.
When you want to create a python package for distribution, you’re faced with many other considerations. There are two competing package structure styles: With or without the package sources inside a “src/” folder. The python packing guide advertises the “src/” folder layout. See https://packaging.python.org/en/latest/tutorials/packaging-projects/#a-simple-project for an example of “<packagename>/src/<packagename>” layout. If in doubt, go with one of the more popular cookiecutter templates.

Install pip packages from git repo

Pip supports installing a python package using a link to a git repository directly. You can specify such a direct link to the package source from the pip command line or a requirements.txt file. The following article gives an overview how.

It can be handy to use pip to install a project dependency directly from a git repository instead of from a Python package index. I’ll show you why you might want to do that and how to do it.

How to pip install from a git repository

Instead of a package name, give pip a git repository URL as parameter:

# general form: pip install git+<repository_url>
# example:
pip install git+https://github.com/arwedus/some-example.git

If this repo does not yet contain a python package struture, you’ll need to add a setup.py at least, so that pip can carry out the install (c.f. cookie-cutter).

You can explicitly call out the package name that you’re installing with #egg=

# general form: pip install git+<repository_url>#egg=<package_name>
# example:
pip install git+https://github.com/arwedus/some-example.git#egg=example-package

There are also a number of different ways to specify a version of the repository that you want to fetch:

# Use a commit SHA
pip install git+https://github.com/arwedus/some-example.git@4045597
# Use a tag
pip install git+https://github.com/arwedus/[email protected]
# Use a branch
pip install git+https://github.com/arwedus/some-example.git@feature/fix-readme

How to include the dependency in a requirements.txt

If you want to share a git repository dependency with other developers, you’ll likely want to add it to your requirements.txt, like so:

# Just put the pip install argument straight into your requirements.txt
package-a==1.2.3
git+https://github.com/arwedus/[email protected]
package-b==4.5.6

# Or you can use the preferred PEP 440 direct URL syntax
package-a==1.2.3
some-example @ git+https://github.com/arwedus/some-example.git@deadbeef123

Python Pitfalls: References to the same dictionary

Today, I tried to build a dictionary which contains the contents of a list as keys and empty dictionarys as values. First, I went with the method dict.fromkeys(seq, [value]), then I filled the sub-dictionaries in a for loop, like this:

mydict = dict.fromkeys(lstLabels, {})
for strLabel in dict.keys():
    mydict[strLabel][strLabel] = "x"

What I ended up with was that the sub-dictionaries referenced by the keys from lstLabels all referenced to the same dictionary instance. What this means is, that whenever you add something to dict[lstLabels[0]], it appears in dict[lstLabels[1]], too, and so on.

The correct way to accomplish what I wanted – different instances of subdirectories – was this:

    for strLabel in lstLabels:
        mydict[strLabel] = {}
        mydict[strLabel][strLabel] = "x"

Apparently, the dict.fromkeys() method does something like lstFail = [[] * 3]

Are there more elegant ways to create a dictionary with subdictionarys based on a list of strings than the one presented?

Installing Python packages with easy_install

Python comes with a nice tool called easy_install that enables you to install additional packages as easy as:

easy_install <package_name>

If you are behind a HTTP proxy server, this might fail unless you specify the proxy URL in the environment variable HTTP_PROXY like this:

set http_proxy=”http://www.myproxy.org&#8221;

However, when you are sitting behind a firewall (like we at my company are), you might still not be able to download any packages from the python package repository. Here’s a quick guide how to circumvent the issue:

  1. Create a folder on your local machine that will contain all packages you have to download.
  2. Go to http://pypi.python.org/simple/ and download the package you want to install into your local folder.
  3. In the scripts directory of your python installation, call:
    easy_install -H none -f d:\Backup\Programs\python-packages <package_name>
  4. When you disallow all foreign hosts with “-H none”, easy_install will not be able to fetch any packages your package depends on, so you have to resolve all dependency errors by repeating step 1 – 3 for the missing package if easy_install gives you a message like this:
    “No local packages or download links found for logilab-common>=0.49.0”

Another solution I found on the internet would be to install and use ASProxy, but I’m not going to try that for now.

See also: easy_install guide