Python packaging edge cases

TL;DR: You cannot package a Python module that depends on local wheel dependencies.

Python packaging is one of those things that doesn’t work in some cases, and I’m going to demonstrate one such case in this post. You can find the code examples from this post in my Github repository here.

Photo by Jon Cartagena on Unsplash

How do you build a Python wheel file that depends on another local wheel file?

Let’s say that you have a wheel file that you want to include as part of a new Python package (maybe the wheel file is a company’s internal package that you cannot upload to pypi or to an internal python package hosting platform, for whatever reason).

How would you do it? You could try directly referencing the local wheel file. Here is the code example of setup.py

import os
from setuptools import find_packages, setup

# add wheel file as dependency
install_requires = [
    f"dummy_wheel @ file://localhost/"
    f"{os.path.join(os.getcwd(),'packages', 'dummy_wheel-1.0.0-py3-none-any.whl')}"
]

setup(
    name="project",
    version="1.0.0",
    url="url",
    include_package_data=True,
    packages=find_packages(),
    python_requires=">=3.8, <4",
    install_requires=install_requires,
)

where install_requires is acting as though dummy_wheel is hosted on localhost. You would have your wheel file under the packages directory, like this.

This seems like it should work, but the newly-created package only works for the person who built it, and no one else can install from it. It’s not very useful if I’m the only one who can use it!

Let’s say I (user1) build the Python package project in my local environment (or using CI).

git clone https://github.com/921kiyo/build-wheel-that-depends-on-another-local-wheel.git
cd build-wheel-that-depends-on-another-local-wheel
python setup.py sdist bdist_wheel

This creates a Python package wheel file in dist/project-1.0.0-py3-none-any.whl.

Now I send the project-1.0.0-py3-none-any.whl file to my colleague (say, user2), and they try to pip install from the wheel file with pip install project-1.0.0-py3-none-any.whl in their environment. They would get an error as follows:

ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '//Users/<user1>/build-wheel-that-depends-on-another-wheel/packages/dummy_wheel-1.0.0-py3-none-any.whl'

Okay, so why are they getting an error relating to user1’s path? This is because when packaging project in user1’s environment, Python will find all the dependencies and include them in the METADATA file. But of course user1’s path for installing dummy_wheel won’t work in user2’s environment. You can check METADATA using importlib.metadata (https://docs.python.org/3.8/library/importlib.metadata.html) as follows (as long as project is already installed):

>>> from importlib import metadata
>>> dict(metadata.metadata("project"))
{'Metadata-Version': '2.1',
 'Name': 'project',
 'Version': '1.0.0',
 'Summary': 'UNKNOWN',
 'Home-page': 'url',
 'License': 'UNKNOWN',
 'Platform': 'UNKNOWN',
 'Requires-Python': '>=3.6, <4',
 'Requires-Dist': 'dummy-wheel @ file://localhost//Users/<user1>/build-wheel-that-depends-on-another-local-wheel/packages/dummy_wheel-1.0.0-py3-none-any.whl'}

Look at the Requires-Dist value in the dictionary. This is where user1’s local path has been included in the package, meaning the packaged dependency dummy-wheel won’t work in user2’s environment.

Okay, so how about using a relative path to the wheel dependency dummy-wheel in order to avoid having the Users/<user1>/... path in the METADATA? Here is the modification.

In setup_modified.py…

from setuptools import find_packages, setup

with open("requirements.txt", "r", encoding="utf-8") as f:
    install_requires = [x.strip() for x in f]

setup(
    name="project",
    version="1.0.0",
    url="url",
    include_package_data=True,
    packages=find_packages(),
    python_requires=">=3.8, <4",
    install_requires=install_requires,
)

…add the relative path in requirements.txt:

./packages/dummy_wheel-1.0.0-py3-none-any.whl

This looks good, so let’s do the packaging with python setup_modified.py sdist bdist_wheel. However, this gives us an error as follows:

python setup_modified.py sdist bdist_wheel
error in project setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Parse error at "'./packag'": Expected W:(abcd...)

Unfortunately you cannot use the relative path, because PEP-508 prohibits the use of relative paths when specifying dependencies. The dependencies need to be an absolute path.

There are three workarounds:

1. Instead of using a binary distribution (bdist_wheel), you could use a source distribution instead (sdist). This means user2 will have to directly install from dummy_wheel’s source code.
2. user2 could directly pip install the dependency wheel file (dummy-wheel) separately before doing pip install project-1.0.0-py3-none-any.whl.
3. You could upload the wheel file dummy_wheel-1.0.0-py3-none-any.whl to somewhere, such as pypi, so user2 can add dummy_wheel in requirements.txt. Then running python setup_modified.py sdist bdist_wheel would fetch dummy_wheel from pypi and include it as part of the dependencies.

Basically, in Python you cannot package a module that depends on another local wheel as a dependency. If anyone has a better workaround, please let me know. :)

Subscribe to my blog

* indicates required

Email Address *

Thank you for reading my blog post! If you liked it, please subscribe to receive new posts. I won't give your address to anyone else, won't send you any spam, and you can unsubscribe at any time.