python pipy.org and pyproject.toml

Date: 2024/02/03 (initial publish), 2024/02/23 (last update)

Source: en/note-00063.md

Previous Post Top Next Post

TOC

I realized the main stream packaging practice now uses pyproject.toml instead of setup.cfg or setup.py.

I wanted to upload a package imediff to pipy.org repository while converting it to use pyproject.toml. Getting confused by the documentation sites which are still actively updated to catch up with the new style to use pyproject.toml etc., I managed to upload a package to pypi.org.

Here are the list of sites which I visited:

In retrospective, I found “An Updated Guide to Setuptools and Pyproject.toml (2023-12-19)” which documents this very concisely.

These documents and the example as the following lead me to sucess:

The end result of my imediff package is available as version 2.8 at:

The source for the debian deb-package is at debian branch.

Key point for successful upload

Use of twine check dist/* was good way to get it acceptable to pypi.org .

Please note that the LICENSE file needs to be not only a clean UTF-8 file but also remove some ASCII control characters (TAB, LF, CR are OK to be there But no FF = ^L like GPL2).

Pypi.org package upload procedures

Python virtual environment

Create and start virtual environment in ~/.venv:

 $ python3 -m venv ~/.venv
 $ . ~/.venv/bin/activate

Please note . is significant here.

pip

Install a pip package <package_name> with:

 $ pip install -U <package_name>

build package

We need to set up access to the upload tool twine:

keyring package

We need to set up access to the keyring tool keyring (backend secret-tool):

twine package

We need to set up access to the upload tool twine:

Update all packages to the latest

Update all packages in virtual environment:

 $ pip list --outdated |tail -n +3|cut -d ' ' -f 1|xargs -n1 pip install -U

(I used this to avoid issues of using older tool chain while facing failed upload as described below. It seems the root cause was not the older tool chain.)

Build package

Build a python package:

 $ cd path/to/project_root
 $ python3 -m build
 $ twine check dist/*

Get API token to the keyring

I got API tokens following their instructions:

I followed Twine: Keyring Support to store API tokens into keyring.

For test pypi:

 $ keyring set https://test.pypi.org/legacy/ __token__
Password for '__token__' in 'https://test.pypi.org/legacy/': pypi-token-value

For real pypi:

 $ keyring set https://upload.pypi.org/legacy/ __token__
Password for '__token__' in 'https://upload.pypi.org/legacy/': pypi-token-value

These are essential steps to make smooth uploads of packages without prompted by twine as mentioned in “Python Packaging User Guide: Uploading the distribution archives”.

Use of ~/pypirc to avoid being prompted for your username and password every time as mentioned in Setting up TestPyPI in .pypirc doesn’t work on my Debian system. So use of keyring mentioned here is needed.

Upload of a python package

Test upload of a python package:

 $ twine upload --verbose --repository testpypi dist/*

Official upload of a python package:

 $ twine upload --verbose --repository pypi dist/*

Leave python virtual environment

Deactivate virtual environment:

 $ deactivate

Retrospective on the upload failures

The real root cause of issue was the existance of FF codes in the GPL2 LICENSE file. See https://github.com/pypa/sampleproject/issues/209

Note on packaging shell command git-ime, manpages, …

Initially, Packaging Python Projects didn’t give me easy path to package shell command git-ime, manpages, … into pipy.org compatible wheel package, I looked around how others do.

My reading of this thread is:

The Python packaging standards are focused on packaging Python code in a portable way - shell scripts are inherently platform-specific, and as such the current thinking is that they should be shipped using platform-specific mechanisms (.deb/.rpm files or similar) rather than Python wheels.

But, nothing stopping you from shipping a wheel (with a custom build backend, or manually put together) with the shell script in the .data/scripts directory.

I realize I can have a manually run additional script as my post-installer by creating an entry point imediff_install in addition to the main imediff as:

[project.scripts]  # Optional
imediff = "imediff:main"
imediff_install = "imediff:install"

Interesting programs

cibuildwheel

cibuildwheel runs on your CI server - currently it supports GitHub Actions, Azure Pipelines, Travis CI, AppVeyor, CircleCI, and GitLab CI - and it builds and tests your wheels across all of your platforms.

sh

sh is a full-fledged subprocess replacement for Python 3.8 - 3.11, and PyPy that allows you to call any program as if it were a function.

Previous Post Top Next Post