Devpi: Your own pypi index

This is a pre-cursor to a longer follow up post I am currently working on covering namespace packaging in Python 2.7+.

The Devpi documentation covers the topic of setting up and deploying your own pypi index, what I would like to cover is why I like to have my own pypi index.

Managing complexity is one of those things developers have to deal with at every level. You have to worry about whether a line, function, class, module or package has the right amount of responsibility. The larger your code base becomes, the more you start to worry about the package level of complexity. This is when I like to split packages out into their own libraries. These libraries can be maintained, tested and versioned independently.

Versioning Packages

Semver with a sprinkling of of PEP-440

Formalising on Semver:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes, MINOR version when you add functionality in a backwards-compatible manner, and PATCH version when you make backwards-compatible bug fixes.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format. > - http://semver.org/

It is sane approach versioning. You can read more about usage and benefits on the Semver site.

Where I do deviate from semver is with the additional labels of pre and post releases. I instead follow PEP-440 with the following release formats as the hyphen separator of semver is not valid with the PEP:

X.YaN        # Alpha release
X.YbN        # Beta release
X.YrcN       # Release Candidate
X.Y          # Final release

X.Y.postN    # Post-release (ignored)

I have never come up on an occasion to reach for PEP-440 post-release recommendations as Semver reserves the final number only for PATCH releases.

Multiple Indexes

Currently we are using our own Pypi index server implementation and are mid way through migrating to Devpi. We simply do not have the resources to maintain our own implementation when there is an open source and actively maintained alternative available. I would like to briefly cover how I plan to utilise Devpi’s multiple indexes for some added protection.

Devpi has index inheritance:

index inheritance: Each index can inherit packages from another index, including the pypi cache root/pypi. This allows to have development indexes that also contain all releases from a production index. All privately uploaded packages will by default inhibit lookups from pypi, allowing to stay safe from an attacker who could otherwise upload malicious release files to the public PyPI index. > - Devpi Docs

So the plan is to have a dev and prod index. Prod will be a child of root/pypi and dev will be a child of prod. Our production and staging environments will use the prod index to fetch its packages. All other environments will look at the dev index.

Alpha/Beta/Release Candidate pre-release packages will only be deployed to the dev index, making them inaccessible in production and staging environments. There will be no possibility for a non-final package to make it into the staging or production environment - a useful free sanity check.

We also keep our prod index clean of non-final mutable packages. I say mutable because often we do overwrite existing pre-release alpha packages intentionally. We do not make any promises that an alpha package is any way reliable - it is an alpha after all.