TL;DR Most Python projects require quite some tooling to be installed on your laptop before you can build them. Why isn’t it just as easy as
npm install && npm test? Learn how pyprojectx can add reproducible, plug-and-play builds to your Python projects.
The Python ecosystem provides lots of nice development tools like code formatters, linters, dependency managers, etc. Seasoned Pythonistas may have forgotten, but installing them can be daunting for noobs, even frustrating for developers coming from other platforms like Nodejs or Java.
My team recently kicked off its first Python project, using Poetry for dependency management. Half of the team (5 out of 10) struggled with the installation, caused by broken Homebrews, shell issues or … Windows.
What can we do about it?
Stop whining and RTFM
We can document all prerequisites and expect everyone to just follow the instructions, but:
- Potential contributors of your OS project will drop out when it’s too much hassle to create a simple pull request.
- Teammates end up using different versions of tools. But it works on my machine.
Using a docker image, that provides all the required tools, has become popular for CI/CD servers. Fol local development, it’s kind of heavy and as such less popular. The latest license changes of Docker Desktop won’t make it more popular.
With Python, all your (recursive) dependencies end up in the same flat dependency list. So adding dev dependencies impacts your production dependencies. It will typically downgrade some libraries to older (slower, flawed?) versions.
Or worse: they cause dependency conflicts. F.e. I onc tried to add AWS SAM as dev dependency, but failed because of a boto3 version conflicts.
Npm works with a nested dependency tree and can install multiple versions of the same library. That’s why it’s safe to add dev dependencies in npm.
The Gradle way
Gradle, the most popular build tool in Java land, treats reproducible builds as top priority. They release (breaking) changes at rapid pace, but nevertheless projects (or commits) relying on older versions are not impacted.
The principle to achieve this, is as brilliant as it is simple: use wrapper scripts that intercept all commands, install the required Gradle version if necessary and delegate the command to that version. These wrapper scripts (a *nix and a Windows script) are committed to git (or other VCS), together with a config file and a small bootstrap jar.
So when you need to patch an older version of your project that relies on another Gradle version, you don’t have to worry
about it as the correct Gradle will be used, both on developer machines and on CI/CD servers. Just use
in stead off
gradle build and you’re future-proof.
The wrapper scripts however don’t solve dependency conflicts among libraries used by build scripts. Java also lacks the nested dependency trees that npm has. Gradle solves this by grouping dependencies in isolated configurations, allowing different tools to use different versions of common libraries.
By combining some ideas from Gradle and npm, pyprojectx turns a standard pyproject.toml config file into an executable and self-contained build script, with python 3.7+ being the only prerequisite.
You only need to download the wrapper scripts into your project directory (or any empty directory)
- osx / linux :
curl -LO https://github.com/houbie/pyprojectx/releases/latest/download/wrappers.zip && unzip wrappers.zip && rm -f wrappers.zip
- windows: unpack the wrappers zip
and specify which tool(sets) you want to use in pyproject.toml:
[tool.pyprojectx] pdm = "pdm==1.12.6" black = "black==21.7b0" https = "httpie"
./pw --init project to create a new pyproject.toml or to append sample config if the file already exists.
Now you can use the configured tools as usual by merely typing
pw on Windows) in front of them:
./pw pdm --help ./pw https POST pie.dev/post message='Pyprojectx is awesome!'
You can alias commands and simple shell scripts that you use often:
[tool.pyprojectx.aliases] install = "poetry install" run = "poetry run" test = "poetry run pytest" clean = """\ rm -r .venv rm -r dist""" build = "pw@install && pw@test && pw@poetry build"
Aliases also come in handy when writing CI/CD scripts because they are easy to test on your local machine and avoid debugging on the CI/CD server. See pyprojectx own github action build script
Pyprojectx eats its own dog food, so you can try it out yourself if you have python 3.7+ installed:
git clone https://github.com/houbie/pyprojectx.git && cd pyprojectx && ./pw build
How it works
The pw script creates a .pyprojectx directory where all the tools (including pyprojectx itself) are installed in isolated virtual environments (cfg. npm’s node_modules).
The tool.pyprojectx values are expected to be in Requirements File Format, leaving it free to/how specify versions and to install multiple tools together. A hash of the requirements is used to determine the virtual environment’s directory. So if you change the version of a tool, it will get re-installed.
The pw script guesses the virtual environment to use based on the provided command, so
./pw https --args delegates
to the https script in the virtual environment containing httpie.
Invoking other scripts provided by the same virtual environment can be achieved with aliases that explicitly select
a tool.pyprojectx tool entry:
[tool.pyprojectx.aliases] # httpie provides both http and https scripts http = "@https:http" post-json = "@https:http POST pie.dev/post message=Awesome!"
./pw http POST pie.dev/post message='a message' will fail because the http alias doesn’t cope with the quotes and
spaces in the command. This limitation still needs to be solved, but as workaround you can either create an extra
http = "httpie" or create an alias that includes all arguments (as in the post-json example above).
Pyprojectx can bring the power of npm dev dependencies and npm scripts to Python, helping to get developers faster onboarded.
No more make files, finally!