Developer Workflow¶
Overview¶
This repository uses custom tooling based on shell (Bash) scripts and Docker to unify local and CI/CD workflows. The idea is that by running the developer workflow in a container, when those commands are run on CI/CD servers they will be executed in the exact same environment. While this does create an explicit dependency on a container runtime like Docker or Podman, it reduces the variance between local and remote environments and thus reduces the likelihood of “well, it works on my machine.” Unifying local and remote workflows is not a novel concept, but the build system in this repo is custom-built. This document describes in detail how to use and customize it. To skip the explanation and get started immediately, skip straight to Initializing the Repo.
Requirements¶
The build system currently only supports Linux (native or WSL) and macOS, and requires Docker or another container runtime with a similar CLI. If you’re using an editor like Vim/Neovim and want to create a local virtual environment for code completion, you’ll also need a local Python installation. That’s it! All other development dependencies are installed in the build container.
Build Container¶
The build container, defined in build-support/docker/Dockerfile, is based on Alpine Linux. The repository files are mounted into the container before each command is run, so the container is not rebuilt after any source files are changed. To add tools or make other changes to the build container, edit the build or any preceding stage in the Dockerfile and rebuild it (using the command ./run.sh build-base, see below).
Build CLI¶
All development commands are run via the run.sh script, which comes with many common commands built-in for Python development (detailed below). Certain build settings are configurable via config.sh, and each setting is documented in that file. If you are not using Docker, make sure to update the CONTAINER_RUNTIME setting in config.sh before continuing.
Initializing the Repo¶
Once you have installed a container runtime and it’s working properly, clear the workspace and build the build container:
./run.sh -l clean && ./run.sh build-base
Initialize the project with a project name and description, module name, author, and source code license:
./run.sh init
If you are using an editor like Vim/Neovim and need a virtual environment for code completion, run the editor-venv command:
./run.sh editor-venv
This creates a virtual environment in build-support/python/virtualenvs/editor-venv/. See .ycm_extra_conf.py (for the YouCompleteMe plugin) for how to point a Vim/Neovim plugin to the virtual environment.
Now you’re ready to start hacking!
The Inner Loop¶
The common set of steps you might want to perform while writing code are formatting, type checking, linting, testing, building, and publishing. The default command, build, checks the format and types, lints the code, runs unit tests, and builds a distribution package:
./run.sh # equivalent to ./run.sh build
This command should be run before comitting any code to the repo, and can be used in precommit hooks or CI/CD scripts. To format code before checking, run the fmt command before build:
./run.sh fmt \
&& ./run.sh build
Once you’re ready to publish your code to PyPI or another registry, and compile the documentation, run:
./run.sh publish \
&& ./run.sh make-docs
Build CLI Commands¶
The table below contains all the built-in commands, their usage, and a brief description.
Command | Usage | Description |
|---|---|---|
build |
| Build distribution packages. Before building, this command will check the code format, run the type checker and linter, and run unit and integration tests. This is the default command. |
build-base |
| Build the build container image. |
check |
| Type check code with MyPy. |
clean |
| Clean the workspace by removing the build virtual environment, compiled documentation and packages, and local caches. |
editor-venv |
| Create or update a local virtual environment for an editor (i.e., Vim/Neovim). |
exec |
| Execute arbitrary shell command ( |
fmt |
| Format code with Black. |
init |
| Initialize repository with project name and description, module name, author, and source code license (should only be run once - see Initializing the Repo). |
lint |
| Lint code with Pylint. |
make-docs |
| Compile documentation with Sphinx. |
publish |
| Publish distribution packages to PyPI (or another registry). |
push-base |
| Push the build container image to the configured container registry (in config.sh). |
shell |
| Sart a Python shell in the build virtual environment. |
test |
| Run unit and documentation tests with Pytest. |
update-deps |
| Update direct and dev dependencies in build virtual environment. |
version |
| If a version ( |
Adding Commands¶
Adding a command to the build CLI is designed to be simple, and only requires adding a function named run-COMMAND to run.sh. For example, the following snippet adds the command bandit to run the Bandit security tool (after adding it as a dev dependency to pyproject.toml):
run-bandit() {
info "Running security scan with Bandit"
poetry run bandit
}
By default, commands will be run use the default Python version set in config.sh. To run a command in every support Python version, or locally-only, edit the run-command function in run.sh. After adding a command, make sure to update the usage in the print-usage() function and the commands table above.
Writing Tests¶
Tests are run by Pytest, and should be placed in the tests/<module_name> directory. All files named test_*.py in that directory that contain tests will be discovered by Pytest. See the Pytest discovery documentation for more information. Examples in docstrings are also tested by doctest automatically, which keeps examples up-to-date and accurate.
NOTE: Support for Dataclass Transforms was not added until Python 3.11. Consequently, using the @taskclass decorator in this package will cause MyPy to wine about invalid task constructor arguments ([call-arg]). These errors can safely be ignored.