# Developer Workflow

## Overview

This repository uses custom tooling based on shell (Bash) scripts and Docker to unify local and CI/CD workflows.
The idea is that by running the developer workflow in a container, when those commands are run on CI/CD servers they will be executed in the exact same environment.
While this does create an explicit dependency on a container runtime like Docker or Podman,
it reduces the variance between local and remote environments and thus reduces the likelihood of "well, it works on my machine."
Unifying local and remote workflows is not a novel concept, but the build system in this repo is custom-built.
This document describes in detail how to use and customize it. To skip the explanation and get started immediately, skip straight to [Initializing the Repo](#initializing-the-repo).

## Requirements

The build system currently only supports Linux (native or WSL) and macOS, and requires [Docker](https://www.docker.com/) or another container runtime with a similar CLI.
If you're using an editor like Vim/Neovim and want to create a local virtual environment for code completion, you'll also need a local Python installation.
That's it! All other development dependencies are installed in the build container.

## Build Container

The build container, defined in [build-support/docker/Dockerfile](../../build-support/docker/Dockerfile), is based on [Alpine Linux](https://hub.docker.com/_/alpine).
The repository files are mounted into the container before each command is run, so the container is not rebuilt after any source files are changed.
To add tools or make other changes to the build container, edit the `build` or any preceding stage in the [Dockerfile](../../build-support/docker/Dockerfile)
and rebuild it (using the command `./run.sh build-base`, see below).

## Build CLI

All development commands are run via the [run.sh](../../run.sh) script, which comes with many common commands built-in for Python development (detailed below).
Certain build settings are configurable via [config.sh](../../build-support/shell/run/config.sh), and each setting is documented in that file.
If you are not using Docker, make sure to update the `CONTAINER_RUNTIME` setting in `config.sh` before continuing.

### Initializing the Repo

Once you have installed a container runtime and it's working properly, clear the workspace and build the build container:

```bash
./run.sh -l clean && ./run.sh build-base
```

Initialize the project with a project name and description, module name, author, and source code license:

```bash
./run.sh init
```

If you are using an editor like Vim/Neovim and need a virtual environment for code completion, run the `editor-venv` command:

```bash
./run.sh editor-venv
```

This creates a virtual environment in `build-support/python/virtualenvs/editor-venv/`. See [.ycm_extra_conf.py](../../.ycm_extra_conf.py)
(for the YouCompleteMe plugin) for how to point a Vim/Neovim plugin to the virtual environment.

Now you're ready to start hacking!

### The Inner Loop 

The common set of steps you might want to perform while writing code are formatting, type checking, linting, testing, building, and publishing.
The default command, `build`, checks the format and types, lints the code, runs unit tests, and builds a distribution package:

```bash
./run.sh # equivalent to ./run.sh build
```

This command should be run before comitting any code to the repo, and can be used in precommit hooks or CI/CD scripts.
To format code before checking, run the `fmt` command before `build`:

```bash
./run.sh fmt \
    && ./run.sh build
```

Once you're ready to _publish_ your code to [PyPI](https://pypi.org/) or another registry, and compile the documentation, run:

```bash
./run.sh publish \
    && ./run.sh make-docs
```

### Build CLI Commands

The table below contains all the built-in commands, their usage, and a brief description.

| Command     | Usage                        | Description                                                                                                                                                                              |
|-------------|------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| build       | `./run.sh build`             | Build distribution packages. Before building, this command will check the code format, run the type checker and linter, and run unit and integration tests. This is the default command. |
| build-base  | `./run.sh build-base`        | Build the build container image.                                                                                                                                                         |
| check       | `./run.sh check`             | Type check code with [MyPy](https://mypy.readthedocs.io/en/stable/introduction.html).                                                                                                    |
| clean       | `./run.sh clean`             | Clean the workspace by removing the build virtual environment, compiled documentation and packages, and local caches.                                                                    |
| editor-venv | `./run.sh editor-venv`       | Create or update a local virtual environment for an editor (i.e., [Vim](https://www.vim.org/)/[Neovim](https://neovim.io/)).                                                             |
| exec        | `./run.sh exec COMMAND`      | Execute arbitrary shell command (`COMMAND`) in virtual environment (using Poetry).                                                                                                       |
| fmt         | `./run.sh fmt`               | Format code with [Black](https://black.readthedocs.io/en/stable/).                                                                                                                       |
| init        | `./run.sh init`              | Initialize repository with project name and description, module name, author, and source code license (should only be run once - see [Initializing the Repo](#initializing-the-repo)).   |
| lint        | `./run.sh lint`              | Lint code with [Pylint](https://pylint.pycqa.org/en/latest/intro.html).                                                                                                                  |
| make-docs   | `./run.sh make-docs`         | Compile documentation with [Sphinx](https://www.sphinx-doc.org/en/master/).                                                                                                              |
| publish     | `./run.sh publish`           | Publish distribution packages to PyPI (or another registry).                                                                                                                             |
| push-base   | `./run.sh push-base`         | Push the build container image to the configured container registry (in [config.sh](../../build-support/shell/run/config.sh)).                                                                 |
| shell       | `./run.sh shell`             | Sart a Python shell in the build virtual environment.                                                                                                                                    |
| test        | `./run.sh test`              | Run unit and documentation tests with [Pytest](https://docs.pytest.org/en/latest/).                                                                                                      |
| update-deps | `./run.sh update-deps`       | Update direct and dev dependencies in build virtual environment.                                                                                                                         |
| version     | `./run.sh version [VERSION]` | If a version (`VERSION`) is specified, updates the package version. Otherwise, shows the current package version.                                                                        |

### Adding Commands

Adding a command to the build CLI is designed to be simple, and only requires adding a function named `run-COMMAND` to [run.sh](../../run.sh).
For example, the following snippet adds the command `bandit` to run the [Bandit security tool](https://bandit.readthedocs.io/en/latest/)
(after adding it as a dev dependency to [pyproject.toml](../../pyproject.toml)):

```bash
run-bandit() {
    info "Running security scan with Bandit"
    poetry run bandit
}
```

By default, commands will be run use the default Python version set in `config.sh`. To run a command in every support Python version, or locally-only, edit the `run-command` function in `run.sh`.
After adding a command, make sure to update the usage in the `print-usage()` function and the [commands table](#build-cli-commands) above.

## Writing Tests

Tests are run by [Pytest](https://docs.pytest.org/en/latest/), and should be placed in the `tests/<module_name>` directory.
All files named `test_*.py` in that directory that contain tests will be discovered by Pytest.
See the [Pytest discovery documentation](https://docs.pytest.org/en/6.2.x/example/pythoncollection.html) for more information.
Examples in docstrings are also tested by [doctest](https://docs.python.org/3/library/doctest.html) automatically, which keeps examples up-to-date and accurate.

**NOTE:** Support for [Dataclass Transforms](https://peps.python.org/pep-0681) was not added until Python 3.11.
Consequently, using the `@taskclass` decorator in this package will cause MyPy to wine about invalid task constructor arguments (`[call-arg]`).
These errors can safely be ignored.