Pymantic Release, Part 1: One Repo, Sixty Modules, and the Lie of a Single Version Number

There is a particular flavour of meeting that happens in every infrastructure team eventually. Someone says the words “we should probably version the modules properly,” everyone nods, and then nothing happens for eight months because the alternative — actually doing it — involves reading the semantic-release source code and discovering it was written for npm and assumes exactly one artefact per repository.

terraform-modules has fifty-seven of them.

modules/
├── application_gateway/
├── availability_set/
├── bootstrap/
├── container_registry/
├── key_vault/
├── key_vault_seeder/
├── load_balancer/
├── network/
├── network_peering/
└── ... (another 48 of these)

Each one is consumed independently. estate pulls bootstrap into every subscription it manages. network is pulled by roughly everything. key_vault_seeder is pulled by three things and understood by one person. These modules change at wildly different rates, for wildly different reasons, and — this is the important part — breaking one of them must not force a version bump on the other fifty-six.

The Layout

Everything in this series depends on two repositories, so here is the shape of both. They’re private, and the scripts are the portable part — the layout is what makes the scripts make sense.

The module repository is the producer. It holds the modules, the fixtures that test them, and the release tooling:

terraform-modules/
├── modules/                          ← the released artefacts, one directory per module
│   ├── bootstrap/
│   ├── network/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   ├── variables_region.tf  →  ../../shared/vars/variables_region.tf   (symlink)
│   │   └── README.md                 ← regenerated by terraform-docs
│   └── ... (55 more)
├── examples/                         ← one per module: test fixtures, never shipped
│   ├── bootstrap/
│   └── network/
│       ├── vnet.tf
│       └── unit.tftest.hcl
├── shared/                           ← variable + locals definitions, symlinked into modules
│   ├── locals/
│   └── vars/
│       ├── variables_region.tf
│       └── variables_resource_group.tf
├── bin/
│   ├── pymantic-release.py           ← the subject of this post
│   └── pr-release-summary.py
└── .github/workflows/release.yml

Two branches matter: main, where work lands, and release, where tags get published. Nothing else in the repo is load-bearing for versioning.

The consumer repository is a separate repo that pulls those modules by tag. It gets its own post, but it’s worth knowing it exists:

estate/
├── module_versions.json              ← which tag of each module this estate runs
├── subscriptions/
│   ├── app-a/prod/
│   │   ├── network.tf
│   │   ├── bootstrap.tf
│   │   └── terragrunt.hcl
│   └── ... (twenty-ish more)
└── bin/
    └── module_configurator.py

The only connection between them is a git tag and a ?ref= string. That’s the whole interface, and it’s why the tag scheme matters so much.

The Problem, Stated Plainly

Terraform consumes a module from git like this:

module "bootstrap" {
  source = "git::https://github.com/example-org/terraform-modules.git//modules/bootstrap?ref=bootstrap/v1.14.2"
}

The ?ref= is a git ref. A tag. That’s the entire versioning contract — Terraform doesn’t have a registry protocol here, doesn’t do version constraints, doesn’t do resolution. It does git checkout <ref> and reads a subdirectory. That’s it.

So the unit of versioning is the tag, and the unit of consumption is the subdirectory. Which means the version scheme has to be:

<module name>/v<major>.<minor>.<patch>

key_vault/v1.0.0
network/v1.8.0
bootstrap/v1.14.2

Fine. Easy. Now go and generate ~57 of those, correctly, on every release, based on what actually changed, forever, without a human deciding it.

Why the off-the-shelf tools were out

We looked. Genuinely, we looked.

• semantic-release — one version per repo. The plugin ecosystem for monorepos exists, is npm-shaped, and would have required a package.json per Terraform module. I am not putting a package.json in modules/network/ and neither are you.
• release-please — closer, has real monorepo support via a manifest. Also wants you to adopt Conventional Commits properly, and wants to own a .release-please-manifest.json that it commits back to your default branch. We already had a promotion branch model, which it has opinions about.
• Doing it by hand — this was the incumbent. git tag key_vault/v1.2.0 && git push --tags. Works great until someone tags v1.2.0 instead of key_vault/v1.2.0 and every consumer’s ?ref= lookup starts resolving to a tag that means nothing.

The actual dealbreaker for all of the above was subtler: none of them can answer the only question that matters here, which is “did this commit change the bytes that a consumer of module X actually pulls?” They reason about commit messages. We needed something that reasons about commit messages and file paths.

So I wrote one. About 300 lines of Python, sat in bin/, run once per module by the release workflow.

How It Actually Works

The mental model is deliberately dumb, which is the nicest thing you can say about a release tool.

There are two long-lived branches. main is where work lands. release is where releases happen. Promotion is a PR from main → release, and merging that PR is what publishes tags. Nothing is released until someone with the right permissions clicks a green button, which is a property you appreciate a great deal the first time a release goes wrong.

For each module, on each run, the script does five things:

1. Find where the module currently is.

def get_tags(self):
    """Get all tags from repository, filtered to contain the module name, and sorted."""
    return sorted(
        [tag.name for tag in self.repository.tags
         if self.module_name == tag.name.split(self.version_separator)[0]],
        key=self.sort_versions
    )

If there are no tags for this module, the “latest tag” is a synthetic <module>/v0.0.0, which is the flag for this has never been released and triggers an Initial release at v1.0.0.

2. Walk every commit since that tag. git log <latest_tag>..HEAD, via GitPython.

3. Classify each commit by its subject line. This is the bit that maps human intent to a version component:

lookup_type = {
  "Feature(Breaking):": "Major",
  "Update(Terraform):": "Major",
  "Major:"            : "Major",
  "Feature:"          : "Minor",
  "Update(Provider):" : "Minor",
  "Fix(Bug):"         : "Minor",
  "Minor:"            : "Minor",
  "Fix(Hot):"         : "Patch",
  "Refactor:"         : "Patch",
  "Patch:"            : "Patch",
  "Test:"             : "None",
  "Docs:"             : "None",
  "None:"             : "None",
}

Yes, it’s Conventional Commits wearing a different hat. No, we didn’t use Conventional Commits, because feat: and fix: do not carry the distinction between a bug fix that changes behaviour (Fix(Bug): → Minor) and a bug fix that restores intended behaviour (Fix(Hot): → Patch). That distinction matters enormously when your consumers are production networks and your “patch” release just changed a subnet delegation.

4. Ask whether the commit is relevant to this module. More on this below, because it is where all the blood is.

5. Take the maximum. Every relevant commit gets a weight — Major 3, Minor 2, Patch 1, None 0 — and the release type is the highest weight in the set. Lower components reset to zero on a bump, which is the one piece of actual semver arithmetic in the file:

for idx, value in enumerate(['Major', 'Minor', 'Patch']):
    if null_flag:
        versions.update({value: 0})
    elif value == self.release_type:
        versions.update({value: int(version_set[idx]) + 1})
        null_flag = True
    else:
        versions.update({value: int(version_set[idx])})

If the highest weight is 0, the script prints No release required. and exits 0. Most modules, on most releases, do exactly this. That’s the point — a release run touches fifty-seven modules and publishes four tags.

git log network/v1.7.1..HEAD → release type

1. 01 Docs: regenerate README tables None · 0
2. 02 Refactor: tidy locals Patch · 1
3. 03 Fix(Hot): restore default route on rebuild Patch · 1
4. 04 Feature: subnet delegation model Minor · 2
5. 05 Test: add tftest fixtures None · 0

max(weight)

None

Patch

Minor

Major

release

network/v1.7.1

network/v1.8.0

Minor · lower components reset

The Workflow

The GitHub Actions side is unglamorous, which is correct. It enumerates examples/ to build a matrix — one job per module — and runs the script once per module in parallel:

  get-examples:
    outputs:
      dirs: ${{ steps.list_dirs.outputs.dirs }}
    steps:
      - name: List Directories
        working-directory: ./examples
        run: |
          dirs=$(tree -J -d -L 1 | jq -c '.[0].contents | map(.name)')
          echo "dirs=$dirs" >> $GITHUB_OUTPUT

  release-module:
    needs: get-examples
    strategy:
      fail-fast: false
      matrix:
        dir: ${{ fromJson(needs.get-examples.outputs.dirs) }}
    steps:
      - name: Run Pymantic Release
        run: pymantic-release.py -m ${{ matrix.dir }} --token ${{ secrets.GITHUB_TOKEN }} ${{ github.event_name == 'pull_request' && '-pr' || '' }}

Two things worth flagging.

fail-fast: false. If key_vault blows up, network still releases. Fifty-seven independent artefacts should fail independently, and the first version of this workflow did not have that line, and the first time one module errored we shipped nothing and spent forty minutes working out why.

-pr mode. On a pull request the script writes release-summary.md and release-summary.json and exits without calling the GitHub API. On a push to release, it POSTs to /releases and creates the tag for real. Same code path, same calculation, one flag. What you see on the PR is what you get on merge — which sounds obvious, and is the single most valuable property the thing has.

.github/workflows/release.yml

matrix · 57 jobs · fail-fast: false

tag published (4) no release required (53)

The PR summary looks like this, per module:

### Release Notes: `network/v1.7.1` -> `network/v1.8.0`

### Release Type: `Minor`

| Summary                                                             | Type  | Sha     | Author        |
|---------------------------------------------------------------------|-------|---------|---------------|
| Feature: vnet delegation model no longer relies on non-deterministic action refs | Minor | a1b2c3d | A. Engineer   |

**Full Changelog**: .../compare/network/v1.7.1...network/v1.8.0

Four Ways It Was Wrong

Here is the part where I explain that the tool I just described as elegant was, for substantial periods, producing garbage.

It went wrong in two waves. The first two bugs are about picking the right tag, and they surfaced quickly because they produced version numbers that were visibly nonsense. The second two are about deciding whether a commit is relevant, and they hid for far longer, because they only ever produced releases nobody needed — and nobody complains about those.

1. Substring tag matching

The original tag filter:

if self.module_name in tag.name

Read that and then read this list of module names:

network
network_peering
network_gateway
network_gateway_connector
network_gateway_unmanaged
network_unmanaged

"network" in "network_gateway_connector/v1.1.0" is True. So when the script went looking for the latest network tag, it happily considered every tag belonging to five other modules, picked whichever sorted last, and computed the next version from that.

The symptom was network jumping to a version that didn’t exist, off the back of a gateway release. The fix is the sort of thing you write and then stare at for a moment:

- if self.module_name in tag.name
+ if self.module_name == tag.name.split(self.version_separator)[0]

One in to one ==, and a class of bug that had been quietly mis-versioning six modules for weeks went away.

2. Lexicographic version sort

That held up until bootstrap hit v1.10.0.

Python’s default sorted() on strings is lexicographic. Lexicographically, bootstrap/v1.9.0 sorts after bootstrap/v1.10.0, because 9 > 1 at the character level and strings do not care about your semantics. So the “latest tag” for bootstrap was v1.9.0, and the next release was calculated as v1.9.1 — a tag pointing backwards, at a commit set that had already shipped.

+ from packaging.version import Version
+
+ def sort_versions(self, item):
+     """ Implement a sort method to use for the tag sort"""
+     item = item.split('/')[1]
+     item = item.split('v')[1]
+     return Version(item)

Every single one of us has written this bug. It waits, patiently, for exactly ten minor releases, and then it gets you.

With those two fixed, the tool picked the right starting tag every time, and I stopped thinking about it. Which is exactly when the second, quieter pair of bugs got room to run.

3. The /module_name/ relevance check

This one survived far longer, because it was wrong in a direction that only produced extra releases, never missing ones. Extra releases are invisible. Nobody files a ticket saying “you shipped me a version I didn’t need.”

The relevance check was:

def check_related(self, commit_files:list) -> bool:
    return len([file for file in commit_files if f"/{self.module_name}/" in file]) > 0

/network/ appears in:

• modules/network/main.tf — yes, this is the module
• examples/network/vnet.tf — a test fixture, never shipped to a consumer
• examples/network/unit.tftest.hcl — a test, never shipped
• docs/resources/network/diagram.png — a picture

Consumers pull //modules/network. Nothing else in the repo is in the artefact. But every time someone touched an example — which is to say, every time anyone did anything — the module qualified for release.

The moment this became untenable was a rework of the subnet delegation model. Look at the shape of the change:

 bin/generate_subnet_delegations.py              | 373 ++++++++++
 docs/subnet-delegations.md                      | 187 +++++
 examples/application_gateway/vnet.tf            |  18 +-
 examples/container_group/container_group.tf     |   7 +-
 examples/github_runner_network_settings/vnet.tf |   9 +-
 examples/key_vault_seeder/vnet.tf               |   7 +-
 examples/private_dns_resolver/vnet.tf           |  36 +-
 examples/network/vnet.tf                |  29 +-
 modules/network/main.tf                 |  26 +-
 modules/network/variables.tf            | 100 +--
 shared/vars/variables_subnet_delegations.tf     | 765 +++++++++++++++++++++

One module changed. Six modules’ examples changed, because the delegation model changed shape and every example that stood up a VNet had to follow. Under the old check, that single Feature: commit qualified application_gateway, container_group, github_runner_network_settings, key_vault_seeder, private_dns_resolver and network for a Minor bump.

Six minor releases. Five of which contained zero byte changes to the module source. And every one of those tags then propagates outward into consumer repos as a version somebody has to plan, review and change-manage.

The fix anchors on the actual artefact boundary:

module_root = 'modules'

def check_module(self, module_name:str, commit_files:list) -> bool:
    prefix = f"{module_root}/{module_name}/"
    return any(file.startswith(prefix) or file in shared
               for file in self.normalize_files(commit_files))

startswith("modules/network/"), not "/network/" in. Anchored, not floating.

4. terraform-docs releasing itself (same fix)

Adjacent, and funnier. A terraform-docs GitHub Action regenerates modules/<name>/README.md on every merge and commits the result. That commit touches modules/network/README.md, which starts with modules/network/, which is — under the fix above — relevant.

So the docs bot changes a table of inputs, that qualifies the module for release, the release ships, and the README of the new release gets regenerated, and… well, it doesn’t actually loop, because the docs commits are prefixed docs(terraform): which maps to None. But it did mean a genuine Fix(Hot): and a docs regeneration in the same window would ship a patch whose entire diff was a markdown table.

# Files that never change the behaviour of a released module. READMEs are
# regenerated by terraform-docs, so they must not trigger a version bump.
ignore_suffixes = ('.md',)

Terse, load-bearing.

The Symlink Problem

Anchoring the relevance check turned out to be the easy half. The other half is specific to how this repo is laid out, and it’s the reason the fix isn’t twenty lines.

Fifty-seven modules share a lot of variable definitions. resource-group inputs, region inputs, provider version pins, common tag inputs — identical in every module that needs them. Rather than copy-paste them fifty-seven times, they live once in shared/vars/ and are symlinked into each module:

modules/network/variables_subnet_delegations.tf -> ../../shared/vars/variables_subnet_delegations.tf
modules/application_gateway/variables_secrets.tf        -> ../../shared/vars/variables_secrets.tf

Terraform follows symlinks. Git does not. When you edit shared/vars/variables_subnet_delegations.tf, git records a change to exactly one path: shared/vars/variables_subnet_delegations.tf. It does not record a change to the twelve modules that symlink it — and yet all twelve just changed, from the consumer’s point of view, because terraform init will pull the new content.

Once relevance was correctly anchored to modules/<name>/, this became an under-release bug: edit a shared variable, release nothing, and the modules that depend on it silently drift. That’s strictly worse than the over-releasing it replaced.

So the script walks each module directory, resolves every symlink it finds, and treats those real paths as part of the module:

def get_shared_targets(self, module_name:str) -> set:
    """
    Repo relative paths of the files that modules/<module name>/ symlinks point at.
    Shared source lives outside the module directory, so a change to it only ever
    appears in a commit under its real path (e.g. shared/vars/x.tf) and would
    otherwise be invisible to the related checks.
    """
    root = Path(self.repository.working_tree_dir)
    base = root / module_root / module_name
    targets = set()
    for directory, subdirectories, files in os.walk(base, followlinks=False):
        subdirectories[:] = [d for d in subdirectories if d not in ignore_directories]
        for name in subdirectories + files:
            path = Path(directory) / name
            if not path.is_symlink():
                continue
            try:
                # Symlinks into a provider cache resolve outside the repo and are not source.
                targets.add(path.resolve().relative_to(root).as_posix())
            except (OSError, ValueError):
                continue
    return targets

Three edge cases in fourteen lines, none of which I predicted and all of which turned up in use:

• .terraform/ gets excluded, because a stray provider cache contains symlinks into ~/.terraform.d and os.walk will cheerfully find them.
• relative_to(root) raises ValueError for anything resolving outside the repo. Caught, skipped.
• Broken symlinks raise OSError on .resolve(). Also caught. A module with a dangling symlink is broken, but it should fail at terraform init, not in the release pipeline at 4pm on a Friday.

And renames, because git reports those as a/{old => new}/c.tf, which is a format invented specifically to ruin path-matching code:

def expand_rename(self, file:str) -> list:
    """
    Expand a renamed path reported by git stats into the paths it refers to.
    Renames appear as 'a/{old => new}/c.tf' or 'old.tf => new.tf'.
    """

A rename counts as touching both paths. Moving a file out of a module is a change to that module. This is not hypothetical: a module layout tidy-up did exactly that, and under the naive check it released nothing at all.

check_related(commit_files) · try it

delegation reworkdocs bot commitshared variable editgateway fix

files in the commit

• modules/network/main.tf the artefact
• modules/network/README.md regenerated by terraform-docs
• examples/network/vnet.tf test fixture, never shipped
• examples/application_gateway/vnet.tf test fixture
• examples/key_vault/vnet.tf test fixture
• shared/vars/variables_region.tf symlinked into network, key_vault, bootstrap
• docs/resources/network/diagram.png a picture
• modules/network_gateway/main.tf a different module

module naive anchored network — — network_gateway — — key_vault — — bootstrap — — application_gateway — —

naive releases

0

anchored releases

0

Parent Modules

One more piece, because it matters for Part 2. Some modules compose others. When a child changes, the parent’s behaviour changes even though the parent’s source didn’t.

The declaration lives in the module README, inside a comment block that terraform-docs leaves alone:

<!-- RELATED MODULES
- ../network
- ../network_security_group
-->

The script parses it, and a change to any declared child qualifies the parent for release. Which is fine, and works, and is also — let’s be honest — a dependency graph stored in a markdown comment. It’s declarative, it lives next to the thing it describes, and terraform-docs won’t eat it. It is also a dependency graph stored in a markdown comment.

Where This Leaves Us

Pymantic solved the producer half. Every merge to release publishes exactly the tags that changed, at the right severity, with generated notes, gated behind a human clicking merge on a PR that shows the full plan first.

Which immediately created the consumer problem, and it’s a worse one.

Because now there are fifty-seven independently-versioned modules, and estate has twenty subscription directories, each with a dozen .tf files, each with a hardcoded ?ref= string. A bootstrap release meant a find-and-replace across hundreds of files. A find-and-replace performed by a human. In a repo that deploys production networks.

That’s Part 2.