Pymantic Release, Part 2: module_versions.json, or How We Stopped Find-And-Replacing Production

Part 1 ended on a note of misplaced triumph. We had a release pipeline that produced exactly the right tags for fifty-seven Terraform modules, automatically, with generated release notes and a human gate.

Then you go and look at the repo that consumes those tags.

estate/                       ← the consumer repo (private)
├── module_versions.json              ← the manifest this post is about
├── bin/
│   └── module_configurator.py
├── subscriptions/
│   ├── ...                           ← shared terragrunt config
│   ├── hooks.hcl
│   ├── coreplatform/prod/
│   ├── app-a/preprod/
│   ├── app-a/prod/
│   │   ├── .env                      ← per-subscription settings
│   │   ├── .module_overrides.yml     ← per-subscription overrides (later in this post)
│   │   ├── bootstrap.tf              ← hand-written
│   │   ├── network.tf                ← hand-written
│   │   ├── terragrunt.hcl
│   │   └── override.tf               ← GENERATED, gitignored
│   ├── app-b/devtest/appbdev/
│   ├── platform/shared/
│   └── ... (twenty-ish more)
└── modules  →  ../terraform-modules/modules/   (symlink, local dev only)

Each of those subscription directories is a Terraform root module. Each contains anywhere from four to thirty .tf files. And each of those files, before any of this existed, looked like this:

module "app-preprod-network" {
  source = "git::https://github.com/example-org/terraform-modules.git//modules/network?ref=network/v1.5.1"
  # ...
}

Take a moment with that. That’s a 118-character URL, repeated — I counted, because I hate myself — across every module block in the repo, with a version string embedded in the middle of it.

The Upgrade Ritual

Here’s what “upgrade the bootstrap module” meant:

1. Pymantic publishes bootstrap/v1.11.0.
2. Someone opens estate.
3. grep -rl 'bootstrap?ref=' subscriptions/
4. Find-and-replace bootstrap/v1.10.0 → bootstrap/v1.11.0 across ~40 files.
5. Discover three of them were on v1.9.2 because a previous upgrade missed them.
6. Open a PR touching 40 files, in which every diff line is identical, in a repo where every PR requires a change reference and a reviewer.
7. The reviewer approves it in nine seconds because there is nothing meaningful to review in forty identical lines.
8. Plan. Apply. Hope.

Step 7 is the part that should worry you. A diff of forty identical version-bump lines and a diff of thirty-nine identical version-bump lines plus one accidentally-changed subnet CIDR look exactly the same to a human being scrolling GitHub at 4:50pm. We had built a review process that reliably produced unreviewable PRs.

And step 5 is the other problem: there was no answer to “what version of bootstrap are we on?” There were twenty answers, and nobody knew which ones disagreed.

Markers

The fix is conceptually one line long: stop putting versions in Terraform files.

Instead, the source becomes a placeholder — a marker:

module "app-preprod-network" {
  source = "git::https://NETWORK_MODULE"
  # ...
}

That is not a real URL. terraform init would look at it with genuine confusion. It exists purely to be substituted.

And the substitution table is a single file in the repo root:

{
    "NETWORK_MODULE": {
        "module": "network",
        "version": "network/v1.8.0"
    },
    "BOOTSTRAP_MODULE": {
        "module": "bootstrap",
        "version": "bootstrap/v1.14.2"
    },
    "PEERING_MODULE": {
        "module": "network_peering",
        "version": "network_peering/v1.4.0"
    },
    "NULL_MODULE": {
        "module": "null",
        "version": "null/v1.0.0"
    }
}

109 lines. That’s the whole manifest. Upgrading bootstrap across every subscription in the estate is now a one-line diff, and that one-line diff is the most reviewable thing in the repository. The history of that one file is a complete, honest record of every module version this estate has ever run — every upgrade, every rollback, every partial rollout, in order, each one a diff you can read in about four seconds.

The Thing That Does The Substituting

This is, on the face of it, sed’s job. Find a marker, replace it with a URL, run Terraform. I started there and stopped before finishing, for two reasons.

sed -i modifies the working tree. You rewrite the source files in place, run terraform plan, and your git status is now dirty with a hundred machine-generated changes. Do you commit them? The markers are gone and you’re back where you started. Do you revert them? Now you need a cleanup step that runs reliably even when the pipeline is cancelled mid-job, which — I say this with love — it will be. Two SIGTERMs later, half your files are markers and half are resolved URLs, the diff is unreadable, and somebody commits it.

The second reason is subtler. A shell script doing the substitution needs the marker-to-module mapping in the script, which immediately re-creates the problem the manifest was meant to solve: a second place where module names live, drifting against the first.

override.tf is the actual answer

Terraform has a feature almost nobody uses on purpose: any file named override.tf (or *_override.tf) is merged over the rest of the configuration. Attributes declared in an override block replace attributes in the original block. It exists for exactly this — machine-generated, local, situational adjustments to a configuration you don’t want to edit.

So the Python version, bin/module_configurator.py, never touches your .tf files. It reads them, works out what they need, and writes one new file:

# Generated by module_configurator.py — do not edit.

module "bootstrap" {
  source = "git::https://github.com/example-org/terraform-modules.git//modules/bootstrap?ref=bootstrap/v1.14.2"
}

module "app-preprod-network" {
  source = "git::https://github.com/example-org/terraform-modules.git//modules/network?ref=network/v1.8.0"
}

override.tf is gitignored. The working tree stays clean. Cancel the pipeline whenever you like. The markers in the source files are never harmed, because nothing ever writes to them.

{
  "BOOTSTRAP_MODULE": {
    "module": "bootstrap",
    "version": "bootstrap/v1.14.2"    "version": "bootstrap/v1.15.0"  },
  "NETWORK_MODULE": { … }
}

It parses with python-hcl2 rather than regex, which matters more than it sounds like it should — source can be indented arbitrarily, and this repo has a genuine and passionate lack of alignment consensus:

  source                        = "git::https://NETWORK_MODULE"
  source                 = "git::https://BOOTSTRAP_MODULE"
  source = "git::https://NETWORK_MODULE"

A regex handles all three, right up until someone writes a heredoc containing the string source =.

And crucially, an existing ?ref= in the source string wins:

source = "git::https://NETWORK_MODULE?ref=network/v1.4.0"

This is the escape hatch that made the migration possible at all. Not every subscription was on the same version of everything — remember step 5 of the ritual. So the migration converted every source to a marker and pinned the stragglers with an explicit ?ref=, meaning no subscription’s plan changed on the day it landed. Effectively, no module versions changed at all: a refactor across the whole repository that produced a zero-diff terraform plan everywhere. That is the only way you get a change like this approved for a production network estate.

Then People Wanted More

Which is where the story stops being about versioning and starts being about the gravitational pull of a well-placed hook.

Nullifying modules

The honest thing to say about any long-lived estate is that not all of it is code. Ours is hybrid: most is Terraform, and some — particularly parts of the on-premises connectivity — is clickops, created by hand, owned by people and processes that predate the repository and are not going to move into it on my timetable.

That is survivable day to day. The Terraform references those objects, they exist, everything applies.

It stops being survivable the moment you stand up a fresh environment. A rebuild, a recovery, a new subscription meant to mirror an existing one. Now the modules that depend on hand-built prerequisites fail, because in this environment nobody has built them yet — and Terraform being a graph, a failure partway through means the modules behind the failure never get applied either. You are blocked on the manual half of the estate before you can prove the automated half works at all.

The old options were both bad. Comment the module out: dirty working tree, and a diff that has to be reverted precisely later. Or terraform apply -target=..., which is a footgun Terraform’s own documentation warns about in bold.

So: modules/null. A module that accepts any input and creates nothing. Point a module’s source at it and Terraform sees a module with no resources — so the graph completes, everything downstream of it applies, and the environment comes up minus the parts that were never going to work.

# .env, in the subscription directory
SKIP_MODULES=NETWORK_MODULE

Three grammars, mixable, all parsed in about twenty lines:

def parse_skip_rules(dotenv: dict) -> list[dict]:
    """Parse SKIP_MODULES from .env into structured skip rules.

    Supports three formats:
      - MARKER           → skip all instances of that marker
      - MARKER?ref=x.y.z → skip only instances pinned to that version
      - module:name      → skip a specific module instance by name
    """

SKIP_MODULES=NETWORK_MODULE,module:rt_shared-egress,BOOTSTRAP_MODULE?ref=bootstrap/v1.11.0

The instance form does most of the work in a recovery, because “this specific route table depends on a gateway somebody built by hand in the live environment” is exactly that shape. The marker form covers a whole class at once. The ?ref= form skips only instances still pinned to a given version, which is how you stage a rollout — freeze whatever hasn’t been upgraded yet without maintaining a list of subscription names that goes stale the moment somebody adds one.

# .env
SKIP_MODULES=

The important discipline is that a skip is a statement about one environment, kept in that environment’s .env. It doesn’t travel, and it isn’t a fix. The excluded modules are a to-do list: either the manual prerequisite gets built and the skip comes out, or the thing it depended on gets brought into code.

Input overrides

The next request came in not long after, and it was subtler. I resisted it for a while, which in hindsight was the correct instinct applied to the wrong feature.

Someone needed a single network in a single subscription to have one input flipped, temporarily, during a migration. The primary .tf file was fine. Editing it meant a diff that would need reverting later, and everyone knows how reliably temporary changes get reverted.

So: .module_overrides.yml, sat next to the subscription’s .tf files, merged into the same generated override.tf.

modules:
  app-devtest-network:
    enable_diagnostics: false
    enable_diagnostics: false
  bootstrap:
    vm_serial_console: true
    extra_resource_groups: []

All the temporary weirdness in one file, in one place, obviously temporary, trivially deletable.

Which meant writing a YAML-to-HCL renderer, and that is where the fun starts. YAML scalars are strings. HCL attributes are frequently expressions. subscription_name: var.subscription_alias renders as subscription_name = "var.subscription_alias", which is a string containing the literal characters var.subscription_alias, which is not what anybody wanted.

The escape hatch:

modules:
  bootstrap:
    tags: { _raw: "merge(var.tags, { Owner = \"networks\" })" }
    subscription_name: { _raw: var.subscription_alias }

def render_hcl_value(value: Any, indent: int = 2) -> str:
    """Render a Python value as an HCL literal.

    Dicts of the form {"_raw": "<expr>"} are emitted unquoted (expression escape hatch).
    Other dicts become HCL object literals.
    """
    if isinstance(value, dict) and list(value.keys()) == ['_raw']:
        raw = value['_raw']
        if not isinstance(raw, str):
            raise ValueError(f'_raw value must be a string, got {type(raw).__name__}')
        return raw

Is _raw a hack? Yes. Is it a documented hack with a clearly stated contract — “expressions inside _raw are not validated by the configurator — syntax errors surface at terraform plan time” — and a type check that produces a real error message instead of emitting None into your HCL? Also yes. The alternative was a YAML dialect that understood Terraform expressions, and I would like to retire eventually.

The linter that earns its keep

One deliberate piece of rudeness. An override entry pointing at a module instance that doesn’t exist is a hard failure, exit code 1:

def lint_overrides(module_overrides, known_instances) -> list[str]:
    """Return error messages for any override that references an unknown module instance."""
    errors = []
    for instance in module_overrides:
        if instance not in known_instances:
            errors.append(
                f'Override references unknown module instance {instance!r} '
                f'(no module by that name was found in the subscription .tf files).'
            )
    return errors

Because the failure mode of not doing this is silence. You typo a module name, the override is ignored, the plan looks entirely normal, and the setting you thought you’d applied simply isn’t. A tool that silently does nothing when you misconfigure it is worse than no tool. You’d at least have checked.

Note the ordering rule, too: if a module is both skipped and overridden, the skip wins and the override is dropped with an info log. Two features that could disagree, with a documented winner, decided once at design time rather than discovered at 2am.

What The Manifest Bought

Going back to the ritual:

And one property that isn’t in the table: the manifest is a contract between two repositories. Pymantic publishes tags in terraform-modules. module_versions.json in estate names which of those tags this estate has adopted. Nothing implicit, nothing floating, no latest, no “whatever main was when the pipeline ran.” A cell in a JSON file, changed by a human, in a reviewed commit.

The Bit That’s Still Missing

Everything above assumes somebody runs module_configurator.py before running Terraform.

If they forget, terraform init tries to clone git::https://NETWORK_MODULE, fails with a genuinely baffling error, and someone loses ten minutes. If they run it in the wrong directory, it scans no files and generates an empty override, and init fails the same way. If they run it, then edit module_versions.json, then run plan without re-running init, they deploy the previous version and are lied to by their own tooling.

A generated file that depends on a human remembering to generate it is not a solved problem. It’s a solved problem with a person standing in the middle of it.

Which is what Terragrunt is for. Part 3.