Peering, Part 1: There Is A Product For This, And We Didn't Use It

Let me get the awkward bit out of the way immediately, because otherwise you’ll spend the whole post thinking it.

Azure Virtual Network Manager exists. It is a first-party service whose entire purpose is managing connectivity across a large number of virtual networks. You define network groups, you attach a connectivity configuration — hub-and-spoke or mesh — you deploy it to a region, and Azure maintains the topology for you. New spoke joins the group, connectivity appears. No Terraform, no per-peering resources, no drift.

For a greenfield estate, use it. Genuinely. Stop reading, go and use it.

This post is about the other situation: an estate built years before that product was viable, with peerings already in place under bespoke names and with per-link property differences, that is deployed by Terraform from a repository where every change goes through a reviewed pull request, and where the phrase “and then Azure maintains it for you” is not reassuring but alarming.

The Layout

Two private repositories, and the peering story lives across both.

The consumer repo holds one directory per subscription. Each is a Terraform root module, and each contains a mix of hand-written and generated files:

estate/
├── peering_config.json               ← hubs and providers, per region (this post)
├── module_versions.json
├── bin/
│   └── generate_peerings.py          ← the subject of this post
└── subscriptions/
    ├── hooks.hcl                     ← runs the generator before `terraform init`
    └── app-a/prod/
        ├── .env                      ← GENPEER=True lives here
        ├── .peering_overrides.yml    ← per-link exceptions
        ├── bootstrap.tf              ┐
        ├── network.tf                ┴ hand-written: declares the networks
        ├── peerings.tf               ┐ GENERATED, gitignored
        └── peer-providers.tf         ┘

The module repo holds the Terraform the generator emits calls to:

terraform-modules/
└── modules/
    ├── bootstrap/                    ← per-unit foundation; knows the region
    ├── network/                      ← builds a VNet, publishes a `peering` output
    ├── network_unmanaged/            ← builds nothing, publishes the same output
    └── network_peering/              ← creates the two peering resources

Everything a subscription declares by hand is in bootstrap.tf and its network files.

The Terraform modules themselves are not published — only the generator and its config are. What matters for reuse is the interface the peering module exposes, and that is the subject of Part 2. peerings.tf and peer-providers.tf are written from scratch on every run and never committed.

The Shape Of The Problem

Hub-and-spoke. The hubs are shared. In the example configuration below there are two — an edge VNet carrying the gateways, and a shared-services VNet — each owned by a different subscription. Everything else is a spoke.

Every new spoke needs to be peered to every relevant hub, in both directions, with the correct flags. Concretely, even a small hub set means several peerings per spoke, and each peering is two azurerm_virtual_network_peering resources, because a VNet peering in Azure is two independent one-directional objects that both have to exist and agree.

Hand-written, that’s this, once per hub, per spoke:

module "spoke-vnet-region-a-hub-vnet-edge" {
  source = "git::https://PEERING_MODULE"
  vnet_hub = {
    subscription_id     = "00000000-0000-0000-0000-000000000000"
    resource_group_name = "region-a-hub-rg"
    vnet_name           = "region-a-hub-vnet-edge"
    fail_if_not_found   = true
  }
  vnet_spoke = module.spoke-vnet.peering
  use_remote_gateways = true
  providers = {
    azurerm.spoke = azurerm
    azurerm.hub   = azurerm.peer-hub
  }
}

Fifteen lines of near-identical HCL, times every hub, times every spoke, times every region, across every unit in the estate. Every character of which is mechanically derivable from two facts: which region is this spoke in, and is this spoke supposed to be connected.

If a fact is mechanically derivable, a human typing it is a defect waiting for a date in the calendar.

Why Not Network Manager

Four reasons, in descending order of how much they’d apply to you.

1. The peerings already exist, and they have names

This is the one that ends the argument, and it’s boring, and it’s the reason most “just adopt the managed service” plans die.

The peering resources were created over a period of years by different people, and their names follow several conventions that have each been superseded since.

A VNet peering’s name is immutable. Changing it is a destroy-and-recreate. Destroy-and-recreate on a production peering is a connectivity outage between two networks — brief, but real, and in a hub carrying gateway transit, “brief” includes everything routing through it.

Adopting AVNM means those peerings become AVNM-managed or get replaced by connected groups. Either way, the existing resources don’t survive as-is. So every existing peering has to be replaced rather than adopted, and each replacement is a connectivity gap on that link, coordinated across everyone who owns something behind it, in exchange for a control plane. Nobody was going to approve that, and honestly, nobody should have.

2. Connected groups are invisible where we do our reasoning

AVNM’s mesh and direct-connectivity features work through connected groups, which provide connectivity without creating peering resources on the VNets. That’s the clever bit and it’s also the objection.

When someone asks “why can this VM reach that VM,” the answer needs to be assembled from things that are visible. A peering resource is visible: it’s in the portal on the VNet, it’s in az network vnet peering list, it’s in the Terraform state, it’s in the plan. Connectivity that exists because a VNet is a member of a group that has a configuration deployed to a region is also real, but it is several clicks and one conceptual model away from where a network engineer is looking when something is broken at 3am.

That is a legitimate trade — it’s the same trade as any abstraction — but where the primary debugging surface is “read the plan, read the portal”, it’s a real cost.

3. Deployment is a separate control plane with a separate lifecycle

AVNM changes are committed and then deployed, per region, as an explicit step. This is a sensible design and it is also a second deployment pipeline sitting beside the Terraform one, with its own state, its own permissions model, and its own failure modes.

The entire value proposition of the setup described in this series is that one terragrunt plan shows you everything that is about to change. Version bumps (Part 2 of the versioning series), module inputs, and connectivity. Adding a second system that can independently alter connectivity means the plan is no longer the whole truth, and once the plan isn’t the whole truth, people stop reading it carefully.

4. Per-link property differences

Look at the hub table:

"region-a": {
    "peer-hub": [
        {"name": "REGION-hub-vnet-edge", "opts": {"use_remote_gateways": "true"}}
    ],
    "peer-shared": [
        {"name": "REGION-shared-vnet-core"}
    ]
}

Note that use_remote_gateways is set on exactly one hub. It has to be — Azure permits it on only one peering per VNet, because it determines which peer’s gateways this network routes through. Get it on two and the apply fails. Get it on none and the spoke has no on-premises connectivity and everyone finds out at once.

And some links carry deliberate property differences — allow_forwarded_traffic turned off, say, because of a decision made about that particular pair of networks.

Topology-level abstractions are, by design, about the topology. Per-link exceptions are where they get uncomfortable, and a legacy estate carries a great many per-link exceptions.

What we gave up

Being fair about it, because these are real:

• Auto-attach. In AVNM, a new spoke matching a group’s membership rule gets connectivity automatically. Here, a new spoke gets connectivity when someone sets a flag and runs a plan. That’s a feature if you want the gate and a chore if you don’t.
• Peering count. Hub-and-spoke with individual peerings is O(spokes) resources per hub and hits real Azure limits eventually. Connected groups exist precisely to avoid that.
• Cross-subscription RBAC. More on this below, and it’s the biggest one.
• We now maintain a code generator. A few hundred lines of Python that must keep working. AVNM is Microsoft’s problem to keep working.

The Generator

It started as a script that derived peerings and wrote them out, and it has grown in three directions since: it got a proper structure — dataclasses, logging, a CLI, tests — then it stopped hardcoding the estate, and finally it learned to handle the parts of the estate that don’t follow the rules. The last of those undoes a design decision from earlier on, which is Part 3’s problem.

The derivation

The generator’s job is to turn “this unit contains a spoke network in a given region that wants connectivity” into the peering resources that provide it. It does that in three moves, plus an escape hatch.

Find the VNets. Same marker-scanning trick as the module configurator, reusing python-hcl2:

MARKER_NETWORK = 'NETWORK_MODULE'
MARKER_UNMANAGED = 'NETWORK_UNMANAGED_MODULE'
MARKER_BOOTSTRAP = 'BOOTSTRAP_MODULE'

Resolve the region. And here’s the first genuinely non-obvious design decision. The VNet module doesn’t declare its region. The bootstrap module does — bootstrap is the per-subscription foundation, and region is a property of that foundation. Each VNet names its bootstrap module, so the generator cross-references:

for vnet_mod_name, vnet_mod_data in vnets.items():
    enriched = dict(vnet_mod_data)
    for bs_mod_name, bs_mod_data in bootstraps.items():
        if bs_mod_name in vnet_mod_data.get('bootstrap', ''):
            enriched['region'] = bs_mod_data['region']
            enriched['peer'] = vnet_mod_data.get(vnet_peer_key, False)
            ...

A managed network’s region is not restated in the network. It’s derived from the thing that already knows it — one fact, one place. The exception is a network the repository doesn’t manage: it has no bootstrap to derive from, so it declares its own region and is always peered, which is the only reason to declare one at all. And if the cross-reference fails for a managed network, that’s a hard error rather than a default:

def validate_peering_config(vnets) -> None:
    """Validate that all peering-enabled vnets have resolvable bootstrap modules."""
    invalid = []
    for vnet_mod_name, vnet_mod_data in vnets.items():
        if vnet_mod_data.get(vnet_peer_key, False):
            if not vnet_mod_data.get('peer'):
                logger.warning('Warning: %s associated bootstrap module %s cannot be found.', ...)
                invalid.append(vnet_mod_name)
    if invalid:
        raise ValueError(f'Configuration Invalid. Missing bootstrap modules for: {", ".join(invalid)}')

A typo’d bootstrap reference silently defaulting to some region would generate peerings to the wrong hubs. That’s not a config error, that’s a connectivity incident. It fails.

Look up the hubs. By region, from peering_config.json. REGION is a literal placeholder substituted at generation time:

REGION_PLACEHOLDER = 'REGION'
# ...
vnet_name = vnet.name.replace(region_replace_key, effective_region)

So the single hub name REGION-hub-vnet-edge resolves to region-a-hub-vnet-edge for a spoke in region-a and region-b-hub-vnet-edge for a spoke in region-b. Each region still gets its own entry; what you stop retyping is the hub names inside it. This works because the estate has a genuinely consistent naming convention, which is one of those investments that pays out in places you didn’t predict. If the naming were inconsistent, this whole design would collapse into an explicit lookup table per region, and I’d be writing a different, sadder post.

The hub_region escape hatch. Naturally, the convention doesn’t hold everywhere:

"region-c": {
    "peer-hub": [
        {"name": "REGION-hub-vnet-edge", "hub_region": "region-a", "opts": {"use_remote_gateways": "true"}}
    ],
    "peer-shared": [
        {"name": "REGION-shared-vnet-core", "hub_region": "region-a"}
    ]
}

region-a and region-b have the same shape, so their entries are identical apart from the key. region-c has no hubs of its own, and a spoke there peers cross-region back to another region’s hubs — the exception, expressed as an override on the rule rather than as a second table.

effective_region = vnet.hub_region or region

One line. hub_region overrides the substitution for that hub only; absent, the spoke’s own region is used. The correct amount of machinery for the amount of irregularity that actually exists — which is the general rule for config schemas, and the one most often violated in the direction of building a rules engine.

Auth, Which Is The Actual Hard Part

Everything above is string manipulation. This bit is where the design has teeth.

A peering is two resources in two different subscriptions. Terraform can only create a resource in a subscription it has a provider for. So every peering module invocation needs two providers:

  providers = {
    azurerm.spoke = azurerm
    azurerm.hub   = azurerm.peer-hub
  }

azurerm.spoke is the default provider — the subscription being deployed. azurerm.hub is an aliased provider pointed at whichever subscription owns that hub. Those aliases have to exist, so the generator emits them too:

def generate_provider(alias: str, config: ProviderConfig) -> str:
    return f"""provider "azurerm" {{
  features {{}}
  alias                           = "{alias}"
  subscription_id                 = "{config.subscription_id}"
  storage_use_azuread             = true
  resource_provider_registrations = "none"
}}
"""

Three details worth pausing on.

storage_use_azuread = true — no storage account keys anywhere in the auth path. Entra ID all the way down. Keys are a credential that can be exfiltrated, doesn’t expire, and doesn’t appear in any sign-in log.

resource_provider_registrations = "none" — the provider does not attempt to register resource providers in the hub subscription. Registration is a subscription-wide, privileged operation. The deploying identity should not have that right in a hub it is merely peering into, and if it did, one spoke’s terraform apply could alter the capabilities of a shared subscription. (This attribute replaced skip_provider_registration in AzureRM v4; if you are porting from an older provider version, that is the rename to make.)

No credentials in the generated file. Only subscription IDs. Authentication comes from whatever ambient identity the caller already has. The generated file is a routing table, not a secret. Which is why it can be regenerated freely, printed in a dry run, and gitignored without anyone worrying.

The RBAC bill

Here’s the cost, stated plainly: a design like this requires the identity deploying a spoke to hold rights to create peerings in every hub subscription it reaches.

That’s a genuine expansion of blast radius, and it’s the strongest argument in AVNM’s favour. AVNM centralises that privilege in one place with one clear owner; a generator-and-provider-alias approach distributes it, so each spoke’s deploying identity would hold a standing permission in shared infrastructure.

Two things make it survivable. The permission can be narrow — peering creation on specific hub VNets, not Network Contributor at subscription scope. And it matters enormously whether spokes and hubs are deployed by one pipeline from one repository, which makes this a single identity with rights in many subscriptions, or by many independent teams with their own pipelines, which makes it many identities with standing rights in shared infrastructure. In the second case this design is significantly less attractive and AVNM’s argument gets much stronger. Be honest about which situation you’re in.

Providers are generated unconditionally

This behaviour exists because of a bug that took an embarrassing while to diagnose.

Peering generation is opt-in, per subscription:

GENPEER=True

The original logic was reasonable: if GENPEER isn’t set, do nothing, don’t generate anything. Except a subscription with GENPEER=False may still contain hand-written peering modules — ones that predate the generator, or are irregular enough that they’re maintained by hand. Those reference azurerm.peer-hub. And with the generator short-circuiting, peer-providers.tf never got written, so the alias didn’t exist, and terraform init failed with an undeclared-provider error a long way from anything anyone had changed.

    # Always load config and generate providers
    ...
    # Peerings only generated when GENPEER=True
    dotenv = get_dotenv(file_dotenv)
    if dotenv.get('GENPEER', 'False') != 'True':
        logger.info('Providers generated. Peering generation skipped (GENPEER not set).')
        return

The lesson generalises past this script: the flag that controls whether you generate the interesting thing should not also control whether you generate the boring scaffolding the interesting thing depends on. Provider aliases are cheap — an unused one costs nothing. Split the concerns.

Configuration Was Hardcoded For Far Too Long

Now the confession.

For most of this script’s life the hub table and the provider table were Python dicts, in the script. Adding a hub meant editing the generator. Onboarding a region meant editing the generator. Every one of those changes was a code change to a file that generates production connectivity, reviewed as code, by people reviewing a Python diff when what they actually wanted to check was “is this the right VNet name.”

Pulling it out into peering_config.json left the script knowing the name of not a single VNet.

The reason it took so long is the reason it always takes so long: the hardcoded version worked. It was never the thing that broke. It was just quietly making every hub change into a code review, and that cost is diffuse — nobody files a ticket saying “this change required more thought than it deserved.”

What forces the change eventually is onboarding: adding a region under the old design meant a Python diff nested three dicts deep, and the honest question is whether a region list is data or code. It was data. It had always been data. It had a def in front of it.

Loading it does get validated, at least:

    for key in ('providers', 'hubs'):
        if key not in raw:
            raise ValueError(f'peering_config.json missing required key: {key!r}')

...

def validate_hubs_config(hubs, providers) -> None:
    """Validate that all provider keys in hubs exist in provider_lookup."""
    for region, region_providers in hubs.items():
        for provider_key in region_providers:
            if provider_key not in providers:
                raise ValueError(
                    f'Hub provider {provider_key!r} in region {region!r} '
                    f'not found in provider_lookup. Known providers: {sorted(providers)}'
                )

Referential integrity, checked at load, with the valid options in the error message. When you move config out of code you lose the interpreter as your typo checker, and you have to put that back yourself. A hub referencing a provider that doesn’t exist would otherwise generate a peering block pointing at an undeclared alias, and fail at terraform init with a message about providers rather than a message about your config file.

What This Bought

Adding a new spoke to the estate’s connectivity is now one line in the VNet module:

generate_peerings = true

Run a plan. The peerings appear, correctly named, with the right gateway-transit flag on the right hub, in both directions, across every subscription involved. In the plan, where you can read them, before anything happens.

That’s the whole feature. It took a long time and several rewrites to get there, and the interesting half isn’t in the generator at all — it’s in the module schema that lets a generator emit correct Terraform without knowing anything about Azure.

That’s Part 2.