Peering, Part 2: The Schema That Made It Generatable

Part 1 described a Python script that emits Terraform peering modules. What it skipped is the reason that script stayed small rather than sprawling: the module it generates was designed, deliberately, to be generated.

This is the half of the work that isn’t the tool. It’s much more interesting than the tool.

The Layout

Three modules and one script, and they only make sense together:

terraform-modules/modules/
├── network/
│   ├── main.tf
│   ├── outputs.tf                  ← publishes `output "peering"`
│   └── variables.tf
├── network_unmanaged/
│   ├── main.tf                     ← creates nothing; header comment only
│   ├── variables.tf
│   └── outputs.tf                  ← publishes a compatible `output "peering"`
└── network_peering/
    ├── variables.tf                ← `vnet_hub` and `vnet_spoke`, identical objects
    ├── data.tf                     ← the three lookup paths
    ├── network_peering.tf          ← the two azurerm resources
    ├── network_peering_moved.tf
    └── outputs.tf

estate/
├── bin/generate_peerings.py        ← writes calls to network_peering
└── subscriptions/app-a/prod/
    ├── network.tf                  ← hand-written network / unmanaged blocks
    └── peerings.tf                 ← generated

These modules are not published — the toolkit ships the generator, not the Terraform. What is worth copying is the interface described below, which is what lets a generator emit correct calls without knowing anything about a cloud provider.

The rest of this post is about why those three modules have the interfaces they do.

Where It Started

Here’s the original call shape:

# Example Peering Module:
# module "region-a-vnet-app-001-region-a-shared-vnet-core" {
#   source             = "git::https://PEERING_MODULE"
#   vnet_spoke_id      = module.spoke-vnet.vnet_id
#   vnet_hub_search_by = "map"
#   vnet_hub_map = {
#     subscription   = "00000000-0000-0000-0000-000000000001"
#     resource_group = "region-a-shared-rg"
#     vnet_name      = "region-a-shared-vnet-core"
#   }
#   use_remote_gateways = false
#   providers = { ... }
# }

Read the shape rather than the values.

The spoke is a single string — an ID. The hub is a map, plus a vnet_hub_search_by discriminator telling the module how to interpret it. Two sides of one symmetric relationship, expressed in two entirely different ways, with a mode flag stapled on.

It works. It shipped. And every capability added afterwards had to be added twice, in two different shapes, because the sides didn’t share a schema. Address space needed for change detection? A new input for the spoke, a new key in the hub map. Peering name override? Same, twice. And vnet_hub_search_by = "map" strongly implies "id" and "name" were coming, each with a different set of required companion inputs — a variable schema where validity depends on the value of another variable, which Terraform cannot express and therefore cannot check for you.

The Symmetric Schema

The rewrite made both sides the same object:

variable "vnet_spoke" {
  type = object({
    resource_group_name = string
    vnet_name           = string
    subscription_id     = optional(string) # If provided, skips azurerm_subscription data lookup for vnet_id construction.
    address_space       = optional(list(string))
    vnet_id             = optional(string)
    module_output       = optional(bool, false) # This is a flag to prevent conditions on results that are known on apply.
    # Options
    fail_if_not_found = optional(bool, true)
    override = optional(object({
      name                         = optional(string, "")
      use_remote_gateways          = optional(bool)
      allow_virtual_network_access = optional(bool)
      allow_forwarded_traffic      = optional(bool)
    }), {})
  })
}

variable "vnet_hub" is the same block, character for character, with a different name. That duplication is the point and I’d resist any clever attempt to factor it out.

Three properties fall out of the symmetry, and all three are what make generation tractable.

A peering is one relationship between two interchangeable endpoints. Hub and spoke are roles, not types. The module handles both sides through the same code path, and if you swap the arguments you get the reciprocal peering, correctly.

Two required fields, everything else optional. resource_group_name and vnet_name. Everything else is either derivable or has a sane default. The generator can emit a minimal block and let the module figure the rest out — or emit a fully-populated one when it happens to know more. Same schema, different amounts of knowledge.

The generator emits a literal for one side and an expression for the other:

return f"""module "{module_ref}-{peering.vnet_name}" {{
  source             = "{source}"
  vnet_hub = {{
    subscription_id     = "{peering.subscription}"
    resource_group_name = "{peering.resource_group}"
    vnet_name           = "{peering.vnet_name}"
    fail_if_not_found   = {'false' if force_no_fail else str(not unmanaged).lower()}{hub_override}
  }}
  vnet_spoke = {spoke_line}{options_line}
  providers = {{
    azurerm.spoke = azurerm
    azurerm.hub   = azurerm.{peering.provider}
  }}
}}
"""

The hub is built from the config table. The spoke is module.<name>.peering — a reference the generator does not need to understand, because the VNet module is going to hand over an object of exactly the right shape.

Which is the actual trick.

Organic Connectivity: The peering Output

The VNet module has an output whose only reason to exist is to be passed to the peering module:

output "peering" {
  description = "The peering information for the virtual network."
  value = {
    resource_group_name = azurerm_virtual_network.main.resource_group_name
    vnet_name           = azurerm_virtual_network.main.name
    address_space       = azurerm_virtual_network.main.address_space
    vnet_id             = azurerm_virtual_network.main.id
    module_output       = true # This is a flag to prevent conditions on results that are known on apply.
  }
}

So the generated spoke side is:

  vnet_spoke = module.spoke-vnet.peering

One line. Not seven inputs the generator has to look up, construct, and keep consistent with the module’s schema — one reference to a contract the module publishes about itself.

Every consequence of this is good:

• The generator does not know a VNet’s ID, address space, or resource group. It doesn’t ask. It couldn’t answer — those are known at apply time, and the generator runs before init.
• Terraform’s dependency graph gets the edge for free. The peering waits for the VNet because it references it. No depends_on, no ordering hints in generated code.
• Change the VNet module’s internals and the peering module doesn’t care, as long as the output shape holds. That’s a real interface, versioned by the tags from the release pipeline.
• The address space arrives without an API call. Terraform already knows it — it just made the thing.

module_output, the ugliest name in the schema

module_output = optional(bool, false) # This is a flag to prevent conditions on results that are known on apply.

It reads like an implementation detail leaking into a public interface, and that’s precisely what it is. It’s also load-bearing, and it’s the fix for a class of Terraform problem that anyone who has written conditional data sources has hit.

The module wants to decide whether to look a VNet up. The natural way is to test whether you were given the data:

# What you want to write. Do not write this.
locals {
  needs_lookup = var.vnet_spoke.address_space == null ? 1 : 0
}

But address_space comes from module.x.peering, which comes from a resource that doesn’t exist yet. During plan, its value is (known after apply). Terraform cannot evaluate count or for_each on a value that isn’t known at plan time, and it will tell you so, at length, in an error that has ruined many afternoons.

So the flag is not “did you give me data.” It’s “do you promise you will have given me data by apply time” — a statement about provenance, not value, which is knowable at plan time because it’s a literal boolean written by whoever wired the modules together.

true from the real VNet module: the data comes from a resource, it’s unknown now, it will be known later, don’t look anything up. false from a hand-written block: the data may genuinely be missing, go and look.

Terraform then plans cleanly:

locals {
  vnet_spoke_id_set            = var.vnet_spoke.module_output
  vnet_spoke_address_space_set = var.vnet_spoke.module_output
  ...
  vnet_spoke_skip_all = local.vnet_spoke_address_space_set
}

I have tried several times to give this variable a better name — data_known_at_apply, provided_by_module, defer_lookup. All of them are more honest and none of them survived, because by the time you’re renaming it, it’s in the API of a module with a lot of consumers and the rename is a Major version and a coordinated migration. Name your flags carefully the first time. This is a public service announcement.

Three Lookup Paths

Making that flag explicit exposed something the module had been doing badly, invisibly, at scale.

 * ## Data Lookup Paths
 *
 * The module uses three data lookup strategies per side (spoke/hub), selected automatically:
 *
 * - **Skip all** (`module_output = true`): All data is provided directly. No API calls are made.
 * - **Direct lookup** (`module_output = false`, `fail_if_not_found = true`): Looks up the vnet by
 *   name and resource group. Terraform will error if the vnet does not exist. (1 API call)
 * - **Discovery** (`module_output = false`, `fail_if_not_found = false`): Lists all vnets in the
 *   subscription to safely check existence. If not found, the peering is silently skipped. (N+1 API calls)

The API call counts are in the documentation because they are the entire reason the paths exist.

Every unit generates a peering module per hub, and before this work each one did an azurerm_subscription lookup and a VNet lookup per side — four data sources apiece, every plan, every apply, multiplied by every unit in the pipeline. Azure Resource Manager rate limits are generous but they are not infinite, and plans spend their time waiting on the API with the CPU idle.

Fixing it took three passes: stop looking things up when the data was already to hand, stop asking Azure for a subscription ID the caller had typed into the config file, and filter the discovery search so it stopped returning networks nobody had asked about.

The middle one is the cheapest win available in Terraform and worth stealing wholesale. The module needed a subscription ID to build a VNet resource ID, and it got one by querying data.azurerm_subscription, which is an API call to learn something the caller already knew:

data "azurerm_subscription" "spoke" {
  provider = azurerm.spoke
  count    = local.vnet_spoke_id_set || var.vnet_spoke.subscription_id != null ? 0 : 1
}

locals {
  vnet_spoke_id = local.vnet_spoke_id_set ? var.vnet_spoke.vnet_id : (
    var.vnet_spoke.subscription_id != null
    ? "/subscriptions/${var.vnet_spoke.subscription_id}/resourceGroups/.../providers/Microsoft.Network/virtualNetworks/..."
    : "${data.azurerm_subscription.spoke[0].id}/resourceGroups/..."
  )
}

Pass the ID, skip the call. Azure resource IDs are a documented, stable string format. Constructing one from parts you have is not a hack; querying an API to discover a value you typed into a config file five lines earlier is the hack.

Discovery, and the 404 that ruined the point

The discovery path is the fiddliest and the most interesting, because it exists to support fail_if_not_found = false — this VNet might not exist, and that’s fine, skip the peering.

The obvious implementation is data.azurerm_virtual_network with a try(). It doesn’t work: a data source that can’t find its target is a hard error in Terraform, and try() doesn’t catch it.

So: list, then filter.

data "azurerm_resources" "spoke_vnets" {
  provider = azurerm.spoke
  count    = local.vnet_spoke_discovery ? 1 : 0
  type     = "Microsoft.Network/virtualNetworks"
  name     = var.vnet_spoke.vnet_name
}

data "azurerm_virtual_network" "spoke" {
  for_each = local.vnet_spoke_discovery ? {
    for vnet in data.azurerm_resources.spoke_vnets[0].resources : vnet.name => vnet
    if lower(vnet.resource_group_name) == lower(var.vnet_spoke.resource_group_name)
  } : {}
  ...
}

An empty list is not an error. for_each over an empty map creates nothing. Existence becomes a value you can branch on rather than an exception that kills the plan.

Now read the comment above it, which is the good bit:

# Discovery path (fail_if_not_found=false): subscription-wide search by type+name,
# filtered to the target RG on the consumer for_each. The RG is intentionally NOT
# passed to azurerm_resources: the RG-scoped list API 404s on a missing RG, which
# would defeat fail_if_not_found=false. Subscription-wide search returns [] safely.

The natural optimisation is to scope the list to the resource group. It’s narrower, it’s faster, it’s obviously correct. And it reintroduces exactly the failure you were avoiding — because when a whole subscription is being built for the first time, the resource group doesn’t exist either, and the RG-scoped list API returns 404 rather than an empty list.

I made that optimisation, shipped it, and it broke the first greenfield build that came along afterwards — a code path you exercise rarely, which is exactly long enough to forget why the slower version was the right one. That comment exists so nobody re-applies the optimisation in eight months, and it says why, not what — the only kind of comment worth the keystrokes.

Note also lower() on both sides of the resource group comparison. Azure resource group names are case-insensitive for lookup and case-preserving for display, so the same RG can come back with different casing depending on which API you asked. This is the sort of thing you learn once.

network_unmanaged: A Module That Creates Nothing

Here is the piece I’m most pleased with, and it’s a module that deploys zero resources.

The problem: not every VNet a spoke needs to peer with is managed by this repository. Some belong to other owners entirely. Some are legacy, built by hand years ago, and touching them is out of scope forever. Some are about to be adopted but aren’t yet.

The generator’s spoke side is module.<name>.peering. If a VNet isn’t a module, there’s no output to reference, and the whole organic-connectivity design falls over.

So make it a module. Just not one that builds anything.

/**
 * # Virtual Network Unmanaged Terraform Module
 *
 * This module does not create any resources, but is used to define a virtual network not managed
 * within the customer deployment repository.
 *
 * In effect, it allows automation (peering generator) to reference the virtual network, as though
 * it were created by the virtual network module.
 */

Its entire functional surface:

output "peering" {
  description = "The peering information for the virtual network."
  value = {
    subscription_id     = var.subscription_id
    resource_group_name = var.resource_group_name
    vnet_name           = var.vnet_name
    module_output       = false # This is a flag to prevent conditions on results that are known on apply.
    override            = var.vnet_peering_name_override != null ? { name = var.vnet_peering_name_override } : null
  }
}

Same output name, and every attribute the peering module needs. module_output = false — because this VNet’s data comes from user input describing something Terraform doesn’t manage, so the peering module must go and look it up, and must tolerate not finding it.

The generator needs one conditional for the whole concept:

fail_if_not_found = {'false' if force_no_fail else str(not unmanaged).lower()}

Read that carefully, because the flag is set on the hub block while the value comes from whether the local network is managed. That is deliberate, and it is the bit worth explaining: a locally-managed network is part of an estate that builds its own hubs, so a missing hub means something is badly wrong and the run should stop. A network the repository doesn’t manage is by definition sitting in territory it doesn’t control, so a hub that isn’t there yet is a normal state and the peering should be skipped quietly rather than failing the apply.

And because it’s a module with a marker in its source, everything upstream just works — the generator scans for NETWORK_UNMANAGED_MODULE alongside NETWORK_MODULE, and unmanaged VNets are peering-enabled unconditionally, since declaring one has no other purpose:

    for vnet_mod_name, vnet_mod_data in vnets_unmanaged.items():
        enriched = dict(vnet_mod_data)
        enriched['region'] = vnet_mod_data['peer_region']
        enriched['peer'] = True
        enriched['unmanaged'] = True

Note the region handling. A managed VNet’s region is derived from its bootstrap module (Part 1). An unmanaged one has no bootstrap — it isn’t ours — so it declares its own, validated:

variable "peer_region" {
  description = "The region group to peer with..."
  type        = string
  validation {
    condition     = can(regex("^(region-a|region-b|region-c)$", var.peer_region))
    error_message = "The peer_region must be one of the following: region-a, region-b, region-c."
  }
}

The regex is a maintenance liability — it will drift from peering_config.json and someone will spend twenty minutes confused. It has also caught every typo anyone has made in this field. Worth it, for now.

The use case I didn’t design for

Two use cases are documented. The first is the obvious one. The second was discovered:

 * 2. To allow migration of a virtual network from unmanaged to managed, by first defining the
 *    virtual network in an unmanaged state, and then later moving it to a managed state without
 *    needing to update references to the virtual network in other modules (peerings, subnets, etc.).
 *    This allows for a more seamless migration with less risk of errors.

Because both modules publish a peering output that the peering module accepts — same name, mutually optional attributes — adopting a legacy VNet is a module swap in one file:

# Before — describing something we don't own
module "legacy-vnet" {
  source              = "git::https://NETWORK_UNMANAGED_MODULE"
  subscription_id     = "00000000-0000-0000-0000-000000000002"
  resource_group_name = "legacy-rg"
  vnet_name           = "legacy-vnet"
  peer_region         = "region-a"
}

# After — importing and managing it for real
module "legacy-vnet" {
  source            = "git::https://NETWORK_MODULE"
  bootstrap         = module.bs_region-a
  generate_peerings = true
  # ... full VNet configuration
}

Every peering block referencing module.legacy-vnet.peering is unchanged. The regenerated peerings.tf is byte-identical apart from fail_if_not_found flipping to true. What changes is that module_output goes from false to true, and the peering module quietly stops making API calls because Terraform now knows the answers.

That property wasn’t designed. It’s what happens when two modules honour the same output contract, and it turned “adopt a legacy VNet” from a multi-stage change into a one-file swap plus a terraform import.

One caveat the shape hides: the unmanaged module can carry a peering name override, and the managed one cannot. If the legacy VNet’s peerings have names that don’t match the convention, move them into .peering_overrides.yml before the swap — otherwise the regenerated peering falls back to the default name, and a peering name is immutable.

Two Bugs Worth Your Time

The reverse trigger

Azure VNet peerings cache the remote network’s address space. Add a range to one side and the peering does not notice — the peer stays on the old view of the address space, silently, and traffic to the new range blackholes. In the portal this surfaces as a peering needing to be re-synced. In Terraform, it surfaces as nothing at all, which is worse.

The module forces replacement when the remote address space changes:

resource "azurerm_virtual_network_peering" "hub" {
  ...
  triggers = {
    remote_address_space = join(",", local.vnet_spoke.address_space == null ? [] : local.vnet_spoke.address_space)
  }
}

resource "azurerm_virtual_network_peering" "spoke" {
  ...
  triggers = {
    remote_address_space = join(",", local.vnet_hub.address_space == null ? [] : local.vnet_hub.address_space)
  }
}

Read those carefully, because the first implementation didn’t. The hub resource triggers on the spoke’s address space, and vice versa. A peering resource describes a view of the other network, so it must be replaced when the other network changes. Triggering each side on its own address space — which is the version that got written first, because it reads more naturally — produces a module that recreates peerings when nothing meaningful changed and, critically, does nothing at all in the one case the feature exists for.

There is no clever lesson here. Two symmetric resources, two crossed references, and a bug that a unit test caught only once someone thought to write the test for the remote side.

The null-guard is the second half: during discovery on a VNet that doesn’t exist, address space is null, and join() on null is an error rather than an empty string. A module that plans fine for existing networks and explodes on new ones is a module that only breaks on greenfield builds. See also: the 404, above. There’s a pattern here, and the pattern is test the empty case.

moved, or how to add count without an outage

Conditional creation — count = local.create_peering — arrived long after the module was already in production across the estate.

Adding count to an existing resource renames its address in state. azurerm_virtual_network_peering.hub becomes azurerm_virtual_network_peering.hub[0]. Terraform, seeing an address it doesn’t recognise and one it no longer has, plans a destroy and recreate. Several hundred times. On production connectivity.

The entire fix is a file:

moved {
  from = azurerm_virtual_network_peering.hub
  to   = azurerm_virtual_network_peering.hub[0]
}

moved {
  from = azurerm_virtual_network_peering.spoke
  to   = azurerm_virtual_network_peering.spoke[0]
}

Eight lines, committed as network_peering_moved.tf, in its own file so it’s obvious it’s a migration artefact rather than configuration. Every consumer upgrading to that version got a no-op plan instead of an outage, without knowing anything had happened.

This is the single highest-leverage feature in Terraform for anyone maintaining shared modules, and it is chronically underused. If your module version bump requires consumers to run terraform state mv, you have shipped a defect and outsourced it.

The Contract, Summarised

The generator writes the arrow. That’s all it does. It doesn’t know what a VNet is, what an address space is, or how to build an Azure resource ID — because the schema was designed so it never has to.

Which leaves one problem. Everything above produces identical peerings from a uniform estate. A legacy estate is not uniform. It has peerings whose names don’t match the convention, links that need forwarded traffic disabled, and hubs that some units must not reach at all.

That’s Part 3.