The Cloud Has No Undo Button

Azure has an entire product line for backup. Recovery Services vaults, Backup Center, point-in-time restore, geo-redundant recovery points, soft delete with a fourteen-day grace period. It will back up a virtual machine, a managed disk, a SQL database, an Azure Files share, a blob container.

It will not back up your network.

Not the VNets. Not the subnets, the NSGs, the route tables, the peerings, the ExpressRoute circuits, the firewall policies, the private DNS zones, or the carefully-worked-out route entries that make traffic go the right way. The configuration of the platform itself — the part that took years to get right and that everything else sits on top of — has no backup product, no restore point, and no undo.

Delete a route table and it’s gone. Immediately. There is no vault it lands in, no recycle bin, no fourteen-day soft delete. Azure will ask you to confirm, and then it will do exactly what you asked.

”But Surely—”

Let’s go through the things people suggest, because I went through them too.

“The Activity Log.” Ninety days of retention, and it records that a change happened — who, when, which operation, success or failure. It does not record what the resource looked like beforehand. The request body is sometimes there, sometimes truncated, and never for the resource you actually need. It answers “who deleted it,” which is useful for the post-incident review and useless at 2am when you’re trying to put it back.

“Resource Graph.” Wonderful tool, and the foundation of everything below. It tells you what exists right now. Its change tables retain about a fortnight, which is not the horizon you need — you cannot ask it what the estate looked like last quarter.

“Azure Resource Manager templates / export.” Exporting a resource group’s template is a genuinely useful feature that produces something nearly redeployable, drops properties it doesn’t understand, and has to be run by a human, per resource group, on purpose, in advance. Nobody has ever done that on a schedule across a whole estate.

“Terraform state.” This is the good answer, and it’s the one that nearly works. If everything is in Terraform, terraform state pull is a point-in-time snapshot of everything Terraform manages.

Everything Terraform manages. That’s the load-bearing clause. In any estate older than about eighteen months, a meaningful fraction of the network was built by hand — during an incident, during a migration, or back when the Terraform provider didn’t support that resource yet. Those resources are exactly the ones nobody understands, exactly the ones with no documentation, and exactly the ones a backup would be most useful for. Terraform state, by definition, cannot see them.

“Policy / DeployIfNotExists.” Prevents drift going forward. Doesn’t tell you what you had.

So: no product. Fine. The requirement is small enough to build.

What I Actually Wanted

One question, answerable in under a minute:

What did this NSG look like three weeks ago?

And its more urgent siblings — what changed in the platform overnight, what did that route table contain before someone tidied it up, which subscriptions gained a subnet this month.

Notice what this is not. It’s not a restore tool. It doesn’t need to re-create resources, resolve dependency order, or handle a region failure. It needs to answer questions about the past, accurately, quickly, and without anyone having remembered to do anything in advance.

Which reframes the problem entirely. This isn’t a backup problem. It’s a version control problem.

Network configuration has three properties that make it perfect for git and terrible for a backup vault:

1. It’s small. The entire network control plane of a large estate is a modest amount of JSON. The VM disks are terabytes; the config describing where those VMs sit is not.
2. It’s text. Structured, diffable, greppable text.
3. It barely changes. Day to day, 99% of it is byte-identical. Which means a content-addressed store deduplicates it to almost nothing, and the days it does change are exactly the days you care about.

Point 3 is the one that makes this work. A daily backup of something that rarely changes is nearly free in git and ruinously expensive in anything that stores full copies.

The Layout

One repository, private, that is 99% data:

estate-backup/
├── subscriptions/                    ← the backup itself, one file per resource
│   └── <subscription-guid>/
│       └── resource_groups/
│           ├── <rg-name>.json        ← the resource group's own metadata
│           └── <rg-name>/
│               └── microsoft.network/<type>/
│                   └── <resource-name>.json
├── subscription_lookup.json          ← guid → friendly name
├── bin/
│   └── backup_resources.py           ← the script
└── .github/workflows/backup.yml      ← the cron that commits it

The path is the Azure resource ID. Deliberately. An Azure resource ID looks like this:

/subscriptions/<guid>/resourceGroups/<rg>/providers/microsoft.network/virtualNetworks/<name>

and the file lands at:

subscriptions/<guid>/resource_groups/<rg>/microsoft.network/virtualnetworks/<name>.json

Same information, same order, one is a URL and one is a filesystem path — lowercased, because that is how Resource Graph returns type. That single decision means you never need an index to find something. If you have the resource ID — and during an incident, the resource ID is the one thing you always have, because it’s in the error message — you have the file path. cat it. Done.

Version 1: Short, and a Long Wait

The first version was about as simple as it gets. Query every microsoft.network resource in every subscription, write each one to its path, done.

The one decision worth explaining is Azure Resource Graph rather than the per-resource ARM APIs.

The ARM approach would be: list subscriptions, list resource groups, list resources, then GET each one at the correct API version for its provider. That’s tens of thousands of individual HTTP calls, each needing the right API version — and API versions differ per resource type, change over time, and return different property shapes depending on which one you pick.

Resource Graph is one KQL-ish query language over a pre-indexed copy of every ARM resource in the tenant:

q = "resources | where type contains 'microsoft.network'"

That’s the whole backup query. One expression, every network resource, every subscription the identity can see, with the full properties blob already denormalised. No API versions to track, no per-provider special cases, no thousands of round trips.

There are two things it doesn’t cover, and both needed special handling.

Resource groups aren’t resources. They live in a separate table, so a second query fetches them by name and writes the <rg-name>.json sitting alongside each <rg-name>/ directory. The RG’s tags and location matter — they’re often where the ownership information lives.

DNS records aren’t resources either. Private DNS record sets live in dnsresources, and there are a lot of them. Storing one file per A record would be tens of thousands of files carrying almost no information each. So they get packed per zone:

dnsresources
| extend Packed = pack_all()
| summarize properties = make_list(Packed) by subscriptionId, resourceGroup, managedBy
| extend type = 'microsoft.network/privatednszones', name = strcat(managedBy, '_recordset')

One synthetic <zone>_recordset.json per zone, containing every record in it. A DNS zone is one thing conceptually, and one file diffs beautifully when three records change.

Version 1 worked. It was also strictly sequential, and against a whole estate of paged results it took long enough that I stopped watching.

Making It Survive Contact With a Real Estate

Three things had to change.

Paging

Resource Graph caps results per response and hands back a skip token:

    st = None
    page = 1
    l = 100
    while True:
        rsp = run_query(q, [subscription], l, st)
        print(f"Subscription: {subscription} - Processing Page: {page} of {div_roundup(rsp.total_records, l)}")
        process_response(rsp.data)
        page += 1
        st = rsp.skip_token
        if not st:
            break

Unremarkable, except for the logging. Page 3 of 47 in a log that runs unattended every morning is the difference between “it’s still going” and “it’s hung,” and I have wasted real time on jobs that didn’t tell me which.

Throttling

Resource Graph rate limits per tenant, and an estate’s worth of paged queries will find that limit:

    for attempt in range(retry_attempts):
        try:
            response = rgc.resources(request)
            break
        except HttpResponseError as error:
            # Only a throttle is worth retrying. A malformed query fails the same way
            # every time, so retrying it just delays a useful error message.
            if error.status_code != 429 or attempt == retry_attempts - 1:
                raise
            backoff = wait_seconds * (2 ** attempt)
            print(f"Throttled, attempt {attempt + 1} of {retry_attempts}, waiting {backoff} seconds")
            time.sleep(backoff)

The first version of this caught HttpResponseError broadly and waited a flat five seconds, which meant a malformed query was retried three times before failing — three times the wait to reach an error message that was never going to change. Checking the status code first, and backing off exponentially, costs two lines and makes the failure mode match the failure.

Concurrency

Subscriptions are independent, so they can run in parallel. The Azure SDK’s synchronous client gets wrapped and run in a thread pool:

async def submain(subscription:str) -> None:
    """ Process a subscription """
    loop = asyncio.get_event_loop()
    await loop.run_in_executor(None, process_sub, subscription)

Then dispatched in batches of five:

    group_tasks_by = 5
    tasks = [ submain(sub) for sub in subs_list]
    for i in range(0, len(tasks), group_tasks_by):
        results_range = i+group_tasks_by if i+group_tasks_by < len(tasks) else len(tasks)
        print(f"Processing Group: {group_number} of {number_of_groups} ({i} to {results_range})")
        await asyncio.gather(*tasks[i:results_range])

Five is not a tuned number. Five is “high enough to be worth doing, low enough that the throttle retry above almost never fires.” Given that the alternative is finishing slightly sooner, there was no reason to push it.

This is worth naming honestly: it’s asyncio used as a batch scheduler around blocking calls, not real async I/O. The whole thing is run_in_executor wrapping a synchronous SDK. The concurrency is real and the runtime is dominated by waiting on Azure either way, so a true async rewrite would make it tidier rather than faster — but it is not the async rewrite the asyncio import suggests.

resources | where type contains 'microsoft.network'

$ git add subscriptions subscription_lookup.json
$ git diff --staged --quiet || git commit -m "Automated scan …"

The Bit That Makes The Diffs Readable

Two arguments, and they matter more than anything else in this post:

def write_file(file_path:str, data:dict) -> None:
    """ Write data to a file """
    with open(file_path, 'w', encoding='utf-8') as f:
        json.dump(data, f, indent=4, default=str, sort_keys=True)

indent=4 and sort_keys=True.

Without those, the whole thing is worthless. Azure does not guarantee key ordering in its responses, and it does not need to — JSON objects are unordered. So the same unchanged NSG, serialised on two consecutive days, can produce two different files. Byte-different, semantically identical.

Do that across the whole estate and every daily commit shows thousands of changes, none of which mean anything. The signal drowns instantly, nobody reads the diffs, and you have built an expensive way to store JSON.

With sorted keys and stable indentation, serialisation is deterministic: identical config produces an identical file, every time. Which means:

• A file that appears in a diff has genuinely changed.
• Git deduplicates the rest to nothing. Daily snapshots of a large estate, and the object store stays small, because almost every blob is identical to yesterday’s and gets stored once.
• git log -p -- <path> on any resource is its complete change history, in order, with dates and diffs.

That last one is the entire product. Every question I set out to answer collapses into a git command:

# What did this NSG look like three weeks ago?
git show $(git rev-list -1 --before="3 weeks ago" main):subscriptions/<sub>/resource_groups/<rg>/microsoft.network/networksecuritygroups/<nsg>.json

# What changed across the whole platform overnight?
git diff HEAD~1 --stat

# When did this route table last change, and to what?
git log -p -- subscriptions/<sub>/resource_groups/<rg>/microsoft.network/routetables/<rt>.json

No product, no console, no query language. The tooling was already on the machine.

One Commit A Day

The pipeline is deliberately boring:

on:
  schedule:
    - cron: "0 3 * * *"

jobs:
  Scan:
    permissions:
      id-token: write
      contents: write
    steps:
      - name: 'Az CLI login'
        uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}

      - name: Scan
        run: backup_resources.py

      - name: Commit
        run: |
          git config --global user.email "github-actions[bot]@users.noreply.github.com"
          git config --global user.name "github-actions[bot]"
          git add subscriptions subscription_lookup.json
          git diff --staged --quiet || git commit -m "Automated scan $(date -u +%Y-%m-%dT%H:%M:%SZ)"
          git push

OIDC federated credentials — id-token: write, no client secret anywhere. Give the identity Reader at the scope you want covered, and nothing else. A backup system that can only read is a backup system that cannot be turned into an attack on the thing it’s backing up, and read-only is genuinely sufficient here.

The scan overwrites every file it fetches; git works out what actually changed. That’s the whole design. There’s no state, no incremental logic, no “what did we have last time” bookkeeping — the previous run’s output is sitting in the working tree, and git status is the diff engine. Re-running it twice in a row produces no commit at all, which is the correct behaviour for something whose job is to record a state.

That git diff --staged --quiet || is load-bearing: git commit exits non-zero with nothing staged, so without it a day of genuinely zero change across the estate turns a healthy run red.

Where It Bit Me

The last batch

After the resources are written, the script collects the resource groups it saw and fetches their metadata in batches of a thousand:

    for idx, rg in enumerate(resource_groups):
        rg_string += f"\"{rg['name']}\", "
        ...
        if idx % rg_group_by == 0:
            # emit a batch
        elif idx == len(resource_groups):
            # emit the final partial batch

Both branches are wrong.

idx % 1000 == 0 is true at idx == 0, so the very first iteration emits a batch containing a single resource group. And enumerate yields indices 0 to len-1, so idx == len(resource_groups) is never true — the final partial batch is silently dropped. So you get a batch of one, then full batches, and whatever remainder is left at the end never refreshes at all.

The corrected version is the obvious one:

            if (idx + 1) % rg_group_by == 0 or idx == len(resource_groups) - 1:

I want to flag why this survived as long as it did, because the mechanism is more interesting than the off-by-one. It only affects the resource-group metadata files, not the resource files — so the backup was 99% correct, every day, and the missing 1% was the least interesting 1%. There was no error, no warning, no failed run. A bug that makes your job fail gets fixed the same morning. A bug that makes your job slightly incomplete can run for a long time, and this one did.

Nothing about “back up your infrastructure” tells you to reconcile the output against the source. Something should.

A great many files

One file per resource across a whole estate adds up fast. Git handles the history beautifully — deduplication does its job. Git handles the working tree less beautifully: git status has to stat every one of them, cloning is not quick, and any tool that walks the tree unprepared will sit there for a while.

If I were starting again I’d look hard at whether DNS-style packing should apply more widely — one file per resource is a lovely property for cat and a rough one for the filesystem. It’s a real trade and I picked the side that makes incidents easier, which I’d pick again, but the file count is something to decide on deliberately rather than arrive at.

It Stopped Being A Backup

Here’s the part I didn’t plan.

After a few months, the repo had accumulated something more valuable than backups: a complete, structured, machine-readable description of every network resource in the estate, in a stable schema, on disk, queryable without touching Azure at all.

Which turns out to be exactly what you need to solve a completely different problem — brownfield onboarding. Taking the resources that were built by hand years ago, that nobody understands and that Terraform can’t see, and bringing them under management.

The usual approach is terraform import plus a human transcribing the portal into HCL, resource by resource, getting it subtly wrong, and finding out at terraform plan. That’s weeks of work per subscription and it’s the reason brownfield estates stay brownfield.

But the backup already has the data. So the same repository grew an adapter that reads it and emits Terraform:

Discover  →  scan the JSON, index every resource across every subscription
Correlate →  group resources into module-shaped sets using ID cross-references
Render    →  resolve fields via YAML manifests, emit native HCL

module "app_network" {
  source        = "git::https://NETWORK_MODULE"
  name_override = "app-vnet"
  address_space = ["198.51.100.0/24"]
  subnets = [{ name_override = "app-subnet", nsg = true … }]
}
import { to = module.app_network… id = "/subscriptions/…" }

The output is module calls against the module library from the versioning series, complete with the version markers the configurator resolves:

module "example_network" {
  source        = "git::https://NETWORK_MODULE"
  name_override = "<vnet-name>"
  address_space = ["198.51.100.0/24"]
  subnets = [{
    name_override    = "<subnet-name>"
    address_prefixes = ["198.51.100.0/26"]
    nsg              = true
    delegation = {
      "Microsoft.Sql/servers" = {
        name    = "Microsoft.Sql/servers"
        actions = ["Microsoft.Network/virtualNetworks/subnets/join/action"]
      }
    }
  }]
}

Generated. From a backup. Of a resource nobody had written any Terraform for.

It emits import {} blocks alongside, so the acceptance test is the blunt one: import, plan, and the plan should be empty. Each generated module carries a confidence score — HIGH when every field came from Azure data, MEDIUM when something was defaulted, LOW when a field couldn’t be inferred — so a human reviews the uncertain ones instead of all of them. Adding support for a new module type is a YAML manifest, not Python.

The adapter is not published — it is estate-shaped in a way the backup script is not, and it deserves its own post rather than a paragraph. The point here is narrower: the backup became the source of truth that made the migration possible. Not because it was designed to, but because “a complete, deterministic, structured snapshot of your infrastructure” is a genuinely useful thing to have lying around, and it’s useful for far more than the disaster you built it for.

What To Steal

Check what your cloud provider actually backs up. It’s less than you think, and the gap is usually the control plane — the configuration rather than the data. Go and look for a restore point for your route tables. I’ll wait.

A backup you can git log beats a backup you can restore, for the failure you’ll actually have. Total loss of a VNet is rare. “Someone changed something and now it’s broken” is Tuesday. Optimise for Tuesday.

Determinism is the whole feature. sort_keys=True is the single most important line in the codebase. Non-deterministic serialisation turns a change log into noise, and noise is not read.

Mirror the provider’s identifier scheme in your storage layout. The resource ID is in every error message. If it’s also the file path, nobody needs to learn your index.

Let git be the state machine. No incremental logic, no bookkeeping, no “what did we have last time.” Overwrite everything, commit, and let content addressing sort it out. The whole scanner is stateless as a result.

Read-only is enough. A backup identity that cannot write cannot be turned into a weapon against the thing it protects.

Reconcile the output against the source. The batching bug ran for months because the job succeeded every time. Count what you fetched, count what you wrote, and complain when they disagree.

Azure should ship this. It hasn’t. It’s about 250 lines of Python and a cron entry, and I’d rather have the git history anyway.