DaaC: The Lost XaaC Nobody Talks About

I needed three new VMs. Fine, easy, except I needed to know which IPs in the subnet were actually free first, and the "system of record" for that was, depending on the day, either a NetBox instance nobody had touched since the person who set it up left, or a Google Sheet that three people had "the current version" of. So I did what everyone does when they don't trust the doc: I started SSHing into random addresses in the range, one at a time, waiting to see who picked up. Got a response, that one's taken. Connection refused, maybe free, maybe just firewalled, who actually knows. Twenty minutes of digital door-knocking just to provision three boxes that, in a sane world, should have taken thirty seconds.

That specific flavor of task, the one that requires real, sustained effort but that nobody actually owns, nobody gets credit for, and everybody silently resents, is one I think every infra person has a version of. Maybe yours is your boss asking, out of nowhere, for "a quick report of all the assets we have in the company," and you realize there is no button for that, there's just you, three tools, and an afternoon. Maybe it's opening NetBox to update one VM's metadata and getting pulled into a small, personal blackhole, because sitting right next to the box you meant to edit are four other entries for VMs that got decommissioned eight months ago and nobody deleted the record. I have something close to OCD about this specific thing, and I mean that mostly seriously, stale entries sitting next to real ones make my skin crawl in a way that's probably not entirely rational, but also, if a source of truth has ghosts living in it, is it actually a source of truth, or just a haunted spreadsheet with extra steps?

I'm not telling you this because it's a dramatic story. None of these are. That's exactly why they never get fixed, they're all just annoying enough to eat an afternoon and never quite annoying enough to justify stopping and building something. Until, apparently, one of them was.

The spreadsheet was already lying, we just hadn't caught it yet

Here's the thing about that IP hunt that bugged me for days afterward, not the twenty minutes it cost, but why it cost twenty minutes. We had monitoring. We had Grafana dashboards you could get lost in for an afternoon. We had, God help us, a spreadsheet, tab-separated by cluster, color-coded, someone's genuine labor of love at some point in the past, plus a NetBox instance that was one part accurate and two parts graveyard. What we didn't have was one boring, ground-truth answer to "what is actually running on this network, right now, according to the machines themselves."

Every source we had was either declared (the spreadsheet, NetBox, someone's memory) or aggregated after the fact and immediately stale (a monitoring tool that only knows about hosts that already have its agent installed, which is a wonderfully circular way to find out about hosts that don't). Nothing was actually walking every reachable box and asking it, directly, "what are you, and is anyone using you," no opinions, no cache, no ghost entries from a VM that got torn down in March.

So I built that thing. I called it Auto-Doc, because I have never once in my life come up with a clever name for anything and I've made peace with it.

What it actually does, and why the boring parts are the point

Auto-Doc is an Ansible playbook. You give it subnets. It scans them, finds every machine that answers, SSHes in with nothing but raw shell commands, cat, grep, df, lsblk, and writes out flat JSON files describing each host's OS, CPU, memory, storage, network, installed tools with exact versions, and running Docker containers. One command. No inventory file to hand-maintain. No agent to install anywhere, ever.

ansible-playbook main.yaml

That's genuinely it. No -i flag. Subnets live in a subnets var at the top of the file, and the very first thing that happens is discovery: scan, find what's alive, resolve its real identity, then immediately document it, all in the same run.

We've named basically every "as Code" pattern that matters by now. Infrastructure as Code. Configuration as Code. Policy as Code. Pipeline as Code. We turned the entire discipline into a family of acronyms so we'd stop treating each of those things as some special, hand-maintained artifact and start treating them like source, versioned, reviewed, diffable, boring in the best way. And then somehow the one artifact that describes what's actually running, the thing every incident retro eventually needs, never got the same treatment. It stayed a wiki page. A spreadsheet. Somebody's memory. The lost XaaC nobody named: Documentation as Code, DaaC, if we're going to commit to the bit. Not documentation generated once and left to rot, documentation that's re-derived from the machines themselves, every run, committed, diffed, trusted the same way you'd trust a Terraform plan.

That's genuinely the whole design philosophy underneath Auto-Doc, once I actually put a name to it. And it turns out that once you actually commit to that philosophy, "just SSH in and read some files" stops being simple almost immediately. Every single component, IP resolution, storage, tools, all of it, has a naive version that breaks on real infrastructure within about a day, and a not-naive version that has to actually earn the right to call itself DaaC. What follows is basically a tour of those fights, one component at a time.

Fight one: which IP is actually yours?

The naive version of subnet discovery is: ping-sweep, SSH in, ask for the hostname, done. That works until a machine answers on more than one IP, which happens constantly the moment you have anything resembling a floating IP, a secondary NIC, a management interface alongside a service interface. Which IP is the "real" one? The naive answer picks whichever one happened to respond first, which is basically a coin flip dressed up as a decision, and a coin flip is exactly the kind of thing you do not want silently baked into a document people are going to trust.

So the actual logic goes deeper than that, in three deliberate layers. For every hostname discovered on more than one candidate IP, it checks the box's own netplan config first, actually reads /etc/netplan/*.yaml on the target and looks for a static IP assignment matching one of the candidates. That's the strongest possible signal, the box itself has a config file declaring "this is my address." If exactly one candidate matches, that's the answer, high confidence, done. If netplan doesn't resolve it cleanly, either there's no netplan config, or more than one candidate matches it, it falls back to asking the box which IP it would use to reach the outside world, ip route get 1.1.1.1, and checks if that source address matches one of the candidates. That's a weaker signal than a declared static config, but it's still the kernel's own routing table telling you something real about which interface it actually considers primary. And if that doesn't resolve it either, it falls back to just picking the numerically lowest candidate IP as a last resort, and, this is the part I actually care about most, it explicitly labels that result as a guess and logs a warning naming the exact host and every IP it saw. Nothing gets silently decided. If the tool had to guess, you know it guessed, and you know exactly what it guessed between.

Why go to that trouble instead of just always trusting ip route? Because on a box with policy routing, VPN interfaces, or a management network that doesn't carry default-route traffic, the "IP the kernel would use to reach the internet" and "the IP everyone actually SSHes to and calls this machine by" are sometimes two completely different addresses. Netplan's static declaration is closer to human intent. The route table is closer to kernel behavior. They usually agree. When they don't, that disagreement is itself useful information, and burying it behind a silent pick would be exactly the kind of quiet, confident wrongness that got me into this mess in the first place with that spreadsheet.

Fight two: storage, or, why df and lsblk both lie a little

If you've ever tried to write something that reports "what disks does this machine have" in a way that's actually true across bare metal, VMs, LVM setups, and network mounts all at once, you already know this is a nightmare with about six different flavors of nightmare hiding inside it. This is the densest file in the whole project, and every line in it exists because some earlier, simpler version of it produced a wrong or misleading answer on a real machine.

Start with the source. /proc/mounts is the ground truth for "what's actually mounted right now," but it's noisy. On any Kubernetes node, every single pod's emptyDir, secret, and configMap volume shows up in there too, which means a naive scan of a K8s worker returns dozens of irrelevant entries under /var/lib/kubelet/pods for things that aren't storage in any meaningful sense, they're ephemeral pod plumbing. So that path gets explicitly excluded, along with /dev/loop* devices, which are almost always squashfs mounts from snap packages, not real storage anyone cares about documenting.

Then there's the LVM problem. When a filesystem sits on a logical volume, the device name you get back is something like /dev/mapper/ubuntu--vg-ubuntu--lv, which tells you precisely nothing about the physical disk underneath it. So the script walks /sys/block/<device>/slaves/, the kernel's own record of which raw block devices back a given logical device, and resolves the mapper name down to the actual disk. Without that step, every LVM-backed host in the fleet would report storage as living on a device that doesn't correspond to anything you could point to in a datacenter.

Vendor, model, and transport type all cascade through the same pattern, three separate fallback chains, because no single source reliably has all of it. lsblk gets asked first, since it's usually right and it's cheap. When it comes back empty, which happens more often than you'd hope, especially on virtio devices, the script reads straight from /sys/block/*/device/vendor and /sys/block/*/device/model instead, the same files lsblk itself would have read, just accessed directly. And when even that's empty, it falls back to inference from the device name itself: a scsi_device directory under sysfs means sata or scsi, an nvme*-prefixed name means nvme, a vda*-prefixed name means virtio. None of those individually is bulletproof. Together, stacked in that order, they cover essentially every disk you'll actually meet in the wild.

And then two special cases that exist purely because I got burned by them specifically: anything with rbd in its source path gets explicitly labeled Ceph, RADOS Block Device, transport network/rbd, because Ceph's naming otherwise sails straight through every heuristic above without matching any of them. And anything mounted with an nfs, ceph, or cifs filesystem type gets bucketed generically as network_mount, vendor network_vendor, model remote_share, because a network share genuinely doesn't have a vendor or model in the way a physical disk does, and reporting unknown for something that's supposed to be unknown, by nature, felt like the more honest answer than pretending otherwise.

Fight three: sixteen tools, sixteen different opinions about what --version means

This is the fight that's the least glamorous and probably the most tedious to have actually had. Every one of the roughly sixteen tools Auto-Doc knows how to check for, Docker, containerd, nginx, Redis, Teleport, Zabbix Agent 2, k3s, kubelet, and so on, formats its own version output slightly differently, and "slightly differently" is doing a lot of work in that sentence.

Teleport prints its version with a leading v that has to get stripped. k3s prints a v prefix and a trailing +-something build suffix, both of which need stripping before you get a clean semantic version. Zabbix Agent 2's version string is buried inside parentheses in the middle of a longer line. Each of these got its own small, specific awk/sed extraction rule, because there was no shortcut, no --version --json flag any of these tools consistently support. This is just what happens when you commit to reporting real, exact, comparable version numbers instead of "yes, it's installed" as a boolean.

The naming mismatches were their own small hell. Zabbix Agent 2's binary is zabbix_agent2, underscore, but its systemd service is registered as zabbix-agent2, hyphen. A naive systemctl is-active zabbix_agent2 silently reports "inactive" forever on a host where it's actually running perfectly fine, because you asked about a service name that doesn't exist. The check explicitly special-cases this one binary-to-service-name translation, because I hit it in exactly one production host and it was enough to make me paranoid about every other tool on the list too.

And figuring out how something got installed, apt-managed versus a standalone binary someone dropped in by hand, turned out to have a cleaner answer than I expected: ask dpkg whether it recognizes the file at all, dpkg -S <path>. If dpkg claims ownership of that exact path, it's apt-managed. If dpkg has never heard of it, it's standalone. No need to know in advance how each tool is typically installed across sixteen different tools and however many Linux distributions, dpkg already keeps that ledger, you just have to ask it the right question.

Fight four: the IP inventory doesn't assume you only have one subnet

I described the IP inventory's shape as an honest tradeoff before, and it is one, it's a pivoted table, wide rows keyed by host octet, not a flat list, which does mean anything downstream that expects flat JSON has to reshape it first. But I undersold something about it while I was busy being self-critical: the pivot isn't hardcoded to a single subnet. It derives its columns dynamically, at run time, from whatever unique subnets actually show up in network.json. Scan one subnet, you get one subnet_<segment>_ip / subnet_<segment>_target_name / subnet_<segment>_target_group triplet of columns. Scan five, you get five triplets, automatically, with zero configuration telling it how many to expect. The jq that builds it reduces over whatever subnet keys it actually finds and generates the column structure from that reduction, rather than a person having to declare, somewhere, "this fleet has N subnets, expect N sets of columns." It grows with your fleet without you touching it.

Fight five: not letting one bad host ruin the whole run

This one's less a clever trick and more a decision I almost got wrong the first time. The obvious way to write a fleet-wide documentation playbook is to just let Ansible do what it does by default: if a host is unreachable or a task fails on it, that host drops out, sometimes loudly, sometimes in a way that eats the rest of the run's attention. Across forty, fifty, a hundred-plus hosts, on any given run there is always one box that's mid-reboot, or firewalled differently than the rest, or just having a bad day.

So the documentation play runs with ignore_unreachable: true at the play level and ignore_errors: true wrapped around the whole block of component tasks. One flaky host gets skipped, quietly, and the other ninety-nine still get fully documented in the same run. And the SSH timeout settings are tuned specifically for this workflow, not a generic default: ConnectTimeout=120, generous, because you've likely just discovered this host seconds ago in the same run and its SSH banner might be slow to answer, but ConnectionAttempts=1, because retrying a host that's genuinely down just burns time you'd rather spend documenting the hosts that are actually up.

Fight six: discovery that doesn't require you to have installed anything first

The scan itself tries nmap first, because it's fast and it's good at this exact job. But it doesn't hard-require it. If nmap isn't on the control node, discovery falls back to a pure Bash port-scan using /dev/tcp, Bash's built-in ability to open a raw TCP connection without any external tool at all, with a manually implemented concurrency cap using job control, jobs -r | wc -l, so it doesn't try to open two hundred and fifty simultaneous connections and fall over. It's slower than nmap. It also means the very first time you ever run this against a fresh control node, before you've apt-installed a single thing, it still works.

And there's a small side effect of the discovery step I didn't originally think of as a feature, but I've come to appreciate as one: it writes hosts.yaml, a genuinely valid, grouped Ansible inventory file, as a record of what it found, even though the live run never reads that file back in, it works entirely from in-memory add_host calls instead. You get a real, usable static inventory as a byproduct of a run that technically didn't need one, in case you want to point some other, more targeted playbook at this exact fleet later without re-discovering it from scratch.

And the dashboard side doesn't cache anything either

The Grafana half of this is a genuinely tiny Go program, json-exposer, that globs confs/*.json and serves each file over HTTP. The detail worth mentioning: it re-reads every file straight from disk on every single HTTP request. No in-memory cache, no "restart the container to pick up new data." Run the Ansible playbook again while Grafana's sitting open in a browser tab, hit refresh, and the new numbers are just there. And its index route, the plain HTML page listing what's currently being served, is generated from whatever's actually present in the folder at request time, so adding a ninth component file later means it shows up in that list automatically, no code change required on the exposer side at all.

Where this actually sits, next to everything else that sort of does this

I've said a version of this before in passing, that Auto-Doc is adjacent to a handful of tools rather than competing with any of them, and I want to actually put that in a table instead of just asserting it, because "adjacent" is a claim that should survive being checked.

A few things I want to say plainly about that table instead of letting it sit there implying more than it should.

Nobody actually owns the full row. That's the actual point: phpIPAM knows nothing about what's running on a host. NetBox knows nothing about what's actually installed versus what was declared months ago. Wazuh correlates packages against CVEs beautifully but has no concept of IP allocation or physical resource footprint at all. Getting the full picture, "which IP, which specs, which package versions, all keyed to the same hostname, in one place," otherwise means running four separate stacks and manually joining their outputs yourself. That join is the thing that's actually missing from the ecosystem, and it's the thing Auto-Doc does by construction, for free, just by using the same hostname and IP as the join key across every single component file.

The cost and footprint really aren't close, either. NetBox is a Django app plus Postgres. phpIPAM is PHP plus MySQL. Wazuh is a manager, an indexer, and a dashboard, three services minimum, plus a persistent agent on every host you want covered, which is a little funny given that keeping agents patched is usually the exact problem you're trying to get ahead of in the first place. Auto-Doc's entire footprint is one Ansible run, on-demand or on a cron, static files at the end of it, and, only if you actually want the Grafana view, one ten-megabyte Go container that caches nothing.

And it's genuinely more to-the-point because it's descriptive instead of administrative. NetBox and phpIPAM need someone to declare correct data and then keep it correct by hand, which is a second source of drift sitting right on top of the infrastructure it's supposedly describing. Auto-Doc just asks the machine what's true right now. For the tool-version-tracking row specifically, that discovered-truth model is actually a better fit than Wazuh's CVE-only correlation for one particular kind of problem: version deprecation, not vulnerability. A package can be perfectly CVE-free and still be three major versions past its support window, and that's a version comparison question, not a vulnerability-feed question. There's no correlation layer for that in Auto-Doc yet either, to be clear, I'm not going to pretend a shell script grew one overnight. But the discovered, ground-truth data it's already collecting is the right foundation for building that layer properly, someday, on top of something that's actually true, instead of on top of a spreadsheet with ghosts in it.

Seeing it

I built a small Grafana dashboard on top of this, Infinity datasource pointed straight at the JSON files, one tab per component:

Both of those are running against fake demo data, obviously, forty-five fictional hosts I generated myself, not our real fleet. But the shape of it is exactly what you'd get pointed at your own subnets. docker compose up -d, import the dashboard, and ten minutes after cloning the repo you're looking at a fully populated view without touching a single real host.

Where this actually leaves things

The next time I need to spin up a few new VMs, or my boss asks for that "quick asset report," I want the honest answer to take five seconds and a jq one-liner, not another twenty minutes of digital door-knocking through a subnet, and definitely not another trip through a NetBox instance with ghosts in it. That's the whole project, really, start to finish. Every fight above exists because the easy version of that fight produced a wrong answer somewhere on a real machine, and a slightly harder version didn't. We named every other "as Code" that mattered. This one was just waiting for someone to bother.

Links

Repo: github.com/hatam-abolghasemi/Auto-Doc
Originally published on Medium