From Three Dead Nodes to a Network with a Heartbeat

On the night of October 25th, a power surge killed three computers on my home network. When the power came back, nothing restarted on its own. And I discovered that the fastest way to tell whether a PC under the desk is on is to bend down and look at the LED.

It wasn't even the first time. A few days earlier, one of the three nodes had stopped booting and I had reinstalled it from scratch. That night, another one had frozen so thoroughly that it took me two manual reboots just to get back in via SSH. Crawling under the desk, in the dark, I wrote two lines in my journal: “Never again.”

The next evening, at 6:40 PM, I rebooted the hub — broker, database, API and dashboard all together — without touching a keyboard. Four minutes later, everything was online again. The day of the incident had become the test suite the system passes on its own.

But the real story isn't the reboot. It's how I got there without losing my mind, starting from a single certainty: AI on its own produces noise.

Hero: the three nodes under the desk, the night of the incident

The Ghost of the Weather Agents

Some time ago I played with AI agents for weather. Instructive experience: every time I asked for a change, the generated file was different. Layout flipped, UI rebuilt, structure reshuffled. Every iteration was a little Groundhog Day: starting almost from scratch for a ten-line fix.

They call it “vibe coding”: uncontrolled, chaotic, non-reproducible. Fun for a prototype, unsustainable for anything else.

The problem was already clear back then, but that night under the desk it took concrete shape: if I'm building a system that monitors something, I must first be able to monitor the system itself. Monitoring you have to rebuild after every change isn't monitoring, it's an anxiety generator.

The alternative to vibe coding is specification-driven development: controlled, orderly, extensible. You define what the system must do before asking the AI to do it. End of sentiment, beginning of rationality and method.

Building the Rails, Not the Train

Honest disclaimer: I can't program, and I'm not a sysadmin. I am, at best, a power user with plenty of curiosity and a soft spot for method.

So the first move wasn't writing code, let alone prompts. It was asking the AI the right questions: what are the points to define? Which parameters do I need to decide on? It helped me explore the problem and evaluate possible solutions. Those answers became my decisions: the architecture design, the goals, the system's boundaries.

Only afterwards did OpenSpec enter the stage — a framework for writing specifications and guiding development with AI: proposal, specs, design, tasks. Every code change goes through it — no more “fix this thing for me” shouted into the void. Code generation and validation happened inside the rails of the specifications, not in the inspiration of the moment.

Here's the point: the skills I lacked weren't delegated to the AI. I acquired them through the AI and then crystallized them into specifications. I built rails for the AI to move within. Not a train that leaves on its own: a route I decided, even where I didn't yet know exactly where it led.

The Map: How the Pieces Talk to Each Other

The architecture that came out of that guided exploration is boring on purpose. And that's its strength.

At the center, an MQTT (Mosquitto) message broker working like a message board: producers post, consumers pick up, and neither knows about the other. Around the board:

The message board: sentinels, collector, database and dashboard A message board at the center: sentinels publish, the collector archives, the dashboard reflects. All decoupled.

A detail I like: even the hub monitors itself through the same board as the other nodes. One path for the same data — so database, APIs and dashboard always tell the same story.

The principle holding it all together is decoupling: producers and consumers don't talk to each other, they talk to the board. And the architectural choices weren't driven by monitoring alone: they were also the chance to equip myself with a reusable setup for other projects. That message bus is destined to become the backbone of my home systems — more on that at the end.

The Roller Coaster

The rest of the day was an obstacle course lasting about seven hours. Three nodes to set up, six bugs encountered, and the AI as a teammate under the supervision of the specifications.

I won't walk you through the bugs in detail — the raw documentation lives in the repository's knowledge base — but the interesting part is the pattern: each bug was born, the method caught it, the diagnosis became a knowledge base entry with its own root cause, and we moved on.

The day's takeaway, in the end, wasn't “the AI got it wrong six times”. It was “six times the method turned an error into an asset”.

Laziness as an Engineering Force

Having installed the first node by hand, looking at the two remaining ones I paused, doubtful: the same sequence of manual commands twice more, hating it a little more each time.

That's when the lazy devops golden rule kicks in: if you have to do it more than once and you hate doing it, you automate it.

Thus was born the installer: a single script embedding the code, asking for credentials securely, verifying they actually work against the live system before touching the disk, repairing what it finds broken, and declaring itself done only when it sees the first heartbeat in the log.

Adding a fourth node now would cost one command. But I'll say that with pride at the end of the post.

The Queen's Test

A monitoring system that doesn't pass tests isn't monitoring, it's good faith. So, the final round of proofs:

The night three nodes died became the test those same three nodes now pass on their own. There's a symmetry in this that I deeply appreciate.

The rails: the human with the blueprint decides, the robot lays the tracks Specification-driven development: the human decides the route, the AI lays the rails to move within.

What I Learned

AI can write code, but shipping a software product is a different thing. You need to guide it onto the right rails, with method and control — otherwise it's chaos, in a slightly fancier sauce than my under-the-desk chaos of October 25th, but still chaos you can't afford.

The realization that surprised me most: the engineering is in writing the specifications, not the code. The code, by now, AI knows how to write. What separates a system from a pile of files is what you decide before the code: contracts, boundaries, falsifiable tests. And the good news for anyone who, like me, isn't a programmer: those skills can be acquired together with the AI, if you use it in the right direction, building the rails for it to run on.

Then there's the future. The message bus at the center of the architecture wasn't born just for monitoring: it's the asset I intend to converge my home systems on. The next step is home automation — sensors and devices producing and consuming messages on the same board as the sentinels. Then notifications, new events to monitor, and maybe a fourth node. Which, I repeat casually, would now cost one command. Basically, imagination is the limit!

The repository is public on Codeberg: code, method and knowledge base.

(Read this in Italian)

“Free technology isn't just code — it's the freedom to look under the hood.”

👤 Lorenzo | ItsMeLarry 🧪 AI & Open Source Experiments | 🕹️ Retro Gaming & Graphic Novels 🎸 Rock, Folk, Jazz, Punk | 🔒 Privacy First 📍 Tuscany / Digital

🔗 noblogo.org/itsmelarry-en | 🐙 Codeberg