How We Architected a Distributed IoT Sensor Network

(When the Edge Hates You)

In 2018, we set out to build a distributed, multi-modal sensor network for urban and agricultural environments. If you spend enough time building standard cloud applications, you start taking things for granted: reliable network connections, homogenous compute, and ephemeral state.

Building a data-fusion system across hostile physical environments completely shatters those assumptions. The physical edge is chaotic, and it forced a permanent mandate for our team: build deterministic, reproducible infrastructure, or fail.

This post describes how we built it, where standard cloud paradigms broke down, and how we survived the edge.

Lean by Default, Docker by Choice

Some background: at the edge, hardware heterogeneity is the rule, not the exception. Our fleet was a wild mix of Raspberry Pis and Nvidia Jetsons, requiring a multi-architecture approach across ARM and x86_64.

We initially designed the node systems to be extremely lean. By default, you could just download the code, set up your Python environment, and run it directly on bare metal. This was actually great for squeezing out maximum performance.

But relying on end-users to manually manage environments on a dozen different edge devices is a recipe for dependency hell and endless friction. To mitigate this, we pushed Docker as the primary deployment path to guarantee standardized, conflict-free environments. Bare-metal execution was always an option if overhead constraints demanded it, but containerization was our shield against environment drift.

"Trustless" Networking

Standard discovery protocols are great... until you're dealing with line-of-sight drops and physical signal jamming in a field. We knew we needed an SD-WAN solution to traverse networks and find endpoints, so we built our own custom overlay network.

We self-hosted a central control endpoint. When a device woke up and caught a WAN connection, it registered with this endpoint, which then dynamically brokered VPN-tunneled connections to geographically proximate nodes.

We tested it, made it incredibly robust, and moved on. It ties back to a core engineering ethos: build the foundation so solidly that you don't waste time diagnosing trivial session-layer disconnects later. By building extreme fault-tolerance at the physical and link layers, upper-layer debugging actually became easier. We designed the system from the ground up to be tolerant of errors, whether from the environment or inside the code.

Step by step: Data, AI, and Infrastructure

With the networking layer stable, we had to tackle data durability, sensor fusion, and deployments. We broke it down into three phases:

1. Data Durability

When the network drops, data cannot simply vanish into a failed queue. We split the job in two:

• Live Telemetry: We used Redis Pub/Sub for real-time queueing and at-least-once message delivery for actively connected clients.
• Physical Edge Caching: True persistence lived on the device. Each sensor node maintained a localized copy of its raw telemetry. If a node went dark, it cached everything. Upon reconnection, or if we physically retrieved it for a batch download, the pipeline seamlessly ingested the backlog. No data was lost.

2. Pragmatic AI

Our core objective was generating a "Common Operating Picture" (COP) by fusing Electro-Optical (EO) and Infrared (IR) imagery with NASA SAR data. We used templated language generation to translate complex, multi-modal tensor data into plain-language text strings for rapid human analysis.

Then came the language decision. We made a hard architectural call: stick with Python. We explicitly avoided C++ to prevent team bottlenecks (you try finding a specialized C++ edge dev on short notice when it's mainly you building the core).

More importantly, we ruled out JavaScript. In 2019, WebGL backends in libraries like TF.js were quantizing weights differently than native runtimes, causing silent precision drift where the exact same inference would yield two different detections. Standardizing on native Python ensured the inference model behaved identically across every single device. We also bypassed workflow tools like Airflow and Node-RED; while powerful, their footprint is simply too massive for resource-constrained edge devices.

3. Infrastructure via Bash

Today, the industry defaults to Infrastructure as Code (IaC) like Terraform for everything. Back then, we relied on good, old-fashioned bash and batch scripts.

We used these scripts to download images, configure the code, and provision cloud-side coordinating nodes on DigitalOcean Droplets to handle the heavy aggregation. The edge devices (a mix of BYOD/BYOC) were bootstrapped via these scripts to automatically point back to the coordinators.

The end result was a highly composable machine learning pipeline. The system automated its own spin-up and teardown, and laid the groundwork for what a robust, drag-and-drop data processing system looks like when properly paired with modern IaC. We even maintained a decentralized UI layer: every edge node hosted a copy of the frontend, allowing engineers to physically connect to a device in the field and reconfigure the ML pipeline locally.

A moving target

This project was a crucible that made future software architecture challenges pale in comparison. Building for the edge forces you to respect the physical constraints of your hardware and the hostility of your network.

But even with this optimized architecture, the edge is always a moving target. We eventually realized we needed an adapter strategy to handle changes in pipelines across edge, processing, and central nodes... but that's a story for another post.