A backend developer can make an IoT pilot look finished by accepting a few device messages into an API. The harder test is whether a device can lose its connection, retry safely, and show up as healthy again without someone repairing the database by hand. I would build that recovery path before choosing a broker or ordering more hardware, because it exposes the assumptions that usually make a pilot misleading.
One simulated device can expose the wrong architecture before hardware arrives
Start with a device contract, not a cloud diagram. Define a device identity, a monotonically increasing sequence number, a sample value, and the time the sample was recorded. A backend cannot distinguish a delayed reading from a current one without a device timestamp, so include it in the eventual contract even if the first local endpoint stores only identity, sequence, and value. Write down who assigns each identity and what happens if a device is replaced; otherwise a replacement may appear to be an inexplicable gap in the old device’s history.
For the first exercise, let a laptop script stand in for a device. Set a trial rate of one sample every 60 seconds; that interval is a value to tune to the signal being sampled, not a rule of IoT design. Give the simulator one identity and make it persist its next sequence number to a local file. Restart it halfway through a run. If it starts numbering at zero again, fix that behavior before connecting a physical sensor, because the server cannot reliably identify retries when the device forgets what it sent.
Make the device’s send queue persistent too. At the trial rate, a 24-hour outage produces 1,440 unsent samples per device. That is a capacity calculation, not a measured storage requirement: timestamps, serialization, and queue metadata add bytes. Put a hard size limit on the queue and specify whether a full queue drops the oldest samples or stops sampling. Either choice can be defensible, but an unspecified choice becomes data loss that nobody can explain.
The threshold thesis (Cheap IoT rollouts break at 20 devices – what to fund instead) is a useful budget prompt, but I would test recovery before funding a larger fleet because a single device can reveal an unbounded queue or ambiguous identity. I would not buy radios to validate the ingestion API: a local simulator makes disconnections and repeats easier to reproduce, while hardware adds failure modes before the server contract is stable.
A small HTTPS endpoint is enough to prove ingestion, but not field security
Use HTTPS POST as the first transport unless a device constraint rules it out. It maps onto familiar backend behavior: a request either receives an acknowledgement or must be retried. Run the following local example with Python 3.11, FastAPI 0.115.12, Uvicorn 0.34.2, and Pydantic 2.11.5 pinned for a reproducible trial. Install the three packages, save the snippet as app.py, set DEVICE_TOKEN to a nonempty local secret, and start Uvicorn on 127.0.0.1. Send POST /events with JSON containing device, seq, and value, plus an X-Device-Token header.
import os, sqlite3, secrets
from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel
app = FastAPI()
class Event(BaseModel):
device: str
seq: int
value: float
@app.post("/events")
def ingest(e: Event, x_device_token: str = Header(default="")):
if not secrets.compare_digest(x_device_token, os.environ["DEVICE_TOKEN"]):
raise HTTPException(401)
with sqlite3.connect("events.db", timeout=5) as db:
db.execute("CREATE TABLE IF NOT EXISTS events (device TEXT, seq INTEGER, value REAL, PRIMARY KEY(device, seq))")
db.execute("INSERT OR IGNORE INTO events VALUES (?,?,?)", (e.device, e.seq, e.value))
return {"accepted": True}
The primary key makes a repeated POST harmless to the stored event count, so the simulator can resend the same sequence number after a lost response. A successful response here means the row was committed by SQLite; it does not prove that a downstream consumer processed the sample. The code uses a shared token and trusts the device field, so keep it on loopback. Before exposing an endpoint, put TLS 1.3 at a reverse proxy such as Nginx, issue a distinct credential per device, and bind that credential to its permitted identity; otherwise anyone holding the shared token can submit data under any name.
This is also the point to compare HTTPS POST with MQTT 3.1.1 explicitly. HTTPS wins for a first setup when devices can initiate ordinary web requests and your team already operates an API; its cost is a request and connection overhead for each batch, plus client-side queue and retry code. MQTT wins when long-lived subscriptions or constrained links make its session model valuable; its cost is a broker such as Eclipse Mosquitto, topic authorization, session-state decisions, and another service to operate. MQTT QoS 1 permits duplicate delivery, so it does not remove the need for the sequence-number rule. I would not select MQTT merely because it is associated with IoT; that association does not settle which failure behavior this deployment needs.
An offline replay should decide whether the pilot works
Now make the simulator perform a failure drill. Send a few samples, stop the API, keep generating numbered samples locally, restart the API, and replay the queue. Treat a timeout as an unknown outcome rather than a rejection: the server may have committed the row before the response disappeared. Resend the same sequence number until it is acknowledged, and delete a queued sample only after receiving that acknowledgement. Changing the sequence number on retry would turn one reading into two apparently valid readings.
For the trial, set the client’s retry delay to start at 2 seconds and cap it at 60 seconds, with jitter so reconnecting devices do not all retry together. These are initial settings to tune against battery use and acceptable delay. Put a timeout on each HTTP request as well; without one, a stalled connection can prevent the queue from advancing even after the server is healthy. Record the outage start, the first successful replay, and the time the backlog reaches zero. Those observations are more informative than a dashboard that merely reports the API as up.
Choose an acceptance target before running the drill: for example, p95 delivery lag below 120 seconds during normal operation, with backlog drained after the simulated outage. This is a proposed service target, not an observed result. Measure lag from the device’s recorded time to the server’s receipt time, but first check clock offset against an NTP-synchronized reference; an inaccurate device clock makes that metric misleading. Also record duplicate POSTs, accepted rows, queue depth, and the oldest queued sample’s age. The counts should reconcile: retries may exceed stored rows, but accepted unique sequences should not silently disappear.
The roadmap-cost warning (IoT Connectivity Mistakes That Blow Up Your Roadmap Cost) is worth challenging with this drill, because a replay trace shows which costs are real for this device contract and which are still guesses. Test a full queue and a credential rejection separately. A full queue is a data-retention decision; an HTTP 401 is a provisioning problem, and retrying it forever only burns power. Write both outcomes into the simulator’s log so an operator can tell them apart without reading server code.
A working pilot needs a replacement and restore test, not just live traffic
Once replay works, replace the simulated device identity as though a physical unit had failed. Decide whether the replacement continues the old sequence or receives a new identity; I favor a new identity because it preserves an unambiguous history of which unit produced each sample. Store the relationship between old unit, new unit, and installed location separately from the event primary key. Then revoke the old credential and verify that it can no longer submit events. This test catches a common gap between an application’s device field and its actual authentication rules.
Next, stop the API and copy the SQLite database using SQLite’s backup mechanism rather than assuming that copying an open database file is safe. Restore it into a fresh local directory, start the service there, and replay the last acknowledged sample. The row count should stay unchanged because the primary key rejects the duplicate. SQLite is appropriate for this single-process trial because it removes an external database from the first recovery test; move to PostgreSQL when concurrent writers, operational backups, or query load require a database service. That move has a cost in deployment and administration, but it should not change the device’s retry contract.
Keep a short record of what the trial actually measured: device count, simulated outage duration, maximum queue depth, duplicate requests, lost unique sequences, and time to drain. Do not present the proposed lag target as a result. Record the versions and configuration used as well, because a later change to the client queue or proxy timeout can alter replay behavior without changing the API schema. A pilot is working when another developer can reproduce the failure drill and explain every missing or repeated sequence.
The first real task is to write a sequence number to disk
Before creating a cloud account or ordering a device, write the simulator’s persistent sequence counter and a queue that survives restart. Point it at the local endpoint, interrupt the connection, and check whether every unique sequence eventually appears once in SQLite. That small test gives you a concrete device contract to take into hardware selection—and a failure you can reproduce when the first field unit behaves differently.
