A small IoT budget is dangerous because it tempts a tech lead to save money in the one place that compounds: fleet control. My position is blunt: a 15-30 person engineering organisation should buy or adopt the boring connectivity layer early, then economise on dashboards, analytics, and custom workflow code.
The cheap plan should buy time, not create a second product
The usual “small budget” move is to run Mosquitto, store device rows in PostgreSQL, and let a worker process send commands, but that becomes a second product because every retry, certificate rotation, and firmware exception needs product-grade behaviour. I would challenge one growth-oriented reference, IoT Connectivity and Device Management for Scalable Apps, because scalable application thinking can make a young team optimise the cloud side before it has proved device discipline in the field.
Cheap is acceptable when it is deliberately temporary. A Dockerised Mosquitto 2.0.18 broker, MQTT 3.1.1 or MQTT 5.0 topics, JSON payloads, and Prometheus black-box checks are fine for a lab or a 30-device partner trial because the blast radius is human-manageable. Cheap becomes reckless when the same stack holds customer devices, because a broker outage then becomes a field-service event rather than a backend incident.
For the first shipped slice, I would set a planning assumption of 500 devices over the next two quarters, not because 500 is magic, but because it is large enough to expose certificate, reconnection, and support problems while still being small enough for one tech lead to reason about. I would tune MQTT keep_alive around 60 seconds on mains-powered devices, because shorter intervals waste radio and broker capacity while longer intervals delay dead-connection detection. On constrained networks, I would tune reconnect backoff between 5 and 120 seconds, because synchronized reconnect storms are more damaging than slow recovery.
This is the kind of cheap experiment I would allow, and only as a contract test for topics and payloads:
docker network create iot-demo || true
docker run -d --rm --name broker --network iot-demo eclipse-mosquitto:2.0.18 mosquitto -c /mosquitto-no-auth.conf
docker run --rm --network iot-demo eclipse-mosquitto:2.0.18 mosquitto_sub -h broker -t 'devices/+/state' -C 1 &
sleep 1
docker run --rm --network iot-demo eclipse-mosquitto:2.0.18 mosquitto_pub -h broker -t 'devices/dev-001/state' -m '{"fw":"1.4.2","rssi":-67}'
docker stop broker
docker network rm iot-demo
I would not ship the above pattern to production, because anonymous access, no TLS, no retained device identity, and no command acknowledgement turn a harmless demo into an incident generator. The useful lesson is narrower: a tech lead can validate topic shape, payload size, and client behaviour before arguing about platforms.
Self-hosting wins only after the load is boring
The explicit comparison is AWS IoT Core versus EMQX 5.x self-hosted on Kubernetes. AWS IoT Core wins when the team has uncertain device growth, weak operations coverage, or strict X.509 provisioning needs, because the service includes managed MQTT endpoints, policies, fleet indexing, and certificate lifecycle hooks. Its cost is per-message and per-connection pricing, plus architectural lock-in to AWS rules, IoT Jobs, and IAM; a vendor-published reference price in us-east-1 has often listed MQTT messaging around $1 per million 5 KB messages, so chatty devices punish sloppy payload design.
EMQX 5.x self-hosted wins when traffic is steady, payloads are high-volume, and the team already runs Kubernetes well, because the marginal message cost can be lower once brokers, disks, networking, and operations are amortised. Its cost is not “free”; EMQX Open Source removes licence spend, but you still pay for nodes, load balancers, persistent volumes, backups, alerting, upgrades, and someone who understands MQTT session behaviour at 02:00. If you need enterprise clustering, audit controls, or vendor support, EMQX Enterprise changes the licence equation, so the open-source spreadsheet can be misleading.
- AWS IoT Core: choose it when you need managed X.509 identity, policy enforcement, and fleet jobs before you need broker customisation; it costs usage fees, AWS coupling, and some topic-rule complexity.
- EMQX 5.x on Kubernetes: choose it when message volume is predictable and your platform team already owns ingress, TLS, metrics, and rolling upgrades; it costs operational attention, capacity planning, and failure testing.
Mosquitto is still useful, but I would keep it in CI, labs, edge gateways, or very small private deployments because it is simple and well understood. I would not make Mosquitto the central production control plane for a commercial fleet unless the business accepts limited management features, because adding registry, shadow state, OTA orchestration, and audit trails beside it recreates the managed platform in fragments.
Azure IoT Hub deserves the same kind of assessment as AWS IoT Core, because its device twins and DPS provisioning are strong if the rest of the company is already in Azure. Eclipse Ditto 3.x can model device state well, but it is rarely the cheapest first move for a 20-person engineering group because operating another distributed system is more expensive than admitting that device shadowing is a managed-service feature.
Proper starts with identity and update semantics, not dashboards
Proper IoT connectivity begins with a device identity that survives manufacturing, replacement, and support mistakes. X.509 client certificates over TLS 1.3 are boring for a reason: they give you revocation, mutual authentication, and policy binding without trusting a shared password that will eventually leak. Pre-shared keys can be defensible for a closed appliance fleet, but they are a poor default because one leaked key can flatten the trust boundary across many devices.
A smart-system reference, IoT Connectivity and Device Management for Smart Systems, rightly pushes the conversation toward coordinated devices, but I would still put rollback and revocation ahead of orchestration because a coordinated outage is worse than a dumb device missing one feature. LwM2M 1.2 is worth considering for constrained device management because it standardises bootstrap, objects, firmware update, and telemetry over CoAP RFC 7252; MQTT is worth keeping when application events dominate, because its pub/sub model is simpler for backend developers and cloud integrations.
Firmware update semantics should be decided before the first field pilot. Mender, Eclipse hawkBit, SWUpdate, RAUC, and Uptane/TUF solve different parts of the problem, and the wrong shortcut becomes expensive because failed updates require truck rolls, RMAs, or support scripts. I would require A/B partitions or an equivalent rollback mechanism on any device that cannot be physically recovered cheaply, because “retry the update” is not a recovery strategy when the network stack is broken.
A practical target to tune is 99% successful OTA completion within 30 minutes for online devices in a pilot cohort, because that threshold is high enough to expose update design flaws without pretending hostile networks behave like CI. Another value to tune is an MQTT command acknowledgement deadline of 10 seconds at p95 for interactive actions, because users perceive command uncertainty before they care about the exact transport protocol. If your measured p95 reconnect after an access-point reboot is 3.2 seconds in the lab, record the firmware, broker, Wi-Fi chipset, and RSSI conditions, because a number without test context becomes folklore.
Zephyr 3.7, FreeRTOS, mbedTLS, wolfSSL, and OpenSSL 3.x all influence the connectivity budget more than a dashboard library does, because RAM, flash, cipher support, and certificate storage dictate what can be secured in production. On the cloud side, configure explicit MQTT 5.0 behaviour such as clean_start=false, session_expiry_interval, QoS 1 for commands that must arrive, and idempotency keys in payloads, because “exactly once” language rarely saves you from duplicated device-side effects.
A small fleet still deserves production telemetry
Telemetry for the connectivity layer should be small, opinionated, and boring. I would track device reconnect rate, p95 command latency, OTA success rate, certificate-expiry horizon, broker publish errors, MQTT QoS 1 retry count, payload bytes per device per day, and last-seen age because those metrics map directly to incidents a small engineering team can act on. I would not start with a general analytics lake, because most early fleets need operational truth more than exploratory reporting.
Prometheus, Grafana, OpenTelemetry, Loki, and Alertmanager are enough for many teams because they give metrics, logs, traces, and paging without forcing a data-platform project. Datadog or New Relic can be better on a small team if nobody wants to maintain observability plumbing, because the subscription may cost less than an engineer debugging Prometheus retention during a customer incident. The budget question is not open source versus paid; it is whether the tool reduces the number of unknowns during a device failure.
Set alerts on symptoms rather than components. “Broker CPU above 80%” is weak by itself because CPU can spike harmlessly during retained-message churn; “command acknowledgement p95 above 10 seconds for 15 minutes” is stronger because it describes user-visible degradation. A measured monthly support baseline, such as 12 device-connectivity tickets before a firmware change and 4 after, is more persuasive than a clean architecture diagram because it links engineering choices to operational load.
Payload discipline is also a budget control. MQTT topics such as devices/{deviceId}/state, devices/{deviceId}/events, and devices/{deviceId}/cmd/{commandId}/ack are boring but debuggable, while clever topic hierarchies leak product assumptions into infrastructure. Use Protocol Buffers 3 or CBOR when payload size matters, because a few hundred unnecessary bytes per minute becomes real cloud spend across a fleet. Keep JSON early if debugging speed matters more than bandwidth, because a 20-person team can waste more money decoding binary payloads during support than it saves on messages.
Security metrics belong here too. Track certificates expiring in fewer than 45 days as an alert threshold to tune, because emergency certificate rotation steals the same engineers who should be building product. Use NIST SP 800-193 as a resilience reference if firmware integrity matters, because it frames protection, detection, and recovery rather than treating secure boot as a checkbox. Use SBOM tooling such as Syft and vulnerability scanners such as Grype or Trivy, because embedded dependency drift is quiet until a customer asks for evidence.
Start by killing one risky shortcut this week
Pick one production shortcut and remove it before choosing another platform. If devices share credentials, create a per-device identity path. If updates lack rollback, stop adding features until recovery is real. If nobody can name p95 command latency, add that metric. A small budget can survive a plain interface; it cannot survive a fleet you cannot revoke, update, or observe.
